Chief Executive Officer Toshiro Muto sounded a grave note speaking at a meeting with officials of the International Paralympic Committee.
"I am seriously worried that the spread of the infectious disease could throw cold water on the momentum toward the games," Muto said, speaking in Japanese. "I hope that it will be stamped out as soon as possible."
Saburo Kawabuchi, the mayor of the Athletes Village where 11,000 Olympians will stay, showed his apprehension.
"I truly hope that the infectious disease will die down somehow so that we will be able to operate the Paralympics and Olympics smoothly," he said. "In the worst case — we will do our utmost for the athletes so that they will be able to concentrate on performing their best."
Tokyo organizers have repeatedly said there are no plans to cancel the Olympics. That position has been repeated by the Switzerland-based International Olympic Committee.
But the problems deepen with each passing day. Some Olympic qualifying events have been cancelled, or relocated. Travel restrictions will add more confusion, and traveling fans are sure to be fearful.
Japan has not reported any deaths from the virus. But deaths in mainland China rose to 490 on Wednesday
"In Japan, we are facing all sorts of problems including coronavirus infections, cyber security and transportation systems," Toshiaki Endo, an organizing committee vice president, said on Tuesday at a news conference.
"The IOC is satisfied with our preparations," he added.
Craig Spence, a spokesman for the Paralympic committee, tried to be reassuring.
"We have full confidence that the relevant authorities, in particular here in Japan and the World Health Organization, will take all the necessary measures to address the situation."
Japanese Prime Minister Shino Abe was asked Monday in the national legislature about the viral outbreak and the impact on the Olympics. He brushed aside worries.
But Yurkio Koike, the government of Tokyo, sounded more concerned in recent comments.
"We must firmly tackle the new coronavirus to contain it, or we are going to regret it," she said.