In a simulated test, most of seven robot models built on vision-language AI could not act on commonsense and world knowledge that the original vision-language models handle well. The finding comes from a preprint, not yet peer reviewed, by Nikita Kachaev, Alexey K. Kovalev and colleagues. The models answered two-choice quiz questions by placing a cube on the picture they chose. They reliably picked the right colour but mostly performed at coin-flip level on richer questions, such as those about emotions or time. The authors read this as a cost of the standard way robot models are built.

The standard approach produces vision-language-action models, or VLAs, which take camera images and a text instruction and output motor commands. They are typically made by fine-tuning a vision-language model (VLM), which answers questions about images in text, on robot data. Models covered earlier in this series, such as Ai2's MolmoAct2, likewise build on a vision-language model. The authors argue that robot benchmarks usually report only whether a whole task succeeded. That single figure mixes perception, knowledge, motor control and the environment, so a failure is hard to pin on missing knowledge. Making a robot model's VLM component answer in text instead shows whether knowledge can still be put into words, but not whether it can guide an action.

The authors' test, Act2Answer, borrows questions from established VLM benchmarks and sorts them into 12 categories, ranging from colour, shape, counting and symmetry to emotion, traffic and celebrities. Each question becomes a short episode in a simulated tabletop environment built on Simpler. Two candidate images lie on the table, the question is rephrased as the instruction, and the robot must put a cube on the image it chooses. An episode counts as correct if the cube stops within a set distance of the right image's centre. Each of the 1,720 items is also run with the answers swapped left and right, giving 3,440 episodes, so a robot that always reaches one way gains nothing. Chance is 50%. The VLAs, including π0, OpenVLA, Magma and SmolVLA, were tested as released, without task-specific fine-tuning.

On colour the VLAs scored 86–91%. On shape they ranged from 83% for SpatialVLA down to 49% for π0. On the ten richer categories most hovered around chance: OpenVLA scored 45–52% on every one, and π0 scored 38–51%. Magma was the clear exception, reaching 94% on celebrities, 88% on public information and 80% on traffic, though it managed only 37% on symmetry and 51% on counting. Nine VLMs were shown the scene's first frame and asked in text whether a named answer board was visible and where it was. The authors report that across most domains the original VLMs outscored the VLAs built on them by roughly 20 to 40 points. They treat this as a rough upper-bound estimate.

To see where the knowledge goes, the authors used linear probes. These are very simple classifiers trained on a model's internal activity at one layer, to test whether the correct answer can be read out there. Across four categories, the answer was often readable above chance in the middle layers of the vision-language backbone. It faded towards the final layers, often to near random. How much reached the action-generating part varied. Magma's best probe accuracy slipped only from 75.2% in the backbone to 72.6% in the action part, while π0's fell from 65.0% to 55.4%. Three VLAs that were also trained on image question-answering data did better overall than three trained mostly on robot data, though the authors present this as an association with uneven gains. Further fine-tuning OpenVLA on a small pick-and-place dataset did not help. Supervised training on example demonstrations cut its colour score from 89% to 82% and its shape score from 64% to 53%. Reinforcement learning left every category within three points of the original.

Much remains unsettled. The authors say the protocol reduces the influence of motor control but does not fully isolate knowledge. Many results sit between 45% and 58%, where the verdict rests on a statistical margin around chance. The setup is narrow: one tabletop, one cube-placing action and 12 categories standing in for seven broad domains. The co-training comparison rests on three models against three. The fine-tuning result rests on one model, and the probing covers only four categories. The results indicate that these robot models, as released, mostly could not act on such distinctions in this setting. Why remains an open question.