World models take the wheel

ECCV 2026: Notes on physical AI — robotics, autonomous driving and the foundation models underneath both.

Generative video and world models led ECCV 2026 — as simulators, evaluators, and candidate policies for embodied AI.

Computer vision is turning its focus to physical AI. At ECCV 2026 all three keynotes were about models that act in the physical world rather than just look at it: Grauman on 4D egocentric understanding, LeCun on action-conditioned world models for planning, Shotton on learning over engineering in a decade of driving. The idea underneath much of the discussion was models that predict what happens next and, when action-conditioned, support planning and control — not just static perception. At least six workshops were built around world models, and similar video and VLM backbones kept showing up in robot manipulation, egocentric video, and autonomous driving.

My own lens is driving, which Jamie Shotton called the first proving ground for embodied AI, but most of what follows transfers to robotics. Here are eleven takeaways, grouped into four areas.

The program at a glance

10,473 submissions · 2,883 accepted (27.5%) · 86 workshops

The building blocks of physical AI dominate the program: image and video generation plus vision-language models account for roughly two in five papers, and 3D is the largest single subfield and every main-track award went to 3D or geometry work. Robotics and embodied AI (~6.5%) and driving (~6%) are small by paper count, but they had all the keynotes and many of the largest workshops.

Ten largest subfields — share of 1,131 categorized papers.

Highlighted: embodied AI and driving subfields.

Shares from a third-party categorization of 1,131 of the 2,883 accepted papers (papernotes).

First, what's a world model?

The term world model gets slapped on many different things, and the conversation gets muddled when they’re treated as the same thing. One clean taxonomy to tell different models apart is the input-output structure. The same split applies whether the agent is a robot arm or a car.

Three stacks in physical AI — video generator, action-conditioned world model, and world-action model — told apart by where the action sits: absent, an input, or an output. A typical VLA maps image and text to actions while a WAM often maps image and text to future images and actions.

Most of the pixel-generating models in all three columns share a latent diffusion/flow transformer backbone, and unified foundation models like Cosmos 3 now support each mode. The left column is generative video in the usual sense — pretraining and synthetic data — not an agent’s transition model until actions are wired in. One real architectural split among the action-aware stacks is generative versus joint-embedding (JEPA-style) models, which predict latent states instead of pixels.

ECCV also debated policies built on video-centric generative pretraining (WAM) versus VLM-centric pretraining (VLA) versus hybrids such as StructVLA. Both show up in manipulation, driving, and navigation — as onboard policies, simulators, or teachers — and head-to-head comparisons across domains are still rare. The eleven takeaways below cover policy, perception, simulation, and data engines. Driving is the running example, but most points apply to robotics.

Policy

01 · WAM vs VLA as policies

For robot policy applications ECCV had plenty of WAM and VLA-based works on display, but direct comparisons were rare and often unfairly matched. In robotics, a robustness study finds WAMs ahead on RoboTwin-Plus (74.2% vs 58.6% for π0.5) but behind on LIBERO-Plus (82.2% vs 85.7%), and NVIDIA reports DreamZero leading RoboArena. In driving, OmniDreams has a WAM beating VLA-based Alpamayo 1.5 at ~1/5 the compute. Indoor navigation in Habitat mostly repeated the same split — navigation world models versus instruction-following VLAs — while open-world outdoor nav stayed thin.

PRISM finds VLM supervision helps driving through the latents it shapes, not through reasoning, and Scaling Verification (Finn, Pavone) shows a test-time verifier beating the same data spent on more VLA pre-training. While VLMs and video generators are both large vision-language transformers their pretraining objectives are quite different (VQA vs video reconstruction). What happens after pretraining is also important: closed-loop RL or on-policy distillation instead of open-loop imitation, verifiers at test time, and evaluation in closed-loop sim or paired OOD benchmarks (Fail2Drive-style), not single-scenario leaderboards.

More to read: Sim-to-real in dexterous VLAs

02 · Closed-loop policy learning

Wayve, NVIDIA, Waymo, Zenseact and KE:SAI all gave the same diagnosis: behavior cloning learns predictions, not behaviors. Bernhard Jaeger’s talk laid out the two recipes that work: RL post-training on top of an imitation policy (what ships today), and a privileged RL teacher distilled on-policy into a sensor-based student (the research frontier). The same split is playing out in robot learning. NVIDIA’s survey of closed-loop training is the best overview.

LEAD traced the CARLA plateau to what the privileged expert sees that the student can’t: occluded actors, noiseless state, and a richer route. Removing those gaps gave +8 driving score on Bench2Drive. The same mismatch hurts any policy distilled from a privileged teacher in sim or on a robot: the student is trained to mimic actions while missing part of the state the teacher used. Perception has a parallel pattern when a stronger view supervises a weaker one — for example +44% long-range BEV from an overhead teacher to a street-level model.

Whatever the teacher sees that the student can't (occluded actors, clean state, a richer route) ends up in the target the student is asked to reproduce.

More to read: BeyondDrive · CaRL · UniDrive-WM · WTA→GMM forecasting · SparseDriveV2

03 · Self-play overfits

Self-play can scale fast — TerraZero runs 1.3M agent-steps/s and leads the InterPlan long-tail benchmark with no human demonstrations — but BehaviorBench shows pure self-play policies overfit to their own traffic and crash more against other agent models. Evaluate against a panel of behaviors, not one.

More to read: ZTRS · Scaling Self-Play · CENTAUR

Perception

04 · Geometry-first perception backbones

Two kinds of pretrained model are becoming perception backbones: feed-forward 3D reconstruction (point maps, splats) and weights transferred from generative video.

Reconstruction models. Map-Det3D detects objects directly in the metric 3D space that a feed-forward reconstructor builds from video. MV2GF replaces ground-plane projection with Depth Anything 3 pointmaps — +7.9 MODA, and with only two cameras 57.6 vs 34.0. GeoUP adapts a VGGT-style latent to calibrated streaming multi-camera driving for unified depth, detection, and occupancy. Hybrid scene memory follows the same pattern: feed-forward or recurrent reconstruction, then refine at test time (ReSplat, TTT3R).

Three reconstruction regimes, each with its own failure mode, and the hybrids that combine them: start feed-forward or recurrent, then refine at test time.

Video generators (perception transfer). GenCeption fine-tunes text-to-video diffusion into a single perception model, matching or beating Depth Anything 3 and SAM3 on depth, normals, pose, and segmentation with 7–500× less data. ICDepth and GeoNeXt do the same for video depth from 0.8M frames and 59K samples. These are not dynamics models in the fig02 sense; they reuse generative video weights for single-frame discriminative vision tasks.

Geometry world models (latent dynamics). VGGT-World autoregressively forecasts VGGT token trajectories, not pixels — a feedforward geometry backbone plus temporal flow matching (fig02’s action-conditioned stack in feature space). Manipulation video WMs. RoMan-4D is a flow-matching video generator whose transport is aligned to a geometry foundation model, then mapped to actions via inverse dynamics.

Both families care more about generalization and label efficiency than winning every public leaderboard, using supervision from SfM, synthetic renders, LiDAR logs, or raw video. Representations still differ — generative VAE latents, JEPA-style abstract states, explicit geometry (PointDiT on point maps); RAE-NWM predicts navigation rollouts in dense DINOv2 tokens (~79% on Habitat image-goal). For metric safety (collision checking, lane geometry), explicit 3D still looks more reliable than decoded pixels alone.

More to read: PhysMani · What if? (world models for spatial reasoning) · Flow4R · Thermo-JEPA

05 · Open-world perception

Generalizing to what wasn’t in the training set is the other big frontier. Fail2Drive pairs every test route with an unseen twin — new objects, layouts and behaviors — and SOTA driving policies lose 22.8% success on average, sometimes ignoring objects clearly visible in the LiDAR. 360CityArena is the navigation counterpart: real Akihabara panoramas in a photorealistic city benchmark, where Gemini 2.5 Flash reaches 17.1% next to 77.3% for humans. Most ECCV fixes used foundation models in straightforward ways: BEVOpen3D distills a 2D vision-language model into an open-world LiDAR detector, open-vocabulary BEV segmentation adds 3D geometric constraints, and NegAS cuts out-of-distribution false positives by 25% on OpenImages. TokenGraph3D shows a frozen self-supervised point encoder already separates object instances without any class labels.

More to read: real-time OOD detection · Is Single-View Mesh Reconstruction Ready for Robotics?

06 · Occlusion and object permanence

An agent has to keep track of what it can’t currently see. StEvo-Bench tests whether action-conditioned generative video models evolve the state of the world independently of what’s in view — insert an occluder or move the camera — and finds most of them let hidden objects disappear, a problem for any agent that plans with one. BeyondSight tackles the same principle in end-to-end driving: persistent actor hypotheses that survive occlusion, with nuScenes-Permanence for training and evaluation. Related driving work includes HC-MTS, which searches worst-case trajectories for hidden vehicles consistent with past visibility, and localizing out-of-sight pedestrians through radar and camera reflections.

Object permanence: a car hidden behind a truck still exists. Many generative video models drop it once it's out of view.

More to read: RePL

Simulation

07 · Manufacturing the long tail

Real logs contain almost none of the scenarios that matter, so they have to be manufactured. World Engine finds real failures, rebuilds them in 3DGS, generates harder variants and RL-post-trains on them: 45.5% fewer collisions, ~200 km on-road with zero disengagements. Adversarial generators are getting sharper too (CompoSIA raises planner collisions 173%).

The World Engine loop: find where the deployed model fails, rebuild those scenes, generate harder variants of them, and post-train on those variants.

More to read: ReinDriveGen

08 · Fidelity vs reactivity

Simulators moved from physics engines (CARLA and robotics equivalents) to neural reconstruction (3DGS twins like HUGSIM) to generative models that reach beyond collected data. Today world models are used mostly for policy evaluation, data generation, and RL teachers — not yet as on-robot MPC policies.

Existing simulator struggle to achieve high fidelity/realism while also being reactive, expressive, and controllable. Log replay and 3DGS twins are faithful but can’t react; box-world sims react but look nothing like sensors; generative models are expressive but drift. CausalDrive is the current bar on reactivity (12.4 FPS, traffic that yields when prompted), and hybrids — 3DGS backgrounds plus diffusion, or AlpaSim on reconstruction or generation — aim for both realistic sensors and closed-loop response (top-right of the schematic). A generator is only as controllable as what you condition on: PrITTI uses primitives instead of voxels, and Trajectory Forcing makes diffusion go layout → object → detail. Benchmarks follow the same arc: NAVSIM → NAVSIM v2 → AlpaSim.

No simulator has both yet: replay and 3DGS twins are faithful but can't react, box-world sims react but have no realism, and hybrids and new benchmarks are moving toward the corner (schematic).

More to read: SPEAR simulator · Waymo World Model · LiSTAR · Cosmos-Transfer1

09 · Generative sim train/serve gap

Generative simulators are trained on expert actions and clean context, then driven by a learner’s actions and their own outputs. Expert-only models (GenAD, Vista) break on non-expert actions; ReSim fixes that by mixing in simulated bad driving. Rollouts drift unless trained on imperfect histories (CausalDrive, DAgger, diffusion forcing). Robot sim rollouts from demonstration-trained world models hit the same mismatch: logs ground appearance, synthetic interaction supplies coverage.

Generative simulators are trained on expert actions and clean context but used with a learner's actions and their own outputs; both gaps have known fixes.

More to read: OmniDreams · VATIX scaling laws

Data engines

10 · Borrow experience from cheaper embodiments

Robot and vehicle data is expensive; broader internet video data and dashcams are not. The recurring trick is to align cheap video to the target sensor layout — body pose for egocentric transfer, calibration for multi-camera driving, and so on. Grauman’s EgoExo-WM trains egocentric world models from third-person video using body pose as the bridge. DexWM learns dexterous hand–object world models from human video and transfers zero-shot to a robot hand, beating Diffusion Policy by over 50%. In driving, Waymo’s Sensor2Sensor turns dashcam video into multi-camera + LiDAR logs for a target vehicle, and roadside sensors can act as teachers. Yunzhu Li made the general case: structured world models as scalable data engines for robotics.

More to read: One Demonstration Is Enough for Real-World Robotic RL

11 · Data engines for the long tail

Dragomir Anguelov’s line stuck with me: “yesterday’s long tail is today’s torso.” The tail moves as capability improves, so chasing failures only after deployment is a losing game. José Alvarez argued for test-driven development instead of reactive patching, and Fail2Drive (same benchmark as in open-world perception above) gives a concrete eval: paired routes the model has never seen (a 22.8% average success drop for SOTA).

Reactive: wait for a failure on the road, then patch. Proactive: list the scenario space, generate and test it in simulation, and fix failures before deployment.

TTCov makes the concept space explicit — an LLM-built atlas of concepts — and picks training data to match deployment frequency. That underweights rare, severe cases by design; keep the atlas, change the target to a floor per concept weighted by severity. Semantic-Drive helps fill it (0.966 vs 0.475 recall over CLIP for long-tail mining).

Every stage now has an automated tool — enumerate (TTCov), mine (Semantic-Drive), generate (CompoSIA, ReinDriveGen), auto-label (TokenGraph3D, RePL), train (World Engine) and validate (Fail2Drive, AlpaSim). Less clear is a shared taxonomy of rare scenarios that every group could use to drive the pipeline — in driving and, even more, in open-world robotics.

More to read: 123D · the common task framework argument · CooperScene

What I'm watching next

• Controlled ablations on whether video pretraining or language pretraining helps more on the same downstream robot or driving task.
• Policies (VLA or WAM) that fuse multi-camera video with depth, LiDAR, radar, or force at control rates, not just RGB.
• More real closed-loop scoreboards like RoboArena and the AlpaSim closed-loop challenge (closing October 31).
• A shared, public taxonomy of rare driving and robotics scenarios so data-engine pipelines can target the same list.

If you only read three things: CausalDrive, World Engine and Do World Action Models Generalize Better than VLAs?.

Workshops worth following: DriveX, World Models in the Loop, How to Build Effective World Models for Embodied AI, Safe World Models. Numbers are as reported by the authors; closed-loop scores depend on the simulator and are not comparable across benchmarks.