VT-WM vs Cosmos3: the same windows, the same decoder, the same metrics
Every number and every frame below comes from the evaluation protocol that produced the
Cosmos3 tables: 16-frame windows, 2 history + 2 predicted latents at 15 fps, metrics on
the 8 predicted frames, ground truth read from the cached latents. The case list is theirs.
Read this before the numbers
The two metrics disagree on the camera streams and agree on tactile.
That is the distortion–perception tradeoff: squared error is minimised by the conditional
mean, i.e. by blur, and blur is cheap in PSNR and expensive in LPIPS.
Aggregate — short horizon
One window, teacher-forced. All of the published cases per task.
Aggregate — long horizon
15 chained windows = 120 frames = 8 s, pixel-space closed loop
(decode → re-encode → feed back). Cosmos3 is re-aggregated on exactly the cases VT-WM
ran rather than quoted, because the VT-WM long runs use a subset.
What the metric is measured against
The protocol scores decoded prediction against decoded ground-truth
latent, so the codec's own error sits on both sides and cancels. This is how far that
target is from the raw camera frames.
Per-case rollouts
Five streams across, three rows down: ground truth, VT-WM, Ours.
All three decoded by the same VAE, so sharpness is comparable down a column. Frames
0–7 are the given history; 8–15 are predicted, and the label strip says which.
What the 20→30 epoch extension bought
VT-WM at three points in its own training, same windows and the
same decoder as above. Rows: ground truth, epoch 10, epoch 20, epoch 30;
streams across. The run past epoch 20 happened because action_gain was
still climbing there — this is what it did to the pixels.