An Empirical Study of Robot Policies Trained on UMI-Style Data

After building a working path from UMI-style data to robot policies in LeRobot, we reached the practical question that follows every successful training run: which model should we actually put on the robot? UMI provides a way to learn robot behavior from handheld human demonstrations [1], while LeRobot gives us a common framework for datasets, policies, and deployment [2]. That makes it easy to train many policies—but not necessarily easy to choose among them.

We found surprisingly few public, systematic empirical studies describing this selection process for a real project. Papers introduce strong architectures, and tutorials show how to train them, but the engineering experience between a finished checkpoint and a deployment decision is shared less often: which metrics were useful, which apparent improvements survived a common evaluation, and which offline behaviors later mattered for physical motion.

That is the reason for this study and this post. We are sharing the evidence that guided model selection in our strawberry-picking project, including results that changed our mind and metrics that proved more useful than training loss.

This article is an empirical study of one dataset, one task family, and our implementation choices. It is not intended as a general ranking of robot-policy architectures. Results that hold for our strawberry-picking demonstrations may change with another task, embodiment, camera layout, dataset size, or controller.

We tested several ideas on the same SROI V2 strawberry-picking dataset: 1,459 handheld UMI demonstrations for training and 100 held-out episodes for evaluation. This post focuses on Action Chunking with Transformers (ACT) [3], SmolVLA [4], π0.5 [5], and the official OpenPI implementation [6], together with controlled ACT input and capacity ablations. The complete experiment record contains more models and diagnostics than we cover here.

Rather than reproduce the chronological research report, this post is organized around four standalone model-selection questions:

1. What metrics are useful for selecting a policy for physical deployment?
2. How did the policy families and training settings compare under one protocol?
3. Did increasing ACT capacity help, and which part of ACT benefited?
4. What temporal context did ACT need, especially from proprioceptive state history?

The report's generative-objective investigation is not discussed here.

We answer each from the comparisons whose evidence was clean enough to interpret. Experimental branches with unfinished runs, weak controls, or unclear conclusions are omitted.

All results below are open-loop evaluations of predicted action chunks. They are useful for rejecting bad designs and selecting robot candidates, but they do not establish closed-loop task success.

Experimental setup

The dataset runs at 30 Hz. Each observation contains the current wrist-view camera image and a relative end-effector state derived from the handheld trajectory. The policy predicts a 30-step UMI-style relative end-effector action chunk.

The cross-family comparison evaluates the complete chunk, covering about one second. The input ablations and physical-dynamics study also inspect the first ten actions to isolate near-term prediction behavior. Whenever a result comes from that shorter window, it is stated explicitly rather than compared directly with a full-chunk number.

The canonical evaluation uses 500 fixed queries from 100 episodes. We aggregate by episode and report 95% episode-bootstrap intervals where they are important. This captures variation across validation episodes, but it does not replace training multiple seeds.

Training loss is not used for cross-family ranking. Different policies optimize different objectives, and a lower training loss does not necessarily mean a better decoded end-effector trajectory. Instead, every prediction is converted into the same physical representation before evaluation.

Q1: What metrics are useful for selecting a deployable policy?

Our answer is that no single offline metric is sufficient. We use three layers together:

1. decoded trajectory accuracy, including endpoint, chunk-wide, and thresholded component metrics;
2. within-prediction motion dynamics, especially acceleration and jerk;
3. adjacent-frame prediction change, which measures how much independently generated chunks move when the policy is queried again one frame later.

The first layer tells us whether the prediction follows the demonstration. The second and third tell us whether the resulting stream of predictions is physically plausible to execute. Training loss is useful for debugging one model family, but not for choosing across families with different objectives.

Let the decoded prediction at step $kk$ be $({\widehat{\mathbf{x}}}_k,{\widehat{\mathbf{R}}}_k,{\widehat{g}}_k)(\backslash hat\{\backslash mathbf\{x\}\}\_k,\; \backslash hat\{\backslash mathbf\{R\}\}\_k,\; \backslash hat\{g\}\_k)$ and the demonstrated target be $({\mathbf{x}}_k,{\mathbf{R}}_k,g_k)(\backslash mathbf\{x\}\_k,\; \backslash mathbf\{R\}\_k,\; g\_k)$. Here $\mathbf{x}\in {\mathbb{R}}^3\backslash mathbf\{x\}\backslash in\backslash mathbb\{R\}^3$ is translation, $\mathbf{R}\in SO(3)\backslash mathbf\{R\}\backslash in\; SO(3)$ is orientation, and $gg$ is normalized gripper opening. All policies are decoded to this common representation before scoring, including policies that internally predict rot6d.

XYZ endpoint error

XYZ endpoint error is the Euclidean distance between the predicted and demonstrated translation at the final evaluated step $HH$:

$$e_{\mathrm{x}\mathrm{y}\mathrm{z},\mathrm{e}\mathrm{n}\mathrm{d}}={\parallel {\widehat{\mathbf{x}}}_H-{\mathbf{x}}_H\parallel }_2\mathrm{.}e\_\{\backslash mathrm\{xyz,end\}\}\; =\; \backslash left\backslash |\backslash hat\{\backslash mathbf\{x\}\}\_H-\backslash mathbf\{x\}\_H\backslash right\backslash |\_2.$$

It is reported in millimeters, and lower is better. This is the most intuitive measure of how far the predicted end effector finishes from the demonstrated target. It says nothing about the path taken before the endpoint, so it must not be used alone.

Rotation endpoint error

Rotation endpoint error is the geodesic angle between the predicted and target orientations:

$$e_{\mathrm{r}\mathrm{o}\mathrm{t},\mathrm{e}\mathrm{n}\mathrm{d}}={\cos }^{-1}\text{\hspace{0.17em}⁣}\left(\frac{\mathrm{tr}({\widehat{\mathbf{R}}}_H^{\mathsf{T}}{\mathbf{R}}_H)-1}{2}\right)\mathrm{.}e\_\{\backslash mathrm\{rot,end\}\}\; =\; \backslash cos^\{-1\}\backslash !\backslash left(\; \backslash frac\{\backslash operatorname\{tr\}(\backslash hat\{\backslash mathbf\{R\}\}\_H^\{\backslash mathsf\; T\}\backslash mathbf\{R\}\_H)-1\}\{2\}\; \backslash right).$$

It is reported in degrees, and lower is better. Unlike subtracting two axis-angle vectors, this measures the actual shortest angular distance on $SO(3)SO(3)$ and is independent of the policy's internal rotation representation.

Chunk-mean XYZ and rotation error

The endpoint metrics examine only the final action. Their chunk-mean counterparts average the same physical errors over all evaluated steps:

$$e_{\mathrm{x}\mathrm{y}\mathrm{z},\mathrm{c}\mathrm{h}\mathrm{u}\mathrm{n}\mathrm{k}}=\frac{1}{H}\sum _{k=1}^H{\parallel {\widehat{\mathbf{x}}}_k-{\mathbf{x}}_k\parallel }_2,e\_\{\backslash mathrm\{xyz,chunk\}\}\; =\backslash frac\{1\}\{H\}\backslash sum\_\{k=1\}^\{H\}\; \backslash left\backslash |\backslash hat\{\backslash mathbf\{x\}\}\_k-\backslash mathbf\{x\}\_k\backslash right\backslash |\_2,$$

with an analogous mean of the geodesic rotation error. These metrics answer whether the complete predicted path stays near the demonstration. A policy can have a good endpoint but a poor chunk mean if it takes an incorrect route and returns near the target at the end.

Per-component L1 error

XYZ L1 per dimension averages the absolute coordinate error over time and the three translation axes:

$${\mathrm{L}1}_{\mathrm{x}\mathrm{y}\mathrm{z}}=\frac{1}{3H}\sum _{k=1}^H\sum _{j=1}^3\mid {\widehat{x}}_{k,j}-x_{k,j}\mid \mathrm{.}\backslash mathrm\{L1\}\_\{\backslash mathrm\{xyz\}\}\; =\backslash frac\{1\}\{3H\}\backslash sum\_\{k=1\}^\{H\}\backslash sum\_\{j=1\}^\{3\}\; \backslash left|\backslash hat\{x\}\_\{k,j\}-x\_\{k,j\}\backslash right|.$$

It is reported in millimeters per dimension. Rotation L1 is computed similarly after decoding both orientations to three axis-angle components and converting the component differences to degrees. These metrics are close to the coordinate-wise regression errors optimized by many policies. Rotation-component L1 is not the same as physical geodesic error, so the geodesic endpoint metric remains the primary orientation measure.

Per-component MSE

MSE squares each coordinate error before averaging:

$${\mathrm{M}\mathrm{S}\mathrm{E}}_{\mathrm{x}\mathrm{y}\mathrm{z}}=\frac{1}{3H}\sum _{k=1}^H\sum _{j=1}^3({\widehat{x}}_{k,j}-x_{k,j}{)}^2\mathrm{.}\backslash mathrm\{MSE\}\_\{\backslash mathrm\{xyz\}\}\; =\backslash frac\{1\}\{3H\}\backslash sum\_\{k=1\}^\{H\}\backslash sum\_\{j=1\}^\{3\}\; (\backslash hat\{x\}\_\{k,j\}-x\_\{k,j\})^2.$$

XYZ MSE is reported in ${\mathrm{m}\mathrm{m}}^2\backslash mathrm\{mm\}^2$ per dimension, and rotation-vector MSE in ${\mathrm{d}\mathrm{e}\mathrm{g}}^2\backslash mathrm\{deg\}^2$ per dimension. Squaring makes this metric more sensitive to occasional large errors than L1. L1 and MSE are reported together because a similar L1 with a much larger MSE usually indicates a heavier error tail.

Acc@0.5

These thresholded accuracies compare different action dimensions on a common dataset-derived scale. For each decoded action coordinate $jj$, the scale is

$$s_j=\frac{q_{99,j}-q_{01,j}}{2},s\_j=\backslash frac\{q\_\{99,j\}-q\_\{01,j\}\}\{2\},$$

where $q_{01,j}q\_\{01,j\}$ and $q_{99,j}q\_\{99,j\}$ are computed from the pooled demonstrated evaluation chunks. A coordinate is counted as correct when

$$\frac{\mathrm{\mid }{\widehat{a}}_{k,j}-a_{k,j}\mathrm{\mid }}{s_j}\le \tau \mathrm{.}\backslash frac\{|\backslash hat\{a\}\_\{k,j\}-a\_\{k,j\}|\}\{s\_j\}\backslash leq\backslash tau.$$

Acc@\(\tau\) is the fraction of step-coordinate pairs satisfying that condition. The reported action accuracy pools the decoded XYZ, axis-angle rotation, and gripper coordinates; component-specific XYZ and rotation accuracies are also retained in the evaluation artifacts. Acc@0.5 is a coarse tolerance and often approaches saturation; Acc@0.1 is stricter and usually separates mature checkpoints more clearly. Higher is better. These values are specific to this dataset and evaluation set. They are not task-success rates, and they are not necessarily the same normalization used to train a policy.

Gripper error

Gripper error is the absolute difference $\mathrm{\mid }{\widehat{g}}_k-g_k\mathrm{\mid }|\backslash hat\; g\_k-g\_k|$ in normalized gripper opening. We record both the final-step error and the mean over the chunk. In our convention, the range is zero to one, so an error of 0.1 means ten percent of the calibrated opening range. Gripper error is kept separate from SE(3) pose error because it describes a different actuator and has different physical consequences.

Within-chunk second difference

The report's original “jerk” column is more precisely a second-difference motion-texture metric. For translation, it measures the mean norm of

$${\mathrm{\Delta }}^2{\mathbf{x}}_k=({\mathbf{x}}_{k+2}-{\mathbf{x}}_{k+1})-({\mathbf{x}}_{k+1}-{\mathbf{x}}_k)\mathrm{.}\backslash Delta^2\backslash mathbf\{x\}\_k\; =\; (\backslash mathbf\{x\}\_\{k+2\}-\backslash mathbf\{x\}\_\{k+1\})\; -(\backslash mathbf\{x\}\_\{k+1\}-\backslash mathbf\{x\}\_k).$$

Rotation uses the corresponding change between consecutive relative rotations and reports the mean geodesic angle in degrees. Lower values indicate a smoother discrete trajectory, but lower is not automatically better: an inaccurate constant trajectory would be perfectly smooth. We therefore compare this metric with the demonstration and with pose accuracy.

Physical velocity, acceleration, and jerk

For physical dynamics, finite differences are divided by the real timestep $\mathrm{\Delta }t=1\mathrm{/}30\backslash Delta\; t=1/30$ s. Translation uses first, second, and third differences of position, reported in $\mathrm{m}\mathrm{m}\mathrm{/}\mathrm{s}\backslash mathrm\{mm/s\}$, $\mathrm{m}\mathrm{m}\mathrm{/}{\mathrm{s}}^2\backslash mathrm\{mm/s^2\}$, and $\mathrm{m}\mathrm{m}\mathrm{/}{\mathrm{s}}^3\backslash mathrm\{mm/s^3\}$. Rotation first computes geodesic angular speed between consecutive orientations, then differentiates it to angular acceleration and jerk, reported in $\mathrm{d}\mathrm{e}\mathrm{g}\mathrm{/}\mathrm{s}\backslash mathrm\{deg/s\}$, $\mathrm{d}\mathrm{e}\mathrm{g}\mathrm{/}{\mathrm{s}}^2\backslash mathrm\{deg/s^2\}$, and $\mathrm{d}\mathrm{e}\mathrm{g}\mathrm{/}{\mathrm{s}}^3\backslash mathrm\{deg/s^3\}$.

These metrics describe motion texture rather than accuracy. We report them relative to the demonstrated trajectory because neither the smallest nor the largest value is inherently desirable.

Adjacent-frame direct prediction-change MSE

This diagnostic asks how much the policy output changes when it is queried one frame later. The policy is run independently at times $tt$ and $t+1t+1$, and the two decoded chunks are compared at the same chunk indices. We do not align actions by their future world timestamp or compensate for the changed relative-pose anchor.

We report normalized action MSE, XYZ MSE in ${\mathrm{m}\mathrm{m}}^2\backslash mathrm\{mm\}^2$ per dimension, and rotation geodesic MSE in ${\mathrm{d}\mathrm{e}\mathrm{g}}^2\backslash mathrm\{deg\}^2$. For stochastic policies, the two calls also contain independent sampler variation. Lower values therefore mean that two adjacent queries produced more similar outputs—not that either output was accurate. A constant but incorrect policy would score perfectly, which is why this diagnostic is interpreted only beside trajectory error.

What physical deployment taught us

The error metrics above are necessary, but our robot experiments made the motion metrics equally important. A policy can have a low endpoint error and still produce a trajectory that is difficult for the arm to track smoothly.

We found two distinct physical-motion properties worth checking.

The first is within one predicted chunk. Without downstream interpolation or trajectory optimization, a deployment candidate should not produce acceleration and jerk substantially above the demonstrated trajectory. For our setup, reaching approximately the ground-truth level or below was a useful minimum condition for smooth execution. This is a project-specific deployment heuristic, not a universal optimality claim: an extremely low value may also indicate an over-smoothed and unresponsive policy.

The second is adjacent-frame direct prediction change. The robot repeatedly queries the policy as observations change. Even if each chunk is smooth internally, large changes between predictions at $tt$ and $t+1t+1$ can cause the commanded trajectory to jump whenever the action queue is refreshed or chunks are blended. This metric also includes the changed relative-pose anchor and, for stochastic policies, a fresh sampler draw. It therefore measures the stability of the complete inference output, not just image sensitivity.

The representative physical sweep showed clear family signatures: ACT was generally smoother than the demonstrations, SmolVLA remained higher-frequency, and the mature π0.5 port sat closer to the demonstrated acceleration and jerk profile. These are motion descriptions, not a standalone ranking; they must be read together with decoded accuracy.

Figure 1. Representative rotational and translational velocity. Dashed references show the demonstrated trajectory.

Figure 2. Representative physical acceleration under the same evaluation window.

Figure 3. Representative true third-derivative jerk at 30 Hz.

The same view across all evaluated runs shows that these signatures were not isolated to one hand-picked checkpoint:

Figure 4. Velocity across the complete evaluated inventory; the y-axis is logarithmic.

Figure 5. Acceleration across the complete evaluated inventory.

Figure 6. Jerk across the complete evaluated inventory.

Figure 7. Predicted-to-demonstrated ratios across velocity, acceleration, and jerk. Each grouped triplet is one checkpoint, not a training trajectory.

Training budget also changed motion differently across policy families. The combined view is the most useful overview; the three following plots provide larger views of each derivative order.

Figure 8. Velocity, acceleration, and jerk versus training budget. Dashed lines are demonstrated references.

Figure 9. Velocity versus training steps.

Figure 10. Acceleration versus training steps.

Figure 11. Jerk versus training steps.

Finally, adjacent-frame prediction change exposes a different failure mode from within-chunk jerk. Deterministic ACT occupied the low-change region. The π0.5 port moved toward that region with training, while SmolVLA retained larger changes over the full chunk. Lower is not automatically better—a constant but wrong prediction would score perfectly—so this plot must be interpreted beside endpoint and chunk accuracy.

Figure 12. Direct same-index change between independent predictions at adjacent frames. The panels report normalized action, XYZ, and rotation MSE; lower means less output change, not greater accuracy.

Taken together, the metrics gave a more useful selection picture than any single leaderboard column. ACT R50 combined strong full-chunk accuracy with smooth, low-change predictions, although it could be over-smoothed. The mature π0.5 port combined similarly strong endpoint accuracy with dynamics closer to the demonstrations. SmolVLA reached good near-term accuracy but retained a larger full-chunk error, higher acceleration and jerk, and larger adjacent-frame changes. Those tradeoffs are the reason our final candidate decision used all three layers rather than endpoint error alone.

Q2: How did the policy families and training settings compare?

The complete 30-step prediction is the more demanding open-loop test because errors can grow farther into the chunk. The lowest observed full-chunk endpoint band belonged to mature ACT R50 and our π0.5 port, both around 21–22 mm. Their bootstrap intervals overlap, so the data does not establish a clear winner between them.

The main training and evaluation inventory was:

The epoch counts are approximate effective dataset passes, computed as steps × batch size / 140,522 training frames. Training uses sampled frames rather than literal sequential epochs, so this column is a scale comparison, not an exact record of how many times every frame was visited.

Figure 13. Full 30-step budget curves under one evaluation protocol. The panels include XYZ and rotation endpoint error, per-component L1, and Acc@0.5/0.1. Some additional experimental arms appear in the figure for completeness but are not interpreted in this article.

Training budget affected the families differently. ACT R50 improved from 23.24 mm at 100k to 21.22 mm at 600k, then remained in roughly the same range through 1M. The π0.5 port was effectively mature by approximately 350k–650k; extending 700k to 1M changed its endpoint from 21.77 to 21.75 mm. SmolVLA improved more gradually, from about 27.3 mm at 100k to 26.3 mm at 1M, but retained a larger far-end error.

Several smaller settings were less influential than model family and budget. For SmolVLA, rot6d and axis-angle actions produced nearly identical full-chunk endpoint error, around 26.9–27.0 mm in the matched notation comparison. Training only the physical action subspace versus the full padded action width also produced overlapping endpoint curves; masking gave a modest late smoothness improvement but not a meaningful endpoint gain. These controls kept us from over-interpreting representation details as architecture improvements.

These rows provide context, not a controlled architecture ranking. The policies differ in pretraining, parameter count, optimization, batch size, and trainable parameter subsets, so nominal training steps are not directly comparable. The official OpenPI and local π0.5-port rows also retain stack-specific differences in state construction, normalization, dataset layout, and numerics. The OpenPI result is useful evidence of strong early fine-tuning, but it is not a pure JAX-versus-PyTorch comparison.

The useful conclusion is limited to our case: ACT R50 and the π0.5 port were the strongest candidates under the full-chunk offline protocol, while the remaining families occupied a higher-error range. Closed-loop testing is still required to determine whether those differences affect task success.

The policy choice also depends on deployment priorities. ACT is deterministic, relatively small, and inexpensive to run. The π0.5 port brought pretrained VLA features and reached a similar mature endpoint region, but at a higher systems cost. Official OpenPI learned quickly at its early budget. SmolVLA was competitive closer to the current observation but retained more error toward the end of a one-second chunk. We therefore treated the table as a candidate filter rather than a universal ranking.

Q3: Did increasing ACT capacity help?

Our production ACT model used a ResNet-18 visual backbone and had been trained for 3 million steps. Its full-chunk endpoint error eventually plateaued around 23.3 mm.

A standard ACT with ResNet-50 reached that region much earlier and continued improving. With the initialization recipe held fixed, its full-chunk endpoint error reached 23.2 mm at 100k steps and about 21.2–21.3 mm at 600k–1M steps. This control matters: the improvement remained when R18 and R50 used the same initialization recipe, so it was not merely a pretrained-weight artifact.

Figure 14. ResNet-50 reaches the historical ResNet-18 full-chunk plateau much earlier and remains better at mature checkpoints.

Increasing capacity elsewhere was less useful. A widened and deepened 145M-parameter ACT transformer did not provide a meaningful endpoint improvement over standard R50 at the tested budget, while increasing latency and memory. For this dataset, spending capacity on visual features was more effective than broadly enlarging ACT.

For our setup, ResNet-50 became the practical ACT backbone when its roughly 25% inference-cost increase was acceptable. A simpler direct-regression ACT remained useful when latency and memory mattered more. These are choices for this system, not evidence that ResNet-50 is the best visual encoder for ACT in general.

Q4: What temporal context does ACT need?

The main temporal-context question was not whether to add another image. It was whether ACT needs the relative end-effector state and how much recent state history is useful.

For UMI training, the model-facing proprioceptive state is a short history of end-effector poses expressed relative to the current pose. It is derived from the handheld trajectory during training and can be reconstructed from robot forward kinematics during deployment. We evaluated $W\in \{0,2,3,4,5,10\}W\backslash in\backslash \{0,2,3,4,5,10\backslash \}$, where $W=0W=0$ is the image-only model and $W=2W=2$ is the original previous-plus-current state.

The first result is that proprioception itself matters. In the near-term protocol, $W=0W=0$ was worse than $W=2W=2$ at every matched budget. At 100k steps, XYZ endpoint error was 13.55 mm without state and 9.20 mm with two poses. At 500k, it was 13.19 versus 10.25 mm. The episode-bootstrap intervals were separated across the curve, Acc@0.1 dropped by roughly three to four percentage points without state, and additional training did not close the gap. For this task, the wrist image did not make end-effector state redundant.

The near-term endpoint result changed as a step rather than a smooth trend. $W=3W=3$ remained close to $W=2W=2$, while $W=4W=4$ entered the same broad performance band as $W=5W=5$ and $W=10W=10$. At 500k steps, endpoint error was 10.25 mm for $W=2W=2$, 9.05 mm for $W=4W=4$, 8.50 mm for $W=5W=5$, and 8.59 mm for $W=10W=10$. The best individual rows were $W=10W=10$ at 100k and 200k, at 7.96 and 7.86 mm.

Figure 15. State-window sweep over $W\in \{0,2,3,4,5,10\}W\backslash in\backslash \{0,2,3,4,5,10\backslash \}$. The $W=0W=0$ curve measures the no-proprioception condition. Four poses enter the longer-window accuracy band; adding more poses does not produce a clear monotonic gain.

At 30 Hz, four poses cover about 100 ms of end-effector history. That provides direct information about recent velocity and acceleration. The result remains provisional: each window length was trained with one seed, and the live deployment path supported only the two-pose state at the time of the study. The gain was also concentrated in the near-term evaluation. Under full 30-step scoring, the $W=4W=4$ arm remained around 21.0–22.1 mm and its confidence intervals overlapped the ordinary R50 curve.

A smaller visual-history control

A single image does not directly reveal velocity, so stacking the previous camera frame seemed like an obvious ACT improvement. We trained a two-frame ResNet-50 ACT using channel-stacked images at t-1 and t, with the first convolution adapted from the same pretrained backbone.

It did not help the full-chunk result. Across five matched checkpoints, the endpoint gap between the one-frame and two-frame models was at most 0.30 mm, with overlapping bootstrap intervals. Rotation endpoint error, component-wise error, and thresholded accuracy also followed the same budget curve.

Figure 16. Matched one-frame and two-frame ACT budget curves. The figure includes the report's additional near-term and motion diagnostics; the full-chunk conclusion is based on the separately scored 30-step comparison.

The result is dataset-specific, not a claim that temporal vision never helps. Our wrist-view camera runs at 30 Hz, and adjacent images often contain highly redundant visual information. In this study, direct EE-pose history was the useful temporal signal; adding one more RGB frame was not.

What remains to be tested

The most important limitation is that this is open-loop evidence. It does not measure compounding control error, contact behavior, recovery, grasp success, safety events, or how predicted chunks interact with the robot controller. A model with slightly lower offline endpoint error may not be the model that succeeds most often on the arm.

The next step is therefore a controlled robot study on an untouched test distribution. It should randomize initial poses and occlusions, report task and substage success, include recovery and intervention tests, and record execution latency and safety failures. The number of trials should be chosen to resolve practically meaningful differences rather than to produce a few illustrative rollouts.

Several offline questions also remain. The principal comparisons need more balanced training seeds, and evaluation on new sessions, operators, scenes, and manipulation tasks is needed before treating any result here as a general property of UMI policies.

The study did give us a clearer experimental discipline: compare decoded physical trajectories, keep the execution protocol fixed, report uncertainty, and distinguish a controlled ablation from a cross-family comparison. Those lessons are more transferable than any single model ranking.

References

1. C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. “Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots.” arXiv:2402.10329, 2024.
2. R. Cadene et al. “LeRobot: An Open-Source Library for End-to-End Robot Learning.” ICLR 2026; arXiv:2602.22818.
3. T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware.” arXiv:2304.13705, 2023. Introduces Action Chunking with Transformers.
4. M. Shukor et al. “SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics.” arXiv:2506.01844, 2025.
5. Physical Intelligence et al. “π0.5: A Vision-Language-Action Model with Open-World Generalization.” arXiv:2504.16054, 2025.
6. Physical Intelligence. “OpenPI: Open-source models and packages for robotics.” GitHub repository.