CERES: your new best friend (if you're into egocentric capture)

Recording egocentric video is easy. Recording a demonstration that you can still interpret six weeks later, less so. The awkward work is keeping video, pose, instructions, task boundaries, failed attempts and tracking failures on the same clock, and then turning all that into a dataset you can actually use.

As a robotics team working for some of the world's leading enterprise customers, from hyperscalers to mining giants, we deal with egocentric capture both for research and project use. We built CERES (short for Capturing Egocentric Recordings with Ease and Speed) because nothing on the market could guarantee the reliability, auditability and task assurance that you need when the stakes are high. CERES is a browser-first acquisition system for VR headsets, currently supporting the Meta Quest family (more comping soon!). It records outward-facing video, optional microphone audio, head pose and both 25-joint hands against an explicit task protocol, then exports retained episodes in a LeRobotDataset v3-compatible format.

What we learned through our day-to-day work is that the useful trick is not putting a camera on someone's head -- GoPros have been bravely managing that since 2004 -- but creating a research-useful, task-guided product. CERES records task boundaries, retries, timing and explicit gaps while the demonstration is happening, rather than asking an annotator to reconstruct them later from a swamp of MP4s and whatever VLA happens to grace their cache.

The main way to use CERES is Solo. One person can define a run, perform it in Quest, review the result and export it locally or to the Hugging Face Hub. There is no capture director, second browser or native application in the loop. Duet keeps the same recorder and task model but adds a second browser for live direction when the protocol, demonstrator or cost of a failed take warrants supervision.

Try it: open Solo in Quest Browser; local capture requires no installation or account. No Quest? Have a look at the public demos, ideally in the Hugging Face based dedicated Viewer, to see what a CERES capture looks like.

CERES 0.9.0, Xuesen, is the current pre-1.0 release. This post follows a Solo run from task definition through the browser journal to a dataset on the Hub.

The unit of collection should be a task, not a file

Most egocentric capture pipelines start with an undifferentiated recording and create structure afterwards. That works, but it throws away the moment when structure is cheapest and most reliable: while the demonstrator and protocol still agree (mostly) as to what is happening.

We built CERES to start with a versioned task specification. A run contains an ordered set of:

• Open tasks, which continue until the demonstrator advances them
• Timed tasks, which end after a defined active duration
• Pauses, which create deliberate unrecorded intervals (these are sometimes known as RESETS)
• Repetitions, reset windows and complete run cycles (think gym)

Each recorded task occurrence becomes a segment inside the episode. The segment retains the task identity, repetition, take, outcome, annotations and exact recorder bounds. PASS and FAIL annotations exist to let demonstrators or capture directors weigh in on what they just saw. RETRY preserves the original attempt in the durable source journal but replaces it in the exported dataset with the next attempt for that repetition.

We found that this structure best supported long-horizon work. "Pack a grocery bag" is not one homogeneous action but a sequence of reaching, grasping, grouping, placing and resetting, often with repeated attempts. If those units exist only in the demonstrator''s head, the dataset team gets to rediscover them frame by frame later, usually by means of expensive soothsaying (no shade on VLMs). Post-hoc annotation is not wrong, but it is an expensive way of reconstructing facts the acquisition system already knew.

Instead of relying on VLM augury, in CERES every retained row receives a task_index. A training pipeline can then join that index to the task text; repetition and take boundaries remain available in ceres/episode-metadata.json.

What can you do with it?

Collect long-horizon and bimanual demonstrations

Solo is our beloved 'single player mode' that removes the second operator and browser from the capture run. CERES canonicalises and hashes the task specification, so the same versioned protocol can be shared across demonstrators. Each person sees the current instruction, countdown, task state and recording state in XR, and the resulting episodes retain which exact specification was used.

That is useful for household manipulation, laboratory work, assembly, equipment set-up and other procedural tasks where the order of operations matters. We bundled some examples with CERES, ranging from placing a cup and folding laundry to washing dishes and packing groceries, but the CERES task grammar is pretty easy to extend to your own use cases -- go wild! Open tasks accommodate naturally variable completion times, timed tasks bound an attempt, repeat counts create controlled repetitions, reset windows keep scene preparation outside the recorded timeline.

As researchers, we rely on demonstrators to give us the raw material that someday will found robotic policies. We owe it to them to respect their time and effort by specifying task descriptions in a way that will allow them to execute those ordinary, annoying yet structurally rich tasks that drive robotics once the cube has been picked up for the ten-thousandth time.

Learn egocentric representations with aligned human motion

Because video and trajectories share timestamps, an action-recognition model can train against task_index, a forecasting model can predict future hand pose or pinch state and a multimodal encoder can align outward frames with head and hand motion. CERES does not supply object contacts or semantic success labels out of the box -- it is a data recorder, leaving the interpretation to you. We are not opinionated as to whether a delicious but collapsing burrito should be considered a success (yes, we have a burrito folding task).

The recorder is deliberately honest about missing evidence. One source observation can fill at most one recorder slot. If no source observation arrives for a scheduled slot, CERES writes an explicit gap; a frame with missing hand tracking remains a row with its tracked flags unset. It does not duplicate the previous pose to make the time series look complete. A filled spreadsheet is not the same thing as a measured trajectory, and we won't pretend it is so.

Use human demonstrations upstream of robot learning

The exported state contains head pose and both tracked hands, the action field contains left and right pinch distance. CERES does not pretend to gather kinematics the way UMI capture devices might be able to. However, CERES data can instead feed inverse-dynamics and embodiment-retargeting pipelines. After conversion into an embodiment's action space, task boundaries and annotations can also condition progress models or task-conditioned policies. For example, we have retargeted the author's valiant attempts at folding a t-shirt to Allegro, Shadow, LEAP and SCHUNK SVH hand models.

Use Solo for independent capture, add Duet when supervision matters

Solo is the default when one trained demonstrator can run a defined protocol independently. Duet is the supervised alternative: a capture director builds the run, pairs a Quest by link or QR code, watches a lossy live view and telemetry, and can pause, annotate, retry or advance the take. The demonstrator sees the same authoritative task state in the headset.

Duet is appropriate for unfamiliar demonstrators, costly sessions, strict study protocols and tasks where live intervention matters. The present system pairs one director with one demonstrator, with fleet supervision coming to the commercial version in 1.1.0 and open source in 1.5.0.

Solo: the complete loop on one headset and five minutes

In Solo, the Quest owns the run state, durable journal, episode review and export queue. This makes it the best way to get started right now, no installs needed. The web-based ops panel handles configuration and episode management, the HUD handles the active task. A typical run looks like this:

1. Open CERES Solo in Quest Browser.
2. Create a run in the browser or import a versioned task specification from a sample, public Gist or local file.
3. Choose the outward camera, optional microphone capture and the task at which the run should begin. These are your privacy settings, and we would want you and all persons present to be comfortable with what was chosen.
4. Get set up, and as soon as 'REC' comes on, start your task.
5. Advance, annotate or retry tasks while the event is still fresh.
6. Finish the run and inspect the episode and capture-health summary.
7. Export to browser storage, a selected folder or an authorised Hugging Face dataset repository.

The recorder normally runs at 30 Hz and owns the application clock independently of rendering, live monitoring, speech recognition and interface refresh. Camera chunks, optional source audio, sensor frames, explicit gaps and task events enter one ordered binary journal. The result is recoverable after a page reload or interrupted session, provided the browser profile and its origin-private storage have not been cleared (at which point we infer you might have bigger worries).

CERES was devised with capture quality monitoring in mind (and by way of full disclosure, we continue to tweak this as we get more and more research teams contributing their perspectives), so overspeed alerts, hand visibility alerts and assistive visualisations are all available to aid in creating a quality capture. Normals, for instance, show the visible aspect of the hand, while motion trails and motion colouring shows direction and velocity of hand movement.

Egocentric capture can record faces, voices, screens and private spaces. Obtain consent and any required study approval before collection, review or redact captures before sharing, keep unreviewed material in a private Hub repository.

What reaches the Hub

The production exporter runs in a dedicated browser worker and uses a Rust/WebAssembly implementation to reconstruct the retained timeline and write a LeRobot v3-compatible dataset. Python and a native FFmpeg installation are not required on the Quest.

Each export includes:

The exported video is silent. Optional microphone audio remains in the durable CERES source but is not currently emitted into the LeRobot media product. Reset windows are paused out of the recorder timeline. Retried ranges are removed from both telemetry and video, while the original bytes remain available in the source journal for audit and recovery.

CERES also manages a bounded section of the dataset card so that observation Parquet files appear in the standard Hugging Face Dataset Viewer without accidentally treating LeRobot metadata or CERES sidecars as training rows. The full CERES dataset viewer Space replays the video, pose, hands, task segments and gaps together. A small Quest 3 demonstration dataset is already available on the Hub.

For a quick inspection through datasets:

from datasets import load_dataset

rows = load_dataset(
    "chrisvoncsefalvay/ceres-demos",
    split="train",
)

print(rows.features)
print(rows[0]["task_index"])
print(rows[0]["ceres.source_gap"])

For robot-learning workflows, the CERES exporter writes LeRobotDataset v3-compatible episode shards: Parquet for state and action data, MP4 for visual observations, and relational metadata for episodes and tasks. As normal Hub artefacts, they can be revision-pinned, kept private and inspected through the Hugging Face Dataset Viewer and CERES Space.

What CERES does not do

CERES is a capture system, not a motion-capture laboratory, teleoperation stack or general sensor-ingestion framework. Quest hand tracking and outward-camera registration remain device- and browser-dependent. The browser does not expose force, tactile state, object state or robot joint commands. We want CERES to do one thing, and one thing only: be the easiest, most convenient way for you to organise and run capture sessions that make the most of your time, and your demonstrators'.

Is CERES for you?

It's important that we are quite honest about this (which we can do, considering CERES is available for the low, low price of $free.00): CERES is overkill for some jobs. If you need twenty clips for qualitative review, if the recording will be watched once and discarded, or if an instrumented robot or teleoperation stack already writes synchronised video, state, action and provenance, the full task-specification-to-LeRobot path buys little. A Quest is also the wrong instrument when a study needs calibrated metrology, tactile sensing, full-body motion or robot-native control signals. Sometimes the correct capture system is a camera and a sensibly named folder.

The dividing line is not how impressive the task looks but what will be asked of the data afterwards. Will episodes be pooled across people or sites? Used to train a policy? Compared across protocol versions? Revisited after the capture team has moved on? Asked to support a paper or safety claim? At that point, clip_047_final_reallyfinal.mp4 stops being cute and becomes a liability. Even teams that do not need CERES (yet!) should want its guarantees: one authoritative clock, explicit missingness rather than invented samples, task and attempt boundaries recorded at acquisition, retries preserved rather than quietly overwritten, raw evidence kept distinct from derived training data, versioned and inspectable exports, and a clear answer to when bytes leave the device. Those are not CERES-specific virtues but properties of a dataset expected to outlive the afternoon.

CERES is not the right instrument for every application. But we believe that if another instrument is used, it should be able to answer the same questions. Use CERES, borrow the acquisition contract, or show that your existing stack provides equivalent guarantees. "We have the video" is a starting point, not a data strategy.

Try it, then be unkind to it

You can try CERES right now - start with one useful multi-step protocol in Solo, record a few cycles, deliberately retry one attempt, inspect the gaps, export to a private Hub repository and open it in the viewer. If the task needs another pair of eyes, repeat it in Duet with a capture director.

The public CERES research release is licensed under CC BY-NC 4.0. Have a go and tell us where it breaks. We are looking forward to inputs from teams in research and industry alike. We'd also love to hear from you about what you want from a tool like this, and where CERES does not yet meet your needs.

Links

• Launch CERES
• Start Solo capture
• CERES public research repository
• CERES demonstration dataset
• CERES dataset viewer Space
• LeRobotDataset v3 documentation