The 45-min poll deadline cancelled shards that were merely QUEUED waiting
for a10g capacity — throwing away their GPU queue position — then, after
the retry rounds, marked the row failed while leaving jobs running, so
their outputs completed late and stranded in staging. This failed
otherwise-successful submissions under GPU load.
- Poll loop never cancels healthy QUEUED/RUNNING shards; only ERROR shards
re-dispatch. Each job's EVAL_JOB_TIMEOUT is the real RUNNING bound.
- Replace the per-round reaper with one generous env-tunable backstop
(default 6h) that only stops an unbounded loop.
- Reconcile-before-fail: always attempt the merge from whatever landed;
a shard flagged ERROR/timeout may still have written all its fixtures
(outputs are the source of truth). Only fail on genuinely-missing ones.
- Cancel orphan jobs on the true-failure path so nothing keeps billing.
- Boot sweep now reconciles stranded pending rows (finalize if artifacts
are complete) instead of blindly fai…
Bill eval jobs to lvwerra instead of michaelr27
Michael Rabinovich
190633b
Bump cadgenbench pin to 8ae1432 (revert 0.2.0 edit-baseline rework; back to 0.1.0)
Michael Rabinovich+1: Cursor
182f8fc
Co-authored-by: Cursor <cursoragent@cursor.com>
submit: drop per-candidate OCC parse from validation gate
Michael Rabinovich+1: Cursor
98bc085
The gate no longer rejects on candidate content (eval re-validates and scores
invalid STEPs 0), so parsing every STEP at submit time was pure wasted work --
slow and OOM-prone on large/malformed files (e.g. a 58 MB STEP), which made
validation crawl and could error the whole submission. Keep only the cheap
presence + non-empty checks; validity is the evaluator's job.
Co-authored-by: Cursor <cursoragent@cursor.com>
submit: raise concurrency_limit to 16 so submissions run in parallel
Michael Rabinovich+1: Cursor
770de2c
handle_submit blocks for the whole eval (dispatch + poll). With Gradio's
default concurrency of 1, submissions serialized and a single stuck eval froze
the Submit button for everyone. Allow up to 16 concurrent.
Co-authored-by: Cursor <cursoragent@cursor.com>
Bump cadgenbench pin to 2abaea8 (cadquery optional; fixes libTKIVtk/VTK clash)
Michael Rabinovich+1: Cursor
f4b0f2f
Co-authored-by: Cursor <cursoragent@cursor.com>
Submit gate: accept unparseable candidates (eval scores them invalid/0)
Michael Rabinovich+1: Cursor
987f284
A present-but-unloadable output.step (empty, or parses to no geometry) was
rejecting the entire submission. The evaluator's CAD-validity gate already
scores any non-valid solid as cad_score=0 (status=invalid), so a bad candidate
is a legitimate zero-scoring outcome, not a submission error. The gate now logs
and lets it through instead of blocking the whole zip.
Co-authored-by: Cursor <cursoragent@cursor.com>
The last rebuild floated pyvista (pulled transitively by cadgenbench) and the
headless vtk-osmesa swap to latest, producing a mismatched pair that failed at
import (libvtkWrappingPythonCore3.12.so not found) and broke the Space. Pin both
to the known-good versions.
Co-authored-by: Cursor <cursoragent@cursor.com>
Drop mesh-file submissions; STEP-only candidates
Michael Rabinovich+1: Cursor
04e4262
Remove the triangle-mesh submission path (and its OpenSCAD framing) from the
submit gate, UI copy, and metrics page; candidates are STEP/BREP only. Pin the
eval image's cadgenbench to a47c951.
Co-authored-by: Cursor <cursoragent@cursor.com>
Retry straggler shards on poll-deadline instead of failing the submission
Michael Rabinovich+1: Cursor
1fd03de
When the Space-side shard poll window elapses with shards still
non-terminal (typically the tail shard stuck QUEUED waiting for
a10g-large GPU capacity, not a compute failure), cancel and re-dispatch
those stragglers and reset the window, up to SHARD_DEADLINE_RETRY_ROUNDS
(2) rounds, before failing. All-or-nothing is preserved; shard uploads
are idempotent so a re-dispatch is safe. Fixes large (full-81) runs that
intermittently failed with "Space-side poll deadline exceeded (2700s)".
Co-authored-by: Cursor <cursoragent@cursor.com>
Bump cadgenbench Docker pin to e7a60e0
Michael Rabinovich
b430299
Bump cadgenbench Docker pin to 1924090
Michael Rabinovich
9c0fafb
Bump cadgenbench Docker pin to f284fbd
Michael Rabinovich
7fa5cee
Accept mesh candidates in leaderboard submissions
Michael Rabinovich
0f9004d
Update gallery default fixture
Michael Rabinovich+1: Cursor
85bb21f
Co-authored-by: Cursor <cursoragent@cursor.com>
Bump CADGENBENCH_SHA to 037eade; add GT edit-diff report backfill tool
Michael Rabinovich+1: Cursor
b885669
Pins the leaderboard image to the cadgenbench commit that renders the GT
answer-key edit-diff report column. Adds tools/backfill_report_gt_edit_diff.py,
the one-off (idempotent) patcher that swaps existing reports' editing GT
columns to the answer-key turntable without re-evaluating.
Co-authored-by: Cursor <cursoragent@cursor.com>
Gallery: show GT "answer-key" edit-diff for editing fixtures
Michael Rabinovich+1: Cursor
49e27be
The ground-truth row now renders the blue edit-diff turntable (the
correct change vs the input) for editing fixtures instead of a plain GT
turntable, so the reference reads the same way as the candidate
red/amber diff in the same column. Adds a gt_diff_resolver pointing at
the one-time GT-dataset webp (<fixture>/renders/edit_diff_gt.webp, via
the existing /gt proxy) and updates the compare modal caption/legend.
tools/generate_gt_edit_diff.py renders/uploads those webps for the 32
editing fixtures (one-time, per data revision).
Co-authored-by: Cursor <cursoragent@cursor.com>
Add missing header logo PNG and tighten title spacing
Michael Rabinovich+1: Cursor
ae4b377
The deployed app.py already references assets/logo.png (via
_logo_data_uri), but the binary was never committed, so the Space
build crashed with FileNotFoundError. Commit the transparent,
content-cropped wordmark and snug up the title block (smaller logo
height + collapsed flex gap) so it sits directly above the tabs.
Co-authored-by: Cursor <cursoragent@cursor.com>
Update leaderboard Space URL after rename to CADGenBench
Michael Rabinovich+1: Cursor
7922a9c
The Space moved from HuggingAI4Engineering/cadgenbench-leaderboard to
HuggingAI4Engineering/CADGenBench; point the hardcoded references at the
new URL.
Co-authored-by: Cursor <cursoragent@cursor.com>
Bump CADGENBENCH_SHA to ee8b257 (deterministic aligner)
Michael Rabinovich+1: Cursor
884294a
Co-authored-by: Cursor <cursoragent@cursor.com>
Bump pinned cadgenbench SHA to 0cf5913 (sample-named run_summary keys)
Michael Rabinovich+1: Cursor
77afed9
Co-authored-by: Cursor <cursoragent@cursor.com>
Rename fixture-named result keys to sample-named keys
Michael Rabinovich+1: Cursor
be6fa3d
Read/write per_sample_scores, per_sample_breakdown, n_samples,
per_sample_results across submit/gallery/admin and tools. Also link the
canonical submission docs from the Submit tab.
Co-authored-by: Cursor <cursoragent@cursor.com>
feat(admin): hide the entire admin panel from non-admins
Michael Rabinovich+1: Cursor
335cdfa
The Admin tab previously showed everyone the table + (disabled) controls.
Actions were already gated server-side (is_admin re-check in every handler),
but the UI and the read-only table were visible to all, which looked
unprofessional and streamed submission rows to non-admins.
Wrap all admin UI in a Column that stays hidden unless the logged-in user
is in CADGENBENCH_ADMINS; non-admins (and logged-out visitors) now see only
the login/logout button and a status line. The table starts empty and the
load/refresh/timer handlers return an empty frame for non-admins, so no
submission data is sent behind the hidden panel. Server-side gating on the
mutating handlers is unchanged (defense in depth).
Co-authored-by: Cursor <cursoragent@cursor.com>
bump cadgenbench -> 68e9337 (clamp edit-diff zoom to full view)
bump cadgenbench -> 2b1c939 (surface-deviation edit-diff field + Surface Distance F1 rename)
Michael Rabinovich+1: Cursor
5b57471
Co-authored-by: Cursor <cursoragent@cursor.com>
metrics page: rename Point Cloud F1 -> Surface Distance F1
Michael Rabinovich+1: Cursor
c8f3957
Match the renamed shape metric and describe it as point-to-surface
(closest point on the other mesh), not nearest point on the other cloud.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: fix tasks iframe fit (use body.scrollHeight)
Michael Rabinovich+1: Cursor
326d20e
documentElement.scrollHeight floors at the iframe viewport height, so the
previous fit never shrank a short detail card below the 90vh fallback (the
blank gap persisted). body.scrollHeight reflects the real content height;
verified the iframe now matches content on both desktop and mobile.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: fit tasks iframe to content (kill blank space)
Michael Rabinovich+1: Cursor
6905c70
The Tasks browser iframe was a fixed 90vh, so a short detail card left a large
blank gap above the Refresh button. Mirror the gallery's approach: size the
iframe to the document height on load / resize / view switch (and via a
ResizeObserver for lazy render images), so it shrinks to the detail card and
grows for the summary grid.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: fix metrics 'On this page' links on mobile
Michael Rabinovich+1: Cursor
528723d
When embedded on huggingface.co the Space is in a cross-origin outer
frame that owns the scroll, so a content-sized metrics iframe couldn't be
scrolled from inside and the anchor links silently did nothing. Give the
mobile metrics iframe its own bounded height so it scrolls its own
document; scrollIntoView then works for the 'On this page' links. Desktop
unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: axis gizmo on task render grids; bump cadgenbench
Michael Rabinovich+1: Cursor
1a5a9a2
tasks.py overlays the shared XYZ orientation gizmo on the editing task's
starting-shape render grid (imported from cadgenbench.common.axes_gizmo).
Bump CADGENBENCH_SHA to 193f2f5 so the Space's report generator picks up the
gizmo + headline-pill changes and tasks.py can import the helper.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: metrics tab single-scroll + polish on mobile
Michael Rabinovich+1: Cursor
f07f508
On phones the Metrics tab's fixed 85vh iframe made a nested scroll-box.
Auto-size the embedding iframe to the page content on narrow screens so
it's a single page scroll (desktop keeps its fixed height, which reads
well). In-page anchor links use scrollIntoView so they still work when
the iframe has no internal scroll. Also tighten padding/formula/table
sizing and shrink the vh end-spacer on mobile.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: tighten mobile gallery box to a single scroll
Michael Rabinovich+1: Cursor
4090e8a
Increase the mobile chrome reserve so the capped box fits the phone
viewport (kills the residual outer page scroll on top of the box's own
scroll). GT row stays pinned; shows the GT + a couple of models with the
rest reachable by scrolling the box.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: mobile uses the same aligned table as desktop
Michael Rabinovich+1: Cursor
0c0ddd8
Revert the mobile card layout (it broke the GT-vs-output comparison by
putting GT in its own card). On phones now show the same table: sticky
sample-column headers + pinned ground-truth row, with each model's four
renders aligned in the same columns directly under the GT. To fit phone
width the per-model name+score becomes a slim bar above that model's
render row (no room for a left identity column plus four renders). The
box still caps + scrolls internally so the GT row stays pinned.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: dedicated mobile card layout for the gallery
Michael Rabinovich+1: Cursor
a636039
A horizontal-scroll table is unusable on phones (one render at a time).
On narrow screens hide the table and render one card per model instead
(plus a ground-truth card), each showing the four renders in a labelled
2x2 grid, so the whole comparison reads top-to-bottom with a single
vertical scroll.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: fix mobile gallery (GT row wrap + double scroll)
Michael Rabinovich+1: Cursor
00f76c2
- The GT row wrapped on mobile because a base rule (.grow.gt-row .rank)
re-showed the rank cell only on that row (7 items in a 6-col grid);
force-hide rank on mobile so the row lays out correctly.
- On phones, drop the vertical cap so the page scrolls once vertically
and fixtures are reached by horizontal swipe (frozen Submission
column), instead of a tiny capped box plus a confusing outer scroll.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: reclaim Gradio chrome + responsive mobile gallery
Michael Rabinovich+1: Cursor
2ec97dd
- Hide the Gradio footer and tighten outer padding (the unused space
between Refresh and the footer) and grow the gallery box to use it, so
more rows fit in one viewport.
- Mobile: the 4-sample matrix no longer clips. On narrow screens the
table scrolls horizontally (swipe), the rank column is dropped, the
Submission column is frozen to the left, and the score breakdown is
hidden -- so the renders are actually visible and you can tell which
model each belongs to, with the header + GT row still pinned.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: squeeze header/caption, grow gallery to show more rows
Michael Rabinovich+1: Cursor
882faa0
- Drop the gallery caption and collapse the Gradio title to a single
compact line, and cut the gallery body padding (removes the dead gap
between the last row and the Refresh button).
- With that freed vertical space, lower the box chrome reserve and raise
the cap so the gallery shows as many rows as fit in one viewport while
staying a single scroll (harness-verified across screen sizes).
Co-authored-by: Cursor <cursoragent@cursor.com>
- Raise the box height cap (760->900) so larger screens use their
vertical space and show more rows, keeping the chrome reserve so it
still fits in one viewport (single scroll).
- Add a clear scroll affordance: a persistent (styled) scrollbar plus a
'scroll for more models' fade cue at the bottom that hides once the
list is scrolled to the end, and a caption note that the GT row stays
pinned.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: widen gallery chrome reserve to kill residual outer scroll
Michael Rabinovich+1: Cursor
5fb80c6
Live verification showed a constant ~56px outer-page overflow on smaller
screens (real HF chrome is a bit taller than the harness estimate). Bump
CHROME_RESERVE 560->620 and lower the box floor so the whole gallery fits
the viewport across screen sizes (single scroll).
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: gallery scroll box sized off screen.availHeight
Michael Rabinovich+1: Cursor
a435f45
Verified the real cause with a headless harness reproducing HF's nested
iframes: HF auto-resizes the Space iframe to full content height, so
inside the gallery window.innerHeight / parent.innerHeight / vh are all
inflated to the content height (live: 1642 vs a 900 viewport) and only
the cross-origin top window scrolls -- which is why no inner cap or
sticky-against-the-iframe ever worked and the GT row scrolled away.
screen.availHeight is the one viewport-ish measure that survives the
nesting, so size the gallery's own scroll box from it (availHeight minus
a chrome reserve, clamped). The column header + GT row are sticky inside
that box, so they stay locked while the submission rows scroll, and the
box is the single scroller (harness confirms no outer-page overflow
across laptop/desktop/large screens).
Co-authored-by: Cursor <cursoragent@cursor.com>
metrics: tighten Metrics page copy
Michael Rabinovich+1: Cursor
2a0d6fb
Fix the "Editing tasks" anchor landing short (bottom spacer), drop the
per-section "Full derivation" links in favor of a single docs/metrics.md
pointer at the end, trim wording (Boolean kernel, "(pieces)", "(e.g.
through-holes)", "not the mean", the report-overlay note), and remove
em-dashes.
Co-authored-by: Cursor <cursoragent@cursor.com>
deploy: bump cadgenbench to b76a55a + ship metrics assets
Michael Rabinovich+1: Cursor
0c1ec25
Pins the Space to cadgenbench b76a55a (report title with submission
name, click-to-zoom images, metric pill deep-links) and copies the new
assets/ dir into the image so the Metrics page's mating-group WebP is
served by the /metrics-assets route.
Co-authored-by: Cursor <cursoragent@cursor.com>
New self-contained Metrics explainer (metrics_page.py) served at
/metrics and embedded in a Metrics tab: validity gate, the three axes,
editing renormalization, with formulas and the interface mating-group
illustration (vendored under assets/metrics/ via LFS, served by the
/metrics-assets route). Submit's merge path passes the submission name
and the /metrics base URL to the report generator so hosted reports
title themselves and their metric pills deep-link to the explainer.
Co-authored-by: Cursor <cursoragent@cursor.com>
fix(oauth): use real HF OAuth on this Docker Space instead of mock login
Michael Rabinovich+1: Cursor
f17ac64
Gradio's get_space() only returns truthy when SYSTEM=="spaces", which HF
sets on Gradio-SDK Spaces but not on sdk: docker Spaces like this one.
Without it, mount_gradio_app wired up Gradio's MOCK OAuth routes: they
never contact hf.co and log every visitor in as the container token's
owner (our HF_TOKEN account). Because that account is in CADGENBENCH_ADMINS,
this leaked the "Logout (michaelr27)" identity to all visitors and granted
admin to anyone who completed the (mock) login.
Set SYSTEM=spaces when running on a Space (SPACE_ID present) so the real
hf_oauth: true flow runs; left unset locally so the local mock login still
works for dev.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: make the gallery the single scroller
Michael Rabinovich+1: Cursor
5eaf6b8
Sizing the iframe to a fixed 80vh left two scrollbars (the iframe body
and the outer Gradio page), which was confusing. Now fitIframe measures
the available height from the iframe's own top down to the bottom of the
viewport (via the same-origin parent) and sizes the iframe to
min(content, available), reserving room for the Refresh button. Short
content shrinks the iframe (nothing scrolls); tall content fills the
viewport so only the iframe body scrolls and the sticky header + GT row
stay locked. The outer page no longer needs its own scrollbar.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: lock GT row via in-iframe scroll capped by max-height
Michael Rabinovich+1: Cursor
a0462d9
The previous attempt capped the gallery scroll box from JS using
window.parent.innerHeight, but inside HF's nested iframes that is not the
browser viewport, so the cap came out huge: the box never scrolled, the
iframe grew to full height, and the page scrolled the GT row away.
Instead, scroll inside the iframe itself (the column header + GT row are
sticky against the iframe viewport, which reliably works) and cap the
iframe with `max-height: 80vh` in app.py, where vh is the Gradio doc
viewport. fitIframe still shrinks the iframe to content for a compact
page when there are few rows; once content exceeds the cap the iframe
body scrolls and the GT row stays pinned. Drops the unreliable JS cap
and the parent-aware modal positioning (back to plain fixed centering).
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: lock GT row + header in a capped gallery scroll box
Michael Rabinovich+1: Cursor
5dd0c03
Make the gallery a scroll container capped at ~78% of the visible
viewport so the column header and ground-truth row stay pinned (sticky)
at the top while submissions scroll inside it. The cap is set from the
parent viewport in JS and the iframe is sized to the (bounded) content,
so the page stays compact and shows no scrollbar when the rows are
shorter than the cap.
Co-authored-by: Cursor <cursoragent@cursor.com>
leaderboard: fixed 4-sample gallery, auto-size iframe, center modal
Michael Rabinovich+1: Cursor
08eae45
Replace the gallery's interactive fixture picker with a fixed
comparison sheet: two generation + two editing samples, one Medium and
one Hard per task. Difficulty is chosen once from the Claude Opus 4.8
baseline's per-sample CAD scores (median = Medium, 20th percentile =
Hard) and frozen into FIXED_FIXTURES; the GUI no longer recomputes it
as the leaderboard evolves, trading a moving ground truth for a simple,
stable interface. Adds tools/pick_gallery_fixtures.py to regenerate the
constant.
Size the gallery iframe to its content (no more 90vh fixed box / nested
scrollbar) and pin the compare modal to the visible part of the iframe
so it opens centered in view instead of far below the fold.
Co-authored-by: Cursor <cursoragent@cursor.com>
deploy: bump cadgenbench to 72f6e82 (scale-relative edit-diff tol, viz polish)
