VN Vu Nguyen Founding MTS / Emulated Resume
Founding Member of Technical Staff / Emulated

A third of the autoresearch bench spend buys nothing.

I went through the public run feed: 419 completed runs, $11,794 of metered compute. 30% of that spend comes after a run's last public improvement, and the share is about the same for every model. Below you can replay pause rules on those same runs, ask a bounded agent about the work, and read the full write-up.

Replay a pause rule

Pause a run after N scored submissions with no public gain.

Each run's pick is its best-public submission. "Runs worse" counts runs whose held-out pick would have been lower had the run stopped at the pause. Cost is spread evenly over each run's wall time, because the feed has one cost total per run.

Loading 419 runs…

Saved----
Held-out score kept--mean over runs
Runs worse----
Runs paused--can be resumed later
Share of spend saved vs held-out score kept, N = 1 to 15

Pause is not kill. With checkpoint and resume, a wrong pause costs delay, not the result, so the "runs worse" pool is what a second decision (which paused runs deserve a resume) has to recover.

pillbox / §0 · attributed session stream
not started

Ask about the analysis, the runtime, or the gaps.

A per-visitor Durable Object binds the actor, scopes, budget, ordered event log and expiry before the first model token is spent. Three turns, five minutes.

Write-up

Can the stopping point be called, and what is it worth in dollars?

9,710 checkpoints, one per scored submission. The run improves again after 77% of them. Predictors are scored leave-one-board-out.

Predictor of "will improve again"AUC
Submissions since the last gain (a rule)0.61
Logistic regression on 6 progress features + model0.68
Gradient boosting on the same0.73
PolicySavedRuns worseHeld-out kept
Pause after 3 with no gain$4,888 · 41%12594.1%
Pause after 8 with no gain$2,063 · 17%4998.8%
Gradient boosting, pause when P(more gains) < 0.25$1,326 · 11%4798.8%
Gradient boosting, < 0.35$1,824 · 15%8298.1%
  • The learned model ranks checkpoints better than the rule (AUC 0.73 against 0.61) but does not yet beat it in dollars. At about the same regret, pausing after 8 saves $2,063 and the model saves $1,326.
  • A controller that learns while it schedules falls into a one-sided-label trap: learning only from runs it kept, held-out score kept drops from 97.3% in the first third to 93.6% in the last. Resuming a random 15% of paused runs brings it back to 97.9%.
  • The controller only sees six progress numbers per checkpoint. Whether it can beat the rule when it sees the work itself (diffs, notes) is untested, because the feed does not carry them.
  • Limits: cost is assumed linear in time, policies are replayed on finished runs, and thresholds were not tuned on held-out boards.
Live run, small

Real agents, real pause and resume, a sealed grader.

Four pi + GLM variants, three turns each, one speed-up task. The grader ran outside the agents' sandboxes, and held-out scores went to a sealed file read only after every worker stopped.

57%

Cost of the scheduled phase

Pausing the bottom half each turn cost 57% of running all eleven turns. Paused runs resumed later from their checkpoints.

checkpoint → pause → resume
200x

Public leader

The cheapest variant led on public inputs and scored 51x held-out. Two of its later turns were wrong on held-out inputs the public set could not see.

public ≠ held-out
95x

Real winner, paused

The best held-out run was paused at turn 2. Running everything and picking by public score misses it too.

pick = argmax(public)
1 task

What it shows

A demonstration, not a statistic. Public-only pausing is only as good as the public set; a broader set or a checkpoint judge is what to test next.

n = 4 variants
The systems

Environments and graders that hold up.

The runtime the live run used, and the harness work behind it.

Systems and data for nine years. Agent runtimes and graders this year.

Meta, MetaMask, Squarespace and Phantom, then Pillbox, Stratus and Onoda. Not an ML researcher; I'd be learning the training side. Happy to move to SF.