Cost of the scheduled phase
Pausing the bottom half each turn cost 57% of running all eleven turns. Paused runs resumed later from their checkpoints.
checkpoint → pause → resumeI went through the public run feed: 419 completed runs, $11,794 of metered compute. 30% of that spend comes after a run's last public improvement, and the share is about the same for every model. Below you can replay pause rules on those same runs, ask a bounded agent about the work, and read the full write-up.
Each run's pick is its best-public submission. "Runs worse" counts runs whose held-out pick would have been lower had the run stopped at the pause. Cost is spread evenly over each run's wall time, because the feed has one cost total per run.
Loading 419 runs…
Pause is not kill. With checkpoint and resume, a wrong pause costs delay, not the result, so the "runs worse" pool is what a second decision (which paused runs deserve a resume) has to recover.
A per-visitor Durable Object binds the actor, scopes, budget, ordered event log and expiry before the first model token is spent. Three turns, five minutes.
9,710 checkpoints, one per scored submission. The run improves again after 77% of them. Predictors are scored leave-one-board-out.
| Predictor of "will improve again" | AUC |
|---|---|
| Submissions since the last gain (a rule) | 0.61 |
| Logistic regression on 6 progress features + model | 0.68 |
| Gradient boosting on the same | 0.73 |
| Policy | Saved | Runs worse | Held-out kept |
|---|---|---|---|
| Pause after 3 with no gain | $4,888 · 41% | 125 | 94.1% |
| Pause after 8 with no gain | $2,063 · 17% | 49 | 98.8% |
| Gradient boosting, pause when P(more gains) < 0.25 | $1,326 · 11% | 47 | 98.8% |
| Gradient boosting, < 0.35 | $1,824 · 15% | 82 | 98.1% |
Four pi + GLM variants, three turns each, one speed-up task. The grader ran outside the agents' sandboxes, and held-out scores went to a sealed file read only after every worker stopped.
Pausing the bottom half each turn cost 57% of running all eleven turns. Paused runs resumed later from their checkpoints.
checkpoint → pause → resumeThe cheapest variant led on public inputs and scored 51x held-out. Two of its later turns were wrong on held-out inputs the public set could not see.
public ≠ held-outThe best held-out run was paused at turn 2. Running everything and picking by public score misses it too.
pick = argmax(public)A demonstration, not a statistic. Public-only pausing is only as good as the public set; a broader set or a checkpoint judge is what to test next.
n = 4 variantsThe runtime the live run used, and the harness work behind it.
Coding agents in microVMs with credentials swapped in at the host boundary. Candidates are graded in a separate microVM on a locked copy of tests and CI config, so an agent can't edit its way to a pass.
github.com/vu1n/pillbox ↗Reference values in a store the agent side cannot write, harness changes only through pre-registered gates. False successes went from 15 to 0; a method reconstructed from a paper did not beat the baseline, reported that way.
Resume ↗Checkpoint, copy and resume an agent session across a microVM, a container or a Cloudflare Sandbox. Each turn gets one sealed packet, so blind workers see only their own task and score.
Demo reel ↗Meta, MetaMask, Squarespace and Phantom, then Pillbox, Stratus and Onoda. Not an ML researcher; I'd be learning the training side. Happy to move to SF.