Claude Code Autoresearch Loops for Prompts and Queries

Your evals already contain an optimizer

If you have a number you care about and a file that moves it, you can have an agent tune that file while you sleep. That's the whole trick behind Andrej Karpathy's autoresearch repo, and it's far more useful outside ML training than people assume. Most of us have at least one of these sitting in a repo right now: a system prompt with an eval set, a slow query with a benchmark, a bundle with a size budget.

I spent a weekend pointing Claude Code at a support-ticket classifier prompt using this pattern. It worked. It also tried to cheat, overfit, and celebrate noise. This post is the setup I ended up with and the guardrails I wish I'd started with.

How autoresearch is put together

The original repo is small on purpose. There's a prepare.py that handles data and fixed utilities, a train.py that holds a compact GPT model and its training loop, and a program.md written by the human. The agent is only allowed to edit train.py. Every experiment trains for a fixed five minutes of wall-clock time, then gets scored on validation bits per byte. If the score improves, the change is committed. If not, it's thrown away and the agent tries something else. Around twelve experiments an hour, roughly a hundred overnight.

Three design choices do most of the work, and they're the ones to copy:

• One editable file. The search space is bounded. The agent can't "improve" the score by rewriting the evaluator.
• A fixed budget per run. Every experiment costs the same, so comparisons are fair and the loop has a predictable pace.
• Git as the memory of what worked. Keep means commit, discard means reset. The history is the research log.

Karpathy has said his program.md is fairly rough, and he's right that it barely matters. The structure is the product, not the prompt.

The general shape: metric, lever, loop

Strip away the GPU and you're left with something any backend dev recognises. You need an objective metric you can compute without a human, a single lever the agent is allowed to pull, and a loop that measures, compares and either keeps or reverts.

People are already running this on marketing stuff like email copy, landing pages and ad creatives, using whatever API reports the conversion number. Those work, but they have a slow, noisy feedback loop measured in hours or days. Developer problems are often a much better fit because the metric is cheap and fast:

• An LLM prompt scored against a labelled eval set (seconds to minutes per run).
• A MongoDB aggregation pipeline scored by explain("executionStats") time on a fixed dataset.
• A Next.js route scored by build output size or a Lighthouse run.
• A SQL Server query scored by logical reads from SET STATISTICS IO ON.

The faster the loop, the more experiments you get per rupee and per hour. A five-minute loop gives you a hundred tries overnight. A loop that needs a day of traffic gives you seven tries a week, and at that point you might as well run the A/B test by hand.

A working example: tuning a classifier prompt

Here's the setup. I have a prompt that classifies support tickets into bug, feature, question or billing. I have around 400 labelled tickets split into a dev set the agent gets to learn from and a holdout set it never sees. The lever is prompt.md. The metric is accuracy.

ticket-classifier/
  prompt.md          # the only file the agent may change
  program.md         # instructions for the agent
  eval.mjs           # fixed scorer, agent never touches it
  loop.mjs           # driver: run agent, score, keep or revert
  data/dev.jsonl     # {"text": "...", "label": "bug"} per line
  data/holdout.jsonl
  .gitignore         # notes.md, results.tsv, failures.json

The fixed scorer

This is the prepare.py equivalent. It runs the prompt over a split and prints one JSON line. On the dev split it also writes the first 20 failures to a file so the agent has something concrete to reason about.

// eval.mjs  (npm i @anthropic-ai/sdk, needs ANTHROPIC_API_KEY)
import fs from "node:fs";
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();
const MODEL = process.env.EVAL_MODEL ?? "claude-haiku-4-5";
const LABELS = ["bug", "feature", "question", "billing"];
const split = process.argv[2] ?? "dev";

const system = fs.readFileSync("prompt.md", "utf8");
const cases = fs
  .readFileSync(`data/${split}.jsonl`, "utf8")
  .split("\n")
  .filter(Boolean)
  .map((line) => JSON.parse(line));

async function classify(text) {
  const res = await client.messages.create({
    model: MODEL,
    max_tokens: 10,
    temperature: 0,
    system,
    messages: [{ role: "user", content: text }],
  });
  const block = res.content[0];
  const out = block?.type === "text" ? block.text.trim().toLowerCase() : "";
  return LABELS.find((l) => out.startsWith(l)) ?? "invalid";
}

let correct = 0;
const failures = [];
for (const c of cases) {
  const got = await classify(c.text);
  if (got === c.label) correct++;
  else failures.push({ text: c.text, expected: c.label, got });
}

if (split === "dev") {
  fs.writeFileSync("failures.json", JSON.stringify(failures.slice(0, 20), null, 2));
}
console.log(JSON.stringify({ split, score: correct / cases.length, n: cases.length }));

It's sequential on purpose. Two hundred Haiku calls take a minute or two, which doubles as the fixed time budget. If you parallelise it, keep the concurrency constant across runs so timing doesn't drift.

The agent's instructions

<!-- program.md -->
You are improving prompt.md, a system prompt that classifies support
tickets into exactly one of: bug, feature, question, billing.

Each run:
1. Read results.tsv to see past experiments and their scores.
2. Read notes.md for hypotheses already tried. Do not repeat one.
3. Read failures.json for the current misclassifications.
4. Form ONE hypothesis about why the prompt fails on those cases.
5. Make one focused edit to prompt.md that tests it.
6. Append to notes.md: the hypothesis and what you changed, two lines max.

Rules:
- Edit only prompt.md and notes.md.
- Do not paste ticket text from failures.json into prompt.md.
- The model must still answer with the bare label only.
- Keep prompt.md under 60 lines.

The "don't paste failures into the prompt" rule is there because the first thing my agent did was add four of the failing tickets as few-shot examples. Dev accuracy jumped. Holdout didn't move. Classic.

The driver

You could run all of this inside one interactive Claude Code session and let it loop on its own. I prefer a dumb outer script that calls claude -p in headless mode once per iteration. Each run starts with a clean context, and the keep-or-revert decision lives in code the agent can't argue with.

// loop.mjs  usage: node loop.mjs 15
import { execFileSync } from "node:child_process";
import fs from "node:fs";

const ITERATIONS = Number(process.argv[2] ?? 10);
const MIN_GAIN = 0.02; // 4 tickets out of 200, below this is noise

const run = (cmd, args) =>
  execFileSync(cmd, args, { encoding: "utf8", stdio: ["ignore", "pipe", "inherit"] });
const score = (split) =>
  JSON.parse(run("node", ["eval.mjs", split]).trim().split("\n").pop()).score;
const log = (line) => fs.appendFileSync("results.tsv", line + "\n");

const program = fs.readFileSync("program.md", "utf8");
let best = score("dev");
const holdoutStart = score("holdout");
log(`0\t${best.toFixed(3)}\tbaseline`);

for (let i = 1; i <= ITERATIONS; i++) {
  run("claude", [
    "-p", program,
    "--allowedTools", "Read,Edit(prompt.md),Edit(notes.md)",
    "--max-turns", "10",
  ]);

  const changed = run("git", ["diff", "--name-only"]).split("\n").filter(Boolean);
  if (changed.some((f) => f !== "prompt.md")) {
    run("git", ["checkout", "--", "."]);
    log(`${i}\t-\trejected: touched ${changed.join(",")}`);
    continue;
  }
  if (changed.length === 0) {
    log(`${i}\t-\tno change`);
    continue;
  }

  const s = score("dev");
  if (s >= best + MIN_GAIN) {
    run("git", ["commit", "-am", `autoresearch ${i}: dev ${best.toFixed(3)} -> ${s.toFixed(3)}`]);
    log(`${i}\t${s.toFixed(3)}\tkept`);
    best = s;
  } else {
    run("git", ["checkout", "--", "prompt.md"]);
    log(`${i}\t${s.toFixed(3)}\tdiscarded`);
  }
}

const holdoutEnd = score("holdout");
console.log(`dev ${best.toFixed(3)} | holdout ${holdoutStart.toFixed(3)} -> ${holdoutEnd.toFixed(3)}`);

Start from a clean git tree with prompt.md committed. notes.md, results.tsv and failures.json are gitignored, so they survive reverts. That matters: a failed experiment is still a lesson, and you want the next run to know about it.

The git diff check is the real enforcement. The --allowedTools rule should keep Claude Code inside those two files, but permission syntax shifts between versions and I don't want my evaluator's integrity resting on a flag I'd have to re-check after every update. Run claude --help on your install to confirm the flag names. If anything outside prompt.md shows up in the diff, the whole iteration gets thrown out.

Where this goes wrong

Noise looks like progress

This is the big one, and it's where most write-ups on this pattern go quiet. A jump from 2.4% to 2.5% on a few hundred samples means nothing. On my 200-ticket dev set, one ticket is half a percentage point. Even at temperature 0 the model isn't perfectly deterministic, so before you start the loop, run the baseline eval three times and see how much it wobbles. I saw about one percentage point of spread. That's why MIN_GAIN is 0.02. Without it the agent will happily "improve" the prompt six times in a row on pure luck, and each lucky commit makes the next real gain harder to find.

The agent optimises the metric, not your intent

Goodhart's law shows up fast. My agent's second trick, after the few-shot stunt, was adding a rule like "if unsure, answer question", because question was the most common label in the dev set. Accuracy went up. The classifier got worse at the thing I actually cared about, which was catching billing issues. If you have a class that matters more, weight it in the score or track per-class recall and refuse commits that drop it. The holdout set is your only honest number. Look at it before you merge anything.

The notes file turns into sludge

The learnings file is what makes run 40 smarter than run 4. It's also an ever-growing chunk of context. After a few dozen runs it fills up with near-duplicate hypotheses, and the agent starts re-reading its own old ideas instead of thinking. Every 20 or so iterations I have a separate claude -p call condense it into a short list of what worked, what didn't, and what's still untested. Treat it like a log you rotate, not a diary.

Cost adds up quietly

Each iteration is one agent session plus one full eval. With a cheap model doing the scoring and a capped --max-turns, my 15-iteration overnight run stayed in the low hundreds of rupees. Swap the eval model for a large one, or let the eval set grow to a few thousand cases, and that number climbs quickly. Put a hard iteration cap on the loop, not just a time window.

Some problems just don't fit

If the metric needs a human to judge it ("does this explanation read well?"), you'll end up using an LLM as the judge, and now you're optimising one model's taste with another model. That can work, but the drift is subtle and you need a human spot-check on the winners. If the feedback takes days, the loop is too slow to beat a person making a thoughtful change. And if there's no API or file to change the input, the agent has nothing to pull.

What I'd actually use this for

My honest take: this isn't self-improving AI in any deep sense. It's hill climbing with a very capable mutation step. That's still a big deal, because the mutation step used to be you, at 11pm, guessing what to tweak next.

The places I'd reach for it first are prompts with a real eval set, query and pipeline tuning against a fixed benchmark, and config search where the space is too big to grid over by hand. In each case the expensive part isn't the loop. It's building an eval you trust. If you don't have one, start there. A good eval set is useful even if you never run the agent against it, and once you have one, the loop above is about a hundred lines of glue.

Run it on a branch, read the git log in the morning, check the holdout number, and keep the commits that make sense to you. The agent does the grinding. You still decide what ships.