2026-08-10 · Loop for Quant Research · 1

Deterministic and Probabilistic

Every system built around a language model has to answer one question before any other: which of its decisions are allowed to be a sample from a distribution? Answer it loosely and you get a system that cannot be debugged. Answer it well and you get a different problem, which is the subject of the second half of this note.

The loop

One iteration of the system this note is about runs a fixed sequence: form an economic hypothesis about stock prices in plain language, write code expressing it, replay sixteen years of market history against that code, score the result, decide whether the score means anything, write down what it learned. One pass is a trial; the replay is a backtest. Nobody is watching — over one campaign it completed 1,203 of them.

Two scores decide whether a trial is kept. The in-sample score is computed on the first fourteen years, which the loop is allowed to optimise against; the holdout score is computed on the last two, which the model never sees. Both are an information ratio (IR) — the strategy's return in excess of a market benchmark, divided by the volatility of that excess. The gate is the rule that reads those two numbers and returns a verdict.

What nondeterminism costs, and where

The obvious way to build a loop like this is to make everything an agent. A proposer agent, a coder agent, a reviewer agent, a judge. It is obvious, it demos well, and it produces a system whose failures you cannot debug.

The reason is worth stating precisely, because "nondeterminism is bad" is not the reason and is not even true. A sampled component is fine in isolation; you run it again and get a different answer, which is sometimes exactly what you want. The cost appears when sampled components are composed. A failure in a chain of them cannot be attributed, because re-running the chain does not reproduce the failure, and changing one link does not isolate it — the other links moved too. You lose the ability to say this component did the wrong thing, which is the ability every debugging technique is built on.

So the question is not how much nondeterminism to have. It is which decisions genuinely need it. And there is a clean test: a decision needs a model exactly when you cannot write down the rule. If you can state the threshold, the retry budget, the schema, the ordering — you have already made the decision, and handing it to a model means paying sampling variance for a choice you had made anyway.

That is one direction of failure. The other is less discussed and it is the one that actually bit me. The design principle at the top of the project's instructions names both:

Thresholds, retry budgets, chain orchestration, schema validation are scripts. Hypothesis generation, code writing, narrative, and intuition are agent. When the boundary is violated — determinism encroaching on judgment (rule-list bloat) or judgment absorbing determinism (counters carried in the agent's head) — the system degrades toward either rigid exploration or unreliable execution.

Read the second half again. Most writing on this subject warns only about judgment absorbing determinism — the agent asked to keep a counter in its head. The mirror case, determinism encroaching on judgment, is the one this note ends on, and it is harder to see because it does not look like anything.

Where the model is, exactly

Three places. The model writes a hypothesis in natural language; it writes the strategy code that expresses the hypothesis; and after the backtest it writes the causal story explaining the result. That is the entire surface.

Everything else is a script. I checked this rather than assuming it — grepping the two mechanical directories for any model invocation, SDK import, or subprocess call to a language model returns exactly one hit, and it is inside a docstring.

The pipeline runs a fixed sequence on every iteration. Each stage's decision rule is worth stating precisely, because precision is the point:

That last row matters. The analysis stage computes per-year returns, factor regressions, holdings statistics, correlations against previously banked strategies — and then makes no decision at all. Its own source says so: information only, no gate, no threshold; interpretation lives in the agent's methodology. Facts are cheap and mechanical. Meaning is not, so meaning is the model's job and only the model's job.

The agent does not count

The clearest test of whether you have drawn the boundary correctly is: who holds the counters?

When a trial crashes or turns out to be a duplicate, it gets retried. The budget is three attempts. That number does not live in a prompt. It is a module constant, and the count is written to a JSON file on disk on every single increment — which means it survives the agent's process dying, a session restarting, or the model simply forgetting. On the third failure the orchestrator emits a different status, and a separate script sees that status, synthesises a placeholder reflection, marks the trial abandoned, and moves on without ever asking the model what it thinks.

The outer loop is the same shape. It decides whether an iteration made progress by counting lines in the trial log — not by reading the agent's self-report, and not by trusting its exit code. The runner's own comment on that is blunt: the model's return code is advisory only. Three consecutive iterations with no new log line and the loop aborts itself.

I did not get this right the first time. An early version fired a consolidation job every N iterations, where N was counted in a shell variable that reset whenever the loop restarted. One campaign ran 279 trials and triggered that job twice. The model spent hundreds of iterations reading a stale snapshot of what had already been discovered, and nothing broke loudly enough to notice. A counter in a volatile scope does not throw; it just quietly starves whatever depends on it.

The firewall

There is one more thing the deterministic layer does that no amount of prompt instruction could achieve: it withholds information from the model.

Every strategy is scored on two windows — an in-sample period the loop is allowed to optimise against, and an out-of-sample period it is not. The out-of-sample numbers exist. The gate reads them. The model never sees them. A frozen set of keys is stripped from the record before it reaches the agent, along with anything whose name ends in _os, so a fourteen-year backtest and a two-year holdout collapse, from the model's point of view, into a handful of in-sample figures and one boolean.

This is not politeness. If the model could read the holdout score it would optimise against it — not by cheating, but by doing exactly what it is for, which is noticing what correlates with success. The only way to keep a holdout clean across two thousand queries is to make it structurally unreadable.

The gate that lies

Now the part worth the reader's time.

The duplicate check is meant to stop the loop degenerating into a parameter sweep. A research loop that can tweak a lookback from 126 days to 252 days and call it a new experiment will do that forever, because it is easy and it occasionally works. So the gate blocks a trial whose column set overlaps a previous trial by 80% or more and whose scoring function is structurally identical.

Three separate places in the codebase describe this gate as blocking pure parameter tweaks. One of them is the project's own architecture documentation.

It does not block them. The structural comparison is a serialisation of the function's syntax tree, and that serialisation includes the value of every numeric literal. Change 126 to 252 and the serialisation differs, the equality check fails, and the verdict comes back novel. Same columns, complete overlap, one digit changed — and the loop treats it as new science. Renaming a local variable defeats it too.

So it is a copy-paste detector documented as a parameter-sweep detector. That would be a minor wart, except for what was built on top of it. An earlier version of the pipeline had a second mechanism — a lineage classifier that tracked whether each trial was a fresh idea or a refinement of a previous one, plus a plateau rule that intervened when refinements stopped paying. Both were deleted. The stated reason for deleting them was that they were redundant, because the duplicate gate blocks parameter tweaks upstream.

It does not. The removal rests on a premise the code does not satisfy.

I want to be precise about the size of this. In practice the loop does run almost entirely on structural novelty — the same commit that removed the plateau rule noted that across 391 accepted trials the classifier had fired zero times. The mechanism was removed because it was never triggering, and it was probably never triggering because the model was not, in fact, trying to sweep parameters. So the practical damage is likely nil. But the reasoning was wrong, and the reasoning is what I would have relied on if the model's behaviour had drifted.

This is the first of the two failure directions the principle warns about, and it is the more insidious one. Rule-list bloat is visible: you can see the rules pile up. Determinism over-claiming looks like nothing at all. A gate that quietly does less than its docstring says is worse than no gate, because you stop watching the thing it was supposed to be watching.

Two smaller ones, for calibration

The retry budget is keyed on the trial's file number. The remedy the agent is told to apply after a duplicate rejection is to revise and rerun — and the next trial number is defined as one above the highest existing file. So writing the revision as a new file resets the counter to one, and the cap of three never binds. The only real backstop is the outer no-progress check, which measures committed results rather than attempts.

And there is a script making a judgment call it should not. A regex flags a strategy as long/short if it finds a negative numeric literal after an equals sign, an opening bracket, or a comma. A perfectly ordinary .clip(-0.9, 0.9) trips it. The project knows: the look-ahead linter's source cites this regex by name as the cautionary precedent for why its own checks were scoped narrowly to specific methods and arguments. The lesson was learned, written down, applied to the new code — and never applied back to the old code.

What the discipline actually buys

Reproducibility, mostly, and a debuggable failure surface. When an iteration goes wrong I can tell within a minute whether the problem is in the third of the system that thinks or the two-thirds that counts, because the two-thirds that counts produces the same output for the same input every time.

But the sharper benefit is that it forces you to say what you actually believe. Writing a threshold into a script requires committing to a number. Writing it into a prompt lets you write "reasonably good", and a system full of reasonably-good is a system nobody can check — including you.

The corollary is the part people skip. If the deterministic layer is where your commitments live, then the deterministic layer has to be right, and it earns exactly as much trust as it has been tested for. Three docstrings agreeing with each other is not a test. I found the duplicate gate's real behaviour by running it on two files that differed by one digit, which took under a minute, and which nobody had done — because the code looked obviously correct and the comment above it was reassuring.

Put the agent only where you must. Then go and check that the rest of it does what you think.