2026-08-10 · Loop for Quant Research · 3

Overfit and Dynamic

A search loop needs something to optimise against. In quantitative research that something can only be built out of the past, and the thing the past describes does not hold still. This note is the chain of reasoning that starts from that mismatch and ends, several steps later, at a slime mould — and then at the specific point where the borrowed mechanism stops working.

The loop, and the object it verifies

Start with the thing being optimised, because everything below is a consequence of what it is. A strategy is a function that decides, for each date, how much of a portfolio to hold in each stock. A backtest replays history against it: on every simulated day it applies the weights that strategy would have chosen using only information available on that day, and accumulates the returns that would have followed. Out comes a single equity curve, and from the curve, scores.

Writing those strategies is automated. A hypothesis is stated in plain language, compiled into code, backtested, scored, and the reason it worked or failed is written down for the next attempt to start from. One pass is a trial; a campaign is roughly a thousand of them, unattended.

This is a genuinely good property, and it is worth pausing on how unusual it is. Most research does not get a verifiable object. Here you do: the hypothesis is compiled into code, the code is executed against a fixed dataset, and a number falls out. Two people running the same strategy over the same window get the same number. The loop can be automated at all only because of this — there is something for a script to check.

A few of the scores, since they recur below. In-sample is the stretch of history the search is allowed to optimise against; the holdout is a later stretch kept back, whose numbers the model never sees. The information ratio (IR) is the strategy's return in excess of a market benchmark, divided by the volatility of that excess — how much the strategy beat the market by, per unit of the risk it took doing so. The gate is the pass/fail rule that decides whether a trial is kept, and it is two IR comparisons and nothing else.

The object is verifiable, and it is also dynamical

Here is the mismatch the whole design is built around.

The object you verify is a simulation of a fixed past. The thing you are trying to say something about is the future of a market. And a market is not a fixed system being sampled — it is a system whose regularities are consumed by the act of being exploited. A relationship can exist in 2014, become widely known, be traded away, and stop existing by 2019, with nothing announcing the transition. The verifiable object is a photograph of something that moves.

That interacts badly with automation. Run a few thousand attempts against sixteen years of daily returns and some will look excellent for no reason at all — this is ordinary multiple comparisons, and every field that searches has it. What makes the quantitative case worse is that the two problems disguise each other. A strategy that fit noise and a strategy whose regime ended produce the same evidence: a good historical score and a disappointing future. Looked at from any single point in time, overfitting is not merely easy, it is the default outcome of trying hard.

The obvious responses are statistical — correct the threshold for the number of trials, hold out more data. Both help. Neither is sufficient, because they treat the score of an individual trial and the problem is not there. The problem is where the trials go. A greedy search concentrates: once a region of the space pays off, every subsequent proposal drifts toward it, and the loop spends its budget re-measuring one idea under different names until some measurement clears the bar by luck. Tightening the bar does not stop this. It just requires more attempts, which an unattended loop is happy to supply.

So: define the problem as explore, not exploit

That reframing is the actual design decision, and everything mechanical follows from it.

If the target were stationary, exploitation would be correct — find the good region, work it out. Because the target is dynamical, the durable output of a search is not a strategy, it is a map: which parts of the space have been looked at, which paid, which are closed. A strategy decays. A map of where you have already been does not, and it is what makes the next campaign cheaper than the last. So the loop is built to cover ground, and accepts a worse hit rate to do it.

But pure exploration is also wrong. A loop that samples uniformly from a space this large learns nothing from its own results and wastes most of its budget. Some exploitation has to be nudged back in — enough to bias the next proposal toward ground that has recently paid, not enough to let it settle there.

Two constraints on where that nudge could live. It had to sit at the data layer — operating on which inputs the next trial reaches for, rather than on the hypothesis itself, because the hypothesis is the part that must stay the model's judgment. And it had to be shallow: a hint the proposal step is free to ignore, not a controller. A deep optimiser over trial history would be one more thing fitting the past, which is the problem we started with.

And the nudge itself has to decay

A static memory of what paid would reintroduce the whole problem one level up. If a region stays warm because it paid in 2015, the loop is being steered by a fact about a market that no longer exists, and the steering gets stronger the longer the run goes, because early successes accumulate and nothing removes them. Any intelligence layer over a dynamical target has to forget, or it becomes a record of the target's history rather than a guide to its present.

So the requirement, written out: a shallow, non-binding, data-layer preference that strengthens where results arrive and weakens everywhere else, continuously, with no explicit expiry rule. I did not want to invent that. Decay-based coordination is old and much-studied, so the question was which existing mechanism has exactly these properties — shallow, local, no central plan, and forgetting built into the substrate rather than bolted on.

Which is what a slime mould does

Ants and slime moulds solve this with stigmergy: coordination through marks left in the environment rather than a central plan. A forager deposits a trail on a productive path, other foragers are biased toward strong trails, and the trail evaporates. Evaporation is the whole trick. Without it the first path found becomes permanent and the colony stops looking. With it, a path has to keep paying to keep existing, and quiet ground cools back down and becomes explorable again.

Every property on the requirement list is there, and none of them is an add-on: the trail is shallow (a scalar, not a model), local (deposited where the forager actually went), non-binding (a bias, not a route), and it forgets by default rather than by rule. That last one is the reason to borrow this rather than build a decaying cache — in stigmergy, persistence is the thing that must be earned, and absence is the resting state.

The translation, and where it is exact

The loop implements a direct translation. Every visited point in the search space carries a score — call it a pheromone level — and when the model is about to propose the next experiment, it asks the map for a suggestion. The mechanics are worth stating exactly, because the interesting properties all live in the details:

• Every committed trial multiplies all region scores by 0.95. That is the evaporation constant. Note the unit: it is per trial, not per unit of time.
• Only a trial that passes the quality gate thickens its own region. Merely running to completion earns nothing.
• Any region whose score falls below 1e-4 is deleted outright, rather than lingering as a near-zero attractor.
• 25% of the map's suggestions are a forced jump. Instead of naming a warm region, the map returns the twenty busiest data columns in the run's history and instructs the agent to try something outside that set.

The second bullet was not the original design. An earlier version also warmed a region for any trial that merely completed, and the reason recorded for removing that is the clearest sentence in the module: it "turned the field into a recency ramp instead of a productivity map."

That distinction is the difference between a working stigmergy and a decorative one. If activity deposits pheromone, the map converges on wherever the agent has recently been — a function of the agent's own attention, not of the world. The map then tells the agent to go where it has been going, and the feedback loop closes with no information entering from outside. Only paying for results keeps the map measuring something external. A productivity map can go cold everywhere, which is uncomfortable and correct; a recency ramp never can, which is comfortable and useless.

The suggestion is also explicitly non-binding. The module's own description calls it "a soft pull, not a command: the thesis still leads. An empty map means propose exactly as before." That matters for the division of labour — the hypothesis is still the model's job, and the map is one more input to it, not a controller.

A map, and a picture of a map

There is a structural point here I got wrong in my own head for a while, and it is worth being unsentimental about. A region's identity is a hash: a SHA1 over the sorted set of data columns the strategy touches, plus a serialisation of its scoring function's syntax tree, truncated to sixteen hex characters. That is a perfectly good lookup key. It is not a position. Two regions any researcher would call adjacent — the same idea with one extra input, or the same inputs combined slightly differently — land at unrelated points in the hash space. No metric, no neighbourhood, no gradient.

Which means "explore near a warm region" is not a computable instruction here. It exists only as prose advice to the model, which can reason about proximity because it understands what the columns mean. The map can say this exact configuration paid and these columns are over-worked. It cannot say go two steps left.

A human looking at the run sees a force-directed graph of these regions, and it looks like a landscape. It is not one; the layout is an artefact of the drawing algorithm and distance on that screen encodes nothing. Real stigmergy is spatial — the ant's trail is embedded in the same metric space the ant walks through, which is why following a gradient works. Strip the space out and you keep the evaporation and lose the navigation. What remains is a decay-weighted table of visited configurations: genuinely useful, and considerably less than the picture suggests.

The overfitting defences are structural, not statistical

None of the anti-overfitting machinery in this system is a significance correction. All of it works by removing choices from the agent.

The evaluation window is frozen in the execution slot, not chosen by the trial: sixteen years, split into a fourteen-year in-sample block and a two-year holdout. Whatever dates a strategy file declares, the runner ignores them, so every trial is measured on the same ground and none can quietly pick a flattering period.

The holdout numbers are then stripped out of what the model sees. A frozen key set is removed from the result record, along with anything whose name ends in _os, so a two-year holdout reaches the model as exactly one boolean. The gate reads the real numbers. The model reads pass or fail. You cannot fit to a bit you receive once per trial, or at least you can fit to it only very slowly.

A third component is often mistaken for another defence, and its own docstring goes out of its way to say it is not. A hook intercepts writes to strategy files and rewrites any date literal beyond the data ceiling. That is not a leak guard; it only blocks references to dates beyond available data. Nothing in the code prevents a signal from referencing the holdout span. Agent discipline does that, and only agent discipline.

Forced novelty, with teeth

The 25% jump is a nudge — the map suggests, the model decides. In one mode the jump is enforced: the loop samples some number of previously untouched data columns, pins them to a file, and the linter rejects any trial that does not reference at least one of them. There is no reasoning around it; a strategy that stays on familiar ground does not run.

The part I find most interesting is a prompt instruction rather than a mechanism: in that mode the agent is told to skip reading the belief wiki entirely, because inherited beliefs anchor you onto already-mined ground and the point is to leave it. It is deliberate, scoped amnesia — the accumulated intuition that makes the loop efficient is exactly what makes it conservative, so the exploration mode switches it off. Every other mode reads the wiki first.

Exhaustion, and how the loop noticed

Over one campaign of 536 trials, the frontier — the family of data the sampler draws from — was re-pointed three times as each family was mined out. The escalation pattern is more interesting than the fact.

The first family produced three consecutive near-barren offers, then a note that the remaining unmined columns were dominated by memo lines and non-equity entities, that the family was "approaching exhaustion", and a switch. Clean.

The second degraded far more slowly and was flagged repeatedly: at eight consecutive barren offers, then again at ten, then eleven, then at "zero non-dead columns" with an explicit escalation addressed to a human operator. The eight-consecutive flag sat unactioned for roughly forty trials. The reason is mundane and general: the escalation channel is prose in a markdown file, and nothing in the code reads it. A warning written to a document that only a model and a human ever open gets acted on when one of them looks.

The third reached six consecutive fully-dead offer sets, then seven. All three families ended exhausted. The run terminated on budget rather than on target — the goal was fifty qualifying strategies and it banked seven.

Here is the part that surprised me: there is no threshold constant anywhere in the code for any of this. No maximum-consecutive-barren, no exhaustion rule, nothing. Every re-pointing decision was made by the model counting barren offers in prose, comparing the pattern against a named earlier precedent, and then either editing the plan file itself or escalating to a human. For a system whose stated principle is that counting belongs to scripts, that is judgment absorbing determinism — the counter carried in the agent's head.

Twice the log names the missing feature precisely, which is the most useful thing in it. The column sampler lacks a population check at the trading universe's size, and lacks a screen against already-rejected sibling names, so it keeps offering columns that are empty for the stocks actually traded and columns from families already closed. The loop diagnosed its own tooling gap correctly and repeatedly, and again, nothing downstream reads a diagnosis.

Did any of it work

Hit rate, meaning the fraction of trials that clear the gate, across four campaigns: 11.36% over 679 trials, 7.08% over 1,090, 4.43% over 1,246, and 1.31% over 536 — the last being the forced-novelty sweep.

Note the ordering. The mode that forces exploration hardest has the lowest hit rate, by a factor of five against the least constrained one. I would rather state that plainly than bury it: coverage costs yield. A sweep mechanically prevented from returning to productive ground spends most of its trials on ground that is not productive, which is what "unexplored" means. Its output is not strategies, it is a map of where not to look, and 529 failures is a reasonable price for knowing three data families are closed.

A related number I find more reassuring than any hit rate: one campaign's belief archive holds 233 blocks marked refuted against 169 marked as having hit a ceiling, alongside 492 one-line rejection bullets. Refutation outnumbers confirmation. A search that mostly kills things is working; the failure mode to fear is the archive that fills up with successes.

The forward record

The output of one campaign is a composite of six strategies, with mean pairwise return correlation 0.26 between members. Three windows:

Every figure below is gross. This framework has no slippage model and no borrow cost, and the fee parameter was left at zero, so nothing here is net of trading. The book is long-only and rebalances monthly, which is the cheap end of the range — but the haircut has not been measured, and an unmeasured haircut is not a small one, it is an unknown one.

Read the live column carefully, because it is the column designed to mislead you. A Sharpe of 2.79 looks superb and means almost nothing here. The book runs at a beta of 1.70, the alpha t-statistic is 0.0024, and 85 trading days cannot separate skill from luck under any test. The entire live return is market exposure — the composite went up because the market went up, levered. Of the three t-statistics, the in-sample 3.57 is the only one that clears anything, and it is in-sample, which is where a t-statistic is worth the least. The honest summary is that the holdout is consistent with the in-sample result without confirming it, and the live record is too short to say anything at all.

Where the analogy breaks

Everything above is the stigmergy working. Here is what it does not do — and the failures are one failure seen from different sides.

The map fades per trial, not per calendar day. A region found six years ago and one found last week decay at the same rate, and both decay because other trials happened. Elapsed time does not enter the calculation. In a slime mould that is fine: the maze is not going anywhere, and the only reason to forget a path is that better paths exist. Here, staleness in the world is what you most need to represent, and it is the one thing the field cannot express.

The scoring function is frozen for the life of the run. Sixteen years of one particular mix of regimes is the target being optimised against, unchanged from the first trial to the last. Whatever a strategy learns, it learns about that mixture.

The gate is regime-blind. Two scalars over two fixed blocks: no per-year stability requirement, no rolling consistency test, no structural-break test anywhere in the gate path. A strategy that earned everything in one eighteen-month stretch and nothing since passes identically to one that earned steadily throughout.

Regime appears as diagnosis, never as control. The analysis stage computes a per-year return fingerprint, the model is asked whether it matches the story it told, and the model can log a regime tag. Nothing downstream reads those tags. No region cools because a regime tag changed; no proposal is discouraged because the regime it depends on has not appeared in four years. The system can say "this worked because rates were falling" and then do nothing whatsoever with the sentence.

The one genuinely adaptive component sits outside the loop. A drift monitor, with watch and explain thresholds, runs on deployed books. It is real and it works, and it monitors a portfolio that has already been chosen. It cannot re-point the frontier, fade a region, or veto a thesis. Adaptation here exists strictly downstream of search.

And the consequence, observed rather than predicted: when the pheromone had fully evaporated — two warm regions, both under 0.19, across 535 trials — the loop did not re-explore. It kept drawing dead columns from a sampler that, by its own logged complaint, was missing the two screens it needed. An empty map is supposed to mean "propose freely". An empty map plus a broken sampler means "propose from a bad list, forever", and nothing in the design distinguishes those two states.

One word for two things

The pheromone evaporates, but the maze never changes shape. That is the whole limitation, and it is not a bug in the implementation — it is inherited from the organism the design was copied from. A slime mould's forgetting solves congestion: do not let the colony pile onto the first adequate path. It does not solve obsolescence, because in a maze nothing becomes obsolete. A market does, constantly. So the loop's map decays over trial count while the thing it maps decays over calendar time, and those two clocks are unrelated.

Which leaves the loop unable to make one distinction it needs badly. When a region stops paying there are two possible reasons: the idea was never right and the measurement finally caught up with it, or the idea was right and the regime it depended on ended. Those call for opposite responses. The first says abandon the thesis. The second says keep it and wait, because the conditions will come back.

The loop has one word for both, and the word is dead.