2026-08-10 · Loop for Quant Research · 5

Look-Ahead Bias and Path Dependency

There is a class of program whose output you cannot check, because you have no idea what the right answer is. A simulation of the past is one of them. This note is about what testing looks like when the usual move — assert that the output equals the expected value — is unavailable, and about the two bugs that survived the first attempt.

The loop, and the thing it cannot check

The program in question is a market simulator. Feed it a strategy — a function that says, for each date, how much of a portfolio to hold in each stock — and it replays sixteen years of history against that function, applying on every simulated day the weights the strategy would have chosen from the information available on that day, and accumulating the returns that would have followed. Out comes a score. Around it sits a loop that writes strategies, runs them through, keeps the ones that clear a bar, and repeats without supervision.

Everything in that arrangement rests on one clause: using only the information available on that day. Violate it and the machine still runs, still produces a score, and the score is better. There is no crash, no exception, no anomaly — a simulation allowed to see the future is not a broken simulation, it is a very successful one. The failure mode of this program is to look like it is working unusually well.

You cannot state the expected output

Here is the testing problem, and it is not specific to finance.

An ordinary unit test names an input and the output it should produce. That requires you to know the answer independently of the program. For a simulation of sixteen years of market history, you do not. Nobody has the correct portfolio weights for an arbitrary Tuesday in 2014 sitting in a fixture file; if such a thing existed you would not have needed the simulator. The program is the only source of the answer it is being asked for. Testing software in this position — scientific simulations, compilers, search engines, machine-learning pipelines — runs straight into what the testing literature calls the oracle problem: there is no oracle to compare against.

Reading the code instead is a losing game, and worse than it looks. The strategy is a small expression sitting on a large stack, and the future can enter through any layer of it: a loader that rebuilds its data panel out to the end of whatever window you asked for, a fill rule, a universe assembled with hindsight, a helper three call frames down that reindexes in the wrong direction. None of those live in the thing the researcher thinks of as "the strategy". The person best placed to review the logic is structurally unable to see the leak, and in this framework is not even permitted to read the layers where it lives.

So state a relation instead of a value

The move that works when you cannot name the correct output is to name a relation between outputs. Run the program twice on inputs that differ in a controlled way, and assert how the two results must relate — not what either one is. You never need to know the right answer; you only need to know how the right answer would have to change, or fail to change, under that particular perturbation.

This is metamorphic testing, and it is the standard answer to the oracle problem. The familiar example is a search engine: you cannot say what results a query should return, but you can say that adding a restrictive term must not increase the number of hits. The assertion is on the relation, and it holds without anyone knowing the correct result set.

So the question becomes: which perturbation, and which relation? For a causal simulation there is an obvious one. The property we actually care about is causality — a weight assigned on a given date must be a function only of information available on that date. Restated as a metamorphic relation: deleting data from after that date must not change the weight. That is checkable without knowing what the weight should have been, and without understanding a single line of the stack it came from.

A second perturbation falls out of the same reasoning. If the weights at time t depend only on a bounded window ending at t, then where the simulation started ought not to matter once the windows have warmed up. Two runs beginning on different dates must converge on the stretch they share.

Two relations, two perturbations, one program treated as a black box throughout. The rest of this note is what happened when they were implemented.

Three runs of the same strategy

The check runs the same strategy function three times with nothing changed but the window.

• B, the baseline, over the full window.
• A, identical to B except that the last 60 trading days are cut off the end.
• D, identical to B except that it starts 250 trading days later, ending on the same date.

B against A detects look-ahead. A and B start on the same day and see the same inputs up to A's end. The only difference is that B was additionally shown the final 60 days. If the weight on some earlier date depends only on information available on that date, deleting data from after that date cannot change it — so every cell the two runs share must be bit-identical. A nonzero difference is not a warning sign or a statistical anomaly; it is direct evidence that a weight was computed from its own future.

That is why the tolerance is 1e-10 rather than some reasonable-looking band. No legitimate mechanism produces a nonzero difference here, so the tolerance is not a judgment about how much error is acceptable — it is a declaration that only floating-point noise is admissible and anything else is a defect.

B against D detects path dependency. D begins 250 trading days later and ends on the same date. If the strategy is stateless — if the weights at time t are a function of a bounded window ending at t — then once D's rolling windows have warmed up the two runs must converge on the shared tail. The first 60 days after D's start are excluded as burn-in for that reason. If the two still differ after that, today's book depends on where you happened to begin the simulation, which is an artefact rather than an edge. The tolerance here is 0.01 on the mean absolute weight difference, and it is not zero for reasons that are the substance of the second half of this note.

The comparison grid, and the bug in it

Having decided what to compare, you have to decide where to compare it, and this is where the first version was wrong.

The obvious implementation intersects the two runs' rebalance dates and compares the rebalance matrices on the overlap. It reads as the careful choice — only compare what both runs actually produced — and it is the opposite of careful. A rebalance date present in one run and missing in the other does not mean "no comparison available"; it means the strategy held something different that day, which is precisely the evidence the check exists to find. And a rebalance goes missing for one specific reason: the signal came out all-NaN in one run and the row dropped. An intersection join discards exactly the rows where the evidence lives.

The consequence is the sentence this section is built around: the leaky test fixture passed because of that bug. The repository keeps a synthetic strategy that deliberately peeks twenty days into the future, whose entire job is to be caught. It was not caught. The check reported PASS on a strategy written to fail, and did so while comparing plenty of honest rows, because the dishonest ones had been filtered out of the join.

The fix changes what is compared, not how strictly. Take the union of both runs' rebalance dates and of their instrument columns, forward-fill each run's standing book onto that grid, and compare the positions actually held rather than the rebalance instructions issued. A date only one run rebalanced on is now a real row: one side shows the new book, the other shows what it was still carrying.

The incident is pinned by a regression test that does not touch the database. It builds a three-row frame, deletes the middle rebalance from one copy, and asserts that the comparison still reports three rows and a maximum difference of 1.0. Under the old join it reported two rows and no difference. Four lines of fixture and two assertions are the only thing standing between the current implementation and a silent return to a rubber stamp.

The measurement

Run against the deployed composite over a sixteen-year window, the three runs cover 4,085, 4,025 and 3,834 trading days and produce 196, 193 and 184 rebalances, compared across 7,237 instruments.

Twelve of those 181 path-dependency rows are dates the baseline rebalanced on and the late-starting run did not. Under the intersection join they would not have been rows at all.

Why the path tolerance is not zero

Because this engine holds positions between rebalances. A name not explicitly re-stated on a rebalance date stays in the book at its existing weight; only an explicit zero closes it. That one convention makes the book at any moment a function of the entire rebalance history rather than of today's signal.

D never takes the positions B took during the first 250 days. Anything B entered in that period and carried forward is simply absent from D's book for as long as B holds it. Stops and forced liquidations that fired on B-only positions never fire in D, because the positions were never opened. Demanding exact equality on the shared tail would fail every honest strategy that uses hold semantics, which in this framework is all of them.

The asymmetry with the other leg is not a double standard. Zero is defensible for look-ahead because no legitimate mechanism produces a difference; it is indefensible here because several do.

What is path-dependent here, concretely: unstated positions persist, and forgetting to write an explicit zero is the most common bug in the framework; positions carry between rebalances by forward-fill; a holding that becomes untradeable is force-liquidated at the previous bar's close, which the strategy did not request and cannot observe; and the strategy cannot read engine state at all — no realised fill price, no cash balance, no liquidation notice.

That last item is the argument for testing from outside, in one line. A strategy cannot compensate for path effects even in principle, because the interface does not let it see them.

The static half, and the form neither half catches

The re-run is one of two instruments. The other is a linter that walks the syntax tree before the strategy is executed and rejects four things: a change-or-difference computation with a negative period, a centred rolling window, backward-fill and interpolation, and a fill method of backfill. It separately bans manual shifting of the weight matrix — not because shifting looks forward, but because the engine already applies a one-bar execution delay and a manual shift would double it.

The two are scoped to each other's blind spots, deliberately. The re-run catches leaks of unbounded reach with certainty: a full-window statistic, an expanding window, a ranking along the time axis. Any of those changes when you delete the last sixty days, everywhere, so the comparison lights up. The linter does not attempt to catch time-axis aggregation at all, and its own comment says why: the reach is unbounded so the re-run is guaranteed to get it, and a syntactic rule broad enough to flag it would kill legitimate row-wise operations too.

The linter's job is the complement — finite forward windows, the ones that differ from a correct expression by a sign. And here is the strongest thing in the codebase, which the project states plainly in its own golden rules: the linter cannot catch a hand-rolled forward slice. A lag helper written with raw array slicing has a correct form that assigns from the past into the present, and an inverted form that drags the future into the present. The inverted form invalidates the entire backtest. It is not syntactically distinguishable from the correct one by any rule the linter applies — and the golden rules go on to say that the re-run cannot catch it either.

The stated reason is precise. A forward window of k bars leaves a footprint only in the final k bars of the run, because everywhere else there is real data on both sides. At that boundary the forward slice has nothing to read, so the value does not come out wrong — it comes out absent. The row becomes all-NaN, drops out of the rebalance set, and never enters the comparison. The leak disguises itself as a missing row, which is the one thing a comparison of values cannot see.

Either way the shape is the point: two instruments, each written with the other's blind spot in mind, and the residual gap recorded rather than papered over.

The other channels, briefly

Four more places the future could enter, closed by construction rather than by testing:

• Fundamentals carry a fixed four-month publication offset applied at load, so a filing becomes visible four calendar months after its fiscal date. Strategies must therefore not add a lag of their own, which is the other half of why manual shifting is banned.
• Execution is delayed one bar by the engine, and the fill price is the next bar's close. The strategy expresses intent; it does not choose a price.
• The evaluation window is fixed in the execution slot regardless of what the trial file asks for, so every trial shares one holdout and one stable universe. A strategy cannot quietly select a flattering period, because its declared dates are ignored.
• A pre-write hook rewrites any date literal beyond the data ceiling before the file is written to disk. Its own docstring insists it is a post-history guard and not an in-sample/holdout leak guard. That distinction is worth praising: a tool that overstates its own coverage is more dangerous than one that has none, because people stop watching the thing it claimed to be watching.

A check that detected the wrong thing

All three runs are pinned to a single fundamentals snapshot. That line looks like a configuration detail and is actually the most instructive part of the implementation.

Without the pin, the loader rebuilds its data panel out to each window's end. Vendors restate past figures. A restatement landing between the windows is applied retroactively to a past date in one run and not in another — so the same historical day carries different fundamentals in B and in A, the weights differ, and the check reports a look-ahead failure that has nothing to do with the strategy. This is not hypothetical. It happened, and the footprint was three cells across two income-statement columns.

Pinning one snapshot across all three runs fixes it. The price is exact: restatements themselves become undetectable, because now all three runs see the same restated values. A real form of historical contamination has been traded away for the ability to measure the one the check is about. That is the right trade, and it is only defensible if you say so.

Publishing what you did not check

So the checker emits, alongside its two verdicts, a list of what it does not cover. Two entries. Contamination uniform across the whole window — a window comparison sees only differences, so if all three runs are wrong in the same way the difference is zero and the check is blind by construction. And vendor restatement, for the reason above.

Marking either as PASS would be a lie. Omitting them would be a quieter lie, the kind that reads as a clean bill of health. So they go out in the result object next to the verdicts, are displayed wherever the verdicts are, and a regression test asserts the list is present rather than swallowed. A verification tool that ships a list of what it does not verify is doing the thing this whole series is about: the value of a check is bounded by how honestly its scope is stated.

One more leak, and it is not about time

A separate sweep applies one of the linter's predicates across every strategy file in the archive. The predicate looks for a cross-sectional operation — a rank, a median, a quantile taken across instruments — performed before the tradable universe has been applied.

This is a correctness bug, not a style preference. A cross-sectional rank is a function of the entire cross-section by definition, so including names you could never have traded changes the percentile of every name you could. Rank over four thousand instruments when three thousand are tradable and the denominator of your top decile is wrong; the book you build from it is not the book the thesis describes. Unlike a look-ahead leak this one has nothing to do with time — it is a leak of the untradeable — and it survives any amount of downstream masking, because masking the output does not repair a score already computed from the wrong population.

Close

The leaks that matter are not in the strategy. They are in the loader, the fill rule, the hold semantics, the vendor's revision history — layers the person writing the signal did not write, cannot see, and in this framework is forbidden from reading. That is not a flaw in the interface; it is what makes the interface usable. But it does mean no amount of care applied inside the strategy can establish that the strategy is clean.

So the check lives outside and imposes a property on the whole stack rather than inspecting any part of it. Two comparisons, two tolerances — one exact because nothing legitimate can move it, one a band because several things can. And then the part that is easy to skip: a written list of what the check does not cover, emitted with the verdicts, so that a PASS means the specific thing it means and not the general thing a reader would like it to mean.