All notes
EngineeringOct 20, 2026

Why our validation harness has certified nothing

Ten signals tested out of sample, deflated for everything we tried. None beat holding. Here is how the gate works, and why we publish the zero.

A cluttered desk late at night under a green lamp

Every trading product has a backtest. Most of them look wonderful. Ours do too, on the data they were fit on. That is exactly the problem.

Fit on one stretch, judged on another

Every signal the floor might lean on is fit on one stretch of history and then judged on a later stretch it never saw. Dev, the out-of-sample tester, watches the gap between the two. For almost every idea we have tried, the gap is large: a Sharpe that looks like 1.5 in sample comes back nearer 0.5 out of it.

Deflated for everything we tried

Try forty variants of an idea and one of them will look good by luck. Priya, the multiple-testing statistician, discounts every result for how many things were tried before it. After that discount, nothing we have tested is distinguishable from chance, and nothing has beaten plain buy-and-hold.

The best honest annualised result we have measured out of sample is about 10.8%. Holding did at least as well.

Why we publish the zero

Because you would find out anyway. The validation gate is a screen on every client's desk, and the straight talk is on our home page, not in a footnote. A floor that argues in the open has to be honest about what it does not know.

What changes if one passes

Less than you would think. A validated signal can carry more weight in the floor's argument. It still goes through every limit and every veto, and on a client desk it still waits for the client's click.

A small number of clients at a time.

Tell us a little about you. We reply personally, usually within a few days. If it is a fit, the questionnaire comes next, and your account after that.

Already a client? Sign in to get your install line