Every trading product has a backtest. Most of them look wonderful. Ours do too, on the data they were fit on. That is exactly the problem.
Fit on one stretch, judged on another
Every signal the floor might lean on is fit on one stretch of history and then judged on a later stretch it never saw. Dev, the out-of-sample tester, watches the gap between the two. For almost every idea we have tried, the gap is large: a Sharpe that looks like 1.5 in sample comes back nearer 0.5 out of it.
Deflated for everything we tried
Try forty variants of an idea and one of them will look good by luck. Priya, the multiple-testing statistician, discounts every result for how many things were tried before it. After that discount, nothing we have tested is distinguishable from chance, and nothing has beaten plain buy-and-hold.
The best honest annualised result we have measured out of sample is about 10.8%. Holding did at least as well.
Why we publish the zero
Because you would find out anyway. The validation gate is a screen on every client's desk, and the straight talk is on our home page, not in a footnote. A floor that argues in the open has to be honest about what it does not know.
What changes if one passes
Less than you would think. A validated signal can carry more weight in the floor's argument. It still goes through every limit and every veto, and on a client desk it still waits for the client's click.
Why our validation harness has certified nothing
Approve-first, in code, not in a setting
What the floor does on a bad day
Why the default broker is now Alpaca