Comparing runs

A useful comparison answers a specific question: what changed in behavior when one input changed? Record the baseline before editing and name the intended difference in the next run.

Keep a run record

Use this as a note template; it is not an importable app configuration or an API payload:

Question:
Baseline run / source revision:
Candidate run / source revision:
Instrument and contract:
Data interval, start, end, timezone:
Dataset identity or saved-data reference:
Strategy settings and configured models:
Capital, quantity, fill delay, session filters:
Runtime / simulation path:
Intended change:
Observed signal and execution differences:
Unexplained differences:

For app-editor tests, save the source and settings alongside the result. For Agent Lab, retain both the run ID and the associated revision. Comparing the current editor buffer with an older result can pair the wrong source and outcome.

Dataset identity is only one dimension

Within an Agent Lab experiment, the dataset identity covers instrument, interval, date range, and timezone. It does not include source, quantity, capital, fill delay, or every execution setting. Matching dataset IDs are useful evidence of a shared bar selection, not proof that two runs are otherwise equivalent.

Source-only revisions reuse the stored experiment bars after the dataset artifact is established. This improves comparisons but also means those runs are not fresh downloads from the provider. Additional service/history calls made by a strategy need their own reproducibility assumptions.

Compare the crossover in stages

1. Repeat the same source and settings to establish whether behavior is reproducible.
2. Change quantity while retaining the same signal code and data. Inspect both requests and capital/execution eligibility.
3. In a separate comparison, change fill delay. Inspect how pending orders alter subsequent entry and exit decisions.
4. In another comparison, add optional L3 oversight and identify the requests its gate rejects.

Do not combine all those edits into one candidate if you want to explain which change caused the result. A higher P&L alongside fewer records still needs an execution-level explanation.

Compare decisions before totals

Find the first timestamp where signals, requests, or outcomes diverge. A difference in initial warmup can propagate through the entire position sequence. A difference in fills can cause the same later signal to see a different position.

Also compare the selected range's end: run-end close attempts and lack of later fill opportunities can change the final records. Extending the end date is an experimental change, not merely a larger chart view.

Keep development and later evaluation separate

After revising source against one historical window, evaluate the saved revision on other windows without changing it in response to each result. Label exploratory tuning and subsequent evaluation clearly. The current comparison metrics do not automatically implement out-of-sample splits, walk-forward analysis, or statistical confidence estimates.

Continue with Limitations, or return to Reading results.