Backtesting limitations

Historical testing is most useful when its assumptions are explicit. These are current behavior boundaries to account for when designing and reading an experiment.

Generic simulated execution

The shared memory fill path uses quote-close price checks. It does not reconstruct the exact order of intrabar high/low events, exchange queue position, or volume-driven partial fills. Fill delay is an elapsed-market-time eligibility check, not a complete latency or slippage model.

Capital and trade limits are simplified runtime controls. Do not equate them with a broker's full margin, financing, or instrument-specific execution requirements. Do not infer transaction costs merely from a metric named “Net P&L.”

Run completion includes an attempt to close remaining positions, but it is not a flat-position guarantee. A gated or delayed close can remain unresolved when no further observation is available.

Agent Lab's current historical path

Agent Lab accepts supported bar-only configurations and runs them in the generic memory session. It rejects tick, book, book-heartbeat, and interpolation modes, and does not accept every specialized app family, including Polymarket.

The current worker caps input at 100,000 bars and serialized result output at 15 MB. Completed-run logs are stored separately, read in bounded pages, and expire according to their retention period. Other model-facing artifact reads remain bounded. Plan validation, run reservations, and deadlines can add further constraints; these caps are not a promise that every smaller request will complete.

Saved result artifacts contain selected bars, trades, and chart data rather than the entire runtime object. Completed-run logs are attached through their separate expiring log file. Absence of a field in an artifact does not mean that field was absent during execution.

Data and warmup

Missing periods and duplicate timestamps require explicit handling. Sorting and finite-value checks do not prove complete coverage. Calendar/session filtering can make delivered strategy observations differ from loaded rows.

Memory sessions do not fetch the normal live external warmup. A source revision tested on a short selected window can therefore start with different state from a live session. Frozen Lab replay bars also do not automatically freeze every external query a script might make.

Interpolation is not recorded tick history

Some non-Lab engine paths support interpolation when tick processing and interpolation are both enabled. Constructing intermediate observations from bars does not recover the actual market's event sequence. Label such runs separately from recorded bar or tick data, and do not compare them as identical datasets.

AI authoring versus strategy execution

Agent Lab's assistant can help edit and inspect a strategy. Real model-provider calls inside an L3 strategy are disabled during backtesting and historical warmup. Deterministic policy logic and supported fake-provider responses can exercise the decision flow without those calls.

Historical versus live behavior

Live feeds can update an unfinished bar repeatedly. Brokers can reject, delay, or fill requests differently from memory simulation. Reconnection, competing strategies, and broker order reconciliation also require behavior beyond a local one-attempt-per-timestamp guard.

Use historical runs to establish explainable behavior, then inspect a running session's actual data and execution path separately. See Backtest, inspect, and launch when moving to that workflow.