Historical market data

Data selection is part of the experiment. Record the exact instrument, interval, dates, timezone, and the resulting coverage before comparing strategy revisions.

Requested range versus delivered frames

A selected date range does not guarantee that every expected bar exists. Holidays, session gaps, provider coverage, and contract boundaries can affect the returned data. The replay loop can also skip work because of session rules or configured strategy cadence.

For supported CME futures, the memory loop consults the engine's session calendar gate; uncovered instruments/dates retain fallback behavior. Regular-hours settings and other loop time rules also affect processing. A gap in SJS observations is therefore not, by itself, proof of a gap in the stored dataset.

Use market timestamps for historical reasoning. Display timezone changes how a time is presented, while session timezone can affect filtering. Keep both the underlying timestamps and relevant timezone settings in your run notes.

Agent Lab data handling

Agent Lab identifies its dataset selection using the instrument, interval, start date, end date, and timezone. After a successful execution produces a dataset artifact, subsequent runs with that identity in the same experiment reuse its stored bars. Editing strategy source alone does not refresh them.

The worker normalizes OHLCV values, filters rows to the selected bounds, and sorts by timestamp. It rejects empty or non-finite data. These checks do not establish complete exchange-session coverage or remove every possible duplicate timestamp.

The frozen artifact covers the supplied replay bars. Do not assume that an additional historical query or external service used by your script is automatically frozen with it. Prefer the self-contained SJS example while establishing reproducibility.

Observe frames without placing orders

This complete SJS diagnostic replaces the trading script temporarily. It observes only frames reaching SJS, logs the first valid frame and timing anomalies, and keeps bounded counters. It assumes one-minute observations and reports gaps longer than two minutes.

export async function app({ context, bar, log }) {
  const state = (context.sjsState ??= {}).backtestDataCheck ??= {
    observed: 0, accepted: 0, invalid: 0, nonIncreasing: 0, gaps: 0,
  };
  state.observed += 1;
  const timestamp = bar?.date == null ? NaN : new Date(bar.date).getTime();
  if (!Number.isFinite(timestamp) || typeof bar?.close !== "number"
    || !Number.isFinite(bar.close)) {
    state.invalid += 1;
    log?.("backtest-data", { issue: "invalid timestamp or close", observed: state.observed });
    return;
  }
  if (state.lastTimestamp != null && timestamp <= state.lastTimestamp) {
    state.nonIncreasing += 1;
    log?.("backtest-data", { issue: "non-increasing timestamp", timestamp });
    return;
  }
  if (state.lastTimestamp == null) {
    state.firstTimestamp = timestamp;
    log?.("backtest-data", { issue: "first valid frame", timestamp });
  } else {
    const gapMs = timestamp - state.lastTimestamp;
    if (gapMs > 120000) {
      state.gaps += 1;
      log?.("backtest-data", { issue: "gap over two minutes", timestamp, gapMs });
    }
  }
  state.lastTimestamp = timestamp;
  state.accepted += 1;
}

The observer does not fill gaps, change timestamps, or validate all OHLCV fields. Its accepted counter means accepted by this diagnostic, not accepted orders. Restore the original source and use a fresh run after inspecting the data.

Continue with Warmup and Debugging.