Backtests

Backtests provide pre-computed, point-in-time correct performance analysis for each Snowtrail signal. They are designed to answer the question: how well does this signal capture the market condition it was built to detect?

Golden Rule

Backtest against the question the product answers.

Each product's backtest measures whether the signal's regime classifications correspond to meaningfully different market outcomes, not whether they predict price direction.

Current Validation Status

A headline is significant only if both the block-permutation p-value and the structure-preserving null p-value (headline_p_value_structure) are below 0.05 (rule in force since 2026-09-29). Latest results as of 5 October 2026:

Product	Status	Headline
GLMI	Validated, coincident	Marginal-basin score vs same-month JKM-TTF spread level: rho -0.596, block p 0.0002, structure p 0.019, n 105 months. No measured lead.
WSSI-US	Validated (inverse by sign convention), in-sample	Forecast HDD anomaly vs signed EIA storage surprise: rho -0.51 all weeks (-0.77 heating season, about 0 off-season), p < 0.001, structure p 0.027, n 156 weeks. Winter 2026-27 hold-out pre-registered.
GBSI-EU	Not validated	Data product. score_rho -0.126, block p 0.001, structure p 0.072, n 2,248: not significant; TTF hold-out +0.08, p 0.25.
WRSI	Not validated	Classifier. Coincident vol ratio 1.43, block p 0.16, structure p 0.028: not significant. Pre-registered forward tests failed.
GBSI-US	No forward claim	Point-in-time fundamentals index. The summary row carries a monitoring statistic (band-occupancy max deviation 3.1 pp), not evidence.
PEMI	No price claim	Grid-stress data feed. Pre-registered price tests found no effect; no backtest is persisted as a claim.

See each product page for the full wording and limitations.

Available Datasets

Every product provides three backtest datasets:

Dataset	Description	SDK Method
Summary	Pre-computed backtest metrics including regime persistence, flip-flop rate, and vol multiplier	backtest_summary()
Outcomes	Per-regime outcome metrics: realised volatility, vol multiplier, tail lift	backtest_outcomes()
Transitions	Regime transition matrix showing probabilities of moving between states	backtest_transitions()

GLMI additionally provides backtest_combinations() for two-signal analysis (4 basin regimes x 3 price contexts).

Key Metrics

• Vol Multiplier: How much more volatile the market is during stressed regimes vs calm regimes (descriptive only; not a sizing input; read headline_* for the tested claim)
• Tail Lift: Probability of extreme moves during stressed vs calm regimes (descriptive only; not a VaR input)
• Regime Persistence: How stable regime classifications are over time
• Flip-Flop Rate: Frequency of rapid regime changes (lower is better)

Methodology

Backtests use:

• Point-in-time inputs: Signals use only data knowable at the time; where a descriptive metric's truth window overlaps the pre-publication period, lookahead_flag is set
• Block bootstrap confidence intervals: Statistical robustness for metrics
• Block permutation tests: The headline p-value is computed by permuting whole blocks, so serial correlation survives the null. Block permutation is used rather than block bootstrap because a bootstrap holds subgroup size fixed and therefore cannot detect sample-size artefacts on heavy-tailed data.
• Add-one p-value estimator: (1 + exceedances) / (1 + permutations). A resampling test cannot support a p-value of exactly zero; where no permutation is as extreme as the observation the honest statement is p < 1/n.
• Structure-preserving null: A second null (for example a circular shift) that keeps the signal's own persistence; reported as headline_p_value_structure
• False Discovery Rate (FDR): Controls for multiple testing
• Minimum sample enforcement: A backtest below the minimum refuses to emit metrics rather than returning placeholder values that read as a genuine null.
• Native-grain evaluation: Each product is scored at the frequency it actually updates. A monthly signal merged onto daily prices would otherwise multiply one observation into ~21 and inflate significance.

Example

from snowtrail import Snowtrail
client = Snowtrail(api_key="your-api-key")

# Get backtest summary for GBSI-US.
# Read the headline_* columns first -- they carry the ONE statistic this
# product's claim rests on, and they are named identically across all six
# products so you can compare like with like.
summary = client.gbsi_us.backtest_summary()
print(summary[["backtest_date", "headline_metric", "headline_value",
               "headline_p_value", "headline_direction", "flip_flop_rate_2w"]])

# Get per-regime outcomes
outcomes = client.gbsi_us.backtest_outcomes()
print(outcomes[["regime_label", "sample_n", "vol_multiplier", "vol_2d_median"]])

# Get transition matrix
transitions = client.gbsi_us.backtest_transitions()
print(transitions)