Skip to main content

How Snowtrail Works

Snowtrail processes raw market data through a systematic pipeline to produce research-ready outputs for energy and commodity markets.

Data Sources​

Snowtrail ingests data from a broad set of sources. A few of them:

  • Government and regulatory agencies: EIA (US Energy Information Administration)
  • Industry bodies: GIE (European gas and LNG storage)
  • Weather and climate: NOAA (GEFS/GFS forecast ensembles), ECMWF, ERA5 reanalysis
  • Market infrastructure: Nord Pool, CFTC COT (positioning data)

These sources are monitored continuously and ingested on schedules that reflect each source's publication cadence, from intraday to weekly.

Data Pipeline​

Snowtrail transforms raw data through a structured, multi-stage pipeline:

  1. Raw data ingestion: Data from external sources is collected, validated, and stored with full provenance. Every raw record carries str_effective_at and str_ingested_at timestamps from the moment it enters the system.

  2. Feature construction: Raw data is cleaned, aligned, and transformed into derived features. Feature pillars run in parallel across supply, demand, storage, weather, flows, and infrastructure, each producing research-ready metrics with stable definitions.

  3. Signal generation: Features are combined into interpretable signals that describe the state of each market. Each product has its own signal architecture tailored to the market it covers (2-4 signals per product), but all share a common principle: the primary signal is fundamentals-only. Price is never an input to the core regime signal. Where price context or positioning signals exist, they are evaluated separately so the fundamental signal can be assessed on its own merits.

  4. Event detection: Discrete developments that materially affect market conditions are identified and classified, providing structured context around disruptions, transitions, and notable shifts.

  5. Backtesting: Signals and events are validated against historical outcomes using a rigorous backtesting framework, including walk-forward analysis and statistical robustness checks.

  6. Delivery: Outputs are made available via REST API, Python SDK, or S3 file delivery.

Daily Orchestration​

The pipeline runs daily through automated orchestration. Each product has its own pipeline that is triggered on a schedule, ensuring data is refreshed and available to consumers by the start of each trading day. Intraday updates are produced for time-sensitive data sources.

Point-in-Time Integrity​

All Snowtrail data maintains strict point-in-time integrity. Every row in every table, from raw through features, signals, events and backtests, carries three timestamps:

  • str_effective_at: the date this row became knowable, being the business date it describes plus the source's publication lag where one applies
  • str_observed_at: the date Snowtrail could first have held the value, derived from the arrival time of the raw data that fed it
  • str_ingested_at: the UTC timestamp of the run that last wrote this row

This design means:

  • Data is never surfaced earlier than the date its source actually published it
  • Research and backtesting can be performed without look-ahead bias from publication lag
  • Queries can be replayed as at any historical date using the as_of query parameter

The derived layers are bitemporal. When a source revises a figure, the recomputed row is appended as a new version rather than overwriting the old one, so as_of returns the value you would have acted on at the time, not today's restated figure. See Point-in-Time Integrity for the limits, including how far back reconstruction reaches.

Signal Design Philosophy​

Snowtrail signals are built on a principle of separating fundamentals from price. The system stress signal is derived entirely from physical market data (storage levels, flows, production, demand), so that it provides an independent view of market conditions. Price is treated as an outcome to be evaluated against that view, never as an input that could introduce circularity.

This separation allows users to ask questions like "How did the market behave during periods of elevated fundamental stress?" without conflating cause and effect.

Update Frequency​

Data is updated on schedules appropriate to each market and data type, ranging from intraday to daily updates. Most signals and features are available by 06:00 UTC each day.