Guide
Online vs offline feature stores: what changes between training and inference?
The same feature has to serve two workloads that could hardly be more different. Understanding both — and what must stay identical between them — is most of feature-infrastructure design.
Two workloads, one feature
Take one feature — payment_to_income_ratio. During model development it must be computed for a million historical applications, as of each application’s decision date. In production it must be computed for one applicant, now, inside a latency budget. Same definition, radically different access patterns. “Offline” and “online” feature infrastructure are the two halves of serving both without letting the definition fork.
The offline workload
The offline side is history-shaped:
- Scale over latency. Millions of rows; seconds or minutes are fine.
- Historical materialization. Executing feature logic across landed source history and persisting the results — including backfills when features or sources change, which is why idempotent, re-runnable jobs matter more here than raw speed.
- Training datasets. Assembling per-decision feature vectors anchored to observation times, under point-in-time constraints so no row sees data that arrived after its decision moment.
- Consumers: training pipelines, model validation, analysis.
Typical backing: a database or warehouse. In ClearFeature, feature history and job metadata live in PostgreSQL, with raw source payloads in S3-compatible object storage.
The online workload
The online side is request-shaped:
- Latency over scale. One entity, one vector, within the decision path’s budget.
- Latest values. For lookup-style access, a fast key-value store holds the most recent feature values per entity — Valkey, in ClearFeature’s case.
- Request-time computation. For features that must reflect data arriving with the request (the transaction being scored, the report just fetched), the logic executes on demand.
- Consumers: live scoring services and decision systems.
Materialization vs request-time computation
The online side has two supply strategies, and most mature systems use both:
Precompute (materialize), then look up. Feature values are computed ahead of time and pushed to the online store. Retrieval is fast and computation cost is amortized. The cost is freshness: the value is as current as the last materialization, and the request’s own data cannot participate.
Compute at request time. The feature executes when asked, over current inputs — fresh by construction, and able to use request-scoped data. The cost is compute in the decision path, and the crucial engineering constraint: the request-time implementation must be the same logic as the training implementation, or you have manufactured training-serving skew at the point of maximum damage.
Rule of thumb: slow-moving aggregates over deep history favor precomputation; anything derived from decision-time inputs favors request-time execution.
Latency vs reproducibility
The online store optimizes for now; the offline store is also the substrate for proving what happened. If latest values simply overwrite in a cache with no availability-aware history behind them, you can serve decisions but not reconstruct them. Systems in regulated or high-stakes domains need the offline side to be an honest, availability-stamped record — not merely a training convenience. (Why this matters →)
Why consistency between the two sides is the whole game
Offline/online is an infrastructure split, but the model must never perceive it: training vectors and serving vectors must come from the same definitions. There are two ways to get that. Keep two implementations synchronized — with parity tests, tolerance dashboards, and permanent vigilance — or have one implementation that both sides execute. The second approach is ClearFeature’s: one versioned Feature Project, one dependency DAG, executed by the platform for historical materialization, PIT training datasets, and request-time computation, with latest values kept in Valkey. The offline/online split remains — as storage and access patterns — while the feature logic stops being split at all.
Do you need one, the other, or both?
- Batch-only scoring (decisions made in nightly runs): offline infrastructure may be all you need.
- Live decisions from precomputable features: offline plus an online lookup store.
- Live decisions from request-scoped data (applications, transactions, market state): request-time execution — and the strongest possible reason to keep it training-identical.
Most decision systems in lending, fraud, and trading end up in the third category — which is why the execution question, not the storage question, decides the architecture. (What real-time actually requires →)
Further reading
- What is a feature store? — the category overview.
- Feature stores for real-time ML — the online path in depth.
- The ClearFeature platform — one execution model across both workloads.
Designing your offline/online split?
See how one execution model serves both workloads without forking the feature logic.