Solutions · Training-serving skew
Stop maintaining separate training and serving feature logic.
Use one versioned Python feature DAG for historical computation and live decisions instead of keeping SQL, notebook, and production-service implementations in sync manually.
What training-serving skew is
Training-serving skew is any systematic difference between the feature values a model was trained on and the values it receives in production. The model itself can be byte-identical in both places — if the features differ, its behavior differs. Offline metrics stop predicting online performance, and nobody changed the model.
How organizations manufacture skew
Almost nobody decides to build three implementations of a feature. It happens one reasonable step at a time: the notebook proves the idea, the batch pipeline makes training data at scale, the online service meets the latency budget. Each rewrite is locally sensible. Globally, the organization now maintains three definitions of the truth.
Before — one feature, three implementations
Notebook Python
research and model development
Historical SQL / Spark
backfills and training pipelines
Online service code
re-implemented for production latency
Three codebases drift apart — silently.
With ClearFeature — one implementation
One Feature Project
Python UDFs · declarative DAG · versions · tests
Historical materialization & PIT training data
Live request-time computation
The same tested logic executes on both paths.
Why tests alone do not close the gap
Teams try to pin implementations together with comparison tests — sample requests replayed offline, tolerance thresholds, parity dashboards. These help, but they test points, not semantics: the null case that never appeared in the sample, the timezone handled differently, the rounding applied in one code path stays invisible until it moves a decision. Parity testing between two implementations is permanent work; a single implementation makes the entire category structurally unnecessary.
One executable implementation
In ClearFeature, a feature exists once: a Python UDF, registered with a version, tested against golden cases. Historical materialization executes it. Training-dataset construction executes it. Request-time scoring executes it. There is no port to maintain and no drift to detect, because there is nothing to drift from.
Dependencies are where drift compounds
Skew is worst in dependent features. If 'payment burden' differs slightly between paths, every feature built on it inherits and amplifies the difference. ClearFeature's registry defines the dependency graph once, and the platform resolves it identically in batch and online execution — the graph cannot fork by accident.
Code skew is not the only skew
Two distinct problems hide under one name. Code skew: implementations differ. Point-in-time skew: training rows see information the live system did not have at decision time. Fixing the first without the second still leaves offline metrics inflated. ClearFeature addresses both — one implementation for code skew, availability-aware datasets for temporal skew.
Migrating without a big bang
You do not fix skew company-wide in one quarter. The practical path is one feature workflow: pick a model whose divergence hurts, move its features into a Feature Project, run historical and live paths from that one DAG, and measure what changed. Expansion follows evidence.
How many implementations of your riskiest feature exist?
If the answer is more than one, we should talk.