A feature store doesn’t have to be a heavy product: well designed on Microsoft Fabric, it reduces scoring latency, duplicated work and operational costs — without large upfront investments.
Why a Lightweight Feature Store Is Essential for Real-time Scoring
Data and AI teams face a practical dilemma: without a single source of truth for features, reproducing preprocessings between training and production becomes manual and error-prone. Each new team tends to reimplement similar transformations, which creates divergence in calculations and complaints from product teams when models start to degrade. The lack of consistency translates into model drift, high latencies and constant support requests from business teams — costs that are hard to quantify but real: teams spending 20% of their time fixing feature discrepancies are not optimizing model value.
A lightweight feature store solves this without imposing a new heavyweight platform. By centralizing feature logic in Delta tables in the Microsoft Fabric Lakehouse and materializing read-optimized versions, we can ensure consistency between training and production, speed up inference and reduce operational costs. In typical organizations, this can reduce duplicated featurization effort by 40–60% and the duration of incidents related to inconsistencies by 70%.
It’s important to emphasize that lightweight does not mean primitive. It means opting for Fabric-native components and simple engineering patterns: canonical tables, scheduled pipelines, read-optimized materializations and clear fallback policies. This approach allows starting with a prototype in 4 weeks and scaling in a controlled way as usage is proven.
Practical Architecture with Microsoft Fabric
A minimal and pragmatic architecture proposal relies on Fabric-native components: Lakehouse for storage of canonical features, Spark Notebooks for compute and updates, SQL Warehouses for low-latency queries and Pipelines for orchestration. We keep it simple: we do not introduce external services until necessary.
Typical flow: ingestion of events/raw -> batch/stream transformation -> write to Delta tables in the Lakehouse (canonical features) -> materialization jobs (read-optimized tables) -> Warehouse SQL endpoints or caches for the inference service. This architecture leverages OneLake as a unified storage layer and the Spark engine for heavy processing. Optionally, for very fast updates, we can integrate a lightweight KV store (for example Redis) as a last-mile cache, but this is complementary and does not replace the Lakehouse.
From an operational point of view, the architecture has three clear zones: ingestion zone (raw events, CDF enabled to detect changes), canonical featurization zone (Delta tables with history and metadata) and serving zone (materialized tables and read-optimized caches). Each zone has defined SLAs — for example, ingestion with 1–2 minutes of acceptable lag, batch materialization with 30–60 minute cadence for aggregated features, and Warehouse query latency below 100 ms for key-based queries.
Feature Design: Batch, Incremental and Online
Not all features are equal. Classifying features by update frequency and computation cost is crucial. I propose three practical categories: 1) static features (user attributes), 2) batch-aggregated features (daily sums, counts) and 3) low-latency/online features (minute-level counters, last interaction).
For each category we define clear rules: schema, time window, latency tolerance and update policy. For example, a type 2 feature may be recalculated every 4 hours with a 30-minute tolerance; a type 3 feature may require updates every 60s and have a fallback to the last calculated window.
Concrete examples:
- Static feature: user_age_bucket. Updated only when the user edits the profile; 24-hour tolerance. Simplicity and determinism are essential.
- Aggregated feature: purchases_30d_count. 30-day rolling window recalculated every 30 minutes. For 6M monthly active users, an hourly job that aggregates events in batch can process 20M events/day in 40–60 minutes on a medium Spark cluster (8–16 cores), producing a table with ~6M rows.
- Online feature: last_click_timestamp_seconds. Should reflect the last interaction in minutes; sub-1s update via write-through to Redis and eventual write to the Delta table for medium-term consistency.
By defining these categories, usage indicators that guide materialization decisions are also defined: if a column is queried in less than 5% of predictions, it may not be worth materializing permanently; prefer computing it on demand or keeping it in cold storage.
Implementation: From the Lakehouse to Low-latency Queries
Step 1 — Canonical tables: create a set of Delta tables in the Lakehouse for each entity (user, product, session). These tables contain raw events and natural key columns (e.g., user_id, timestamp) and normalized fields for features. Use date partitioning for bulk ingestion and hash partitioning by user_id on materialized tables for serving. For 100M records, a strategy of 256 partitions by user_id hash usually balances I/O and parallelism.
Step 2 — Featurization jobs: Spark notebooks that execute transformations and write results to feature tables. Use checkpoints and an idempotency token to ensure re-executions do not duplicate results. Write with MERGE for incremental updates and leverage the Delta Change Data Feed to process only new events. Practical example: a job consuming 20M events/day using 16 cores and 64 GB RAM can complete hourly aggregations in 25–45 minutes, consuming around 8–12 CPU hours per day, depending on optimization.
Step 3 — Materialization for serving: create read-optimized tables in the Warehouse. For example, generate a daily_features_user table with 100M rows partitioned by user_id hash and with z-ordering on the most queried columns. Configure permissions and caching in the Warehouse to reduce query latency. The Warehouse cache typically improves latencies of 200–400 ms to under 50 ms for very selective key-value queries; tune cache size to reflect access patterns (for example, 10–20% of the working set in memory).
Step 4 — Read strategies for inference: for batch models, read materialized tables directly. For real-time inference, combine a query to the Warehouse (fast cache, target <100 ms for key selections) with a fallback to features computed in-memory or by the application service when not available. A common approach is the following chain: 1) query the in-memory cache (hit target 85–95%), 2) if miss, query the Warehouse (≤100 ms), 3) if still miss, compute inline or use a default feature. This model ensures scoring latency remains predictable and bound by SLOs.
Operations, Versions and Reproducibility
Tracking feature versions is vital. Include a featurization manifest with notebook hash, output schema and pipeline version. Each feature table should include metadata: feature_version, generated_at, source_job_id. Thus, a reproducible training run explicitly points to feature_version X. Use Delta time travel as an emergency mechanism to recover previous states in case of erroneous writes — for example, rollback a wrong materialization in the last 24 hours.
Operational alerts: implement simple quality checks (for example, null counts, quantile distributions) that run after each materialization. Practical rules:
- Alert if the percentage of nulls for a critical feature exceeds 2%.
- Alert if the cardinality of a primary key varies more than 5% between consecutive windows.
- Alert if the job execution time exceeds 2x the historical average for that window.
Mini Practical Case: Real-time Scoring in an E‑commerce Company
Context: e‑commerce company with 120 employees, 2 web apps and a recommendation engine that scores on each visit to personalize the carousel. Volume: 20M events per day; 6M monthly active users; ranking model with 75 features.
Challenge: original scoring latency was ~420 ms per request (queries to multiple APIs and ad-hoc feature computations), which degraded experience and increased bounce rate on pages with dynamic carousels. We implemented a lightweight feature store on Fabric with these measures:
- Building canonical tables: ingest events to the Lakehouse (20M/day), partitioned by day and by user_id hash for serving.
- Batch featurization: daily and hourly aggregates generated with Spark, updated every 30 minutes using Change Data Feed to process only new events.
- Low-latency materialization: table partitioned by user_id with z-ordering and Warehouse cache for frequent key-value queries.
- Online fallback: session counter in Redis for sub-1s updates when needed and write-through to Delta for eventual persistence.
Results after 8 weeks:
- Average scoring latency reduced from 420 ms to 85 ms (including Warehouse call and prefetch of the 75 features). At peaks, latencies were under 150 ms, compared with previous peaks of >1s.
- Production error rate related to feature inconsistencies dropped 78%. Monthly incidents fell from 9 to 2, with mean time to resolution reduced from 4h to 1h30m.
- Operational cost (compute for featurization) reduced 35% by eliminating duplicate jobs and better sharing of results between training and production. We estimated a direct annual saving of ~€45k in compute consumption and avoided engineering hours.
This example shows that with pragmatic choices (correct materialization, caching and simple fallback), substantial gains are possible without a monolithic project or high upfront costs. ROI became evident in 2 months, justifying expansion to other feature pipelines.
Centralizing feature logic is less about technology and more about reducing friction between teams: less duplication, fewer incidents, more trust in models.
Best Practices and Pitfalls to Avoid
Best practice 1: keep transformations deterministic and versioned. If a notebook depends on date/time, record the window clearly to avoid non-deterministic results between training and production. Use fixed seeds for random operations and document time windows in feature versions.
Best practice 2: start small. Identify the 10 highest-value business features and implement them first. Test latency and costs before materializing hundreds of columns. A typical pilot with 10 features allows validating benefits with reduced investment — for example, a prototype that processes 2M events/day and serves 100k requests/day gives a clear picture of impact without large financial exposure.
Pitfall to avoid: trying to materialize everything by default. Rarely used features can increase storage costs and degrade performance. Use usage metrics to decide what to materialize. Another pitfall is not having a clear retention policy: without cleanup, Delta tables can grow indefinitely; define retentions and run controlled vacuum to manage storage cost.
In Summary
- Build a lightweight feature store in the Microsoft Fabric Lakehouse using Delta tables, Spark notebooks and SQL Warehouses to serve low-latency features.
- Categorize features as batch, incremental and online; implement materializations and fallback to combine consistency and performance.
- Version transformations and include automated quality checks to ensure reproducibility and reduce regressions.
- Start with a small set of high-impact features and measure latency and costs before scaling.
Practical next steps: identify the 10 critical features in your business, implement them as canonical Delta tables in OneLake, and create an idempotent Spark notebook for the first materialization. Measuring before and after (latency, cost, number of incidents) turns hypotheses into measurable results.
Would you like to explore an implementation plan tailored to your reality — number of users, latency windows and operational budget — and validate expected gains with a 4-week prototype?
