Multi-Store Demand Models: Features First, Then One LightGBM Per Store×Product

Part 1 of this series argued that a shelf-price system should not spit out “the right sticker.” It should score a grid with a demand curve underneath. This post is the curve: how I build a multi-store demand forecasting model that predicts units, not dollars — with shared feature engineering and one LightGBM per store × product slice.

One sentence: I define features once, force own-price into the model, train a separate LightGBM for each store×SKU family with monotonic constraints, and I never trust a random train/test split on a time series.

Examples use Harbourline Mart — a fictional multi-store grocer I made up for this series, not a real company. Engineers get the feature contract and split rules. Managers get one scoreboard: can we explain next week’s volume miss without blaming “the algorithm said so”?

Backbone for the series: Shelf Price Decision Systems: Demand, Grid, and Gate. Part 3 scores the grid; part 4 ships jobs on Azure. Here we stay in data + model.

What a demand forecasting model owns (and what it does not)

Analogy first. Think fantasy sports: every player card uses the same stats (points, minutes, opponent strength). Each stadium still needs its own game-plan binder because the crowd, weather, and rival roster change. Harbourline’s shared playbook is the feature contract. Each store × SKU family gets its own LightGBM binder.

What the demand model owns:

• Predict units sold (or a basket-volume proxy) for a store × SKU family on a day
• Respond sensibly when we plug in a candidate shelf price at score time
• Ship an artifact bundle the optimiser can load without reverse-engineering notebooks

What it deliberately does not own:

• Picking the shelf sticker (that is the grid + penalties + gate)
• Category politics, planogram space, or supplier deal paperwork
• “Being right” on a random 20% holdout while leaking next week into train

Shared feature engineering, local models

I refuse one mega-model for “all milk in all stores.” That is multi-store demand on purpose: elasticity, rival set, and lag scale differ by location. I also refuse copy-pasted feature scripts per store — that is how column drift kills you in month three.

Layer	Shared once	Local per store × SKU
Feature names + dtypes	Yes — one contract	Values differ; schema does not
Competitor distance weights	Formula + radius policy	Which rivals fall inside the radius
LightGBM booster	Hyperparameter defaults	One model file per slice
Monotone constraints	Policy: own-price ↓ demand	Applied on that slice’s columns
House brand vs branded twin	How we name families	Separate slices — do not pool units

Feature families I actually ship

Feature engineering production is boring on purpose. Fancy transforms that need a research paper to debug do not survive a category manager asking “why did West store milk drop?”

Family	Examples	Why it is there
Own price	Today’s shelf price, lags, Δ vs 7d median	The lever we will sweep on the grid
Demand history	Units lags 1/7/28, rolling means, yoy proxy	Base rate before price moves
Calendar / promo	DOW, school holidays, promo flag, pay cycle	Grocery is calendar-shaped
Competitor price features	Distance-weighted rival shelf mean, min, Δ vs own	Anchor the demand context, not the later penalty
Competitor response stats	How often rivals moved after our last change	Optional; keeps score-time assumptions honest
Cost / margin band	Unit COGS, margin band id	Baseline volume tables key off weekday × band
Cross-price (twins)	Sibling SKU shelf price when house vs brand split	Substitution without pooling the target

I build a weekday × margin-band baseline table of expected units. Part 3 uses it for shortfall penalties. The demand model itself still predicts absolute units; the baseline is a second contract the optimiser reads.

Mutual information, then force own-price

Raw candidate columns easily hit 80–150. I run mutual information (or a cheap impurity rank) against units, keep a top-N set, then force own-price features back in even if MI ranks them mid-pack on a quiet month.

Why force? Because at score time we only vary price on the grid. If the booster barely saw own-price during feature select, û(p) becomes a flat line and the optimiser collapses into pure penalty theatre. Importance after fit is a report for humans; forced columns are a contract for the system.

# Pseudocode — feature select contract (not a library tutorial)
cands = mi_rank(X_all, y_units, top_n=40)
must = {"own_price", "own_price_lag_7", "own_price_delta_7"}
selected = ordered_unique(must + cands)
# write features.json with names, dtypes, mono_map

LightGBM production choices that matter

I use LightGBM because it is fast on tabular slices, handles missingness without drama, and supports monotone constraints. “LightGBM production” here means three boring rules, not a Kaggle bag of tricks.

Choice	What I do	Failure if I skip it
One booster per store × SKU family	Fit only that slice’s rows	Averaged elasticity that fits no store
Monotone on own-price features	Constraint map: price up → units down	Physics-breaking tails on the grid
Recent-row weights	Up-weight last 4–8 weeks lightly	Model lives in last year’s promo regime
Time-aware holdout	Last N weeks frozen; no shuffle	Pretty metrics, ugly next Monday
Artifact bundle	model + features.json + metrics.json	Score job guesses column order

# Pseudocode — train one slice (shape only)
params = {
  "objective": "regression",
  "monotone_constraints": mono_map_from_features_json,  # -1 on own-price cols
  # ... depth / leaves kept boring and stable
}
dtrain = lgb.Dataset(X_train, label=y_train, weight=recency_weights)
model = lgb.train(params, dtrain, valid_sets=[dholdout_time])
save_bundle(slice_id, model, features_json, metrics_json)

I do not chase leaderboard RMSE with five stacked models. I chase a demand curve that is monotone in price, stable week to week, and explainable when West store’s rival cuts 10 cents.

The random-split trap

If you shuffle grocery days into 80/20, adjacent days leak seasonality and promo state into both sides. The demand forecasting model looks brilliant and then fails the first real week after a holiday.

Rule I write into the train job:

• Hold out the last N complete weeks (often 2–4) as a time block
• Never sample holdout rows from inside the train window
• Report MAE / bias / coverage on the holdout and a simple naïve baseline (last-week same DOW)
• Refuse promote if holdout loses to naïve on quiet SKUs without a documented reason

That is classical ML discipline, the same boundary thinking as in AI architecture boundaries and evals — the eval contract is part of the system, not a screenshot from a notebook.

Score-time contract: û(p), nothing else

When the frequent score job runs, for each store × SKU family and each candidate shelf price p on the small grid:

1. Build the feature row with own-price fields set to p (other features from latest daily build)
2. Load that slice’s bundle
3. Predict û(p) units
4. Optionally apply a simple competitor-response assumption (rival moves toward p) — document it; do not hide it inside the booster
5. Hand the curve to the optimiser (part 3)

# Pseudocode — score one candidate
row = latest_features(store, sku_family)
row = set_own_price_fields(row, candidate_price=p)
u_hat = booster.predict(align(row, features_json))
# optimiser later: contribution(p) - shortfall - anchor

If û(p) is negative, I clamp at a small epsilon before the optimiser — silent negatives poison contribution. Calibration hard-fails belong in part 4’s quality gate; here the rule is: never ship a raw negative demand into margin math.

Scoreboard for engineers and managers

Question	Healthy answer	Smell
What does the model output?	Units at price p	A single “recommended price”
How many boosters?	One per store × SKU family	One global model “for scale”
Own-price in features?	Forced + monotone	Dropped by auto-select
Holdout?	Last N weeks, time block	Shuffled 80/20
House vs brand twin?	Separate slices + optional cross-price	Pooled target “to get more data”
Artifact?	model + feature JSON + metrics	Pickle of a notebook session

Where this sits in the series

• Part 1 — decide with a grid, not a price-regressor: hub post
• Part 2 (this post) — demand forecasting model, features, LightGBM slices
• Part 3 — objective = contribution − shortfall − anchor; dual change gate (both $ and % legs)
• Part 4 — Azure ML jobs, storage handoffs, production gaps

Related craft on this blog: AI solution architect decisions that survive review, RAG failure modes (different domain, same “contracts over vibes” habit), and the Azure integration series starting at microservices integration patterns when you need config-driven plumbing beside the ML path.

FAQ

What is a demand forecasting model in a price decision system?
A model that predicts units (or volume proxy) at candidate shelf prices for a store × product slice. It does not output the sticker. The optimiser and gate do (gate = both dollar and percent lift vs current shelf — see part 1).

Why one LightGBM per store × product instead of a single global model?
Rivals, lag scale, and elasticity differ by location. A shared feature playbook keeps ops sane; local boosters keep demand honest.

Why force own-price features if mutual information ranks them lower?
Score time only sweeps price. If own-price is missing or weak, the demand curve flattens and commercial penalties dominate for the wrong reason.

What are competitor price features vs the anchor penalty?
Competitor features condition the demand prediction (context). The anchor penalty (part 3) punishes candidate stickers that look insane vs rivals after the curve is scored. Different layers.

Is random train/test OK for grocery demand?
No. Use a time-aware holdout block. Random splits leak the future and inflate confidence.

How do house brand and branded twins work?
Separate slices so targets are not pooled. Optionally add the sibling’s shelf price as a cross-price feature so substitution shows up without merging demand histories.

What must be in the artifact bundle?
The booster, an ordered features.json (names, dtypes, mono flags), and metrics.json from the time holdout. Score jobs load the bundle; they do not re-derive column order from tribal knowledge.

Where do shortfall and competitive-anchor penalties live?
In the optimiser (part 3), not inside LightGBM leaves. Policy stays explicit and auditable.