Structural Language ModelOverview
Structural Language Model treats price action as a language. Each bar is tokenised into one of five structural symbols, and a low-order Markov model learns the grammar — the probability of what comes next given the recent context. Instead of "match the nearest historical shape" (fragile, overfit-prone k-NN), it estimates P(next token | last k tokens): a nonparametric conditional-move model that proves or disproves itself, live, on your symbol. It is a research/forecast read, not a signal service.
The five-symbol grammar
Every bar becomes one token, built from robust intrabar primitives (gap-immune, no fragile sweep/FVG detection), with adaptive thresholds so the alphabet stays balanced across symbols and timeframes:
X− down impulse · d ordinary down · c compression / indecision · u ordinary up · X+ up impulse
The model then learns grammar like c → X+ (breakout), X+ → X− (reversal), runs of u/X+ (trend), X+ → c (exhaustion), using order-1 or order-2 transition counts with Laplace smoothing, updated online.
Why these parts are one tool
The tokeniser turns raw OHLC into a balanced, information-rich alphabet — without it the Markov counts are dominated by whatever token is most common.
The Markov model reads out, each bar, a directional bias (P up-ish − P down-ish), a predictability score (how peaked the next-token distribution is, via normalized entropy), a structural-surprise spike (−log P of the token that just printed — a grammar break), and the full next-bar probability ladder.
The harness is the part that makes it honest. It's prequential (predict-then-update: each transition is scored from counts that exclude its own outcome, so every score is out-of-sample), it runs a walk-forward in-sample vs out-of-sample split with Wilson 95% intervals, and it draws a reliability curve — binning OOS predictions by predicted P(up) and showing the realized up-rate per bin. A rising, significant curve = real calibrated information; a flat one = none. Remove any part and you can no longer answer "is this model actually calibrated on this market?"
How to use it
Read the directional bias line against its conviction bands as context, not a trigger, and check predictability for how peaked the forecast is. Then read the harness — the model is only worth trusting where the out-of-sample up-lean lift is above 1 and/or the reliability spread is positive and significant (✓sig). A flat or insignificant curve means there's no calibrated edge here; treat it as descriptive only, or try another symbol/timeframe. The dashboard has a Compact layout (default: forecast + the one calibration line that matters) and a Pro layout (the full ladder, in/out-of-sample lift, and the three-bin reliability curve). Bias-turn crosses are optionally mirrored on the price chart. It is never a standalone signal.
Non-repainting
Tokens and counts update only on confirmed bars, and the score for bar t uses counts as they stood before bar t's transition was added — nothing reads its own future. The live next-bar forecast naturally refines as the current bar forms (it's a forecast, not a settled statistic). All harness figures are out-of-sample by construction.
Honest limits
OHLCV only. A per-bar tokeniser maximises samples but is coarser than a swing/event grammar (a documented future extension). Any edge is typically modest and market/timeframe-dependent — directional forecasting on noisy price is hard, and no indicator has an inherent edge. That's exactly why the harness is built in: validate it before trusting it.
Outputs for other scripts
Generic EXP_* plots — bias, predictability, structural surprise, live P(next up-ish), and the OOS lift — are published to the Data Window for use from other scripts via input.source().
Concept credits
Markov chains / n-gram language models — A. Markov (1913); C. Shannon (1948)
Prequential (predict-then-update) evaluation — A. P. Dawid (1984)
Additive (Laplace) smoothing — P.-S. Laplace
Entropy — C. Shannon (1948)
Wilson score interval — E. B. Wilson (1927)
Synthesis and Pine implementation are the author's own; no third-party Pine code reused.
Disclaimer
Research and education only. Not financial advice, not a signal service, not a guarantee of future results. Validate with your own testing, apply realistic costs, and manage risk.
Indicator
