Your strategy version hash does not pin the code that runs it

A strategy version hash makes a backtest reproducible only if it covers everything that can change the answer. Almost every versioning scheme hashes the strategy definition — the YAML, the JSON, the parameter block — and stops. That is one of the two hashes you need.

Here is the whole problem in four lines, konis_core/strategy/indicators.py:151:

@register("rsi", pane="oscillator")
def _rsi(ns: Namespace, source: str = "close", period: int = 14) -> pd.Series:
    s = _series(ns, source)
    return _talib(talib.RSI, s.index, s, timeperiod=int(period))

The strategy document says rsi. It does not say what rsi means. Replace talib.RSI with a Wilder smoothing you wrote yourself and every document in the catalogue that calls rsi starts producing different signals — under an unchanged version hash, with nothing in the system noticing.

What the document hash covers, and what it does not

Strategy.version_id (konis_core/strategy/schema.py:99) is a content hash of the parsed document:

payload = json.dumps(self.to_dict(), sort_keys=True, separators=(",", ":"))
return hashlib.sha256(payload.encode("utf-8")).hexdigest()

sort_keys=True means key order in the YAML is not a change. separators means whitespace is not a change. It hashes the parsed structure, not the file, so a reflow or a reordered mapping keeps the key stable. For what it claims — an edit is a different key — it is exactly right, and two caches are built on that claim.

StrategyCatalogue detects reloads from a directory signature of (name, st_mtime_ns, st_size) per file (strategy/catalogue.py:70-84) and keys cached strategies by content rather than by name, because, as its header puts it, keying a cache by (pair, strategy_name) cannot detect an edit.

The signal store leans on it harder. From signals/store.py:12-15:

Keyed by STRATEGY VERSION, not strategy name. The previous cache decided validity with count_documents(query) > 0, which has no notion of which version produced the rows — edit a strategy and the stale frame is served forever.

That is a real bug class closed outright. A count of matching documents answers "are there rows?" and is silently asked in place of "are these the rows?". A content hash cannot go stale because an edit is a different key.

It closes the bug in one dimension. The failure it cannot see is the one where the document is byte-identical and the code underneath it is not.

The second hash

konis_core/strategy/engine_fingerprint.py is two hash functions and a pair of dataclasses. The inner one:

def _digest(parts: Iterable[str]) -> str:
    h = hashlib.sha256()
    for part in parts:
        h.update(part.encode("utf-8"))
        h.update(b"\x00")
    return h.hexdigest()

The \x00 between parts is the usual defence against concatenation ambiguity — without it, ["ab", "c"] and ["a", "bc"] hash the same.

One indicator:

def indicator_digest(name: str) -> str:
    fn = INDICATORS.get(name)
    if fn is None:
        return "missing"
    try:
        outputs = ",".join(outputs_of(name))
    except Exception:
        outputs = "?"
    return _digest([name, outputs, _normalised_source(fn)])

The vocabulary a document reaches:

selected = sorted(set(names)) if names is not None else sorted(INDICATORS)
parts = [f"{n}:{indicator_digest(n)}" for n in selected]
parts.extend(_normalised_source(module) for module in _SHARED_MODULES)
return _digest(parts)

where _SHARED_MODULES = (expression, candle_transforms) — the rule evaluator and the candle derivations every document passes through no matter which indicators it names.

That is small enough to verify by hand. Against this checkout:

indicator_digest("ema") = 31991e2bfcb9f7da505b514b207b4c32b1ad800735271416803327790d6d1e52
indicator_digest("rsi") = 3d973d497916d1f8cb2cb494d1e1572da59a16dbd8de257146cfb56942654edf

parts = ["ema:31991e2b…", "rsi:3d973d49…",
         normalised(expression), normalised(candle_transforms)]
sha256, each part followed by a NUL byte:

  hand-rolled                        906c4c2f742e35bc73ad0561a68ea99c893e33740ea3bfaf8d48791a066ebd57
  vocabulary_digest(["rsi", "ema"])  906c4c2f742e35bc73ad0561a68ea99c893e33740ea3bfaf8d48791a066ebd57

vocabulary_digest(["ema", "rsi"]) gives the same value: the sorted(set(...)) means the order a document happens to list its indicators in is not part of the engine's identity. The two shared modules contribute 6,637 and 2,116 characters of normalised source respectively.

Every hex value in this post is for one commit of one repository. On any other checkout they are different, which is the entire point of quoting them.

What normalisation throws away

_normalised_source parses the function, strips docstrings, and unparses. Here is chopiness as written:

@register("chopiness", pane="oscillator")
def _chopiness(ns: Namespace, period: int = 14, **sources: str) -> pd.Series:
    """Choppiness Index: is the range being traversed or merely revisited?

    100 * log10(sum(TR, n) / (max(high, n) - min(low, n))) / log10(n). High
    means the bars cover a lot of distance inside a narrow range -- chop -- and
    low means they are going somewhere.

    Spelled the way the strategies spell it, missing 'p' and all, because that
    is the name they call and this vocabulary exists to match them.
    """
    f = _ohlcv(ns, **sources)
    n = int(period)
    tr = _talib(talib.TRANGE, f.index, f["high"], f["low"], f["close"])
    span = f["high"].rolling(n).max() - f["low"].rolling(n).min()
    ratio = tr.rolling(n).sum() / span.replace(0.0, np.nan)
    return 100.0 * np.log10(ratio) / np.log10(n)

and here is what actually gets hashed:

@register('chopiness', pane='oscillator')
def _(ns: Namespace, period: int=14, **sources: str) -> pd.Series:
    f = _ohlcv(ns, **sources)
    n = int(period)
    tr = _talib(talib.TRANGE, f.index, f['high'], f['low'], f['close'])
    span = f['high'].rolling(n).max() - f['low'].rolling(n).min()
    ratio = tr.rolling(n).sum() / span.replace(0.0, np.nan)
    return 100.0 * np.log10(ratio) / np.log10(n)

Four things went away, and each one is a decision rather than an accident of the tooling.

Comments. They never reach the AST, so ast.unparse drops them for free. This is the exclusion that makes the rest worth having: a digest that moves when someone fixes a typo in a comment is noise, and a noisy drift report is one people learn to skip. The module docstring states the rule directly — the digest has to move when behaviour moves and stay still when a comment is fixed.

Docstrings. Comments are free; docstrings are not, because they are ast.Expr nodes in the body. The walk removes the leading string constant from every FunctionDef, AsyncFunctionDef, ClassDef and Module, substituting ast.Pass() when that empties the body. Nine lines of prose here, describing the arithmetic without being it.

Formatting. Falling out of ast.unparse, not chosen, but worth knowing you have it: double quotes became single, period: int = 14 became period: int=14. Running a formatter over the indicator modules does not invalidate a single pin.

The symbol's own name. _chopiness became _. This one is a deliberate rewrite:

if isinstance(tree, ast.Module) and len(tree.body) == 1:
    top = tree.body[0]
    if isinstance(top, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
        top.name = "_"

The name a document calls is rsi — the registry key — and that is hashed separately as the first field of indicator_digest. The private symbol implementing it is provenance. Renaming _rsi to _relative_strength must not read as a change in what rsi computes, or the report cries wolf on a refactor.

One implementation detail carries a scar. The dedent is textwrap.dedent, not inspect.cleandoc, with the reason inline: cleandoc treats the first line differently from the rest, which turns a nested def into a syntax error and drops you silently into the raw-source fallback. A fingerprint that quietly degrades to hashing whitespace is worse than one that fails loudly.

What is deliberately kept

The declared output names. outputs_of("bollinger") is ('upper', 'mid', 'lower'), and those three strings are hashed alongside the source. They are not arithmetic. They are the contract a rule binds to: rename mid to middle and every document comparing close < bollinger.mid breaks, without a line of the computation changing. An engine fingerprint that only covered the maths would call that a no-op.

The shared evaluation path. Two indicators can both be untouched while the expression evaluator's comparison semantics move underneath them. Hashing expression and candle_transforms into every vocabulary digest catches the change that belongs to no single indicator.

The decorator line. @register('chopiness', pane='oscillator') survives normalisation, which means pane is inside the digest — change which chart pane an indicator draws in and the engine digest moves, even though the computed series is identical. That is a false positive. So is the name rewrite's narrowness: the rewrite only fires when the parsed tree has exactly one top-level statement, which is true of a function and false of a module, so renaming a private helper inside expression.py moves the digest too. Both err toward reporting drift that did not happen, which is the direction the fallback path also picks:

except (OSError, TypeError):
    # A C extension or a builtin. Pin what we can identify instead.
    module = getattr(obj, "__module__", "?")
    qualname = getattr(obj, "__qualname__", repr(obj))
    return f"<unparsed:{module}.{qualname}>"

with the comment: a digest that is too sensitive is still safe; one that silently covers nothing is not. An unknown name digests to the literal string "missing" rather than raising, and an indicator that got into INDICATORS without going through register digests with outputs "?" rather than raising — because an indicator the fingerprint refuses to cover is a hole in exactly the audit it exists for, and an exception here takes down a whole worker pass.

Why the digest is narrow

The registry holds 37 indicators. konis-v3 names four.

konis-v3  version e4563022622ad85b…  engine 7703cdedb92290df…
          ('ema', 'rolling_max', 'rolling_min', 'rsi')

If engine_digest covered the whole registry, shipping a thirty-eighth indicator that nobody calls would invalidate every pin in the system. Every stored result would read as drifted; the report would be all rows and no information; within a week it would be muted. Measured, by registering a new indicator at runtime and re-digesting:

registry size                        37 -> 38
vocabulary_digest(["rsi", "ema"])    906c4c2f…  ->  906c4c2f…   unchanged
vocabulary_digest()                  25be2a5c…  ->  39191573…   changed

Both behaviours are wanted. The narrow form is what a pin uses; the whole-vocabulary form (names=None) is what a fleet-wide health check wants, and it is what api/services/authoring_vocabulary.py:129 stamps on the vocabulary catalogue the builder reads.

The converse also matters, and it shows up straight away in the catalogue. Four documents name the same four indicators, so they carry four different version_ids and one engine_digest:

The 2021 variant differs from its siblings only by also naming rsi, and that one extra name is enough to give it a different engine digest — it now depends on an implementation the others do not touch. Two axes: many documents to one engine, and one document to many engines over time.

Drift is a report, not an exception

current = execution_pin(strategy)
for name in sorted(set(pin.indicator_digests) | set(current.indicator_digests)):
    was = pin.indicator_digests.get(name, "absent")
    now = current.indicator_digests.get(name, "absent")
    if was != now:
        drift.append(EngineDrift(f"indicator:{name}", was, now))

if not drift and pin.engine_digest and pin.engine_digest != current.engine_digest:
    drift.append(EngineDrift("evaluator", pin.engine_digest, current.engine_digest))

The per-indicator digests are stored on the pin — not just the rolled-up one — so the report can name the indicator that moved instead of saying only that something did. The union of both key sets with an "absent" sentinel means an indicator that disappeared from the registry reads as a change rather than a KeyError. The evaluator row is emitted only when no individual indicator moved, so a shared-path change is reported once instead of smeared across every indicator.

A version_id mismatch is appended as a row too, not raised — the caller is usually auditing a historical record, and "was this even the same document?" is a legitimate line in the same report. Against the seven documents in this catalogue and this checkout, detect_drift returns [] for all of them, which is the only interesting thing a drift check can say on the day it is written.

Where the pin is stamped — and the two places it is not checked

Stamped:

• PortfolioResult.engine_digest at backtest/portfolio.py:161, from vocabulary_digest(sorted({s.fn for s in strategy.indicators})), and emitted by to_dict(). Its field comment states the failure directly: a stored result keyed on strategy_version alone can be served back after the indicators it called were rewritten.
• SignalRow.engine_digest (planes/matching.py), with a comment explaining why it is deliberately not part of key: subscriptions match on the document, and folding the engine into the stream key would silence every bot the moment an indicator was touched.
• signal_worker.StreamSpec.engine_digest, memoised per version_id because computing it reads Python source through inspect — cheap once per document, wasteful per bar. Note what that memo key implies: an engine change inside a running process is not picked up until restart. The docstring says so.

Now the two gaps.

The signal store drops the digest on write. The worker computes it, passes it to rows_from_frame, and every SignalRow carries it. Then _document() at signals/store.py:80 builds the Mongo document field by field:

def _document(row: SignalRow) -> dict[str, Any]:
    return {
        "strategy_version": row.strategy_version,
        "pair": row.pair,
        "timeframe": row.timeframe,
        "timestamp": row.timestamp,
        ...
    }

No engine_digest. since() reconstructs rows with SignalRow(**{k: v for k, v in doc.items() if k != "created_at"}), so every row read back out gets the dataclass default — the empty string. The stream key genuinely does not need the digest; the audit trail does, and that is the thing being discarded. The evidence is computed correctly and thrown away at the last step.

The published-backtest query has no fingerprint guard at all. publishedBacktests() in mr-konis-ai, backend/src/services/strategy-backtest-service.ts:259, is the query behind the public strategy table:

const docs = await db
  .collection(COLLECTION)
  .find({
    strategy: { $in: [...labelByStrategy.keys()] },
    status: 'ok',
    pair_group: { $in: PAIR_GROUPS },
  })
  .sort({ run_at: -1 })
  .limit(200)
  .toArray();

A strategy name, a status flag, a pair-group filter. No version_id, no engine_digest, no floor on run_at. The metrics are then read straight off the stored document. A row computed by an engine that no longer exists is, at this query, indistinguishable from one computed this morning — and it will be served, because nothing in the filter can tell them apart. docs/strategy-authoring/backtest-and-honesty.md records the same gap in its status block and names the fix: gate publication on the engine fingerprint the engine already stamps.

That is the honest state. The mechanism exists and is correct; the backtest engine stamps it; the live signal path computes it. The two places where a stored number is read back out and shown to someone are the two places it is not checked.

The shape of it

Two hashes, two questions:

• version_id — is this the same document? A content hash of the normalised document. Cheap, and nearly universal.
• engine_digest — does the same document still mean the same thing? A content hash of the implementations that document actually reaches, plus the shared evaluation path. Requires deciding, function by function, what counts as behaviour: comments out, docstrings out, formatting out, the symbol's own name out, declared output names in, the shared evaluator in, and every indicator the document never touches out.

A pin that answers only the first question tells you the recipe has not changed. It says nothing about the kitchen. And a hash that nothing checks on the read path is a comment with extra steps.