The benchmark repositories are designed to make narrow product claims testable. They do not claim that terminology drift is solved for every project, model, or corpus.
The same 10 coding tasks were run with and without a canonical vocabulary brief: 3 repeats, 2 conditions, 60 runs total.
Synthetic paired benchmark, Claude Sonnet 4.6, temperature 0. The independent scorer reads raw model output.
Open benchmark repository →SkeinRank discovered cross-version terminology candidates without a gold term list, known-renames file, release-note hints, or labels during discovery. Human labels were applied after ranking.
The harder Airflow 2.3 → 2.7 window remains published at P@5 40% and P@10 20% rather than being hidden.
Open benchmark repository →For Airflow 2.7.0 → 3.0.0, the top-ranked candidates included asset, auth manager, AssetAlias, ObjectStoragePath, asset events, bundles, outlet_events, dag bundles, and JWT.
The maintained true label airflow.sdk was discovered by core but excluded by the prose-only review policy because all 79 occurrences were code context. The report records that failure stage instead of treating the candidate as unknown.
The OSS drift report measures terminology residue and naming transitions in real open-source histories. It complements the product benchmarks by showing why renamed concepts remain operationally difficult long after a code rename is complete.
Always state:
