Cache hit rate is 98%, so why does the release keep shipping an old version?

Final text.

If the words "98% cache hit rate" are taped to your retro board, it's time to read that number in reverse. A hit rate doesn't measure correctness. It only counts how heavily you reused old results. There is no guarantee anywhere that those old results are still the right answer. Most incidents where releases keep shipping old versions start exactly at this point.

The higher the hit rate, the longer you stay stale

In pipelines that adopt a dependency cache, releases slipping by a week is not rare. The typical case is when the key doesn't look at the lock file. If the actions/cache key only reads the package.json hash, the key stays the same even after package-lock.json is updated. At the restore step, last week's entire node_modules gets pulled up. If you run npm ci here, you dodge the damage. ci rebuilds the tree from scratch based on the lock, so the cache doesn't break reproducibility. The real danger is a setup that skips the install step or keeps using a partial restore as-is. Such a pipeline treats new dependencies as already installed, and ships the old dependency versions with a green checkmark. The failure leaves no trace in the logs. Tests pass too. It just ships "the old thing" about every other time.

These incidents are caught late. What shows up in the run summary is just a green badge and a Cache hit marker, so the problem only becomes visible when you compare against the next release. Meanwhile, the cache masquerades as a "well-running optimization" and survives for weeks more. The moment the hit rate looks most reassuring is exactly the moment to be most suspicious. A cache hit means, after all, "I used yesterday's answer unchanged today."

When the lock isn't in the key, an invisible old version gets shipped

It's worth splitting up what gets cached. Caching node_modules in one piece and caching a package store like ~/.npm are completely different in nature. The npm cache is content-addressed and the lock file has integrity hashes baked in, so even if it's slightly stale, wrong versions rarely leak through. A node_modules snapshot has no such safety net. If your target is the latter, you need to hold the key tightly.

The key should only hash files that fully determine the dependency tree. package.json, the lock file, and .npmrc are enough. Adding the CI script hash on top of that only crashes your hit rate. Putting files whose hash changes often into the key is the same mistake. Even changing just the description field in package.json grinds the key, and as a result the cache almost never hits. Conversely, if a file left out of the key actually changes the tree, the lock update quietly gets buried.

One more thing to watch for: the lock file is in the key, but that lock is not actually being honored. If you started with no lock in the repo and someone pulled new dependencies with npm install, the lock that's committed to git and the tree at run time are already out of sync. The key sees the intact original lock and rules it a hit. Only the installed result gets cached as new. In this case, npm ci actually makes things worse. If the lock is skewed, ci itself fails. Making ci always pass comes before cache design, not after.

A matrix key has to carve the environment into it

Leaving matrix variables out of the key is the most common and most expensive mistake. When jobs with different OS and Node versions share the same key, one side's saved node_modules gets restored by the other. Native modules break on the spot. sharp, esbuild, and the node-gyp family are the classic examples. The error suddenly appears mid-build and produces lines like "it worked until yesterday." Cases of macos runner artifacts being restored on ubuntu are easy to find.

key: ${{ runner.os }}-${{ matrix.node-version }}-dep-${{ hashFiles('**/package-lock.json') }}

That's basically the whole skeleton of the rule. You keep only the changing environment (OS, version) and the determinant of the dependency tree (lock) in the key. Add restore-keys on top and one more trap appears. restore-keys automatically finds previous caches when the key misses. As convenient as it is, it's blunt. Even when a new key is created because the lock changed, the fallback pulls up a cache from ten days ago. In a workflow that uses ci, the shield comes back up here, but a setup that keeps using install ends up with dependencies diverging forever. The scenario where a minor version update drops out of releases for days repeats at this exact point. If you don't want fallback, either drop restore-keys entirely or append an explicit version number to the value so previous caches are only allowed when you want them.

While we're cleaning up the matrix, let's also look at splitting dependency layers. The cache should use different targets and keys for each build stage. Dependency installation and bundle build produce unrelated outputs, so mixing them in the same cache lets a change in either side drag down the entire hit rate. Store the install stage under a lock-based key and the build stage under a source-hash-based key separately. If they're mixed, you get a trade-off that's hard to keep: more cache saves and less usefulness.

A build cache trusts the artifacts wholesale

Setups that store the whole bundler incremental cache (.next/cache, esbuild, tsc incremental) are also common. If the key is independent of the source hash, the cache mistakes it for "nothing changed." Intermediate artifacts get restored and promoted to artifacts as-is. If a broken incremental cache is restored, the next build can struggle with unexplained parsing errors. For caches like these, lifecycle rules matter more than the key. Save only after a successful build, and discard when the source hash changes. Never save the artifacts of a failed build.

What's often missed here: attaching the full list of input files to the incremental cache key isn't worth the cost. Instead, put a single source-tree hash in the key and handle the discard condition at the job level. For example, running a cold-build job periodically to flush out the incremental cache is lower maintenance.

Older keys can also get evicted because of cache size limits. When you hit the per-repository cache cap, the hit rate suddenly drops to 0 and a build that usually took 40 seconds shoots back up to 5 minutes. To diagnose that day's failure as a cache problem, checking the restore step logs for whether a Cache not found message appears is the right order. Storing binary artifacts also means weighing compression efficiency. Even bundled as a tarball, node_modules easily exceeds several hundred MB, so you have to do the math between the 10GB storage cap of actions/cache and per-runner restore time. If the upload and restore time outweighs the time the cache saves, a clean install beats using a cache.

PRs that only fail when the cache is emptied

All of these incidents come from builds that only pass when a cache is present. So validation should also be done without the cache. Add a job that periodically injects a date into the key to force a full cold build. Once a month is enough. What's more useful than whether it passes is comparing artifacts. If the cold build's artifacts and the usual artifacts have different hashes, you need to chase down which side is stale. Usually it's not hard. Print the package version string or bundle header inside the artifact and compare it against the expected version, and the answer appears immediately. Running this comparison weekly narrows down the point where the cache went bad to the commit level.

There's a reaction you meet when first introducing this validation: "I'm saving two minutes a run thanks to the cache; does it make sense to spend twenty minutes on validation?" There are two answers here. The time saved repeats on every run, while a broken validation only has to fire once to cost more than that. For a fair cost estimate, measure build time for two weeks before and after introducing the cache, convert the time saved to an average, and set it against the time lost to a bad release. Since the validation job is a diagnostic rather than a build, the sensible setup is one where a failure doesn't block deployment.

A hit rate is just a number that feels good. If download time was cut in half, that's a success. But you shouldn't drag that number out as evidence of correctness. If you have to pick a metric, choose "are there PRs that only fail when the cache is emptied" over the cache hit rate. If you can't yet answer "no" to that question, the safer move is to reduce the information going into the key. The less information a key holds, the fewer places it can go wrong. It starts with a single lock file.

Comments

Loading comments.