In late August I released version 0.9.3 of einvoicing, my e-invoicing gem. It adds and fixes no feature. Version 0.9.2 would not load in a project that did not already have bigdecimal, while its 317 tests passed on my machine.

Works on my machine

The gem uses bigdecimal for amounts and rexml to read XML. Ruby used to install them itself, they now have to be declared as dependencies, and I had not declared them. On my workstation, other gems already pulled them in. Anywhere else, require "einvoicing" failed right away, and Peppol validation failed as soon as it read XML.

The tests run in the repository. The published package only holds the files listed in the gemspec and the declared dependencies, and it installs on a Ruby I do not control. No test looked at that package.

Two blocks joined by the gem build command. On the left, the repository: the files, the gems already on the workstation, a Gemfile.lock resolved on Ruby 4.0, checked by the 317 tests. On the right, the package: only the listed files, only the declared dependencies, installed on the user's Ruby from 3.2, checked by the new check. Below, what can go missing on the way: a dependency, a file, a 3.2-compatible version. — The tests check the repository, the user installs the package.

Three gaps between the repository and the package

A dependency present on the workstation and missing on the user’s side. That is the gap that shipped.
A file read at runtime but missing from the gemspec. The gem embeds an sRGB profile in every PDF/A-3: drop it from the list and every test stays green while the published gem can no longer produce a compliant invoice. I caused it on purpose, and no check saw it.
A lockfile resolved on Ruby 4.0, pinning versions that require Ruby 3.3 while the gem promises to run on 3.2.

The checks added

The first builds the gem as for a release, installs it in an empty environment and loads it. It then compares, file by file, what the repository uses at runtime with what the package contains. An undeclared dependency fails the load, and a missing file is identified.

The second walks the lockfile and verifies that every version accepts Ruby 3.2. A gem it could not verify, for instance when RubyGems rate-limits the requests, counts as a failure.

Those checks had holes of their own, flagged in review. The first loaded the package without looking at its contents, so removing a translation file left everything green. Once fixed, it verified that a gemspec glob matched something instead of checking every expected file. I reproduced each hole by breaking the package on purpose before fixing it.

Beyond Ruby

The same gap exists wherever you test one thing and ship another: an installer tried on the machine that compiled it, a Docker image built with a local cache, a mobile app validated on the developer’s phone.

If you have software built for you, ask what the last check before delivery runs on. If it runs on the contractor’s machine rather than on what you receive, it does not cover what you receive.

Version 0.9.3 and version 0.4.0 of einvoicing-connect have been out since August 31. Their continuous integration now runs the tests on Ruby 3.2 and 4.0, installing the package in an empty environment on both, and the lockfile check.