A rebuild could clear 28% of our client’s container CVEs. The rest needed evidence.

Two of the three arguments I gave a client for why a vulnerable library couldn’t be reached were wrong. I had offered to test them the following week, tested them that afternoon instead, and found input from outside reaching the library on a path I’d assumed was switched off. Had the argument stood, the next step would have been a not_affected statement in a file the client gives its customers, whose scanners use it to hide the finding.

That’s the hard part of clearing container CVEs for a compliance audit. Patching turned out to be the smaller share of the work. Most findings had no fix, and each one needed a decision the client could defend to its auditor and to every customer who scans its images. This is what we learned doing it, written as the playbook I’d follow next time.

The client ships container images that its customers run themselves. Its compliance platform had marked the test for critical and high container vulnerabilities overdue, at 21 critical and 109 high findings, a few weeks before its SOC 2 Type 2 observation window closed, and whatever the test showed in those weeks was what the auditor could sample. Customers can’t rebuild the images, and the client’s platform team put the consequence plainly in a CI file: the customer’s scanner finds a new CVE before the company does, and the first the company hears of it is an audit letter.

I took the obvious route first. I added apt-get upgrade and apk upgrade to nine Dockerfiles, bumped the nginx base, and was ready to rebuild and redeploy.

Then I pulled the scanner’s raw findings for every production image and sorted them by one field: whether a fixed package exists. Our count was 116, the critical and high findings on the image each production service actually runs. Only 32 of them, 28%, had a fixed package to upgrade to, and the rebuild cleared 20 of those, because the service carrying the other 12 was retired. For most of the other 84, there was nothing newer to upgrade to.

Only the first bar can be fixed by a rebuild.

01/ 09Sort by whether a fix exists before you touch a Dockerfile

GCP’s Container Analysis records whether each vulnerable package has a fixed version. You have to ask for it per digest and read the field. This is the filter from our triage script, trimmed:

VJSON=$(gcloud artifacts docker images describe "$IMGREF" \          --show-package-vulnerability --format=json) # fixAvailable lives at .vulnerability.fixAvailable in newer gcloud and at# .vulnerability.packageIssue[].fixAvailable in older. Treat either as fixable.# A won't-fix is also signalled by fixedVersion.kind == "MAXIMUM".FLAT=$(echo "$VJSON" | jq -c '  (.package_vulnerability_summary.vulnerabilities // {}) as $v  | [ $v | to_entries[] | .value[]?      | { sev: (.vulnerability.effectiveSeverity // .vulnerability.severity),          fix: ( (.vulnerability.fixAvailable == true)                 or ((.vulnerability.packageIssue // []) | map(.fixAvailable == true) | any) ),          cve: (.vulnerability.shortDescription // .noteName // ""),          pkg: ((.vulnerability.packageIssue // [{}])[0].affectedPackage // ""),          fixed: ((.vulnerability.packageIssue // [{}])[0].fixedVersion.fullName // "") } ]')

That gives a JSON array per image. The script then writes every element to a CSV, one row per image and CVE, with the image and digest in front:

echo "$FLAT" | jq -r --arg reg "$REG" --arg img "$SHORT" --arg dig "$DIG" \  '.[] | [$reg,$img,$dig,.sev,(.fix|tostring),.cve,.pkg,.fixed] | @csv' >> "$CSV"

The CSV keeps every severity. Keeping the critical and high rows for the latest production digest of each service gave the 116, and fix says whether a rebuild can help. The MAXIMUM signal in the comment is in the record too, but we filter on the boolean. Each row then goes down the same path:

The rest of this post is mostly about the orange question: what counts as evidence.

02/ 09The scanners never agreed on a number

Before fixing anything, I had to decide which count I was fixing. They didn’t match.

The compliance platform’s count moved on its own. During our sweep its critical findings went from 21 to 15 to 17, and its highs from 109 to 76 to 79, while a live scan of the registry on the same day showed 27 critical and 96 high. The platform reads the registry on its own schedule, so it lags. The registry also held more than 380 stale image digests that no environment ran, and every raw count included them. Weeks later the client’s security lead saw the platform reporting findings deduplicated by CVE, a different count again, and asked the question that matters: which number does the auditor actually see?

The fix flag needed checking too. GCP’s flag answers “is there a fix in this image’s Debian release?”, not “does a fix exist?”. The critical CVE in the reachability section below had no fix in Debian 12 and one in Debian 13, so the registry called it unfixable while the real fix was a base image upgrade.

The platform’s own total was 130, because it also counted images outside the production services. So the 116 is our own count: critical and high findings on the latest digest each of the ten production services actually runs, one row per image and CVE, with the date of the scan. We reported that definition alongside the number every time.

03/ 09Ask whether the package needs to be there

I had expected the rebuild to do most of the work, so the first bucket surprised me most. Every fixable OS finding sat on just two images, the web frontend on Alpine and one Debian-based service. The Python images had none: every OS package flagged on them was one Debian hadn’t fixed yet. So most of my Dockerfile edits changed nothing the platform was counting. And the Debian service carrying 12 of the 32 was retired a couple of weeks later, so the rebuild itself cleared 20.

The Python dependencies were the easy part. They lived in uv.lock, where no base image can reach them, and the fix was a lockfile bump and a normal deploy. cryptography, four of our 18, is the one that catches people. Its wheels bundle their own OpenSSL, so bumping the system libssl does nothing for it.

For the biggest bucket there was no fix at all, so the useful question was whether the package needed to be there. Perl alone accounted for 28 findings, four CVEs on each of seven Debian images, and 14 of the criticals, and none of these services runs Perl. It’s there because Debian marks perl-base as Essential, which I didn’t fully appreciate until I tried removing it from the real base image and apt-get remove refused without --allow-remove-essential. The rest were graphics libraries pulled in by a document-conversion service, plus attr, acl and one small command-line utility. You can’t remove Perl from a Debian image without forcing dpkg past its own safety check, but on distroless it never arrives. Both services already on distroless scanned at zero, which was the best argument I had for moving the Python images the same way. That move hadn’t happened by the time of the register below.

The last nine I didn’t want to clear myself. They were four libraries on two images that outside input might reach, so we kept them open for the client’s security owner.

04/ 09Pin what builds your images, including the scanner

I’d also rebuild differently now. An apt-get upgrade line means the same commit builds different bytes on different days. I flagged that in our spec and accepted it for speed, and about two weeks later the client’s platform team made the better call. They banned the upgrade line and pinned every base image by digest, with Renovate bumping the digests. The two Renovate settings from our proposal that do the work are these. The first pins every base image by digest and the second holds back releases younger than three days:

docker: { pinDigests: true },minimumReleaseAge: "3 days",

To keep the fixable count from growing back, we also proposed a Trivy gate in CI. Our version scanned each built image and referenced the action as aquasecurity/trivy-action@0.28.0, a version tag. Months earlier, attackers had repointed most of trivy-action’s tags at credential-stealing commits (CVE-2026-33634), and 0.28.0 was in the affected range. The bad tags were live for about half a day in March and had been restored long before our proposal, so the tag we used was clean by then, but pinning a scanner by tag was still a mistake. Our Renovate config would have pinned it by digest on its first run, but the workflow file we proposed still named the tag, and that’s what would have run first.

The team merged a different shape. It scans the repo’s lockfiles and pins the action by commit SHA. It ran report-only at first, because blocking on day one would have failed every PR for findings its author didn’t cause. This is the core of the step after they made it blocking, once the fixable backlog was clear:

uses: aquasecurity/trivy-action@<commit SHA>  # v0.36.0with:  scan-type: fs  severity: CRITICAL,HIGH  ignore-unfixed: true  exit-code: "1"

Notice that this gate scans lockfiles, so a new fixable OS package in an image won’t fail a PR; pair it with an image scan. Trivy also isn’t the auditor’s scanner. With ignore-unfixed, Trivy skips the won’t-fix findings that Artifact Registry still counts, and its database had gaps too: at one point it reported nothing across the repo’s lockfiles while Dependabot had three high alerts open against the Python one.

05/ 09Treat every VEX statement as a claim you’ll have to defend

I made most of my mistakes in the VEX file.

A VEX statement is a machine-readable claim that a product isn’t affected by a CVE, with a justification code, and scanners that read VEX can suppress the finding. We wrote one per CVE, each listing the images it covered. Here is the shape, with the CVE, registry and prose removed:

{  "vulnerability": { "name": "<a perl CVE>" },  "products": [    "pkg:oci/<image-a>?repository_url=<registry>",    "pkg:oci/<image-b>?repository_url=<registry>"  ],  "status": "not_affected",  "justification": "vulnerable_code_not_in_execute_path",  "impact_statement": "<why this package's vulnerable code can't run here>"}

Every impact statement followed one template: the package came with the Debian base, and the service never invokes it. For Perl the claim was at least checked, with a code search and a look inside each image: nothing in these services executes the perl binary. Nobody ran the services and watched for it, though, and I come back to that below. For the small utility it was wrong twice. During the next evidence pass, I found it was installed on purpose in a sandbox service, and that the service can run it.

My first correction kept not_affected and switched the justification to inline_mitigations_already_exist, since the sandbox is isolated. One of the client’s engineers pushed back, and they were right. OpenVEX reserves that justification for controls that prevent exploitation, and isolation only limits the damage after a bug fires. That’s an argument for accepting a risk, and I hadn’t thought about the difference until then.

The utility became a risk acceptance with a named owner and a review date, and its statement came out of the file. The graphics-library statements went the same way: running without a display makes that code less likely to run, without showing that it never does. Each of those 15 CVEs sat on a single image, so they became the 15 risk-accepted rows. Six statements survived: four Perl CVEs plus attr and acl. Each applied to the six Debian images still running (the seventh was the retired service), so that’s 36 rows.

VEX files get passed to customers, and their scanners suppress whatever you mark not_affected, so I went back over the six that survived. Writing this up, I found one more problem. The same reasoning applies to Perl, attr and acl on that sandbox, since it can run those binaries too. Those six sandbox rows were still not_affected in the final register, and they belong in the risk register next to the utility. We raised the six sandbox rows with the client’s security team, and they’ve since been fixed in the sandbox image itself, so the file no longer has to argue them. (Our file also listed products as plain strings, the older OpenVEX shape, where version 0.2 expects {"@id": ...} objects.)

06/ 09Give every decision an owner and a date

For every finding we couldn’t patch, the client needed a decision that would hold up in front of the auditor: removed, not affected with evidence, or accepted by someone with the authority to accept it.

Each risk acceptance had a named owner and an expiry date, 90 days out for the first batch. The client’s own audit response explains why the date matters more than the wording: an item held without a date can’t be told apart from an item forgotten, and a forgotten item is what the original finding was. The same goes for anything you claim reduces the risk. Our first draft listed compensating controls, and the client’s CTO removed the ones he checked and found weren’t actually in place in production.

Timing mattered as well. One of the fixes merged on the last day of the observation window. The scan that proved it ran three days later, just outside the window, so the evidence and the period it covered had to be explained together. Evidence screenshots also had to show the computer’s clock, and they expire.

07/ 09Test reachability before you argue it

A later scan turned up a new CVE, not one of our 116, in one of those four libraries. I had a reasonable-sounding case that it couldn’t be reached, and I’d written it up for the client’s security lead. It rested on three arguments about which code the service runs and how it’s deployed. I offered to test it “next week.”

I tested it that same afternoon instead. I pulled the production image by its exact digest and ran it on my own machine, logged every library load with LD_DEBUG, and preloaded a small shim with one wrapper per parsing entry point. This is the pattern, simplified, with the library’s function name and arguments replaced by generic ones:

#define _GNU_SOURCE#include <dlfcn.h>#include <stddef.h> /* Excerpt. LD_PRELOAD wrapper: log the buffer, then call the real function. * note_item() is the shim's logging helper (not shown); it appends * "name len=<n> pfx=<first 16 bytes as hex>" to a log file. */int target_parse(const unsigned char *p, size_t n) {    static int (*real)(const unsigned char *, size_t) = NULL;    if (!real) real = dlsym(RTLD_NEXT, "target_parse");    note_item("target_parse", p, n);    return real ? real(p, n) : -1;}

Build the full shim with gcc -shared -fPIC -o trace.so trace.c -ldl and run the target with LD_PRELOAD=./trace.so LD_DEBUG=libs. Because each log line carries the first 16 bytes of the buffer, you can match it against your own test data, and a call that logs your bytes was caused by your input.

I built three inputs that exercise the library, and two controls that don’t:

Every logged prefix matched my test data, and I sent the inputs through the service’s real HTTP endpoint, the way a user’s file would arrive. One of my arguments was a setting that turns the relevant feature off. Another was a file that feature needs, which the image doesn’t ship. The third was that the container has no outbound network. The first two assumed that feature was the only way into the library, and it wasn’t. Any uploaded file that carries a digital signature gets its certificate decoded on an ordinary request, whatever the setting says, and the decoding happens in memory, so the missing file never came into it. No outbound network was true, but this bug is triggered by data a user uploads, so it didn’t need one. I hadn’t proven the vulnerable function runs, but I had shown that bytes an outside user controls reach the library, which is enough to rule out not_affected.

One limit before you reuse this: a shim only sees calls that cross into the library, and any test, including a breakpoint or a uprobe on the vulnerable function itself, only covers the inputs you tried. If a test finds nothing, your inputs didn’t get there, and that’s all you’ve learned. For not_affected I’d want the vulnerable code absent from the image, or no way for outside input to reach it at all. I handed the results to the client’s security owner. In the end that CVE didn’t need his signature. The next Debian release had the fix, so we rebased the image onto it, compared the service’s output against the old image, and withdrew the risk acceptance that had been waiting for him.

08/ 09What left the image, and what we argued away

Patched (orange) and retired with a service (dark) add up to the 58 findings that left the images.

This is the client’s register three weeks after the finding, which we reviewed with them. Nobody has rescanned all 116 rows since, so “patched” here means the fix shipped, not that a scan confirmed it. By the register’s account, 58 findings left the images. Patching took 40 of those, including the 20 the rebuild itself cleared, and the retired service took 18. 36 stayed as not_affected, and I now think six of those rows are wrong. 15 became risk acceptances with an owner and a review date, and 7 stayed open.

Two of the patched rows need explaining. Both were among the nine we’d held back, and both were fixed upstream after our scan. A later rebuild installed the fixed version, but the register kept listing them as open, because someone had refreshed the version string without checking whether it was the fixed one. The register was corrected on the day of this snapshot, and the figure shows the corrected state.

Meanwhile the register kept growing. In the same three weeks, newly published CVEs against packages still in the images added 38 more not_affected rows. 18 were Perl. Fifteen of those rest on the same reasoning as the original Perl statements, so they’re in the same class, argued but untested, though it’s a strong argument for a binary nothing calls. The other three are on the sandbox and have the same problem as the six above.

The other 20 are for a library that the code links but, according to a code trace, never calls. That claim is weaker, and the test above showed how easily a reasonable argument about code paths turns out wrong. Those 20 rows need a test before I’d stand behind them.

After this snapshot, the rebase from the test section also took one of the held-back libraries out of that image entirely. For the sandbox utility and a held-back library that still had no fix anywhere, the client’s security owner told us he’d rather write “reachable, no fix published” and own it than mislabel it, and he put the risk acceptances together on that basis. The client’s platform team also planned to retire the service that carried most of the open rows. Testing the 20 linked-library rows was outside our scope, and so was the first 90-day review, so I can’t tell you how either landed.

09/ 09The playbook

1. Define the count before you fix it: critical and high findings on the digest each production service actually runs, one row per image and CVE, with the scan date. Prune stale digests.
2. Sort by whether a fix exists in the image’s own release before touching a Dockerfile.
3. For findings with no fix, ask whether the package needs to be in the image at all. Distroless removes what Debian won’t let you uninstall.
4. Pin base images by digest and the scanner action by commit SHA. Gate CI on fixable findings, report-only until the backlog is clear, and scan both images and lockfiles.
5. Treat every not_affected statement as a claim your customers’ scanners will act on. Cite isolation in a risk acceptance, where it belongs.
6. Test reachability before you argue it, and remember a test that finds nothing only covers the inputs you tried.
7. Give everything left a named owner and an expiry date, and check that every compensating control you cite is running in production.

I did this work myself as part of a XORS engagement. Thanks to the client’s platform and security team, who reviewed every PR and caught the VEX mistake above.