December 2021 delivered Log4Shell, and the subsequent weeks were chaos. A month later, we’re reflecting on what worked, what didn’t, and what we’re changing permanently.

The Problem

The vulnerability itself was severe—remote code execution with trivial exploitation. But the real challenge was answering a simple question: “Are we affected?”

Surprisingly difficult. Log4j wasn’t just a direct dependency. It was nested inside other libraries, bundled in commercial products, embedded in tools we’d forgotten we deployed. Our initial inventory missed half the instances.

Even after identifying affected systems, patching wasn’t straightforward. Some applications bundled Log4j in ways that couldn’t be updated without vendor releases. Others had complex dependency trees where multiple versions coexisted.

The vulnerability disclosure happened on a Friday. By Monday, active exploitation was widespread. Response time mattered.

Our Solution

Software Bill of Materials (SBOM) generation became mandatory for all builds. We integrated Syft into CI pipelines to produce SBOMs automatically. When the next vulnerability hits, we’ll query the SBOM database rather than searching filesystems.

Container image scanning expanded beyond build-time. We enabled continuous scanning in our registry, alerting on newly discovered vulnerabilities in existing images.

Dependency pinning and review tightened. No more pulling “latest” tags. Explicit versions in lock files. Automated PRs for dependency updates with security context.

Incident response runbooks now include supply-chain scenarios. Who owns the relationship with each vendor? What’s the fallback if a patch isn’t available? How do we implement compensating controls?

Network egress filtering was already in place but proved valuable. Even vulnerable applications couldn’t reach attacker infrastructure when egress was restricted.

The Benefits

The next critical vulnerability won’t require manual archaeology. SBOM queries can identify affected components within minutes rather than days.

Vendor conversations are more productive. We can tell them exactly which product versions we run and ask specific questions about their remediation timeline.

The security team earned credibility with the business. Rapid, coordinated response during Log4Shell demonstrated the value of security investment. Future security initiatives face less resistance.

Log4Shell was painful, but it forced improvements we’d been postponing. The next incident—because there will be a next incident—will find us better prepared.