In the past couple weeks, I’ve learned more about renovate, SBOMs, provenance and attestations than I ever wanted to know. (But if I’m being honest, I do enjoy learning a bit more about it.)
Backstory is that I decided to make CNPG-Extensions an actually serious project. The original name was “Not-CNPG” as a joke about CNCF’s restrictive licensing policies which forbid hosting open source software with licenses like GPL. https://github.com/cnpg-extensions/
As a “serious” project I wanted to provide provenance info so users can more have assurance about the contents of a container image, and so that scanners can accurately report licenses and compare software versions against vulnerability databases. This week I also started exploring support for pgrx extensions with full rust dependency graphs in the SBOM so that tools like trivy can flag RUSTSEC vulns even on packages buried in the dependency tree.
Example Trivy output for a Debian-based extension:
Example Trivy output for a pgrx-based extension (this is not final):
A few things I’ve learned along the way:
- Renovate auto-update problem: some extensions (MySQL FDW, PL/Debugger) have a sql version that’s completely different from the package version. There’s no way to know the SQL version outside of manually inspecting source code or firing up a full test container.
- Renovate auto-update decision: I don’t want to promote “release candidate” or “beta” versions on channels that users consider to be stable releases. How this is reflected in a version string varies by extension; requires manual check before promotion. But I want stuff as automated as possible ~ generally I don’t want to have to be approving PRs all the time. I might do a little research and only disable auto-update for extensions that have had beta/rc versions in the past.
- Two ways to sign attestations: Cosign/Sigstore or GitHub Artifact Attestations. CNPG currently uses Cosign. I ended up trying out GitHub Artifact Attestations… until a few days later when I stumbled across a random GH Issue on the ParadeDB project which pointed out that GitHub might have paywalled some of the functionality here behind Enterprise subscriptions (I think for people mirroring repos). So then I went and migrated all my working branches back to Cosign, re-ran tests, etc.
- SBOM problem: docker/moby buildkit uses syft to generate an SBOM. latest version of syft still has open issues around debian packages and basically it can’t detect licenses for a huge number of them. so the default docker SBOM is always missing a bunch of license info.
- my workaround: adding CodeScan. then CodeScan promptly crashed on the PL/R license file because the file was too big. so i split the file into chunks using a marker at the beginning of each license and scanned each file. this worked and gave me a super highly reliable license scan, though I had to manually re-assemble an SBOM.
- If you install the latest rust using the method where it compiles cargo-about and cargo-cyclonedx, those two packages take… a loooooong time. For two large pgrx builds (pg-parquet and pg-search), my GitHub action workflows hit the six hour timeout and got killed.
- Turns out trivy can also generate SBOMs including Cargo.lock parsing… but you need the latest version because even something as recent as 0.66.0 has bugs where it couldn’t parse Cargo.lock if the project used workspaces. https://github.com/aquasecurity/trivy/issues/10007
- Even the latest version of Trivy has a bug where it misses git-only deps. AI told me there wasn’t an existing issue… hopefully was right; I filed a new one https://github.com/aquasecurity/trivy/discussions/11236
- Trivy doesn’t seem to have any capability to emit CONTAINS records in the SPDX SBOM. So I still need syft because I want to work backwards from files actually copied to the extension scratch container, to figure out which packages are included.
- “Modern” software languages (um, like less than a couple decades old?) that handle dependencies are wonderful. But I wanted to add pg-duckdb which is in C++ and so now I need bespoke pg-duckdb dependency processing build scripts… at least this gets easier with AI agents to help do the coding… but yeah not wanting to commit myself to maintaining any custom build systems unless it’s really worthwhile (AI or not)
- GitHub CI being as powerful as it is – and offering free compute – is a much more significant contribution to Open Source than they get credit for. My project to build, test, host and distribute a bunch of kubernetes postgres extensions with strong provenance across two architectures, two major Debian OS base containers, three versions of CloudNativePG… this can trigger some rather impressive counts of GitHub jobs! Like hundreds! And I’m amazed just how much compute GitHub gives away for free, which supports Open Source projects like this.
There are still some big open questions in my mind around how to best manage Postgres Extensions with CloudNativePG. How high of a bar for contributions? How much review is needed? What level of commitment from a contributor is expected – do we want to avoid drive-by contributions of big chunks of code which could become a liability – and how to decide whether to trust someone?
Maybe my CNPG-Extensions project can be an option for a lower bar, in addition to the licensing concerns. But I’m not sure. How important is provenance? I’ve put a lot of effort into it this past week. I’m not sure about random debian packages downloaded from GitHub; how do we know they were built correctly? Do we care? If a package is in the official Debian or PGDG repositories then I tend to have a little more trust in it – is that justified?
And then there are the AI related topics: just because you can ask an AI agent to rewrite the operating system on your laptop, doesn’t actually change the fundamentals of computing that much. Decades ago I had fun running my own wordpress site. I learned a bit and it fueled my enthusiasm. But after the third time cleaning up a hack I decided I’d rather spend my time elsewhere, and I started paying someone else to manage that part. AI agents are getting lots of people excited about coding again which is great. But eventually it comes back to boring and well maintained platforms.
Anybody can throw some code on the internet. That doesn’t mean anybody will maintain the code – ensuring that things are rebuilt after Log4Shell happens again, two years from now. AI is great at writing code but you still better pay attention to what human is at the steering wheel and whether they seem like they are paying attention, or seem like they have any level of commitment to stick around. Or alternatively you need to be willing to take full ownership/responsibility yourself.
I like the OpenSSF scorecard for this; it’s worth a read. https://scorecard.dev/
Just a reminder that AI doesn’t change this – it’s more important than ever.
And a final random musing… here’s a picture of my whiteboard right now where I’m trying to track a hierarchy of five different work streams that I have going all at the same time! Each is a separate git branch, forked from the branch above it (and requiring a rebase whenever an upper branch changes something). Agents mean I can fire off long-running tests and come back 12 hours later to check results and make design decisions, but having five different work streams in parallel is a lot of mental overhead – it’s a little tiring!
Ardent Performance Computing 
Discussion
No comments yet.