Allow per target registry based cache

[dfunkt]

Description

Allow the use of a per target registry based cache instead of the gha cache. (with zstd compression if possible)

[crazy-max]

You mean have support for registry cache ? If so we would need to have signature and verification support for this exporter similar to:

• gha: add signed cache support moby/buildkit#6397
• sign github actions cache blobs #60

Also note that registry cache requires authentication to the registry so would not work on pull request from a fork as secrets are not exposed.

[dfunkt]

Yes, that is what I meant. I didn't realize that the implementation would be so involved, so if you believe that it isn't feasible right now feel free to close this issue.

[crazy-max]

We can keep this issue opened, it should be feasible yes.

Btw do you have a link to a public repo using our reusable workflow?

[dfunkt]

I've been experimenting with it in a few repos, here's one:
https://github.com/dfunkt/cloudflared/tree/experiment
Overall it's nice, but it seems slightly slower than what I was using before with the distributed builds (https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners).

A few other things that I find missing are:

1. The attestation isn't being uploaded to the Attestations section on GitHub
2. No easy way to extract the binaries from the build when pushing the image to the registry. Usually I'd have these uploaded in a target specific artifact

[crazy-max]

Thanks for your feedback!

Overall it's nice, but it seems slightly slower than what I was using before with the distributed builds (https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners).

Looking at this run: https://github.com/dfunkt/cloudflared/actions/runs/21512188664

It takes 2m51s

Old one with classic distributed build: https://github.com/dfunkt/cloudflared/actions/runs/21221275716

It takes 2m14s

I think major overhead is signature verification of all dependencies involved in our reusable workflows but also signing your attestations and verifying them. That's a tradeoff with signing involved.

[dfunkt]

I forgot to add something to my missing things list. A way to fetch the tags for the git repo would also be nice for some of my repos (something like fetch-depth: 0 from actions/checkout)

[crazy-max]

The attestation isn't being uploaded to the Attestations section on GitHub

We were thinking about using the attestation api to push the provenance for local output similar to what gh cli does: https://github.com/cli/cli/attestations. Not yet tracked but feel free to open an issue for this feature.

No easy way to extract the binaries from the build when pushing the image to the registry. Usually I'd have these uploaded in a target specific artifact

For this you can add an extra job with output: local, we are doing this in our own repos if you're interested: https://github.com/docker/buildx/blob/b7bb516036b65eaaec45188c0fa3a79655b86f40/.github/workflows/build.yml#L349-L363

We will document this use case in the future.

I forgot to add something to my missing things list. A way to fetch the tags for the git repo would also be nice for some of my repos (something like fetch-depth: 0 from actions/checkout)

Yes that's something I would like to be fixed as well: crazy-max/diun#1544 (comment)

[dfunkt]

For this you can add an extra job with output: local, we are doing this in our own repos if you're interested:

This isn't really a problem for something small like cloudflared, but for something bigger like vaultwarden this feels like a rather poor usage of compute resources. (a fresh build for vaultwarden takes ~15m, ~10m with cache)
Essentially building the same thing multiple times just to get the binaries.

[crazy-max]

This isn't really a problem for something small like cloudflared, but for something bigger like vaultwarden this feels like a rather poor usage of compute resources. (a fresh build for vaultwarden takes ~15m, ~10m with cache)
Essentially building the same thing multiple times just to get the binaries.

If multi staging is properly done in the Dockerfile to produce a runnable image and artifacts, then you could use the same cache scope for both image and artifacts.

[dfunkt]

Thanks for the suggestion, I made some modifications to the Dockerfile and the problem is now solved.

[bo0tzz]

I'd love to see this option so we can switch to github-builder from our own bespoke workflow. The GHA cache backend's quota is just too small for us, unfortunately. If there's anything I can do to help get this implemented I'd be happy to hear it.