commit signing in 2023 is kinda wack

30 Dec 2023

I’ve been taking a look at authorisation processes for version control systems at work and had three colleagues independently ask about scope for git commit signing, but they were surprised to discover I’d been soured toward traditional approaches for git commit signing and generally advise shying away from signing due to complexity.

I’ve previously tried using GnuPG for commit signing a few years ago but was frustrated by impracticalities caused by pinning trust to long-lived cryptographic identities, so have kept a hoping ear toward advancements in commit signing.

A personal blog post is better for clout than an internal Confluence page (and besides, it’s probably responsible to spend time proving my assumptions about current commit signing options + seek advice from internet friends before recommending waiting on signing enforcement).

wolfgirls can have a little crypto (as a treat)

Before we get deep in the weeds, I’d like to explicitly note I’m not here to spoil anyone’s fun.

I get it- it’s fun to cosplay as a cryptography enthusiast. I used to sign my emails with GPG (before xkcd#1181 lol), I’d travel for key signing parties at conferences, I even carry Yubikeys on my collar these days.

As much as I’d like to believe in web of trust as a mechanism to bootstrap trust in folks’ cryptographic identities, I kept getting burnt by the impracticality of maintaining a long-lived cryptographic identity.

Key compromise happens, key loss happens, and identities change over time.

I gave up on the web of trust after my identity drifted from my government-issued documentation (🏳️‍⚧️) and key signing parties became awkward. I could create a new GPG key and sign it with my old one [1, 2], but the cracks beginning to show in popular keyserver infrastructure discouraged me from bothering.

wolfgirls literally only want three things and they’re disgusting

We’ll cover GPG’s interactions with git in more detail later on, but let’s take a step back and outline which properties I’m hoping to find in a commit signing solution:

• short-lived cryptographic identities
  • …so signing processes should avoid binding trust to something an engineer knows or directly manages (private keys)
• clear revocation procedures (backed by trusted timestamps)
  • …so cryptographic material compromise is a non-event (laptop/security key theft, malware)
• straightforward local verification (no web UI badges)
  • …so deployment infrastructure can audit new code before admission to production

These goals are inspired by the attacks on @ledgerhq/connect-kit (2023), git.php.net (2021), and github.com/gentoo (2018).

our options

I’m not sure the options currently made available to software engineers through popular git platforms allow us to bestow these guarantees on commits, instead mostly relying on enforcing commits signature validity when admitting pushes and accepting older commits will eventually become invalid.

In an effort to prove this, we’ll look at GitHub and GitLab’s offerings and try to demonstrate pitfalls in cryptographic key material management. As of writing, both platforms permit on-platform verification for commits using GPG, SSH, or S/MIME keys.

gpg keys

GPG is cryptographically trash. I won’t spend time here rehashing arguments made by people far more qualified on the matter, but I can recommend Moxie Marlinspike’s GPG And Me (2015) if you’ve not had a chance to read it before.

Still- it’s the oldest mechanism for git commit signing, so we’ll take a moment to run through GPG’s setup, creating signed commits, and validating them.

I’ll be following GitHub’s advice for generating a new GPG key to set up my local environment for git commit signing with GPG. This’ll help us ensure we use an algorithm type that’s supported by GitHub and GitLab.

First, let’s create a fresh key:

$ gpg --default-new-key-algo rsa4096 --gen-key

gpg --gen-key output (expand)

gpg (GnuPG) 2.3.8; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: directory '/Users/htw/.gnupg' created
gpg: keybox '/Users/htw/.gnupg/pubring.kbx' created
Note: Use "gpg --full-generate-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: Harley Watson
Email address: [email protected]
You selected this USER-ID:
    "Harley Watson <[email protected]>"

Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /Users/htw/.gnupg/trustdb.gpg: trustdb created
gpg: directory '/Users/htw/.gnupg/opengpg-revocs.d' created
gpg: revocation certificate stored as '/Users/htw/.gnupg/opengpg-revocs.d/BFFF2D87B274A1F4D82499F0FAE54A86444FA9AA.rev'
public and secret key created and signed.

Note that this key cannot be used for encryption.  You may want to use
the command "--edit-key" to generate a subkey for this purpose.
pub   rsa4096 2023-01-22 [SC] [expires: 2025-01-21]
      BFFF2D87B274A1F4D82499F0FAE54A86444FA9AA
uid                      Harley Watson <[email protected]>

Next, we’ll export the key to our pasteboard and add it to GitHub and GitLab.

$ gpg --armor --export BFFF2D87B274A1F4D82499F0FAE54A86444FA9AA | pbcopy

We can now tell git about our GPG key, making it available for commit signing.

$ git config --global --unset gpg.format
$ git config --global user.signingkey BFFF2D87B274A1F4D82499F0FAE54A86444FA9AA

We can now make a test commit that’s signed with GPG.

$ mkdir gpg
$ echo "OK" > gpg/example.txt
$ git add gpg/example.txt
$ git commit -S -m "add GPG-signed example.txt" && git push

We’re also able to verify the commit as appropriately signed through VCS web UIs.

Now, let’s throw a wrench into the situation and assume both my GPG and SSH keys have been compromised (allowing an attacker to sign + push commits).

Despite a lack of trusted timestamps in git commits, we’ll try be fair to GPG and cut a revocation certificate at the current timestamp, rather than using the one precreated for us at key generation.

$ gpg --output revoke.asc --gen-revoke BFFF2D87B274A1F4D82499F0FAE54A86444FA9AA

gpg --gen-revoke output (expand)

sec  rsa4096/FAE54A86444FA9AA 2023-01-22 Harley Watson <[email protected]>

Create a revocation certificate for this key? (y/N) y
Please select the reason for the revocation:
  0 = No reason specified
  1 = Key has been compromised
  2 = Key is superseded
  3 = Key is no longer used
  Q = Cancel
(Probably you want to select 1 here)
Your decision? 1
Enter an optional description; end it with an empty line:
>
Reason for revocation: Key has been compromised
(No description given)
Is this okay? (y/N) y
ASCII armored output forced.
Revocation certificate created.

Please move it to a medium which you can hide away; if Mallory gets
access to this certificate he can use it to make your key unusable.
It is smart to print this certificate and store it away, just in case
your media become unreadable.  But have some caution:  The print system of
your machine might store the data and make it available to others!

In the meantime, (after the revocation certificate was generated but not yet uploaded) an attacker has pushed a malicious commit using the stolen GPG key.

$ echo "BAD" > gpg/example.txt
$ git commit -a -S -m "add bad GPG-signed example.txt" && git push

We definitely want to replace the key on GitHub and GitLab with the revoked one… well, if we could, anyway.

We can try removing the existing key from GitHub, but this causes our previous “good” commit to also be flagged as unverified.

Unfortunately, trying to readd the GPG key (having imported the revocation certificate & embedding it into the key) causes GitHub to return an error and leaves both commits permanently unverified.

The situation for GitLab isn’t much different. We’re provided with a “Revoke” button inside acccount settings, but pressing it warns us:

Attempting to delete the key presents us with a slightly different warning:

Theoretically GitLab’s implementation only verifying commit validity at push would allow previously signed commits to remain flagged as “Verified” if no malicious commits have been pushed, but local verification would remain difficult. As a side effect, repos pushed with signed commits before GPG keys are added to GitLab will always have their commits listed as unverified, even after keys are added.

I’m unconvinced this is a useful situation- we can either accept old commits will become impossible to locally validate as time goes on (and place trust in VCS hosts being capable of validating new commits during pushes) or we accept this doesn’t provide significant benenfit beyond enforcing branch protection rules and restricting pushes based on SSH keys.

ssh key signing

git added support for commit signing using SSH public crypto in 2.34.0, allowing software engineers to reuse their SSH push key management procedures for commit signing (without using GPG).

Similar to GPG signing, GitHub recommends setting up SSH signing through git’s global configuration.

$ git config --global gpg.format ssh
$ git config --global user.signingkey /PATH/TO/.SSH/KEY.PUB

Both GitHub and GitLab require engineers to add signing SSH keys to their accounts, separate from any push SSH keys that may have been configured. Though user.signingkey can be the same as your push SSH key, it can be an entirely different key.

With git configured for SSH, we can now make a test commit using the same -S flag as earlier.

$ mkdir ssh
$ echo "OK" > ssh/example.txt
$ git add ssh/example.txt
$ git commit -S -m "add SSH-signed example.txt" && git push

We’re also able to verify the commit as appropriately signed (with an SSH key!) through VCS web UIs.

Unfortunately, we hit bumpy roads with SSH keys as we approach key revocation.

Within their documentation for commit signature verification, GitHub notes:

Generating a GPG signing key is more involved than generating an SSH key, but GPG has features that SSH does not. A GPG key can expire or be revoked when no longer used. GitHub shows commits that were signed with such a key as “Verified” unless the key was marked as compromised. SSH keys don’t have this capability.

Attempting to delete the SSH key as a signing key from GitHub produces a warning clarifying the implications on our previously signed commits.

GitLab presents similar “revoke”/”remove” options for SSH keys as it does with GPG keys, but this still relies on engineers removing compromised signing keys before any unexpected commits are pushed.

While it’s disappointing neither VCS host allows leveraging git’s gpg.ssh.allowedSignersFile to define valid-after/valid-before constraints (see also, Allowed Signers), this would be understandably difficult to manage in practice since git commits lack trusted timestamps- an attacker could simply create commits older than the valid-before constraint if guardrails aren’t enforced by time-of-push.

s/mime

exists? I’m not sure many people actually use this for git in production.

S/MIME was historically used as an alternative to GPG for email security, particularly in corporate environments (where Public Key Infrastructure could be centrally managed, like Microsoft Exchange’s usage of Active Directory Certificate Services).

GitHub doesn’t allow custom CAs and relies on the the Debian ca-certificates package for key verification, but GitLab appears to allow custom CAs on self-installations.

Some commercial CAs still issue S/MIME certificates (a web search for “S/MIME certificate class 1 free” will do the trick, though some CAs charge ~USD$11/yr).

WebPKI CAs are required (by CA/B Forum Baseline Requirements § 4.9.1.1) to provide quick revocation for compromised keys, usually indicated through a Certificate Revocation List (CRL) and Online Certificate Status Protocol (OSCP).

Still- it’s unclear whether GitHub or GitLab routinely check CRL/OSCP for certificate validity. GitLab’s documentation explicitly mentions a rake task to re-check commits, but GitHub lack any details on revalidating signatures for commits.

the future

The aforementioned solutions fundamentally rely on software engineers being capable of safeguarding cryptographic material throughout its validity.

Building on Certificate Transparency’s success in the WebPKI space, we’ll explore some work-in-progress proposals that aspire to provide codebase security by leveraging transparency logs and short-lived cryptographic identities.

gitsign

Part of the Sigstore project (“backed by the Open Source Security Foundation/OpenSSF under the Linux Foundation, with contributions from Google, Red Hat, Chainguard, GitHub and Purdue University”), Gitsign allows software engineers to “sign git commits with an OpenID Connect identity” and embed these signing details into “the transparency log Rekor for subsequent verification”.

Gitsign’s focus on short-lived identities presents a significantly different flow for signing commits.

We’ll be following Gitsign’s Quick Start instructions to get a feel for its developer flow, while setting up a repo whose signatures we can inspect later.

First, we’ll install Gitsign using Homebrew and configure git to use it for signing.

$ brew tap sigstore/tap && brew install gitsign
$ gitsign --version
gitsign version 0.8.0
$ git config --global gpg.x509.program gitsign  # Use Gitsign for signing
$ git config --global gpg.format x509  # Gitsign expects x509 args

We can also define our “Connector ID” now and pre-select our identity provider for signing commits. As of writing, Sigstore’s public Certificate Authority (Fulcio) will accept one of these values:

• https://github.com/login/oauth
• https://accounts.google.com
• https://login.microsoftonline.com

I’ll select GitHub for our demonstration.

$ git config --global gitsign.connectorID https://github.com/login/oauth

We can now make a test commit that’s signed with Gitsign.

$ mkdir gitsign
$ echo "OK" > gitsign/example.txt
$ git add gitsign/example.txt
$ git commit -S -m "add Gitsign-signed example.txt" && git push

Next, our terminal returns something like the following, asking us to complete an OpenID Connect flow with GitHub.

Your browser will now be opened to:
https://oauth2.sigstore.dev/auth/auth?access_type=online&client_id=sigstore&code_challenge=*snip*&code_challenge_method=S256&connector_id=https://github.com/login/oauth&nonce=*snip*&redirect_uri=http://localhost:57883/auth/callback&response_type=code&scope=openid%20email&state=*snip*

The commit is written as normal once we complete the OpenID Connect flow, but an additional line precedes git’s output.

tlog entry created with index: 60059585

The same line can be observed when running git-verify-commit.

$ git verify-commit HEAD
tlog index: 60059585
gitsign: Signature made using certificate ID 0x8cb44217e92234c853d13bc93192cd11feb815a4 | CN=sigstore-intermediate,O=sigstore.dev
gitsign: Good signature from [[email protected]](https://github.com/login/oauth)
Validated Git signature: true
Validated Rekor entry: true
Validated Certificate claims: false
WARNING: git verify-commit does not verify cert claims. Prefer using `gitsign verify` instead.

We’re warned this didn’t “verify cert claims”: since we don’t tell git-verify-commit which identity to expect, Gitsign is only able to ensure the signature is plausibly valid & included in the transparency log (Rekor, thus the tlog index mentions we’ve seen so far).

gitsign verify can be used to declare an expected identity (e.g., known developers or CI infrastructure) for checking operations.

$ gitsign verify --certificate-oidc-issuer "https://github.com/login/oauth" --certificate-identity "[email protected]"
tlog index: 60059585
gitsign: Signature made using certificate ID 0x8cb44217e92234c853d13bc93192cd11feb815a4 | CN=sigstore-intermediate,O=sigstore.dev
gitsign: Good signature from [[email protected]](https://github.com/login/oauth)
Validated Git signature: true
Validated Rekor entry: true
Validated Certificate claims: true

We can also manually check Rekor for our commit’s transparency log entry by using the tlog index (see also, Get Entry).

$ brew install rekor-cli
$ rekor-cli get --log-index 60059585 --format=json | jq .

rekor-cli output (expand)

{
  "Attestation": "",
  "AttestationType": "",
  "Body": {
    "HashedRekordObj": {
      "data": {
        "hash": {
          "algorithm": "sha256",
          "value": "c876096f859c8ba3f5f569ba14e36fce5874519cdf01779439b13f42efd6241a"
        }
      },
      "signature": {
        "content": "MEYCIQDxRhS2OHpSUGro9pVVGWqtxHSSZMd9UIOX6zESHsyThQIhAOC8vya1KWqO/Fx1rLEYxb4/mvPTmO6YLvKfEkGVToOF",
        "publicKey": {
          "content": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5akNDQWxDZ0F3SUJBZ0lVS3RqSHN2dnVGMW41dFFEOWRaSTZsT0xuS1Q4d0NnWUlLb1pJemowRUF3TXcKTnpFVk1CTUdBMVVFQ2hNTWMybG5jM1J2Y21VdVpHVjJNUjR3SEFZRFZRUURFeFZ6YVdkemRHOXlaUzFwYm5SbApjbTFsWkdsaGRHVXdIaGNOTWpNeE1qSTVNVGN3TlRBMFdoY05Nak14TWpJNU1UY3hOVEEwV2pBQU1Ga3dFd1lICktvWkl6ajBDQVFZSUtvWkl6ajBEQVFjRFFnQUVuQk5zY2xrVWtQK2QrRE4wMDdRWXA2ZTdOaGhiNUUwNjNwT0kKSjFPSml3d0YxRENzaXZCdEE2bEc0bzZ0SWJEV3ZHQThxM2RTUkRBUGZlZlBacXB2K0tPQ0FXOHdnZ0ZyTUE0RwpBMVVkRHdFQi93UUVBd0lIZ0RBVEJnTlZIU1VFRERBS0JnZ3JCZ0VGQlFjREF6QWRCZ05WSFE0RUZnUVVUa29JCjFEbEFVNEFkY2JYQkF0SWl5dks5RXpRd0h3WURWUjBqQkJnd0ZvQVUzOVBwejFZa0VaYjVxTmpwS0ZXaXhpNFkKWkQ4d0dRWURWUjBSQVFIL0JBOHdEWUVMYUhSM1FHeHZZbWt1ZEc4d0xBWUtLd1lCQkFHRHZ6QUJBUVFlYUhSMApjSE02THk5bmFYUm9kV0l1WTI5dEwyeHZaMmx1TDI5aGRYUm9NQzRHQ2lzR0FRUUJnNzh3QVFnRUlBd2VhSFIwCmNITTZMeTluYVhSb2RXSXVZMjl0TDJ4dloybHVMMjloZFhSb01JR0tCZ29yQmdFRUFkWjVBZ1FDQkh3RWVnQjQKQUhZQTNUMHdhc2JIRVRKakdSNGNtV2MzQXFKS1hyamVQSzMvaDRweWdDOHA3bzRBQUFHTXRvbGJvZ0FBQkFNQQpSekJGQWlCWU5ZZTlGbnpxNU8xeXRnK2MvWUo2eHlCM0N4WWFOVlBQNVZmcnBHMHVTd0loQU9iRk4yc1p0NGlJCmRoeFlYTUczZyt4YS9qTzkwOXJRTkRwc1Frd1BtVHFLTUFvR0NDcUdTTTQ5QkFNREEyZ0FNR1VDTUhPVGpoVjUKZHY3Ty9aalRleE1BWXhJQzZ3bHRWWmdvWWpEcWE1MEloWTV4Zk9Gck5yUC82UGpEckRBRVNjTTRiQUl4QU96TwpZMGIxdGdJeWZSUmtJS2IwazJ5VUltVnUxb2llNHZoUkVBUllDL2tEQzFMQmZyNk9waVJFTVZoeUJDb2dCZz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
        }
      }
    }
  },
  "LogIndex": 60059585,
  "IntegratedTime": 1703869505,
  "UUID": "24296fb24b8ad77acd4659b1fbb64daf674e77058ed14f8091e8b53905f4fba798ffc324bac1f75b",
  "LogID": "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"
}

Inspecting .Body.HashedRekordObj.signature.publicKey.content, we can also decode the certificate used to sign this commit (see also, Verifying the Commit Signature).

$ cert=$(rekor-cli get --log-index 60059585 --format=json | jq -r .Body.HashedRekordObj.signature.publicKey.content)
$ echo $cert | base64 --decode | openssl x509 -text

openssl output (expand)

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            2a:d8:c7:b2:fb:ee:17:59:f9:b5:00:fd:75:92:3a:94:e2:e7:29:3f
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: O = sigstore.dev, CN = sigstore-intermediate
        Validity
            Not Before: Dec 29 17:05:04 2023 GMT
            Not After : Dec 29 17:15:04 2023 GMT
        Subject:
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:9c:13:6c:72:59:14:90:ff:9d:f8:33:74:d3:b4:
                    18:a7:a7:bb:36:18:5b:e4:4d:3a:de:93:88:27:53:
                    89:8b:0c:05:d4:30:ac:8a:f0:6d:03:a9:46:e2:8e:
                    ad:21:b0:d6:bc:60:3c:ab:77:52:44:30:0f:7d:e7:
                    cf:66:aa:6f:f8
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                Code Signing
            X509v3 Subject Key Identifier:
                4E:4A:08:D4:39:40:53:80:1D:71:B5:C1:02:D2:22:CA:F2:BD:13:34
            X509v3 Authority Key Identifier:
                DF:D3:E9:CF:56:24:11:96:F9:A8:D8:E9:28:55:A2:C6:2E:18:64:3F
            X509v3 Subject Alternative Name: critical
                email:[email protected]
            1.3.6.1.4.1.57264.1.1:
                https://github.com/login/oauth
            1.3.6.1.4.1.57264.1.8:
                ..https://github.com/login/oauth
            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : DD:3D:30:6A:C6:C7:11:32:63:19:1E:1C:99:67:37:02:
                                A2:4A:5E:B8:DE:3C:AD:FF:87:8A:72:80:2F:29:EE:8E
                    Timestamp : Dec 29 17:05:04.418 2023 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:20:58:35:87:BD:16:7C:EA:E4:ED:72:B6:0F:
                                9C:FD:82:7A:C7:20:77:0B:16:1A:35:53:CF:E5:57:EB:
                                A4:6D:2E:4B:02:21:00:E6:C5:37:6B:19:B7:88:88:76:
                                1C:58:5C:C1:B7:83:EC:5A:FE:33:BD:D3:DA:D0:34:3A:
                                6C:42:4C:0F:99:3A:8A
    Signature Algorithm: ecdsa-with-SHA384
    Signature Value:
        30:65:02:30:73:93:8e:15:79:76:fe:ce:fd:98:d3:7b:13:00:
        63:12:02:eb:09:6d:55:98:28:62:30:ea:6b:9d:08:85:8e:71:
        7c:e1:6b:36:b3:ff:e8:f8:c3:ac:30:04:49:c3:38:6c:02:31:
        00:ec:ce:63:46:f5:b6:02:32:7d:14:64:20:a6:f4:93:6c:94:
        22:65:6e:d6:88:9e:e2:f8:51:10:04:58:0b:f9:03:0b:52:c1:
        7e:be:8e:a6:24:44:31:58:72:04:2a:20:06
-----BEGIN CERTIFICATE-----
MIICyjCCAlCgAwIBAgIUKtjHsvvuF1n5tQD9dZI6lOLnKT8wCgYIKoZIzj0EAwMw
NzEVMBMGA1UEChMMc2lnc3RvcmUuZGV2MR4wHAYDVQQDExVzaWdzdG9yZS1pbnRl
cm1lZGlhdGUwHhcNMjMxMjI5MTcwNTA0WhcNMjMxMjI5MTcxNTA0WjAAMFkwEwYH
KoZIzj0CAQYIKoZIzj0DAQcDQgAEnBNsclkUkP+d+DN007QYp6e7Nhhb5E063pOI
J1OJiwwF1DCsivBtA6lG4o6tIbDWvGA8q3dSRDAPfefPZqpv+KOCAW8wggFrMA4G
A1UdDwEB/wQEAwIHgDATBgNVHSUEDDAKBggrBgEFBQcDAzAdBgNVHQ4EFgQUTkoI
1DlAU4AdcbXBAtIiyvK9EzQwHwYDVR0jBBgwFoAU39Ppz1YkEZb5qNjpKFWixi4Y
ZD8wGQYDVR0RAQH/BA8wDYELaHR3QGxvYmkudG8wLAYKKwYBBAGDvzABAQQeaHR0
cHM6Ly9naXRodWIuY29tL2xvZ2luL29hdXRoMC4GCisGAQQBg78wAQgEIAweaHR0
cHM6Ly9naXRodWIuY29tL2xvZ2luL29hdXRoMIGKBgorBgEEAdZ5AgQCBHwEegB4
AHYA3T0wasbHETJjGR4cmWc3AqJKXrjePK3/h4pygC8p7o4AAAGMtolbogAABAMA
RzBFAiBYNYe9Fnzq5O1ytg+c/YJ6xyB3CxYaNVPP5VfrpG0uSwIhAObFN2sZt4iI
dhxYXMG3g+xa/jO909rQNDpsQkwPmTqKMAoGCCqGSM49BAMDA2gAMGUCMHOTjhV5
dv7O/ZjTexMAYxIC6wltVZgoYjDqa50IhY5xfOFrNrP/6PjDrDAEScM4bAIxAOzO
Y0b1tgIyfRRkIKb0k2yUImVu1oie4vhREARYC/kDC1LBfr6OpiREMVhyBCogBg==
-----END CERTIFICATE-----

This certificate is issued by Sigstore’s Fulcio in accordance to their certificate specification. Notably, it is only valid for 10minutes and embeds my identity as reported by GitHub.

While tokens issued by an OpenID Connect provider can usually be publicly verified by using /.well-known/openid-configuration to retrieve signing keys, Fulcio maintains its own certificate transparency log (distinct from Rekor’s) to track issued certificates.

Fulcio’s transparency log allows certificates to be verified long after an OpenID Connect provider rotates their signing keys.

We can also use our Fulcio certificate (alongside the commit’s corresponding Rekor entry) to prove a commit as being created at a particular time. In the event of OIDC account compromise, Rekor’s transparency logs could be searched to track down unexpected commit signatures.

Similarly, Fulcio’s short certificate validity provides confidence that additional signatures can’t be issued after OIDC account credentials are rotated.

Unfortunately, this means anyone observing Rekor or Fulcio’s transparency logs can determine when commits have been made (even for private repos).

Worse still, GitHub and GitLab currently don’t implement Gitsign’s validation mechanisms or trust Fulcio’s CA, causing commits to appear as Unverified.

openpubkey

OpenPubkey is a draft specification and implementation backed by BastionZero, Docker, and the Linux Foundation[3] that aspires to reuse OpenID Connect identities as cryptographic identities through proof-of-possesion, achieved by using the OIDC ID token’s nonce to indicate a public key[4].

Since the nonce claim is a standard field that is “passed through unmodified from the Authentication Request to the ID Token”[5], OpenPubkey doesn’t require any modifications to existing OpenID Connect providers.

While the OpenPubkey draft implementation doesn’t currently provide a direct equivalent to Sigstore’s Gitsign, and its authors note the two “cannot really be compared”, they explain OpenPubkey could be used within Fulcio’s CA infrastructure to mitigate trust in Fulcio as an OIDC verifier[6].

Several contributors to Sigstore have noted concerns with OpenPubkey’s proposed model[7, 8], notably towards privacy (unexpected OIDC claims being included in the certificate, like name), expiration times being implicit, and ability to verify signatures in the long-term.

OpenPubkey relies on the /.well-known/openid-configuration lookup alluded to in the previous section for client-side verification of signatures, but it’s unclear how to handle OIDC provider key rotation without relying on a transparency log like Sigstore’s Fulcio.

I think OpenPubkey’s reuse of the OIDC nonce claim is a very cool trick, but I’d be hesitant to use it today outwith a self-managed OIDC environment. I’m curious to see how it’ll evolve in the future, particularly if the concerns around long-term validation can be addressed.

git push signing

Both of the aforementioned “future” options (Gitsign and OpenPubkey) imply internet access to complete an OpenID Connect flow and create an identity certificate. Since software engineers are likely to be online while pushing, push signing (added in git 2.2.0) presents a compromise to commit signing.

Passing --signed=true to git push will use the configured signing mechanism to sign the overall push request[9]. git-receive-pack (on the server side) can use the signature as a factor while deciding whether to accept the push and/or can record it as part of a push transparency log for later auditing.

The Linux Kernel’s canonical git repository, hosted on kernel.org, maintains a push transparency log of all git-receive operations[10].

Monitoring projects, like Foxboron’s tlog.linderud.dev, can crawl the transparency logs and detect unexpected signatures which might be indicative of VCS infrastructure compromise.

Unfortunately, signed pushes are not currently supported by GitHub or GitLab, so this isn’t likely to be usable by most engineers in the near future.

git push --signed=true output (expand)

$ git push --signed=true github
fatal: the receiving end does not support --signed push

$ git push --signed=true gitlab
fatal: the receiving end does not support --signed push
fatal: the remote end hung up unexpectedly                                                   
remote: 
remote: ========================================================================
remote: 
remote: rpc error: code = Canceled desc = user canceled the push
remote: 
remote: ========================================================================
remote:

Still- GitHub does provide repository event logs which could be regularly audited for unexpected push events, even if the push events don’t include signatures.

(GitLab requires a Premium license for push logging.)

what do you do then nerd

idk, i don’t sign my commits but i also don’t advocate for silver bullets 🐺

You probably already restrict pushing based on SSH keys and have branch protection rules enforcing PRs for merges, so surprise code should be easy to notice.

I’ll probably start signing my commits once Gitsign gains wider adoption and is suitable for private codebases.

Malware stealing SSH keys is worrying, so I try to protect my git SSH push keys with maxgoedjen/secretive by placing them in my laptop’s secure enclave (which forbids export and requires biometrics before use).

greetz

Massive thank you to everyone who helped me out with drafts of this blog post especially binaryfox, eta, Foxboron, jordixou, thejsa, lukegb, Ninji, and videah.

Of course- thank you, dear reader, for making it through my ramble :)