with
Qubes OS this is extremely easy because of the hardware isolation
model used in Qubes. You can have a VM with no network access, into
which you copy files that you need to sign, then copy them back out
once they have been signed (or you can use qubes-split-gpg, which
allows you to essentially use an airgapped vault VM as a virtual
Yubikey of sorts).
Thanks for the detailed answer! This sounds indeed very interesting (and I agree with your points about the security of open source fw code). My point about building trust in a way that scales is the same even with this approach. You have to factor in that it is much more difficult to collect the necessary information, from a more complex system, required to prove that a particular key was actually created and used with a setup like this **and** cryptographically verify them compared to a hw token/HSM approach.
To give an example, once I have a signature produced by a key generated in a yubikey, an automated service in launchpad could run one command to attest that the key used was indeed generated in a yubikey [1]. We then implicitly put our trust to the yubikey's manufacturer and for many this is enough. With HSMs it is similar, you are essentially paying the HSM manufacturer for trusting them.
We should have clear guidelines about what are the requirements for an alternative method to be considered as trusted as the ones I mentioned above (what are the necessary attestations, how feasible it is to automate verification as a whole/in parts etc). For the Qubes example, I want to see cryptographic proof that the signing happened inside the "air-gapped" qube, proof that the key was indeed generated inside of it and cannot leave it, proof that you have set up your system properly and that your system comes from a manufacturer I consider trusted so that I can trust its hardware isolation mechanisms etc. (turtles all the way down) and I also want a system to do that verification for me as much as possible.
[1]: https://developers.yubico.com/PKI/yubico-ca-certs.txt
To give an example, once I have a signature produced by a key generated in a yubikey, an automated service in launchpad could run one command to attest that the key used was indeed generated in a yubikey [1]. We then implicitly put our trust to the yubikey's manufacturer and for many this is enough. With HSMs it is similar, you are essentially paying the HSM manufacturer for trusting them.
We should have clear guidelines about what are the requirements for an alternative method to be considered as trusted as the ones I mentioned above (what are the necessary attestations, how feasible it is to automate verification as a whole/in parts etc). For the Qubes example, I want to see cryptographic proof that the signing happened inside the "air-gapped" qube, proof that the key was indeed generated inside of it and cannot leave it, proof that you have set up your system properly and that your system comes from a manufacturer I consider trusted so that I can trust its hardware isolation mechanisms etc. (turtles all the way down) and I also want a system to do that verification for me as much as possible.
[1]: https://developers.yubico.com/PKI/yubico-ca-certs.txt