# DigestSet field type specification Set of one or more cryptographic digests, or other immutable references, for a single software artifact or metadata object. ## Schema ```json { "": "", "": "", ... } ``` ## Fields A DigestSet is represented as a _JSON object_ mapping algorithm name to a string encoding of the digest using that algorithm. The named standard algorithms below use lowercase hex encoding. Usually there is just a single key/value pair, but multiple entries MAY be used for algorithm agility. Each entry in a DigestSet MUST be an immutable reference to an artifact. It is STRONGLY RECOMMENDED to use a commonly accepted, cryptographically secure digest algorithm to achieve this immutability. See [Use cases for non-cryptographic, immutable, digests](#use-cases-for-non-cryptographic-immutable-digests) for further guidance. ### Supported algorithms #### `sha256`, `sha224`, `sha384`, `sha512`, `sha512_224`, `sha512_256`, `sha3_224`, `sha3_256`, `sha3_384`, `sha3_512`, `shake128`, `shake256`, `blake2b`, `blake2s`, `ripemd160`, `sm3`, `gost`, `sha1`, `md5` Standard cryptographic hash algorithms using [the NIST names][] (converting to lowercase and replacing `-` with `_`) as keys and lowercase hex-encoded values, for cases when the method of serialization is obvious or well known. #### `dirHash` The [directory Hash1][] function, omitting the "h1:" prefix and output in lowercase hexadecimal instead of base64. This algorithm was designed for go modules but can be used to digest the _contents_ of an arbitrary archive or file tree. Equivalent to extracting the archive to an empty directory and running the following command in that directory: ```bash find . -type f | cut -c3- | LC_ALL=C sort | xargs -r sha256sum | sha256sum | cut -f1 -d' ' ``` For example, the module dirhash `h1:Khu2En+0gcYPZ2kuIihfswbzxv/mIHXgzPZ018Oty48=` would be encoded as `{"dirHash1": "2a1bb6127fb481c60f67692e22285fb306f3c6ffe62075e0ccf674d7c3adcb8f"}`.
Detailed example The go module `github.com/marklodato/go-hello-world@v0.0.1` has module dirhash `h1:Khu2En+0gcYPZ2kuIihfswbzxv/mIHXgzPZ018Oty48=`: ```bash $ curl https://sum.golang.org/lookup/github.com/marklodato/go-hello-world@v0.0.1 ... github.com/marklodato/go-hello-world v0.0.1 h1:Khu2En+0gcYPZ2kuIihfswbzxv/mIHXgzPZ018Oty48= ... ``` To compute the dirhash by hand, first fetch the module archive and extract it to an empty directory: ```bash curl -O https://proxy.golang.org/github.com/marklodato/go-hello-world/@v/v0.0.1.zip mkdir tmp cd tmp unzip ../v0.0.1.zip ``` We can see all of the files in the directory using the first part of the command above: ```bash $ find . -type f | cut -c3- | LC_ALL=C sort | xargs -r sha256sum 3a137eef6458bfb76bb2c63fc29ffc7166604d2d2e09ed9d8250a534122a8364 github.com/marklodato/go-hello-world@v0.0.1/README.md 28e7c942a036902d981759d0bf5704d2bfc7cb500caf68b84711b234af01c6a5 github.com/marklodato/go-hello-world@v0.0.1/go.mod ddc4da627d9a9f45fb29641a1b185d6f53287ecfd921aacbf4fe54b7a86fe8d1 github.com/marklodato/go-hello-world@v0.0.1/main.go ``` The dirhash is the sha256 sum over the output of the previous command: ```bash $ find . -type f | cut -c3- | LC_ALL=C sort | xargs -r sha256sum | sha256sum | cut -f1 -d' ' 2a1bb6127fb481c60f67692e22285fb306f3c6ffe62075e0ccf674d7c3adcb8f ``` This is equivalent to the base64 encoded version: ```bash $ echo '2a1bb6127fb481c60f67692e22285fb306f3c6ffe62075e0ccf674d7c3adcb8f' | xxd -r -p | {printf 'h1:'; base64} h1:Khu2En+0gcYPZ2kuIihfswbzxv/mIHXgzPZ018Oty48= ```
#### `gitCommit`, `gitTree`, `gitBlob`, `gitTag` The lowercase hex SHA-1 (40 character) or SHA-256 (64 character) of a git commit, tree, blob, or tag object, respectively. The `gitTree` and `gitBlob` in particular can be used for arbitrary trees or files, even outside git. This hash is computed over ` SP NUL `, where: - `` is one of `commit`, `tree`, `blob`, `tag` - `SP` is the ASCII space character, 0x20 - `` is the number of bytes in ``, represented as a decimal ASCII number with no leading zeros - `NUL` is the ASCII NUL character, 0x00 - `` is git representation of the object: - For `commit`, the raw commit object ([more info][so-commit][^git-docs]) - For `tree`, the raw tree object, which is a series of ` NUL ` entries, sorted by `` in the C locale ([more info][so-tree][^git-docs]) - For `blob`, the raw file contents - For `tag`, the raw tag object For more information, see [Git Objects] in the Git Book.[^git-docs] Example of `gitBlob` for the file containing the 5 bytes `Hello` (no newline): ```bash $ printf 'blob 5\0Hello' | sha1sum | cut -f1 -d' ' 5ab2f8a4323abafb10abb68657d9d39f1a775057 $ printf 'Hello' | git hash-object -t blob --stdin 5ab2f8a4323abafb10abb68657d9d39f1a775057 ``` ### Guidelines It is RECOMMENDED to use at least `sha256` for compatibility between producers and consumers, unless a different hash algorithm is more conventional (e.g. `gitCommit` for git). Consumers MUST only accept algorithms that they consider secure and MUST ignore unrecognized or unaccepted algorithms. For example, most applications SHOULD NOT accept "md5" because it lacks collision resistance. Two DigestSets SHOULD be considered matching if ANY acceptable field matches. **New algorithms MUST document how the value is encoded, e.g. URL-safe base64, lowercase hex, etc.** ### Use cases for non-cryptographic, immutable, digests While cryptographic digests are the strongly recommended immutable identifier, users might have need to refer to an artifact by some other means. For example, it might be technically infeasible to compute a digest over the content, or the user might interact with the content through an interface that doesn't expose them to the entirety of the content. In these situations, users MAY use a non-cryptographic identifier in a DigestSet so long as the risk of the object being mutated is acceptable for the application. One concrete example of where a non-cryptographic hash can be useful is when referring to Virtual Machine images. Often these images are very large (impractical to run a cryptographic hash over) and users often interact with them via APIs that the platform provides that don't involve the user having complete custody of the content. Platforms like AWS and GCP provide 'ids' for users to use when referring to these images. A user may say something like "create an instance with image 123". In that case the user doesn't actually have the bits that correspond to 'image 123' so they cannot digest it themselves. And by the time the image has started it can be difficult, if not impossible, to digest the original content that was used to boot the instance. These IDs can often be treated as immutable and may be perfectly suited to users threat profiles. Allowing DigestSets to use these types of identifiers allows providers to make statements about the content of these VM images using the identifiers their users have ready access to. In addition, using an ID like this does not preclude including a cryptographic hash in the DigestSet as well. If possible including both may provide the most flexibility for the user's various use cases. Per the specification [guidelines](#guidelines), the value encoding for non-cryptographic digests is identifier-specific. See the `subject` [field details](statement.md#fields) for Statement `subject`-specific requirements. ## Examples - `{"sha256": "abcd", "sha512": "1234"}` matches `{"sha256": "abcd"}` - `{"sha256": "abcd"}` does not match `{"sha256": "fedb", "sha512": "abcd"}` - `{"somecoolhash": "abcd"}` uses a non-predefined algorithm - `{"arn": "arn:aws:ec2:region:123:image/ami-123abcd"}` for a non-cryptographic VM image digest using an Amazon Resource Name (ARN) in the `subject` field of a Statement --- [^git-docs]: At the time of writing (2023-03), git has no official documentation of the internal object format used for hashing. The [Git Objects] chapter of the Git Book is the closest thing to official documentation, but it lacks many details, such as the raw tree object format. The best documentation we have found are the linked Stack Overflow articles. If you can find a better, more official reference, please open an issue. [Git Objects]: https://git-scm.com/book/en/v2/Git-Internals-Git-Objects [directory Hash1]: https://cs.opensource.google/go/x/mod/+/refs/tags/v0.5.0:sumdb/dirhash/hash.go [so-commit]: https://stackoverflow.com/a/37438460 [so-tree]: https://stackoverflow.com/a/35902553 [the NIST names]: https://csrc.nist.gov/projects/hash-functions