Documentation
¶
Overview ¶
Manifests are a list of paths and their hash digests, canonically ordered by path in increasing lexographical order. Manifests are encoded as:
<digest type>:<digest>[SP][SP]<path>[LF]
"shake256" is the only supported digest type. The digest is 64 bytes of hex encoded output of SHAKE256. See golang.org/x/crypto/sha3 and FIPS 202 for details on the SHAKE hash.
Example ¶
package main
import (
"context"
"fmt"
"github.com/bufbuild/buf/private/pkg/manifest"
"github.com/bufbuild/buf/private/pkg/storage/storagemem"
)
func main() {
ctx := context.Background()
bucket, _ := storagemem.NewReadBucket(
map[string][]byte{
"foo": []byte("bar"),
},
)
m, _, _ := manifest.NewFromBucket(ctx, bucket)
digest, _ := m.DigestFor("foo")
fmt.Printf("digest[:16]: %s\n", digest.Hex()[:16])
path, _ := m.PathsFor(digest.String())
fmt.Printf("path at digest: %s\n", path[0])
}
Output: digest[:16]: a15163728ed24e1c path at digest: foo
Index ¶
- func AsProtoBlob(ctx context.Context, b Blob) (_ *modulev1alpha1.Blob, retErr error)
- func BlobEqual(ctx context.Context, a, b Blob) (_ bool, retErr error)
- func NewBucket(m Manifest, blobs BlobSet, opts ...BucketOption) (storage.ReadBucket, error)
- func NewBucketFromManifestBlobs(ctx context.Context, manifestBlob *modulev1alpha1.Blob, ...) (storage.ReadBucket, error)
- func NewFromBucket(ctx context.Context, bucket storage.ReadBucket) (*Manifest, *BlobSet, error)
- func ToProtoManifestAndBlobs(ctx context.Context, manifest *Manifest, blobs *BlobSet) (*modulev1alpha1.Blob, []*modulev1alpha1.Blob, error)
- type Blob
- func NewBlobFromProto(b *modulev1alpha1.Blob) (Blob, error)
- func NewMemoryBlob(digest Digest, content []byte, opts ...MemoryBlobOption) (Blob, error)
- func NewMemoryBlobFromReader(content io.Reader) (Blob, error)
- func NewMemoryBlobFromReaderWithDigester(content io.Reader, digester Digester) (Blob, error)
- type BlobSet
- type BlobSetOption
- type BucketOption
- type Digest
- type DigestType
- type Digester
- type Manifest
- func (m *Manifest) AddEntry(path string, digest Digest) error
- func (m *Manifest) Blob() (Blob, error)
- func (m *Manifest) DigestFor(path string) (*Digest, bool)
- func (m *Manifest) MarshalText() ([]byte, error)
- func (m *Manifest) Paths() []string
- func (m *Manifest) PathsFor(digest string) ([]string, bool)
- func (m *Manifest) UnmarshalText(text []byte) error
- type MemoryBlobOption
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AsProtoBlob ¶
AsProtoBlob returns the passed blob as a proto module blob.
func BlobEqual ¶
BlobEqual returns true if blob a is the same as blob b. The digest is checked for equality and the content bytes compared.
An error is returned if an unexpected I/O error occurred when opening, reading, or closing either blob.
func NewBucket ¶
func NewBucket(m Manifest, blobs BlobSet, opts ...BucketOption) (storage.ReadBucket, error)
NewBucket takes a manifest and a blob set and builds a readable storage bucket that contains the files in the manifest.
func NewBucketFromManifestBlobs ¶
func NewBucketFromManifestBlobs( ctx context.Context, manifestBlob *modulev1alpha1.Blob, blobs []*modulev1alpha1.Blob, ) (storage.ReadBucket, error)
NewBucketFromManifestBlobs builds a storage bucket from a manifest blob and a set of other blobs, provided in protobuf form. It makes sure that all blobs (including manifest) content match with their digest, and additionally checks that the blob set matches completely with the manifest paths (no missing nor extra blobs). This bucket is suitable for building or exporting.
func NewFromBucket ¶
NewFromBucket creates a manifest and all its files' blobs from a storage bucket, with all its digests in DigestTypeShake256.
Types ¶
type Blob ¶
Blob is a blob with a digest and a content.
func NewBlobFromProto ¶
func NewBlobFromProto(b *modulev1alpha1.Blob) (Blob, error)
NewBlobFromProto returns a Blob from a proto module blob. It makes sure the digest and content matches.
func NewMemoryBlob ¶
func NewMemoryBlob(digest Digest, content []byte, opts ...MemoryBlobOption) (Blob, error)
NewMemoryBlob takes a digest and a content, and turns it into an in-memory representation of a blob, which returns the digest and an io.ReadCloser for its content.
func NewMemoryBlobFromReader ¶
NewMemoryBlobFromReader creates a memory blob from content, which is read until completion. The returned blob contains all bytes read. If you are using this in a loop, you might better use NewMemoryBlobFromReaderWithDigester so you can reuse your digester.
func NewMemoryBlobFromReaderWithDigester ¶
NewMemoryBlobFromReaderWithDigester creates a memory blob from content with the passed digester. The content is read until completion. The returned blob contains all bytes read.
type BlobSet ¶
type BlobSet struct {
// contains filtered or unexported fields
}
BlobSet represents a set of deduplicated blobs, by digests.
func NewBlobSet ¶
NewBlobSet receives a slice of blobs, and de-duplicates them into a BlobSet.
type BlobSetOption ¶
type BlobSetOption func(*blobSetOptions)
BlobSetOption are options passed when creating a new blob set.
func BlobSetWithContentValidation ¶
func BlobSetWithContentValidation() BlobSetOption
BlobSetWithContentValidation turns on content validation for all the blobs when creating a new BlobSet. If this option is on, blobs with the same digest must have the same content (in case blobs with the same digest are sent). If this option is not passed, then the latest duplicated blob digest content will prevail in the set.
type BucketOption ¶
type BucketOption func(*bucketOptions)
BucketOption are options passed when creating a new manifest bucket.
func BucketWithAllManifestBlobsValidation ¶
func BucketWithAllManifestBlobsValidation() BucketOption
BucketWithAllManifestBlobsValidation validates that all manifest digests have a corresponding blob in the blob set. If this option is not passed, then buckets with partial/incomplete blobs are allowed.
func BucketWithNoExtraBlobsValidation ¶
func BucketWithNoExtraBlobsValidation() BucketOption
BucketWithNoExtraBlobsValidation validates that the passed blob set has no additional blobs beyond the ones in the manifest.
type Digest ¶
type Digest struct {
// contains filtered or unexported fields
}
Digest represents a hash function's value.
func NewDigestFromBytes ¶
func NewDigestFromBytes(dtype DigestType, digest []byte) (*Digest, error)
NewDigestFromBytes builds a digest from a type and the digest bytes.
func NewDigestFromHex ¶
func NewDigestFromHex(dtype DigestType, hexstr string) (*Digest, error)
NewDigestFromHex builds a digest from a type and the hexadecimal string of the bytes. It returns an error if the received string is not a valid hex.
func NewDigestFromProtoDigest ¶
func NewDigestFromProtoDigest(digest *modulev1alpha1.Digest) (*Digest, error)
NewDigestFromProtoDigest maps a modulev1alpha1.Digest to a Digest.
func NewDigestFromString ¶
NewDigestFromString build a digest from a string representation of it.
type DigestType ¶
type DigestType string
DigestType is the type for digests in this package.
const (
DigestTypeShake256 DigestType = "shake256"
)
type Digester ¶
Digester is something that can digest a content into a digest.
func NewDigester ¶
func NewDigester(dtype DigestType) (Digester, error)
type Manifest ¶
type Manifest struct {
// contains filtered or unexported fields
}
Manifest represents a list of paths and their digests.
func NewFromReader ¶
NewFromReader builds a manifest from an encoded manifest reader.
func (*Manifest) AddEntry ¶
AddEntry adds an entry to the manifest with a path and its digest. It fails if the path already exists in the manifest with a different digest.
func (*Manifest) DigestFor ¶
DigestFor returns the matching digest for the given path. The path must be an exact match. Digest is nil and ok is false if no digest is found.
func (*Manifest) MarshalText ¶
MarshalText encodes the manifest into its canonical form.
func (*Manifest) PathsFor ¶
PathsFor returns one or more matching path for a given digest. The digest is expected to be a lower-case hex encoded value. Returned paths are unordered. Paths is nil and ok is false if no paths are found.
func (*Manifest) UnmarshalText ¶
UnmarshalText decodes a manifest from member.
Use NewManifestFromReader if you have an io.Reader and want to avoid memory copying.
type MemoryBlobOption ¶
type MemoryBlobOption func(*memoryBlobOptions)
MemoryBlobOption are options passed when creating a new memory blob.
func MemoryBlobWithDigestValidation ¶
func MemoryBlobWithDigestValidation() MemoryBlobOption
MemoryBlobWithDigestValidation checks that the passed content and digest match.
Source Files