overlay

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 16, 2026 License: LGPL-2.1 Imports: 26 Imported by: 0

README

mod/overlay

mod/overlay turns a canonical version tree into package-manager artifacts and metadata. It detects Go and Composer packages, rewrites Go module paths for Go proxy artifacts when enabled, plans raw universal archives, and builds artifact bytes from storage.

Place in the runtime

flowchart TB
  tree["canonical tree"] --> overlay["mod/overlay"]
  overlay --> detect["ecosystem detection"]
  overlay --> plan["artifact plan"]
  plan --> gozip["Go module zip"]
  plan --> composer["Composer dist"]
  plan --> universal["zip and tar.gz"]
  gozip --> storage["mod/storage"]
  composer --> storage
  universal --> storage

Responsibilities

  • Detect whether a tree is Go-publishable, Composer-publishable, both, or neither.
  • Rewrite Go module paths and imports for the configured public module path in Go proxy module zips.
  • Reject trees that cannot become valid Go module zips.
  • Build Go proxy .mod, .info, and .zip artifacts.
  • Build Composer metadata and dist archives.
  • Build raw universal .zip and .tar.gz artifacts with top directory <key>-<version>/.
  • Render install snippets and integrity hints used by the HTML UI.

Contracts

  • Detection must be deterministic from tree contents and listener/domain context.
  • Go zip validity follows golang.org/x/mod/zip constraints, including fold collisions and invalid names.
  • Symlink trees are not Go-publishable.
  • Go module path rewrite is size-stable by contract: the size reported to modzip.Create must equal the length of the content later opened for the same file. rewriteContent and rewrittenSize share one scanner to keep this invariant.
  • Builders must not trust staged files after validation; storage reopens and verifies staged content before commit.
  • Artifact identifiers include listener id when output depends on host or scheme; raw universal archives use the global listener id.

Important files

  • detect.go, goviability.go: ecosystem detection and Go zip viability.
  • rewrite.go: Go module path and import rewriting.
  • plan.go, materialize.go: artifact planning and materialization.
  • gorender.go, gozip.go, composer.go, universal.go: domain builders.
  • snippet.go: UI install snippets.
  • interface.go: storage and reader seams.

Operational notes

Overlay code is intentionally strict. A tree that is valid as a universal archive can still be blocked for Go if it would produce a module zip that Go tooling rejects. The version remains published; only Go-specific artifacts are disabled.

Go zip block reasons are grouped into stable categories before they are persisted. Known x/mod/zip invalid-path, collision, and size-limit errors keep their existing labels. Unknown future invalid errors use unclassified module zip errors; mod/rescan counts that label with rescan_go_zip_unclassified_total so dependency wording drift is visible after upgrades instead of being silently merged into invalid paths.

Documentation

Index

Constants

View Source 
const (

	// GoZipFormatVersion is the Go module zip format version.
	GoZipFormatVersion = 2
	// UniversalZipFormatVersion is the universal zip format version.
	UniversalZipFormatVersion = 2
	// UniversalTarGzFormatVersion is the universal tar.gz format version.
	UniversalTarGzFormatVersion = 2
)
View Source 
const (
	// GoZipUnclassifiedLabel is the stable label for x/mod/zip errors this code cannot classify yet.
	GoZipUnclassifiedLabel = "unclassified module zip errors"
)

Variables

This section is empty.

Functions

func BazelSnippet 

func BazelSnippet(name string, url string, sha256Arr []byte, stripPrefix string) string

BazelSnippet returns http_archive over universal.tar.gz with sha256 integrity and top-dir strip_prefix.

func CandidateFromDetection 

func CandidateFromDetection(detectionObj core.DetectionObj) (goCandidate *CandidateObj, composerCandidate *CandidateObj)

CandidateFromDetection restores candidates from saved DetectionObj evidence JSON. RewriteBlobs are not included; serve-time builders receive them separately from storage.RewriteSet.

func CollisionError 

func CollisionError(collisionObj ComposerCollisionObj) error

CollisionError returns a stable stcode error for composer name collisions.

func ComposerRequireSnippet 

func ComposerRequireSnippet(name string, version string, scheme string, host string, basePath string) string

ComposerRequireSnippet returns a mirror install recipe. The mirror has priority for its packages, transitive dependencies stay on Packagist, and http/ygg entries disable secure-http. basePath is the nested route base ("/pkg") or empty in root mode: Composer resolves packages.json under the repository URL, so the URL must carry the prefix.

func DegradedReason 

func DegradedReason(err error) (string, bool)

DegradedReason returns a stable degradation code for non-nil overlay build/detect errors. overlay owns this dictionary; rescan/server only store the returned code. Unknown build errors map to "materialization_error".

func GoInfo 

func GoInfo(version string, ingestTS time.Time) api.GoInfoObj

GoInfo returns the generated `@latest`/`.info` object. mod/server returns it directly for `@latest` and streams it for `.info`; time is local IngestTS.

func GoInstallSnippet 

func GoInstallSnippet(modulePath string, version string, proxyScheme string, host string) string

GoInstallSnippet returns a Go-module consumption recipe: GOPROXY for our entry host, private mirror sumdb settings, optional GOINSECURE for http/ygg, and `go get`.

func RenderGoInfo 

func RenderGoInfo(version string, ingestTS time.Time) ([]byte, error)

RenderGoInfo returns `@latest`/`<version>.info` as canonical JSON bytes through generated jx encoder.

func RenderGoVersionList 

func RenderGoVersionList(versionArr []string) []byte

RenderGoVersionList returns `@v/list`: canonical versions, one per line, sorted descending.

func ZigSnippet 

func ZigSnippet(url string) string

ZigSnippet returns build.zig.zon.url; the client computes the hash through `zig fetch`.

Types

type ArtifactBuilderInterface 

type ArtifactBuilderInterface interface {
	Build(ctx context.Context, writerObj io.Writer) error
}

ArtifactBuilderInterface matches storage.ArtifactBuilderInterface structurally so overlay can pass builders to storage without importing the storage root.

type ArtifactPlanObj 

type ArtifactPlanObj struct {
	MaterializerID stcode.MaterializerType
	ArtifactKind   archive.FormatType
	FormatVersion  uint32
	ListenerID     stcode.ListenerType
	Builder        ArtifactBuilderInterface
}

ArtifactPlanObj is one byte-stable artifact to materialize: identity tuple, listener, and ready builder. overlay owns the build policy; rescan hashes Builder and registers ArtifactObj.

type BlobReaderInterface 

type BlobReaderInterface interface {
	ReadBlob(ctx context.Context, hashObj core.HashObj) ([]byte, error)
}

BlobReaderInterface is a narrow blob read boundary by hash; manifests are small.

type CandidateObj 

type CandidateObj struct {
	Ecosystem    stcode.EcosystemType
	GoModulePath string         // upstream module path from go.mod
	ComposerName string         // vendor/package from composer.json
	RewriteBlobs []core.HashObj // blobs containing upstream module path, host-independent

	// GoZipBlockReason is non-empty when the tree cannot become a valid Go module zip.
	GoZipBlockReason string
}

CandidateObj is one ecosystem detection result for a version.

type ComposerCollisionObj 

type ComposerCollisionObj struct {
	Name           string
	Key            string
	ConflictingKey string
}

ComposerCollisionObj is a composer-name conflict between keys.

func ResolveComposerNames 

func ResolveComposerNames(nameByKey map[string]string) (names []string, nameToKey map[string]string, collisions []ComposerCollisionObj)

ResolveComposerNames enforces uniqueness across keys: sorted unique names, name-to-winner-key map for serve-time p2, and collisions. The lexicographically smaller key wins; rescan owns cross-key aggregation, cache, and ETag.

type ComposerVersionInputObj 

type ComposerVersionInputObj struct {
	Version      string
	IngestTS     time.Time // composer `time` source is local ingest time
	DistURL      string
	DistShasum   string // lowercase hex sha1 universal zip (rescan: hex.EncodeToString(digest.BodySha1))
	ComposerJSON []byte
}

ComposerVersionInputObj is one version build input assembled by rescan/server: version, time, universal dist, and raw composer.json. overlay parses it and builds generated api.* objects.

type DetectionResultObj 

type DetectionResultObj struct {
	Detection    core.DetectionObj
	RewriteBlobs []core.HashObj
	Go           *CandidateObj
	Composer     *CandidateObj
}

DetectionResultObj is the version detection aggregate stored by rescan in PublishObj. Simultaneous Go and Composer matches set Conflict and disable rewrites.

type DetectorInterface 

type DetectorInterface interface {
	Ecosystem() stcode.EcosystemType
	Detect(ctx context.Context, treeArr []core.TreeEntryObj, src BlobReaderInterface) (*CandidateObj, error)
}

DetectorInterface detects one ecosystem from a version tree and returns nil candidate when absent.

type ListenerCtxObj 

type ListenerCtxObj struct {
	ListenerID  stcode.ListenerType
	EntryHost   string
	RoutePrefix string
}

ListenerCtxObj is listener host context for host-sensitive Go output.

func ListenerContexts 

func ListenerContexts(domain string, routePrefix string, yggHost string) []ListenerCtxObj

ListenerContexts returns host contexts for active listeners. It is the shared source for rescan materialization and rebuild/self-test so host-sensitive artifacts are rebuilt under the same EntryHost used for serving.

type MaterializerDescriptorObj 

type MaterializerDescriptorObj struct {
	MaterializerID stcode.MaterializerType
	FormatVersion  uint32
	Kinds          []archive.FormatType
}

MaterializerDescriptorObj is materializer_id, format_version, and artifact kinds for self-test.

type Obj 

type Obj struct {
	// contains filtered or unexported fields
}

Obj is the overlay registry: ordered detectors, config-derived parameters, and archive engine. It is immutable after New so router tables can stay strict and fixed.

func New 

func New(configObj *stconf.ConfigObj) (*Obj, error)

New builds the registry from config and holds no network or disk resources.

func (*Obj) ArtifactPlan 

func (obj *Obj) ArtifactPlan(
	st StorageInterface,
	key string,
	version string,
	treeHashObj core.HashObj,
	detectionObj core.DetectionObj,
	candidateObj *CandidateObj,
	rewriteArr []core.HashObj,
	listenerArr []ListenerCtxObj,
) []ArtifactPlanObj

ArtifactPlan returns byte-stable artifacts for a version under active listeners:

  • universal zip+tar.gz are the raw uploaded tree, host-independent, and always a single global row;
  • go-zip is per listener and only for GoPublishable versions.

Composer dist uses universal zip; served-live JSON metadata is not part of the plan.

func (*Obj) ComposerP2 

func (obj *Obj) ComposerP2(packageName string, inputArr []ComposerVersionInputObj) (*api.ComposerP2Obj, error)

ComposerP2 builds a generated p2 package document sorted descending. mod/server returns it directly. dist is universal zip; shasum is sha1.

func (*Obj) ComposerPackageList 

func (obj *Obj) ComposerPackageList(packageNames []string) *api.ComposerPackageListObj

ComposerPackageList builds generated packages/list.json.

func (*Obj) ComposerPackages 

func (obj *Obj) ComposerPackages(packageNames []string) *api.ComposerPackagesObj

ComposerPackages builds generated packages.json with metadata-url, available-packages, and list.

func (*Obj) Descriptors 

func (obj *Obj) Descriptors() []MaterializerDescriptorObj

Descriptors returns current format versions for byte-stable materializers. The driver rebuilds the smallest artifact for each pair and compares body_hash. Served-live Composer/Go JSON metadata is excluded.

func (*Obj) Detect 

func (obj *Obj) Detect(ctx context.Context, treeArr []core.TreeEntryObj, src BlobReaderInterface) (DetectionResultObj, error)

Detect runs detectors over a version tree and aggregates the result. Go+Composer sets Conflict and disables rewrite. The return value is what rescan stores in PublishObj.

func (*Obj) GoModuleZipBuilder 

func (obj *Obj) GoModuleZipBuilder(
	st StorageInterface,
	key string,
	version string,
	treeHashObj core.HashObj,
	detectionObj core.DetectionObj,
	candidateObj *CandidateObj,
	rewriteArr []core.HashObj,
	listenerCtxObj ListenerCtxObj,
) ArtifactBuilderInterface

GoModuleZipBuilder builds @v/<version>.zip for a key. Module path is always our target, and content is rewritten by the shared rewritePlan. Call only for GoPublishable versions.

func (*Obj) GoPublishable 

func (obj *Obj) GoPublishable(key string, version string, detectionObj core.DetectionObj, candidateObj *CandidateObj, listenerCtxObj ListenerCtxObj) bool

GoPublishable reports whether a version has Go @v routes: unambiguous Go detection, a tree able to form a valid go module zip, canonical `vX.Y.Z`, valid module path with matching major suffix, and rewrite either unnecessary or allowed. This is the shared Go serve gate.

func (*Obj) RenderGoMod 

func (obj *Obj) RenderGoMod(
	ctx context.Context,
	st StorageInterface,
	key string,
	version string,
	treeHashObj core.HashObj,
	detectionObj core.DetectionObj,
	candidateObj *CandidateObj,
	listenerCtxObj ListenerCtxObj,
) ([]byte, error)

RenderGoMod returns `<version>.mod` as rewritten go.mod, or raw go.mod when rewrite is unnecessary or disabled. Rewrite decisions go through rewritePlan.

func (*Obj) TargetModulePath 

func (obj *Obj) TargetModulePath(key string, version string, listenerCtxObj ListenerCtxObj) string

TargetModulePath returns the served Go module path for snippets and pages.

func (*Obj) UniversalBuilder 

func (obj *Obj) UniversalBuilder(st StorageInterface, key string, version string, treeHashObj core.HashObj, format archive.FormatType) ArtifactBuilderInterface

UniversalBuilder builds a raw universal archive with top-dir <key>-<version>/.

func (*Obj) UniversalTopDir 

func (obj *Obj) UniversalTopDir(key string, version string, _ core.DetectionObj, _ *CandidateObj, _ ListenerCtxObj) string

UniversalTopDir returns the raw universal archive top-dir (<key>-<version>/) for Bazel strip_prefix.

type StorageInterface 

type StorageInterface interface {
	ReadTree(ctx context.Context, treeHashObj core.HashObj) ([]core.TreeEntryObj, error)
	ReadBlob(ctx context.Context, hashObj core.HashObj) ([]byte, error)
}

StorageInterface is the narrow storage read boundary required by overlay builders.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.