resource

package
v1.115.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 26, 2026 License: Apache-2.0 Imports: 25 Imported by: 0

Documentation

Overview

Package resource provides the data layer for human-uploaded reference material (samples, playbooks, templates, references). Resources are scoped to global, persona, or user visibility and stored as blobs in S3 with metadata in PostgreSQL.

Index

Constants

View Source
const (
	// DefaultListLimit is used when no limit is specified in a list query.
	DefaultListLimit = 100
	// MaxListLimit caps a client-supplied page size so a single list request
	// cannot pull an unbounded window.
	MaxListLimit = 200
)
View Source
const (
	// SurfaceMCPRead is an MCP resources/read call.
	SurfaceMCPRead = "mcp_read"
	// SurfaceFetch is a search `fetch` of an mcp:resource:<id> reference.
	SurfaceFetch = "fetch"
	// SurfaceDownload is a REST content download (the portal's Download
	// button, and any direct API client).
	SurfaceDownload = "rest_download"
)

Read surfaces a resource's content can be served through. Recorded on every read event so a curator can tell material an agent actually pulls from material a human downloads once and forgets.

View Source
const (
	MaxUploadBytes     = 100 << 20 // 100 MB
	MaxDescriptionLen  = 2000
	MaxDisplayNameLen  = 200
	MaxTagsPerResource = 20
	MaxTagLen          = 50
	MaxCategoryLen     = 31
)

Validation limits.

View Source
const DefaultMaxVersions = 10

DefaultMaxVersions is the number of content revisions a resource keeps, counting the current head. A revision past the cap prunes the oldest version rows and their blobs. Bounded by default because a resource blob is up to MaxUploadBytes (100 MB) and an unbounded trail turns every edit into permanent storage growth.

View Source
const (
	// DefaultSearchLimit is the top-K returned when the caller specifies none.
	DefaultSearchLimit = 20
)

Search result limits, mirroring the asset/prompt/memory ranked surfaces so every ranked surface clamps the same way.

View Source
const DefaultURIScheme = "mcp"

DefaultURIScheme is used when no scheme is configured.

View Source
const MaxContentIndexBytes = 32 << 10

MaxContentIndexBytes bounds how much of a resource's content is extracted into content_text for search. The prefix has to be large enough to cover the part of a document that describes it (a data dictionary's column list, a runbook's procedure) while keeping the embedded text within one provider call and the row small enough that listing resources stays cheap.

View Source
const MaxContentReadBytes = 8 << 20

MaxContentReadBytes bounds the object size the index consumer will pull from blob storage to extract that prefix. The blob API has no range read, so extracting 32 KiB means holding the whole object in memory; an upload may be up to MaxUploadBytes (100 MB) and several index workers run concurrently. A resource larger than this is indexed on its metadata alone rather than risking the sweep's memory on a file whose first 32 KiB is all that would have been kept anyway.

View Source
const MaxInlineContentBytes = 1 << 20

MaxInlineContentBytes is the size at or below which a text resource's content is returned inline rather than as a pointer to the blob. It is shared by the MCP resources/read path and the search `fetch` reference so an agent gets the same answer for the same file whichever door it comes through.

View Source
const MaxMultipartMemory = 10 << 20

MaxMultipartMemory is the max memory for multipart form parsing (10 MB).

View Source
const MinMaxVersions = 2

MinMaxVersions is the floor a configured retention cap is raised to. One version is the head itself, so a cap below 2 keeps no history at all and would make the version panel a list of one row that changes under the reader.

Variables

View Source
var DeniedExtensions = map[string]bool{
	".exe": true, ".sh": true, ".bat": true, ".cmd": true,
	".ps1": true, ".msi": true, ".com": true, ".scr": true,
}

DeniedExtensions lists file extensions that are blocked for upload.

View Source
var DeniedMIMETypes = map[string]bool{
	"application/x-executable":    true,
	"application/x-msdos-program": true,
	"application/x-msdownload":    true,
	"application/x-sh":            true,
	"application/x-shellscript":   true,
	"application/x-bat":           true,
	"application/x-msi":           true,
}

DeniedMIMETypes lists MIME types that are blocked for upload.

View Source
var ErrForbidden = errors.New("forbidden")

ErrForbidden signals that authentication succeeded at the credential level but the request is refused for a policy reason the client can recover from without re-authenticating — specifically a CSRF-token failure on a cookie-authenticated mutation. A ClaimsExtractor returns it so the handler responds 403 (not 401), matching the admin/portal surfaces and preventing the SPA from force-logging-out the user on a recoverable CSRF error.

Functions

func BuildRevisionS3Key added in v1.115.0

func BuildRevisionS3Key(scope Scope, scopeID, resourceID, revisionID, filename string) string

BuildRevisionS3Key constructs the S3 object key for a revision's blob. Each revision gets its own key under a v/<revisionID>/ segment so prior versions remain independently readable and pruning one never touches another's bytes.

The key is keyed by an opaque revision id rather than the version number because the number is assigned by the database inside the insert transaction, after the blob has already been written; deriving the key from the number would require knowing it first, which is exactly the race the in-transaction assignment exists to avoid. Version 1 of a resource uploaded before versioning keeps the flat key BuildS3Key produced at create time.

func BuildS3Key

func BuildS3Key(scope Scope, scopeID, resourceID, filename string) string

BuildS3Key constructs the S3 object key for a resource blob.

func BuildURI

func BuildURI(scheme string, scope Scope, scopeID, category, filename string) string

BuildURI constructs the canonical resource URI from its components.

func CanAccessResource added in v1.115.0

func CanAccessResource(c Claims, r *Resource) bool

CanAccessResource checks whether the caller may see a specific resource at all: it is inside their visible scopes, OR they hold write authority over the scope it lives in (a platform admin, or that persona's admin).

The second clause is what separates this from CanReadResource. VisibleScopes is membership-based and grants an admin no cross-persona read, so a platform admin who uploads a persona-scoped resource — which CanWriteScope explicitly permits — was then refused GET, PATCH and DELETE on it: they could create material they could neither manage nor remove. Use this as the visibility gate on a resource the caller names by id; CanReadResource remains the membership rule for enumeration and for content served into an agent's session.

It deliberately checks CanWriteScope rather than CanModifyResource: the latter also grants the original uploader, and that grant is not re-derived from current authority. An admin who uploaded into another user's scope and then lost their admin role would otherwise keep reading, editing, and deleting that user's private file forever, because the uploader_sub on the row never changes. Every legitimate uploader whose authority came from their own scope (a user uploading to their own user scope) is already covered by CanReadResource.

func CanModifyResource

func CanModifyResource(c Claims, r *Resource) bool

CanModifyResource checks whether the caller can update or delete a resource. The caller must be the original uploader OR have write permission for the scope.

func CanReadResource

func CanReadResource(c Claims, r *Resource) bool

CanReadResource checks whether the caller can read a specific resource.

func CanWriteScope

func CanWriteScope(c Claims, scope Scope, scopeID string) bool

CanWriteScope checks whether the caller has write permission for the given scope.

func GenerateID

func GenerateID() (string, error)

GenerateID returns a cryptographically random 32-character hex string.

func IndexText added in v1.115.0

func IndexText(r Resource, contentText string) string

IndexText composes the text a resource is embedded and lexically indexed on: its display name, description, category, filename, tags, and contentText, the bounded text prefix the index consumer extracted from the uploaded file. The indexjobs resource consumer and the request-path search MUST agree on this composition so a stored embedding lives in the same space as the query; it is defined once here for both. Empty fields are skipped so a sparse resource does not pad the text with blank lines. The lexical arm's resource_fts (migration 000091) composes the same corpus from the same columns.

func IsNotFound added in v1.111.0

func IsNotFound(err error) bool

IsNotFound reports whether an error from a Store read means the resource does not exist, as opposed to the read having failed.

The Postgres store surfaces a missing row as a wrapped sql.ErrNoRows rather than as (nil, nil), so a caller that must distinguish "deleted" from "the database is down" cannot do it by nil-checking the result. Getting that distinction wrong is not cosmetic: a prompt attachment whose resource was deleted has to degrade to a flagged broken link, while a failed read has to fail closed.

func IsObjectNotFound added in v1.115.0

func IsObjectNotFound(err error) bool

IsObjectNotFound reports whether a blob-store GetObject error indicates the object does not exist (an orphaned resource: the metadata row survived its content), as opposed to a transient or permission failure that a retry might resolve.

The mcp-s3 client wraps the underlying AWS/SeaweedFS error without a typed not-found, so detection is by the standard S3 not-found signatures present in the wrapped message. It lives here because both readers of resource blobs depend on the distinction and must draw it identically: the resources/read middleware self-heals a confirmed orphan by pruning the row, and the search index consumer clears a confirmed-orphan's indexed content instead of leaving stale text behind. A per-caller copy of this heuristic would let the two diverge on exactly the case that matters.

func NormalizeMaxVersions added in v1.115.0

func NormalizeMaxVersions(configured int) int

NormalizeMaxVersions clamps a configured retention cap: a non-positive value selects the default, and anything below MinMaxVersions is raised to it.

func PersonaAdminRoles added in v1.111.0

func PersonaAdminRoles(roles []string) []string

PersonaAdminRoles extracts the persona names a role set grants admin authority over, tolerating any role prefix.

func SanitizeFilename

func SanitizeFilename(name string) (string, error)

SanitizeFilename normalizes a filename for storage: lowercase, no spaces, no path separators or shell metacharacters, preserves extension.

func ValidateCategory

func ValidateCategory(cat string) error

ValidateCategory checks that a category matches the required pattern.

func ValidateDescription

func ValidateDescription(desc string) error

ValidateDescription checks description length and content.

func ValidateDisplayName

func ValidateDisplayName(name string) error

ValidateDisplayName checks display name length and content.

func ValidateMIMEType

func ValidateMIMEType(mt string) error

ValidateMIMEType checks that the MIME type is not on the deny list. The base type is extracted (e.g. "text/html" from "text/html; charset=utf-8") before checking the deny list.

func ValidateScope

func ValidateScope(scope Scope, scopeID string) error

ValidateScope checks scope and scope_id consistency.

func ValidateTags

func ValidateTags(tags []string) error

ValidateTags checks tag count, length, and format.

Types

type Claims

type Claims struct {
	Sub             string   // Keycloak subject (user ID)
	Email           string   // user email
	Personas        []string // persona names the user belongs to
	Roles           []string // raw roles from auth (may have prefix, e.g., "dp_admin")
	IsAdmin         bool     // resolved by the caller from persona config
	AdminOfPersonas []string // persona names this user can admin (resolved by caller from role patterns)
}

Claims represents the identity information needed for resource permission checks.

func BuildClaims added in v1.111.0

func BuildClaims(sub, email, persona string, roles []string, isAdmin bool) Claims

BuildClaims assembles permission claims from an authenticated caller's identity. It owns the roles-to-persona-admin mapping so every surface that must apply the resource read rule — the resources middleware, prompt attachment serving (#1013), the attachment REST handler — derives claims identically instead of each reimplementing it.

persona is the caller's single resolved persona; pass "" when none is resolved. It is separate from roles because persona membership is resolved from roles by the platform's persona registry, which this package does not see.

type ClaimsExtractor

type ClaimsExtractor func(r *http.Request) (*Claims, error)

ClaimsExtractor extracts resource Claims from an HTTP request. Provided by the platform auth middleware.

type Deps

type Deps struct {
	Store     Store
	S3Client  S3Client
	S3Bucket  string
	URIScheme string          // defaults to "mcp" if empty
	OnCreate  func(*Resource) // called after successful create to register with MCP
	OnDelete  func(string)    // called after successful delete with URI to unregister

	// Versions records content revisions. Absent on a deployment whose store
	// does not implement VersionStore, which disables the revision and version
	// routes (503) and leaves create, metadata edits, and reads unaffected.
	Versions VersionStore
	// MaxVersions is the retention cap; non-positive selects DefaultMaxVersions.
	MaxVersions int
	// ReadRecorder audits served content reads. Absent when audit is disabled,
	// which silences read events without affecting the reads themselves.
	ReadRecorder ReadRecorder
	// Usage supplies audit-derived read counts for the detail read. Absent when
	// audit is disabled, which leaves the usage field off the response.
	Usage UsageReader
}

Deps holds the dependencies for the resource HTTP handler.

type Filter

type Filter struct {
	Scopes   []ScopeFilter // visibility scopes (derived from claims)
	Category string        // optional category filter
	Tag      string        // optional tag filter
	Query    string        // optional text search in display_name/description
	Sort     Sort          // ordering; empty selects SortUpdated
	Limit    int
	Offset   int
}

Filter specifies criteria for listing resources.

type Handler

type Handler struct {
	// contains filtered or unexported fields
}

Handler provides HTTP endpoints for resource CRUD.

func NewHandler

func NewHandler(deps Deps, extractFn ClaimsExtractor, authMiddle func(http.Handler) http.Handler) *Handler

NewHandler creates a resource handler with auth middleware.

func (*Handler) ServeHTTP

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP implements http.Handler.

type ParsedURI

type ParsedURI struct {
	Scope   Scope
	ScopeID string
	Path    string
}

ParsedURI holds the components extracted from a resource URI.

func ParseURI

func ParseURI(scheme, uri string) (ParsedURI, error)

ParseURI extracts scope, scopeID, and path from a resource URI. Returns an error if the URI does not match the expected format.

type ReadEvent added in v1.115.0

type ReadEvent struct {
	ResourceID string
	URI        string
	// Surface is one of the Surface* constants: which door served the content.
	Surface string
	// Version is the revision served, or 0 when the head was served without
	// naming a version.
	Version   int
	UserID    string
	UserEmail string
	Persona   string
	SessionID string
	RequestID string
}

ReadEvent describes one served read of a resource's content.

type ReadRecorder added in v1.115.0

type ReadRecorder interface {
	RecordRead(ctx context.Context, ev ReadEvent)
}

ReadRecorder records a served read. Every surface that hands a resource's bytes to a caller calls it, so "who read this, through which door" has one answer rather than one per surface.

Implementations are best-effort and must not fail the read: a recorder that cannot write must swallow the failure, because refusing to serve a file because its audit row would not persist trades a working read for a lost log line. Recording is off entirely when audit is disabled, which is why every call site tolerates a nil recorder.

type ReadTracker added in v1.115.0

type ReadTracker interface {
	TouchRead(ctx context.Context, id string, at time.Time) error
}

ReadTracker stamps the durable last-read time on a resource row. It is separate from the audit rollup because it answers a different question: the rollup is bounded by audit retention and cannot be sorted on in SQL, while this column survives retention and is what the admin table orders by.

type Resource

type Resource struct {
	ID            string    `json:"id" example:"res_01HK7R9F"`
	Scope         Scope     `json:"scope" example:"persona"`
	ScopeID       string    `json:"scope_id,omitempty" example:"data-engineer"` // persona name or user sub; empty for global
	Category      string    `json:"category" example:"runbooks"`
	Filename      string    `json:"filename" example:"etl-runbook.md"`
	DisplayName   string    `json:"display_name" example:"ETL Runbook"`
	Description   string    `json:"description" example:"Step-by-step procedures for ETL pipeline operations"`
	MIMEType      string    `json:"mime_type" example:"text/markdown"`
	SizeBytes     int64     `json:"size_bytes" example:"34000"`
	S3Key         string    `json:"s3_key" example:"resources/res_01HK7R9F/etl-runbook.md"`
	URI           string    `json:"uri" example:"mcp://persona/data-engineer/runbooks/etl-runbook.md"`
	Tags          []string  `json:"tags"`
	UploaderSub   string    `json:"uploader_sub" example:"550e8400-e29b-41d4-a716-446655440000"`
	UploaderEmail string    `json:"uploader_email" example:"marcus.johnson@example.com"`
	CreatedAt     time.Time `json:"created_at"`
	UpdatedAt     time.Time `json:"updated_at"`
	// LastReadAt is when the resource's content was last served through any
	// surface, stamped by the read recorder. NULL means never read since the
	// deployment began auditing reads (#1014). It is the durable answer, unlike
	// the audit-derived Usage.LastReadAt, which is bounded by audit retention.
	LastReadAt *time.Time `json:"last_read_at,omitempty"`
	// Usage is the audit-derived read activity of this resource. It is not a
	// stored column: the detail read fills it from the audit rollup, and it is
	// absent everywhere the rollup was not consulted.
	Usage *Usage `json:"usage,omitempty"`
}

Resource represents a human-uploaded reference material entry.

type Revision added in v1.115.0

type Revision struct {
	ResourceID    string
	MIMEType      string
	SizeBytes     int64
	S3Key         string
	UploaderSub   string
	UploaderEmail string
	// RestoredFrom names the version a restore re-promoted, or nil for a fresh
	// upload.
	RestoredFrom *int
}

Revision is a new content revision to record: the blob that was just written and who wrote it. The version number is not an input — the store assigns it — and the resource's id, uri, filename, scope and metadata are untouched, which is the whole point of revising in place instead of delete-plus-re-upload.

type S3Client

type S3Client interface {
	PutObject(ctx context.Context, bucket, key string, data []byte, contentType string) error
	GetObject(ctx context.Context, bucket, key string) (body []byte, contentType string, err error)
	DeleteObject(ctx context.Context, bucket, key string) error
}

S3Client abstracts blob storage operations for resources.

type Scope

type Scope string

Scope defines the visibility level of a resource.

const (
	// ScopeGlobal is visible to every authenticated user.
	ScopeGlobal Scope = "global"
	// ScopePersona is visible to users operating under the named persona.
	ScopePersona Scope = "persona"
	// ScopeUser is visible only to the owning user.
	ScopeUser Scope = "user"
)

type ScopeFilter

type ScopeFilter struct {
	Scope   Scope
	ScopeID string // empty for global
}

ScopeFilter identifies a single scope+id pair for visibility filtering.

func VisibleScopes

func VisibleScopes(c Claims) []ScopeFilter

VisibleScopes returns the set of (scope, scope_id) tuples the caller is allowed to see. Always derived from claims, never from request input.

type ScoredResource added in v1.115.0

type ScoredResource struct {
	Resource Resource `json:"resource"`
	Score    float64  `json:"score"`
}

ScoredResource pairs a resource with its relevance score in [0,1].

type SearchQuery added in v1.115.0

type SearchQuery struct {
	Embedding []float32     // query vector; nil selects lexical-only ranking
	QueryText string        // raw query text for the lexical arm
	Scopes    []ScopeFilter // caller's visible scopes; mandatory
	Limit     int           // max results; clamped into [1, maxSearchLimit]
}

SearchQuery describes a relevance ranking request over managed resources. Visibility is applied in SQL before ranking: Scopes is the caller's visible (scope, scope_id) set as VisibleScopes computes it, so a resource the caller could not list is never ranked. An empty Scopes returns nothing rather than searching unscoped. A nil Embedding selects lexical-only ranking (the graceful-degradation path when no embedding provider is configured); a non-nil Embedding selects hybrid ranking.

func (SearchQuery) EffectiveLimit added in v1.115.0

func (q SearchQuery) EffectiveLimit() int

EffectiveLimit clamps the requested limit into the search bounds.

type Searcher added in v1.115.0

type Searcher interface {
	Search(ctx context.Context, q SearchQuery) ([]ScoredResource, error)
}

Searcher ranks the resources visible to a caller by relevance to a query. It is a capability separate from Store: only a backing store that can rank (the PostgreSQL store with pgvector) implements it, so the feature degrades to absent rather than forcing every Store implementation to carry a ranking query.

type Sort added in v1.115.0

type Sort string

Sort names an ordering for the list path.

const (
	// SortUpdated orders by most recently updated, the default.
	SortUpdated Sort = "updated"
	// SortLastRead orders by most recently read, never-read resources last.
	// It is what a curator hunting dead weight sorts by.
	SortLastRead Sort = "last_read"
)

type Store

type Store interface {
	Insert(ctx context.Context, r Resource) error
	Get(ctx context.Context, id string) (*Resource, error)
	GetByURI(ctx context.Context, uri string) (*Resource, error)
	List(ctx context.Context, filter Filter) ([]Resource, int, error)
	Update(ctx context.Context, id string, u Update) error
	Delete(ctx context.Context, id string) error
}

Store persists and queries resource metadata.

func NewPostgresStore

func NewPostgresStore(db *sql.DB) Store

NewPostgresStore creates a resource store backed by PostgreSQL.

type Update

type Update struct {
	DisplayName *string  `json:"display_name,omitempty"`
	Description *string  `json:"description,omitempty"`
	Tags        []string `json:"tags,omitempty"`
	Category    *string  `json:"category,omitempty"`
}

Update holds mutable fields for a PATCH operation.

type Usage added in v1.115.0

type Usage struct {
	// Reads30d and Reads90d count audited reads in the trailing 30 and 90
	// days. Both are bounded by the audit retention window: a deployment
	// keeping 30 days of audit reports the same number twice.
	Reads30d int64 `json:"reads_30d" example:"42"`
	Reads90d int64 `json:"reads_90d" example:"117"`
	// BySurface30d breaks the 30-day count down by Surface* value.
	BySurface30d map[string]int64 `json:"by_surface_30d,omitempty"`
	// LastReadAt is the most recent audited read within the retention window.
	// The durable answer lives on the resource row (Resource.LastReadAt), which
	// outlives retention; this field is what the rollup itself saw.
	LastReadAt *time.Time `json:"last_read_at,omitempty"`
}

Usage is the audit-derived read activity of a single resource.

type UsageReader added in v1.115.0

type UsageReader interface {
	ResourceUsage(ctx context.Context, resourceIDs []string) (map[string]Usage, error)
}

UsageReader aggregates read events into per-resource usage. The Postgres audit store implements it, the same way it implements prompt usage (#1009): the audit log is already the durable record of who read what, so usage is a rollup of it rather than a second set of counters to keep in sync.

type Version added in v1.115.0

type Version struct {
	ResourceID    string `json:"resource_id" example:"a1b2c3d4e5f67890a1b2c3d4e5f67890"`
	Version       int    `json:"version" example:"3"`
	MIMEType      string `json:"mime_type" example:"text/markdown"`
	SizeBytes     int64  `json:"size_bytes" example:"34000"`
	S3Key         string `json:"s3_key" example:"resources/persona/data-engineer/a1b2/v3/etl-runbook.md"`
	UploaderSub   string `json:"uploader_sub" example:"550e8400-e29b-41d4-a716-446655440000"`
	UploaderEmail string `json:"uploader_email" example:"marcus.johnson@example.com"`
	// RestoredFrom names the version this revision re-promoted, or nil when the
	// revision was a fresh upload. A restore is recorded as a new head revision
	// rather than a rewind so the trail stays append-only.
	RestoredFrom *int      `json:"restored_from,omitempty" example:"1"`
	CreatedAt    time.Time `json:"created_at"`
}

Version is one recorded revision of a resource's content.

It carries only what varies per revision. Filename is absent on purpose: the canonical URI embeds the filename and revision keeps the URI stable, so every version of a resource shares the resource's filename.

type VersionStore added in v1.115.0

type VersionStore interface {
	// AddRevision records a revision and points the resource head at its blob,
	// in one transaction. The version number is assigned inside that
	// transaction as one past the highest recorded, so two concurrent revisions
	// cannot claim the same number, and the stored row is returned.
	AddRevision(ctx context.Context, rev Revision) (*Version, error)

	// ListVersions returns every recorded revision of a resource, newest first.
	ListVersions(ctx context.Context, resourceID string) ([]Version, error)

	// GetVersion returns one recorded revision. A missing revision surfaces as
	// a wrapped sql.ErrNoRows (see IsNotFound).
	GetVersion(ctx context.Context, resourceID string, version int) (*Version, error)

	// PruneVersions deletes the oldest revisions beyond the newest keep and
	// returns the deleted rows so the caller can remove their blobs. It never
	// deletes a row whose S3 key is currently referenced by the resource head,
	// so pruning can never orphan the live content.
	PruneVersions(ctx context.Context, resourceID string, keep int) ([]Version, error)
}

VersionStore persists the content-revision trail of a resource and moves the head to a new revision. The Postgres resource store implements it; it is a separate interface from Store so a deployment or test that only needs metadata CRUD is not forced to model the version trail, and so callers can type-assert for the capability.

The trail and the head move together — a version row whose bytes no resource points at, or a head pointing at bytes no version row records, is a broken state — so AddRevision owns both writes rather than leaving the caller to sequence them.

Directories

Path Synopsis
Package resourceindex is the managed-resource consumer of the shared indexjobs framework (#1012).
Package resourceindex is the managed-resource consumer of the shared indexjobs framework (#1012).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL