Documentation
¶
Overview ¶
Package resource provides the data layer for human-uploaded reference material (samples, playbooks, templates, references). Resources are scoped to global, persona, or user visibility and stored as blobs in S3 with metadata in PostgreSQL.
Index ¶
- Constants
- Variables
- func BuildRevisionS3Key(scope Scope, scopeID, resourceID, revisionID, filename string) string
- func BuildS3Key(scope Scope, scopeID, resourceID, filename string) string
- func BuildURI(scheme string, scope Scope, scopeID, category, filename string) string
- func CanAccessResource(c Claims, r *Resource) bool
- func CanModifyResource(c Claims, r *Resource) bool
- func CanReadResource(c Claims, r *Resource) bool
- func CanWriteScope(c Claims, scope Scope, scopeID string) bool
- func GenerateID() (string, error)
- func IndexText(r Resource, contentText string) string
- func IsNotFound(err error) bool
- func IsObjectNotFound(err error) bool
- func NormalizeMaxVersions(configured int) int
- func PersonaAdminRoles(roles []string) []string
- func SanitizeFilename(name string) (string, error)
- func ValidateCategory(cat string) error
- func ValidateDescription(desc string) error
- func ValidateDisplayName(name string) error
- func ValidateMIMEType(mt string) error
- func ValidateScope(scope Scope, scopeID string) error
- func ValidateTags(tags []string) error
- type Claims
- type ClaimsExtractor
- type Deps
- type Filter
- type Handler
- type ParsedURI
- type ReadEvent
- type ReadRecorder
- type ReadTracker
- type Resource
- type Revision
- type S3Client
- type Scope
- type ScopeFilter
- type ScoredResource
- type SearchQuery
- type Searcher
- type Sort
- type Store
- type Update
- type Usage
- type UsageReader
- type Version
- type VersionStore
Constants ¶
const ( // DefaultListLimit is used when no limit is specified in a list query. DefaultListLimit = 100 // MaxListLimit caps a client-supplied page size so a single list request // cannot pull an unbounded window. MaxListLimit = 200 )
const ( // SurfaceMCPRead is an MCP resources/read call. SurfaceMCPRead = "mcp_read" // SurfaceFetch is a search `fetch` of an mcp:resource:<id> reference. SurfaceFetch = "fetch" // SurfaceDownload is a REST content download (the portal's Download // button, and any direct API client). SurfaceDownload = "rest_download" )
Read surfaces a resource's content can be served through. Recorded on every read event so a curator can tell material an agent actually pulls from material a human downloads once and forgets.
const ( MaxUploadBytes = 100 << 20 // 100 MB MaxDescriptionLen = 2000 MaxDisplayNameLen = 200 MaxTagsPerResource = 20 MaxTagLen = 50 MaxCategoryLen = 31 )
Validation limits.
const DefaultMaxVersions = 10
DefaultMaxVersions is the number of content revisions a resource keeps, counting the current head. A revision past the cap prunes the oldest version rows and their blobs. Bounded by default because a resource blob is up to MaxUploadBytes (100 MB) and an unbounded trail turns every edit into permanent storage growth.
const (
// DefaultSearchLimit is the top-K returned when the caller specifies none.
DefaultSearchLimit = 20
)
Search result limits, mirroring the asset/prompt/memory ranked surfaces so every ranked surface clamps the same way.
const DefaultURIScheme = "mcp"
DefaultURIScheme is used when no scheme is configured.
const MaxContentIndexBytes = 32 << 10
MaxContentIndexBytes bounds how much of a resource's content is extracted into content_text for search. The prefix has to be large enough to cover the part of a document that describes it (a data dictionary's column list, a runbook's procedure) while keeping the embedded text within one provider call and the row small enough that listing resources stays cheap.
const MaxContentReadBytes = 8 << 20
MaxContentReadBytes bounds the object size the index consumer will pull from blob storage to extract that prefix. The blob API has no range read, so extracting 32 KiB means holding the whole object in memory; an upload may be up to MaxUploadBytes (100 MB) and several index workers run concurrently. A resource larger than this is indexed on its metadata alone rather than risking the sweep's memory on a file whose first 32 KiB is all that would have been kept anyway.
const MaxInlineContentBytes = 1 << 20
MaxInlineContentBytes is the size at or below which a text resource's content is returned inline rather than as a pointer to the blob. It is shared by the MCP resources/read path and the search `fetch` reference so an agent gets the same answer for the same file whichever door it comes through.
const MaxMultipartMemory = 10 << 20
MaxMultipartMemory is the max memory for multipart form parsing (10 MB).
const MinMaxVersions = 2
MinMaxVersions is the floor a configured retention cap is raised to. One version is the head itself, so a cap below 2 keeps no history at all and would make the version panel a list of one row that changes under the reader.
Variables ¶
var DeniedExtensions = map[string]bool{ ".exe": true, ".sh": true, ".bat": true, ".cmd": true, ".ps1": true, ".msi": true, ".com": true, ".scr": true, }
DeniedExtensions lists file extensions that are blocked for upload.
var DeniedMIMETypes = map[string]bool{ "application/x-executable": true, "application/x-msdos-program": true, "application/x-msdownload": true, "application/x-sh": true, "application/x-shellscript": true, "application/x-bat": true, "application/x-msi": true, }
DeniedMIMETypes lists MIME types that are blocked for upload.
var ErrForbidden = errors.New("forbidden")
ErrForbidden signals that authentication succeeded at the credential level but the request is refused for a policy reason the client can recover from without re-authenticating — specifically a CSRF-token failure on a cookie-authenticated mutation. A ClaimsExtractor returns it so the handler responds 403 (not 401), matching the admin/portal surfaces and preventing the SPA from force-logging-out the user on a recoverable CSRF error.
Functions ¶
func BuildRevisionS3Key ¶ added in v1.115.0
BuildRevisionS3Key constructs the S3 object key for a revision's blob. Each revision gets its own key under a v/<revisionID>/ segment so prior versions remain independently readable and pruning one never touches another's bytes.
The key is keyed by an opaque revision id rather than the version number because the number is assigned by the database inside the insert transaction, after the blob has already been written; deriving the key from the number would require knowing it first, which is exactly the race the in-transaction assignment exists to avoid. Version 1 of a resource uploaded before versioning keeps the flat key BuildS3Key produced at create time.
func BuildS3Key ¶
BuildS3Key constructs the S3 object key for a resource blob.
func CanAccessResource ¶ added in v1.115.0
CanAccessResource checks whether the caller may see a specific resource at all: it is inside their visible scopes, OR they hold write authority over the scope it lives in (a platform admin, or that persona's admin).
The second clause is what separates this from CanReadResource. VisibleScopes is membership-based and grants an admin no cross-persona read, so a platform admin who uploads a persona-scoped resource — which CanWriteScope explicitly permits — was then refused GET, PATCH and DELETE on it: they could create material they could neither manage nor remove. Use this as the visibility gate on a resource the caller names by id; CanReadResource remains the membership rule for enumeration and for content served into an agent's session.
It deliberately checks CanWriteScope rather than CanModifyResource: the latter also grants the original uploader, and that grant is not re-derived from current authority. An admin who uploaded into another user's scope and then lost their admin role would otherwise keep reading, editing, and deleting that user's private file forever, because the uploader_sub on the row never changes. Every legitimate uploader whose authority came from their own scope (a user uploading to their own user scope) is already covered by CanReadResource.
func CanModifyResource ¶
CanModifyResource checks whether the caller can update or delete a resource. The caller must be the original uploader OR have write permission for the scope.
func CanReadResource ¶
CanReadResource checks whether the caller can read a specific resource.
func CanWriteScope ¶
CanWriteScope checks whether the caller has write permission for the given scope.
func GenerateID ¶
GenerateID returns a cryptographically random 32-character hex string.
func IndexText ¶ added in v1.115.0
IndexText composes the text a resource is embedded and lexically indexed on: its display name, description, category, filename, tags, and contentText, the bounded text prefix the index consumer extracted from the uploaded file. The indexjobs resource consumer and the request-path search MUST agree on this composition so a stored embedding lives in the same space as the query; it is defined once here for both. Empty fields are skipped so a sparse resource does not pad the text with blank lines. The lexical arm's resource_fts (migration 000091) composes the same corpus from the same columns.
func IsNotFound ¶ added in v1.111.0
IsNotFound reports whether an error from a Store read means the resource does not exist, as opposed to the read having failed.
The Postgres store surfaces a missing row as a wrapped sql.ErrNoRows rather than as (nil, nil), so a caller that must distinguish "deleted" from "the database is down" cannot do it by nil-checking the result. Getting that distinction wrong is not cosmetic: a prompt attachment whose resource was deleted has to degrade to a flagged broken link, while a failed read has to fail closed.
func IsObjectNotFound ¶ added in v1.115.0
IsObjectNotFound reports whether a blob-store GetObject error indicates the object does not exist (an orphaned resource: the metadata row survived its content), as opposed to a transient or permission failure that a retry might resolve.
The mcp-s3 client wraps the underlying AWS/SeaweedFS error without a typed not-found, so detection is by the standard S3 not-found signatures present in the wrapped message. It lives here because both readers of resource blobs depend on the distinction and must draw it identically: the resources/read middleware self-heals a confirmed orphan by pruning the row, and the search index consumer clears a confirmed-orphan's indexed content instead of leaving stale text behind. A per-caller copy of this heuristic would let the two diverge on exactly the case that matters.
func NormalizeMaxVersions ¶ added in v1.115.0
NormalizeMaxVersions clamps a configured retention cap: a non-positive value selects the default, and anything below MinMaxVersions is raised to it.
func PersonaAdminRoles ¶ added in v1.111.0
PersonaAdminRoles extracts the persona names a role set grants admin authority over, tolerating any role prefix.
func SanitizeFilename ¶
SanitizeFilename normalizes a filename for storage: lowercase, no spaces, no path separators or shell metacharacters, preserves extension.
func ValidateCategory ¶
ValidateCategory checks that a category matches the required pattern.
func ValidateDescription ¶
ValidateDescription checks description length and content.
func ValidateDisplayName ¶
ValidateDisplayName checks display name length and content.
func ValidateMIMEType ¶
ValidateMIMEType checks that the MIME type is not on the deny list. The base type is extracted (e.g. "text/html" from "text/html; charset=utf-8") before checking the deny list.
func ValidateScope ¶
ValidateScope checks scope and scope_id consistency.
func ValidateTags ¶
ValidateTags checks tag count, length, and format.
Types ¶
type Claims ¶
type Claims struct {
Sub string // Keycloak subject (user ID)
Email string // user email
Personas []string // persona names the user belongs to
Roles []string // raw roles from auth (may have prefix, e.g., "dp_admin")
IsAdmin bool // resolved by the caller from persona config
AdminOfPersonas []string // persona names this user can admin (resolved by caller from role patterns)
}
Claims represents the identity information needed for resource permission checks.
func BuildClaims ¶ added in v1.111.0
BuildClaims assembles permission claims from an authenticated caller's identity. It owns the roles-to-persona-admin mapping so every surface that must apply the resource read rule — the resources middleware, prompt attachment serving (#1013), the attachment REST handler — derives claims identically instead of each reimplementing it.
persona is the caller's single resolved persona; pass "" when none is resolved. It is separate from roles because persona membership is resolved from roles by the platform's persona registry, which this package does not see.
type ClaimsExtractor ¶
ClaimsExtractor extracts resource Claims from an HTTP request. Provided by the platform auth middleware.
type Deps ¶
type Deps struct {
Store Store
S3Client S3Client
S3Bucket string
URIScheme string // defaults to "mcp" if empty
OnCreate func(*Resource) // called after successful create to register with MCP
OnDelete func(string) // called after successful delete with URI to unregister
// Versions records content revisions. Absent on a deployment whose store
// does not implement VersionStore, which disables the revision and version
// routes (503) and leaves create, metadata edits, and reads unaffected.
Versions VersionStore
// MaxVersions is the retention cap; non-positive selects DefaultMaxVersions.
MaxVersions int
// ReadRecorder audits served content reads. Absent when audit is disabled,
// which silences read events without affecting the reads themselves.
ReadRecorder ReadRecorder
// Usage supplies audit-derived read counts for the detail read. Absent when
// audit is disabled, which leaves the usage field off the response.
Usage UsageReader
}
Deps holds the dependencies for the resource HTTP handler.
type Filter ¶
type Filter struct {
Scopes []ScopeFilter // visibility scopes (derived from claims)
Category string // optional category filter
Tag string // optional tag filter
Query string // optional text search in display_name/description
Sort Sort // ordering; empty selects SortUpdated
Limit int
Offset int
}
Filter specifies criteria for listing resources.
type Handler ¶
type Handler struct {
// contains filtered or unexported fields
}
Handler provides HTTP endpoints for resource CRUD.
func NewHandler ¶
func NewHandler(deps Deps, extractFn ClaimsExtractor, authMiddle func(http.Handler) http.Handler) *Handler
NewHandler creates a resource handler with auth middleware.
type ReadEvent ¶ added in v1.115.0
type ReadEvent struct {
ResourceID string
URI string
// Surface is one of the Surface* constants: which door served the content.
Surface string
// Version is the revision served, or 0 when the head was served without
// naming a version.
Version int
UserID string
UserEmail string
Persona string
SessionID string
RequestID string
}
ReadEvent describes one served read of a resource's content.
type ReadRecorder ¶ added in v1.115.0
ReadRecorder records a served read. Every surface that hands a resource's bytes to a caller calls it, so "who read this, through which door" has one answer rather than one per surface.
Implementations are best-effort and must not fail the read: a recorder that cannot write must swallow the failure, because refusing to serve a file because its audit row would not persist trades a working read for a lost log line. Recording is off entirely when audit is disabled, which is why every call site tolerates a nil recorder.
type ReadTracker ¶ added in v1.115.0
ReadTracker stamps the durable last-read time on a resource row. It is separate from the audit rollup because it answers a different question: the rollup is bounded by audit retention and cannot be sorted on in SQL, while this column survives retention and is what the admin table orders by.
type Resource ¶
type Resource struct {
ID string `json:"id" example:"res_01HK7R9F"`
Scope Scope `json:"scope" example:"persona"`
ScopeID string `json:"scope_id,omitempty" example:"data-engineer"` // persona name or user sub; empty for global
Category string `json:"category" example:"runbooks"`
Filename string `json:"filename" example:"etl-runbook.md"`
DisplayName string `json:"display_name" example:"ETL Runbook"`
Description string `json:"description" example:"Step-by-step procedures for ETL pipeline operations"`
MIMEType string `json:"mime_type" example:"text/markdown"`
SizeBytes int64 `json:"size_bytes" example:"34000"`
S3Key string `json:"s3_key" example:"resources/res_01HK7R9F/etl-runbook.md"`
URI string `json:"uri" example:"mcp://persona/data-engineer/runbooks/etl-runbook.md"`
Tags []string `json:"tags"`
UploaderSub string `json:"uploader_sub" example:"550e8400-e29b-41d4-a716-446655440000"`
UploaderEmail string `json:"uploader_email" example:"marcus.johnson@example.com"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
// LastReadAt is when the resource's content was last served through any
// surface, stamped by the read recorder. NULL means never read since the
// deployment began auditing reads (#1014). It is the durable answer, unlike
// the audit-derived Usage.LastReadAt, which is bounded by audit retention.
LastReadAt *time.Time `json:"last_read_at,omitempty"`
// Usage is the audit-derived read activity of this resource. It is not a
// stored column: the detail read fills it from the audit rollup, and it is
// absent everywhere the rollup was not consulted.
Usage *Usage `json:"usage,omitempty"`
}
Resource represents a human-uploaded reference material entry.
type Revision ¶ added in v1.115.0
type Revision struct {
ResourceID string
MIMEType string
SizeBytes int64
S3Key string
UploaderSub string
UploaderEmail string
// RestoredFrom names the version a restore re-promoted, or nil for a fresh
// upload.
RestoredFrom *int
}
Revision is a new content revision to record: the blob that was just written and who wrote it. The version number is not an input — the store assigns it — and the resource's id, uri, filename, scope and metadata are untouched, which is the whole point of revising in place instead of delete-plus-re-upload.
type S3Client ¶
type S3Client interface {
PutObject(ctx context.Context, bucket, key string, data []byte, contentType string) error
GetObject(ctx context.Context, bucket, key string) (body []byte, contentType string, err error)
DeleteObject(ctx context.Context, bucket, key string) error
}
S3Client abstracts blob storage operations for resources.
type ScopeFilter ¶
ScopeFilter identifies a single scope+id pair for visibility filtering.
func VisibleScopes ¶
func VisibleScopes(c Claims) []ScopeFilter
VisibleScopes returns the set of (scope, scope_id) tuples the caller is allowed to see. Always derived from claims, never from request input.
type ScoredResource ¶ added in v1.115.0
ScoredResource pairs a resource with its relevance score in [0,1].
type SearchQuery ¶ added in v1.115.0
type SearchQuery struct {
Embedding []float32 // query vector; nil selects lexical-only ranking
QueryText string // raw query text for the lexical arm
Scopes []ScopeFilter // caller's visible scopes; mandatory
Limit int // max results; clamped into [1, maxSearchLimit]
}
SearchQuery describes a relevance ranking request over managed resources. Visibility is applied in SQL before ranking: Scopes is the caller's visible (scope, scope_id) set as VisibleScopes computes it, so a resource the caller could not list is never ranked. An empty Scopes returns nothing rather than searching unscoped. A nil Embedding selects lexical-only ranking (the graceful-degradation path when no embedding provider is configured); a non-nil Embedding selects hybrid ranking.
func (SearchQuery) EffectiveLimit ¶ added in v1.115.0
func (q SearchQuery) EffectiveLimit() int
EffectiveLimit clamps the requested limit into the search bounds.
type Searcher ¶ added in v1.115.0
type Searcher interface {
Search(ctx context.Context, q SearchQuery) ([]ScoredResource, error)
}
Searcher ranks the resources visible to a caller by relevance to a query. It is a capability separate from Store: only a backing store that can rank (the PostgreSQL store with pgvector) implements it, so the feature degrades to absent rather than forcing every Store implementation to carry a ranking query.
type Store ¶
type Store interface {
Insert(ctx context.Context, r Resource) error
Get(ctx context.Context, id string) (*Resource, error)
GetByURI(ctx context.Context, uri string) (*Resource, error)
List(ctx context.Context, filter Filter) ([]Resource, int, error)
Update(ctx context.Context, id string, u Update) error
Delete(ctx context.Context, id string) error
}
Store persists and queries resource metadata.
func NewPostgresStore ¶
NewPostgresStore creates a resource store backed by PostgreSQL.
type Update ¶
type Update struct {
DisplayName *string `json:"display_name,omitempty"`
Description *string `json:"description,omitempty"`
Tags []string `json:"tags,omitempty"`
Category *string `json:"category,omitempty"`
}
Update holds mutable fields for a PATCH operation.
type Usage ¶ added in v1.115.0
type Usage struct {
// Reads30d and Reads90d count audited reads in the trailing 30 and 90
// days. Both are bounded by the audit retention window: a deployment
// keeping 30 days of audit reports the same number twice.
Reads30d int64 `json:"reads_30d" example:"42"`
Reads90d int64 `json:"reads_90d" example:"117"`
// BySurface30d breaks the 30-day count down by Surface* value.
BySurface30d map[string]int64 `json:"by_surface_30d,omitempty"`
// LastReadAt is the most recent audited read within the retention window.
// The durable answer lives on the resource row (Resource.LastReadAt), which
// outlives retention; this field is what the rollup itself saw.
LastReadAt *time.Time `json:"last_read_at,omitempty"`
}
Usage is the audit-derived read activity of a single resource.
type UsageReader ¶ added in v1.115.0
type UsageReader interface {
ResourceUsage(ctx context.Context, resourceIDs []string) (map[string]Usage, error)
}
UsageReader aggregates read events into per-resource usage. The Postgres audit store implements it, the same way it implements prompt usage (#1009): the audit log is already the durable record of who read what, so usage is a rollup of it rather than a second set of counters to keep in sync.
type Version ¶ added in v1.115.0
type Version struct {
ResourceID string `json:"resource_id" example:"a1b2c3d4e5f67890a1b2c3d4e5f67890"`
Version int `json:"version" example:"3"`
MIMEType string `json:"mime_type" example:"text/markdown"`
SizeBytes int64 `json:"size_bytes" example:"34000"`
S3Key string `json:"s3_key" example:"resources/persona/data-engineer/a1b2/v3/etl-runbook.md"`
UploaderSub string `json:"uploader_sub" example:"550e8400-e29b-41d4-a716-446655440000"`
UploaderEmail string `json:"uploader_email" example:"marcus.johnson@example.com"`
// RestoredFrom names the version this revision re-promoted, or nil when the
// revision was a fresh upload. A restore is recorded as a new head revision
// rather than a rewind so the trail stays append-only.
RestoredFrom *int `json:"restored_from,omitempty" example:"1"`
CreatedAt time.Time `json:"created_at"`
}
Version is one recorded revision of a resource's content.
It carries only what varies per revision. Filename is absent on purpose: the canonical URI embeds the filename and revision keeps the URI stable, so every version of a resource shares the resource's filename.
type VersionStore ¶ added in v1.115.0
type VersionStore interface {
// AddRevision records a revision and points the resource head at its blob,
// in one transaction. The version number is assigned inside that
// transaction as one past the highest recorded, so two concurrent revisions
// cannot claim the same number, and the stored row is returned.
AddRevision(ctx context.Context, rev Revision) (*Version, error)
// ListVersions returns every recorded revision of a resource, newest first.
ListVersions(ctx context.Context, resourceID string) ([]Version, error)
// GetVersion returns one recorded revision. A missing revision surfaces as
// a wrapped sql.ErrNoRows (see IsNotFound).
GetVersion(ctx context.Context, resourceID string, version int) (*Version, error)
// PruneVersions deletes the oldest revisions beyond the newest keep and
// returns the deleted rows so the caller can remove their blobs. It never
// deletes a row whose S3 key is currently referenced by the resource head,
// so pruning can never orphan the live content.
PruneVersions(ctx context.Context, resourceID string, keep int) ([]Version, error)
}
VersionStore persists the content-revision trail of a resource and moves the head to a new revision. The Postgres resource store implements it; it is a separate interface from Store so a deployment or test that only needs metadata CRUD is not forced to model the version trail, and so callers can type-assert for the capability.
The trail and the head move together — a version row whose bytes no resource points at, or a head pointing at bytes no version row records, is a broken state — so AddRevision owns both writes rather than leaving the caller to sequence them.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package resourceindex is the managed-resource consumer of the shared indexjobs framework (#1012).
|
Package resourceindex is the managed-resource consumer of the shared indexjobs framework (#1012). |