storageurl

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 29, 2026 License: Apache-2.0, MIT Imports: 14 Imported by: 0

Documentation

Overview

Package storageurl converts internal storage references into HTTP(S) URLs an external client can load directly.

WeKnora persists files behind two internal reference forms: the stable `resource://<handle>` application identity and legacy/canonical provider paths (`local://…`, `minio://…`, `storage://<backend-id>/cos://…`). Neither is fetchable by a browser or a third-party app, which must otherwise call the authenticated `/files` proxy for every image.

This package is the single implementation of the "give me a loadable link" translation. It is used by the IM channels (which have no way to attach WeKnora credentials to an image fetch) and, opt-in, by the HTTP API so integrators receive ready-to-render URLs.

Index

Constants

View Source 
const (
	// QueryParam is the request query parameter that selects the mode per call.
	QueryParam = "resource_urls"
	// EnvVar sets the deployment-wide default mode for requests that omit
	// QueryParam. It sits alongside APP_EXTERNAL_URL, which is what actually
	// makes `resource://` handles resolvable to a public `/r/<token>` URL.
	EnvVar = "RESOURCE_URL_MODE"
)

Variables

View Source 
var ErrPublicModeForbidden = errors.New(
	"resource_urls=public is not available for a knowledge-base-restricted API key")

ErrPublicModeForbidden is returned when the caller asked for ModePublic but its credentials must not receive anonymous, time-limited file URLs. Callers map it to 403 rather than 400: the request is well-formed, the scope is not.

View Source 
var Pattern = types.StorageReferencePattern

Pattern matches every internal storage reference form: `resource://` handles, legacy `provider://` paths, and canonical `storage://<backend-id>/provider://` paths. The trailing character class stops at Markdown/HTML delimiters so a reference inside `![alt](…)` or `src="…"` is matched without them.

Functions

func BuildFileServiceForProvider 

func BuildFileServiceForProvider(
	tenant *types.Tenant,
	provider string,
	defaultSvc interfaces.FileService,
) interfaces.FileService

BuildFileServiceForProvider selects the FileService for a storage provider. The reference's own scheme wins over the tenant DefaultProvider. It falls back to the process-wide default FileService (STORAGE_TYPE / env) when tenant config is missing — mirrors ImageMultimodalService.resolveFileServiceForPayload (issue #1282).

func FindIncompleteMarkdownImage 

func FindIncompleteMarkdownImage(s string) int

FindIncompleteMarkdownImage returns the byte offset of an unclosed `![alt](url` suffix at the end of s, or -1 if none.

func FindIncompleteRef 

func FindIncompleteRef(s string) int

FindIncompleteRef returns the byte offset of a potentially truncated storage reference at the tail of s, or -1 if none.

func HandleModeForced 

func HandleModeForced(ctx context.Context) bool

HandleModeForced reports whether ctx was pinned by WithForcedHandleMode.

func HoldbackCutoff 

func HoldbackCutoff(chunk string) int

HoldbackCutoff returns the offset at which chunk stops being safe to flush, or len(chunk) when the whole chunk can be emitted.

func IsHTTPURL 

func IsHTTPURL(s string) bool

IsHTTPURL reports whether s is an http(s) URL — the only form an external client can fetch; any provider scheme (oss://, local://, …) is not. Scheme match is case-insensitive per RFC 3986 §3.1: a backend may emit an operator-configured host (e.g. OBS_PROXY_DOMAIN) with an uppercase scheme.

func LocalStorageBaseDir 

func LocalStorageBaseDir() string

LocalStorageBaseDir is the on-disk root used when a reference resolves to the local provider.

func Rewrite 

func Rewrite(ctx context.Context, content string, r Resolver, logPrefix string) string

Rewrite is the one-shot form of Rewriter.String for callers that translate a single self-contained message.

func WithForcedHandleMode 

func WithForcedHandleMode(ctx context.Context) context.Context

WithForcedHandleMode pins a request to ModeHandle. Anonymous surfaces use it: an embed visitor is authorized only by the channel's session handle, so it must keep fetching images through the channel-scoped `/embed/…/files` proxy instead of receiving a shareable, credential-free URL.

Types

type FileServiceResolver 

type FileServiceResolver struct {
	// contains filtered or unexported fields
}

FileServiceResolver resolves and caches one FileService per owner and storage instance. The cache is scoped to a single request or outbound message so a long answer with many images does not re-create an SDK client per reference.

Not safe for concurrent use; Rewriter drives it from one goroutine at a time.

func NewFileServiceResolver 

func NewFileServiceResolver(
	tenant *types.Tenant,
	defaultSvc interfaces.FileService,
	storageResolvers ...interfaces.StorageBackendResolver,
) *FileServiceResolver

NewFileServiceResolver builds a resolver for tenant. defaultSvc is the process-wide fallback for legacy references without an explicit backend. WithResources enables owner/instance routing for resource:// handles.

func (*FileServiceResolver) ResolveFileService 

func (r *FileServiceResolver) ResolveFileService(filePath string) interfaces.FileService

ResolveFileService implements Resolver.

func (*FileServiceResolver) WithAuthorization added in v0.9.0 

func (r *FileServiceResolver) WithAuthorization(authorize func(context.Context, string) error) *FileServiceResolver

WithAuthorization gates every reference before any storage credentials are selected. Model output and editable Markdown are not authorization evidence.

func (*FileServiceResolver) WithContext 

func (r *FileServiceResolver) WithContext(ctx context.Context) *FileServiceResolver

WithContext sets the context used for backend lookups and their logs.

func (*FileServiceResolver) WithResources added in v0.9.0 

func (r *FileServiceResolver) WithResources(
	catalog interfaces.ResourceCatalog,
	tenants interfaces.TenantService,
) *FileServiceResolver

WithResources enables routing resource handles through their recorded owner and storage instance. Callers must already be authorized to expose the references; this only selects storage and does not grant resource access.

type Held 

type Held struct {
	Content string
	Meta    interface{}
}

Held is one released tail: the rewritten content and the metadata carried by the last Push that contributed to it.

type Mode 

type Mode string

Mode selects how stored files are referenced in an API response. 

const (
	// ModeHandle returns the internal `resource://<handle>` reference. Clients
	// must fetch the bytes through the authenticated `/files` proxy. This is the
	// default and the only mode that never depends on external reachability.
	ModeHandle Mode = "handle"
	// ModePublic returns a time-limited HTTP(S) URL the client can load
	// directly, so third-party apps can render images without a second
	// authenticated call. References that cannot be turned into an HTTP URL are
	// left as handles.
	ModePublic Mode = "public"
)

func DefaultMode 

func DefaultMode(ctx context.Context) Mode

DefaultMode returns the deployment-wide default from EnvVar. An unset or unparseable value yields ModeHandle, so a typo degrades to the safe default rather than failing every request.

func ParseMode 

func ParseMode(raw string) (Mode, error)

ParseMode validates a mode supplied by a client or by configuration. An empty value yields ModeHandle so an unset parameter or setting keeps the default.

func ResolveMode 

func ResolveMode(ctx context.Context, queryValue string) (Mode, error)

ResolveMode combines the per-request query value with the deployment default, then applies the two limits that no caller may opt out of.

An explicit query value wins over the deployment default; an invalid one is returned as an error so integrators find their typo instead of silently receiving handles.

Limits:

  • A ctx pinned by WithForcedHandleMode (anonymous embed traffic) is silently downgraded rather than rejected, so an embed client that happens to send the parameter keeps working — it just never gets a public URL.
  • A knowledge-base-restricted API key is rejected with ErrPublicModeForbidden. Such a key is already denied the `/files` proxy, so handing it anonymous file URLs would widen its scope from "chunk text" to "file bytes" through the back door.

type Resolver 

type Resolver interface {
	// ResolveFileService returns nil when no backend can serve ref.
	ResolveFileService(ref string) interfaces.FileService
}

Resolver maps one storage reference to the FileService that owns it. A reference carries its own provider scheme and optional storage backend id, so a single answer may span several backends.

type Rewriter 

type Rewriter struct {
	// contains filtered or unexported fields
}

Rewriter replaces storage references with loadable HTTP URLs.

Resolutions are memoised for the Rewriter's lifetime. Create one per request or per outbound message: a `resource://` handle costs one access-grant row per resolution, and a streamed answer repeats the same image across many chunks.

Safe for concurrent use: mu covers both the memo and the resolver, which is itself single-threaded. Resolution therefore serialises across goroutines, which is what we want — two chunks naming the same image must not each pay for a signature.

Logging policy:

  • A successful rewrite logs the source reference at INFO. The signed URL is DEBUG-only so log aggregation cannot hand out anonymously readable links; operators can raise the log level when verifying reachability.
  • Failure or no-op logs at WARN. The no-op case usually means APP_EXTERNAL_URL is unset, the most common cause of "image broken in the IM channel / in my app" reports.

func NewRequestRewriter 

func NewRequestRewriter(
	ctx context.Context,
	mode Mode,
	defaultSvc interfaces.FileService,
	storageResolver interfaces.StorageBackendResolver,
) *Rewriter

NewRequestRewriter builds the Rewriter for one API request or response stream.

ModeHandle yields a disabled Rewriter, so the default path resolves nothing and costs no access-grant rows. The tenant is taken from ctx because a reference may live on a tenant-configured storage backend rather than the process-wide default.

No extra authorization gate applies here, unlike the `/files` proxy which rejects KB-restricted API keys. That gate exists because `/files` takes an arbitrary caller-supplied path it cannot bind to a KB allow-list. Here the references come from a response the caller is already authorized to receive, so the server — not the client — chooses which resources get a URL.

func NewRewriter 

func NewRewriter(r Resolver, logPrefix string) *Rewriter

NewRewriter returns a Rewriter that resolves references through r. logPrefix tags log lines with the calling surface (for example "IM" or "API"). A nil resolver yields a Rewriter that leaves content unchanged.

func (*Rewriter) CopyData 

func (w *Rewriter) CopyData(ctx context.Context, data map[string]interface{}) map[string]interface{}

CopyData returns a rewritten copy of an SSE metadata map, or data itself when it holds no storage reference. Agent tool results put renderable Markdown into this map, and its shape is tool-defined, so every string leaf is rewritten.

func (*Rewriter) CopyReferences 

func (w *Rewriter) CopyReferences(ctx context.Context, refs []*types.SearchResult) []*types.SearchResult

CopyReferences returns rewritten copies of retrieval results, covering both the chunk text and the structured image_info payload.

Copies rather than in-place edits because an SSE references payload shares its *SearchResult pointers with the stream replay buffer and the assistant message being persisted; rewriting those in place would corrupt both.

func (*Rewriter) CopyValue 

func (w *Rewriter) CopyValue(ctx context.Context, value interface{}) interface{}

CopyValue is CopyData for a decoded JSON value of any shape (object, array or scalar). Unchanged values are returned as-is.

func (*Rewriter) Enabled 

func (w *Rewriter) Enabled() bool

Enabled reports whether this Rewriter can actually rewrite anything.

func (*Rewriter) Ref 

func (w *Rewriter) Ref(ctx context.Context, ref string) string

Ref rewrites a value that is a bare storage reference rather than prose containing one — for example `MessageImage.URL`.

func (*Rewriter) RewriteMessages 

func (w *Rewriter) RewriteMessages(ctx context.Context, messages []*types.Message)

RewriteMessages replaces storage references in a message history response so clients receive loadable image URLs. Messages are mutated in place.

func (*Rewriter) RewriteMessagesResponse 

func (w *Rewriter) RewriteMessagesResponse(ctx context.Context, messages []*types.Message) []*types.Message

RewriteMessagesResponse returns rewritten message copies so callers can keep the originals untouched (for example objects returned from a service cache).

func (*Rewriter) String 

func (w *Rewriter) String(ctx context.Context, content string) string

String replaces every storage reference in content with an HTTP URL. References that are already HTTP, that no backend claims, or that resolve to a non-HTTP location are left untouched so the caller degrades to the authenticated file proxy rather than emitting an unfetchable URL.

type StreamRewriter 

type StreamRewriter struct {
	// contains filtered or unexported fields
}

StreamRewriter rewrites storage references in a stream of content deltas.

Each logical stream is identified by a key (WeKnora uses the SSE event id, the same key clients accumulate on). Push returns only the prefix that is safe to emit now, retaining any tail that may be an incomplete reference until the next Push for that key.

Safe for concurrent use.

func NewStreamRewriter 

func NewStreamRewriter(rewriter *Rewriter) *StreamRewriter

NewStreamRewriter wraps rewriter with per-stream holdback state.

func (*StreamRewriter) Enabled 

func (s *StreamRewriter) Enabled() bool

Enabled reports whether this StreamRewriter can rewrite anything.

func (*StreamRewriter) FlushAll 

func (s *StreamRewriter) FlushAll(ctx context.Context) map[string]Held

FlushAll releases every held tail, rewritten, keyed by stream. Callers use it when a stream ends without a terminal chunk (for example a client disconnect) so buffered content is not silently dropped.

func (*StreamRewriter) Push 

func (s *StreamRewriter) Push(ctx context.Context, key, chunk string, flush bool, meta interface{}) string

Push feeds the next chunk of the stream identified by key and returns the rewritten content that is ready to emit. Set flush on the stream's terminal chunk to release any held tail. meta is retained opaquely alongside the tail and handed back by FlushAll so a late release can carry the same metadata as the event it was cut from.

func (*StreamRewriter) Rewriter 

func (s *StreamRewriter) Rewriter() *Rewriter

Rewriter exposes the underlying Rewriter for stream fields that arrive whole (references, metadata) and therefore need no holdback.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.