webpreview

package
v0.3.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 19, 2026 License: AGPL-3.0 Imports: 25 Imported by: 0

Documentation

Overview

Package webpreview is the browser twin of the phone preview: an overlay-injecting reverse proxy that `clank preview` puts in front of a KindWeb dev server (see internal/host/preview), plus the dictation service the overlay's push-to-talk streams into.

The proxy owns three concerns on one loopback origin so the injected overlay never fights CORS or mixed-origin auth:

  • serve the overlay assets (embedded, /__clank/overlay.js)
  • relay /__clank/api/* to the daemon's unix socket behind a per-run bearer token (the web analog of preview_frontdoor.go's LAN pairing token)
  • rewrite proxied HTML to inject the overlay <script> + config

Dictation runs in this process rather than the daemon on purpose: it lives exactly as long as `clank preview` does, and keeping it out of the daemon means no host API surface until a second client (TUI, mobile-web) wants to share it.

Index

Constants

View Source
const (
	OverlayPath       = "/__clank/overlay.js"
	ChatPath          = "/__clank/chat.js"
	MarkdownPath      = "/__clank/markdown.js"
	TranscriptPath    = "/__clank/transcript.js"
	SettingsPath      = "/__clank/settings.js"
	SourceControlPath = "/__clank/sourcecontrol.js"
	BoxPosPath        = "/__clank/boxpos.js"
	LauncherPath      = "/__clank/launcher.js"
	ResizePath        = "/__clank/resize.js"
	WorkletPath       = "/__clank/worklet.js"
	APIPrefix         = "/__clank/api"

	// NativePreviewUserAgentToken keeps the JS overlay out of clank-mobile's
	// WebView, where the Kotlin prompt box owns the interaction surface.
	NativePreviewUserAgentToken = "ClankNativePreview/1"
)
View Source
const EngineEnvVar = "CLANK_VOICE_ASR_CMD"

EngineEnvVar overrides the dictation engine with a shell command that reads a 16 kHz mono s16le WAV on stdin and prints the transcript to stdout, e.g.

CLANK_VOICE_ASR_CMD='whisper-cli -m ~/models/ggml-base.en.bin -nt -f -'

One process per utterance, final-only (no partials). When unset, the sherpa-onnx engine is used if a clank-voice binary is installed (see sherpa.go); otherwise voice is off.

View Source
const LauncherSeenPath = "/__clank/launcher/seen"
View Source
const ModelSlug = "parakeet-tdt-0.6b-v3-int8"

ModelSlug names the dictation model set, mirroring clank-mobile's ModelManager (MODEL_SLUG there): NVIDIA Parakeet TDT 0.6b v3 int8 for ASR plus Silero VAD for segmentation, both in sherpa-onnx form.

Variables

This section is empty.

Functions

func DefaultModelsDir

func DefaultModelsDir() (string, error)

DefaultModelsDir is where EnsureModels materializes the model set: <CLANK_DIR>/models/<slug>. Living under config.Dir keeps isolated stacks (CLANK_DIR overrides) isolated here too.

func EnsureModels

func EnsureModels(ctx context.Context, dir string, progress ModelDownloadProgress) error

EnsureModels downloads any missing model files into dir. Idempotent and interrupt-safe: each file streams to <name>.tmp and is renamed into place only when complete, so a Ctrl+C mid-encoder re-downloads just that file next time.

func FindClankVoice

func FindClankVoice() (string, error)

FindClankVoice locates the clank-voice binary: next to the running executable first (paired installs), then PATH — the same resolution clank uses for clankd and the daemon uses for clank-host.

func InjectOverlayResponse

func InjectOverlayResponse(resp *http.Response, snippet []byte) error

InjectOverlayResponse rewrites a successful identity-encoded HTML response. Non-HTML, compressed, and oversized bodies pass through unchanged.

func ModelsPresent

func ModelsPresent(dir string) bool

ModelsPresent reports whether every model file exists non-empty in dir. Partial downloads never count: files are written to a .tmp name and renamed only when complete.

func OverlaySnippet

func OverlaySnippet(config map[string]any) ([]byte, error)

OverlaySnippet renders the config and module tags injected into preview HTML.

func ServeOverlayAsset

func ServeOverlayAsset(w http.ResponseWriter, r *http.Request) bool

ServeOverlayAsset serves one embedded overlay module and reports whether the request matched a reserved asset path.

func ShouldInjectOverlay

func ShouldInjectOverlay(userAgent string) bool

ShouldInjectOverlay reports whether a preview page needs the browser UI.

Types

type DictationEngine

type DictationEngine string

DictationEngine selects how the overlay transcribes push-to-talk audio. The value is a user preference persisted across preview runs (see Options.PersistDictationEngine); empty means "not chosen yet", which makes the overlay ask on first dictation.

const (
	// DictationLocal transcribes on this machine via the configured
	// Engine (clank-voice or an exec command) — audio never leaves it.
	DictationLocal DictationEngine = "local"
	// DictationWebSpeech transcribes in the browser via the Web Speech
	// API (SpeechRecognition), which typically uploads audio to the
	// browser vendor's speech service (Google in Chrome, Apple in
	// Safari). No audio touches the /__clank/voice socket.
	DictationWebSpeech DictationEngine = "webspeech"
)

func ParseDictationEngine

func ParseDictationEngine(s string) (DictationEngine, bool)

ParseDictationEngine validates a stored or client-sent engine string. Empty is NOT valid here — "unchosen" is a caller-level state, not an engine.

type Engine

type Engine interface {
	// Open starts a session. The caller owns it and must Close it.
	Open(ctx context.Context) (Session, error)
	// Describe returns a short human-readable engine label for the
	// preview banner and logs.
	Describe() string
}

Engine produces dictation sessions. Implementations may serialize sessions (the sherpa engine funnels everything through one model process); Open blocks until the engine is free or ctx is done.

func EngineFromEnv

func EngineFromEnv() Engine

EngineFromEnv returns the exec-command engine configured via EngineEnvVar, or nil when unset.

type ExecEngine

type ExecEngine struct {
	Cmdline string
}

ExecEngine shells out to a user-configured command per utterance. WAV on stdin (not raw PCM) so off-the-shelf tools — whisper.cpp's whisper-cli, sox pipelines, a custom script — work without flags describing the sample format. Final-only: partials would mean re-decoding the whole utterance per chunk at O(n²) cost.

func (*ExecEngine) Describe

func (e *ExecEngine) Describe() string

func (*ExecEngine) Open

func (e *ExecEngine) Open(ctx context.Context) (Session, error)

type ModelDownloadProgress

type ModelDownloadProgress func(file string, index, count int, done, total int64)

ModelDownloadProgress reports EnsureModels progress. done/total are bytes for the current file; total is -1 when the server sent no Content-Length.

type Options

type Options struct {
	// UpstreamURL is the HTTP(S) origin the proxy fronts. Required.
	UpstreamURL *url.URL

	// DaemonSocketPath is the clank daemon's unix socket; /__clank/api/*
	// relays there. Required.
	DaemonSocketPath string

	// Token gates /__clank/api/* and /__clank/voice. The injected page
	// config carries it; nothing else on the machine learns it. This is
	// the same trust move as the LAN front door's pairing token, scoped
	// down to loopback.
	Token string

	// OverlayConfig is serialized into window.__CLANK_PREVIEW for the
	// overlay (session context: local_path, backend, hostname, name,
	// optional session_id). Token and voice availability are added here.
	OverlayConfig map[string]any

	// Engine powers local dictation; nil marks the local engine
	// unavailable in the overlay config and 503s the voice endpoint.
	// The overlay may still offer the browser's Web Speech API, which
	// it detects client-side.
	Engine Engine

	// DictationEngine is the persisted engine choice injected into the
	// overlay config. Empty means unchosen (the overlay asks on first
	// dictation); anything else must parse via ParseDictationEngine.
	DictationEngine DictationEngine

	// PersistDictationEngine stores a choice made in the overlay's
	// engine picker so it survives preview restarts. nil means choices
	// only last for this run.
	PersistDictationEngine func(DictationEngine) error

	// LauncherSeen suppresses the first-use coachmark. PersistLauncherSeen
	// stores the acknowledgement across auto-assigned preview ports.
	LauncherSeen        bool
	PersistLauncherSeen func() error

	// ListenPort for the proxy on 127.0.0.1; 0 picks a free port.
	ListenPort int

	Log *log.Logger
}

Options configures Start.

type Result

type Result struct {
	Text  string
	Final bool
	// Err reports a failed decode (Final implied). The session may
	// still accept the next utterance unless Results was closed.
	Err error
}

Result is one transcription update for the current utterance. Partials are cumulative (the text so far, not a delta), matching how clank-mobile's recognizer commits VAD segments monotonically.

type Server

type Server struct {
	// URL is the browser-facing address, http://127.0.0.1:<port>.
	URL string
	// contains filtered or unexported fields
}

Server is a running overlay proxy.

func Start

func Start(opts Options) (*Server, error)

Start binds the loopback listener and serves in the background. Loopback-only on purpose: unlike the phone flow there's no LAN peer, and not binding 0.0.0.0 keeps the daemon relay off the network.

func (*Server) Shutdown

func (s *Server) Shutdown(ctx context.Context)

Shutdown stops the listener. The dev server and daemon are the caller's concern, same split as the LAN front door.

type Session

type Session interface {
	// Feed accepts PCM as it arrives from the mic.
	Feed(pcm []byte) error
	// End marks push-to-talk release: flush and decode; exactly one
	// Final Result follows on Results (empty text = heard nothing).
	End() error
	// Cancel discards audio buffered since the last End.
	Cancel() error
	// Results streams partial and final updates. Closed when the
	// session is Closed or the engine dies.
	Results() <-chan Result
	// Close releases the session and, for serializing engines, hands
	// the engine to the next Open.
	Close() error
}

Session is one dictation conversation, typically bound to one overlay WebSocket. PCM is s16le, 16 kHz, mono — the same shape clank-mobile feeds sherpa-onnx.

type SherpaEngine

type SherpaEngine struct {
	// Bin is the clank-voice executable path.
	Bin string
	// Args are passed verbatim (production: --models <dir>). Split out
	// so tests can substitute a fake binary with its own flags.
	Args []string
	// ReadyTimeout caps the model-load wait after spawn. Zero means
	// defaultReadyTimeout.
	ReadyTimeout time.Duration
	Log          *log.Logger
	// contains filtered or unexported fields
}

SherpaEngine drives a clank-voice subprocess (voice-engine module): sherpa-onnx with Silero VAD + the same Parakeet model clank-mobile runs on-device. The subprocess is spawned once and kept warm — the model load is the expensive part (~670 MB of weights), so paying it per utterance or per WebSocket would wreck push-to-talk latency.

The cgo/onnxruntime dependency lives entirely in the voice-engine module; this driver is pure Go, so clank's CGO_ENABLED=0 cross-builds (e.g. the fly.io clank-host artifact) are unaffected.

Sessions are serialized: one model process, one utterance stream at a time. Open blocks until the previous session Closes.

func NewSherpaEngine

func NewSherpaEngine(bin, modelsDir string, lg *log.Logger) *SherpaEngine

NewSherpaEngine returns an engine driving bin with the standard production arguments.

func (*SherpaEngine) Close

func (e *SherpaEngine) Close() error

Close tears down the warm model process. Implements io.Closer so the CLI can release ~1 GB of RSS on preview shutdown.

func (*SherpaEngine) Describe

func (e *SherpaEngine) Describe() string

func (*SherpaEngine) Open

func (e *SherpaEngine) Open(ctx context.Context) (Session, error)

func (*SherpaEngine) Prewarm

func (e *SherpaEngine) Prewarm()

Prewarm starts loading the model in the background so the first push-to-talk doesn't pay the multi-second load. A session opened mid-load simply waits on the spawn lock and proceeds when ready.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL