Documentation
¶
Overview ¶
Package officeengine runs an app's office engine on this server: the Office editor app's x2t (ONLYOFFICE's converter) and its page layout (sdkjs in QuickJS), both compiled to WebAssembly and shipped in the app's release as engine.zip (docs/ONLYOFFICE.md → The office engine without a Document Server, docs/APP-PLUGINS-API.md → engines.office).
It is the second back end of the office engine apps convert with (`engines:office`) and of office thumbnails, beside the ONLYOFFICE Document Server (internal/onlyoffice). The server picks one of them (internal/server, the `engine` capability of Default apps); this package only runs the app's.
Two sides, one binary ¶
filex never links the engine's code (AGPL): it runs it as a separate WebAssembly program, files in and files out, the way it would run a Document Server's x2t. The PARENT side (Runner, in the server) starts `filex office-engine` - a hidden command of this same binary, the CHILD side (Serve) - and talks to it over the child's stdin and stdout, one JSON line per job (Request) and one per answer (Answer). The child holds the wazero runtimes: x2t-server.wasm, an instance per job, mounted on a directory of its own; and a warm layout worker per kind of document (layout-host.wasm running the app's worker.js over sdkjs).
Why a process of its own ¶
- Speed: both modules are compiled WITHOUT wazero's context watch (2-4.6 times faster on x2t, measured), and such a module can only be stopped by ending its process. The parent keeps every job's deadline and kills the process when it passes; the next job starts another.
- Memory: x2t takes 284 MiB at the least and 2.6 GiB for a 200,000-row csv. In a process of its own that memory, a trap and a loop go back to the system with the process, never into filex's heap.
Memory ¶
Every instance is capped (FILEX_OFFICE_ENGINE_MAX_MB, MaxMBDefault): its memory may not grow past the cap. x2t's memory is reserved up to the cap when the instance starts (memory.go), so growing it never copies it - the 200,000-row csv peaked at 6.2 GB of process with copies, 2.6 GiB without. The reservation is address space outside Go's heap: the system backs only what x2t writes and takes it all back when the instance closes, so job after job the process holds what the job used, not the cap. The parent also watches the process as a whole where the system lets it read a process's resident memory (proc_*.go): about twice the cap ends it. A layout worker is started again after WorkerJobs jobs or WorkerMaxBytes of memory; a process left idle for Options.IdleTimeout ends.
Compiling, and "preparing" ¶
wazero compiles x2t-server.wasm in 3.5-5 minutes the first time on a machine; its cache (in filex's data directory) loads it in about a second in every process after that. So a new engine package - an install, an upgrade, a new filex - is PREPARED in the background by one process that compiles both modules and starts a worker of each kind once (which writes the layout's bytecode); until it is done the engine is not Ready, its Status says "preparing", and a Document Server, if one is connected, takes the jobs meanwhile.
Index ¶
- Constants
- Variables
- func ClampMaxMB(mb int) int
- func KindOf(ext string) string
- func Main(stdin io.Reader, stdout io.Writer) int
- func Serve(ctx context.Context, cfg ChildConfig, in io.Reader, out io.Writer) error
- type Answer
- type ChildConfig
- type Conversion
- type Error
- type Hello
- type Options
- type Package
- type Purpose
- type Request
- type Runner
- func (r *Runner) Close()
- func (r *Runner) Convert(ctx context.Context, c Conversion) error
- func (r *Runner) DrawThumbnail(ctx context.Context, t Thumbnail) ([]byte, error)
- func (r *Runner) Ready() bool
- func (r *Runner) Release(app string)
- func (r *Runner) SetPackage(p *Package)
- func (r *Runner) Status() Status
- type State
- type Status
- type Thumbnail
Constants ¶
const ( EngineX2T = "x2t-server.wasm" EngineLayout = "layout-host.wasm" EngineWorker = "layout/worker.js" )
What engine.zip holds (docs/APP-PLUGINS-API.md → engines.office).
const ( KindWord = "word" KindCell = "cell" KindSlide = "slide" )
Kinds of document, as the Document Server's documentType names them.
const ( WorkerJobs = 20 WorkerMaxBytes = 1536 << 20 )
Worker renewal.
const ( // OpConvert turns In (From) into To, written at Out. OpConvert = "convert" // OpThumbnail draws In's first page as a PNG fitted into Size x Size, at // Out. OpThumbnail = "thumbnail" )
The operations.
const ( // CodeUnknown: the engine process ended without an answer, or a failure // that may pass (asked again later). CodeUnknown = -1 // CodeTimeout: the job's deadline passed (the parent ended the process). CodeTimeout = -2 // CodeCorrupt: the document could not be read or converted (x2t's 80, 84, // 86, a trap, a layout that failed). The same bytes fail again. CodeCorrupt = -3 // CodePassword: the document is protected by a password (x2t's 90, 91) // or by DRM (85). CodePassword = -5 // CodeParams: this format is not made from that one (x2t's 82, 88). CodeParams = -7 // CodeDetect: what the input is could not be told (x2t's 89, 95). CodeDetect = -9 // CodeTooLarge: the memory cap, or x2t's own size limits (93, 94, 96). CodeTooLarge = -10 )
The failures, as the Document Server's conversion API numbers them (internal/onlyoffice DS*): the server maps both back ends' failures with one table (onlyoffice.ClassOf) and says them with one set of sentences.
const ( MaxMBDefault = 2048 MaxMBMin = 512 MaxMBMax = 4096 )
The memory cap (FILEX_OFFICE_ENGINE_MAX_MB): every instance's memory, in MiB. x2t starts at 320 MiB and may use 4 GiB, its own maximum.
const ConfigEnv = "FILEX_OFFICE_ENGINE_CONFIG"
ConfigEnv carries the ChildConfig (JSON) to an engine process: the command line stays short and carries no path a process listing would show.
Variables ¶
var ( // ErrNoEngine: no running app brings an office engine. ErrNoEngine = errors.New("office engine: no app brings one") // ErrPreparing: the engine is being compiled (Status says since when). ErrPreparing = errors.New("office engine: being prepared") // ErrFailed: the engine could not be prepared (Status says why). ErrFailed = errors.New("office engine: could not be prepared") // ErrClosed: the runner was closed (filex is stopping). ErrClosed = errors.New("office engine: closed") )
Why a job was not even started.
Functions ¶
func ClampMaxMB ¶
ClampMaxMB is the cap in force for a configured value (0: the default).
func KindOf ¶
KindOf is the editor a document of this extension opens in, "" for none: the kinds the Document Server lays out (a PDF, a picture), and what x2t reads into each editor's format.
Types ¶
type Answer ¶
type Answer struct {
ID int64 `json:"id"`
OK bool `json:"ok"`
// Bytes is the result's size; Pages the pages the layout drew.
Bytes int64 `json:"bytes,omitempty"`
Pages int `json:"pages,omitempty"`
// Code is a failure as the Document Server's conversion API numbers it
// (the Code* constants), so a caller says both back ends' failures the
// same way; What is its short name for a record ("x2t-86", "trap",
// "memory"); Error the longer English, for a log.
Code int `json:"code,omitempty"`
What string `json:"what,omitempty"`
Error string `json:"error,omitempty"`
// Over: the result was larger than MaxBytes, and nothing was written.
Over bool `json:"over,omitempty"`
// Ms is where the time went (to_bin, layout, from_bin, convert).
Ms map[string]int64 `json:"ms,omitempty"`
}
Answer is what came of one Request.
type ChildConfig ¶
type ChildConfig struct {
// Engine is the app's engine.zip, EngineSHA256 the hash it must have.
Engine string `json:"engine"`
EngineSHA256 string `json:"engine_sha256"`
// Package is the app's interface bundle (ui.zip), which holds the
// editor's sdkjs/ and fonts/ under EditorDir; mounted read-only.
Package string `json:"package"`
PackageSHA256 string `json:"package_sha256"`
EditorDir string `json:"editor_dir"`
// Cache is the compilation cache's root (wazero's, and the layout's
// bytecode), shared by every engine process of this filex.
Cache string `json:"cache"`
// Work is the process's own temporary directory; the parent removes it
// when the process ends.
Work string `json:"work"`
// MaxMB caps every instance's memory (MiB).
MaxMB int `json:"max_mb"`
// Prepare: compile, start a worker of each kind once, say Prepared and
// end - no jobs.
Prepare bool `json:"prepare,omitempty"`
}
ChildConfig is what an engine process is started with.
type Conversion ¶
type Conversion struct {
// Src is the input, From its format; To the conversion API's output type.
Src, From, To string
// Delimiter and CodePage are the conversion API's CSV knobs.
Delimiter, CodePage int
// Dst is where the result goes (it must not exist); MaxBytes its ceiling.
Dst string
MaxBytes int64
}
Conversion is one office conversion (the office engine apps use).
type Error ¶
type Error struct {
// Code is the conversion API's number (Code*), What its short name for a
// record ("x2t-86", "trap", "memory", "timeout", "crash").
Code int
What string
// Over: the result was larger than the request's MaxBytes.
Over bool
Err error
}
Error is a job the engine did not do.
type Hello ¶
type Hello struct {
// Ready: the modules are compiled, jobs may come.
Ready bool `json:"ready,omitempty"`
// Prepared: a Prepare run finished (the process ends after it).
Prepared bool `json:"prepared,omitempty"`
// Error: the process could not start (a package that does not hash to
// its pin, a module that does not compile); it ends after saying so.
Error string `json:"error,omitempty"`
CompileMs int64 `json:"compile_ms,omitempty"`
}
Hello is an engine process's first line.
type Options ¶
type Options struct {
// Command is the argv that starts an engine process (default: this
// executable, `office-engine`). Env is added to its environment.
Command []string
Env []string
// CacheDir is the compilation cache (filex's data directory).
CacheDir string
// WorkDir is where each process's temporary directory is made (default:
// the system's).
WorkDir string
// MaxMB is the memory cap in force, read when a process starts.
MaxMB func() int
// Slots is how many processes a pool may run at once (thumbs.office_slots,
// 1-4), read at every job.
Slots func() int
// ProcessLimit is the most memory a process may hold (bytes), read at
// every job; default about twice the cap. 0 from the function: none.
ProcessLimit func() int64
// IdleTimeout ends a process nobody used for that long (default 3 min);
// JobsPerProcess ends one after that many jobs (default 50).
IdleTimeout time.Duration
JobsPerProcess int
// StartTimeout bounds a process's start (default 2 min: the modules come
// from the cache); PrepareTimeout a prepare run (default 30 min);
// RetryPrepare is how long after a failed one it is tried again
// (default 30 min).
StartTimeout time.Duration
PrepareTimeout time.Duration
RetryPrepare time.Duration
Log *slog.Logger
}
Options tune a Runner.
type Package ¶
type Package struct {
App, Version string
Engine string
EngineSHA256 string
UI string
UISHA256 string
EditorDir string
// HostVersion is this filex's version: a new filex (and with it a new
// wazero) prepares the engine again.
HostVersion string
}
Package is the engine an app brings: its engine.zip and the interface bundle that holds the editor's scripts and fonts, each with the hash it was installed with.
type Purpose ¶
type Purpose string
Purpose names the pool a job runs in: thumbnails and conversions each have their own processes, so a long conversion never holds the thumbnails up (and the other way round).
type Request ¶
type Request struct {
ID int64 `json:"id"`
Op string `json:"op"`
// In is the input on this machine, From its format (the extension, lower
// case, no dot).
In string `json:"in"`
From string `json:"from"`
// To is the conversion API's output type, as the Document Server is asked
// for it: an extension, or "pdfa".
To string `json:"to,omitempty"`
// Out is where the result goes. It must not exist: the child creates it.
Out string `json:"out"`
// MaxBytes is the largest result accepted (0: no limit). A larger one is
// not written and the answer says Over.
MaxBytes int64 `json:"max_bytes,omitempty"`
// Delimiter and CodePage are the conversion API's CSV knobs (0: not said):
// the delimiter as the Document Server numbers it (1 tab, 2 semicolon,
// 3 colon, 4 comma, 5 space), the code page as Windows does (65001).
Delimiter int `json:"delimiter,omitempty"`
CodePage int `json:"code_page,omitempty"`
// Size is a thumbnail's box, in pixels.
Size int `json:"size,omitempty"`
// Params are the editor's json parameters (a Document Server's
// m_sJsonParams): a spreadsheet's page layout for its thumbnail.
Params json.RawMessage `json:"params,omitempty"`
}
Request is one job.
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
Runner runs the office engine of the package it is given, in processes of its own.
func (*Runner) Close ¶
func (r *Runner) Close()
Close ends every process; the runner takes no more jobs.
func (*Runner) Convert ¶
func (r *Runner) Convert(ctx context.Context, c Conversion) error
Convert runs one conversion. A failure is an *Error (Over: the result was over MaxBytes), ErrNoEngine / ErrPreparing / ErrFailed when the engine does not run, or ctx's error when the deadline passed.
func (*Runner) DrawThumbnail ¶
DrawThumbnail draws a PNG of the document's first page.
func (*Runner) Ready ¶
Ready reports whether jobs run now. From memory: thumbnails ask it for every listed office file.
func (*Runner) Release ¶
Release ends every process of an app's engine and forgets the package - before the app's files are moved (an upgrade, a removal): on Windows a file a process holds open cannot be moved. SetPackage brings it back.
func (*Runner) SetPackage ¶
SetPackage switches the runner to an engine package (nil: none). The same package again changes nothing; another one ends every process of the previous one and prepares the new one unless this machine already has.
type State ¶
type State string
State is where the engine is.
const ( // StateAbsent: no running app brings an office engine. StateAbsent State = "absent" // StatePreparing: compiling it, in the background (3.5-5 minutes the // first time on a machine). StatePreparing State = "preparing" // StateReady: jobs run. StateReady State = "ready" // StateFailed: it could not be prepared (Status.Error says why); tried // again after Options.RetryPrepare. StateFailed State = "failed" )
The states.
type Status ¶
type Status struct {
State State `json:"state"`
App string `json:"app,omitempty"`
Version string `json:"version,omitempty"`
Error string `json:"error,omitempty"`
Since time.Time `json:"since"`
}
Status is the engine as the administrator's screens show it.
type Thumbnail ¶
type Thumbnail struct {
// Src is the document, From its format.
Src, From string
// Size is the box the page is fitted into; Params the editor's json
// parameters (a sheet's page layout); MaxBytes the largest picture.
Size int
Params json.RawMessage
MaxBytes int64
}
Thumbnail is one picture of a document's first page.