README
¶
sqi (pronounced "sky") is an open source distributed task and render farm manager built for modern production pipelines. It is designed to run simply on a handful of local workstations and scale to hybrid on-premises and cloud infrastructure without changing how you work.
📖 Documentation — quickstart, architecture, configuration, and full reference.
Status: v0.2.0 (Phase 2) released. Builds on the Phase 1 core with products and presets as an authoring layer over OpenJD, a community preset-library integration, product-driven submission, additional path-translation modes, S3-compatible storage, compute locations, and in-application DCC submitters for Maya, Houdini, Nuke, and Blender. Contributions, feedback, and discussion welcome.
Why sqi exists
The render farm management space is in an awkward moment. Legacy on-premises systems are aging out of active development. The primary cloud-native alternative requires deep AWS infrastructure investment, locks you into a single provider, and introduces workflow constraints that don't match production realities. A lot of studios are looking for a third option.
sqi is that option. It draws on years of experience building and operating Smedge (Uberware's long-running render farm product) and applies those lessons to a clean, modern architecture that works wherever your compute lives — local network, cloud, or both.
Design goals
Simple by default. A single binary gets a working farm. Workers on other machines run a second binary and can find the server automatically. No external database, no message broker configuration, no cloud account required to get started.
Scalable when needed. The same software that runs your five-machine studio farm can be deployed in a distributed, high-availability configuration with a separate database, message broker, and workers across multiple cloud providers. The upgrade path is configuration, not migration.
Not tied to any cloud provider. Workers run on Linux, macOS, and Windows — bare metal, VMs, or containers. Cloud compute locations are supported across AWS, GCP, Azure, and any provider that can run a container or a binary. Your control plane runs where you want it.
OpenJD compatible. sqi adopts the Open Job Description format as its native job execution layer. Jobs authored for OpenJD-compatible tools work with sqi without reformatting. This is a real standard designed for portability, not a proprietary format.
General purpose. Rendering is the primary use case and the domain sqi is designed around, but the job model is general. Any workload expressible as a command with defined inputs, outputs, and environment is a valid sqi job — simulation, transcoding, machine learning pipelines, data processing, software development, or anything else a studio runs at scale.
Key features
-
Farms and queues organize work hierarchically. Configuration — scheduling policy, worker affinity, usage pool limits — can be set at the farm level and overridden per queue, so you're not repeating yourself on every job submission. Storage doesn't need this: a storage location already maps each compute location to its own root path, so jobs resolve the right path wherever they run without per-queue configuration.
-
Products and presets are the authoring layer above raw job descriptions. A product defines a class of work (an Arnold render, a Houdini sim, an ffmpeg transcode) in terms of user-friendly parameters and how they map to commands. A preset is a ready-to-use product definition for a specific tool, installable from a community library directly through the web UI.
-
Community preset library is a public repository of product presets covering major DCCs — Arnold, Blender, Houdini, Maya, Nuke, After Effects, Cinema 4D, and more. Browse, install, and customize presets from within the
sqiinterface. Community contributions welcome. -
Named storage locations provide a clean abstraction over where data lives. A location named
nas_showsmight resolve to/mnt/nas/showson a local Linux worker,Z:\showson a Windows worker, ands3://studio-bucket/showson a cloud instance. Jobs reference location names; workers resolve them. Built-in support for S3-compatible object storage (AWS S3, Backblaze B2, Cloudflare R2, MinIO, and others). -
Path translation handles the fact that different applications deal with cross-platform and cross-environment paths differently. Product definitions declare how paths should be resolved — baked into the command line, passed as remapping arguments, set as environment variables, or staged locally before execution.
-
Usage pool management tracks concurrent usage of any shared, finite resource across all workers. Tell
sqiyou have 20 Arnold render licenses (or a per-show task cap, or any other limit) and it ensures no more than that many tasks run at once, globally, across every compute location. Bring-your-own-limit —sqienforces your cap, not its own. -
Pull-based workers register with the farm and pull assigned tasks rather than receiving pushed assignments. Workers can be added, removed, and scaled without the scheduler needing to manage their lifecycle. This makes auto-scaling and ephemeral cloud workers straightforward.
-
Layered authentication — from no auth at all (local network trust, appropriate for small private farms) through local accounts, LDAP/Active Directory, and OAuth2/OIDC for SSO. Designed so simple deployments stay simple and enterprise deployments get what they need.
-
Web UI is the primary interface. No required desktop application. Real-time job and worker status, log streaming, preset management, product configuration, and farm administration — all in a browser.
-
DCC submitters for Maya, Houdini, Nuke, and Blender via the
sqi-submitterPython package — in-application submission with per-host scene extraction and pre-fill, built on top of thesqi-sdkPython library, which covers scripted submission and pipeline integration on its own. -
Optional LLM integration (disabled by default) adds natural-language interfaces for farm management — explain a job failure in plain English, manage workers or job priorities by description, filter and search jobs conversationally. Pluggable provider model: bring your own API key (OpenAI, Anthropic, Azure, or a local Ollama instance).
sqinever requires or bundles an AI service.
Deployment
All-in-one (simple mode)
sqi-server # runs the scheduler, API, web UI, everything
sqi-worker # run on each render node — finds the server automatically
On a local network, workers discover the server via mDNS and connect without any configuration. Open a browser, start submitting. That's it.
See the Quickstart for a full walkthrough (binary or Docker Compose), including creating a farm and queue and submitting your first job.
Distributed (production mode)
A planned configuration (Phase 4) for high availability: a standalone NATS message broker and PostgreSQL for durable state, with workers connecting to the broker directly. The same binaries will support this mode; PostgreSQL, the HA scheduler, and Kubernetes manifests are not yet implemented. Today, deployment uses the single-binary or Docker Compose simple mode above.
Both modes run the same software. The difference is configuration.
Status and roadmap
sqi v0.1.0 delivered the Phase 1 core: scheduler, pull-based workers, OpenJD job execution, and a basic web UI, with a Python client and single-binary or Docker Compose deployment. v0.2.0 completes Phase 2 — products and presets as an authoring layer over OpenJD, the community preset-library integration, a product-driven submission form, additional path-translation modes, S3-compatible storage, compute locations, and in-application DCC submitters for Maya, Houdini, Nuke, and Blender. Authentication and multi-user support (Phase 3) and production-hardening features follow in later phases.
This is a real project with a concrete development commitment, not a design document waiting for funding. Feedback on priorities is welcome — open an issue or start a discussion.
See ROADMAP.md for more information about the design and implementation plan.
Contributing
Contributions are welcome at all stages — code, preset definitions, documentation, bug reports, and design discussion.
The community preset library in particular benefits from people who know their tools well. If you have a working submission setup for a DCC or tool and want to share it as a sqi preset, that's a valuable contribution that doesn't require any Go or server-side knowledge.
See CONTRIBUTING.md for guidelines.
License
sqi is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). This means you can use, modify, and self-host sqi freely. If you offer sqi (or a modified version) as a network service, the AGPL requires you to make your source available.
A commercial license is available for organizations that require it. Contact robin@uberware.net.
About
sqi is built by Uberware, the company behind Smedge, a render farm manager with over two decades of production use. sqi is a clean-break successor designed for the infrastructure realities of the next decade.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
presetgen
command
Command presetgen publishes the repo's authored presets (presets/sqi/*.yaml) into a checkout of the preset-library repo: it writes the definition files under a namespaced subdir and merges their entries into index.json, leaving any foreign entries untouched.
|
Command presetgen publishes the repo's authored presets (presets/sqi/*.yaml) into a checkout of the preset-library repo: it writes the definition files under a namespaced subdir and merges their entries into index.json, leaving any foreign entries untouched. |
|
sqi-server
command
Command sqi-server is the sqi distributed task management server.
|
Command sqi-server is the sqi distributed task management server. |
|
sqi-worker
command
Command sqi-worker is the sqi distributed task management worker agent.
|
Command sqi-worker is the sqi distributed task management worker agent. |
|
internal
|
|
|
api
Package api wires together the sqi-server HTTP surface: the chi router, standard middleware stack, and all route groups.
|
Package api wires together the sqi-server HTTP surface: the chi router, standard middleware stack, and all route groups. |
|
bus
Package bus provides the embedded NATS JetStream message broker and the subject/stream definitions used for sqi-server's internal messaging.
|
Package bus provides the embedded NATS JetStream message broker and the subject/stream definitions used for sqi-server's internal messaging. |
|
config
Package config defines the sqi-server runtime configuration and provides layered loading from built-in defaults, a YAML/JSON file, SQI_* environment variables, and CLI flag overrides — in that override order.
|
Package config defines the sqi-server runtime configuration and provides layered loading from built-in defaults, a YAML/JSON file, SQI_* environment variables, and CLI flag overrides — in that override order. |
|
diag
Package diag holds the server-side in-memory ring buffer of diagnostic (operational) log records gathered from sqi-server itself and from connected workers.
|
Package diag holds the server-side in-memory ring buffer of diagnostic (operational) log records gathered from sqi-server itself and from connected workers. |
|
discovery
Package discovery advertises the running sqi-server on the local network via multicast DNS (mDNS / DNS-SD) so that workers and the sqi CLI can find it without manual address configuration.
|
Package discovery advertises the running sqi-server on the local network via multicast DNS (mDNS / DNS-SD) so that workers and the sqi CLI can find it without manual address configuration. |
|
health
Package health implements liveness and readiness checks for sqi-server.
|
Package health implements liveness and readiness checks for sqi-server. |
|
log
Package log provides slog-based structured logging for the sqi binaries.
|
Package log provides slog-based structured logging for the sqi binaries. |
|
metrics
Package metrics defines and registers all Prometheus metrics for sqi-server.
|
Package metrics defines and registers all Prometheus metrics for sqi-server. |
|
middleware
Package middleware provides net/http middleware for sqi-server.
|
Package middleware provides net/http middleware for sqi-server. |
|
openjd
Package openjd implements parsing, validation, and parameter-space expansion for Open Job Description (OpenJD) job templates.
|
Package openjd implements parsing, validation, and parameter-space expansion for Open Job Description (OpenJD) job templates. |
|
openjd/fmtstring
Package fmtstring resolves Open Job Description (OpenJD) format strings.
|
Package fmtstring resolves Open Job Description (OpenJD) format strings. |
|
presetgen
Package presetgen generates a preset-library index and definition files from the repo's authored presets (presets/sqi/*.yaml) and publishes them into a library checkout.
|
Package presetgen generates a preset-library index and definition files from the repo's authored presets (presets/sqi/*.yaml) and publishes them into a library checkout. |
|
presetlib
Package presetlib is the client for a community preset library: it fetches a static JSON index (cached in memory) and, on install, fetches and verifies a single preset definition.
|
Package presetlib is the client for a community preset library: it fetches a static JSON index (cached in memory) and, on install, fetches and verifies a single preset definition. |
|
product
Package product is the catalog layer over OpenJD templates: it parses product definition files, holds the embedded built-ins, validates inline templates, and overlays read-only built-ins on the store via Catalog.
|
Package product is the catalog layer over OpenJD templates: it parses product definition files, holds the embedded built-ins, validates inline templates, and overlays read-only built-ins on the store via Catalog. |
|
scheduler
Package scheduler implements the sqi-server assignment loop and worker registry — the authoritative component for deciding which task runs on which worker.
|
Package scheduler implements the sqi-server assignment loop and worker registry — the authoritative component for deciding which task runs on which worker. |
|
server
Package server owns the sqi-server component lifecycle: starting and stopping the store, message bus, scheduler, HTTP server, and mDNS responder in the correct dependency order.
|
Package server owns the sqi-server component lifecycle: starting and stopping the store, message bus, scheduler, HTTP server, and mDNS responder in the correct dependency order. |
|
store
Package store defines the storage interface for sqi-server and the domain model types shared across all components.
|
Package store defines the storage interface for sqi-server and the domain model types shared across all components. |
|
store/fake
Package fake provides an in-memory implementation of store.Store for unit tests that must avoid touching the filesystem.
|
Package fake provides an in-memory implementation of store.Store for unit tests that must avoid touching the filesystem. |
|
store/migrations
Package migrations embeds the SQL migration files used by goose to manage the sqi-server SQLite schema.
|
Package migrations embeds the SQL migration files used by goose to manage the sqi-server SQLite schema. |
|
store/sqlite
Package sqlite provides a SQLite-backed implementation of store.Store.
|
Package sqlite provides a SQLite-backed implementation of store.Store. |
|
ui
Package ui serves the embedded web UI bundle over HTTP with single-page application (SPA) routing semantics.
|
Package ui serves the embedded web UI bundle over HTTP with single-page application (SPA) routing semantics. |
|
version
Package version holds build-time metadata injected by the release toolchain.
|
Package version holds build-time metadata injected by the release toolchain. |
|
worker/cancel
Package cancel implements per-task cancel-signal subscriptions for sqi-worker.
|
Package cancel implements per-task cancel-signal subscriptions for sqi-worker. |
|
worker/capabilities
Package capabilities provides auto-detection and manual override of sqi-worker host capabilities reported to sqi-server at registration time.
|
Package capabilities provides auto-detection and manual override of sqi-worker host capabilities reported to sqi-server at registration time. |
|
worker/config
Package config defines the sqi-worker runtime configuration and provides layered loading from built-in defaults, a YAML/JSON file, SQI_WORKER_* environment variables, and CLI flag overrides — in that override order.
|
Package config defines the sqi-worker runtime configuration and provides layered loading from built-in defaults, a YAML/JSON file, SQI_WORKER_* environment variables, and CLI flag overrides — in that override order. |
|
worker/diaglog
Package diaglog implements the sqi-worker diagnostic-log sink.
|
Package diaglog implements the sqi-worker diagnostic-log sink. |
|
worker/discovery
Package discovery provides mDNS-based sqi-server discovery for sqi-worker.
|
Package discovery provides mDNS-based sqi-server discovery for sqi-worker. |
|
worker/envutil
Package envutil provides shared helpers for building process environment variable slices from os.Environ() plus a caller-supplied override map.
|
Package envutil provides shared helpers for building process environment variable slices from os.Environ() plus a caller-supplied override map. |
|
worker/executor
Package executor implements the bare-metal task executor for sqi-worker.
|
Package executor implements the bare-metal task executor for sqi-worker. |
|
worker/fmtres
Package fmtres resolves OpenJD "{{...}}" format strings on the worker, where the session working directory — and therefore Session.WorkingDirectory — is only known at run time.
|
Package fmtres resolves OpenJD "{{...}}" format strings on the worker, where the session working directory — and therefore Session.WorkingDirectory — is only known at run time. |
|
worker/heartbeat
Package heartbeat implements the worker's periodic liveness signal to sqi-server.
|
Package heartbeat implements the worker's periodic liveness signal to sqi-server. |
|
worker/lease
Package lease implements the worker side of the work-lease protocol: it keeps exactly one outstanding lease request per queue and dispatches each assignment the server returns.
|
Package lease implements the worker side of the work-lease protocol: it keeps exactly one outstanding lease request per queue and dispatches each assignment the server returns. |
|
worker/logstreamer
Package logstreamer implements the log-chunk publisher for sqi-worker.
|
Package logstreamer implements the log-chunk publisher for sqi-worker. |
|
worker/metrics
Package metrics defines and registers all Prometheus metrics for sqi-worker.
|
Package metrics defines and registers all Prometheus metrics for sqi-worker. |
|
worker/natsclient
Package natsclient provides the NATS client factory for sqi-worker.
|
Package natsclient provides the NATS client factory for sqi-worker. |
|
worker/obs
Package obs provides the local observability HTTP server for sqi-worker.
|
Package obs provides the local observability HTTP server for sqi-worker. |
|
worker/openjd
Package openjd implements the OpenJD progress-line interceptor for sqi-worker.
|
Package openjd implements the OpenJD progress-line interceptor for sqi-worker. |
|
worker/pathmap
Package pathmap implements resolved-mode path substitution for sqi-worker.
|
Package pathmap implements resolved-mode path substitution for sqi-worker. |
|
worker/protocol
Package protocol defines the versioned JSON wire protocol used between sqi-server and sqi-worker agents.
|
Package protocol defines the versioned JSON wire protocol used between sqi-server and sqi-worker agents. |
|
worker/registration
Package registration manages worker registration with sqi-server over NATS.
|
Package registration manages worker registration with sqi-server over NATS. |
|
worker/session
Package session manages the lifecycle of OpenJD sessions on the worker.
|
Package session manages the lifecycle of OpenJD sessions on the worker. |
|
worker/staging
Package staging implements the stage_locally path delivery: copying job inputs to worker-local scratch before a task runs and outputs back afterward.
|
Package staging implements the stage_locally path delivery: copying job inputs to worker-local scratch before a task runs and outputs back afterward. |
|
worker/status
Package status implements the typed task-status publisher for sqi-worker.
|
Package status implements the typed task-status publisher for sqi-worker. |
|
ws
Package ws defines the typed WebSocket message envelope shared by the server upgrade handler (internal/api) and the fanout layer.
|
Package ws defines the typed WebSocket message envelope shared by the server upgrade handler (internal/api) and the fanout layer. |
|
Package pkg is the root of the sqi public Go API surface.
|
Package pkg is the root of the sqi public Go API surface. |
|
test
|
|
|
integration/s3sync
command
Command s3sync is a TEST-ONLY sync helper for the S3 staging integration test.
|
Command s3sync is a TEST-ONLY sync helper for the S3 staging integration test. |
|
Package web embeds the compiled web UI bundle into the sqi-server binary so the single binary serves its own front-end with no external asset directory.
|
Package web embeds the compiled web UI bundle into the sqi-server binary so the single binary serves its own front-end with no external asset directory. |