docflow

command
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 26, 2026 License: MIT Imports: 5 Imported by: 0

README

BDF Docflow

A document workflow engine: a software-defined document network. Documents enter at virtual printers (CUPS server nodes, the IPP entries) and at the API's uploaders (a document, or a template and the data to fill it with), flow through the nodes an administrator composes — queues that fan out, classes that send to one member in turn, inboxes that hold a job until a person or an application sends it on — and end on display boards (the devices enrolled at a location, such as the tablets of a warehouse), on printers, and in archives that keep a copy nobody changes. Paper is one destination among others. The topology is laid out on the admin page, as the printers and classes of a CUPS server are, with edges drawn between nodes (see Topology below).

The engine is docflow, the web application docflow/web on Popcorn Web; this command serves it with every input and output. It keeps no state in the process: the records and the documents are in a yuruidb store, so the same binary runs as one process on a site's server or as a function of AWS Lambda.

Run it

From the repository's root, either

pw dev

which regenerates the pages and the public tree, rebuilds and restarts on every change, and signs people in at pw's development OpenID Provider ([dev.idp] in popcornweb.toml): it gives Docflow its issuer and a client of the run (AUTH_OIDC_*), so nothing about the IdP is typed. Signing in is picking one of the people of devidp.toml, no password checked: Ada Admin (docflow-admins, the administrators by config.dev.toml), Mina Member (warehouse-a) and Otto Outsider (no groups). Editing the roster takes effect without a restart; TestDevRoster checks that its people still hit the rules. Or

go run ./cmd/docflow

with an IdP of your own in .env.local (copy .env.example; git ignores it). Neither needs a secret in development.

pw build compiles it, the public tree built and embedded, to bin/docflow ([build] in popcornweb.toml: docflow at the root is the package).

In development (no APP_ENV), it reads config.dev.toml at the repository's root (the [oidc] rules above, plain-http cookies), keeps its store in docflow.db and makes a random secret. Configure it with config.{APP_ENV}.toml (the [docflow] table; docflow --generate-config toml prints every key) or with variables:

Variable
DOCFLOW_RECORDS store of the records: bbolt:path, dynamodb://table?region=…, firestore://project/database, s3://bucket?region=…&endpoint=…, r2://bucket?endpoint=…, gcs://bucket
DOCFLOW_DOCUMENTS store of the slips' documents (bbolt, s3, r2, gcs); empty for the records' store
DOCFLOW_SECRET signs the devices' sessions, the apps' tokens and the pages' sessions: 32 bytes or more, the same for every instance
DOCFLOW_PREVIOUS_SECRETS retired secrets that still open what was signed with them, while a rotation goes on (set the new secret, keep the old one here for a day, then drop it)
DOCFLOW_ADMIN_TOKEN bearer token of the API (/api), for scripts (people sign in)
DOCFLOW_TICK_TOKEN bearer token a scheduler calls POST /api/tick with
DOCFLOW_SUBMIT_TOKENS name:secret pairs that may send jobs (IPP Basic, or bearer name:secret); none lets anyone send
DOCFLOW_SERVERLESS refuse a store one process keeps, tick only when called, and open no ports (CUPS servers are reached at /ipp/print/<name>)
DOCFLOW_SIGN_KEY_FILE, DOCFLOW_SIGN_CERT_FILE the private key (PEM) that sign nodes sign with, and its certificate followed by its chain
DOCFLOW_SIGN_TSA URL of a time-stamp authority (RFC 3161) that stamps the signatures of sign nodes
DOCFLOW_TICK_INTERVAL how often a process ticks (default 30s)
DOCFLOW_JOB_KEEP how long a job that ended is kept, with its document once nothing shows it (default 336h, 14 days)
DOCFLOW_ON_ERROR nodes a job goes to when a send or a step fails for good at a node without on_error edges (a notify node)
DOCFLOW_CREDENTIAL_DIR directory of the credentials nodes name, a file each; DOCFLOW_CRED_<name> comes first
DOCFLOW_DEVICE_URL origin the devices of displays open (an enrollment's URL and QR code), e.g. https://docflow.example.com; empty: this server as the admin reached it, at this computer's address on the network for localhost, on DOCFLOW_HTTPS_PORT when there is one
DOCFLOW_HTTPS_PORT, DOCFLOW_TLS_CERT_FILE, DOCFLOW_TLS_KEY_FILE an HTTPS listener serving what the HTTP one does, for devices on the network (a device makes its key with WebCrypto, which needs HTTPS off localhost), and its certificate and key (PEM); in development, without them, one signed by itself and kept in the user's cache (config.dev.toml listens on 8443)

Sign-in

Every page but /tablet is for a person signed in at an OIDC IdP (authorization code with PKCE, through popcornweb's plugin/auth). Anyone the IdP signs in gets in, with no registration here, and sees their own jobs and their groups' jobs, and the nodes they may print to; the admins see everything and get /admin and /audit, the others get 403 there. The IdP is popcornweb's [auth.oidc]; who gets in and who is an admin is [oidc]:

Variable
AUTH_OIDC_ISSUER, AUTH_OIDC_CLIENT_ID, AUTH_OIDC_CLIENT_SECRET the IdP and this client (required outside dev)
AUTH_OIDC_REDIRECT_URL https://<host>/auth/callback, as registered at the IdP (outside dev)
OIDC_GROUPS_CLAIM claim of the groups: groups, roles, or a dotted path such as Keycloak's realm_access.roles (it must be in the ID token)
OIDC_GROUPS_PREFIX take only the groups of this prefix
OIDC_ADMINS admins: issuer#subject, or an email the IdP verified
OIDC_ADMIN_GROUPS groups whose members are admins (one admin rule is required outside dev)
OIDC_TOPOLOGY_ADMINS, OIDC_TOPOLOGY_ADMIN_GROUPS the role topology: the admin pages and the API of the topology (nodes, templates, keys, apps, devices, every inbox, deliveries), short of an admin
OIDC_AUDITORS, OIDC_AUDITOR_GROUPS the role audit: /audit, GET /api/audit, and every job (reading); an admin has both roles
OIDC_ALLOWED_ISSUERS, OIDC_ALLOWED_DOMAINS, OIDC_ALLOWED_GROUPS narrow who gets in (email domain or Google's hd; admins always get in)

The rest of [auth] and [session] is Docflow's (web.ConfigureAuth): the state of a login on its way through the IdP is kept in the records' store, so that any instance finishes it, and the session is a sealed cookie (session.keyring.secret, or made of DOCFLOW_SECRET). The groups are copied at each login; who is an admin is decided on each request. A sign-in after a sign-out asks the IdP to let the person choose who they are: popcornweb (0.5.14 or later) marks /auth/login to carry prompt=select_account login until a sign-in completes, and sends only requests for a page to the sign-in (the icons Safari asks for on its own, /favicon.ico and /apple-touch-icon*.png, get 401), so the IdP does not sign the same person in again without a word.

Everything an application calls is the API of the one server, /api: the nodes reached from outside are under their kind and their name, /api/<kind>/<name> — an uploader takes documents at POST /api/upload/<name>, an inbox is worked at /api/inbox/<name>, a downloader is read at /api/download/<name>, an S3 trigger takes an event's reference at POST /api/s3-trigger/<name>. No node but a CUPS server (IPP: AirPrint, IPP Everywhere, Mopria) has a port of its own.

Apps are the applications that send with credentials of their own (a business system, a WMS): an administrator registers one on /admin/apps (PUT /api/apps/<id> with its scopes; the secret comes back once, POST /api/apps/<id>/secret rotates it), and an uploader, an inbox or a downloader lists the apps it takes (apps). The app takes a token at POST /oauth/token (the client credentials grant: -u id:secret -d grant_type=client_credentials, ten minutes) and sends with it as a bearer to /api/upload/<name>, or prints over IPP with the same id and secret as Basic; it sees and, with the scope jobs.cancel, cancels its own jobs. The scopes are print, print.template, jobs.cancel, inbox (works the inboxes that list it: lists what they hold, their thumbnails, pages and documents, releases and discards: help/en/inboxes.md), download (reads the downloaders that list it: help/en/downloads.md), tick (a scheduler's) and admin (a script's: the API as an administrator). An app's jobs are its group's when the destination gives them none. Disabling an app refuses its credentials and the tokens out. GET /api/openapi.json is the OpenAPI 3.1 document of what the topology publishes, made of it as it is: the uploaders (the formats they take, the JSON Schema of the data their templates fill, the metadata the nodes after them look at), the inboxes (the nodes they release to) and the downloaders; an app reads the ones that list it, ?node= narrows it (help/en/api.md).

A public client is a frontend of one's own in people's browsers (a customer portal, a single-page app on another origin): registered with "public": true and its redirect_uris (https, or http on localhost), it has no secret, and gets a token of the person using it with the authorization code grant and PKCE (S256): the frontend sends the person to GET /oauth/authorize?response_type=code&client_id=…&redirect_uri=…&code_challenge=…&code_challenge_method=S256&state=…, where they sign in at the IdP as for the pages (once a session; the registration is the consent, no screen asks), and comes back with a code, which it exchanges at POST /oauth/token (grant_type=authorization_code, client_id, code, code_verifier, redirect_uri) for a token of an hour. The token is the person, with the client's scopes: it opens the downloaders, the inboxes the person works and /api/me, nothing else of the API. A page of the client's origin may call the token endpoint, the downloaders and the inboxes (CORS).

Every node (a display, a printer, a queue, a class …) has a scope and an owner ("scope":"group", "owner":"warehouse-a", a value of the groups claim; personal and a person's email; shared, the default): who may print to it from the pages, and the group whose members see the jobs sent to it (of a display with none, its fallback printer's). A job records who sent it as verified (submitted_by: the person signed in, the token, the key of an uploader, or anonymous) apart from the name the sender gave (claimed_user, shown only). The sender and the admins cancel a job; its group only sees it.

pw dev's provider is the local IdP. The tests also use print/fakeidp (fakeidp.New(...).Listen("127.0.0.1:9781") serves it by hand), for what the roster cannot say: a nested groups claim (realm_access.roles), an email the IdP did not verify, a token tampered with.

AWS credentials come from the environment (AWS_ACCESS_KEY_ID …), web identity, the container's or the instance's role.

Use it

The pages are in English and Japanese: in the language a person chose with the switch in the bar (English, 日本語; the top page and a device's board have it too), which pw keeps in a cookie for a year, else in the browser's (Accept-Language), else in English. The templates' texts are the catalog messages/pages.yaml, which pw generate compiles and checks against every {t id} of the templates; the scripts write theirs in English in _("…") (docflow/web/public/i18n.js), and public/ja.js has their Japanese, loaded on a Japanese page only (assets_test.go checks that every text has its translation, and every translation its text). The API is not translated: its errors, shown in the pages' toasts, are English.

The manual is in the application: Help in the bar of every page, at /help, in English and Japanese, opening in the language of the pages. Its chapters are the Markdown files of help/en and help/ja, embedded by package help and rendered when served (goldmark, no raw HTML): the pages, the topology and its editor, every kind of node and its settings, what flows along the edges and the metadata, templates and data, the entries (uploaders, CUPS server nodes, apps, IPP), display boards and devices, inboxes (and their API) and deliveries, downloaders, sign-in, roles and audit, the API, and the configuration. A link between chapters is written as jobs.md, so that the files read on GitHub too; every chapter has the same section ids in every language, so that the pages link to the manual without a language (the editor's sidebar links /help/nodes#printer), and help_test.go checks the links, the ids, and that every kind of node has its section. Changing the manual is editing the Markdown: no generation.

In short: a person signs in at the IdP (see Sign-in) and sees /jobs, their jobs and their groups' and a form that prints to a source (an entry: an uploader, a CUPS server) they may send to, and /topology, the network with the activity of their jobs. The role topology prints to any node from the form, to try the topology out, and gets the administrator's pages under /admin (the editor of the topology, apps, jobs, deliveries), /inboxes (every inbox; the members of an inbox's group get it for theirs) and the role audit gets /audit. Every node is PUT /api/nodes/<name> with its kind and the kind's settings — upload, cups, queue, class, filter, assert, branch, template, convert, sign, inbox, display, printer, cups-out, archive, sink, download (help/en/nodes.md) — and POST /api/nodes/<name>/rename renames one, its edges following; IPP enters at cups nodes alone (their port's /printers/<name>, and /ipp/print/<name>); a display's tablets are enrolled with a one-time code from the display's sidebar (help/en/displays.md); an inbox holds a job until someone releases it, on /inboxes or through /api/inbox/<name> (the role topology, the members of the inbox's group, the apps it lists: help/en/inboxes.md); a downloader after an archive or an inbox lets backends and people's frontends fetch the documents at /api/download/<name> (help/en/downloads.md); and the API is help/en/api.md. For example: an audit copy, a site's printers in turn, both warehouses at once, a review before printing, and an application that prints invoices from data:

The kinds are registered with the engine (docflow), and GET /api/kinds lists them with their roles and what they take. The core has the routing, transforming and keeping kinds. Inputs and outputs are adapters under docflow/io: printer, cups (a CUPS server computers print to) and cups-out (a CUPS server's queue jobs are printed to) are docflow/io/ipp; smtp is docflow/io/smtp, s3 and s3-trigger docflow/io/s3, http docflow/io/http, notify docflow/io/notify, and render docflow/render. This application imports them all; the manual (help/en/nodes.md) tells their settings, on_error and the credentials they name.

curl -X PUT -H "Authorization: Bearer $TOKEN" localhost:8080/api/nodes/audit -d '{"kind":"archive","store":"s3://audit?region=ap-northeast-1"}'
curl -X PUT -H "Authorization: Bearer $TOKEN" localhost:8080/api/nodes/invoices -d '{"kind":"queue","to":["audit"]}'
curl -X PUT -H "Authorization: Bearer $TOKEN" localhost:8080/api/nodes/tokyo -d '{"kind":"class","members":["mfp-3f","mfp-4f"]}'
curl -X PUT -H "Authorization: Bearer $TOKEN" localhost:8080/api/nodes/both -d '{"kind":"queue","to":["wh-a","wh-b"]}'
curl -X PUT -H "Authorization: Bearer $TOKEN" localhost:8080/api/nodes/review -d '{"kind":"inbox","to":["wh-a","tokyo"]}'
curl -X PUT -H "Authorization: Bearer $TOKEN" localhost:8080/api/nodes/erp -d '{"kind":"upload","to":["invoices"],"formats":["pdf"]}'
# → {"name":"erp", …, "key":"bdfk_…"}: the application sends with that key
curl -X POST -H "Authorization: Bearer bdfk_…" --data-binary @statement.pdf "localhost:8080/api/upload/erp?name=statement&meta.dept=sales"
# the review inbox: what it holds, and a release (an app with the scope inbox it lists, or the admin token)
curl -H "Authorization: Bearer $TOKEN" localhost:8080/api/inbox/review
curl -X POST -H "Authorization: Bearer $TOKEN" localhost:8080/api/inbox/review/1/release -d '{"to":["wh-a"],"worker":"Sato"}'
# data for a template: a data uploader, and a template node that fills the invoice
curl -X PUT -H "Authorization: Bearer $TOKEN" localhost:8080/api/nodes/forms -d '{"kind":"template","template":"invoice","to":["invoices"]}'
curl -X PUT -H "Authorization: Bearer $TOKEN" "localhost:8080/api/nodes/forms/templates/invoice?name=invoice.xlsx" --data-binary @invoice.xlsx
curl -X PUT -H "Authorization: Bearer $TOKEN" localhost:8080/api/nodes/erp-data -d '{"kind":"upload","accept":"data","to":["forms"]}'
curl -X POST -H "Authorization: Bearer bdfk_…" -H "Content-Type: application/json" "localhost:8080/api/upload/erp-data?name=INV-1" -d '{"customer":"ACME","total":12000}'

On AWS Lambda

From the repository's root:

pw build --target=lambda
docker build .pw/build/lambda/nethttp

builds the function (linux/amd64) and its container image with the AWS Lambda Web Adapter. No config file is shipped: configure the function with variables, DOCFLOW_SERVERLESS=true among them. Keep the records in DynamoDB (a table with the string keys pk and sk) and the documents in S3, and let EventBridge Scheduler call POST /api/tick every minute. To print on a site's printers, put the function in a VPC that reaches them (Direct Connect or a VPN).

A program of its own

The web application is the package docflow/web; this command is web.RegisterConfig() and web.Main(web.Options{}) with every input and output imported (docflow/io/all). A program of its own composes the same application with kinds of node of its own (registered with docflow.RegisterKind, with the fields of their settings, so that the editor lays them out), settings of its own (pw.RegisterConfig), and its own pages and API (web.Options.Routes), and is built with pw build as this one is: examples/docflow-custom is one, with a folder exit. docflow gen-project <dir> writes such a project to start from (the example itself, with the project's name and module path: --module example.com/acme/docflow, --name, and --bdf ../bdf for a checkout of bdf instead of the released module); then go mod tidy and pw dev in it.

Generated files

pw reads popcornweb.toml at the repository's root (it wants it beside go.mod): [generate] names only docflow/web. After changing the pages (docflow/web/pages.pw.html), their texts (docflow/webmessages/pages.yaml) or docflow/web/config.go (the manual in docflow/web/help/ and the scripts in docflow/web/public need nothing: they are embedded as they are), run from the repository's root:

pw generate

and commit what it wrote: the *_pw_gen.go files (the catalog's is docflow/webmessages/messages_pw_gen.go). The module's build runs no generator, so a plain go build embeds what is committed. The scripts and the style are not pw's public tree: docflow/web embeds public and serves it at /_docflow/ itself, so that a program composing it keeps /public for its own assets.

Style

The pages use Tailwind CSS and daisyUI, built into docflow/web/public/app.css (committed, and embedded in the binary: no CDN); the manual's HTML is styled by Tailwind's typography plugin (prose), which the standalone CLI bundles. No Node or npm: the standalone tailwindcss CLI loads daisyUI's release files in docflow/web/tailwind/ (versions and hashes in tailwind/VERSIONS). After changing the pages' markup or classes, with that CLI on the PATH (brew install tailwindcss):

go generate ./docflow/web
pw generate

CI builds both again and fails on a difference.

Documentation

Overview

Command docflow is BDF Docflow, a document workflow engine on Popcorn Web: documents and data enter at uploaders and virtual printers, flow through a topology of nodes, and end on printers, on the tablets of a location (a warehouse), in archives and in downloaders. The engine is package docflow, the web application docflow/web: this is the one that serves every input and output of docflow/io. A program of its own composes the same application with kinds of node, settings and pages of its own (docflow/web.Options; examples/docflow-custom).

People sign in to its pages at an OIDC IdP (popcornweb's plugin/auth, set by web.ConfigureAuth), and see the jobs of their scope; the devices of /tablet keep their own sign-in.

It runs as a process (a container, a server of a site) or as a function (AWS Lambda through the Lambda Web Adapter): with docflow.serverless, the records must be in a store several instances share, and a scheduler calls POST /api/tick instead of the timer.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL