Documentation
¶
Overview ¶
Package filter compiles URL query parameters into sqlb predicates.
It is the second producer of the predicate AST: hand-written Go is the first. Both go through the same builder, so a filter arriving over HTTP is subject to the same compilation, the same bind-parameter discipline and the same query hooks as a query written by hand.
Nothing is filterable, sortable or searchable unless the column declares that capability, and the parser reports the allowed columns when a request asks for one that does not. A request naming an unknown or uncapable column is a 400, never a leak and never a silently ignored parameter.
Grammar:
?status=eq.active operator form
?email=alice@example.com shorthand, equivalent to eq
?age=gte.18&age=lt.65 repeated params conjoin
?tag=in.a,b,c value lists
?deleted_at=isnull null tests
?metadata=hasdoc.{"lang":"de"} jsonb containment
?or=(status.eq.draft,age.lt.18) explicit disjunction
?filter={"op":"and",...} JSON expression tree, for arbitrary nesting
?sort=-created_at,name sorting, "-" for descending
?select=id,name projection
?search=ada fan-out over searchable columns
?page=2&per_page=50 pagination
The same predicates can arrive as a JSON expression tree, in the ?filter= parameter above or — via ParseFilterTree — on their own. It is a second frontend over the same compiler, so a JSON filter is subject to the identical column gate, coercion, bind discipline and MaxFilters budget; the URL grammar is what a query string can spell, not the limit of what the package accepts. A request may carry both, and Parse charges their conditions to one budget.
Index ¶
- Constants
- func Apply[T any](b *sqlb.Builder[T], q *Query) *sqlb.Builder[T]
- func Coerce(s string, t reflect.Type) (any, error)
- func ParseFilterTree(data []byte, opts Options) (sqlb.Pred, error)
- func WriteError(w http.ResponseWriter, err error) bool
- type Error
- type Errors
- type Node
- type Options
- type Query
Examples ¶
Constants ¶
const ( DefaultPageSize = 25 MaxPageSize = 200 MaxFilters = 24 MaxSortTerms = 4 MaxGroupDepth = 3 // MaxListValues bounds one `in`/`nin` list. A list is a single condition // against MaxFilters however long it is, so without this the budget is // bypassed by writing ?id=in.1,2,3,… — one parameter, one predicate, and a // bind parameter per member until the driver's 65535 runs out. MaxListValues = 100 // MaxValueLength bounds one filter value or search term. The pattern // operators pass their operand through unescaped on purpose, so a value is // a lever on how much work a scan does, and a long one is a cheap way to // pull that lever. MaxValueLength = 256 // MaxOffset bounds how far into a result set offset paging may reach. // Offset paging is the one untrusted-input dimension the grammar left // open, and it is the cheapest per-request scan-cost lever it has: // `?page=50000000` asks Postgres to produce and discard ten billion rows // before returning a page of twenty-five. Cursor paging has no such cost, // but it is opt-in per request, so it is not a bound. // // Generous on purpose — a hundred thousand rows is past where offset paging // is a good idea and well past where any human is browsing — because the // point is to have a ceiling, not to pick the right depth for a resource. // Override it per resource like the others. MaxOffset = 100_000 )
Defaults applied when Options leaves a limit unset. They are deliberately conservative: an unbounded list endpoint is a denial-of-service waiting for a client that forgets to paginate.
const ( MaxTreeDepth = 4 MaxTreeNodes = 64 )
Structural limits for a JSON filter tree, the analogue of MaxGroupDepth and the per-request condition budget. A hostile tree is bounded by shape before a single column is resolved, so the cost of rejecting it does not scale with how deep or wide the attacker made it.
const TreeParam = "filter"
TreeParam is the query parameter a JSON filter tree travels in when a request carries the URL grammar and a tree at once (see ParseFilterTree). Parse does not read it — a tree is the REST layer's to compile — but the parameter is reserved so the URL grammar never mistakes it for a column, letting the two filter formats share one request.
Variables ¶
This section is empty.
Functions ¶
func Apply ¶
Apply writes the parsed query onto a builder.
Apply owns the projection. Given ?select it uses those columns; otherwise it projects every non-hidden column. It does not fall back to the builder's default of "all mapped columns", because that would put a Hidden column into a REST response any time a handler forgot to project. A caller wanting a custom projection should apply Where, Order and the limits from the Query fields directly instead.
An expansion is applied as a relation join. Apply does this rather than refusing, and the projection below is why it can: ?select names columns of T, and an expanded relation is not one — it arrives as its own JSON value in a column the scanner recognises, so the row stays exactly as wide as T.
Example ¶
A hidden column stays out of the projection even when the request names no columns, because Apply owns the projection rather than falling back to the builder's default of every mapped column. Forgetting to project cannot leak a password hash.
The trailing `ORDER BY "id"` is Apply making the ordering total. Nothing asked for it, and without it an unsorted list has no stable page boundary — page two may repeat a row from page one or skip one, whether it is reached by offset or by cursor.
q, err := filter.Parse(url.Values{}, exampleOptions())
if err != nil {
panic(err)
}
sql, _, err := filter.Apply(sqlb.Query[Article](), q).SQL()
if err != nil {
panic(err)
}
fmt.Println(sql)
Output: SELECT "id", "title", "body", "status", "views", "author_id", "draft", "published_at", "created_at" FROM "articles" ORDER BY "id" ASC LIMIT 25 OFFSET 0
func Coerce ¶
Coerce converts a URL token into the Go type of its column, so that the driver binds an int as an int rather than as text.
It is exported because a path segment needs the same treatment as a query parameter: `GET /posts/{id}` has to bind a uuid as a uuid, since Postgres will not compare one to text. Parse uses it for every filter value.
func ParseFilterTree ¶
ParseFilterTree decodes a standalone JSON filter tree and compiles it into a single predicate, gated by opts.Model exactly as the URL frontend is. Use it when the tree arrives on its own — a POST body, say — rather than in a query string: a tree in `?filter=` is compiled by Parse, which shares its MaxFilters budget with the URL filters in the same request. On its own the tree has the whole budget to itself.
Every problem is collected rather than reported one at a time, so a malformed tree takes one round trip to fix (ADR-0011). The error is a filter.Errors, so WriteError renders it as the same 400 the URL frontend produces.
func WriteError ¶
func WriteError(w http.ResponseWriter, err error) bool
WriteError writes err as a JSON problem response if it is a parse failure, and reports whether it did. It is the whole error path of a list handler.
Types ¶
type Error ¶
type Error struct {
Param string `json:"param"`
Value string `json:"value,omitempty"`
Reason string `json:"reason"`
Allowed []string `json:"allowed,omitempty"`
}
Error is one rejected query parameter.
It carries the allowed alternatives where there are any, because the caller most likely to read it is a program assembling requests against a schema it only partly knows. "column is not sortable" is a dead end; the same message plus the sortable columns is a fix.
type Errors ¶
type Errors []*Error
Errors is the set of problems found in one request. Parsing collects them all rather than stopping at the first, so a malformed request needs one round trip to fix rather than one per mistake.
func AsErrors ¶
AsErrors extracts parse errors from err, unwrapping as it goes.
Prefer it to a type assertion. Parse returns Errors directly today, but a hook, a middleware or a caller adding context will wrap it, and `err.(filter.Errors)` panics the moment that happens:
if errs, ok := filter.AsErrors(err); ok {
errs.WriteHTTP(w)
return
}
func (Errors) StatusCode ¶
StatusCode is 400: every parse failure is a malformed request.
func (Errors) WriteHTTP ¶
func (e Errors) WriteHTTP(w http.ResponseWriter)
WriteHTTP writes the errors as a JSON problem response.
type Node ¶
type Node struct {
Op string `json:"op"`
Children []Node `json:"children,omitempty"`
Field string `json:"field,omitempty"`
Value any `json:"value,omitempty"`
}
Node is one node of a JSON filter tree: a logical group (Op is "and"/"or", with Children) or a leaf condition (Op is a comparison, with Field and Value). Exactly one shape per node; validateTree enforces it.
type Options ¶
type Options struct {
// Model supplies the columns and their capabilities. Required.
Model *sqlb.Model
DefaultPageSize int
// MaxFilters bounds the number of leaf conditions a request may ask for,
// counting the ones inside `or=`/`and=` groups. Counting top-level
// parameters instead would leave the budget open to a single group holding
// as many conditions as the client cared to write.
MaxFilters int
MaxPageSize int
MaxSortTerms int
// MaxListValues bounds one `in`/`nin` list; MaxValueLength bounds one
// filter value or search term.
MaxListValues int
MaxValueLength int
// MaxOffset bounds how deep ?page= and ?offset= may reach. A request past
// it is refused with a message pointing at ?cursor=, which has no such
// cost.
MaxOffset int
// Expandable lists the relation names ?expand may name. Parsing validates
// against it and Apply performs the join, so a parsed ?expand is never
// silently dropped: a name that is not here is a 400 listing the ones that
// are.
//
// The rest package validates these against the model at startup, so a
// relation that cannot be expanded is a mounting error rather than a
// request-time surprise.
Expandable []string
// Computed lists the computed columns this resource is willing to pay for.
// Empty means none, and none is the default on purpose.
//
// A computed column is declared on the model, which is shared, and wanted
// by one screen. Projecting every declared one attached a correlated
// subquery per column to every read of the model, and a column carrying a
// Needs bind made unrelated reads fail outright (#92). So declaring stays
// global — the expression, its type, its binds — and *selecting* is per
// resource, beside the other things a mount already decides.
//
// A column not listed here is not reachable from this resource at all: not
// projected, not filterable, not sortable, and not nameable in ?select.
// Being unreachable rather than merely unprojected is what keeps the cost
// opt-in, since a filter on a correlated subquery costs what the projection
// would have.
Computed []string
// Columns narrows this resource to the columns it names. Empty means every
// column the model has, which is the default and what almost every resource
// wants.
//
// It is the same per-resource reachability Computed has, generalised to
// stored columns, and it is here because a model is shared in the other
// direction too: one table, two surfaces, and the privileged one is the
// reason the sensitive column exists (#148). A public catalogue and an
// admin panel over the same products differ in which columns each may see,
// and Hidden cannot say that — Hidden is a property of the model, and there
// is one model.
//
// A column not listed is not reachable from this resource at all: not
// projected, not filterable, not sortable, not nameable in ?select, not
// searched by ?search, and not named in the list a rejection offers. That
// last one matters — a narrowed resource that advertised the column it is
// about to refuse would leak the schema it was narrowed to hide.
//
// Names are column names, as Computed's are. The rest package checks them
// against the model at startup, where a typo is a resource missing a column
// rather than a request-time surprise.
Columns []string
// DisableSearch rejects ?search even when columns are searchable.
DisableSearch bool
}
Options configures parsing for one resource.
type Query ¶
type Query struct {
Where []sqlb.Pred
Order []sqlb.Order
Select []string
Expand []string
Search string
Page int
PageSize int
Limit int
Offset int
// Computed names the computed columns this resource selects, copied from
// Options so that Apply projects exactly what parsing validated against.
Computed []string
// Columns is the resource's surface, copied from Options for the same
// reason: the default projection is built in Apply, and a narrowed resource
// whose parser refused a column while its projection selected it anyway
// would read the value out of the database on every request and drop it on
// the way out — which is a narrowing in the response only, and not the one
// Options.Columns describes.
Columns []string
// Cursor is the keyset position `?cursor=` asked to resume from, empty for
// the first page. It is the alternative to Page and Offset rather than an
// addition to them: a request carrying both is refused, since the two
// answer the same question with different answers.
Cursor sqlb.Cursor
}
Query is a parsed request: predicates, ordering, projection and pagination, all already validated against the model.
func Parse ¶
Parse compiles URL query parameters into a Query.
Every problem found is reported, not just the first, so a caller fixing a request sees the whole list at once.
Example ¶
A parsed request compiles into the same predicates hand-written Go produces, so it goes through the same builder, the same bind-parameter discipline and the same query hooks. This is the whole design: one AST, two producers.
values, err := url.ParseQuery("status=eq.published&views=gte.100&sort=-views&per_page=10")
if err != nil {
panic(err)
}
q, err := filter.Parse(values, exampleOptions())
if err != nil {
panic(err)
}
sql, args, err := filter.Apply(sqlb.Query[Article](), q).SQL()
if err != nil {
panic(err)
}
fmt.Println(sql)
fmt.Println(args...)
Output: SELECT "id", "title", "body", "status", "views", "author_id", "draft", "published_at", "created_at" FROM "articles" WHERE ("status" = $1) AND ("views" >= $2) ORDER BY "views" DESC, "id" DESC LIMIT 10 OFFSET 0 published 100
Example (Cursor) ¶
Cursor pagination reads the URL the same way, and the boundary it produces is a predicate like any other — so it goes through the same builder, the same bind parameters and the same hooks as everything else.
The cursor names the position of the last row of the previous page. Because `?sort=-views` is not a total order on its own, Apply has already appended the primary key, and the cursor carries both terms.
Both are descending and neither column is nullable, so the boundary compiles to a row comparison rather than the lexicographic OR-chain. That is the form Postgres can answer with a single index seek on `(views DESC, id DESC)`, which is the entire reason to page this way instead of with OFFSET.
values, err := url.ParseQuery(
"sort=-views&per_page=10&cursor=" +
"eyJrIjpbeyJjIjoidmlld3MiLCJkIjp0cnVlLCJ2IjoxMDB9LHsiYyI6ImlkIiwiZCI6dHJ1ZSwidiI6ImE3In1dfQ")
if err != nil {
panic(err)
}
q, err := filter.Parse(values, exampleOptions())
if err != nil {
panic(err)
}
sql, args, err := filter.Apply(sqlb.Query[Article](), q).SQL()
if err != nil {
panic(err)
}
fmt.Println(sql[strings.Index(sql, "WHERE"):])
fmt.Println(args...)
Output: WHERE ("views", "id") < ($1, $2) ORDER BY "views" DESC, "id" DESC LIMIT 10 OFFSET 0 100 a7
Example (CursorConflict) ¶
A cursor and a page number are two answers to "where does this page start", so a request carrying both is refused rather than having one of them silently ignored.
values, err := url.ParseQuery("cursor=abc&page=3")
if err != nil {
panic(err)
}
if _, err := filter.Parse(values, exampleOptions()); err != nil {
fmt.Println(err)
}
Output: filter: cursor=abc: a cursor and page both say where the page starts; send one or the other
Example (Rejection) ¶
A column that does not declare a capability cannot be reached through it, and the rejection reports what would have been accepted. Every problem in the request is reported at once rather than one per round trip.
values, err := url.ParseQuery("sort=body&internal_note=eq.hunter2")
if err != nil {
panic(err)
}
_, err = filter.Parse(values, exampleOptions())
// AsErrors unwraps to the structured form, which carries the allowed
// alternatives — the difference between a dead end and a fix. Prefer it to a
// type assertion, which panics the moment a caller wraps the error.
errs, ok := filter.AsErrors(err)
if !ok {
panic("expected parse errors")
}
for _, e := range errs {
fmt.Printf("%s: %s\n", e.Param, e.Reason)
fmt.Println(" allowed:", strings.Join(e.Allowed, ", "))
}
Output: internal_note: unknown parameter allowed: id, title, body, status, views, author_id, draft, published_at sort: column is not sortable allowed: title, views, published_at, created_at
Example (Search) ¶
Search fans out over every searchable column as a disjunction, and the user's input is escaped, so typing "50%" searches for that literal string rather than matching every row.
values, err := url.ParseQuery("search=50%25&select=id,title")
if err != nil {
panic(err)
}
q, err := filter.Parse(values, exampleOptions())
if err != nil {
panic(err)
}
sql, args, err := filter.Apply(sqlb.Query[Article](), q).SQL()
if err != nil {
panic(err)
}
fmt.Println(sql)
fmt.Println(args...)
Output: SELECT "id", "title" FROM "articles" WHERE ("title" ILIKE $1) OR ("body" ILIKE $2) ORDER BY "id" ASC LIMIT 25 OFFSET 0 %50\%% %50\%%