rules

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 2, 2026 License: AGPL-3.0 Imports: 11 Imported by: 0

Documentation

Overview

Package rules is KnightLoader's rule engine. It is one engine used twice: as the Packagizer, which rewrites a link's package, folder, name and download options before it is queued, and as the LinkGrabber filter, which decides whether a link is taken at all. The two differ only in what a matching rule does, so they share the types and the matching code.

A Set is compiled once and the resulting Matcher is then asked about every candidate link. Compiling once is what keeps the per-link cost bounded — a paste can be several thousand links — and it is also the only moment at which a rule the user got wrong can still be reported to them. Compile therefore never fails: it returns a working Matcher plus everything it could not use, so a broken rule costs the user that rule and nothing else.

The filter half exists because JDownloader's eats links in silence: something is filtered, nothing says what or why, and the link is simply not there. Check never returns a bare "no" — a rejection always names the rule that made it and carries a reason.

Variables

Every string an action sets is a template. It goes through internal/pathvars first, which resolves <jd:packagename>, <jd:hoster>, <jd:filename> and the date placeholders, and then through this package, which resolves the ones the Packagizer adds on top:

<jd:orgfilename>            the link's file name as it arrived
<jd:orgfilenamewithoutext>  the same, with the extension cut off
<jd:orgfiletype>            the extension without its dot, empty when there is none
<jd:source:N>               the Nth path segment of the source page's URL, counting from 1
<jd:match:FIELD:N>          capture group N of this rule's "matches" pattern on FIELD
<jd:append>                 empty the first time a value is produced, "_2", "_3" ... after

Unknown or out-of-range placeholders are left in the text rather than blanked, which is what internal/pathvars does and for the same reason: a typo that is visible in the folder name can be fixed, and one that quietly collapsed to nothing cannot be found at all.

<jd:source:N> does not mean what JDownloader means by it

In a JDownloader Packagizer template, <jd:source:1> is capture group 1 of the regular expression the rule tests the source URL with. Here it is the first path segment of that URL, and it has been that since the package was written. The two agree often enough to be dangerous and disagree silently: a template copied out of a JD config builds a different folder, with no error and nothing to notice until the files are already somewhere else.

The path-segment meaning is kept, because it is what is already stored in people's rules and because it is the only one of the two that works in a rule with no regular expression in it at all. JD's meaning is spelled <jd:match:FIELD:N>, which says out loud which pattern it reads — JD's version can only ever mean the source, and ours reads a capture group from any field the rule matched with. Compile refuses a rule whose <jd:match:...> names a field the rule has no "matches" condition on, so the JD habit of reaching for a group that is not there fails loudly at the moment the rule is saved rather than quietly in a folder name.

Every variable resolves against the link as it arrived, so rules do not chain onto each other's output: <jd:filename> in the fourth rule is still the name the hoster gave, not whatever the second rule renamed it to. That is why <jd:filename> and <jd:orgfilename> are the same value here. It costs a little expressiveness and buys the property that matters in a list a user edits by hand — what a rule does can be read off that rule alone.

Index

Constants

View Source
const (
	PriorityMin = -3
	PriorityMax = 3
	MaxChunks   = 16
)

Bounds a rule may not exceed. Priority spans the seven values the interface offers (app.Priorities, JDownloader's set) and is the one bound both sides clamp to: narrower here, a rule could not reach the highest and lowest at all while the menu still showed them; wider, a rule could hand a task a priority no control in the interface is able to undo. MaxChunks is a guard rail rather than tuning: connections beyond a handful buy nothing on a hoster that rate-limits per file and are a reliable way to get an account flagged.

Variables

This section is empty.

Functions

func CategoryOf

func CategoryOf(pattern string) string

CategoryOf names the category a stored condition value came from, or "" when the pattern was written or edited by hand. It is what lets the editor reopen a rule showing the chip it was created with instead of a wall of extensions — and, just as importantly, show the raw pattern once somebody has changed it, rather than a chip that no longer says what the rule does.

func CategoryPattern

func CategoryPattern(id string) (string, bool)

CategoryPattern is one category's condition value. The second result is false for an unknown id, so a client asking for a category this build does not have gets nothing rather than a pattern that matches everything.

func Compile

func Compile(s Set) (*Matcher, []Problem)

Compile validates a rule set once and returns a Matcher plus every rule it had to leave out. It never returns nil: a caller that ignores the problems still gets a defined Matcher — one that does nothing — instead of a nil dereference on the first link.

A rule with any problem at all is dropped whole rather than partly applied. A rule missing one of its conditions is not a stricter rule, it is a rule that does something the user never asked for, and for a filter that means links disappearing on a typo. A set switched off compiles to that same empty Matcher and reports nothing, exactly as a switched-off rule does. Off means off: leaving the problems standing would keep a page badge lit about rules that are not being applied, and there would then be no way to silence a rule set short of deleting it. The editor gets its problems from Preview, which validates the set as written whether or not it is switched on, so a set can still be fixed while it is off.

func FileSegment

func FileSegment(value string) string

FileSegment cuts a value down to the one path segment a file name is allowed to be - the very cut Apply gives Action.Filename, exported so a rename typed into the interface and a rename written by a rule cannot disagree about what a name is.

Two cuts drift, and the one that drifts is the one that lets "../../etc/x" through. A value that sanitises away entirely comes back as "file" rather than empty, for the same reason it does inside a template: a rename has to end in a name, and the caller can see this one and correct it.

Types

type Action

type Action struct {
	PackageName string `json:"packageName,omitempty"`
	DownloadDir string `json:"downloadDir,omitempty"`
	Filename    string `json:"filename,omitempty"`
	Comment     string `json:"comment,omitempty"`
	Priority    *int   `json:"priority,omitempty"`
	AutoExtract *bool  `json:"autoExtract,omitempty"`
	Chunks      *int   `json:"chunks,omitempty"`

	// Reject drops the link instead of taking it. Reason is shown to the user
	// alongside the rule's name; when it is empty Check writes one, because a
	// rejection nobody can explain is the behaviour this package exists to
	// avoid.
	Reject bool   `json:"reject,omitempty"`
	Reason string `json:"reason,omitempty"`
}

Action is what a matching rule does. The Packagizer flavour uses the fields above Reject, the filter flavour uses Reject and Reason; a Set is free to use either, and each flavour ignores the other's fields.

Every string field is a template and is expanded through internal/pathvars plus the Packagizer-only variables listed in the package documentation. An empty string means "leave this alone", never "clear it": a later rule that sets only the folder must not wipe the package name an earlier one chose. The three optional values are pointers for the same reason — priority 0, zero chunks and auto-extract off are all real settings, so "unset" needs to be something other than the zero value.

DownloadDir is the only field allowed to spell out path levels. Filename is cut back to a single segment once it has been expanded, because a file name containing a separator is not a name, it is a way out of the folder the caller picked.

type ActionGrammar

type ActionGrammar struct {
	// ID is the JSON field name on Action, so the form addresses the same key the
	// engine reads and a typo is a compile error over here rather than a control
	// that quietly does nothing.
	ID string `json:"id"`
	// Kind is how it is edited: template, int, bool or reject.
	Kind string `json:"kind"`
	// Flavour is which of the two engines honours it: "packagizer", "filter", or
	// empty for both.
	Flavour string `json:"flavour,omitempty"`
	// Min and Max bound an int action, so the form clamps to the same numbers
	// actionProblems refuses outside of.
	Min *int `json:"min,omitempty"`
	Max *int `json:"max,omitempty"`
}

ActionGrammar is one thing a matching rule can do.

type Candidate

type Candidate struct {
	Filename string
	URL      string
	Hoster   string
	Source   string
	Filetype string
	Filesize int64
	Package  string
	Added    time.Time
}

Candidate is the link a rule set is asked about. Hoster and Filetype are derived from URL and Filename when left empty, so the ordinary caller fills in what it has and nothing more.

Added is what the date variables format. It is an argument rather than a clock read inside the package, so a rule set's output is a pure function of its input and a test never has to wait for a second to pass.

type Category

type Category struct {
	ID string `json:"id"`
	// Extensions is what the category covers, without dots, lower case, in the
	// order the editor should list them. It is shown so somebody can check
	// whether their format is in there before trusting the chip.
	Extensions []string `json:"extensions"`
	// Pattern is the exact value the editor writes into Condition.Value with
	// OpMatches on FieldFiletype. Generated here rather than in the interface so
	// that recognising a stored rule as a category is a string comparison against
	// this same field, and cannot drift into "nearly the same pattern".
	Pattern string `json:"pattern"`
}

Category is one named group of extensions, together with the condition value it stands for.

func Categories

func Categories() []Category

Categories is every category, in menu order.

type Condition

type Condition struct {
	Field Field  `json:"field"`
	Op    Op     `json:"op"`
	Value string `json:"value,omitempty"`
	Min   int64  `json:"min,omitempty"`
	// Max of zero means "no upper bound". A half-filled range is the normal
	// shape of "at least 500 MB", and reading the empty box as zero would give
	// the user a rule that can never match anything.
	Max int64 `json:"max,omitempty"`
}

Condition is one test against a candidate link. Value carries the text for the string operators and the byte count for OpEquals/OpEqualsNot on a file size; Min and Max carry the range for OpBetween. Sizes are always plain bytes — turning "700 MB" into a number is the interface's job, and a parser hidden down here would disagree with it sooner or later.

type Effect

type Effect struct {
	Package     string `json:"package,omitempty"`
	Dir         string `json:"dir,omitempty"`
	Filename    string `json:"filename,omitempty"`
	Comment     string `json:"comment,omitempty"`
	Priority    *int   `json:"priority,omitempty"`
	AutoExtract *bool  `json:"autoExtract,omitempty"`
	Chunks      *int   `json:"chunks,omitempty"`
	// Matched names the rules that fired, in the order they fired, so the
	// interface can answer "why did this land here" without re-running anything.
	Matched []string `json:"matched,omitempty"`
}

Effect is what the Packagizer decided for one link. Empty strings and nil pointers mean "no rule had an opinion", so the caller applies only what is set and leaves the rest of the task as it was.

type Field

type Field string

Field is what a condition looks at on a candidate link.

const (
	FieldFilename Field = "filename"
	FieldURL      Field = "url"
	FieldHoster   Field = "hoster"
	FieldSource   Field = "source" // the page a crawl found the link on
	FieldFiletype Field = "filetype"
	FieldFilesize Field = "filesize"
	FieldPackage  Field = "package"
)

type FieldGrammar

type FieldGrammar struct {
	ID Field `json:"id"`
	// Ops is exactly the set compileCondition accepts for this field, in the
	// order the form should list them.
	Ops []Op `json:"ops"`
	// Numeric marks the field whose values are byte counts rather than text, so
	// the form knows to offer a size box instead of a text box. Turning "700 MB"
	// into a number is the interface's job: a parser down here would disagree
	// with the one up there sooner or later.
	Numeric bool `json:"numeric,omitempty"`
	// Groups marks a field a capture group can be read from, which is every field
	// the "matches" operator applies to.
	Groups bool `json:"groups,omitempty"`
}

FieldGrammar is one thing a condition can look at, and what it can be asked.

type Grammar

type Grammar struct {
	Fields    []FieldGrammar  `json:"fields"`
	Operators []OpGrammar     `json:"operators"`
	Actions   []ActionGrammar `json:"actions"`
	Variables []Variable      `json:"variables"`
	// Categories are the file-type shorthands the editor offers on a filetype
	// condition. They expand into ordinary conditions and are described here for
	// the same reason everything else in this file is: the pattern a category
	// stands for has to be the one the engine will actually run.
	Categories []Category `json:"categories"`
	Limits     Limits     `json:"limits"`
}

Grammar is the whole of it.

func Describe

func Describe() Grammar

Describe is the grammar as it stands. It is built rather than stored so that adding a field or an operator above shows up here by the same edit.

type Limits

type Limits struct {
	PriorityMin int `json:"priorityMin"`
	PriorityMax int `json:"priorityMax"`
	MaxChunks   int `json:"maxChunks"`
	MaxPattern  int `json:"maxPattern"`
}

Limits are the numbers a form should stop the user at, taken from the same constants that refuse a rule.

type LinkReport

type LinkReport struct {
	URL      string `json:"url"`
	Filename string `json:"filename"`
	// Matched is the rules that fired, by index, in the order they fired.
	Matched []int `json:"matched"`
	// Effect and Verdict are exactly what the two engines would return, from the
	// same code that staging calls.
	Effect  Effect  `json:"effect"`
	Verdict Verdict `json:"verdict"`
	// Result is where the link would end up, with the fields no rule had an
	// opinion about filled in from the link itself. Effect leaves those empty and
	// means "unchanged", which is right for the caller applying it and useless to
	// somebody asking what the answer is.
	Result Outcome `json:"result"`
}

LinkReport is what would happen to one sample link.

type Matcher

type Matcher struct {
	// contains filtered or unexported fields
}

Matcher is a compiled Set. It is safe for concurrent use, which matters because links are staged from several goroutines and there is exactly one Matcher per rule set.

func (*Matcher) Apply

func (m *Matcher) Apply(c Candidate) Effect

Apply runs the Packagizer flavour. Every matching rule contributes in order and a later rule wins per field, unless the set asked to stop at the first match.

Only the template that survived the whole list is expanded, and it is expanded after the loop rather than inside it. Expanding as we go would spend the work on values a later rule immediately overwrites, and — the reason this is a correctness question rather than a performance one — <jd:append> would count every one of those discarded values. Two rules both writing an <jd:append> package name would hand the very first link "_2", a de-duplication of nothing, and the suffix would then say how many rules touched the field rather than how often that name has been seen.

func (*Matcher) Check

func (m *Matcher) Check(c Candidate) Verdict

Check runs the filter flavour. The first matching rule decides: a reject ends it, and an explicit accept ends it too when the set stops at the first match, which is how a narrow "keep this" placed above a broad "drop that" survives. A link no rule matched is accepted, so an empty or broken filter takes everything rather than nothing.

func (*Matcher) Empty

func (m *Matcher) Empty() bool

Empty reports whether the matcher holds no usable rule, so a caller can skip building a Candidate for every link when nothing is configured.

func (*Matcher) ResetAppend

func (m *Matcher) ResetAppend()

ResetAppend clears the <jd:append> counter, so a caller that treats each paste as a fresh batch can start its numbering over instead of continuing from wherever the last batch left off.

type Op

type Op string

Op is how a condition compares. Every operator except OpMatches folds case, because a user typing "sample" into a web form does not mean to let "Sample.mkv" through. OpMatches is left alone: the pattern carries its own flags and forcing (?i) onto it would overrule what the user wrote.

const (
	OpContains    Op = "contains"
	OpEquals      Op = "equals"
	OpContainsNot Op = "contains-not"
	OpEqualsNot   Op = "equals-not"
	OpMatches     Op = "matches"    // regular expression, unanchored
	OpBetween     Op = "is-between" // numeric, Min..Max, file size only
)

type OpGrammar

type OpGrammar struct {
	ID Op `json:"id"`
	// Value is set when the operator needs the single Value box filled.
	Value bool `json:"value,omitempty"`
	// Range is set when it needs Min and Max instead. Both are never set at once.
	Range bool `json:"range,omitempty"`
	// Regex marks the operator whose value is a pattern, so the form can offer a
	// pattern box and say what a bad one costs.
	Regex bool `json:"regex,omitempty"`
}

OpGrammar says what a form has to collect for one operator.

type Outcome

type Outcome struct {
	Package  string `json:"package"`
	Filename string `json:"filename"`
}

Outcome is the two values a dry run can state outright. The folder is not one of them: when no rule names a folder the link lands in the one the settings say, and this package has no way to know which that is — a guess printed in a preview would be believed.

type Problem

type Problem struct {
	Index   int    `json:"index"` // position in Set.Rules, zero-based
	Rule    string `json:"rule"`  // the rule's name, or its position when unnamed
	Message string `json:"message"`
	// Condition is which condition the problem is about, counting from 1, and 0
	// when it is about the action or the rule as a whole.
	//
	// The number is already in Message, and the editor needs it as data: put a
	// message under the rule and the user still has to find which of six rows has
	// the unparsable pattern in it. The alternative is a client that parses
	// "condition 3 (filename): ..." back out of a sentence, which breaks the first
	// time the wording changes or the interface is translated.
	Condition int `json:"condition,omitempty"`
}

Problem is one rule Compile could not use, in words meant for the user.

func (Problem) Error

func (p Problem) Error() string

type Report

type Report struct {
	// Problems is every rule Compile could not use, keyed to its position in the
	// set so the editor can put the message on the rule instead of in a list at
	// the bottom of the page that nobody connects to anything.
	Problems []Problem `json:"problems"`
	// Rules mirrors the set one entry per rule, in order, including the ones that
	// are switched off or broken. A rule missing from the answer would look like a
	// rendering fault; one that is present and says why it did nothing does not.
	Rules []RuleReport `json:"rules"`
	// Links is one entry per sample, in the order they were given.
	Links []LinkReport `json:"links"`
	// Disabled reports that the set's master switch is off, so the editor can say
	// that nothing below is being applied. The rest of the report is still filled
	// in as though it were on: a set cannot be repaired while it is off if being
	// off also hides what is wrong with it.
	Disabled bool `json:"disabled,omitempty"`
}

Report is what a whole dry run answers with.

func Preview

func Preview(s Set, cands []Candidate) Report

Preview runs a set against a batch of sample links and says what each rule and each link did.

The set is compiled with its master switch ignored, so a set that is switched off can still be worked on; Report.Disabled carries the real state. Disabled rules stay switched off, because a rule switched off is an edit somebody made on purpose and turning it back on in the preview would answer a question nobody asked.

type Rule

type Rule struct {
	Name       string      `json:"name,omitempty"`
	Disabled   bool        `json:"disabled,omitempty"`
	Conditions []Condition `json:"conditions,omitempty"`
	Action     Action      `json:"action"`
}

Rule is one entry in a Set. All of its conditions must hold for it to match, so an "either/or" is written as two rules; that keeps the rule list something a person can read top to bottom without tracking precedence.

A rule with no conditions matches every link. That is deliberate: it is how a catch-all default folder or a blanket reject at the end of a filter is written.

The flag is Disabled rather than Enabled so the zero value is a live rule. A client that posts a rule without the field means to add a rule, and the other way round it would arrive switched off with nothing to explain why.

type RuleReport

type RuleReport struct {
	Index    int    `json:"index"`
	Name     string `json:"name"`
	Disabled bool   `json:"disabled,omitempty"`
	// Problems is this rule's share of Report.Problems. It is duplicated rather
	// than referenced because the editor renders it next to the rule, and a
	// client filtering a flat list by index is a filter every client has to write.
	Problems []Problem `json:"problems,omitempty"`
	// Matched is how many of the samples this rule fired on. Zero on a rule that
	// compiled cleanly is the answer somebody is usually looking for.
	Matched int `json:"matched"`
}

RuleReport is one rule's own verdict on the dry run.

type Set

type Set struct {
	Rules []Rule `json:"rules,omitempty"`
	// Disabled is the master switch for the whole list, so somebody can find out
	// whether a rule set is the reason for what they are seeing without deleting
	// it or switching off nine rules one at a time.
	//
	// It is spelled Disabled and not Enabled for the same reason Rule.Disabled is:
	// the zero value has to be a live set. An Enabled field arrives false in every
	// stored settings file written before it existed, which would switch both the
	// Packagizer and the link filter off on the first boot after the upgrade — and
	// the symptom reads as a matching bug, not as a settings bug, so it is looked
	// for in the wrong place.
	Disabled bool `json:"disabled,omitempty"`
	// StopAfterMatch ends evaluation at the first rule that matches. The
	// Packagizer wants it off, so every matching rule contributes and a later
	// rule wins per field; a filter usually wants it on, so an accept placed
	// above a broad reject actually protects the link.
	StopAfterMatch bool `json:"stopAfterMatch,omitempty"`
}

Set is one ordered rule list, persisted as part of the settings.

type Variable

type Variable struct {
	// Tag is inserted verbatim, with N and FIELD left standing as the parts the
	// user replaces.
	Tag string `json:"tag"`
	// ID keys the description the interface shows beside it.
	ID string `json:"id"`
	// Params names the placeholders inside Tag, so the form can prompt for them
	// instead of leaving the user to notice a literal N in their folder name.
	Params []string `json:"params,omitempty"`
}

Variable is one placeholder a template field can carry.

type Verdict

type Verdict struct {
	Rejected bool   `json:"rejected"`
	Rule     string `json:"rule,omitempty"`
	Reason   string `json:"reason,omitempty"`
}

Verdict is what the filter decided. Rule is set whenever a rule decided the outcome, including an explicit accept, so a link that survived a filter can say what let it through as well as what would have stopped it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL