Documentation
¶
Overview ¶
Package langtag answers one question: may this language track stand in for the one that was asked for, and how far a substitution is that.
The question comes up whenever software holds a language choice a person made and then meets content tagged differently. A media library is the motivating case: a viewer selects a subtitle tagged "nob" on one episode, the next episode carries only "nor", and string equality reports no match even though both name Norwegian Bokmål.
Tags are structured, not opaque ¶
Language identifiers arrive from files, transcoders and third-party APIs in every code system ever published: ISO 639-1 two-letter codes, both the bibliographic and terminological ISO 639-2 variants, ISO 639-3, and BCP 47 tags with script and region subtags. Parse canonicalizes all of them onto one BCP 47 form, so the twenty ISO 639-2 duplicate pairs (ger/deu, fre/fra, chi/zho and the rest) and the deprecated codes (in, iw, ji) stop being distinct identities.
Distance is graded, not binary ¶
Compare reports a Tier: how far a candidate sits from what was wanted. Tiers 0 through 2 are derived from published standards data. The two above them are curated judgments, split because they make different claims: TierIntelligible says two languages are close enough to read across, and TierSharedLiteracy says only that readers of one are broadly literate in the other. A caller sets a floor and nothing beyond it is a match.
want, _ := langtag.Parse("nob")
have, _ := langtag.Parse("nor")
langtag.Compare(want, have) // TierSameLanguage
What this package deliberately will not do ¶
It will not substitute an unrelated language at the intelligible tier or below. CLDR's own distance data rates Basque against Spanish, Welsh against English and Tamil against English as close, because those populations are broadly literate in the second language. That is accurate sociolinguistics and a poor default: a viewer who chose Tamil subtitles did not ask for English ones. Claims of that shape live at TierSharedLiteracy alone, named for what they are, and the table ships one of them.
Every cross-language entry is a claim about reading. Danish and Norwegian are close on the page and much further apart aloud. Software matching an audio track should stop at TierOtherScript; this package cannot tell which kind of track a caller is matching.
It is not a general internationalization library. There is no collation, no formatting and no content negotiation. Display names live in the optional subpackage langtag/name, which is separate because its CLDR tables add roughly 2.5 MB to a binary.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Match ¶
Match reports whether have is an acceptable stand-in for want at or within floor, using the built-in table.
TierNone is not a usable floor and always reports false, in either position. It means "no relationship", so accepting it as a floor would accept every language as a substitute for every other. That matters because it is also what ParseTier returns for input it does not recognise: a mistyped configuration value therefore matches nothing rather than everything.
Example ¶
A floor keeps a caller from reaching further than it intends.
package main
import (
"fmt"
"github.com/cplieger/langtag"
)
func main() {
want := langtag.MustParse("zh-Hans")
have := langtag.MustParse("zh-Hant")
fmt.Println("at same-language:", langtag.Match(want, have, langtag.TierSameLanguage))
fmt.Println("at other-script: ", langtag.Match(want, have, langtag.TierOtherScript))
}
Output: at same-language: false at other-script: true
func Reason ¶
Reason returns the recorded justification for a cross-language substitution under the built-in table. See Comparer.Reason.
Example ¶
Reason explains a tier-3 substitution, so a surprised user can be told why.
package main
import (
"fmt"
"github.com/cplieger/langtag"
)
func main() {
reason, ok := langtag.Reason(langtag.MustParse("nb"), langtag.MustParse("da"))
fmt.Println(ok)
fmt.Println(reason)
}
Output: true Written Bokmål derives from Danish and the two remain close on the page. This is a written-language claim only; the spoken languages are much further apart.
func Valid ¶
Valid reports whether raw names a language. Intended for validating configuration at load time, so that a typo fails loudly at startup instead of silently matching nothing forever.
func ValidateFallbacks ¶ added in v1.0.2
ValidateFallbacks reports why each rejected entry in f was rejected, in order, and returns nil when every entry is usable. WithFallbacks drops the same entries silently, because a table is configuration rather than a runtime input; this is the way to find out what it dropped.
Types ¶
type CloseScript ¶ added in v1.0.4
type CloseScript struct {
// Language is the primary subtag after macrolanguage folding.
Language string
// A and B are the ISO 15924 script codes, in either order.
A, B string
// Reason states why readers move between them.
Reason string
// Provenance records where the claim can be re-checked.
Provenance string
}
CloseScript is a pair of scripts that one language's readers move between freely, so a difference between them is not a barrier the way a script difference usually is.
This is a narrow escape hatch from the structural rule that a script difference is farther than a region difference. That rule is right in general and wrong for a handful of languages written in two scripts whose readers are taught both.
func CloseScripts ¶ added in v1.0.4
func CloseScripts() []CloseScript
CloseScripts returns a copy of the built-in list of script pairs that read as one language.
Unlike Fallbacks, this list is not replaceable. It records where CLDR states an explicit symmetric distance for a script pair, which is a fact about the data rather than a judgment about people, so a deployment has nothing to disagree with. A caller that wants a script difference treated as a barrier anyway floors at TierSameLanguage and gets that.
type Comparer ¶
type Comparer struct {
// contains filtered or unexported fields
}
Comparer answers tier questions against a specific fallback table. Use it when the built-in cross-language judgments do not suit the deployment; the package-level Compare, Match and Best use the built-in table.
A Comparer is immutable after construction and safe for concurrent use.
func WithFallbacks ¶
WithFallbacks returns a Comparer using the supplied cross-language table instead of the built-in one. Passing nil or an empty slice disables both cross-language tiers, so no substitution beyond TierOtherScript can occur at any floor.
Entries name languages as Tag.Language reports them, so "no" rather than "nb" or "nor". An entry naming anything else simply never matches.
Malformed entries are DROPPED rather than accommodated, so a table authoring mistake removes a substitution instead of licensing an unintended one. See ValidateFallbacks to find out which entries were rejected and why. An entry is malformed when it names no language on either side, names one language on both, carries no Kind, or claims a Kind that disagrees with its direction.
Example ¶
A deployment that disagrees with a shipped judgment replaces the table instead of forking the package. Passing nil disables tier 3 outright.
package main
import (
"fmt"
"github.com/cplieger/langtag"
)
func main() {
strict := langtag.WithFallbacks(nil)
nb, nn := langtag.MustParse("nb"), langtag.MustParse("nn")
fmt.Println("built-in:", langtag.Compare(nb, nn))
fmt.Println("no table:", strict.Compare(nb, nn))
}
Output: built-in: intelligible no table: none
func (*Comparer) Compare ¶
Compare reports how far the available tag have sits from the wanted tag want. See the package-level Compare for the argument-order contract.
type Fallback ¶
type Fallback struct {
// Want is the language subtag a reader asked for.
Want string
// Have is the language subtag they can use instead.
Have string
// Reason states why a reader of Want can use Have. It exists so that an
// entry cannot be added without an argument, and so that a surprised user
// can be shown one.
Reason string
// Provenance records where the claim comes from, so a later maintainer can
// re-check it rather than inheriting it on trust.
Provenance string
// Kind decides the tier, and must agree with Both: an interchangeability
// claim runs both ways, a shared-literacy claim runs one way.
Kind Kind
// Both records that the relationship holds in the reverse direction too.
Both bool
}
Fallback is one directed cross-language relationship: a reader who wants the language Want can use a track in the language Have.
Want and Have are primary language subtags after macrolanguage folding, the value Tag.Language reports. One entry therefore covers every spelling of the languages it names: a Want of "no" answers for nor, nob, no and nb alike, and no second entry is needed for the umbrella tag.
type Kind ¶ added in v1.0.1
type Kind uint8
Kind separates the two different claims a cross-language fallback can make. It decides which tier an entry lands on, so that a caller can accept one kind of substitution without accepting the other.
const ( // KindUnset is the zero value, and it is not a usable Kind. An entry must // name its kind, because the two kinds land on different tiers and the // weaker claim must never inherit the stronger tier by default. Entries // carrying KindUnset are dropped by [WithFallbacks]. KindUnset Kind = iota // Intelligible means the two languages are close enough that readers move // between them. Such a relationship runs both ways. Intelligible // literate in the other one, which may be entirely unrelated. Such a // relationship runs one way: the majority-language population does not // reciprocate. SharedLiteracy )
type Tag ¶
type Tag struct {
// contains filtered or unexported fields
}
Tag is a canonical BCP 47 language tag that names an actual language.
The zero Tag is invalid: it is what Parse returns for input that names no language, and it matches nothing at any tier, including another zero Tag. Two tracks both tagged "undetermined" are not known to share a language.
Tag holds only strings, so it is comparable, copyable and carries no reference to the parsing library in its public shape.
func MustParse ¶
MustParse is Parse for identifiers fixed at compile time. It panics when raw does not name a language, so it belongs in package-level tables and tests, never on a path that handles input.
func Parse ¶
Parse canonicalizes one raw language identifier from an untrusted source.
It accepts ISO 639-1, ISO 639-2 (bibliographic or terminological), ISO 639-3 and BCP 47 tags, in any letter case, with surrounding whitespace. ok is false, and the returned Tag is the zero Tag, for the empty string, for the placeholder subtags that name no language (und, zxx, mul, mis), for private-use subtags (qaa through qtz), and for anything the IANA Language Subtag Registry does not know.
Parse never reports an error value. Callers act on ok; there is no recovery from a malformed tag beyond ignoring it, and every call site would otherwise discard the error.
Example ¶
One language spelled two ways is one language.
package main
import (
"fmt"
"github.com/cplieger/langtag"
)
func main() {
for _, raw := range []string{"ger", "deu", "de", "nob", "nor"} {
tag, _ := langtag.Parse(raw)
fmt.Printf("%-4s -> tag %-3s language %s\n", raw, tag, tag.Language())
}
}
Output: ger -> tag de language de deu -> tag de language de de -> tag de language de nob -> tag nb language no nor -> tag no language no
func (Tag) IsZero ¶
IsZero reports whether t names no language. Every comparison involving a zero Tag yields TierNone.
func (Tag) Language ¶
Language returns the primary language subtag after macrolanguage folding: "no" for both nob and nor, "zh" for both cmn and zho. Two tags share a language exactly when this value matches. It is the right key for a per-language cache or a learned-preference map, because it does not fragment across the code systems that name one language several ways.
type Tier ¶
type Tier uint8
Tier is how far an available language sits from the one that was wanted. A caller picks the highest tier it will accept as a floor.
The order is by KIND OF LICENSE, from narrowest to widest: the same tag, then the same language, then the same language in another script, then a close language, then a language readers merely happen to be schooled in. It is deliberately NOT a reading-difficulty scale, and reading difficulty is not monotonic along it. TierOtherScript is where that shows: what remains on it is the script pairs CLDR does not explicitly rate, which it scores at 50, farther than any pair on the two tiers above. A caller serving Chinese content should weigh that before accepting TierOtherScript.
Tiers 0 through 2 follow from published standards data and hold no opinion. The two tiers above them are curated judgments, and they are separated because they make different kinds of claim: TierIntelligible says two languages are interchangeable, while TierSharedLiteracy says only that readers of one are, as a population, literate in the other. The second is a much weaker claim and the substitution it licenses is a different language entirely, so reaching it takes its own opt-in.
const ( // TierIdentical means the two tags canonicalize to the same thing. It // covers the code systems that spell one language several ways: nor and // no, ger and deu, chi and zho, iw and he. TierIdentical Tier = iota // TierSameLanguage means one language a reader takes in without effort. It // covers macrolanguage folding (nob against nor, cmn against zho), regional // variants (es-ES against es-419, pt-BR against pt-PT, en-GB against en-US), // and the script pairs whose readers are taught both, which is Serbian // Latin against Serbian Cyrillic (see [CloseScripts]). TierSameLanguage // TierOtherScript means one language in a script its readers are NOT // generally taught alongside: Simplified against Traditional Chinese, // uz-Latn against uz-Cyrl. Readers of one usually manage the other, with // effort. The pairs CLDR explicitly rates as close sit on the tier below // instead, so what remains here is the set CLDR does not vouch for, which // it scores at 50 against 4 to 20 for every curated close-language pair. // That is why this tier can be a bigger ask than the two above it. TierOtherScript // TierIntelligible means two different languages that readers move between // because the languages themselves are close: Bokmål and Nynorsk, Danish // and Norwegian, Croatian and Bosnian, Czech and Slovak. The relationship // runs both ways, which is the mark of interchangeability rather than of // one population's schooling. // // These are written-language claims. Two languages readable to each other // on the page are not necessarily comprehensible to each other aloud, and // Danish against Norwegian is exactly that case. A caller matching audio // should not reach this tier. TierIntelligible // that readers of the wanted one can use because they are broadly literate // in it. Catalan readers read Spanish. The relationship runs one way only, // and the two languages may be unrelated. // // This tier exists so the claim can be made honestly and separately rather // than smuggled in beside interchangeability. It is never reached unless a // caller asks for it by name, and the table ships one entry: extending it // to the rest of the family is a decision for whoever runs the software, // through [WithFallbacks]. TierSharedLiteracy // TierNone means no relationship worth acting on. It is also the answer // whenever either tag is the zero Tag. TierNone )
func Best ¶
func Best[T any](want Tag, candidates []T, tagOf func(T) Tag, floor Tier) (out []T, tier Tier, ok bool)
Best returns the candidates closest to want and the tier they matched at, using the built-in table. See BestWith.
Example ¶
Best selects from a caller's own type and returns every candidate at the closest tier, leaving the final choice to whatever the caller ranks by.
package main
import (
"fmt"
"github.com/cplieger/langtag"
)
func main() {
type subtitle struct {
file string
codec string
lang langtag.Tag
}
available := []subtitle{
{"ep2.eng.srt", "srt", langtag.MustParse("eng")},
{"ep2.nor.ass", "ass", langtag.MustParse("nor")},
{"ep2.nor.srt", "srt", langtag.MustParse("nor")},
{"ep2.swe.srt", "srt", langtag.MustParse("swe")},
}
want := langtag.MustParse("nob") // the viewer chose Bokmål on episode 1
matches, tier, ok := langtag.Best(want, available,
func(s subtitle) langtag.Tag { return s.lang },
langtag.TierSameLanguage)
if !ok {
fmt.Println("no usable subtitle")
return
}
fmt.Printf("matched at tier %s:\n", tier)
for _, m := range matches {
fmt.Printf(" %s (%s)\n", m.file, m.codec)
}
}
Output: matched at tier same-language: ep2.nor.ass (ass) ep2.nor.srt (srt)
func BestWith ¶
func BestWith[T any](c *Comparer, want Tag, candidates []T, tagOf func(T) Tag, floor Tier) (out []T, tier Tier, ok bool)
BestWith returns every candidate at the closest tier reached, the tier itself, and whether anything was within floor.
All returned candidates share one tier, because the caller is expected to rank within a language by criteria this package knows nothing about: codec, forced and hearing-impaired flags, track title, provider score. Input order is preserved so an existing tie-break stays deterministic.
tagOf extracts the language from a candidate, which lets a caller pass its own type without building a parallel slice and mapping indices back. A candidate whose tag is the zero Tag never matches.
A floor of TierNone or beyond selects nothing, and does NOT widen to the most permissive tier. An unrecognised configuration value parses to TierNone (see ParseTier), and the safe reading of an unusable floor is that no substitution was authorised, not that every substitution was.
func Compare ¶
Compare reports how far the available tag have sits from the wanted tag want, using the built-in table.
The argument order is load-bearing. Cross-language entries are directed, so Compare(want, have) and Compare(have, want) can differ: a Catalan viewer accepts a Spanish track, and a Spanish viewer does not accept a Catalan one. want is always the language a person chose; have is what the content offers.
Example ¶
Compare grades the distance rather than answering yes or no.
package main
import (
"fmt"
"github.com/cplieger/langtag"
)
func main() {
want := langtag.MustParse("nob")
for _, raw := range []string{"nob", "nor", "nn", "sv"} {
have := langtag.MustParse(raw)
fmt.Printf("a viewer wanting nob, offered %-4s: %s\n", raw, langtag.Compare(want, have))
}
}
Output: a viewer wanting nob, offered nob : identical a viewer wanting nob, offered nor : same-language a viewer wanting nob, offered nn : intelligible a viewer wanting nob, offered sv : none
Example (Directed) ¶
Tier-3 relationships are directed, because shared literacy usually runs one way: Catalan readers read Spanish, and Spanish readers do not read Catalan.
package main
import (
"fmt"
"github.com/cplieger/langtag"
)
func main() {
ca, es := langtag.MustParse("ca"), langtag.MustParse("es")
fmt.Println("wanting ca, offered es:", langtag.Compare(ca, es))
fmt.Println("wanting es, offered ca:", langtag.Compare(es, ca))
}
Output: wanting ca, offered es: shared-literacy wanting es, offered ca: none
func ParseTier ¶
ParseTier maps a configuration string onto a Tier. It accepts the spellings String produces, in any letter case, plus the underscore form so that an operator who writes same_language is not punished for it. ok is false for anything else, and for "none", which is not a usable floor: a floor of TierNone would accept every language as a substitute for every other.