README
¶
docutils
Pure-Go (CGO=0) reStructuredText engine — the layer both plain reST tooling and Sphinx build on, not a port of Sphinx itself. See the org capability map and the project memory for the full rationale behind this scope.
Status: early v1, core grammar + a first writer
This is a from-scratch parser modeled on the reference implementation
(docutils.parsers.rst, Python,
public domain) — its states.py state-machine design and pattern tables
were read as the specification, not executed or embedded. A local
docutils 0.23 install serves as a foreign judge during development: test
fixtures are cross-checked against publish_string(..., writer_name= 'pseudoxml') output, but nothing in this module invokes Python at
build or run time.
Implemented: sections (over/underlined titles, arbitrary nesting
depth via first-seen title-style ordering), paragraphs, transitions,
bullet lists, enumerated lists — all five of docutils' own sequences
(arabic, loweralpha/upperalpha, lowerroman/upperroman) in all
three formats (N., (N), N)), plus the auto-enumerator #, with
docutils' own ambiguity-resolution rules (Body.parse_enumerator, read
directly: a bare single roman-charset letter defaults to roman ONLY when
no sequence is already established — "H." then "I." continues an
established upperalpha list as ordinal 9, not a fresh upperroman
one), a malformed roman numeral (iiii, no valid subtractive form)
rejected outright rather than treated as a valid ordinal, enumtype/
prefix/suffix/start attributes, and the "start value not
ordinal-1" INFO + "ends without a blank line" WARNING diagnostics (both
land as SIBLINGS of the <enumerated_list>, never nested inside it,
matching self.parent += msg read directly; the start-value INFO's own
LINE is the list's, absolute — docutils raises it with
base_node=enumlist, and Reporter.system_message reads that node's own
source/line instead of consulting its cursor at all, v0.136.6: passing
the index within the current nesting level is the same number only at the
top level, and seven real-world corpus files had a nested list saying
line="1" for a list on line 704) — enumerator-sequence
validation WITHIN an already-started list (docutils errors on a
non-consecutive ordinal mid-list; this parser's own continuation check
already requires exact +1, so a gap just ends the list instead of
producing that distinct diagnostic) is the one piece of this still not
fully ported, see "Not yet ported" below — field lists
(including a docutils-shaped body-indent quirk: a continuation line
indented less than the marker column, e.g. under :date: 2026-08-30,
still belongs to the field — a FIELD BODY and an OPTION DESCRIPTION
discover their own indent via get_first_known_indented, while a LIST
ITEM's content column is fixed by its marker, so the same shallower line
ENDS a bullet/enumerated list and becomes a block quote with the usual
unindent warning, v0.65.0+; a BARE marker with nothing after it takes
its column from wherever its first indented line starts, narrower or
wider alike, independently of that distinction) — with a leading field list (the first
child that is not PreBibliographic: a <meta>, comment, decoration,
system message, target or substitution definition is stepped over, which
is first_child_not_matching_class(nodes.PreBibliographic) in the
transform itself, v0.136.5 — requiring position 0 exactly made docinfo
promotion and the meta hoist mutually exclusive) promoted to a typed
<docinfo>, registered
bibliographic names (author, authors, organization, address, contact,
version, revision, status, date, copyright, dedication, abstract,
matched the same case/whitespace-insensitive way as any other reST
name) becoming typed children rather than staying generic <field>s;
authors splits a single field body on ; (or, failing that, ,)
into one <author> per name; dedication/abstract become sibling
<topic> elements instead, right after docinfo — definition lists, line blocks (nested by
relative indentation, matching docutils' own sub-stanza grouping),
doctest blocks (kept verbatim, ">>>" prompts included) — a
block runs from its prompt to the next BLANK line, indentation
irrelevant, since the indented lines after a prompt are the
interpreter's own OUTPUT (docutils reads it with get_text_block() and
no flush_left; stopping at the first indented line instead handed that
output to a spurious block quote, v0.62.0), block
quotes, literal blocks (::), comments, directives (an unrecognized name draws docutils' own pair of
diagnostics — an INFO for the failed lookup, then
Unknown directive type "X". carrying the whole directive source as a
<literal_block>; both live in the PARSER, not in a transform, so
Options.ReportUnknownDirectives defaults TRUE, and a consumer that
would rather keep an unknown directive's CONTENT — a Sphinx
.. toctree:: in a document this package was never told about — sets it
false for the structural capture below, v0.68.0+. Note this is what the
corpus's German admonition fixtures need: docutils' own default language
is English, so .. Achtung:: is simply unknown to it too, not a
localization gap. A directive this package DOES implement never draws
that diagnostic even when a particular invocation fails its own
validation — .. raw:: with no format argument is
Error in "raw" directive: ... in docutils, and still the structural
capture here. Structural capture is name, arguments and raw content —
never dispatched to per-directive semantics, with these exceptions: raw (.. raw:: FORMAT, whose content passes through completely unprocessed, tagged
with its target format; see Options.RawEnabled, on by default matching
real docutils' own default despite its confusingly-named --no-raw
flag), table (.. table:: TITLE, dispatching its body through this
project's own existing simple/grid table parser and requiring exactly
one table to result — :class:/:name:/:align:/:width:/:widths:
options, the title itself inline-parsed so it can carry markup or a
dangling-markup warning of its own), list-table (building a
<table> from scratch out of a uniform two-level bullet list — rows,
each a nested list of cells — plus :header-rows:/:stub-columns:; the
colwidths-given/colwidths-auto annotation :widths: adds lands
BEFORE the :class: values for list-table/csv-table and AFTER them
for table, because the first two set it while building their own table
and the third appends it to one the nested parse already built — opposite
orders for the same two options, asked of docutils rather than reasoned
about),
and the nine generic ADMONITIONS — attention/caution/danger/
error/hint/important/note/tip/warning, directive name
matched case-INSENSITIVELY (.. Note::, .. WARNING:: work the same as
.. note::), each just wrapping its own content in a like-named node —
plus the one non-generic admonition directive (a REQUIRED title
argument becoming a <title>, auto-classed admonition-<slug of the title> unless :class: overrides it), topic (a REQUIRED title
argument), and sidebar (an OPTIONAL title argument, plus its own
:subtitle: option — valid only alongside a title) — both with a real
NESTING restriction real docutils itself enforces: a topic/sidebar is
only valid directly inside a document or section (a topic ALSO directly
inside a sidebar); anywhere else — a list item, a block quote, another
topic — is an ERROR, checked against this parser's own notion of "the
container currently being appended to", the same thing real docutils'
own state_machine.node check means; image (a REQUIRED URI argument,
no content permitted, :alt:/:height:/:width:/:scale:/:align:
— validated against left/center/right, real docutils' own
substitution-definition-only vertical-values variant not implemented —
/:loading: — embed/link/lazy — /:class:/:name:/:target:
options; a :target: wraps the <image> in a <reference>, its value
read by the SAME parse_target a .. _name: value hyperlink target
uses, so a plain URI becomes a refuri and other_ or `other name`_
becomes a name+refname pair resolved with every other reference —
note that no mailto: adjustment is applied to it, unlike an inline
embedded URI, and that a :name: beside a :target: stays on the
IMAGE, not on the wrapper) and
figure (same argument, reuses image's own option handling for
everything but its own :figwidth:/:figclass:/:figname:/:align:,
plus an optional caption/legend body: the content's first <paragraph>
becomes <caption>, an empty comment suppresses the caption entirely,
anything else there is an ERROR discarding whatever legend would have
followed, and everything remaining after becomes <legend>; a
.. class::-produced <pending> node ahead of the caption IS reproduced
now, see below); code (an OPTIONAL language argument
becomes a class alongside the fixed code one — even for a language
this parser will never actually highlight, see below — plus
:class:/:name:/:number-lines: options; :number-lines: is fully
ported, independent of lexical analysis: a leading <inline class="ln">
line-number marker, right-padded to the width of the LAST line number,
before the content and after every embedded newline. Real docutils
additionally runs the content through Pygments for syntax coloring
(per-token <inline class="..."> spans) unless the document's own
syntax_highlight setting is "none" — not ported here, the same
"no Pygments equivalent and never will" scope boundary already
documented for the inline :code: role below: content here is always
treated the way real docutils itself treats an unhighlightable
language, matching every corpus fixture's own default-settings output,
none of which exercise a language that would actually highlight
differently); container (an OPTIONAL argument, possibly spanning
several lines joined with a space, becomes the node's own classes —
NOT a :class: option, container only recognizes :name: — content
REQUIRED, parsed as ordinary body elements); compound (no argument
at all — same-line text after :: folds into the directive's own
FIRST content line, exactly like the nine generic admonitions —
:class:/:name: options, content REQUIRED); meta (a run of field markers — reusing the
SAME grammar every field list already does — each becoming a <meta>
whose own attributes come from splitting the marker's own, backslash-
unescaped NAME text on whitespace: the first token is either
attr=value or a bare word, defaulting to name="..."; every token
after the first MUST be attr=value or it's a real per-field ERROR;
parsing stops at the first non-field-marker content line, and anything
left unconsumed there makes the WHOLE directive one more ERROR on top
of whatever fields already parsed. The distinctive part: NONE of a
meta directive's own result nodes stay at their own lexical position —
real docutils splices them straight into the document ROOT's children,
at the front, however deeply the directive itself was nested; this
project defers that same effect to a single post-pass at the end of
parsing rather than mutating the tree mid-walk. The insertion point is
docutils' own first_child_not_matching_class((nodes.Titular, nodes.meta)) (v0.136.5): the meta nodes go after a leading title,
subtitle or rubric and after the meta nodes already hoisted, but AHEAD of
everything else — including a leading field list, which the DocInfo
transform then still finds by stepping over them); :class:/:name: options work
the same way real docutils' own generic option/content-block split does
(Body.parse_directive_block, read directly): the option block is
whatever TRAILING run of :key: value lines the directive's own body
ends with, wherever that starts — not merely a leading run assumed to be
at the very top, the simpler shape this project's table/list-table
options still use, since no corpus case has needed the general form
there — so real content is free to precede the options, and a role
invocation elsewhere in that content is never mistaken for one.
An UNKNOWN option KEY is an ERROR naming it, for every directive listed
in optionspec.go's transcribed table — seventeen of them as of
v0.133.0 (the eleven admonition-shaped ones, container, image,
figure, table, list-table, topic, sidebar, rubric,
parsed-literal, line-block, joining code and math). The table was
complete two dozen versions before its callers were: an entry alone
enforces nothing, which is why TestWiredDirectivesRejectAnUnknownOption
asks each directive rather than reading the table. csv-table is the one
implemented directive still absent from it, and is documented as such by
a test of its own. Option VALUE conversion (a malformed :widths:, a
:height: that is not a length) is still not validated, except
:align:/:loading:/:target: on image/figure, all three directly
corpus-tested — an empty :target: raises the same "argument required
but none supplied" error unchanged_required raises, at option-assembly
time, so no image survives it.
Being faithful here COSTS a rendered block, and the count is the point of
Options.ReportUnknownDirectiveOptions: 28 real-world corpus files lost
one to a sphinx-only option before v0.133.0 and 32 do now, the four new
ones being exactly the four files the corpus newly matches.
go-richdoc/rst sets the flag false for that reason. Matching this
project's own established scope boundary — see "Not yet ported" below, hyperlink
targets with reference resolution — including INDIRECT targets
(.. _a: b_ and the backquoted phrase form .. _a: `b c`_, whose
value is itself another target's name, chased
through however many hops until a real URI is reached; a cycle is left
unresolved rather than looping forever) and ANONYMOUS targets/references
(.. __: uri / x__ / `x`__, matched by DOCUMENT-ORDER
POSITION — the Nth anonymous reference to the Nth anonymous target,
textual order, regardless of which comes first — rather than by name at
all, a genuinely different mechanism from every other target/reference
pair above; an anonymous target's own value may itself be indirect,
.. __: othername_, chased the same way a named indirect target is;
every NAMED target — anonymous or indirect ones excepted — also gets an
id attribute, real docutils' own Target.run behavior, missing
entirely until v0.46.0 caught it) —
footnotes ([1]_/[#]_/[#name]_/
[*]_, with real auto-NUMBERING ([#]_/[#name]_ sharing one
sequence, an explicit [1]_ elsewhere making the sequence skip that
number) and auto-SYMBOL assignment ([*]_, docutils' own fixed
ten-symbol sequence, doubling/tripling/... once it wraps: **, ††,
...) — both matched to their references by document-order position
when unnamed, same mechanism as anonymous targets above), citations
([CITE2002]_, never auto-numbered — the bracketed label is VALIDATED
against docutils' own dispatch patterns, [0-9]+|#|#simplename|\* for a
footnote and a bare simplename for a citation, so a line like
.. [citation label with spaces] text is not a citation at all but
falls through to the explicit-markup comment fallback, v0.53.0+; when
two names slug to the same id, the later one is disambiguated with an
incrementing -N suffix, docutils' create_id). That same label
validation applies to the INLINE reference form too ([CIT 1]_, whose
label is not a simplename, stays plain text instead of becoming a
<citation_reference>, v0.55.0+), and every footnote/citation REFERENCE
carries a parse-time id of its own (footnote-reference-1,
citation-reference-1, ... — document.note_*_ref all call set_id,
so an auto-numbered or symbol reference with no refname at all still
gets one; the two prefixes count independently of each other and of the
<footnote>/<citation> nodes they point at). Substitution definitions/references
(|name| — whitespace immediately INSIDE either pipe disqualifies the
construct entirely and the line becomes an ordinary comment
(.. | bad name |), while a space WITHIN the name is fine
(|good name|), v0.77.0+; a definition followed by a non-blank
unindented line draws the same "Explicit markup ends without a blank
line" warning every other explicit construct already did — the marker's own NAME may itself span multiple physical
lines, real docutils progressively re-matching its own substitution
pattern against the growing text until the closing | is found
(matchPipeLabelMultiline, explicit.go, v0.50.0+) — its content is
always an embedded directive invocation,
real semantics for replace:: (its two content checks are ported too,
v0.73.0+: NO content is
Content block expected for the "replace" directive; none found. and
more than one paragraph is
Error in "replace" directive: may contain a single paragraph only. —
each an ERROR quoting the DIRECTIVE block, followed by the
"empty or invalid" WARNING quoting the whole substitution line; content
is otherwise inline-parsed as one paragraph's worth
of content, nested directly as the substitution's own children — a
prohibited construct inside it, an anonymous reference, an
auto-numbered/auto-symbol footnote reference, or anything carrying its
own name/id, is a real ERROR, matching
disallowed_inside_substitution_definitions; a <problematic> anywhere
in the result, checked first, is a separate ERROR of its own — its own
WARNING messages carry no backref attribute in this specific
reconstruction, unlike their usual shape elsewhere, v0.46.0, checked
directly against the foreign judge; .. replace:: used as an ordinary
TOP-LEVEL directive, rather than embedded inside a substitution
definition, is its own separate "Invalid context" ERROR, v0.46.0),
image::
(the substitution's own name becomes the embedded image's default
:alt:, matching real docutils' option_presets), and raw:: (already
a real directive on its own — nested here rather than flattened onto
attributes, the shape an earlier version of this project mistakenly
produced); any OTHER directive name is captured only as far as
"produced nothing this project can nest here", the same "empty or
invalid" outcome real docutils reaches too by filtering a non-inline
result out — |name|_/|name|__ used AS a hyperlink resolves the
same way a bare/anonymous reference does, just wrapping the substitution
instead of carrying its own display text), and inline markup for **strong**, *emphasis*,
literal , a bare `x` with no role (docutils' DEFAULT
role, title_reference), named/anonymous references both bare (x_,
x__) and backtick-quoted (`x`_, `x`__) including an
embedded URI or indirect-name target (`text <https://example.com>`_, `text <alias_>`_, with mailto: auto-prefixing for an
embedded email address; a real multi-line target — the marker itself
starting on a later physical line, or its own value wrapping across
one, both real, corpus-tested shapes — joins with all REAL whitespace
inside it stripped entirely, while a backslash-escaped space/newline
there becomes exactly one literal space instead; omitting the display
text (`<uri>`_) reuses the ALIAS AS COMPUTED as both display text
and the target's own name (v0.134.0+), which is docutils' one-line "if
not text: text = alias" — so a name alias displays its NORMALIZED name
(`<Feature Negotiation_>`__ shows "feature negotiation", as two
PEPs do) and an email alias displays the whole mailto: URI it links
to; re-deriving that text instead of reusing the computed alias got both
branches wrong, each differently; a NAMED (single _) reference like this
also emits a real <target> sibling, so another reference elsewhere to
the same display text can resolve to it too — an ANONYMOUS (__) one
never does, resolving directly off its own refuri/refname instead of
joining document-order anonymous-target matching), interpreted text with a role, prefix
(:role:`x`) or suffix (`x`:role:), for docutils' built-in
GENERIC roles (emphasis, strong, literal, subscript/sub,
superscript/sup, title-reference/title/t,
abbreviation/ab, acronym/ac), plus its other three always-registered,
non-generic roles: math (a dedicated <math> node holding the raw,
unescaped TeX source, rendered by both writers below); code (real
class-list/highlight-language derivation, roles.py's code_role
ported — the content is preserved raw, backslashes intact, wrapped in
<literal class="...">; since this parser has no Pygments equivalent
and never will, a resolved highlight language always takes docutils'
own LexerError path, which itself distinguishes an EXPLICIT
:language: option, a WARNING + <problematic> — "Cannot analyze code.
Pygments package not found." — from one merely implied by a custom
role's own name, a silent fallback to the same plain, unclassified shape
as no language at all); and pep/pep-reference and rfc/rfc-reference
(numeric validation — a PEP number 0-9999, an RFC number ≥ 1, optionally
with a #section suffix — producing a <reference> to the canonical
page on success or an ERROR + <problematic> otherwise), plus a
.. role:: NAME(BASE)-registered custom role — aliasing a generic role
by tag the same way a built-in does, code (with its own :language:
option, see above), a bare definition with no BASE at all (docutils'
generic_custom_role, a plain <inline>), or (BASE is raw, with a
:format: option) this parser's one INLINE raw construct, mirroring the
raw directive above (the BUILT-IN raw role used directly, with no
.. role:: registration at all and so no :format: reaching it, always
errors instead, matching roles.py's own raw_role) — EVERY one of
these carries a :class: option (real docutils' set_implicit_options:
every role function implicitly supports one), defaulting to the role's
own name exactly like real docutils' Role directive does, rendered as
a class="..." attribute on whatever element the role produces — a
registered custom role's <inline class="..."> is NOT the same shape as
a totally unregistered role name, which still falls back to this
parser's own invented <inline role="name"> rather than docutils'
error (see "Not yet ported" below); rubric (a REQUIRED single-line-or-
final-argument-whitespace argument, inline-parsed directly into the
<rubric> node itself — no wrapping <title>, unlike topic/admonition
— :class:/:name: options; unlike every other directive here, Rubric
declares no content at all, so a real indented body is a genuine
"no content permitted" ERROR, not silently ignored); parsed-literal
(a <literal_block> whose content IS inline-parsed, unlike an ordinary
literal block — the whole joined content parsed in ONE call, so markup
spanning multiple physical lines still resolves as a single node,
matching real docutils' own inline_text call exactly); header/
footer (no argument at all, same shape as the nine generic
admonitions — same-line text folds into the directive's own first
content line — but nested-parsed into the document's own SINGLETON
<header>/<footer>, not a fresh element per invocation: a SECOND
.. header:: anywhere in the document appends more content to the
SAME <header> rather than creating a second one, wrapped together in
one <decoration> at the end of parsing with header always first and
footer always last, regardless of which was declared first in the
source — get_decoration/get_header/get_footer, nodes.py, read
directly). The .. role:: DIRECTIVE itself
(as opposed to a role it registers being used unrecognized later) does
error on a malformed definition, matching Role.run: no content at all,
content not matching the NAME(BASE) pattern, a BASE that isn't a
recognized role name (an INFO plus an ERROR — a base that's itself a
PREVIOUSLY .. role::-defined custom role is accepted, though the
chain only ever resolves ONE level deep at use time, see roleElement's
own doc comment), and a :class:/argument-derived class name that
can't become a valid identifier (a purely-numeric token, make_id's
own leading-digit stripping leaving nothing). .. default-role:: (an
OPTIONAL single-token argument, no options, no content permitted)
changes what BARE interpreted text — no explicit role prefix, no
trailing hyperlink underscore — resolves to for the rest of the
document, reusing the SAME base-role validation .. role:: already
has (an unresolvable name is the identical INFO+ERROR pair); a bare
.. default-role:: with no argument at all RESETS it back to real
docutils' own standard default, title-reference — DefaultRole.run,
misc.py, read directly. .. reStructuredText-test-directive:: (one of the 43 entries in
docutils' own _directive_registry, misc.TestDirective — a real
registered directive, not something its test suite adds. It builds
nothing: it reports back exactly how the directive block was SPLIT,
Directive processed. Type="...", arguments=[...], options={...}, content: plus the content as a <literal_block>, or content: None.
That makes its twelve corpus fixtures a conformance suite for the
argument/option/content split itself — which is why they are worth
having: whether an indented block is the ARGUMENT or the CONTENT turns
entirely on the blank line between it and the directive marker.
optional_arguments=1 with final_argument_whitespace, one option
(:option:, required value), content allowed). .. date:: (usable ONLY inside a substitution
definition — anywhere else is an ERROR with the same "Invalid context"
sentence .. replace:: gets, since misc.Date.run opens by refusing
any state that is not a SubstitutionDef. The directive declares no
arguments, so same-line text folds into its CONTENT, which is a C
strftime format string, defaulting to %Y-%m-%d when there is none.
The whole portable conversion set is implemented (%a %A %b %B %c %C %d %D %e %F %g %G %h %H %I %j %k %l %m %M %n %p %r %R %s %S %t %T %u %U %V %w %W %x %X %y %Y %z %Z %%), each pinned against real C strftime for
three timestamps chosen to separate them; an UNKNOWN conversion is kept
verbatim, % and all, since real docutils delegates to a platform
strftime and the platforms disagree about that case). .. line-block:: (the legacy directive form
of a line block, deprecated in favor of the bare | syntax below but
still supported — :class:/:name: options, content required; unlike
the bare syntax, each content line is inline-parsed INDEPENDENTLY,
never joined across a wrap, so markup can't span two lines the way it
can there — reuses the bare syntax's own nesting algorithm verbatim,
LineBlock.run, body.py, read directly). .. math:: (TeX math
source kept VERBATIM — never inline-parsed, since a backslash or
asterisk in it is math syntax rather than markup — in one or more
<math_block> nodes: content SPLITS ON BLANK LINES into several
siblings from a single directive, and no arguments are declared at all,
so same-line text folds into content's own first line, which is why an
argument PLUS an indented body yields two blocks rather than one;
:class:/:name: options, content required — MathBlock.run,
body.py, read directly. The HTML writer emits MathJax's own
display delimiters \[...\] and the LaTeX writer an equation*
environment, the latter matching real docutils' own latex2e output
exactly; docutils' html5 writer instead emits MathML by default, which
would need a TeX→MathML converter this project deliberately doesn't
carry, the same category of omission as the code directive's own
Pygments highlighting). And
backslash escapes; standalone URI (scheme://...) and email
(user@host) recognition — no backtick quoting or trailing _ needed
at all, e.g. plain https://example.com in running text becomes a
reference on its own, trailing sentence punctuation (, and .)
correctly excluded from the link; SIMPLE tables (=====-bordered, with
an optional ------underlined column-span row and a
multi-line/nested-list cell content — docutils' own SimpleTableParser
docstring example is this parser's own test fixture, verbatim); and
GRID tables (+---+---+-bordered, |-separated columns, an optional
+===+===+ head/body separator, cells spanning multiple ROWS as well
as columns — likewise docutils' own GridTableParser docstring example,
verbatim, traced with the same BFS cell-rectangle algorithm as
upstream: a queue of corner candidates, scanning right/down/left/up
around each cell to close its rectangle and discover the next cells'
starting corners); and OPTION lists (-f, --file=ARG Description.,
man-page-style, comma-separated short/long flag groups with an
argument joined by a space/=/embedded delimiter, reusing the same
marker+indented-continuation machinery as field lists — a marker with
no following content at all, on its own line or indented beneath it,
is not really an option list item and falls back to plain paragraph
text, matching docutils' own TransitionCorrection; the ARGUMENT follows
docutils' own option_argument class from its option_marker pattern,
[a-zA-Z][a-zA-Z0-9_-]* or a <placeholder>, so it must start with a
LETTER — --option=arg,arg, --option=arg=arg and --option= are each
an ordinary paragraph rather than an option list, v0.72.0+); and inline
internal targets (_`text` — a target INSIDE a paragraph, as
opposed to the block-level .. _name: uri hyperlink target above,
docutils' own target pattern in Inliner.patterns). Unlike a
block-level target, this one keeps its content as visible text and
carries no URI of its own — a reference resolving to one now resolves
to a same-document anchor (#slug, the target's own SLUGIFIED id
attribute, docutils/rst v0.32.0+ — matching every other target kind;
before that this parser only gave it a name, and same-document
anchors carried whatever raw, unslugified text the name itself
happened to be, spaces and all) instead, which both writers render
with a real anchor point (HTML <a id="slug">text</a>; LaTeX
\hypertarget/\hyperlink, not \href — a #-prefixed refuri routes
to the internal-link path specifically, since \href's usual URL
escaping would corrupt hyperref's own #-marker convention).
string2lines' per-line rstrip strips whatever Python's str.isspace()
accepts (v0.119.0+) — a no-break space, an em space, an ideographic
space — not just the ASCII space it stopped at, so term\u00a0 is the
term term and not term . unicode.IsSpace was MEASURED against
str.isspace() rather than assumed, after two rounds of exactly that
mistake: the two agree on all 29 code points except U+001C–U+001F, which
Python calls whitespace and Go does not. U+200B is in neither set and
survives, which is the case that keeps the rule from becoming "strip
anything invisible".
A backslash with nothing after it DISAPPEARS (v0.119.0+). escape2null
appends '\x00' + text[i+1:i+2], empty at the end of the text, so
unescape has a lone null to remove — para\ is the paragraph para.
Before another line it was already right, because there the escape had a
newline to eat.
A GRID table that does not FORM a table is now reported rather than
silently abandoned (v0.121.0+). Once the top border matches, docutils has
committed to a table: any failure is Malformed table. plus a detail
(Right border not aligned or missing., Bottom border missing or corrupt.) with the offending block quoted as a literal_block. This
parser reported nothing and let the lines fall back to ordinary block
parsing, so a broken table silently became a paragraph. Only the lines
the block COVERS are consumed; whatever interrupted the table is parsed
as itself afterwards.
Blank line required after table. came with it, and applies to a
well-formed table too: docutils raises it from table_top, the CALLER
of the table parser, which is why neither path had it here.
The SIMPLE table followed in v0.122.0, and that half was about CONTENT
rather than diagnostics: text in the MARGIN between two columns was
sliced away and dropped, so a<TAB>b c — a tab expanded across
the gap — became a two-cell row with the b simply gone. docutils
refuses the whole table for it
(Text in column margin in table line N.). Text past the LAST column
still extends it, which is a documented docutils feature, so that stays
a table.
The block is quoted AS WRITTEN: setup() rewrites the = borders to
- on a COPY there, and doing it in place quoted -------- where the
author typed ========.
The rest of the failure exits followed in v0.136.8, and the reason they
had stayed silent is worth naming: the docutils testsuite records an
EXCEPTION for each of them, because test_SimpleTableParser.py and
test_TableParser.py exercise tableparser directly rather than through a
document. The corpus extraction had no tree to record, so five cases were
skipped — and three of the five were real defects nothing was measuring.
Their inputs are ordinary documents, so a document-level expectation exists
and the corpus now carries it:
| detail | what it used to do |
|---|---|
Bottom border or header rule does not match top border. |
a paragraph, no diagnostic |
No bottom table border found. |
a paragraph, no diagnostic |
No bottom table border found or no blank line after table bottom. |
built a table the reference refuses |
Malformed table; parse incomplete. |
a paragraph, no diagnostic |
Multiple head/body row separators (table lines N and M); only one allowed. |
a paragraph, no diagnostic |
Two of find_head_body_sep's own errors are NOT ported, and the witnesses
say why: "the head/body row separator may not be the first or last line of
the table" cannot be reached through a document parse, because a +=====+
first line is not a grid-table top border at all (the block never becomes a
table), and a +=====+ last line is refused earlier as
Bottom border missing or corrupt.. Both were checked against the
reference rather than assumed.
Reporting these exposed a latent defect the silence had been hiding.
simple_table_top_pat is ANCHORED — =+( +=+)+ *$ — and this package
asked only "does the line contain two or more runs of =", which is true
of ordinary prose: we use u = Unicode object and s = Python string has
two. It cost nothing while a failed table attempt fell through quietly, and
the moment the first diagnostic appeared on that path it became a
Malformed table. ERROR in 68 real-world corpus files.
A table's geometry is measured in CODE POINTS, with East Asian
Wide and Fullwidth characters counting TWO (v0.109.0+ for grid tables,
v0.115.0+ for simple ones; eastasian.go).
Both halves of that matter. docutils is Python, so its line indexing is
per-code-point for free and … is one column; reading the same lines as
Go strings made it three, pushing every later | out of line with its
border and degrading the whole table to a paragraph — no diagnostic, no
partial table. But plain code points would be wrong the other way:
docutils pads Wide/Fullwidth characters before measuring
(StringList.pad_double_width) and strips the padding back out of each
cell, so a table aligned by code point but not by width is one it
REJECTS, and accepting it would trade a parse failure for a silent
disagreement. The rst package has no dependencies and Go ships no East
Asian width data, so the ranges are generated from the reference
interpreter's own unicodedata — the same table the corpus judge
consults:
python3 -c 'import unicodedata as u; \
print(*[hex(c) for c in range(0x110000) \
if u.east_asian_width(chr(c)) in ("W","F")])'
(eastasian.go stores that as sorted ranges; TestIsEastAsianWide
checks the range boundaries rather than their middles.)
The SIMPLE table needed the same thing and did not get it until
v0.115.0. Its borders are ASCII, so only the data rows were wrong — but
extendLastColumnForOverflow compared a row's BYTE length against a
column boundary counted in characters, so one accented letter anywhere
made the last column a column too wide, and a table full of them (PEP
3117 declares its types with mathematical symbols) made it several.
docutils pads simple tables exactly as it pads grid ones:
simple_table_top calls pad_double_width right where grid_table_top
does.
rst.ColumnWidth (v0.116.0+) is the OTHER column count, and the two are
deliberately different because docutils' are: a table's geometry comes
from pad_double_width, which doubles wide characters and says nothing
about combining ones, while utils.column_width doubles wide characters
AND subtracts combining ones. The second is what a title underline's
length is compared against (states.py:2888, :3147).
This package had that as "count the runes that are not unicode.Mn",
wrong twice over. It never doubled wide characters, so a CJK title got no
underline too short warning it had earned; and unicode.Mn is not the
set unicodedata.combining tests — they disagree on 1127 code
points, 1089 of which are in Mn with a combining class of zero. So the
combining ranges are generated from the reference interpreter too:
python3 -c 'import unicodedata as u; \
print(*[hex(c) for c in range(0x110000) if u.combining(chr(c))])'
The arithmetic is two independent steps, not a three-way choice: U+3099
is Wide and combining, so it is 2−1 = 1 column, and a switch that
treats combining as "no column" makes it 0. That was one character in a
1245-character sample against the reference, and the only one the first
attempt got wrong.
A gap that was measured, left open, then closed by asking what it
actually needed (v0.128.0). nodes.make_id lowercases, transliterates
through two tables, then NFKD-normalises and drops whatever is still
non-ASCII — so Ti中tle is title (the character leaves no separator
behind) while Ti…tle is ti-tle (… decomposes to ASCII dots, which
become one). This package turned every non-foldable character into a
separator, giving ti-tle for both, and the note here said telling them
apart "needs NFKD, which Go has no standard-library support for and which
this dependency-free package will not embed".
Both halves of that were true and the conclusion still did not follow.
make_id does not need NFKD; it needs the ASCII residue of NFKD,
which is a pure function of one rune, and the 2166 runes that have one
fit in 833 bytes of data (makeidfold.go, generated from the same
unicodedata 16.0.0 that generated eastasian.go's tables). The port is
now exact: a differential probe of 3992 inputs — one for every rune with
an ASCII residue, 1500 with none, and every printable ASCII character in
three positions — agrees with docutils on all 3992. The same table
removed three WRONG mappings the hand-written fold carried (ß to s
rather than sz, æ to a rather than ae, œ to o rather than
oe) and let class_option's two hand-rolled copies of make_id
become calls to it.
rst.TableColumnWidth exports that metric (v0.110.0+), because a WRITER
needs the identical rule: go-richdoc/rst padded its cells by rune
count, so a CJK cell overflowed its column and a table this package had
just parsed did not survive being written back out. Two implementations
of one rule is what put the byte-vs-code-point version of this bug here
in the first place.
The .. marker's own trailing whitespace is a RUN, not one space
(v0.108.0+, explicitMarkerRest): docutils writes it twice and both
spellings say one-or-more — the state machine's
explicit_markup: r'\.\.( +|$)' and then each of the five construct
patterns' own \.\.[ ]+. Reading it as exactly one space is silent
rather than noisy: every construct pattern simply fails against a rest
that starts with a space, and the line lands on the comment fallback, so
.. [#name] or .. note:: becomes a COMMENT and takes its whole body
out of the document. The marker's width is the marker's alone; body
indentation is still dedented by the first body line's own indent.
Not yet ported (see the rst, explicit.go/fieldlist.go/
lineblock.go/inline.go/table.go/gridtable.go doc comments for
the exact list and why): the "ends without a blank line; unexpected
unindent" WARNING real docutils' own unindent_warning produces for
Every construct that can raise "ends without a blank line; unexpected
unindent." now does (v0.117.0+, unindentWarning). Four were missing it:
a directive on its own line, a directive with an indented body, an
option list and a hyperlink target. The directive case is the one that
explains the shape of the fix — parseDirective had a dozen exits and
exactly ONE of them, the unknown-directive branch, emitted the warning,
so every directive the package actually implements returned silently. It
is now a wrapper with a single exit rather than a dozen places to
remember. The guard is shared too: another line of the same family
immediately after is not an interruption, so two consecutive targets
warn about nothing.
A uniform line of four or more characters only ends a paragraph as its
SECOND line (v0.117.0+). docutils' underline transition belongs to the
Text state, which a block occupies for one line; after that
get_text_block reads to the next blank line whatever the shape, so
Ti\ntle\n===== is one three-line paragraph where this parser produced
a paragraph and a <transition>. A permanent test had been pinning the
wrong answer, and was corrected against the reference.
EVERY explicit-markup construct (a directive, footnote, citation, ...)
that a following, insufficiently-indented line interrupts — this parser
has it for enumerated lists (parser.go, v0.25.0), bullet lists
(parser.go's parseBulletList, v0.47.0 — a genuinely missing case, not
just a wording gap: real docutils treats a DIFFERENT bullet character
as an outright stop, never a continuation, the same way ordinary text
does), footnotes, citations,
and comments (explicit.go's parseFootnoteOrCitation/parseComment,
v0.35.0/v0.40.0 — including the one real subtlety: two adjacent
explicit-markup constructs with no blank line between them are NOT an
abrupt unindent at all, real docutils chains a whole run of them
through one nested state machine and only warns once that chain is
broken by something that ISN'T itself another explicit-markup line,
Body.explicit_list, read directly), literal blocks (parser.go's
tryLiteralBlock, v0.34.0), line blocks, definition lists, field
lists (v0.37.0/v0.38.0/v0.39.0), and topics/sidebars (topics.go,
v0.44.0 — including a NESTED topic/sidebar's own rejection diagnostic,
which used to always report line="1" regardless of its real position
in the document, an index local to the outer topic/sidebar's own
rebased content rather than the real document; runTopicOrSidebar's
own doc comment has the derivation for computing a real absolute line
for topic/sidebar content specifically). The "nested constructs report a
placeholder line" claim this paragraph used to make is no longer true, and
what replaced it is a MEASUREMENT rather than a claim: a differential probe
of 348 cases — 29 diagnostic-producing snippets crossed with 12 nesting
wrappers (block quote, bullet item, enumerated item, definition and field
bodies, note, topic, sidebar, footnote body, and three nested-twice
shapes), at /Users/Shared/rstcorpus/nestprobe — agrees with the reference
on 347 of them. Neither corpus contains ANY of these shapes, which is why a
probe was needed at all: both corpora read the same before and after every
fix it found. A diagnostic about a
construct's own missing content names the LAST LINE THE BLOCK CONSUMED,
which for a blank-terminated block is the blank line itself; since
docutils consumes and counts trailing blank lines too (.. [c]\n\n
reports line 2, .. [c]\n\n\n reports line 3), splitLines splits on every boundary Python's
str.splitlines() recognizes — CR, LF, VT, FF, the three C1
file/group/record separators, NEL, and the two Unicode separators
LINE SEPARATOR (U+2028) and PARAGRAPH SEPARATOR (U+2029) — not just
\n, since that is what docutils' string2lines is built on
(v0.62.0+). It drops only
ONE trailing empty element — the artifact of the final newline, exactly
Python's str.splitlines() — rather than every trailing blank as it
used to (v0.53.0; stripping them all cancelled an off-by-one, so these
line numbers had been coming out right for the wrong reason). Docutils'
standalone PEP/RFC recognition —
bare pep-123/RFC 123 text with no :PEP:/:RFC: role markup at all
(checked against Parser().parse() with default settings — docutils'
own pep_references/rfc_references settings default to off, so
implementing this unconditionally would diverge from upstream's own
default rather than fill a real gap; the :PEP:/:RFC: roles
themselves ARE ported, see above), and — see
Options.ReportUnknownRoles, which defaults TRUE — an unknown
interpreted-text role's rewrite to problematic, the leniency this
paragraph used to describe as permanent now being the flag's false
side, for a consumer that would rather keep a Sphinx :doc: reference as
text.
Eleven role names are KNOWN and still error, which is a third case
neither "unknown" nor "implemented" (v0.136.0+): docutils binds
anonymous-reference, citation-reference, footnote-reference,
index, named-reference, substitution-reference, target,
uri-reference and the aliases i, uri, url to
roles.unimplemented_role, so the lookup SUCCEEDS and the role then
raises Interpreted text role "..." not implemented. with no lookup INFO
in front of it. This package knew one of the eleven and rendered the rest
as a bare <inline role="...">, silently accepting :index:`word`,
which three sphinx corpus files write. They are gated by
ReportUnknownRoles even though, strictly, the lookup succeeds and it is
the role FUNCTION that errors: a consumer turning that flag off wants the
same outcome for both, :index:`word` rendering as "word" rather
than as the raw :index:`word` a <problematic> carries. The twelfth name,
restructuredtext-unimplemented-role, is the exception that says the set
cannot be derived from the registry alone: it is in the registry and NOT
in the language module, so its lookup misses first and it draws the INFO
too.
The unknown-role leniency covers the role being UNRECOGNIZED, not the
role markup being malformed: the two syntax
errors docutils raises in interpreted_or_phrase_ref before it
resolves the role at all ARE ported (v0.54.0+) — a role in both
positions at once (:a:`x`:b:,
"Multiple roles in interpreted text (both prefix and suffix present; only one allowed).") and a role combined with a reference suffix
(:a:`x`_, "Mismatch: both interpreted text role prefix and reference suffix.", the message naming whichever position the role was
in). Both were verified against the reference for :emphasis:, an
ordinary built-in, so neither depends on the role being unknown; the
suffix used to be silently dropped as leftover plain text. A role NAME
is docutils' own simplename, so ., _, + and : are all legal in
it (:very.long-role_name:`x`, and :a:b:`x` is greedily the
single role a:b) — this used to scan letters/digits/- only, missing
such a role entirely and leaving it as plain text beside a bare
<title_reference>. An unresolved reference stays a bare <reference> carrying its
refname, matching docutils' own bare parse; the rewrite to
<problematic> plus a trailing "Docutils System Messages" section is
its DanglingReferences + Messages TRANSFORMS, which run after the
parser and which a caller may never run at all. They are available via
Options.ReportDanglingReferences, off by default (v0.66.0+).
Whether a reference that DOES resolve gets its refuri is a separate
question and a separate option, Options.ResolveReferences, also off
by default (v0.80.0+). That resolution is itself a transform
(docutils' references.Hyperlinks, via ExternalTargets/
InternalTargets), so a bare parse leaves even a resolvable reference
carrying only the refname it points at. Turning it on fills in the
refuri — an external target's URI, or #its-id for an internal one —
and any renderer wants it: the bundled html and latex writers'
own tests set it, and so does go-richdoc/rst. Until v0.80.0 this
was unconditional, documented as what made the package usable without a
transform pipeline; the refname is written either way, so a consumer
could always resolve for itself, and the default now matches every
other transform-derived option instead of being the one exception.
The anonymous attribute is spelled "1",
not "true" — it is a boolean in docutils and starttag renders one as
str(int(value)) (v0.67.0+). Standalone URI recognition (no backtick
quoting or trailing _ needed at all, inline.go) uses docutils' own
two character classes rather than a punctuation heuristic (v0.63.0+):
the body class [-_.!~*'()[\];/:@&=+$,%a-zA-Z0-9] contains neither <
nor >, so a URI inside angle brackets ends at the bracket, and the
final character must be one of [_~*/=+a-zA-Z0-9] OR any body character
followed by > — which is why <http://example.org/x.> keeps a
trailing stop that see http://example.org/x. loses. Known
divergence: a path segment ending in _ (http://a/b_) is one URI
here, where docutils splits it into http://a/ plus a bare reference
b_, its reference pattern winning the tail; matching that means
modelling pattern ALTERNATION order, not just character classes. The SCHEME must be one docutils recognizes
(utils.urischemes, 108 entries, transcribed) — its own fixture states
the rule: "None of these are standalone hyperlinks (their 'schemes' are
not recognized): signal:noise, a:b." — and the hierarchical part is
(//?)?, so two slashes, one, or none all work (http://x,
http:/one-slash, mailto:, news:), v0.76.0+. An email's HOST is
[chars]+ plus a separate final-character group, so it needs at least
two characters and must end on one of [_~*/=+a-zA-Z0-9]: user@host
and a@b-c are addresses, a@b is not. This used to match — real docutils' own URI pattern also
accepts a bare "scheme:path" with no "//" at all (mailto:, news:,
urn: and friends), not yet ported; the SAME schemes work fine as an
EMBEDDED URI inside a phrase reference (`text <mailto:x@y.com>`_), where this limitation doesn't apply. Directive-level argument/option-syntax
validation is likewise not implemented for ANY directive (an
unresolvable role base, an invalid option value, a malformed directive
argument list, a required argument missing entirely — real docutils
raises a distinct diagnostic for each; this parser's own directives,
raw/table/list-table, silently ignore an option they don't
understand rather than erroring, matching the same leniency choice as
the unknown-role case just above) — a real, general gap, not chased
per-directive as each one gets implemented. Title-style consistency (a title style's LEVEL
is fixed by first-seen order across the whole document; skipping more
than one new level at once is an error) and section-title diagnostics
(too-short overline/underline, missing/mismatched underline, incomplete
title, invalid title-or-transition) ARE enforced — matchTitle/
titleDiagnostic/checkSubsectionLevel, rst/parser.go; the
match_titles=false case (a title-looking construct somewhere titles
aren't allowed, e.g. inside a block quote) and enumerator-sequence
validation (docutils errors on a non-consecutive ordinal) are the two
pieces still NOT ported. Paragraph gathering now matches real docutils
(v0.60.0+): once a line has become ordinary paragraph text,
Text.text's get_text_block swallows every subsequent line up to the
next blank line or dedent, never re-examining a LATER line to see
whether it independently looks like a different construct. docutils'
Text state has exactly four transitions — blank, indent, underline,
text — and no bullet/enum/field/explicit-markup/doctest/line-block/table
transition at all, so text\n- item, text\n.. comment,
text\n:field: v, text\n| line and text\n1. one are
each ONE paragraph. consumeParagraph used to break on every one of
those shapes; the checks could only ever fire MID-paragraph, which is
exactly where docutils does not look, since a construct at a genuine
paragraph boundary is preceded by a blank line and handled before this
function is reached. A title underline still splits, because underline
IS one of those four transitions. A table's column-margin violations are never detected
(only the "last column overflows its width" case is handled, since real
content relies on it). A block quote's own indent is discovered the same
way real docutils' StringList.get_indented does: the MINIMUM across a
whole (possibly variable-depth) indented run, not the first line's own
indent — a deeper-then-shallower run correctly NESTS instead of producing
sibling block quotes. A trailing "-- text" / "--- text" / em-dash-prefixed
attribution line, preceded by a blank line and internally
consistently-indented, becomes a real <attribution>, splitting the
region into one <block_quote> per attribution boundary — split_attribution
check_attribution, ported; the diagnostics real docutils emits for a malformed attempt (an inconsistent-indent continuation, an unindent with no blank line first) are deliberately NOT ported, the same scope boundary as title-style/table-column diagnostics just above. A dangling NAMED reference (bare, backtick-quoted, or an embedded indirect alias — matched by name and found nowhere) IS rewritten to<problematic>, with every such message collected into a trailing<section class="system-messages">, docutils' own DanglingReferences + Messages transforms, simplified: no duplicate/ambiguous-name diagnostics, and<problematic>'s content is the reference's own visible text rather than real docutils' verbatim source slice (this parser doesn't track original source text on a node at all). An ANONYMOUS reference/target count mismatch IS covered too — real docutils checks this as a single whole-document condition, not per-reference (AnonymousHyperlinks.apply, read directly): if the counts don't match EXACTLY, in either direction, every anonymous reference in the document becomes<problematic>, all sharing ONE message. An unclosed inline-markup start-string (*x,**x, two backticks with no closing pair, a bare interpreted-text backquote) IS rewritten to<problematic>too —Inliner.inline_obj, ported — a genuinely SEPARATE source from the dangling-reference/anonymous-mismatch cases above: this one fires during inline PARSING itself (inline.go), not a whole-document post-pass over an already-built tree, and its<system_message>is attached directly as a sibling of whatever it came from (a paragraph, a section title, a block quote's attribution, a field name's body, a definition list term, a line block line) at that construct's own point of origin — real docutils'RSTState.paragraph/new_subsection/parse_attribution/field/line_block_line/termallreturn/+=these messages as siblings, states.py read directly — NEVER collected into the trailingsystem-messagessection above.
Where "sibling" stops being the rule is a directive's own TITLE argument,
and docutils decides it PER CONSTRUCT rather than generally (v0.136.11).
BaseAdmonition.run does admonition_node += title and then
admonition_node += messages, and only then parses the content into that
node, so the messages sit between the title and the body.
Topic.run builds node_class(text, *(titles + messages)) — title,
subtitle when there is one, then every message from both. But Table.run
really does return [table_node] + messages, genuinely siblings, and so
does Rubric. This package applied the table's shape to all of them, which
put an admonition's, topic's or sidebar's title messages outside the element
entirely; a 930-case inline probe (30 inline constructs × 31 containers,
/Users/Shared/rstcorpus/nestprobe/inlineprobe.py) found exactly those
three and left the tables and the rubric alone, which is why the tables are
now pinned as controls rather than "fixed" too.
The same probe found the other half of the attribution rule:
parse_attribution passes 1 + line_offset — the attribution's OWN first
line — to inline_text, and this package passed zero, so every inline
diagnostic in an attribution came out with no line attribute at all. Where
the messages GO was already right (Body.block_quote does
elements += messages, so they are the block quote's siblings). The probe
now agrees with the reference on all 930 cases. That section is real docutils' own
transforms.universal.Messages, and it wraps only messages with no
parent at all (if not msg.parent, read directly) — an inline-markup
message already has one the moment it's attached, so it is categorically
excluded, unlike the dangling-reference/anonymous-mismatch messages
(built via document.reporter.error with no tree insertion of their
own, hence genuinely parentless). A substitution_reference ("|x" with
no closing "|") routes through the identical real-docutils mechanism
(inline_obj) and DOES produce this warning — v0.36.0 corrected an
earlier version of this note, which claimed otherwise based on limited
testing that happened not to exercise a substitution reference with no
valid closing "|" anywhere in the text at all (|sub|ref, the closing
"|" immediately followed by a word character); verified directly
against the live reference implementation, not just the corpus, before
trusting the correction.
Whether a */**/two-backtick/backquote/| counts as a genuine markup
start- or end-string at all is docutils' own start_string_prefix/
end_string_suffix rule, ported verbatim (punctuation.go) — not "any
punctuation on either side", this parser's own earlier, simpler
approximation via Go's unicode.IsPunct, which treated an opening and a
closing bracket/quote identically. Real docutils distinguishes them: a
start-string may be preceded only by whitespace, an OPENER, or a
DELIMITER ((*emphasis*), -*emphasis*-), never a CLOSER or a
CLOSING-DELIMITER ()*emphasis*( is NOT markup at all); an end-string may
be followed only by whitespace, a CLOSER, a DELIMITER, or a
CLOSING-DELIMITER (*emphasis*., *emphasis*)), never an OPENER
(*emphasis*( is unclosed, not valid markup). A markup start-string
immediately sandwiched between a real matching open/close pair with
nothing else between ((*)text) is additionally rejected —
Inliner.quoted_start, ported — even though the basic rule alone would
accept it. The four character classes (openers/closers/delimiters/
closing-delimiters) and the open/close quote-pairing table are
docutils.utils.punctuation_chars verbatim, covering the full range of
quotation-mark conventions real docutils itself supports (French, German,
Polish, Hungarian, Greek, CJK, ...) — generated from a live Python
reference and cross-checked against every Unicode code point, not
hand-transcribed (a first attempt using literal glyphs silently
substituted at least two visually-similar-but-distinct characters). A
backslash-escaped space or newline immediately after an end-string is
itself a valid boundary (docutils' own \x00 escape-marker, which
end_string_suffix explicitly allows) and is dropped from the resulting
text entirely rather than rendered as a literal space — docutils' own
unescape(), "backslash-escaped spaces are also removed" — a real bug
this parser had for every kind of escaped whitespace, not just at a
markup boundary, only exposed once markup started closing correctly for
inputs that exercise it. The interpreted-text backquote and the
substitution-reference | each allow an optional trailing _/__ to be
consumed by the SAME end-string-suffix check as part of resolving them as
references (`text`_, |sub|__) — a separate regex in real docutils
from the generic emphasis/strong/literal one, ported as its own boundary
check (findCloseBackquote/findCloseSubstitution) rather than folded
into the shared one, since only these two constructs have it. An unknown
interpreted-text role still stays a plain node instead — deliberately,
see above. A resolved
embedded-link reference doesn't get the extra <target> sibling node
docutils emits alongside it (this parser sets refuri/refname directly
on the <reference> instead; resolution still works the same way since
it's all done by matching names, just without that second node) —
genuinely unported transforms, unlike target/anonymous resolution,
footnote numbering, docinfo promotion, and dangling-named-reference
rewriting above, each a deliberately simplified PORT of the
corresponding real transform, not a parser-level gap (verified by comparing
against Parser().parse(src, document) directly, before docutils' own
transform pipeline runs, not publish_string's fully-transformed
output). A literal span ( ) RESTORES its backslash escapes
instead of stripping them — the ONLY marker with this behavior:
Inliner.literal calls inline_obj(..., restore_backslashes=True),
which (nodes.unescape, read directly) puts the literal backslash back
rather than dropping it, so \literal keeps its own visible
backslash (\literal) where every other construct (emphasis, strong,
a role's own interpreted text, ...) would silently drop it. A
backslash in a literal does not protect the CLOSING backquotes either
(v0.56.0+): docutils spells the two end-strings differently on purpose —
(?<![\s\x00])(\*) for emphasis against (?<!\s)(``) for literal, with
no \x00 in the literal form's lookbehind at all — so literal` `` closes normally and keeps the backslash as content, and `` ``a\\`` `` keeps BOTH backslashes. That is the spec's "backslashes are not escapes inside inline literals" made mechanical, and it is the one place a marker's close must IGNORE an escape this parser otherwise honors everywhere (findCloseLiteral). General marker start-boundary recognition immediately after a backslash-escaped space or newline (m\ a\ r\ k\ u:title:— each construct's own escaped-space prefix now counts as valid preceding whitespace,docutils/rstv0.33.0+) IS ported —validStartBoundaryunescapes a neighboring rune before its own character-class checks, the same "real docutils'\x00-substitution keeps the ORIGINAL character intact for structural matching" principle joinEmbeddedURIalready relies on for embedded links (v0.31.0). An escaped character that would otherwise complete a MULTI-CHARACTER CLOSE delimiter specifically (an escaped first backtick immediately followed by a second, real one — `` ``text\`` `` — real docutils'\x00-substitution still counts the escaped character's own identity toward the "``" pair, only suppressing the backslash from the final text) remains a known, NOT yet ported gap: unlike the start-boundary fix (a single-rune lookbehind, straightforward to unescape), a close search needs to treat ONE escaped rune as if it were TWO characters (docutils' own \x00marker byte plus the original character, occupying two string positions there — this parser'sescapeRune`
collapses both into one rune) to correctly recover the restored
backslash in the surrounding text, not merely recognize the character
identity — deliberately not attempted without dedicated time to get
that representational mismatch right.
Every <system_message> this parser builds now carries level/type
(constants per message kind — inline-markup-time messages are always
level="2" type="WARNING", matching Inliner.inline_obj's
self.reporter.warning(...) exactly, since "start-string without
end-string" is the only text that function ever produces) and, for a
PARAGRAPH or a SECTION TITLE at the top level of the document (i.e. not
nested inside a list item, block quote, field body, definition, or
table cell), line too — the construct's own real 1-indexed source
line, matching real docutils' RSTState.paragraph/new_subsection
exactly (verified against the foreign judge for both an underlined and
an overlined title, where the reported line is the title TEXT's own
line, never the overline's). A message from any OTHER inline-parsing
context, or from a nested one, still omits line — this parser doesn't
track absolute source position through recursion into a rebased
sub-slice of lines (a list item, a block quote's body, a field's body,
a definition, a table cell) anywhere yet, a genuinely separate,
larger undertaking (see parser.currentLine's own doc comment in
parser.go) than threading it through the two top-level call sites
this round covers. A table's <tgroup cols="N">/<colspec colwidth="W"> wrapper
IS produced (verified: it's part of the bare parse, not a later
transform or writer-side addition, unlike the above), the html/
latex writers and go-richdoc/rst all unwrap it transparently.
Every section title IS registered as an implicit hyperlink target too
(docutils' own new_subsection/create_id, ported): a `Some Title`_
reference resolves to a same-document anchor derived from the title, the
id a plain-ASCII slug (accents folded, everything else stripped) and the
name the whitespace-normalized title text — both writers emit the id
as a real anchor (<section id="..."> in HTML, a bare \hypertarget{id}{}
right before the sectioning command in LaTeX). Two sections sharing a
title get distinct ids (title, title-1, ...) with no ambiguous-name
diagnostic, the same "no duplicate/ambiguous-name diagnostics"
simplification as dangling-reference rewriting above.
Section-title recognition and its diagnostics are ported from real
docutils' Line/Text states (states.py, read directly), not just
the well-formed case: a title inset under its overline (leading
whitespace on the title line) is stripped rather than rejected, but
still counts toward the overline-width comparison exactly as docutils
computes it (before the strip, not after — an inset title can trigger
"Title overline too short." on its own even when the stripped text
alone would fit); an adornment under 4 columns is
NOT wrong by itself — ===/One/=== is an ordinary section title
(v0.82.0+). Its length only matters once something ELSE has already
failed: Line.text and Line.underline reach short_overline from
inside their failure branches only, so a short overline that is also
narrower than its title, or has a mismatched underline, or none at all,
or is followed straight by a second adornment, reverts the whole attempt
to plain text with an INFO notice. At 4 columns and up the SAME three
failures are a WARNING ("Title overline too short.", section still
created) or an ERROR (missing, mismatched, or absent-at-EOF underline,
and two overlines with no title text between — no section created).
That is a threshold, not a blanket rule, and checking the length first
suppressed every well-formed short title.
Which handler runs at all is decided by PATTERN, on the line AFTER the
overline, and 'underline' is tried before 'text'
(Text.initial_transitions, inherited by SpecializedText and so by
Line): a uniform punctuation line therefore always reaches
Line.underline — the "Invalid section title or transition marker"
ERROR, or short_overline's rewind under 4 columns — and the three-line
Line.text path never even looks at it (v0.136.7). Testing "is line i+2
an underline equal to line i" first, which this package did, is true of
three IDENTICAL adornment lines: ====/====/==== became a <section>
whose title was ==== with the rest silently dropped, and
.../.../... swallowed its own third line. Only an equal first and
third line reach it, which is why ====/----/Title was always right.
What happens AFTER the demotion is docutils' "bubble-up" (v0.84.0+):
short_overline calls previous_line(), so the demoted line goes back
into the Body state as ordinary text — where it can be a title's OWN
text, underlined by the very line that was rejected as its underline a
moment ago. ... over ... is therefore an INFO and a section titled
..., not a two-line paragraph. The same re-entry applies to the
match_titles=False errors: the line after a reported adornment starts
a fresh block there too, so -----/Title/----- inside a block quote
is the adornment's own error AND "Unexpected section title." for the
title under it, rather than one error and a stray paragraph. Title-STYLE consistency is enforced too: a style's
level is fixed by the order it's first seen in the whole document
(title_styles, ported), reusing an established style returns to that
level (closing any deeper-nested sections), and introducing more than
one new level at once — skipping a level — is
"Inconsistent title style: skip from level X to Y.", an ERROR with no
section created. A numbered line ("1. Numbered Title") is
disambiguated from a genuine enumerated-list item by peeking at the
following line (is_enumerated_list_item, ported): blank, indented, or
starting with the next ordinal's own marker confirms a real list item;
anything else — most commonly a title underline — corrects the
enumerator-looking line back to plain text, letting the title win.
The match_titles=False diagnostics ARE ported (v0.59.0+): inside a
block quote, a list item, a table cell — anywhere that is not the
document or a section — neither a section title nor a transition is
allowed, and each becomes an ERROR carrying the offending source as a
<literal_block> ("Unexpected section title." for an underlined
title, both of whose lines are consumed; "Unexpected section title or transition." for a lone adornment, which used to become a
<transition> regardless). An adornment too short to be either (under
4 characters, ANY length down to one) draws
"Unexpected possible title overline or transition. Treating it as ordinary text because it's so short." and is then treated as text.
All of this applies only at the START of a block: an adornment-looking
line in the MIDDLE of a paragraph is folded into that paragraph with no
diagnostic at all, which is what real docutils does and what an
unguarded first version of this got wrong. Still not ported: enumerator-
sequence validation (docutils warns on a non-consecutive ordinal; this
parser doesn't check).
Docutils' RSTState.unindent_warning — one method raising
"<X> ends without a blank line; unexpected unindent." from nine call
sites — is ported for all of them but Option list, which no corpus
fixture reaches: Block quote (v0.59.0+), Bullet list, Enumerated list,
Field list, Explicit markup, Definition list and Literal block. Nested content now
threads a REAL absolute line number rather than the "unknown"
placeholder, so a diagnostic raised inside one carries a line at last:
block quotes (v0.59.0+), and definition bodies, field bodies, option
descriptions and bullet/enumerated list items (v0.61.0+). The
correspondence is exact wherever the sub-slice comes from
consumeIndentedBlock or gatherListItemLines, since both only dedent,
so entry k is the parent's line i+k — the same derivation v0.44.0
made for topic/sidebar content.
A message raised during INLINE parsing does not follow that derivation at
all, and v0.132.0 is where this stopped being guesswork. docutils binds
Reporter.get_source_and_line ONCE, to the document-level state
machine (RSTStateMachine.run sets it; NestedStateMachine.run does
not), so a message with no line of its own and no already-parented
base_node — which is every duplicate-name notice, since the <target>
it describes is built before it is attached — reports wherever that
machine happens to sit. Three rules cover it, and none of them mentions
the nesting depth:
- outside any nested construct, including inside a section (docutils
parses a section's body with the document machine itself): the
paragraph's own
max(first+1, last); - inside one: frozen at the last line of the block the document-level dispatch collected, however deep the message is raised — a duplicate in the middle of a long block quote reports the quote's LAST line;
- and a list hands only its FIRST item to that dispatch (the rest go to a nested list machine), so every message anywhere in a list reports the end of item one.
This file recorded the whole family as "not derivable" for many rounds.
It became derivable by TRACING the reference — wrapping
get_source_and_line and reading its line_offset/input_offset back —
rather than by reading states.py again: 26 shapes, all 26 agreeing, and
fourteen real-world corpus files. A table cell is the one shape whose
agreement is still a coincidence rather than a derivation.
Those two, and the block-quote gatherer, used to trim trailing blank
lines as well (v0.131.0 stopped them, for what that cost see below).
StringList.get_indented trims nothing from the end — it stops at the
first insufficiently indented NON-BLANK line — and the blank lines after
a nested construct therefore belong to the block that quotes it. An
unknown directive's <literal_block> shows one blank line fewer than
were written (two written, one shown: the last is the text's own
trailing newline), and inside a list item, a field body or a block quote
this package showed none at all. Three real-world corpus files, and the
whole cost of the change was ONE emptiness test: an option marker with
nothing after it (-f alone) now yields a slice holding one blank line
instead of an empty one, and option_list read that as a description.
allBlank, not len() == 0, is the test that does not depend on how
many blank lines the gatherer happened to absorb. A CSV-TABLE cell is no longer among them (v0.136.3): a diagnostic raised
inside one reports the directive's content OFFSET — one less than the first
content line's number — plus the line index WITHIN THE CELL, and nothing
for the row, because docutils hands each cell to nested_parse as its own
StringList starting at zero. Two cells in different rows therefore report
the SAME line while two lines of one quoted cell report different ones,
which is why the rule had to be traced rather than assumed; that is exactly
a line base of contentOffset - 1, so the cells needed no machinery, only
the base this had been passing as "unknown". Still placeholder: a GRID
or SIMPLE table cell (whose content is genuinely not a contiguous parent
slice) and any directive whose body goes through
parseDirectiveBlock, whose fold-back branch breaks the
correspondence.
Directives whose real work happens in a later TRANSFORM parse to a
<pending> placeholder carrying the transform's name and options
(pending.go, v0.64.0+): class/rst-class
(misc.ClassAttribute — but ONLY in its contentless form: given CONTENT
it parses that content and adds the classes to each resulting top-level
node, returning those nodes and no placeholder at all, v0.74.0+), sectnum/section-numbering (parts.SectNum)
and target-notes (references.TargetNotes). This project still has no
transform system and is not getting one — but that was never what these
needed. A parse tree holds the placeholder; nothing has to run, and
nodes.pending.pformat gives its exact shape (details SORTED BY KEY,
values in Python repr form). title is the odd one out: it sets the
document's own title ATTRIBUTE and leaves no node behind at all.
class and title are also the two whose option_spec is None, and
parse_directive_block only looks for options when there IS one — so
their ARGUMENT is the whole block up to the first blank line, and a
field-marker-shaped line inside it belongs to the argument (v0.136.3).
sphinx's extdev/appapi.rst writes .. class:: Sphinx with :no-index:
on the next line, which is the two classes sphinx and no-index; this
read the second line as CONTENT and built a field list out of it, and with
a plain word there it produced a second paragraph beside the real one. The
blank line is what separates the two readings, so the runner is told how
many followed the directive — gatherExplicitBody drops them, and without
that count an argument continuation and a content block look identical.
That argument then goes through directives.class_option, which is one
nodes.make_id per whitespace-separated token and raises ValueError on
any token whose id comes out EMPTY — caught into
Invalid class attribute value for "class" directive: "...", quoting the
WHOLE argument as written, not the offending token (v0.136.5). This
package used to drop the empty ids and apply whatever was left, so
.. class:: 1 produced a class-less block instead of an error. What the
check must NOT do is reject punctuation make_id merely folds away:
sphinx's two cpp-domain pages write .. class:: template<typename T, and
those are the classes template-typename t int n.
target-notes validates its options (v0.74.0+) — an unknown one, or one
given with no value, is docutils' own
Error in "target-notes" directive: .... That was portable for this
directive alone when it was written, because its whole option_spec is a
single :class: entry; the GENERAL per-directive validation described
further down arrived in v0.133.0 and now covers seventeen. Still not
dispatched: contents and date.
include is implemented as of v0.138.0. raw's :file:/:url: forms are
still a deliberate non-target.
An inclusion SPLICES the file's text into the input being parsed, which is
what makes it an inclusion rather than a graft: a section title in the
included file nests under the including document's sections, a construct is
not cut at the file boundary, and ids are assigned in document order.
Parsing the file separately and attaching its nodes gets all three wrong.
:literal:, :code: (with :number-lines:), :start-line:,
:end-line:, :start-after:, :end-before:, :tab-width:, :class: and
:name: are all ported, each checked against what the reference answers for
the same input.
It needs a path, and without one it does nothing. Options.SourcePath
is the document's own path; a relative include resolves against its
directory, and a NESTED include against the directory of the file it is
written in — the reference resolves every include against
document.current_source, which follows the inclusion. With no SourcePath
the directive is disabled, which is the safe default rather than an
omission: a caller that handed this parser a string has not asked it to read
the filesystem, and docutils gates the same directive behind its own
file_insertion_enabled setting for the same reason. Disabled, it reports
the reference's own "include" directive disabled. warning — unless
ReportUnknownDirectives is off, in which case it falls back to the
structural capture like every other directive this parser cannot act on, so
a consumer that converts documents without a path (go-richdoc/rst does)
still sees the directive rather than a diagnostic.
Like csv-table, include is absent from optionspec.go's transcribed
table, so an unknown option KEY on it is ignored rather than reported. That
is the same documented gap, not a new one.
A circular inclusion is a warning naming the whole chain, and the
mechanism is the reference's own: the directive appends an
end of inclusion from "..." comment to the lines it splices, and that
comment pops the log and produces no node. It matters because the simpler
rule — "refuse a file already included" — would also refuse the SAME FILE
TWICE IN SEQUENCE, which is legal.
Three things are NOT ported, each for a stated reason: :parser:, which
runs another docutils parser over the content and has no meaning here; the
<name> form, which reads from docutils' own bundled include directory
(isonum.txt and friends), since this package bundles no such files and
answers with the same error a missing file gets; and the line-length limit,
which this parser has no setting for. And one limitation is real: a message
raised inside included text carries the line number of the SPLICED input,
not of the file it came from. The reference has its own version of that
problem and says so in a TODO.
The real-world corpus is at 1564 of 1564 as of v0.138.0, up from 1553.
The eleven that moved all use .. include:: on a file the corpus's flat
arrangement does not have, and both implementations now report the same
error about it. One normalisation was needed in the sweep to see that, and
it is worth stating: the path inside that message is printed RELATIVE TO THE
PROCESS'S WORKING DIRECTORY — directives.adapt_path ends with
utils.relative_path(None, ...) and says why ("convert to relative path for
shorter system messages") — so its spelling depends on where the run was
started, not on the parser. The recorded judgement was generated from a
different directory than the sweep runs in, which made those eleven differ
over five leading ... The sweep compares the basename, and a unit test
holds the other half: a DIFFERENT file still differs.
The testsuite corpus is at 579 of 579. The one mismatch that stood
there until v0.138.0 was the judge's, and the correction is worth recording
because the earlier diagnosis in this README was wrong. .. role:: custom
calls roles.register_local_role, which writes into a module-level
roles._roles — process-global state that docutils' own test framework
resets between cases. The script that regenerates each case's expectation
with default settings ran all 579 in ONE process without clearing it, so a
case defining :custom: with :class: custom-class leaked into the NEXT
case, whose whole subject is using a role BEFORE defining it: its
regenerated expectation became <inline classes="custom-class"> where the
docutils test file itself expects <problematic> — which is what this
package produces. Clearing the registry per case changed exactly one
expectation out of 579, which is also the evidence that the leak was hiding
nothing else. This parser's own role registry is per-document and always
was, for the reason its comment gives: no cross-document leakage between
concurrent Parse calls.
Five Options fields decide whether a docutils behaviour that is NOT
part of parsing happens at parse time here, and each default follows
one rule rather than taste: does the reference implementation do this in
the PARSER, or in a TRANSFORM?
| option | default | because |
|---|---|---|
ReportUnknownDirectives |
true | Body.unknown_directive is in the parser |
ReportUnknownRoles |
true | Inliner.interpreted is in the parser |
ReportUnknownDirectiveOptions |
true | Body.parse_directive_options is in the parser |
ReportDanglingReferences |
false | DanglingReferences/Messages are transforms |
PromoteDocInfo |
false | DocInfo is a transform |
NumberAutoFootnotes |
false | references.Footnotes is a transform |
ResolveReferences |
false | references.Hyperlinks is a transform |
Three more options are a different kind and not one of those seven: they do
not decide whether a behaviour happens. SourcePath supplies the one piece of
information an inclusion cannot work without (empty disables include),
IncludeRootPrefix confines an absolute include path (docutils' own
root_prefix), and LineLengthLimit is the reference's denial-of-service
guard, 10000 by default and 0 for none. See Security below.
The third one was added in v0.107.0 after MEASURING what faithfulness
costs a renderer rather than assuming it was free: 32 of the 1564
real-world corpus files carry a sphinx-only option on a code block or an
equation (:caption:, :emphasize-lines:, :label:, :linenos:), and
each one loses the whole block to an error paragraph. That is right for
a parser and wrong for a converter, which is exactly what an option is
for.
A consumer picks per field. go-richdoc/rst turns the three report
options OFF (a diagnostic aimed at someone WRITING reST becomes
fabricated content once it reaches a converted document) and
PromoteDocInfo, NumberAutoFootnotes and ResolveReferences ON (its
Document.Meta is built from the first, it renders footnotes, and its
links need somewhere to point). One thing stays unconditional either
way, because it is assigned during PARSING even in docutils: every
footnote gets its id — note_autofootnote calls set_id, so an auto
footnote is footnote-1 with no label at all, and auto and symbol
footnotes share one positional counter.
A directive name is docutils' own simplename (v0.90.0+):
alphanumeric runs joined by -._+:, which is the same grammar
scanSimpleName already applied to ROLE names. The directive path had
its own character class, missing : entirely, so every namespaced
directive — .. rst:directive::, .. py:function::, the shape Sphinx
documentation is full of — fell through to the comment fallback instead
of being reported as an unknown directive. 49 real-world files, and no
docutils testsuite fixture at all.
docutils resolves a written directive or role name through its
language module (languages/en.py) BEFORE the registry — which is
what the lookup-failure INFO means when it names that module by path.
Its English table has four directive aliases and eleven role ones; the
roles were all already present here, rst-class/section-numbering
were handled at their own call sites, and code-block/sourcecode were
missing until v0.88.0. rst-class deliberately stays out of the shared
helper: runClassDirective distinguishes class from its alias by the
name as WRITTEN, so canonicalizing it would erase a real difference.
The space after a field marker is a SEPARATOR, never indentation
(v0.134.0+). :Description: value — two spaces, which is how a PEP
aligns a column of field values — made the value an indented block here,
so it became a <block_quote>, and a continuation line after it drew a
spurious "Block quote ends without a blank line" warning that split one
field body into three nodes. docutils arrives at the plain paragraph from
the other side: get_first_known_indented cuts the first line at the
marker and trims the block by the minimum indent of the lines AFTER it,
so whatever the first line has left over never sets an indent for
anything to be measured against — a continuation indented DEEPER than the
marker joins the same paragraph, which is the case that says this is not
about alignment at all.
An embedded URI gets mailto: only when the WHOLE target is an email
address (v0.125.0+), which is what adjust_uri tests — its email
pattern, anchored with $. The heuristic this replaced ("contains @,
does not contain ://") was wrong in both directions: it prefixed
mailto:core@pytest.org AGAIN, so pytest's own contact page pointed at
mailto:mailto:core@pytest.org, and it prefixed a@b, which docutils
leaves alone because the host half needs two characters. The grammar
settles both without a special case, since emailc excludes :.
A <problematic> node quotes the source AS WRITTEN (v0.123.0+):
docutils builds its text from the rawsource with
unescape(..., restore_backslashes=True), so :file:`PC\\python_uwp.cpp`
keeps both backslashes. Three sites passed the runes still carrying
escapeBackslashes' private-use encoding, so U+F005C — a shifted
backslash, not a character any document contains — went straight into
the tree (PEP 773 came out with one). tryURIScheme had carried a note
about that leak since v0.31.0; these were the sites it did not cover.
A scheme whose path is EMPTY is still a URI (v0.123.0+) when it has a
slash: urilast includes /, so file://, http:// and file:/ end
on a valid final character, while mailto: and news: — a scheme with
no slash and nothing after — do not. The slashes were being skipped
PAST before the span was measured, which made all of them plain text.
A bare hyperlink reference ends on that same class too (v0.112.0+) —
the THIRD place in inline.go to need it, and the last one that was
still testing unicode.IsPunct instead. The two are not the same set
and not even nested: of 39 characters run through real docutils, twelve
disagreed, in both directions. * is Unicode punctuation and not in the
class, so bdist_* to stdlib became a reference to bdist — taking the
underscore and the asterisk out of the visible text with it — where
docutils has a plain sentence; (, [, {, &, %, #, @ did the
same. The other way, > is a math SYMBOL and \ is not punctuation at
all, yet both legitimately end a reference.
A standalone email address may end on the same end_string_suffix
class a standalone URI may (v0.87.0+) — notably >, which Unicode
classifies as a math SYMBOL rather than punctuation, so an ad-hoc
"space or punctuation" test rejects <user@host> while accepting
(user@host). That is how essentially every PEP writes an author, and
it accounted for 386 of the 1564 real-world corpus files on its own.
That address's START boundary followed in v0.111.0, along with the
divergence this paragraph used to record as permanent. emailc is
[-_!~*'{|}/#?^&=+$%a-zA-Z0-9], so /is an email character andcomp.lang.python/python-list@python.orgis ONE address (PEP 20 writes it); it is ASCII, so an accented letter ends the address rather than continuing it; and.is a SEPARATOR between runs,emailc+(.emailc+)*`,
which is why a leading, trailing or doubled dot is not an address at all.
Both scans backtrack (v0.130.0+), which is the other half of that
same end_string_suffix. docutils matches each with one pattern ending
in uri_end plus the suffix, so when the longest span fails the trailing
lookahead the regex engine shortens it until one fits; a scanner that
checks the boundary once and gives up refuses a URI that is merely
FOLLOWED by an unusual character. pytest's changelog template writes
https://github.com/pytest-dev/pytest/issues/{{ value[1:] }}, and { is
neither a URI character nor a closer, so the whole link was plain text
where docutils links the …/issues prefix. The email half needed the
same loop for user@e.org/{ — found only by a 672-case differential
probe, after a six-case check had said that path was already right.
A bare name_ reference bounds an implicit URI the same way an
emphasis start-string does (v0.130.0+): it is another alternative of the
same patterns.initial, matched in the first pass, so file://x/y_ is
the URI file://x/ followed by a reference to y rather than one URI
ending in an underscore. No real-world file writes one; the probe found
it, 12 of its 672 cases.
And when standalone_uri matches but the scheme is not in
urischemes.schemes, docutils raises MarkupMismatch, after which
implicit_inline returns the WHOLE text it was given as one plain
Text node. Two things follow. No email is found inside the refused
URI — a per-position scanner reads svn+ssh://pythondev@svn.python.org/
as a mailto: address, and this one did. And a good URI AFTER it is not
recognised either, until an explicit construct starts a fresh run:
emphasis or an inline literal resets it, a newline inside the same
paragraph does not. Each of those boundaries was checked by running
docutils, not by reading it.
Two things keep that faithful rather than merely strict. The scheme test
happens only once the whole absolute-URI shape has matched, because that
is when docutils applies it — testing it at the colon refuses
Trailing punctuation: https://x.org, which is not a URI at all. And an
implicit match may not cross an explicit start-string (implicitLimit),
which docutils gets structurally: Inliner.parse finds those first and
hands implicit_inline only the text between them. Without that bound,
transcribing emailc faithfully brings ` along with it and
non-@overload-decorated becomes one address swallowing the
literal — PEP 484, caught by set-diffing the corpus within the same
round rather than by the totals, which were rising.
An OPENING backquote in a target's name must CLOSE (v0.120.0+).
Body.patterns.target makes the closing one mandatory once the opening
one is used, and hyperlink_target keeps appending body lines until the
pattern matches or the block runs out — then raises
MarkupError("malformed hyperlink target."), which becomes a WARNING
beside the line captured as an ordinary comment. This parser fell
through to the unquoted branch, which finds the colon and takes
everything before it, so .. _\a: http://e.combecame a target named ``a `` — backquote and all — pointing at the URI: a target invented out
of broken syntax, under a name no reference could ever spell.
A hyperlink target's URI has its whitespace REMOVED and its escapes
processed (v0.118.0+), and so does its NAME. parse_target states the
URI rule once —
' '.join(''.join(unescape(part).split())
for part in split_escaped_whitespace(' '.join(block)))
— and it is not a trim: every unescaped whitespace run disappears, which
is what lets a URI wrap across lines and close up seamlessly, while each
ESCAPED one becomes exactly one space. joinEmbeddedURI has been that
rule since v0.31.0, used for the URI embedded in a phrase reference; a
target's own URI never went through it.
The name is the other half, and the halves disagreeing is what made it
visible: `a\ b`_ is a reference to ab on the inline side, while the
target beside it recorded a\ b, so the two never matched and the
reference stayed unresolved against a target sitting right next to it.
A hyperlink target's name may be backquoted (v0.92.0+), and then the
quotes are delimiters rather than part of the name: .. _\with: colon`:is the single namewith: colon, the colon inside them not terminating it. This parser kept the quotes in the namewhile dropping them from theid (make_id` drops them anyway), so the two disagreed — on every
phrase target pytest's and sphinx's documentation writes.
A diagnostic raised INSIDE a directive's body now carries a real source
line (v0.93.0+, admonitions and container). parseBlockLines takes a
lineBase so a nested construct can map local indices back to absolute
ones — block quotes, list items, field bodies, option lists and
definitions all compute one, and every directive passed -1. The
mapping is exact rather than approximate: a directive's combined block is
[text after "::"] + [the blank lines under it] + [the body], mirroring
lines[i:] one for one, so parseDirectiveBlockAt reports where the
content begins and the caller adds i. Every directive that nests content now
threads one (v0.97.0): admonitions, container, class, figure,
header/footer and the two table directives. A footnote or citation
body followed in v0.113.0 — it was still passing -1, so every
diagnostic raised inside one carried no line at all, which is nine
real-world corpus files and one of the cheapest fixes in this list:
content[k] is lines[i+k], so the base is i+lineBase. The test's
telling case is a SECOND paragraph in the body, which reports its own
first line rather than the footnote's — so the rule cannot be satisfied
by remembering where the footnote started.
A directive's own diagnostics followed in v0.126.0. Fifteen runners
computed lineno := i + 1 — a LOCAL index — against three that used
msgLine(i, lineBase), so a nested code-block reported line 3 where
docutils reports 159: 3 was its offset inside the footnote body rather
than its place in the file. Six of the fifteen already RECEIVED
lineBase and ignored it, which is the clearest form of a mechanism
built and never wired — the parameter was there, the call sites passed
it, and nothing read it. Seventeen sites in all, counted by grepping the
old form afterwards rather than trusting the list drawn up first: two
more turned up that way, in parseDirectiveBody and
parseSubstitutionDef.
Table cells followed in v0.114.0, which leaves one caller passing the
sentinel in the whole package. Every cell in a row needs a DIFFERENT
base, which is why this could never have been one value per table: for a
grid table a cell's content is block rows top+1..bottom-1 and
block[r] is lines[i+r], so the base is i+top+1+lineBase; for a
simple table cell.lineOffset is already the row's first line within the
block, so it is i+lineOffset+lineBase, with no border row to skip.
A csv-table cell passed -1 until v0.136.3, on the reasoning that a
CSV reader does not record which body line each row began on. It does not
have to: docutils does not report the row either. Every diagnostic from
any cell carries the directive's content OFFSET plus the line index WITHIN
the cell, which is a line base of contentOffset - 1 and no bookkeeping
at all — a rule that had to be TRACED, since five probes made it look like
a constant before a sixth (a role on the second line of a quoted cell)
moved it.
Two derivations are
needed, because class and the table directives split their options off
themselves rather than through parseDirectiveBlock, so their content is
a SUFFIX of the body (bodyStartIndex) rather than an offset into the
combined block. Still -1: a GRID or SIMPLE table cell, which needs a per-row
offset — so those table directives' own threading, while correct, cannot
be observed until that is done. A csv-table cell no longer is: see above.
.. csv-table:: (v0.101.0+) reads its content — and its :header:
option — as CSV and lays the result out with the same
tgroup/colspec/thead/tbody assembly the list-table directive
uses. The dialect is docutils' own DocutilsDialect defaults (comma,
" quote, doubled quotes, whitespace after a delimiter discarded),
which is Go's encoding/csv with TrimLeadingSpace. Ragged rows are
padded to the widest rather than rejected. :delim: and :keepspace: ARE ported (v0.136.4+), because both are
fields of that same reader — Comma and TrimLeadingSpace — and the
delimiter takes any spelling single_char_or_whitespace_or_unicode takes:
a character, the words tab/space, or a Unicode code written decimal,
hex (six prefixes) or as an XML entity. sphinx's own latex.rst writes a
table with :delim: ; inside a list item, and refusing it turned the
whole table into a <directive> node holding its own source. Not
ported: :file: and :url:, which read from the filesystem or network
while parsing, and :quote:/:escape:, which need a reader field
encoding/csv does not have — such an invocation, and a :delim: value
that is not a character at all, still fall back to the structural
<directive> capture rather than producing a wrong table.
A directive argument spanning two source lines keeps its LINE BREAK
(v0.100.0+): parse_directive_arguments joins the argument block with
"\n", and a directive declaring final_argument_whitespace passes the
whole text through with its internal whitespace intact — so a two-line
.. rubric:: or admonition title is two lines, not one. The image
URI goes the other way, directives.uri removing all whitespace so a
split URI closes up.
Input lines go through docutils' own string2lines (v0.105.0+): a
VERTICAL TAB or FORM FEED becomes a SPACE (convert_whitespace=True,
whose pattern is exactly [\v\f], so neither is a line boundary —
U+2028 and the C1 separators still are), then each line has its TABS
EXPANDED to the next multiple of 8 and its trailing whitespace
stripped. Without the expansion a leading tab counted as ZERO indent,
so a tab-indented block escaped whatever construct it belonged to —
sphinx's C-domain documentation indents with tabs.
Two different questions are asked about that indentation, and this package
had collapsed them into one (v0.135.0). "Is this line indented?" is
line[0] != ' ' — a PLAIN space. "How deep is it?" is
len(line) - len(line.lstrip()), and str.lstrip() strips every Unicode
whitespace character, a NO-BREAK SPACE included. pytest's own
documentation has a literal block whose first line reads
" pytest …": one plain space deep by the first test, five
CHARACTERS deep by the second, and docutils dedents all five — counting
plain spaces dedented one, leaving four characters of phantom indentation
in a block that preserves it. The two cannot be merged in either
direction: a line indented with NO-BREAK SPACES ALONE is not indented at
all, so a block quote written that way is an ordinary paragraph and a
literal block is "expected; none found". The count is in CHARACTERS,
because the dedent that follows removes the same units the minimum was
taken in.
All three messages an unknown role raises quote the role AS THE AUTHOR
WROTE IT (v0.135.0) — roles.role builds two of them from its own
role_name and Inliner.interpreted the third from role. The
lowercased spelling is for LOOKUP only, and quoting it made
:Class:`x` report "class". The two DIRECTIVE paths that raise the
same pair (.. role:: with an unknown base, .. default-role:: with an
unknown name) already had it right, which is how one corpus file and one
capital letter showed that two of three call sites agreed and the third
did not.
A duplicate-name notice lands INSIDE the innermost container, right
before the paragraph whose inline parsing raised it — docutils appends it
to whatever body element is being filled at that moment. The list of tags
that can hold one named .. admonition:: and not one of the NINE
specific admonitions it shares its content model with (v0.136.2 adds
them), so the notice for a duplicate inside a .. note:: went BEFORE the
note. A family is exactly what a list like that gets wrong, so every
container was then checked against the reference one at a time. header
and footer admit one too and this package still raises none there: the
duplicate-name pass runs before hoistDecoration, and moving the hoist
earlier would make the walk see a header BEFORE the line-1 reference it
follows in the source, where docutils registers names in SOURCE order. No
corpus file has a duplicate name inside a header.
An inline literal whose whole content is a LONE BACKSLASH — ``\``,
which is how PEP 12 documents line continuations — is a literal, not a
<problematic> (v0.103.0+). Inside a literal a backslash is not an
escape, so the escaped backquote lends its backquote to the end string
while the marker stays content; the close therefore lands at the very
first content position, which the "empty content" guard used to reject
on position alone. Four backquotes ARE an empty literal and stay
rejected.
The :math: ROLE keeps its backslashes, alone among the generic roles:
math_role calls unescape(text, restore_backslashes=True), a
backslash in math being TeX syntax rather than reST escaping. :literal:
and :emphasis: still consume them.
.. contents:: (v0.99.0+) produces its PARSE-TIME shape: a <topic>
classed contents (plus local, plus any :class:), an optional
<title> — the argument, or the default label, or NONE at all under
:local: — and the <pending> a later transform replaces with the real
table of contents. Its details carry the parsed options, so :depth:
prints as a bare integer, a flag as None, and :backlinks: none as
None while :backlinks: top keeps its string. Not ported: the
"may not be used within topics or body elements" context check, which
needs a parent-state notion this parser does not keep.
.. epigraph::, .. highlights:: and .. pull-quote:: (v0.98.0+) are
a block quote carrying a class and nothing else: BlockQuote.run calls
state.block_quote() on its own content and appends its class list, so
an attribution line inside one becomes an <attribution> exactly as in
a bare quote. They declare NO options, so a :class: line under one is
CONTENT and parses as a field list inside the quote.
Directive option validation is driven by ONE transcribed table
(v0.106.0+, rst/optionspec.go): each directive's own option_spec
from docutils' registry. code and math consult it, so a Sphinx-only
option — :caption: on a code block, :label: on an equation — is an
unknown option ERROR naming the directive AS WRITTEN. The table lists
every implemented directive, but an entry alone enforces nothing: a
directive is validated only once it calls the validator, which keeps the
table safe to complete ahead of its callers. The scan always stops where
the CONTENT begins.
The code directive VALIDATES its options (v0.96.0+): CodeBlock's
option_spec has exactly class, name and number-lines, and
anything else — sphinx's :caption:, :emphasize-lines: — is an
unknown option ERROR with no <literal_block> produced at all. Both
that message and the missing-content one name the directive AS WRITTEN,
so .. code-block:: says code-block. The option scan stops where the
CONTENT begins: a code block's own content may start with something
field-marker-shaped, which is not an option.
An anonymous hyperlink target may be written __ uri with no ..
prefix (v0.95.0+). It is a Body transition of its own (__( +|$)), so
it fires only at the START of a block — the same text inside a paragraph
stays ordinary text. Both spellings now also carry the id
note_anonymous_target gives them (target-1, target-2, …), which
neither had before.
Ids come from ONE namespace, claimed in document order (v0.94.0+ for
sections, v0.104.0+ for EVERY id-bearing node). A directive's :name:
option and an embedded-URI target used to compute make_id() directly,
so they neither disambiguated against ids already taken nor had a
fallback when make_id came back EMPTY — a name with no
ASCII-alphanumeric start, such as a footnote-style link named 1,
produced id="" and then id="-1". docutils falls back to
make_id(tagname) plus a counter: target-1, note-1, table-1.
document.set_id has a single set per document, and new_subsection
claims a section's id while the section is being parsed — so a section
ABOVE a colliding target keeps the bare id and the target takes the
suffix, and below it the reverse. This parser assigned section ids in a
pass AFTER parsing, from a private map, so .. _get-started: over a
"Get Started" heading gave both nodes id="get-started" (not a valid
document at all), and the ordering of the pair was inverted whenever a
section came first.
A duplicate-name message about an explicit hyperlink TARGET carries the
target's OWN first line (v0.102.0+) — the .. _name: line, even when
the URI continues below it. Footnotes, citations, substitution
definitions and sections already recorded one; a target did not.
A duplicate SECTION name is reported against the title's underline (v0.91.0+) — the last line of the title construct, and so the state machine's own position once it has read the whole thing, overlined form included. Nothing recorded a line for a section before that, so every such message came out with none.
Two line conventions live side by side (v0.85.0+). Inliner.parse
passes its own lineno — the paragraph's FIRST line — to every
diagnostic it raises, so an unclosed-markup or unknown-role message
reports that line however long the paragraph is. A message reported
WITHOUT one (code_role's "Cannot analyze code", set_duplicate_name's
duplicate-name notices, whose base_node is not yet attached and
carries no line) falls back to StateMachine.get_source_and_line()
instead. For a top-level paragraph that is max(firstLine+1, lastLine):
Text.text() steps at least one line past the first looking for a
continuation, then stops on the last line of the block it read. Three
earlier attempts guessed a single formula for both and were each
contradicted by the next probe; what settled it was instrumenting the
reference rather than re-reading it. Inside a NESTED block the position is the enclosing
block's own end, which is what smFreeze holds. That end includes the
blank line(s) that TERMINATED the block, because get_indented reads them
and leaves the cursor on the last one (v0.136.9): a message taking its line
from the cursor was one line early inside a directive or footnote body, and
two early where two blank lines followed. Citation content expected. is
the message that shows it — docutils raises it with no line argument at
all, so it is the cursor by construction, and the locally computed value it
used to carry is the same number only at the top level.
The last divergence that probe found is closed too (v0.136.10), and what
closed it was the fifteen witnesses the previous round said it needed.
Literal block expected; none found. is reported where the nested block
RUNS OUT, so its line depends on whether the enclosing construct's block
kept its trailing blank lines — and the two constructs differ:
get_first_known_indented hands a footnote's or citation's block to the
nested parse WITH them, while parse_directive_block trims a directive's.
So text then :: reports the blank line inside a note or a topic, one
line further inside a footnote, and two further where two blank lines
follow. This package trimmed before parsing and gave the directive's
answer everywhere. The probe now agrees with the reference on all 348
cases, which is the same state both corpora are in: every remaining
difference is named, and each name has a witness.
A NESTED context (docutils' match_titles=False) applies
Text.underline's own checks IN ORDER (v0.136.9). An underline has no
minimum length — the pattern is the line pattern, one punctuation
character repeated — so dup over === is a title attempt and draws
Unexpected section title.; this package tested its transition predicate
instead, whose four-character floor belongs to transitions, and produced a
paragraph. The width rule runs BEFORE the match_titles one: a title wider
than an under-four-character underline is not a title in any context
(TransitionCorrection sends both lines back as text), and above that it
is merely too short, where the WARNING is emitted BEFORE the error. And a
DEMOTED short adornment comes back through the dispatch as a title's own
text, so ... over === draws the "so short" INFO and then the error,
where both lines used to be swallowed into one paragraph — the same
bubble-up that makes the pair a real section at the top level.
An adornment-shaped line is never a CONTINUING definition term
(v0.86.0+): the list ends there on the unindent (with the usual
"Definition list ends without a blank line" warning), the line itself
goes back through the ordinary dispatch — a short one drawing the "so
short" INFO and coming back as the FIRST term of a NEW list, a long one
becoming an "Incomplete section title." ERROR — so two items that look
like one list are two. It may still OPEN a list, which is exactly what a
demoted short adornment does.
One node is excluded from the name machinery for a reason the other
exclusions do not cover: the <directive> placeholder this package builds
for a directive it does not implement (only under
ReportUnknownDirectives: false). docutils has no such node, and its name
attribute is the DIRECTIVE's own name, so letting it claim a target made any
document that uses .. deprecated:: AND has a target or section called
"Deprecated" collide with itself (v0.136.12). One overloaded attribute, three
wrong things: the directive lost its name — a consumer rebuilding the source
then wrote .. :: 9.1, which reads back as a COMMENT, which is how this was
found — a Duplicate explicit target name WARNING was fabricated, and the
REAL target was invalidated so every reference to it dangled. Neither corpus
can see it, since both run with the default options where that node does not
exist; the witness is go-richdoc/rst, 53 of its 1564 files.
Duplicate reference names are diagnosed and resolved (dupnames.go,
v0.57.0+), a full port of docutils' own set_duplicate_name transition
table. What happens when two elements claim one name depends on whether
each is EXPLICIT (a .. _name: target, an inline _`name` target, a
footnote, a citation, a directive's :name:) or IMPLICIT (a section
title, or the target a phrase reference with an embedded URI leaves
behind): two explicit ones invalidate each other with a WARNING, an
explicit one OVERRIDES an implicit one, two implicit ones invalidate each
other, and two naming the same destination keep the first. "Invalidating"
moves the name from name to dupname — the element keeps its id and
its place, but stops being something a reference resolves to. The
message's position follows docutils' own msgnode rule, including the
case where it is BUILT AND DROPPED: inside a line block msgnode is the
<line>, which cannot hold body elements, so a duplicate there is
invalidated silently ("System messages are no longer inserted between
<line>s"). Not matched: the line attribute on these specific
messages, which docutils reports against its own node.line bookkeeping
rather than the duplicate's own source line — probing found no constant
offset (three hypotheses each fitted some shapes and failed others), so
this parser uses the real source line rather than a formula reverse-fitted
to one fixture. The substitution-definition duplicate rule IS ported too (v0.71.0+),
and it runs BACKWARDS compared with every other: note_substitution_def
keeps only the LAST definition, so the OLD node is the one invalidated
to dupname and the newcomer keeps its name.
Sphinx's autodoc extension (and napoleon, downstream of
it) is out of scope entirely: it works by importing and introspecting
live Python code, which is not portable to pure Go.
import (
"github.com/go-docutils/docutils/doctree"
"github.com/go-docutils/docutils/rst"
)
doc := rst.Parse(source)
fmt.Print(doctree.Dump(doc)) // this project's own pseudoxml-like debug format
Dump follows pseudo_quoteattr for attribute values, which is
literally '"%s"' % value and escapes NOTHING — not the double quote,
not the backslash, not a newline or a tab (v0.124.0+). It used to escape
newline and tab, on the reasoning that this dump is line-oriented, with
the note that no corpus file exercised it. Six real-world files carry a
multi-line :alt:, so that stopped being true; the shape argument was
about OUR format, and the reference's answer is the one the corpus
compares against.
Writers
Both writers' CONTENT fidelity is measured over the 1564-file real-world corpus
for the first time (v0.136.13, /Users/Shared/rstcorpus/htmlprobe): every
character of the tree's own text must appear, in order, in the rendered output.
The question is deliberately not "does this match docutils' writer", which
neither writer aims at (see each package's SCOPE note) — it is "does the writer
DROP anything", since a node with no case in the switch takes its whole subtree
with it.
Both are now 1564 of 1564. Getting there found one defect in each writer,
and both were a table's own TITLE (.. table:: Caption): the latex writer
DROPPED it -- the tabular came out alone -- and the html writer emitted it as an
<h1> INSIDE the <table>, which is not valid HTML at all. A tabular is not a
float, so the caption needs a table environment around the pair (vanilla LaTeX,
no package); an uncaptioned table stays a bare tabular rather than being wrapped
in a float it never asked for. The three LaTeX escapes that REPLACE their
character with a macro (\textasciitilde{} and friends) are mapped back in the
probe, since a character-level comparison cannot see them otherwise.
The probe was shown able to FAIL before either number was believed: breaking one inline case in the html writer made 493 files report at once.
A content probe answers "nothing is missing", not "everything is where HTML
allows it" — it said the former while a table's caption sat in an <h1> INSIDE
the <table>, which is not an allowed child. So there is a second measurement:
every element the fragment emits is checked against the CONTENT MODEL of its
parent (/Users/Shared/rstcorpus/validprobe), for the containers this writer
emits. 1560 of 1564 valid, 0 violations; the other 4 embed raw html of the
author's own — sphinx's domains page opens <span> anchors it never closes — and
nothing downstream can validate that. Falsifiable too: making a <ul> emit a
<p> child made 959 files report.
That probe is also what turned up the comment escaping. It flagged a <http>
"tag" inside a comment, which was the probe not understanding comments — and
checking THAT showed the real defect: an odd run of dashes
(.. a comment ---> arrow) produced - -->, so the comment ended early and its
tail leaked into the document as visible text. docutils' rule is a space after
each dash FOLLOWED by another, which ReplaceAll("--", "- -") is not.
It also has a blind spot worth naming — it is whitespace-INSENSITIVE, so it
cannot see words running together. That is exactly the defect reading its output
found: a LaTeX tabular cell holding a bullet list came out
"Table cellscontainbody elements.", docutils' own GridTableParser docstring
example. doctree.AsText concatenates every descendant with nothing between,
which is right for inline content and wrong across blocks.
Validity for the latex writer means something the HTML side cannot borrow:
the output is only a document if a TeX engine can READ it. So the third
measurement COMPILES all 1564 (/Users/Shared/rstcorpus/texprobe, through
go-tex/engine's gotex -offline). 1553 of 1564 compile. The 11 that do
not are all named: 8 are EMPTY sources — an empty .rst gives a document with
nothing to typeset, and this engine calls that an error where pdftex only warns
— and 3 carry the author's own .. raw:: latex with sphinx-only markup
(\sphinxsetup, \dimeval) or & alignment in a formula the reference also
writes into equation*.
Compiling found two defects, and neither was visible to either probe above.
The first is a control word that swallowed the next word. A footnote opened
with \par\noindent and its own text followed immediately, so TeX read
\noindentFirst — a backslash takes the LONGEST run of letters — and stopped.
Every character was present, in order, and in the right container; the document
just could not be read. 2130 sites in the corpus were correct for an accidental
reason (a name puts a \hypertarget next, and its backslash ends the word),
which is why only 12 files failed. A census of the writer's own vocabulary
confirmed \noindent was the ONLY command it emits that can run into text:
every other is followed by {, a newline, or another backslash.
The second is the math environment. It was hardcoded to equation* and
verified against the reference — on a one-line formula, which exercises one side
of a rule that BRANCHES. pick_math_environment gives align* when the formula
carries a top-level line break, and amsmath rejects \\ inside equation*, so
a multi-line formula was LaTeX that cannot compile. The rule is ported with its
toplevel_code intact, because the naive reading is wrong for a real case: a
\\ inside \begin{matrix} stays equation*. This corpus has no such
formula, so the witness comes from running the reference on the three shapes
rather than from the corpus.
The compile sweep was shown able to fail in both directions: the 12 failures
went to 0 with the fix and nothing regressed (set-diff, not totals), and making
the writer emit one bogus command made 912 files report it. Read with
-lenient -report-skipped — which walks the WHOLE document instead of stopping
at the first error, the way a compile does — the entire corpus leaves exactly
two undefined commands, both the author's.
What the writers had no case for at all
Three measures said the writers were sound and a fourth question had never been
asked: which doctree tags OCCUR in the corpus and have no case in each writer's
switch, falling through to default: renderChildren? A census of all 1564 files
answers it, and the answer separates into two kinds.
For most of them the TEXT survives and only the container is lost: a note
renders as an indistinguishable paragraph, a topic's title as an <h1>. That
is a presentation question, inside this writer's declared scope, and it is left
alone — but now measured and named rather than unnoticed.
For four of them it was content loss, and two probes were blind to each:
An <image> has no children, so it rendered as NOTHING. 130 image nodes in
47 files, 120 of them outside a substitution_definition and therefore meant to
be rendered. A content probe cannot see one missing — an image carries no text —
and a content-model probe has nothing to check. Both writers now emit them, and
120 of 120 reach the output. The first sign of this was not a probe at all
but reading a compiled PDF's LaTeX: \href{https://github.com/python/peps/actions}{},
a link with no clickable text, which is what PEP's CI badge became once the
image inside its :target: was dropped.
A block whose children are INLINE lost its boundary. A rubric or a
caption rendered its words with no element around them, so they ran into the
neighbouring block's: docutils' own test-markup-rubric fixture, a run of
consecutive rubrics, came out as This is a rubricThis is another. Every
character present, in order, in valid HTML. So there is a fourth probe
(/Users/Shared/rstcorpus/blockprobe): text sitting directly inside a
container, or at the fragment root, belongs to a block that lost its element.
35 files and 76 runs before, 0 after.
The <img> output matches the reference byte for byte on every shape tried,
which took reading image_size: a measure with a UNIT becomes a style
declaration and a unitless one a plain attribute, and :scale: multiplies
whichever measures were declared. One branch is deliberately not ported — with
:scale: and fewer than two measures docutils reads the missing dimension out of
the image FILE, and a writer handed nothing but a doctree has no file to read.
The LaTeX side reads the same attributes differently, which is the kind of thing
only running the reference tells you: a unitless measure is BIG POINTS
(width=200bp), and a percentage is a fraction of \linewidth. Where latex2e
emits one of its own DU* macros this writer emits the FALLBACK DEFINITION
docutils itself provides for it (\DUrubric → \subsubsection*{\emph{#1}},
DUlegend → {\small …}), since there is no preamble here to define a macro in.
Taking docutils' own expansion is the difference between following the reference
and guessing at it.
Two things are worth stating plainly about the numbers.
\usepackage{graphicx} and \includegraphics made the compile sweep go DOWN,
1553 → 1540. Every one of the 13 new failures is a file whose image reference
nothing local can satisfy: a remote badge URL, a data: URI, an absolute
/_static/… path, or sphinx's own image.* language glob. The probe now writes
a placeholder for every reference a local file COULD satisfy — otherwise it
would be measuring the corpus's asset completeness rather than the writer's
LaTeX — and reports the four unsatisfiable classes as their own class. docutils'
latex2e writer emits all four verbatim too, so the drop is the documents saying
what they reference, not a regression.
The validity probe grew a rule at the same time, because a rule it does not have
is a rule it passes: <figure>'s own content model (at most one <figcaption>,
first or last child) was added when the writer started emitting figures at all,
and shown able to fail — moving the <figcaption> off the end made 20 files
report.
A bare target chained onto a URL
.. _name: with no reference of its own hands its ids and names to the next
node (PropagateTargets). collectTargets read that as "a same-document
anchor, resolve it to #id" — right for an inline _text`` target and for one
preceding a real node, and wrong when the next node is ANOTHER TARGET, where the
name lands on that target and inherits what it points at. Asked about
.. _pythondoc:
.. _gendoc: http://example.com/gendoc
the reference answers <reference name="pythondoc" refuri="http://example.com/gendoc">; this parser answered #pythondoc, a
fragment nothing in the document carries.
Neither corpus sweep can see this, and neither moved when it was fixed: both
compare against docutils' own BARE parse, where a reference carries its refname
and no refuri at all. The defect only exists once ResolveReferences is on,
which is how every consumer reads it — it surfaced as a dangling link in
go-richdoc/rst, not here.
The rule's other side is pinned too, and is also the reference's answer: a bare
target chained onto another BARE one keeps its own id
(<section ids="section-one b a">, <reference name="a" refid="a">), because
the node they both precede carries every one of the ids.
Two things about the fix are worth recording. resolveIndirect answers from
direct FIRST, so redirecting a name into the indirect map is not enough —
collectTargets' own #name has to be removed, or the chain is unreachable and
the old answer comes out anyway. That was found by printing the two maps: both
were right, and the wrong answer appeared regardless. And the tests' first
version matched the refuri ANYWHERE in the dump, where the chained target
carries it too — so three of four cases passed with the defect still present.
Each expectation now names the <reference> element.
html: html.Render(doc) string renders a doctree to an HTML
fragment — body content only, no <!DOCTYPE>/<html>/<head>, no
stylesheet, no CSS classes or ids beyond the few this parser can
actually populate (a footnote/citation's own id, a role's name as a
class). This is a deliberate, bounded v1: docutils' own HTML writer
(writers/_html_base.py + html5_polyglot/__init__.py, ~2300 lines)
embeds a full default CSS stylesheet and a CSS-class vocabulary Sphinx
themes build on — replicating that byte-for-byte would be roughly
another parser's worth of work, for a stylesheet Sphinx doesn't even
use (it has its own Jinja2 templates). Tag choices follow
html5_polyglot where there's an obvious correspondence
(section/h1-h6/p/ul/ol/li/blockquote/table/thead/tbody/tr/td/th,
em/strong/code/cite/sub/sup/abbr; a grid-table cell's column/row span
becomes colspan/rowspan, HTML's own native primitives for exactly
this; an option list becomes a <dl>, each item's comma-separated
flags joined into one <dt>, e.g. <dt>-f, --file=FILE</dt>; a
:math: role becomes the raw TeX source wrapped in \(...\), the
MathJax inline-delimiter convention, plain text with no wrapping tag
or script dependency — MathJax auto-detects it with no markup of its
own to hook into); a directive with no semantic dispatch of its own
renders as <pre class="directive" data-directive="name"> rather than
being silently dropped; an unresolved reference/substitution-reference falls back to
plain text since there's nothing to link to or substitute. Verified
structurally against docutils' --writer=html5 output on representative
documents (not byte-for-byte, given the scope above) plus a tag-balance
check over a document exercising every implemented construct together.
import "github.com/go-docutils/docutils/html"
fmt.Println(html.Render(doc)) // e.g. "<p>Hello <em>world</em>.</p>"
latex: latex.Render(doc) string renders a doctree to a complete,
standalone, compilable .tex document — meant as input to a LaTeX
engine such as go-tex. Unlike html.Render
(a fragment meant to be embedded), LaTeX has no equivalent to dropping a
fragment into a hosting page, so a full document — a fixed
\documentclass{article} with a minimal preamble (hyperref only, for
working links/anchors) — is the useful unit. Also deliberately NOT a
port of docutils' latex2e writer (writers/latex2e/__init__.py, ~3486
lines: multiple document classes, syntax-highlighted listings, real
LaTeX \footnote-machinery bridged across the doctree's separate
footnote-definition/-reference nodes via custom preamble macros,
docinfo-to-titlepage conversion). This uses only vanilla LaTeX
constructs (itemize/enumerate/quote/verbatim/description/
verse/tabular), so it always compiles without a custom macro
package — a field list, a definition list, AND an option list (its
comma-separated flags joined into one \item[{...}]) all share the
same description environment, since none of the three has a native
LaTeX construct of its own. A :math: role renders as core $...$
inline math mode — its content is TeX source already, written
verbatim rather than through the usual text-escaping pass, which
would otherwise corrupt the very characters (^, _, \) math mode
depends on. A table's cell content is flattened to
plain text — a nested
list or multi-paragraph cell would need a p{width} column + minipage
to stay valid LaTeX, not implemented here. A grid-table cell's column
span renders as \multicolumn (plain LaTeX, no package); its ROW span
does NOT — plain tabular has no rowspan primitive without the
multirow package, which this writer deliberately never depends on, so
a row-spanning cell's content still appears but isn't merged, which can
visually misalign a later row that relied on the merge (real row/column
spans both work correctly in html.Render, since HTML has native
primitives for this and no such package constraint). Footnotes/citations don't use
LaTeX's native \footnote (it wants inline content at the reference
point, docutils' doctree has them as separate nodes); a reference
renders as a \hyperlink jump to a labeled paragraph where the
definition appears in the document's normal flow, not a page-bottom
note. Verified by actually compiling representative output (special
characters, nested sections past LaTeX's 5 native depths, every
implemented construct together) with
tectonic during development
— real PDFs, zero errors — not just structural comparison; that step
isn't part of go test itself since a LaTeX engine isn't a build
dependency of this module (same "reference tool, not a runtime
dependency" rule as the docutils foreign judge).
import "github.com/go-docutils/docutils/latex"
os.WriteFile("out.tex", []byte(latex.Render(doc)), 0644)
// tectonic out.tex (or any other LaTeX engine, incl. go-tex)
Security
This section is the result of an audit, not a statement of intent: every claim below was produced by running something. Where the behaviour is the reference's, that is said so explicitly — this package's purpose is fidelity, and a divergence would be a defect even when it looks safer.
What touches the filesystem
Exactly one thing: the include directive, and only when Options.SourcePath is
set. With no path it is disabled (the reference's own
"include" directive disabled., or the structural capture when
ReportUnknownDirectives is off), so a caller that handed this parser a string
cannot be made to read a file. raw's :file:/:url: forms are not implemented,
and nothing else opens anything.
Options.IncludeRootPrefix (docutils' root_prefix) is the confinement knob: an
ABSOLUTE include path resolves under it instead of at the real root. It does not
confine a RELATIVE path that climbs with .., because the reference does not either
— a caller that needs that guarantee leaves SourcePath empty.
Denial of service
| guard | default | what it stops |
|---|---|---|
Options.LineLengthLimit |
10000 (the reference's) | a document with one enormous line; the parse is refused with the reference's own Line N exceeds the line-length-limit. ERROR and produces no tree. The included lines of an include are checked too, and refuse the INCLUSION with a warning naming the file |
| circular inclusion | always | a file that includes itself, directly or through a chain; the same file twice in SEQUENCE is still allowed, which is why the guard is a chain and not a set |
A complexity attack was found by this audit and fixed: 20000 hyperlink targets
sharing one name took 14 seconds for 340 KB, while 20000 DISTINCT names took 26ms.
Four O(n)-per-duplicate operations were responsible — searching the body for the
element's inline host, searching it again for its top-level ancestor, a slice
insertion that copied the whole tail, and an id allocator that rescanned name-1,
name-2, … from 1 for every collision. The same document now parses in 70ms, and
TestADuplicateNameIsNotQuadratic holds the bound as a RATIO (4× the input must not
cost more than 8× the time) rather than as a wall-clock budget. Restoring the old id
allocator makes it report 4x the duplicates cost 16.1x the time, which is what a
regression test for a complexity bound has to be able to say.
What is still unbounded, as in the reference: the total size an inclusion chain may
read, and how deep it may nest. A caller parsing untrusted documents with
SourcePath set should bound those itself.
Measured for shape, not just for speed: a 2 MB single line, 50000 consecutive *
or `, 20000 unterminated grid-table rows and 2000 levels of nested bullets all
parse without a panic, in a second or less each.
What is passed through verbatim
The writers escape TEXT and HTML attribute values. These probes are the evidence:
| input | HTML output |
|---|---|
`click <javascript:alert(1)>`_ |
<a href="javascript:alert(1)">click</a> |
:alt: of "><script>alert(1)</script> |
alt=""><script>alert(1)</script>" |
a & in a URL |
& |
A URI scheme is not filtered, and the reference does not filter one either — asked
the same question it also writes href="javascript:alert(1)". Serving HTML built from
untrusted reST therefore needs a scheme allow-list in the CONSUMER; this package will
not quietly diverge from the reference to provide one.
The same holds for LaTeX, more sharply. Text is escaped (\textbackslash{}, \{,
\$), but a URI or an image path goes into \href{…}/\includegraphics{…}
verbatim, so a path containing } closes the argument early — in this package and in
docutils alike (\href{http://x/a}\input{b}}{link} is the reference's own output for
that input). .. raw:: latex and :math: are TeX by definition and pass through
untouched, the first gated by Options.RawEnabled (default true, as in the
reference). Do not compile a .tex generated from an untrusted document, and never
with shell-escape enabled.
Testing
go test ./.... Fixtures in rst/parser_test.go were generated from
this parser's own output, then eyeball-verified against the docutils
foreign judge (see the package doc comment) before being frozen — not
hand-transcribed (for footnotes/citations/substitutions, "docutils
foreign judge" means Parser().parse(src, document) directly rather
than publish_string, to see the tree before docutils' own transforms
run — see the rst package doc comment). Coverage as of this writing:
doctree 97%, rst 93%, html 89%, latex 87%. go vet ./... and
gofmt -l . clean.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package doctree defines the document tree produced by the reST parser, modeled on docutils.nodes (Body, Structural, Inline element categories).
|
Package doctree defines the document tree produced by the reST parser, modeled on docutils.nodes (Body, Structural, Inline element categories). |
|
Package html renders a doctree.Element into an HTML fragment.
|
Package html renders a doctree.Element into an HTML fragment. |
|
Package latex renders a doctree.Element into a standalone LaTeX document, meant as input to a LaTeX engine such as go-tex.
|
Package latex renders a doctree.Element into a standalone LaTeX document, meant as input to a LaTeX engine such as go-tex. |
|
Package rst is a reStructuredText parser producing a doctree.Element document tree, modeled on docutils.parsers.rst.states.
|
Package rst is a reStructuredText parser producing a doctree.Element document tree, modeled on docutils.parsers.rst.states. |