article

Overview ¶

Command article finds a page's article body by scoring elements as it streams past them, and then emits that element's subtree and nothing else.

$ article < page.html
<div class="post-body"><p>The first paragraph…</p>…</div>

scored 214 elements; the winner is <div class="post-body"> at bytes 4218-18902
  text            8214 characters
  link text        412 characters in 9 links
  paragraphs        14
  score           7390
  runner-up       <div id="content"> with 3120

Why this cannot be one pass ¶

The winner is not known until the document ends, and a rewrite cannot write to a position it has already passed. So the first pass scores and the second emits: two passes, which the library's documentation prices and which is unavoidable here rather than a convenience.

What makes the second pass cheap is that the winner can be named by where it was. lolhtml.SourceLocation offsets are absolute and unaffected by how the document was written in - one byte at a time gives the same numbers as one call - so the byte range from the first pass identifies the element in the second, whatever the second pass's read sizes are. Nothing has to be buffered but the scores.

The scoring ¶

Text that is not inside a link, minus twice the text that is, plus a bonus per paragraph. That is the shape of every readability heuristic and it is not the interesting part; what the library decides is how the counting has to be written:

A text chunk does not know what element it is in, so the count is kept per open element and added to all of them - a paragraph's text is its own and its container's, which is what makes a container score above its children.

A text chunk cannot tell that it is inside a link either, so the depth of open anchors is counted. There is no selector for "not inside an <a>".

An element's score is complete only at its end tag, so lolhtml.Element.OnEndTag is where a candidate is recorded - and an element whose end tag is missing never records one, which is why the report says how many elements were skipped for that reason rather than pretending the document was tidy.