randogoth/sites
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
sites/plugins/smolnet.py
randogoth 9817e7190c feat: make the capsule subscribable over Gemini 
Link labels in every capsule index are now ISO 8601 and lead with the date,
which is what Gemini's subscription convention reads a feed entry from: the
first ten characters of a label must parse as YYYY-MM-DD or the line is
ignored. The board's `09 Oct 26` parsed as nothing, so the front page, the
archive and all six areas registered zero entries and the capsule was
invisible to every aggregator in Geminispace. `/archive` now carries 36.

The Connect page advertised `/feeds/all.rss.xml` and `/feeds/all.atom.xml`,
which exist only in the HTML build and 404 over Gemini. That paragraph now
sits behind an itsybitsy `only html` format gate, with prose outside it that
is true on both faces. The gate can only cut that way round: Pelican does not
implement the directive and renders it as an inert comment, so a gate
subtracts from the capsule and never from the site.

Also adds dormant `SMOLNET_COMMENTS` support, for a per-article thread link
gated to the text formats. Off until kapara is deployed, since setting it
would put a dead link under every article.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-09 17:19:50 +03:00

481 lines
20 KiB
Python
Raw Blame History

"""Render a Pelican site as a small-net capsule beside its HTML.
A capsule is a directory of plain Markdown that a protocol daemon serves over
Gemini, Spartan, Nex, Gopher and WAP, converting per request. This writes one
out of the same content the HTML build uses, so the two never drift.
Written against `itsybitsy`, which parses **no frontmatter at all**: a document
is pure Markdown and its metadata lives beside it in a per-directory sidecar.
Pelican's own metadata is a block of bare `Title:`/`Date:` lines at the head of
every file, so served unchanged it becomes the first paragraph of every page.
Stripping that is most of what this plugin does.
Nothing here is specific to one site. Everything that differs between sites is a
setting, and every setting has a default, so a site can switch it on with just
`PLUGINS = ['smolnet']`.
SMOLNET_OUTPUT 'smolnet' directory under OUTPUT_PATH
SMOLNET_SECTION_DIR 'section' subdirectory holding one index per category
SMOLNET_LATEST 15 entries on the front page; 0 drops the section
SMOLNET_META [date, category] fields for the line under an article title
SMOLNET_SECTION_ORDER {} category name -> sort key, else alphabetical
SMOLNET_MIN_SECTIONS 2 below this, the section index is dropped
SMOLNET_LABELS {} override any heading, see LABELS
SMOLNET_ASSETS None directories to copy; defaults to STATIC_PATHS
SMOLNET_SIDECAR '.itsybitsy.toml'
SMOLNET_ROT13_ATTR None attribute holding a rot13 address to reveal
SMOLNET_TAGLINE None one line at the foot of the index
SMOLNET_COMMENTS None thread URL per article, `{slug}` substituted
SMOLNET_COMMENTS_FORMATS formats that get it, default gemtext and text
Why link-only paragraphs rather than lists: gemtext has one kind of link, a line
beginning `=>`, so a renderer has to lift links out of the prose. itsybitsy's
gemtext renderer emits a bare `=>` line for a paragraph that holds nothing but
links, so a run of links comes out as a clean run of link lines. A Markdown list
would instead print a bullet *and* a link line for every entry, listing
everything twice.
Gemini clients number link lines themselves, which means a menu of areas is
dialled by number without this plugin writing a single bracket.
The links are separated by **hard breaks**, which is two trailing spaces. gemtext
lifts each link out either way, but the inline formats lay a paragraph out as
flowing text, so with soft breaks a six-item menu renders as one run-on line in a
browser and on a handset. A hard break becomes `<br/>` there and costs gemtext
nothing. The `text` renderer used for Nex and Gopher flattens both break kinds to
a space and still reflows, which is the one format this shape does not serve; a
site whose main face is Gopher should emit lists instead.
"""
import os
import re
import shutil
from pelican import signals
from pelican.generators import Generator
LABELS = {
"sections": "Sections",
"latest": "Latest",
"archive": "All entries",
"pages": "Pages",
"back": "Back to the index",
"comments": "Comments",
}
# Pelican's metadata block: bare `Key: value` lines from the first line on,
# with indented continuations, ending at the first blank line. This is
# Python-Markdown's Meta extension, not YAML, and it has no delimiters.
META_LINE = re.compile(r"^[A-Za-z][A-Za-z0-9_-]*:")
META_CONT = re.compile(r"^\s+\S")
# Pelican's internal link syntax. Left alone these reach the capsule verbatim.
INTERNAL = re.compile(r"\{(filename|static|attach)\}([^)\s]+)")
# Links written against the HTML build, e.g. ](/go-west.html). Pelican's
# {filename} form is the portable one, but plenty of content predates knowing
# that and hardcodes the output URL instead. A capsule has no .html, and the
# daemon resolves a name carrying an extension literally, so these 404 unless
# they are rewritten.
HARDCODED = re.compile(r"\]\(\s*(/[^)\s]*?\.html)([^)]*)\)")
SUMMARY_MARKER = "<!-- PELICAN_END_SUMMARY -->"
# Markdown image syntax, for dropping the ones whose host is gone. Matched on
# the source rather than the rendered HTML because this plugin reads the
# Markdown directly, so lostimages.py, which works on Pelican's output, never
# sees the capsule.
MD_IMAGE = re.compile(r'!\[[^\]]*\]\(\s*(https?://[^\s)]+)[^)]*\)')
def rot13(text):
"""The obfuscation used on addresses in the HTML templates."""
out = []
for ch in text:
if "a" <= ch <= "z":
out.append(chr((ord(ch) - 97 + 13) % 26 + 97))
elif "A" <= ch <= "Z":
out.append(chr((ord(ch) - 65 + 13) % 26 + 65))
else:
out.append(ch)
return "".join(out)
def strip_metadata(text):
"""Drop the leading metadata block, returning the body alone.
Stops at the first line that is neither a `Key:` line nor an indented
continuation of one, so a document that happens to open on prose is
returned untouched rather than having its first paragraph eaten.
"""
lines = text.splitlines()
i = 0
seen = False
while i < len(lines):
line = lines[i]
if not line.strip():
if seen:
i += 1
break
if META_LINE.match(line):
seen = True
elif seen and META_CONT.match(line):
pass
else:
break
i += 1
if not seen:
return text
return "\n".join(lines[i:]).lstrip("\n")
def rewrite_internal(text, by_source, warn):
"""Resolve `{filename}other.md` against the documents actually built.
The target is a path relative to the referring file, which for a flat
content directory means a bare basename. Anything that does not resolve is
reported rather than silently emitted, because a literal `{filename}` in a
capsule is both broken and ugly.
"""
def sub(match):
kind, target = match.group(1), match.group(2)
key = os.path.basename(target)
if kind == "filename" and key in by_source:
return "/" + by_source[key]
warn("smolnet: unresolved {%s}%s", kind, target)
return "/" + key.rsplit(".", 1)[0]
return INTERNAL.sub(sub, text)
def rewrite_html_links(text, slugs, warn):
"""Point links written against the HTML build at the capsule instead.
`/pages/glossary.html` and `/go-west.html` both become `/<slug>`, since the
capsule is flat. A target whose slug is not among the documents built is
reported and left alone rather than silently rewritten into a second broken
link.
"""
def sub(match):
target, rest = match.group(1), match.group(2)
slug = os.path.basename(target)[: -len(".html")]
if slug in slugs:
return "](/%s%s)" % (slug, rest)
warn("smolnet: link to %s matches no document", target)
return match.group(0)
return HARDCODED.sub(sub, text)
def reveal_rot13(text, attr):
"""Replace a block of HTML carrying a rot13 address with the address itself.
Address obfuscation defends against web crawlers and depends on JavaScript,
so over Gemini it is both pointless and impossible. The renderer drops raw
HTML blocks outright, which would take the address with them: on a page
whose purpose is to carry it, that is a silent loss rather than a tidy-up.
The whole block is replaced rather than the tag that carries the attribute.
Matching tags means guessing where the element ends, and the first attempt
at this stopped at the first closing tag and left the rest of the markup
stranded in the output. Markdown already separates block-level HTML by blank
lines, so the block is a boundary the format gives us for free.
"""
if not attr:
return text
carrier = re.compile(r"\b" + re.escape(attr) + r'="([^"]+)"', re.IGNORECASE)
blocks = re.split(r"(\n[ \t]*\n)", text)
out = []
for block in blocks:
found = carrier.search(block)
out.append(rot13(found.group(1)) if found else block)
return "".join(out)
def drop_dead_images(text, dead):
"""Remove images hotlinked from hosts that no longer exist.
It matters more here than on the web. A browser shows a broken image as a
silent gap, but gemtext has no inline images: every one becomes a visible
`=>` link line, so the page would advertise a dead host as though it were
somewhere worth going.
Shares LOSTIMAGES_DEAD_HOSTS with lostimages.py so there is one list, and
matches on the host rather than fetching, because a build must not depend on
the network.
"""
if not dead:
return text
def sub(match):
host = match.group(1).split("//", 1)[1].split("/")[0].lower().split(":")[0]
if any(host == d or host.endswith("." + d) for d in dead):
return ""
return match.group(0)
text = MD_IMAGE.sub(sub, text)
# a line that held nothing but the dropped image would leave a blank
return re.sub(r"\n[ \t]*\n[ \t]*\n+", "\n\n", text)
def toml_escape(value):
return str(value).replace("\\", "\\\\").replace('"', '\\"')
def sidecar(titles):
"""A `.itsybitsy.toml` naming each file beside it.
The daemon falls back to the first heading and then to the filename, so
this is belt and braces, but it is the only place a title can live that
survives a document having no heading of its own.
"""
lines = []
for name in sorted(titles):
lines.append('[page."%s"]' % toml_escape(name))
lines.append('title = "%s"' % toml_escape(titles[name]))
lines.append("")
return "\n".join(lines)
def link(label, href):
return "[%s](%s)" % (label.replace("]", ""), href)
def gate(block, formats):
"""Wrap a block in an itsybitsy format gate.
A gate names output format ids, not protocols: `gemtext` is the one Gemini,
Spartan, Nex and Gopher are all served from, so there is no gating on a
protocol name. Blank lines because a gate encloses *blocks*, and a
directive run together with the line under it is one block.
Note which way this can cut. Pelican does not implement the directive and
renders it as an inert HTML comment, so a gate can only ever subtract from
the capsule, never from the site. That is the right direction for anything
this plugin writes, because the capsule is all it writes.
"""
return "<!-- only %s -->\n\n%s\n\n<!-- end -->" % (" ".join(formats), block)
class CapsuleGenerator(Generator):
"""Writes the capsule once the content generators have read everything."""
def generate_context(self):
pass
def generate_output(self, writer=None):
s = self.settings
out = os.path.join(self.output_path, s.get("SMOLNET_OUTPUT", "smolnet"))
section_dir = s.get("SMOLNET_SECTION_DIR", "section")
labels = dict(LABELS, **s.get("SMOLNET_LABELS", {}))
order = s.get("SMOLNET_SECTION_ORDER", {}) or {}
latest_n = s.get("SMOLNET_LATEST", 15)
rot_attr = s.get("SMOLNET_ROT13_ATTR")
dead_hosts = {h.lower().lstrip(".") for h in
s.get("LOSTIMAGES_DEAD_HOSTS", [])}
sidecar_name = s.get("SMOLNET_SIDECAR", ".itsybitsy.toml")
# kapara serves the same remark42 thread the HTML widget shows, over
# Gemini, Spartan and Nex. Articles only: the widget is on articles
# only, and a page with a thread nobody can reach from the site is a
# second conversation, which is the one thing the gateway exists to
# avoid. Gated because this tree can also be served as WML or
# XHTML-MP, and a handset cannot open a gemini:// link.
comments = s.get("SMOLNET_COMMENTS")
comments_formats = s.get("SMOLNET_COMMENTS_FORMATS", ("gemtext", "text"))
articles = [a for a in self.context.get("articles", [])]
pages = [p for p in self.context.get("pages", [])]
if not articles and not pages:
return
# Map source basename -> slug, so {filename} can be resolved. Built from
# everything published, pages included, since an article may link one.
by_source = {}
slugs = set()
for doc in articles + pages:
by_source[os.path.basename(doc.source_path)] = doc.slug
slugs.add(doc.slug)
os.makedirs(out, exist_ok=True)
root_titles = {}
def emit(doc, trail, thread=False):
body = strip_metadata(self._read(doc))
body = body.replace(SUMMARY_MARKER, "")
body = rewrite_internal(body, by_source, self._warn)
body = rewrite_html_links(body, slugs, self._warn)
body = reveal_rot13(body, rot_attr)
body = drop_dead_images(body, dead_hosts)
# Blank lines, not newlines: a heading and the line under it are
# separate blocks only if a blank line divides them, otherwise the
# date runs into the first sentence as one paragraph.
head = "# %s\n\n" % doc.title
if trail:
head += "%s\n\n" % trail
text = head + body.strip() + "\n"
if thread and comments:
text += "\n%s\n" % gate(
link(labels["comments"], comments.format(slug=doc.slug)),
comments_formats,
)
name = "%s.md" % doc.slug
with open(os.path.join(out, name), "w", encoding="utf-8") as fh:
fh.write(text)
root_titles[name] = doc.title
# Which metadata goes under the title. Default is date and category,
# but a site that files by tag has no useful category and may well have
# a field that means more: zonetoast carries `era`, which is the point
# of the site in a way that Pelican's default `misc` is not.
meta_fields = s.get("SMOLNET_META", ["date", "category"])
for article in articles:
bits = []
for field in meta_fields:
value = getattr(article, field, None)
if not value:
continue
if field == "date":
bits.append(value.strftime("%d %b %Y"))
else:
bits.append(str(value))
emit(article, " · ".join(bits), thread=True)
for page in pages:
emit(page, "")
# One index per category.
sections = sorted(
self.context.get("categories", []),
key=lambda pair: (order.get(str(pair[0]), 99), str(pair[0])),
)
# A site that files by tag rather than category still has one category,
# Pelican's DEFAULT_CATEGORY, holding everything. Offering "misc, 14" as
# the only way in is worse than offering nothing, so one bucket is not a
# taxonomy and the whole section index goes.
if len(sections) < s.get("SMOLNET_MIN_SECTIONS", 2):
sections = []
if sections:
os.makedirs(os.path.join(out, section_dir), exist_ok=True)
section_titles = {}
for category, items in sections:
lines = ["# %s" % category, ""]
lines.append(self._links(sorted(items, key=lambda a: a.date, reverse=True)))
lines += ["", link(labels["back"], "/")]
name = "%s.md" % category.slug
with open(os.path.join(out, section_dir, name), "w", encoding="utf-8") as fh:
fh.write("\n".join(lines).rstrip() + "\n")
section_titles[name] = str(category)
self._write_sidecar(os.path.join(out, section_dir, sidecar_name), section_titles)
by_date = sorted(articles, key=lambda a: a.date, reverse=True)
# The full listing, so the index can stay short without hiding anything.
archive = ["# %s" % labels["archive"], "", self._links(by_date), "",
link(labels["back"], "/")]
with open(os.path.join(out, "archive.md"), "w", encoding="utf-8") as fh:
fh.write("\n".join(archive).rstrip() + "\n")
root_titles["archive.md"] = labels["archive"]
index = ["# %s" % s.get("SITENAME", "index"), ""]
if sections:
index += ["## %s" % labels["sections"], ""]
index.append(" \n".join(
link("%s · %d" % (category, len(items)),
"/%s/%s" % (section_dir, category.slug))
for category, items in sections
))
index.append("")
if latest_n and by_date:
index += ["## %s" % labels["latest"], "",
self._links(by_date[:latest_n]), "",
link(labels["archive"], "/archive"), ""]
if pages:
index += ["## %s" % labels["pages"], "",
" \n".join(link(p.title, "/%s" % p.slug) for p in pages), ""]
tagline = s.get("SMOLNET_TAGLINE")
if tagline:
index += ["---", "", tagline, ""]
with open(os.path.join(out, "index.md"), "w", encoding="utf-8") as fh:
fh.write("\n".join(index).rstrip() + "\n")
root_titles["index.md"] = s.get("SITENAME", "index")
self._write_sidecar(os.path.join(out, sidecar_name), root_titles)
self._copy_assets(out)
def _links(self, docs):
"""One link per line in a single paragraph.
Deliberately not a Markdown list: a list item renders as a bullet *and*
a link line, so every entry would appear twice.
Targets are root-relative. A section index sits one directory down, so
relative targets would need `../` and would break again at any further
depth; `/slug` is correct from anywhere and the daemon resolves it
against the site root.
The date is ISO 8601 and it leads the label, which is protocol surface
rather than a style choice. Gemini's subscription convention takes a
link line for a feed entry when the first ten characters of its label
are a `YYYY-MM-DD` date, reads the rest as the title, and ignores every
line where they are not, so this one format is what makes each index a
feed: the front page, the archive and all six areas, subscribable in a
client and submittable to an aggregator. A board's `09 Oct 26` parses
as nothing and registers zero entries.
The separator is the one the convention's own examples use. Clients are
asked to sanitise whatever sits between the date and the title, but not
all of them do, and a `·` that survives arrives as part of the title.
"""
rows = []
for doc in docs:
label = doc.title
if getattr(doc, "date", None):
label = "%s - %s" % (doc.date.strftime("%Y-%m-%d"), doc.title)
rows.append(link(label, "/%s" % doc.slug))
# Hard breaks, which is two trailing spaces in CommonMark. gemtext lifts
# each link to its own => line either way, but the inline formats lay a
# paragraph out as flowing text, so without these the whole menu renders
# as one run-on line in a browser and on a handset.
return " \n".join(rows)
def _read(self, doc):
with open(doc.source_path, encoding="utf-8") as fh:
return fh.read()
def _write_sidecar(self, path, titles):
with open(path, "w", encoding="utf-8") as fh:
fh.write(sidecar(titles))
def _copy_assets(self, out):
"""Copy the static directories in, so the capsule stands on its own.
A capsule whose images are fetched over HTTPS from the web site is not
really a capsule, and link targets are resolved against the capsule
root, so pointing outside it would not work anyway.
"""
s = self.settings
names = s.get("SMOLNET_ASSETS")
if names is None:
names = [n for n in s.get("STATIC_PATHS", []) if not n.startswith(".")]
for name in names:
src = os.path.join(self.path, name)
if not os.path.isdir(src):
continue
shutil.copytree(src, os.path.join(out, name), dirs_exist_ok=True)
def _warn(self, fmt, *args):
try:
from pelican.log import logger
logger.warning(fmt, *args)
except Exception:
pass
def get_generators(pelican_object):
return CapsuleGenerator
def register():
signals.get_generators.connect(get_generators)
Reference in a new issue View git blame Copy permalink