docs: condense the README, fold the audit summary in, and tighten the declaration
This commit is contained in:
randogoth
parent
19e7cfaca3
commit
bb7a5111a1
3 changed files with 68 additions and 289 deletions
234
README.md
234
README.md
|
|
@ -1,32 +1,24 @@
|
|||
# 🕸 itsybitsy
|
||||
|
||||
[](LICENSE) [](https://ai-declaration.md)
|
||||
[](LICENSE) [](https://ai-declaration.md) 
|
||||
|
||||
Serve folders of Markdown to different domains over HTTP, [Gemini](https://geminiprotocol.net/docs/specification.gmi), [Spartan](https://portal.mozz.us/gemini/spartan.mozz.us/), [Nex](https://nightfall.city/nex/info/specification.txt) and [Gopher](https://www.rfc-editor.org/rfc/rfc1436), rendered on request with no build step.
|
||||
|
||||
A Rust re-implementation of [smolweb](https://code.randogoth.com/randogoth/smolweb), which serves one folder from one process with its output formats hardcoded and its configuration in per-file frontmatter. itsybitsy changes all three: one process serves many domains, each output format is a separate crate behind a cargo feature, and configuration lives in a TOML file per content directory.
|
||||
Serve folders of Markdown to different domains over HTTP, [Gemini](https://geminiprotocol.net/docs/specification.gmi), [Spartan](https://portal.mozz.us/gemini/spartan.mozz.us/), [Nex](https://nightfall.city/nex/info/specification.txt) and [Gopher](https://www.rfc-editor.org/rfc/rfc1436), rendered on request with no build step. One process serves many virtual hosts, each output format is a separate crate behind a cargo feature, and configuration lives in TOML files. Gemini expects TLS to be terminated in front of it; that is the one deployment requirement itsybitsy does not satisfy by itself.
|
||||
|
||||
## Status
|
||||
The output formats are Rust ports of previous projects: the standalone Markdown to esoteric text format converter [md2txt](https://code.randogoth.com/randogoth/md2txt) and a spinoff from that for WML, [wapdown](https://code.randogoth.com/randogoth/wapdown).
|
||||
|
||||
It serves. All five protocols work across five output formats, with virtual hosting wherever the protocol carries a hostname. Gemini expects TLS to be terminated in front of it, which is the one deployment requirement itsybitsy does not satisfy by itself.
|
||||
## Deployments
|
||||
|
||||
| Area | State |
|
||||
| --- | --- |
|
||||
| Server configuration, virtual hosts, `--check` | implemented |
|
||||
| Path resolution, traversal containment, live reload | implemented |
|
||||
| Markdown parsing, includes, ASCII art, card breaks | implemented |
|
||||
| gemtext, XHTML-MP and HTML output | implemented |
|
||||
| HTTP, Spartan and Nex listeners, virtual hosting | implemented |
|
||||
| Fixed-width text output (Nex, later Gopher) | implemented |
|
||||
| FIGlet banners and hyphenation, behind features | implemented |
|
||||
| WML decks and card sub-documents, behind a feature | implemented |
|
||||
| Gopher, with dot-stuffing and the RFC 4266 root menu | implemented |
|
||||
| Gemini, behind a TLS terminator | implemented |
|
||||
| Built-in TLS, so no terminator is needed | not planned for now |
|
||||
Itsybitsy can be seen in action running the Gemini/Spartan/Nex/Gopher content for https://smol.place, the Gemini/Spartan capsules for https://randogoth.com and https://wap.randogoth.com
|
||||
|
||||
## Configuration
|
||||
## Usage
|
||||
|
||||
Two files. A server file names the virtual hosts and the listeners:
|
||||
```bash
|
||||
itsybitsy --config /etc/itsybitsy.toml # serve
|
||||
itsybitsy --config /etc/itsybitsy.toml --check # validate configuration only
|
||||
```
|
||||
|
||||
A server file names the sites and the listeners:
|
||||
|
||||
```toml
|
||||
version = 1
|
||||
|
|
@ -48,167 +40,32 @@ formats = ["gemtext"]
|
|||
site = "smol"
|
||||
```
|
||||
|
||||
A page is rendered into the union of what the enabled listeners can serve, and no more: a configuration with no HTTP listener never builds any HTML. HTTP and Spartan route by the hostname the request carries, falling back to `default_site`; Nex carries none, so its listener names one site outright.
|
||||
HTTP and Spartan route by the hostname the request carries, falling back to `default_site`; Nex and Gopher carry none, so their listener names one site outright. Every format a listener serves must be compiled in, which `--check` reports. A `.itsybitsy.toml` in a content directory sets rendering options for that directory — `[defaults]` for all its Markdown files, `[page."name.md"]` for one — with no inheritance from parent directories. Editing it re-renders on the next request.
|
||||
|
||||
A `.itsybitsy.toml` in any content directory says how the Markdown in **that directory** renders. There is no inheritance: a subdirectory without its own file uses the built-in defaults rather than its parent's. Repetition in a deep tree is the price of never having to look elsewhere to know how a page renders.
|
||||
### Formats and protocols
|
||||
|
||||
```toml
|
||||
[defaults]
|
||||
max_age = 3600
|
||||
margin_left = 2
|
||||
h1_style = "underline:="
|
||||
| Format | Served as | Behind feature |
|
||||
| --- | --- | --- |
|
||||
| `gemtext` | `text/gemini` | default |
|
||||
| `xhtmlmp`, `html` | WAP and desktop markup | default |
|
||||
| `text` | `text/plain`, wrapped to 80 columns | default |
|
||||
| `wml` | WML 1.3 decks for WAP 1.x handsets | `wml` |
|
||||
| FIGlet banners, hyphenation for `text` | | `figlet`, `hyphenation` |
|
||||
|
||||
[page."index.md"]
|
||||
title = "Notes"
|
||||
max_age = 300
|
||||
```
|
||||
HTTP negotiates formats from `Accept` and `?format=<id>` overrides it. Two decorative text features are off by default to save binary size: `cargo build --release --features "figlet hyphenation"`.
|
||||
|
||||
`[defaults]` applies to every Markdown file in the directory; a `[page."name.md"]` table overrides it for one file. `title` is per-page by nature, so setting it in `[defaults]` is an error; left unset, it is derived from the page's first level-1 heading.
|
||||
### Markdown
|
||||
|
||||
The fixed-width text format reads `margin_left`, `margin_right`, `paragraph_spacing`, `h1_style` through `h6_style`, `blockquote_bars`, `list_indent`, `code_block_line_numbers` and `wrap_code_blocks`. A heading style is `underline`, `underline:<char>`, `figlet`, `figlet:<font>`, `markers` or `plain`; by default the first three levels are underlined with `=`, `-` and `~`, and deeper ones keep their `#` markers, since underline characters run out before heading levels do. Editing the file re-renders that directory's pages on the next request, and a file that fails to parse makes them error rather than silently falling back to defaults.
|
||||
|
||||
Because the name begins with a dot, the same rule that keeps `.secret.md` out of the URL space keeps this file out of it too.
|
||||
|
||||
`--check` validates the configuration and reports the routing it resolved, without binding a port:
|
||||
|
||||
```console
|
||||
$ itsybitsy --config /etc/itsybitsy.toml --check
|
||||
sites:
|
||||
smol: /srv/smol/content
|
||||
hosts:
|
||||
smol.place -> smol
|
||||
www.smol.place -> smol
|
||||
listeners:
|
||||
web: http on 0.0.0.0:8080 [xhtmlmp, html] by host, default smol
|
||||
formats rendered per page: html, xhtmlmp
|
||||
```
|
||||
|
||||
Everything the schema cannot express is checked here rather than on first request: a format no enabled feature provides, two sites claiming one hostname, a content root that does not exist, a server file sitting inside a content root where it would be served.
|
||||
|
||||
## Markdown
|
||||
|
||||
Markdown is parsed once per document into one representation that every output format reads, rather than once per format. Everything itsybitsy adds to CommonMark is either a construct other tools already understand or invisible to them, so a document stays portable:
|
||||
Markdown is parsed once per document and rendered to every format. Everything itsybitsy adds to CommonMark stays portable:
|
||||
|
||||
| Written | Means |
|
||||
| --- | --- |
|
||||
| `![[path]]` | splice that file's lines in here |
|
||||
| `` | an image whose target is a text file, inlined verbatim |
|
||||
| `<!-- card Title -->` | a card divider, for formats that paginate |
|
||||
| `---` | an untitled divider |
|
||||
| `<!-- center -->`, `<!-- right margin=4 -->` | alignment for the block that follows |
|
||||
|
||||
Only the include is non-standard, and it is the spelling Obsidian and its relatives established; CommonMark renders it as literal text, which is why it is the one construct handled before parsing. The rest are ordinary HTML comments and images, interpreted after parsing — so an HTML comment that is not a recognised directive stays a comment, at the cost of a misspelled one doing nothing rather than complaining.
|
||||
|
||||
Alignment only reaches the fixed-width text formats; gemtext and HTML have no way to express it. An art label may carry its own `:center`, `:left` or `:right` token, which is removed from the label so it does not leak into the fallback text a gemtext or HTML client shows.
|
||||
|
||||
Include and art targets resolve relative to the file holding the reference and must stay inside the content root. Expansion is bounded on three axes — nesting depth, total lines, and total bytes — because a cycle check alone does not stop a long chain, and neither stops a diamond where two branches include the same file without ever repeating one on a single path.
|
||||
|
||||
### Differences from smolweb
|
||||
|
||||
Configuration keys are not carried over verbatim either: md2txt accepts three spellings of `paragraph_spacing`, and its `cache_control` is named for the header it lands in rather than the value it holds, which here is `max_age`. Its `text` and `nex` renderers differ only in heading decoration and in whether links are inlined or numbered — and the numbered form never writes the reference list its numbers point at, so there is one text renderer here rather than two. Tables, which both of its text renderers drop entirely, are rendered as aligned columns.
|
||||
|
||||
wapdown's own parser produces neither tables nor lists, so its WML decks render a bullet list as literal text and drop tables entirely; both come out as real markup here, lists as marked lines and tables as `<table>`, which WML has.
|
||||
|
||||
Gemini diverges in three places. smolweb answers 59 (bad request) to any URL that is not `gemini://`, which tells a client its request was malformed when an `https://` URL is merely for somewhere this server does not fetch from; that is 53 (proxy request refused) here, and 59 is kept for a URL that genuinely does not parse. smolweb builds absolute redirect targets from the port it is bound to, which is wrong behind a terminator; these are relative. And smolweb terminates TLS itself with a certificate named per site, so `tls` is not a configuration key here at all — naming one is an error rather than a setting that quietly does nothing.
|
||||
|
||||
smolweb inherits five invented directives from its two renderer libraries: `{.card}`, `{.include}`, `![[ ]]`, `#[label](art.txt)` and MultiMarkdown attribute lists (`{: .center}`) — two incompatible brace grammars, with art alignment expressible two different ways in one line. None of it appeared in real content, so itsybitsy expresses the same capabilities with standard constructs instead. Parsing once rather than once per library also means the text formats gain setext headings and the WAP formats gain tables and ASCII art, none of which their original parser handled.
|
||||
|
||||
## Output formats
|
||||
|
||||
Each format is a crate implementing one trait, compiled in behind a cargo feature. A renderer is handed the parsed document and may not open files or sockets, which is what keeps every path check in one place rather than in each format.
|
||||
|
||||
| Format | Served as | Notes |
|
||||
| --- | --- | --- |
|
||||
| `gemtext` | `text/gemini` | Unwrapped; Gemini and Spartan clients wrap for themselves |
|
||||
| `xhtmlmp` | `application/vnd.wap.xhtml+xml` | Well-formed XML with the Mobile Profile doctype |
|
||||
| `html` | `text/html` | The same markup without the XML declaration |
|
||||
| `text` | `text/plain` | Wrapped to 80 columns, because Nex and Gopher clients do not wrap |
|
||||
| `wml` | `text/vnd.wap.wml` | WAP 1.x decks, paginated to a per-card byte budget (feature `wml`) |
|
||||
|
||||
HTTP negotiates between them from `Accept`. Only a literal media type counts as a match, so a browser's `*/*` can never be read as willingness to receive a WAP format; `?format=<id>` overrides negotiation outright, and naming an id the listener does not serve is a bad request rather than a silent fallback.
|
||||
|
||||
A third party adds a format by writing a crate that depends on `itsybitsy-core`, adding it to the workspace and one `#[cfg]` arm in the binary's registry, then naming its id in `formats`. Every listener's formats are checked against the registry at startup, so a format switched off by a feature is a configuration error naming the missing id rather than a server error on the first request.
|
||||
|
||||
## Optional features
|
||||
|
||||
Two decorative capabilities for the text format are off by default, because most pages leave them alone and both cost binary size:
|
||||
|
||||
| Feature | Adds | Cost |
|
||||
| --- | --- | --- |
|
||||
| `wml` | WML 1.3 decks for WAP 1.x handsets | +37 KiB |
|
||||
| `figlet` | `h1_style = "figlet"` banners, fonts `small` and `standard` | +60 KiB |
|
||||
| `hyphenation` | `hyphenate = true`, English patterns | +127 KiB |
|
||||
| `hyphenation-all` | every language the pattern crate carries | +3.0 MiB |
|
||||
|
||||
```bash
|
||||
cargo build --release --features "figlet hyphenation"
|
||||
```
|
||||
|
||||
A banner that will not fit the line, names a font this build lacks, or is asked for by a build without `figlet` falls back to the level's underline — a heading that cannot be decorated should not be lost. An unknown font name is logged, since that is almost always a typo. Likewise `hyphenate` has no effect without the feature, and an unknown `hyphen_lang` leaves the text unhyphenated; a ragged right edge is the plain-text convention anyway.
|
||||
|
||||
## Protocols
|
||||
|
||||
| Protocol | Routes by | Notes |
|
||||
| --- | --- | --- |
|
||||
| HTTP | `Host` header | `GET` and `HEAD`, negotiated by `Accept` |
|
||||
| Spartan | the host field of its request line | One format per listener |
|
||||
| Nex | nothing; its listener names one site | No status line at all |
|
||||
| Gemini | the authority of the URL it is sent | TLS terminated in front; one format per listener |
|
||||
| Gopher | nothing; its listener names one site | Item type 0 and the root menu |
|
||||
|
||||
A protocol with no hostname in its requests cannot be routed by one, so its listener names a single site outright and configuration refuses to leave that implicit once more than one site exists. HTTP, Gemini and Spartan fall back to `default_site` when a request names a host this server does not know.
|
||||
|
||||
Neither Nex nor Gopher has a redirect status, so a canonical target is resolved server-side and the real content comes back on the first request rather than a bounce.
|
||||
|
||||
### Gemini and TLS
|
||||
|
||||
itsybitsy does not speak TLS. The Gemini listener expects plaintext, so a TLS wrapper terminates in front of it and forwards to a loopback port:
|
||||
|
||||
```toml
|
||||
[listener.gemini]
|
||||
protocol = "gemini"
|
||||
# The terminator owns 1965; this is its back end.
|
||||
bind = "127.0.0.1:11965"
|
||||
formats = ["gemtext"]
|
||||
default_site = "smol"
|
||||
```
|
||||
|
||||
```ini
|
||||
; /etc/stunnel/gemini.conf
|
||||
[gemini]
|
||||
accept = 1965
|
||||
connect = 127.0.0.1:11965
|
||||
cert = /var/lib/itsybitsy/gemini.pem
|
||||
```
|
||||
|
||||
`stunnel` and `ghostunnel` are built for exactly this and support SNI, so each virtual host can present its own certificate. Use a long-lived self-signed certificate rather than an ACME one: Gemini clients pin the fingerprint they first saw, and a renewal that changes it is indistinguishable from an interception.
|
||||
|
||||
Two things follow from terminating in front, and neither is a limitation that can be configured away. Every connection appears to come from the terminator, so request logs show its address unless it speaks the PROXY protocol, which itsybitsy does not yet read. And a client certificate never reaches us, so statuses 60 to 62 are not implementable — itsybitsy serves static documents and has nothing to authenticate, so only the logs are poorer for it.
|
||||
|
||||
Redirects are relative references, which the spec permits and which are the only correct form here: the port this listener is bound to is the terminator's back end, not a port any client reached, so an absolute URL built from it would point somewhere unpublished.
|
||||
|
||||
Gopher serves item type 0 (text) and one menu: a prefix-less `gopher://host/` means item type 1 by RFC 4266, so an empty selector gets a one-item menu pointing at `/` rather than the root document, which would be the wrong type. Text items are dot-stuffed and terminated with a lone dot; binary items are sent raw, since dot-stuffing would corrupt them and a terminator would become part of the file. There are no generated directory listings and no type 7 search.
|
||||
|
||||
## Card sub-documents
|
||||
|
||||
WML is the one output that paginates: a WAP 1.x handset has a hard per-card byte budget and refuses a deck that exceeds it. So a long page becomes a chain of screens, and a page with dividers becomes a menu card linking to the rest.
|
||||
|
||||
With `deck_per_card`, each card also gets its own URL under the page's:
|
||||
|
||||
| Request | WML client | Any other format |
|
||||
| --- | --- | --- |
|
||||
| `/trail` | the menu deck | the whole page |
|
||||
| `/trail/weather` | that card's own deck | redirects to `/trail` |
|
||||
| `/trail/nonexistent` | redirects to `/trail` | redirects to `/trail` |
|
||||
| `/about/whatever` | not found | not found |
|
||||
|
||||
That URL space belongs to the format that claims it. A format which does not address a sub-document redirects to the parent rather than substituting something else, and a page with no dividers has no such URLs at all — `/about/whatever` is not a sub-document just because `/about` exists. Nex and Gopher have no redirect status, so they resolve to the parent's content directly instead of bouncing.
|
||||
|
||||
The deck keys are `split_level`, `split_on_rule`, `max_card_bytes`, `menu`, `menu_style`, `deck_per_card`, `template_nav`, `nav_next_label`, `nav_prev_label`, `nav_back_label`, `home_label` and `images`.
|
||||
|
||||
## URLs
|
||||
|
||||
Links should be root-relative and extensionless (`[about](/about)`, not `about.md`), so the same link resolves identically from every protocol.
|
||||
Links should be root-relative and extensionless (`[about](/about)`), so the same link resolves identically from every protocol:
|
||||
|
||||
| Request | Serves |
|
||||
| --- | --- |
|
||||
|
|
@ -217,11 +74,46 @@ Links should be root-relative and extensionless (`[about](/about)`, not `about.m
|
|||
| `/foo.md` | redirects to `/foo` |
|
||||
| `/img.png` | the file itself, by media type |
|
||||
|
||||
Nothing outside the content root is reachable. A request target is percent-decoded before it is normalised, so an encoded `..` becomes a real one and gets clamped at the root rather than quietly matching nothing; any path component beginning with a dot is refused outright; and whatever survives is canonicalised and required to still be inside the root, which is what defeats a symlink pointing out of it. There are no generated directory listings — only `index.md`.
|
||||
Nothing outside a content root is reachable, dot-prefixed components are refused, and include expansion is depth-, line- and byte-capped. There are no generated directory listings — only `index.md`.
|
||||
|
||||
URLs the server generates are percent-encoded. A file name can contain a space, a `?`, a `#` or even a CR LF, and these URLs are emitted as redirect targets, so an unencoded one would truncate the path or end the response header and let a crafted file name inject a header of its own into every protocol.
|
||||
### Gemini and TLS
|
||||
|
||||
A document is also refused if it nests blocks more than 100 levels deep, or if it is larger than the 8 MiB an expansion may produce. Both are about the stack and the heap rather than about the content: every pass over a parsed document recurses, and a stack overflow aborts the process instead of unwinding, so one pathological file would take every virtual host down with it. Includes are separately capped at 16 levels and 200 000 lines, with the byte cap catching the diamond fan-out that a per-stack cycle check cannot see.
|
||||
itsybitsy does not speak TLS. A TLS wrapper (`stunnel`, `ghostunnel`) or a load balancer owns 1965 and forwards plaintext to a loopback port the Gemini listener binds; it supports SNI, so each virtual host can present its own certificate. Use a long-lived self-signed certificate rather than an ACME one: Gemini clients pin the fingerprint they first saw. A TLS terminator in front is also how the container deployment below runs.
|
||||
|
||||
## Deployment
|
||||
|
||||
The image builds from the repository root and runs on any OCI container host:
|
||||
|
||||
```bash
|
||||
docker build -t itsybitsy .
|
||||
docker run -p 8080:8080 -p 3000:3000 -p 1900:1900 -p 7070:7070 itsybitsy
|
||||
```
|
||||
|
||||
It serves the demo site with `docker/itsybitsy.toml`, which binds `0.0.0.0` where the checkout's own server file binds loopback, so published ports are reachable. A real site mounts over those paths:
|
||||
|
||||
```bash
|
||||
docker run -p 8080:8080 \
|
||||
-v /srv/mysite.toml:/srv/itsybitsy/itsybitsy.toml:ro \
|
||||
-v /srv/mysite:/srv/content:ro itsybitsy
|
||||
```
|
||||
|
||||
The container runs as an unprivileged user. Port 1965 stays unexposed on the host: the Gemini listener is plaintext and a TLS terminator must own it in front, as always.
|
||||
|
||||
The flake deploys without Docker:
|
||||
|
||||
```bash
|
||||
nix run . -- --config ./itsybitsy.toml # build and serve
|
||||
nix build .#static # fully static musl binary
|
||||
scp result/bin/itsybitsy host:/usr/local/bin/
|
||||
```
|
||||
|
||||
`packages.static` links against musl with no C dependency, so the one binary runs unchanged on any Linux host, including one that has never seen Nix; `nix run .#release-static` publishes it as a release asset.
|
||||
|
||||
## Security audit
|
||||
|
||||
An adversarial audit of integration, containment, protocol abuse and load (2026-10-04, 138 checks over raw sockets) found three defects, all fixed with regression tests: deeply nested documents overflowed the stack and aborted the process (parsing now refuses past 100 levels), a file name with CR LF could inject a response header via redirect targets (generated URLs are now percent-encoded), and a large Markdown file was read into memory before the 8 MiB expansion cap rejected it (reads are now capped). After the fixes, 135 traversal attempts across all five protocols returned nothing outside the content roots and include bombs were bounded, at roughly 8–9k req/s with flat memory.
|
||||
|
||||
Outstanding: the render cache is unbounded (6.2× the source size across formats), an oversized request header yields a TCP reset instead of a 400, a listener's `width` is ignored, and there is no graceful shutdown. The TLS leg in front of Gemini was not tested, and the threat model assumes the content tree is author-controlled.
|
||||
|
||||
## Development
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue