randogoth/itsybitsy
1
0
Fork 0
Code Issues Pull requests Projects Releases 3 Packages Wiki Activity Actions

feat: serve gopher with dot-stuffing and a synthetic root menu

Browse source
This commit is contained in:
randogoth 2026-10-04 21:20:38 +03:00
parent 1de4674b09
commit dc2114a4d2
8 changed files with 334 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

22
README.md
View file

@ -2,13 +2,13 @@
[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE) [![AI-DECLARATION: copilot](https://img.shields.io/badge/%E4%B7%BC%20AI--DECLARATION-copilot-fee2e2?labelColor=fee2e2)](https://ai-declaration.md)
Serve folders of Markdown to different domains over HTTP, [Spartan](https://portal.mozz.us/gemini/spartan.mozz.us/) and [Nex](https://nightfall.city/nex/info/specification.txt), rendered on the fly with no build step.
Serve folders of Markdown to different domains over HTTP, [Spartan](https://portal.mozz.us/gemini/spartan.mozz.us/), [Nex](https://nightfall.city/nex/info/specification.txt) and [Gopher](https://www.rfc-editor.org/rfc/rfc1436), rendered on request with no build step.
A Rust re-implementation of [smolweb](https://code.randogoth.com/randogoth/smolweb), which serves one folder from one process with its output formats hardcoded and its configuration in per-file frontmatter. itsybitsy changes all three: one process serves many domains, each output format is a separate crate behind a cargo feature, and configuration lives in a TOML file per content directory.
## Status
It serves. HTTP, Spartan and Nex are working across five output formats, with virtual hosting on all three. Gemini and Gopher are the remaining protocols.
It serves. HTTP, Spartan, Nex and Gopher are working across five output formats, with virtual hosting wherever the protocol carries a hostname. Gemini is the one protocol left.
| Area | State |
| --- | --- |
@ -20,7 +20,8 @@ It serves. HTTP, Spartan and Nex are working across five output formats, with vi
| Fixed-width text output (Nex, later Gopher) | implemented |
| FIGlet banners and hyphenation, behind features | implemented |
| WML decks and card sub-documents, behind a feature | implemented |
| Gemini and Gopher | planned |
| Gopher, with dot-stuffing and the RFC 4266 root menu | implemented |
| Gemini | planned |
## Configuration
@ -142,6 +143,21 @@ cargo build --release --features "figlet hyphenation"
A banner that will not fit the line, names a font this build lacks, or is asked for by a build without `figlet` falls back to the level's underline — a heading that cannot be decorated should not be lost. An unknown font name is logged, since that is almost always a typo. Likewise `hyphenate` has no effect without the feature, and an unknown `hyphen_lang` leaves the text unhyphenated; a ragged right edge is the plain-text convention anyway.
## Protocols
| Protocol | Routes by | Notes |
| --- | --- | --- |
| HTTP | `Host` header | `GET` and `HEAD`, negotiated by `Accept` |
| Spartan | the host field of its request line | One format per listener |
| Nex | nothing; its listener names one site | No status line at all |
| Gopher | nothing; its listener names one site | Item type 0 and the root menu |
A protocol with no hostname in its requests cannot be routed by one, so its listener names a single site outright and configuration refuses to leave that implicit once more than one site exists. HTTP and Spartan fall back to `default_site` when a request names a host this server does not know.
Neither Nex nor Gopher has a redirect status, so a canonical target is resolved server-side and the real content comes back on the first request rather than a bounce.
Gopher serves item type 0 (text) and one menu: a prefix-less `gopher://host/` means item type 1 by RFC 4266, so an empty selector gets a one-item menu pointing at `/` rather than the root document, which would be the wrong type. Text items are dot-stuffed and terminated with a lone dot; binary items are sent raw, since dot-stuffing would corrupt them and a terminator would become part of the file. There are no generated directory listings and no type 7 search.
## Card sub-documents
WML is the one output that paginates: a WAP 1.x handset has a hard per-card byte budget and refuses a deck that exceeds it. So a long page becomes a chain of screens, and a page with dividers becomes a menu card linking to the rest.

7
bin/Cargo.toml
View file

@ -12,10 +12,9 @@ name = "itsybitsy"
path = "src/main.rs"
[features]
default = ["http", "spartan", "nex", "gemtext", "wap", "text"]
http = []
spartan = []
nex = []
# Every protocol listener is always compiled in: they are a few hundred lines of
# line parsing between them, with no dependency of their own to gate.
default = ["gemtext", "wap", "text"]
gemtext = ["dep:itsybitsy-gemtext"]
wap = ["dep:itsybitsy-wap"]
# WML 1.3 decks for WAP 1.x handsets.

165
bin/src/proto/gopher.rs Normal file
View file

@ -0,0 +1,165 @@
//! Gopher, RFC 1436: a selector in, a text file or a menu out.
//!
//! Only item type 0 (text) and the one menu described below. There are no
//! gophermap directory listings and no type 7 search, because neither has a
//! source in a folder of Markdown.
//!
//! Like Nex there is no status line and no redirect, so an error is a plain body
//! and a canonical target is resolved server-side rather than bounced.
use std::io::{BufRead, BufReader, Read, Write};
use std::net::TcpStream;
use anyhow::Result;
use itsybitsy_core::site::{Resolution, Resource};
use crate::proto::for_log;
use crate::serve::Listener;
/// Gopher selectors are short; the cap is smolweb's.
const MAX_REQUEST: usize = 512;
pub fn serve(listener: &Listener, mut stream: TcpStream) -> Result<()> {
let selector = {
let mut reader = BufReader::new(stream.try_clone()?);
read_selector(&mut reader, MAX_REQUEST)
};
let Some(selector) = selector else {
log::warn!("{}: unreadable or oversized selector", listener.name);
return text(&mut stream, b"Bad request\n");
};
// Gopher carries no hostname, so the listener names its site outright.
let site = listener.only_site();
log::info!("{} gopher {}", listener.name, for_log(&selector));
// A prefix-less `gopher://host/` means item type 1 by RFC 4266, so a client
// asking for the root expects a menu rather than a text file. Serving the
// root document here would be the wrong type; a one-item menu pointing at
// `/` gets the client to the same place with the type it asked for.
if selector.is_empty() {
return menu(&mut stream, listener, &[("0", "Home", "/")]);
}
match site.resolve_flat(&selector) {
Ok(Resolution::Found(Resource::Document { page, .. })) => {
match page.body(&listener.formats[0]) {
Some(body) => text(&mut stream, body),
None => text(&mut stream, b"Internal error\n"),
}
}
Ok(Resolution::Found(Resource::Raw { path, .. })) => {
// A binary item is sent raw and ended by closing: dot-stuffing it
// would corrupt it, and a terminator would be part of the file.
let mut file = std::fs::File::open(&path)?;
std::io::copy(&mut file, &mut stream)?;
Ok(())
}
// resolve_flat follows redirects and sub-documents to real content.
Ok(_) => text(&mut stream, b"Not found\n"),
Err(err) => {
log::warn!("{} gopher {}: {err}", listener.name, for_log(&selector));
text(&mut stream, b"Internal error\n")
}
}
}
/// Read a selector: everything up to the first tab or line ending.
///
/// A tab separates a selector from a search term, which this server has no use
/// for but must not read as part of the path.
fn read_selector<R: std::io::Read>(reader: &mut BufReader<R>, cap: usize) -> Option<String> {
let mut line = Vec::new();
let mut limited = reader.take(cap as u64 + 1);
limited.read_until(b'\n', &mut line).ok()?;
if line.len() > cap {
return None;
}
let end =
line.iter().position(|byte| matches!(byte, b'\t' | b'\r' | b'\n')).unwrap_or(line.len());
String::from_utf8(line[..end].to_vec()).ok()
}
/// Send a text item: dot-stuffed, then the lone-dot terminator.
fn text(stream: &mut TcpStream, body: &[u8]) -> Result<()> {
stream.write_all(&dot_stuff(body))?;
// A line holding only a dot ends the item, which is why one in the content
// has to be doubled above.
stream.write_all(b".\r\n")?;
Ok(())
}
/// Send a menu: tab-separated items, then the terminator.
fn menu(stream: &mut TcpStream, listener: &Listener, items: &[(&str, &str, &str)]) -> Result<()> {
let mut out = Vec::new();
for (kind, display, selector) in items {
out.extend_from_slice(
format!(
"{kind}{display}\t{selector}\t{}\t{}\r\n",
listener.advertised_host, listener.advertised_port
)
.as_bytes(),
);
}
stream.write_all(&out)?;
stream.write_all(b".\r\n")?;
Ok(())
}
/// Double a leading dot on any line, so no content line can end the item.
fn dot_stuff(body: &[u8]) -> Vec<u8> {
let mut out = Vec::with_capacity(body.len());
let mut at_line_start = true;
for byte in body {
if at_line_start && *byte == b'.' {
out.push(b'.');
}
out.push(*byte);
at_line_start = *byte == b'\n';
}
out
}
#[cfg(test)]
mod tests {
use super::*;
fn read(input: &[u8]) -> Option<String> {
read_selector(&mut BufReader::new(input), MAX_REQUEST)
}
#[test]
fn reads_a_selector_up_to_its_terminator() {
assert_eq!(read(b"/about\r\n").as_deref(), Some("/about"));
assert_eq!(read(b"/about\n").as_deref(), Some("/about"));
assert_eq!(read(b"\r\n").as_deref(), Some(""));
}
#[test]
fn a_tab_separates_a_search_term_that_is_not_part_of_the_path() {
assert_eq!(read(b"/about\tsearch words\r\n").as_deref(), Some("/about"));
}
#[test]
fn refuses_an_oversized_selector() {
let mut long = vec![b'a'; MAX_REQUEST + 1];
long.extend_from_slice(b"\r\n");
assert_eq!(read(&long), None);
}
#[test]
fn dot_stuffing_protects_a_leading_dot_on_every_line() {
// A line holding only a dot is the terminator, so one in the content has
// to be doubled or the item would end early.
assert_eq!(dot_stuff(b".\n"), b"..\n");
assert_eq!(dot_stuff(b"a\n.b\nc\n"), b"a\n..b\nc\n".to_vec());
// A dot that is not at the start of a line is ordinary text.
assert_eq!(dot_stuff(b"a.b\n"), b"a.b\n".to_vec());
assert_eq!(dot_stuff(b"plain\n"), b"plain\n".to_vec());
}
#[test]
fn dot_stuffing_handles_a_dot_at_the_very_start() {
assert_eq!(dot_stuff(b".hidden"), b"..hidden".to_vec());
}
}

1
bin/src/proto/mod.rs
View file

@ -1,5 +1,6 @@
//! Shared plumbing for the protocol listeners.
pub mod gopher;
pub mod http;
pub mod negotiate;
pub mod nex;

32
bin/src/serve.rs
View file

@ -31,6 +31,10 @@ pub struct Listener {
/// The single site served by a protocol that carries no hostname.
pub site: Option<String>,
pub timeout_secs: u64,
/// Hostname to advertise in a Gopher menu, which names the host to fetch each
/// item from rather than assuming the client remembers.
pub advertised_host: String,
pub advertised_port: u16,
max_connections: u32,
open: AtomicU32,
}
@ -84,7 +88,11 @@ pub fn bind(
for (name, spec) in &config.listener {
let socket = TcpListener::bind(&spec.bind)
.with_context(|| format!("listener '{name}' binding {}", spec.bind))?;
bound.push(Bound { listener: Arc::new(listener(name, spec, &registry, &sites)), socket });
// Built after the bind so it can carry the port it actually got, which a
// listener bound to port 0 only learns here.
let local = socket.local_addr().ok();
let listener = listener(name, spec, config, &registry, &sites, local);
bound.push(Bound { listener: Arc::new(listener), socket });
}
Ok(bound)
}
@ -120,9 +128,22 @@ pub fn run(config: &ServerConfig, registry: Arc<Registry>, sites: Arc<SiteSet>)
fn listener(
name: &str,
spec: &ListenerSpec,
config: &ServerConfig,
registry: &Arc<Registry>,
sites: &Arc<SiteSet>,
local: Option<std::net::SocketAddr>,
) -> Listener {
// A Gopher menu has to name a host reachable from outside, so the site's own
// first hostname is better than whatever address the socket is bound to.
let advertised_host = spec
.site
.as_deref()
.or(spec.default_site.as_deref())
.and_then(|name| config.site.get(name))
.and_then(|site| site.hosts.first().cloned())
.or_else(|| local.map(|addr| addr.ip().to_string()))
.unwrap_or_else(|| "localhost".to_string());
Listener {
name: name.to_string(),
protocol: spec.protocol,
@ -132,6 +153,8 @@ fn listener(
default_site: spec.default_site.clone(),
site: spec.site.clone(),
timeout_secs: spec.timeout_secs,
advertised_host,
advertised_port: local.map(|addr| addr.port()).unwrap_or(70),
max_connections: spec.max_connections,
open: AtomicU32::new(0),
}
@ -178,9 +201,10 @@ fn handle(listener: &Listener, stream: TcpStream) {
Protocol::Http => proto::http::serve(listener, stream),
Protocol::Spartan => proto::spartan::serve(listener, stream),
Protocol::Nex => proto::nex::serve(listener, stream),
// Validation refuses these until their listeners exist.
Protocol::Gemini | Protocol::Gopher => {
unreachable!("{} is rejected at startup", listener.protocol.as_str())
Protocol::Gopher => proto::gopher::serve(listener, stream),
// Configuration validation refuses this, so it cannot be reached.
Protocol::Gemini => {
unreachable!("gemini is refused at startup until its listener exists")
}
}
}));

97
bin/tests/listeners.rs
View file

@ -27,6 +27,8 @@ impl Server {
write(dir.path(), "one/about.md", "# About\n\nAbout the first.\n");
write(dir.path(), "one/img.png", "not really a png");
write(dir.path(), "one/.itsybitsy.toml", "[defaults]\nmax_age = 120\n");
// A line starting with a dot, which Gopher has to double.
write(dir.path(), "one/dotted.md", "# Dotted\n\n .hidden\n");
write(dir.path(), "two/index.md", "# Two\n\nSecond site.\n");
write(
dir.path(),
@ -69,6 +71,12 @@ protocol = "nex"
bind = "127.0.0.1:0"
formats = ["gemtext"]
site = "two"
[listener.gopher]
protocol = "gopher"
bind = "127.0.0.1:0"
formats = ["gemtext"]
site = "one"
"#,
one = dir.path().join("one").to_str().unwrap(),
two = dir.path().join("two").to_str().unwrap(),
@ -408,6 +416,95 @@ fn nothing_outside_the_root_is_reachable_over_any_protocol() {
assert_eq!(server.send("nex", b"/.itsybitsy.toml\r\n"), "Not found\n");
}
// -- Gopher --------------------------------------------------------------
//
// Ported from smolweb's tests/test_gopher.py, where the exact wire bytes are the
// assertion: Gopher has no status line, so framing is all a client has.
#[test]
fn an_empty_selector_gets_a_one_item_menu() {
// RFC 4266: a prefix-less `gopher://host/` means item type 1, so a client
// asking for the root expects a menu rather than a text file.
let server = Server::start();
let port = server.addr("gopher").port();
let response = server.send("gopher", b"\r\n");
assert_eq!(response, format!("0Home\t/\tone.test\t{port}\r\n.\r\n"));
}
#[test]
fn the_root_selector_serves_the_document_as_text() {
let server = Server::start();
let response = server.send("gopher", b"/\r\n");
assert!(response.starts_with("# One"), "{response:?}");
assert!(response.ends_with(".\r\n"), "{response:?}");
}
#[test]
fn a_text_item_ends_with_the_lone_dot_terminator() {
let server = Server::start();
let response = server.send("gopher", b"/about\r\n");
assert!(response.contains("About the first."), "{response:?}");
assert!(response.ends_with("\n.\r\n"), "{response:?}");
}
#[test]
fn a_missing_selector_is_a_well_formed_text_item() {
// No status to report with, so the error is the item's content. smolweb says
// "Not found." here and "Not found" over HTTP and Nex; one wording is used
// across every protocol instead.
let server = Server::start();
assert_eq!(server.send("gopher", b"/nope\r\n"), "Not found\n.\r\n");
}
#[test]
fn a_selector_without_a_leading_slash_still_resolves() {
// Clients built from a menu line send the selector verbatim, and a menu may
// carry it either way.
let server = Server::start();
assert!(server.send("gopher", b"about\r\n").contains("About the first."));
}
#[test]
fn a_tab_separated_search_term_is_not_part_of_the_selector() {
let server = Server::start();
let response = server.send("gopher", b"/about\tignored words\r\n");
assert!(response.contains("About the first."), "{response:?}");
}
#[test]
fn gopher_resolves_a_redirect_server_side() {
// There is no redirect item type to bounce with.
let server = Server::start();
let response = server.send("gopher", b"/about.md\r\n");
assert!(response.contains("About the first."), "{response:?}");
}
#[test]
fn a_binary_item_is_sent_raw_with_no_terminator() {
// Dot-stuffing it would corrupt it, and a terminator would become part of
// the file.
let server = Server::start();
let response = server.send("gopher", b"/img.png\r\n");
assert_eq!(response, "not really a png");
}
#[test]
fn a_leading_dot_in_the_content_is_doubled() {
let server = Server::start();
let response = server.send("gopher", b"/dotted\r\n");
// The source line is `.hidden`; undoubled it would end the item early.
assert!(response.contains("\n..hidden"), "{response:?}");
assert!(response.ends_with(".\r\n"), "{response:?}");
}
#[test]
fn an_oversized_selector_is_refused() {
let server = Server::start();
let mut request = vec![b'a'; 600];
request.extend_from_slice(b"\r\n");
assert_eq!(server.send("gopher", &request), "Bad request\n.\r\n");
}
// -- Sub-documents -------------------------------------------------------
/// A page that paginates addresses sub-documents under its own URL. Only a

15
core/src/config.rs
View file

@ -125,6 +125,15 @@ impl Protocol {
pub fn negotiates(self) -> bool {
matches!(self, Protocol::Http)
}
/// Whether a listener exists for this protocol.
///
/// Gemini needs a TLS stack and per-host certificate selection, which is its
/// own piece of work; naming it in a configuration is refused here rather
/// than panicking on the first connection.
pub fn is_implemented(self) -> bool {
!matches!(self, Protocol::Gemini)
}
}
/// What validation established, and what `--check` prints.
@ -254,6 +263,12 @@ impl ServerConfig {
fn validate_listeners(&self, available: &[&str]) -> Result<BTreeSet<String>, Error> {
let mut formats = BTreeSet::new();
for (name, spec) in &self.listener {
if !spec.protocol.is_implemented() {
return Err(Error::config(format!(
"listener '{name}': {} is not implemented yet",
spec.protocol.as_str()
)));
}
if spec.formats.is_empty() {
return Err(Error::config(format!("listener '{name}': formats must not be empty")));
}

6
itsybitsy.toml
View file

@ -21,3 +21,9 @@ protocol = "nex"
bind = "127.0.0.1:1900"
formats = ["text"]
site = "demo"
[listener.gopher]
protocol = "gopher"
bind = "127.0.0.1:7070"
formats = ["text"]
site = "demo"