feat: serve gopher with dot-stuffing and a synthetic root menu

README.md

@@ -2,13 +2,13 @@
 
 [![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE) [![AI-DECLARATION: copilot](https://img.shields.io/badge/%E4%B7%BC%20AI--DECLARATION-copilot-fee2e2?labelColor=fee2e2)](https://ai-declaration.md)
 
-Serve folders of Markdown to different domains over HTTP, [Spartan](https://portal.mozz.us/gemini/spartan.mozz.us/) and [Nex](https://nightfall.city/nex/info/specification.txt), rendered on the fly with no build step.
+Serve folders of Markdown to different domains over HTTP, [Spartan](https://portal.mozz.us/gemini/spartan.mozz.us/), [Nex](https://nightfall.city/nex/info/specification.txt) and [Gopher](https://www.rfc-editor.org/rfc/rfc1436), rendered on request with no build step.
 
 A Rust re-implementation of [smolweb](https://code.randogoth.com/randogoth/smolweb), which serves one folder from one process with its output formats hardcoded and its configuration in per-file frontmatter. itsybitsy changes all three: one process serves many domains, each output format is a separate crate behind a cargo feature, and configuration lives in a TOML file per content directory.
 
 ## Status
 
-It serves. HTTP, Spartan and Nex are working across five output formats, with virtual hosting on all three. Gemini and Gopher are the remaining protocols.
+It serves. HTTP, Spartan, Nex and Gopher are working across five output formats, with virtual hosting wherever the protocol carries a hostname. Gemini is the one protocol left.
 
 | Area | State |
 | --- | --- |
@@ -20,7 +20,8 @@ It serves. HTTP, Spartan and Nex are working across five output formats, with vi
 | Fixed-width text output (Nex, later Gopher) | implemented |
 | FIGlet banners and hyphenation, behind features | implemented |
 | WML decks and card sub-documents, behind a feature | implemented |
-| Gemini and Gopher | planned |
+| Gopher, with dot-stuffing and the RFC 4266 root menu | implemented |
+| Gemini | planned |
 
 ## Configuration
 
@@ -142,6 +143,21 @@ cargo build --release --features "figlet hyphenation"
 
 A banner that will not fit the line, names a font this build lacks, or is asked for by a build without `figlet` falls back to the level's underline — a heading that cannot be decorated should not be lost. An unknown font name is logged, since that is almost always a typo. Likewise `hyphenate` has no effect without the feature, and an unknown `hyphen_lang` leaves the text unhyphenated; a ragged right edge is the plain-text convention anyway.
 
+## Protocols
+
+| Protocol | Routes by | Notes |
+| --- | --- | --- |
+| HTTP | `Host` header | `GET` and `HEAD`, negotiated by `Accept` |
+| Spartan | the host field of its request line | One format per listener |
+| Nex | nothing; its listener names one site | No status line at all |
+| Gopher | nothing; its listener names one site | Item type 0 and the root menu |
+
+A protocol with no hostname in its requests cannot be routed by one, so its listener names a single site outright and configuration refuses to leave that implicit once more than one site exists. HTTP and Spartan fall back to `default_site` when a request names a host this server does not know.
+
+Neither Nex nor Gopher has a redirect status, so a canonical target is resolved server-side and the real content comes back on the first request rather than a bounce.
+
+Gopher serves item type 0 (text) and one menu: a prefix-less `gopher://host/` means item type 1 by RFC 4266, so an empty selector gets a one-item menu pointing at `/` rather than the root document, which would be the wrong type. Text items are dot-stuffed and terminated with a lone dot; binary items are sent raw, since dot-stuffing would corrupt them and a terminator would become part of the file. There are no generated directory listings and no type 7 search.
+
 ## Card sub-documents
 
 WML is the one output that paginates: a WAP 1.x handset has a hard per-card byte budget and refuses a deck that exceeds it. So a long page becomes a chain of screens, and a page with dividers becomes a menu card linking to the rest.

bin/Cargo.toml

@@ -12,10 +12,9 @@ name = "itsybitsy"
 path = "src/main.rs"
 
 [features]
-default = ["http", "spartan", "nex", "gemtext", "wap", "text"]
+# Every protocol listener is always compiled in: they are a few hundred lines of
-http = []
+# line parsing between them, with no dependency of their own to gate.
-spartan = []
+default = ["gemtext", "wap", "text"]
-nex = []
 gemtext = ["dep:itsybitsy-gemtext"]
 wap = ["dep:itsybitsy-wap"]
 # WML 1.3 decks for WAP 1.x handsets.

bin/src/proto/gopher.rs

@@ -0,0 +1,165 @@
+//! Gopher, RFC 1436: a selector in, a text file or a menu out.
+//!
+//! Only item type 0 (text) and the one menu described below. There are no
+//! gophermap directory listings and no type 7 search, because neither has a
+//! source in a folder of Markdown.
+//!
+//! Like Nex there is no status line and no redirect, so an error is a plain body
+//! and a canonical target is resolved server-side rather than bounced.
+
+use std::io::{BufRead, BufReader, Read, Write};
+use std::net::TcpStream;
+
+use anyhow::Result;
+use itsybitsy_core::site::{Resolution, Resource};
+
+use crate::proto::for_log;
+use crate::serve::Listener;
+
+/// Gopher selectors are short; the cap is smolweb's.
+const MAX_REQUEST: usize = 512;
+
+pub fn serve(listener: &Listener, mut stream: TcpStream) -> Result<()> {
+    let selector = {
+        let mut reader = BufReader::new(stream.try_clone()?);
+        read_selector(&mut reader, MAX_REQUEST)
+    };
+    let Some(selector) = selector else {
+        log::warn!("{}: unreadable or oversized selector", listener.name);
+        return text(&mut stream, b"Bad request\n");
+    };
+
+    // Gopher carries no hostname, so the listener names its site outright.
+    let site = listener.only_site();
+    log::info!("{} gopher {}", listener.name, for_log(&selector));
+
+    // A prefix-less `gopher://host/` means item type 1 by RFC 4266, so a client
+    // asking for the root expects a menu rather than a text file. Serving the
+    // root document here would be the wrong type; a one-item menu pointing at
+    // `/` gets the client to the same place with the type it asked for.
+    if selector.is_empty() {
+        return menu(&mut stream, listener, &[("0", "Home", "/")]);
+    }
+
+    match site.resolve_flat(&selector) {
+        Ok(Resolution::Found(Resource::Document { page, .. })) => {
+            match page.body(&listener.formats[0]) {
+                Some(body) => text(&mut stream, body),
+                None => text(&mut stream, b"Internal error\n"),
+            }
+        }
+        Ok(Resolution::Found(Resource::Raw { path, .. })) => {
+            // A binary item is sent raw and ended by closing: dot-stuffing it
+            // would corrupt it, and a terminator would be part of the file.
+            let mut file = std::fs::File::open(&path)?;
+            std::io::copy(&mut file, &mut stream)?;
+            Ok(())
+        }
+        // resolve_flat follows redirects and sub-documents to real content.
+        Ok(_) => text(&mut stream, b"Not found\n"),
+        Err(err) => {
+            log::warn!("{} gopher {}: {err}", listener.name, for_log(&selector));
+            text(&mut stream, b"Internal error\n")
+        }
+    }
+}
+
+/// Read a selector: everything up to the first tab or line ending.
+///
+/// A tab separates a selector from a search term, which this server has no use
+/// for but must not read as part of the path.
+fn read_selector<R: std::io::Read>(reader: &mut BufReader<R>, cap: usize) -> Option<String> {
+    let mut line = Vec::new();
+    let mut limited = reader.take(cap as u64 + 1);
+    limited.read_until(b'\n', &mut line).ok()?;
+    if line.len() > cap {
+        return None;
+    }
+    let end =
+        line.iter().position(|byte| matches!(byte, b'\t' | b'\r' | b'\n')).unwrap_or(line.len());
+    String::from_utf8(line[..end].to_vec()).ok()
+}
+
+/// Send a text item: dot-stuffed, then the lone-dot terminator.
+fn text(stream: &mut TcpStream, body: &[u8]) -> Result<()> {
+    stream.write_all(&dot_stuff(body))?;
+    // A line holding only a dot ends the item, which is why one in the content
+    // has to be doubled above.
+    stream.write_all(b".\r\n")?;
+    Ok(())
+}
+
+/// Send a menu: tab-separated items, then the terminator.
+fn menu(stream: &mut TcpStream, listener: &Listener, items: &[(&str, &str, &str)]) -> Result<()> {
+    let mut out = Vec::new();
+    for (kind, display, selector) in items {
+        out.extend_from_slice(
+            format!(
+                "{kind}{display}\t{selector}\t{}\t{}\r\n",
+                listener.advertised_host, listener.advertised_port
+            )
+            .as_bytes(),
+        );
+    }
+    stream.write_all(&out)?;
+    stream.write_all(b".\r\n")?;
+    Ok(())
+}
+
+/// Double a leading dot on any line, so no content line can end the item.
+fn dot_stuff(body: &[u8]) -> Vec<u8> {
+    let mut out = Vec::with_capacity(body.len());
+    let mut at_line_start = true;
+    for byte in body {
+        if at_line_start && *byte == b'.' {
+            out.push(b'.');
+        }
+        out.push(*byte);
+        at_line_start = *byte == b'\n';
+    }
+    out
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn read(input: &[u8]) -> Option<String> {
+        read_selector(&mut BufReader::new(input), MAX_REQUEST)
+    }
+
+    #[test]
+    fn reads_a_selector_up_to_its_terminator() {
+        assert_eq!(read(b"/about\r\n").as_deref(), Some("/about"));
+        assert_eq!(read(b"/about\n").as_deref(), Some("/about"));
+        assert_eq!(read(b"\r\n").as_deref(), Some(""));
+    }
+
+    #[test]
+    fn a_tab_separates_a_search_term_that_is_not_part_of_the_path() {
+        assert_eq!(read(b"/about\tsearch words\r\n").as_deref(), Some("/about"));
+    }
+
+    #[test]
+    fn refuses_an_oversized_selector() {
+        let mut long = vec![b'a'; MAX_REQUEST + 1];
+        long.extend_from_slice(b"\r\n");
+        assert_eq!(read(&long), None);
+    }
+
+    #[test]
+    fn dot_stuffing_protects_a_leading_dot_on_every_line() {
+        // A line holding only a dot is the terminator, so one in the content has
+        // to be doubled or the item would end early.
+        assert_eq!(dot_stuff(b".\n"), b"..\n");
+        assert_eq!(dot_stuff(b"a\n.b\nc\n"), b"a\n..b\nc\n".to_vec());
+        // A dot that is not at the start of a line is ordinary text.
+        assert_eq!(dot_stuff(b"a.b\n"), b"a.b\n".to_vec());
+        assert_eq!(dot_stuff(b"plain\n"), b"plain\n".to_vec());
+    }
+
+    #[test]
+    fn dot_stuffing_handles_a_dot_at_the_very_start() {
+        assert_eq!(dot_stuff(b".hidden"), b"..hidden".to_vec());
+    }
+}

bin/src/proto/mod.rs

@@ -1,5 +1,6 @@
 //! Shared plumbing for the protocol listeners.
 
+pub mod gopher;
 pub mod http;
 pub mod negotiate;
 pub mod nex;

bin/src/serve.rs

@@ -31,6 +31,10 @@ pub struct Listener {
     /// The single site served by a protocol that carries no hostname.
     pub site: Option<String>,
     pub timeout_secs: u64,
+    /// Hostname to advertise in a Gopher menu, which names the host to fetch each
+    /// item from rather than assuming the client remembers.
+    pub advertised_host: String,
+    pub advertised_port: u16,
     max_connections: u32,
     open: AtomicU32,
 }
@@ -84,7 +88,11 @@ pub fn bind(
     for (name, spec) in &config.listener {
         let socket = TcpListener::bind(&spec.bind)
             .with_context(|| format!("listener '{name}' binding {}", spec.bind))?;
-        bound.push(Bound { listener: Arc::new(listener(name, spec, &registry, &sites)), socket });
+        // Built after the bind so it can carry the port it actually got, which a
+        // listener bound to port 0 only learns here.
+        let local = socket.local_addr().ok();
+        let listener = listener(name, spec, config, &registry, &sites, local);
+        bound.push(Bound { listener: Arc::new(listener), socket });
     }
     Ok(bound)
 }
@@ -120,9 +128,22 @@ pub fn run(config: &ServerConfig, registry: Arc<Registry>, sites: Arc<SiteSet>)
 fn listener(
     name: &str,
     spec: &ListenerSpec,
+    config: &ServerConfig,
     registry: &Arc<Registry>,
     sites: &Arc<SiteSet>,
+    local: Option<std::net::SocketAddr>,
 ) -> Listener {
+    // A Gopher menu has to name a host reachable from outside, so the site's own
+    // first hostname is better than whatever address the socket is bound to.
+    let advertised_host = spec
+        .site
+        .as_deref()
+        .or(spec.default_site.as_deref())
+        .and_then(|name| config.site.get(name))
+        .and_then(|site| site.hosts.first().cloned())
+        .or_else(|| local.map(|addr| addr.ip().to_string()))
+        .unwrap_or_else(|| "localhost".to_string());
+
     Listener {
         name: name.to_string(),
         protocol: spec.protocol,
@@ -132,6 +153,8 @@ fn listener(
         default_site: spec.default_site.clone(),
         site: spec.site.clone(),
         timeout_secs: spec.timeout_secs,
+        advertised_host,
+        advertised_port: local.map(|addr| addr.port()).unwrap_or(70),
         max_connections: spec.max_connections,
         open: AtomicU32::new(0),
     }
@@ -178,9 +201,10 @@ fn handle(listener: &Listener, stream: TcpStream) {
             Protocol::Http => proto::http::serve(listener, stream),
             Protocol::Spartan => proto::spartan::serve(listener, stream),
             Protocol::Nex => proto::nex::serve(listener, stream),
-            // Validation refuses these until their listeners exist.
+            Protocol::Gopher => proto::gopher::serve(listener, stream),
-            Protocol::Gemini | Protocol::Gopher => {
+            // Configuration validation refuses this, so it cannot be reached.
-                unreachable!("{} is rejected at startup", listener.protocol.as_str())
+            Protocol::Gemini => {
+                unreachable!("gemini is refused at startup until its listener exists")
             }
         }
     }));

bin/tests/listeners.rs

@@ -27,6 +27,8 @@ impl Server {
         write(dir.path(), "one/about.md", "# About\n\nAbout the first.\n");
         write(dir.path(), "one/img.png", "not really a png");
         write(dir.path(), "one/.itsybitsy.toml", "[defaults]\nmax_age = 120\n");
+        // A line starting with a dot, which Gopher has to double.
+        write(dir.path(), "one/dotted.md", "# Dotted\n\n    .hidden\n");
         write(dir.path(), "two/index.md", "# Two\n\nSecond site.\n");
         write(
             dir.path(),
@@ -69,6 +71,12 @@ protocol = "nex"
 bind = "127.0.0.1:0"
 formats = ["gemtext"]
 site = "two"
+
+[listener.gopher]
+protocol = "gopher"
+bind = "127.0.0.1:0"
+formats = ["gemtext"]
+site = "one"
 "#,
             one = dir.path().join("one").to_str().unwrap(),
             two = dir.path().join("two").to_str().unwrap(),
@@ -408,6 +416,95 @@ fn nothing_outside_the_root_is_reachable_over_any_protocol() {
     assert_eq!(server.send("nex", b"/.itsybitsy.toml\r\n"), "Not found\n");
 }
 
+// -- Gopher --------------------------------------------------------------
+//
+// Ported from smolweb's tests/test_gopher.py, where the exact wire bytes are the
+// assertion: Gopher has no status line, so framing is all a client has.
+
+#[test]
+fn an_empty_selector_gets_a_one_item_menu() {
+    // RFC 4266: a prefix-less `gopher://host/` means item type 1, so a client
+    // asking for the root expects a menu rather than a text file.
+    let server = Server::start();
+    let port = server.addr("gopher").port();
+    let response = server.send("gopher", b"\r\n");
+    assert_eq!(response, format!("0Home\t/\tone.test\t{port}\r\n.\r\n"));
+}
+
+#[test]
+fn the_root_selector_serves_the_document_as_text() {
+    let server = Server::start();
+    let response = server.send("gopher", b"/\r\n");
+    assert!(response.starts_with("# One"), "{response:?}");
+    assert!(response.ends_with(".\r\n"), "{response:?}");
+}
+
+#[test]
+fn a_text_item_ends_with_the_lone_dot_terminator() {
+    let server = Server::start();
+    let response = server.send("gopher", b"/about\r\n");
+    assert!(response.contains("About the first."), "{response:?}");
+    assert!(response.ends_with("\n.\r\n"), "{response:?}");
+}
+
+#[test]
+fn a_missing_selector_is_a_well_formed_text_item() {
+    // No status to report with, so the error is the item's content. smolweb says
+    // "Not found." here and "Not found" over HTTP and Nex; one wording is used
+    // across every protocol instead.
+    let server = Server::start();
+    assert_eq!(server.send("gopher", b"/nope\r\n"), "Not found\n.\r\n");
+}
+
+#[test]
+fn a_selector_without_a_leading_slash_still_resolves() {
+    // Clients built from a menu line send the selector verbatim, and a menu may
+    // carry it either way.
+    let server = Server::start();
+    assert!(server.send("gopher", b"about\r\n").contains("About the first."));
+}
+
+#[test]
+fn a_tab_separated_search_term_is_not_part_of_the_selector() {
+    let server = Server::start();
+    let response = server.send("gopher", b"/about\tignored words\r\n");
+    assert!(response.contains("About the first."), "{response:?}");
+}
+
+#[test]
+fn gopher_resolves_a_redirect_server_side() {
+    // There is no redirect item type to bounce with.
+    let server = Server::start();
+    let response = server.send("gopher", b"/about.md\r\n");
+    assert!(response.contains("About the first."), "{response:?}");
+}
+
+#[test]
+fn a_binary_item_is_sent_raw_with_no_terminator() {
+    // Dot-stuffing it would corrupt it, and a terminator would become part of
+    // the file.
+    let server = Server::start();
+    let response = server.send("gopher", b"/img.png\r\n");
+    assert_eq!(response, "not really a png");
+}
+
+#[test]
+fn a_leading_dot_in_the_content_is_doubled() {
+    let server = Server::start();
+    let response = server.send("gopher", b"/dotted\r\n");
+    // The source line is `.hidden`; undoubled it would end the item early.
+    assert!(response.contains("\n..hidden"), "{response:?}");
+    assert!(response.ends_with(".\r\n"), "{response:?}");
+}
+
+#[test]
+fn an_oversized_selector_is_refused() {
+    let server = Server::start();
+    let mut request = vec![b'a'; 600];
+    request.extend_from_slice(b"\r\n");
+    assert_eq!(server.send("gopher", &request), "Bad request\n.\r\n");
+}
+
 // -- Sub-documents -------------------------------------------------------
 
 /// A page that paginates addresses sub-documents under its own URL. Only a

core/src/config.rs

@@ -125,6 +125,15 @@ impl Protocol {
     pub fn negotiates(self) -> bool {
         matches!(self, Protocol::Http)
     }
+
+    /// Whether a listener exists for this protocol.
+    ///
+    /// Gemini needs a TLS stack and per-host certificate selection, which is its
+    /// own piece of work; naming it in a configuration is refused here rather
+    /// than panicking on the first connection.
+    pub fn is_implemented(self) -> bool {
+        !matches!(self, Protocol::Gemini)
+    }
 }
 
 /// What validation established, and what `--check` prints.
@@ -254,6 +263,12 @@ impl ServerConfig {
     fn validate_listeners(&self, available: &[&str]) -> Result<BTreeSet<String>, Error> {
         let mut formats = BTreeSet::new();
         for (name, spec) in &self.listener {
+            if !spec.protocol.is_implemented() {
+                return Err(Error::config(format!(
+                    "listener '{name}': {} is not implemented yet",
+                    spec.protocol.as_str()
+                )));
+            }
             if spec.formats.is_empty() {
                 return Err(Error::config(format!("listener '{name}': formats must not be empty")));
             }

itsybitsy.toml

@@ -21,3 +21,9 @@ protocol = "nex"
 bind = "127.0.0.1:1900"
 formats = ["text"]
 site = "demo"
+
+[listener.gopher]
+protocol = "gopher"
+bind = "127.0.0.1:7070"
+formats = ["text"]
+site = "demo"