diff --git a/README.md b/README.md index 9c11f53..1973886 100644 --- a/README.md +++ b/README.md @@ -2,13 +2,13 @@ [![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE) [![AI-DECLARATION: copilot](https://img.shields.io/badge/%E4%B7%BC%20AI--DECLARATION-copilot-fee2e2?labelColor=fee2e2)](https://ai-declaration.md) -Serve folders of Markdown to different domains over HTTP, [Spartan](https://portal.mozz.us/gemini/spartan.mozz.us/) and [Nex](https://nightfall.city/nex/info/specification.txt), rendered on the fly with no build step. +Serve folders of Markdown to different domains over HTTP, [Spartan](https://portal.mozz.us/gemini/spartan.mozz.us/), [Nex](https://nightfall.city/nex/info/specification.txt) and [Gopher](https://www.rfc-editor.org/rfc/rfc1436), rendered on request with no build step. A Rust re-implementation of [smolweb](https://code.randogoth.com/randogoth/smolweb), which serves one folder from one process with its output formats hardcoded and its configuration in per-file frontmatter. itsybitsy changes all three: one process serves many domains, each output format is a separate crate behind a cargo feature, and configuration lives in a TOML file per content directory. ## Status -It serves. HTTP, Spartan and Nex are working across five output formats, with virtual hosting on all three. Gemini and Gopher are the remaining protocols. +It serves. HTTP, Spartan, Nex and Gopher are working across five output formats, with virtual hosting wherever the protocol carries a hostname. Gemini is the one protocol left. | Area | State | | --- | --- | @@ -20,7 +20,8 @@ It serves. HTTP, Spartan and Nex are working across five output formats, with vi | Fixed-width text output (Nex, later Gopher) | implemented | | FIGlet banners and hyphenation, behind features | implemented | | WML decks and card sub-documents, behind a feature | implemented | -| Gemini and Gopher | planned | +| Gopher, with dot-stuffing and the RFC 4266 root menu | implemented | +| Gemini | planned | ## Configuration @@ -142,6 +143,21 @@ cargo build --release --features "figlet hyphenation" A banner that will not fit the line, names a font this build lacks, or is asked for by a build without `figlet` falls back to the level's underline — a heading that cannot be decorated should not be lost. An unknown font name is logged, since that is almost always a typo. Likewise `hyphenate` has no effect without the feature, and an unknown `hyphen_lang` leaves the text unhyphenated; a ragged right edge is the plain-text convention anyway. +## Protocols + +| Protocol | Routes by | Notes | +| --- | --- | --- | +| HTTP | `Host` header | `GET` and `HEAD`, negotiated by `Accept` | +| Spartan | the host field of its request line | One format per listener | +| Nex | nothing; its listener names one site | No status line at all | +| Gopher | nothing; its listener names one site | Item type 0 and the root menu | + +A protocol with no hostname in its requests cannot be routed by one, so its listener names a single site outright and configuration refuses to leave that implicit once more than one site exists. HTTP and Spartan fall back to `default_site` when a request names a host this server does not know. + +Neither Nex nor Gopher has a redirect status, so a canonical target is resolved server-side and the real content comes back on the first request rather than a bounce. + +Gopher serves item type 0 (text) and one menu: a prefix-less `gopher://host/` means item type 1 by RFC 4266, so an empty selector gets a one-item menu pointing at `/` rather than the root document, which would be the wrong type. Text items are dot-stuffed and terminated with a lone dot; binary items are sent raw, since dot-stuffing would corrupt them and a terminator would become part of the file. There are no generated directory listings and no type 7 search. + ## Card sub-documents WML is the one output that paginates: a WAP 1.x handset has a hard per-card byte budget and refuses a deck that exceeds it. So a long page becomes a chain of screens, and a page with dividers becomes a menu card linking to the rest. diff --git a/bin/Cargo.toml b/bin/Cargo.toml index 82e4052..16e0fe2 100644 --- a/bin/Cargo.toml +++ b/bin/Cargo.toml @@ -12,10 +12,9 @@ name = "itsybitsy" path = "src/main.rs" [features] -default = ["http", "spartan", "nex", "gemtext", "wap", "text"] -http = [] -spartan = [] -nex = [] +# Every protocol listener is always compiled in: they are a few hundred lines of +# line parsing between them, with no dependency of their own to gate. +default = ["gemtext", "wap", "text"] gemtext = ["dep:itsybitsy-gemtext"] wap = ["dep:itsybitsy-wap"] # WML 1.3 decks for WAP 1.x handsets. diff --git a/bin/src/proto/gopher.rs b/bin/src/proto/gopher.rs new file mode 100644 index 0000000..e2ba940 --- /dev/null +++ b/bin/src/proto/gopher.rs @@ -0,0 +1,165 @@ +//! Gopher, RFC 1436: a selector in, a text file or a menu out. +//! +//! Only item type 0 (text) and the one menu described below. There are no +//! gophermap directory listings and no type 7 search, because neither has a +//! source in a folder of Markdown. +//! +//! Like Nex there is no status line and no redirect, so an error is a plain body +//! and a canonical target is resolved server-side rather than bounced. + +use std::io::{BufRead, BufReader, Read, Write}; +use std::net::TcpStream; + +use anyhow::Result; +use itsybitsy_core::site::{Resolution, Resource}; + +use crate::proto::for_log; +use crate::serve::Listener; + +/// Gopher selectors are short; the cap is smolweb's. +const MAX_REQUEST: usize = 512; + +pub fn serve(listener: &Listener, mut stream: TcpStream) -> Result<()> { + let selector = { + let mut reader = BufReader::new(stream.try_clone()?); + read_selector(&mut reader, MAX_REQUEST) + }; + let Some(selector) = selector else { + log::warn!("{}: unreadable or oversized selector", listener.name); + return text(&mut stream, b"Bad request\n"); + }; + + // Gopher carries no hostname, so the listener names its site outright. + let site = listener.only_site(); + log::info!("{} gopher {}", listener.name, for_log(&selector)); + + // A prefix-less `gopher://host/` means item type 1 by RFC 4266, so a client + // asking for the root expects a menu rather than a text file. Serving the + // root document here would be the wrong type; a one-item menu pointing at + // `/` gets the client to the same place with the type it asked for. + if selector.is_empty() { + return menu(&mut stream, listener, &[("0", "Home", "/")]); + } + + match site.resolve_flat(&selector) { + Ok(Resolution::Found(Resource::Document { page, .. })) => { + match page.body(&listener.formats[0]) { + Some(body) => text(&mut stream, body), + None => text(&mut stream, b"Internal error\n"), + } + } + Ok(Resolution::Found(Resource::Raw { path, .. })) => { + // A binary item is sent raw and ended by closing: dot-stuffing it + // would corrupt it, and a terminator would be part of the file. + let mut file = std::fs::File::open(&path)?; + std::io::copy(&mut file, &mut stream)?; + Ok(()) + } + // resolve_flat follows redirects and sub-documents to real content. + Ok(_) => text(&mut stream, b"Not found\n"), + Err(err) => { + log::warn!("{} gopher {}: {err}", listener.name, for_log(&selector)); + text(&mut stream, b"Internal error\n") + } + } +} + +/// Read a selector: everything up to the first tab or line ending. +/// +/// A tab separates a selector from a search term, which this server has no use +/// for but must not read as part of the path. +fn read_selector(reader: &mut BufReader, cap: usize) -> Option { + let mut line = Vec::new(); + let mut limited = reader.take(cap as u64 + 1); + limited.read_until(b'\n', &mut line).ok()?; + if line.len() > cap { + return None; + } + let end = + line.iter().position(|byte| matches!(byte, b'\t' | b'\r' | b'\n')).unwrap_or(line.len()); + String::from_utf8(line[..end].to_vec()).ok() +} + +/// Send a text item: dot-stuffed, then the lone-dot terminator. +fn text(stream: &mut TcpStream, body: &[u8]) -> Result<()> { + stream.write_all(&dot_stuff(body))?; + // A line holding only a dot ends the item, which is why one in the content + // has to be doubled above. + stream.write_all(b".\r\n")?; + Ok(()) +} + +/// Send a menu: tab-separated items, then the terminator. +fn menu(stream: &mut TcpStream, listener: &Listener, items: &[(&str, &str, &str)]) -> Result<()> { + let mut out = Vec::new(); + for (kind, display, selector) in items { + out.extend_from_slice( + format!( + "{kind}{display}\t{selector}\t{}\t{}\r\n", + listener.advertised_host, listener.advertised_port + ) + .as_bytes(), + ); + } + stream.write_all(&out)?; + stream.write_all(b".\r\n")?; + Ok(()) +} + +/// Double a leading dot on any line, so no content line can end the item. +fn dot_stuff(body: &[u8]) -> Vec { + let mut out = Vec::with_capacity(body.len()); + let mut at_line_start = true; + for byte in body { + if at_line_start && *byte == b'.' { + out.push(b'.'); + } + out.push(*byte); + at_line_start = *byte == b'\n'; + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + fn read(input: &[u8]) -> Option { + read_selector(&mut BufReader::new(input), MAX_REQUEST) + } + + #[test] + fn reads_a_selector_up_to_its_terminator() { + assert_eq!(read(b"/about\r\n").as_deref(), Some("/about")); + assert_eq!(read(b"/about\n").as_deref(), Some("/about")); + assert_eq!(read(b"\r\n").as_deref(), Some("")); + } + + #[test] + fn a_tab_separates_a_search_term_that_is_not_part_of_the_path() { + assert_eq!(read(b"/about\tsearch words\r\n").as_deref(), Some("/about")); + } + + #[test] + fn refuses_an_oversized_selector() { + let mut long = vec![b'a'; MAX_REQUEST + 1]; + long.extend_from_slice(b"\r\n"); + assert_eq!(read(&long), None); + } + + #[test] + fn dot_stuffing_protects_a_leading_dot_on_every_line() { + // A line holding only a dot is the terminator, so one in the content has + // to be doubled or the item would end early. + assert_eq!(dot_stuff(b".\n"), b"..\n"); + assert_eq!(dot_stuff(b"a\n.b\nc\n"), b"a\n..b\nc\n".to_vec()); + // A dot that is not at the start of a line is ordinary text. + assert_eq!(dot_stuff(b"a.b\n"), b"a.b\n".to_vec()); + assert_eq!(dot_stuff(b"plain\n"), b"plain\n".to_vec()); + } + + #[test] + fn dot_stuffing_handles_a_dot_at_the_very_start() { + assert_eq!(dot_stuff(b".hidden"), b"..hidden".to_vec()); + } +} diff --git a/bin/src/proto/mod.rs b/bin/src/proto/mod.rs index db2a67c..ba73f40 100644 --- a/bin/src/proto/mod.rs +++ b/bin/src/proto/mod.rs @@ -1,5 +1,6 @@ //! Shared plumbing for the protocol listeners. +pub mod gopher; pub mod http; pub mod negotiate; pub mod nex; diff --git a/bin/src/serve.rs b/bin/src/serve.rs index 2196add..ecf9edd 100644 --- a/bin/src/serve.rs +++ b/bin/src/serve.rs @@ -31,6 +31,10 @@ pub struct Listener { /// The single site served by a protocol that carries no hostname. pub site: Option, pub timeout_secs: u64, + /// Hostname to advertise in a Gopher menu, which names the host to fetch each + /// item from rather than assuming the client remembers. + pub advertised_host: String, + pub advertised_port: u16, max_connections: u32, open: AtomicU32, } @@ -84,7 +88,11 @@ pub fn bind( for (name, spec) in &config.listener { let socket = TcpListener::bind(&spec.bind) .with_context(|| format!("listener '{name}' binding {}", spec.bind))?; - bound.push(Bound { listener: Arc::new(listener(name, spec, ®istry, &sites)), socket }); + // Built after the bind so it can carry the port it actually got, which a + // listener bound to port 0 only learns here. + let local = socket.local_addr().ok(); + let listener = listener(name, spec, config, ®istry, &sites, local); + bound.push(Bound { listener: Arc::new(listener), socket }); } Ok(bound) } @@ -120,9 +128,22 @@ pub fn run(config: &ServerConfig, registry: Arc, sites: Arc) fn listener( name: &str, spec: &ListenerSpec, + config: &ServerConfig, registry: &Arc, sites: &Arc, + local: Option, ) -> Listener { + // A Gopher menu has to name a host reachable from outside, so the site's own + // first hostname is better than whatever address the socket is bound to. + let advertised_host = spec + .site + .as_deref() + .or(spec.default_site.as_deref()) + .and_then(|name| config.site.get(name)) + .and_then(|site| site.hosts.first().cloned()) + .or_else(|| local.map(|addr| addr.ip().to_string())) + .unwrap_or_else(|| "localhost".to_string()); + Listener { name: name.to_string(), protocol: spec.protocol, @@ -132,6 +153,8 @@ fn listener( default_site: spec.default_site.clone(), site: spec.site.clone(), timeout_secs: spec.timeout_secs, + advertised_host, + advertised_port: local.map(|addr| addr.port()).unwrap_or(70), max_connections: spec.max_connections, open: AtomicU32::new(0), } @@ -178,9 +201,10 @@ fn handle(listener: &Listener, stream: TcpStream) { Protocol::Http => proto::http::serve(listener, stream), Protocol::Spartan => proto::spartan::serve(listener, stream), Protocol::Nex => proto::nex::serve(listener, stream), - // Validation refuses these until their listeners exist. - Protocol::Gemini | Protocol::Gopher => { - unreachable!("{} is rejected at startup", listener.protocol.as_str()) + Protocol::Gopher => proto::gopher::serve(listener, stream), + // Configuration validation refuses this, so it cannot be reached. + Protocol::Gemini => { + unreachable!("gemini is refused at startup until its listener exists") } } })); diff --git a/bin/tests/listeners.rs b/bin/tests/listeners.rs index 6ae0e69..4c6c675 100644 --- a/bin/tests/listeners.rs +++ b/bin/tests/listeners.rs @@ -27,6 +27,8 @@ impl Server { write(dir.path(), "one/about.md", "# About\n\nAbout the first.\n"); write(dir.path(), "one/img.png", "not really a png"); write(dir.path(), "one/.itsybitsy.toml", "[defaults]\nmax_age = 120\n"); + // A line starting with a dot, which Gopher has to double. + write(dir.path(), "one/dotted.md", "# Dotted\n\n .hidden\n"); write(dir.path(), "two/index.md", "# Two\n\nSecond site.\n"); write( dir.path(), @@ -69,6 +71,12 @@ protocol = "nex" bind = "127.0.0.1:0" formats = ["gemtext"] site = "two" + +[listener.gopher] +protocol = "gopher" +bind = "127.0.0.1:0" +formats = ["gemtext"] +site = "one" "#, one = dir.path().join("one").to_str().unwrap(), two = dir.path().join("two").to_str().unwrap(), @@ -408,6 +416,95 @@ fn nothing_outside_the_root_is_reachable_over_any_protocol() { assert_eq!(server.send("nex", b"/.itsybitsy.toml\r\n"), "Not found\n"); } +// -- Gopher -------------------------------------------------------------- +// +// Ported from smolweb's tests/test_gopher.py, where the exact wire bytes are the +// assertion: Gopher has no status line, so framing is all a client has. + +#[test] +fn an_empty_selector_gets_a_one_item_menu() { + // RFC 4266: a prefix-less `gopher://host/` means item type 1, so a client + // asking for the root expects a menu rather than a text file. + let server = Server::start(); + let port = server.addr("gopher").port(); + let response = server.send("gopher", b"\r\n"); + assert_eq!(response, format!("0Home\t/\tone.test\t{port}\r\n.\r\n")); +} + +#[test] +fn the_root_selector_serves_the_document_as_text() { + let server = Server::start(); + let response = server.send("gopher", b"/\r\n"); + assert!(response.starts_with("# One"), "{response:?}"); + assert!(response.ends_with(".\r\n"), "{response:?}"); +} + +#[test] +fn a_text_item_ends_with_the_lone_dot_terminator() { + let server = Server::start(); + let response = server.send("gopher", b"/about\r\n"); + assert!(response.contains("About the first."), "{response:?}"); + assert!(response.ends_with("\n.\r\n"), "{response:?}"); +} + +#[test] +fn a_missing_selector_is_a_well_formed_text_item() { + // No status to report with, so the error is the item's content. smolweb says + // "Not found." here and "Not found" over HTTP and Nex; one wording is used + // across every protocol instead. + let server = Server::start(); + assert_eq!(server.send("gopher", b"/nope\r\n"), "Not found\n.\r\n"); +} + +#[test] +fn a_selector_without_a_leading_slash_still_resolves() { + // Clients built from a menu line send the selector verbatim, and a menu may + // carry it either way. + let server = Server::start(); + assert!(server.send("gopher", b"about\r\n").contains("About the first.")); +} + +#[test] +fn a_tab_separated_search_term_is_not_part_of_the_selector() { + let server = Server::start(); + let response = server.send("gopher", b"/about\tignored words\r\n"); + assert!(response.contains("About the first."), "{response:?}"); +} + +#[test] +fn gopher_resolves_a_redirect_server_side() { + // There is no redirect item type to bounce with. + let server = Server::start(); + let response = server.send("gopher", b"/about.md\r\n"); + assert!(response.contains("About the first."), "{response:?}"); +} + +#[test] +fn a_binary_item_is_sent_raw_with_no_terminator() { + // Dot-stuffing it would corrupt it, and a terminator would become part of + // the file. + let server = Server::start(); + let response = server.send("gopher", b"/img.png\r\n"); + assert_eq!(response, "not really a png"); +} + +#[test] +fn a_leading_dot_in_the_content_is_doubled() { + let server = Server::start(); + let response = server.send("gopher", b"/dotted\r\n"); + // The source line is `.hidden`; undoubled it would end the item early. + assert!(response.contains("\n..hidden"), "{response:?}"); + assert!(response.ends_with(".\r\n"), "{response:?}"); +} + +#[test] +fn an_oversized_selector_is_refused() { + let server = Server::start(); + let mut request = vec![b'a'; 600]; + request.extend_from_slice(b"\r\n"); + assert_eq!(server.send("gopher", &request), "Bad request\n.\r\n"); +} + // -- Sub-documents ------------------------------------------------------- /// A page that paginates addresses sub-documents under its own URL. Only a diff --git a/core/src/config.rs b/core/src/config.rs index bc9bfcb..b65d1af 100644 --- a/core/src/config.rs +++ b/core/src/config.rs @@ -125,6 +125,15 @@ impl Protocol { pub fn negotiates(self) -> bool { matches!(self, Protocol::Http) } + + /// Whether a listener exists for this protocol. + /// + /// Gemini needs a TLS stack and per-host certificate selection, which is its + /// own piece of work; naming it in a configuration is refused here rather + /// than panicking on the first connection. + pub fn is_implemented(self) -> bool { + !matches!(self, Protocol::Gemini) + } } /// What validation established, and what `--check` prints. @@ -254,6 +263,12 @@ impl ServerConfig { fn validate_listeners(&self, available: &[&str]) -> Result, Error> { let mut formats = BTreeSet::new(); for (name, spec) in &self.listener { + if !spec.protocol.is_implemented() { + return Err(Error::config(format!( + "listener '{name}': {} is not implemented yet", + spec.protocol.as_str() + ))); + } if spec.formats.is_empty() { return Err(Error::config(format!("listener '{name}': formats must not be empty"))); } diff --git a/itsybitsy.toml b/itsybitsy.toml index 7072731..7f954a2 100644 --- a/itsybitsy.toml +++ b/itsybitsy.toml @@ -21,3 +21,9 @@ protocol = "nex" bind = "127.0.0.1:1900" formats = ["text"] site = "demo" + +[listener.gopher] +protocol = "gopher" +bind = "127.0.0.1:7070" +formats = ["text"] +site = "demo"