feat: port pneumatube to Rust on fumi-core

README.md

@@ -1,10 +1,12 @@
 # pneumatube
 
-A gateway bridging a smolmail account and an SMTP mailbox: smolmail messages carrying an `Email-To` frontmatter field are relayed to that address over SMTP, and mail sent to `<username>@<email domain>` is relayed as a smolmail message to `<username>` if that user is registered on the configured smolmail server. See `src/pneumatube/__main__.py` for the full environment variable list, `docker-compose.yml`/`docker/postfix/` for a Postfix-fronted container deployment, and `flake.nix` for a NixOS deployment with no containers.
+A gateway bridging a smolmail account and an SMTP mailbox: smolmail messages carrying an `Email-To` frontmatter field are relayed to that address over SMTP, and mail sent to `<username>@<email domain>` is relayed as a smolmail message to `<username>` if that user is registered on the configured smolmail server. See `src/main.rs` and `src/config.rs` for the full environment variable list, `docker-compose.yml`/`docker/postfix/` for a Postfix-fronted container deployment, and `flake.nix` for a NixOS deployment with no containers.
+
+The implementation is Rust on [fumi-core](../fumi) — the smolmail client library that also powers the `fumi` CLI — reached as a Cargo path dependency on a sibling checkout (`../fumi/core`, the same convention the nix flake and container build spell out). The email side is self-contained: a hand-rolled LMTP listener, a minimal outbound SMTP client (STARTTLS and AUTH LOGIN included, rustls underneath), and a `mail-parser` based body extractor. The gateway's own smolmail state (`gateway.db`) is a fumi store, so every fumi CLI command works against it for bootstrap and manual review.
 
 ## Using the gateway as a registered smolmail user
 
-This assumes a smolmail account already exists, and that the gateway's smolmail address (e.g. `gateway@smol.example.org`) and the email domain it answers to (e.g. `example.com`) are known — the operator can provide both. Everything below uses the reference client, `smolmail.py`, the same way it's used for anything else.
+This assumes a smolmail account already exists, and that the gateway's smolmail address (e.g. `gateway@smol.example.org`) and the email domain it answers to (e.g. `example.com`) are known — the operator can provide both. Everything below uses `fumi`, the same way it's used for anything else.
 
 ### Sending email through the gateway
 
@@ -12,7 +14,7 @@ Send the gateway a message with an `Email-To` frontmatter field naming where it
 
 ```
 echo "Hi there, this is coming in from smolmail." | \
-  smolmail.py send gateway@smol.example.org --subject "Hello" --header "Email-To: someone@example.com"
+  fumi send gateway@smol.example.org --subject "Hello" --header "Email-To: someone@example.com"
 ```
 
 A few things that aren't obvious from the outside:
@@ -26,8 +28,8 @@ A few things that aren't obvious from the outside:
 Mail sent to `<username>@example.com` reaches that registered user as a smolmail message, but by default only once the user has accepted the gateway:
 
 ```
-smolmail.py accept gateway@smol.example.org
-echo "hi" | smolmail.py send gateway@smol.example.org
+fumi accept gateway@smol.example.org
+echo "hi" | fumi send gateway@smol.example.org
 ```
 
 Both steps matter, not just the first. Accepting the gateway only changes the user's own server's tiering for mail from it; the gateway has no way to know that happened until it sees it reflected in a message from the user (same §5.8 mechanism as above), which means sending it at least one message. Until both are done, inbound email addressed to that user is rejected outright — not queued, not quietly demoted to some low-priority folder, just refused before it's even accepted.
@@ -40,32 +42,26 @@ Once that's done, the smolmail messages received this way won't say who original
 
 ### Unit tests
 
-`uv run pytest` (or `devbox run test`) — 27/27 passing. Covers address validation and header-injection rejection (`smtp_client`), frontmatter sanitization (`email_to_smol`), MIME body extraction, the accepted-recipient gate (`email_to_smol`), per-recipient LMTP reply assembly and the accepted-recipient RCPT reject (`smtp_server`), the Postfix `tcp_table(5)` lookup protocol including the same gate (`lookup_server`), and failure isolation plus the accepted-sender tier gate in the smolmail → email poll loop (`smol_to_email`).
+`cargo test` (or `devbox run test`) — 32 passing, offline (stubs plus fumi's in-memory store). Covers address validation including the CR/LF/NUL/unicode-whitespace injection battery and the CR/LF header-value backstop (`smtp_client`), frontmatter sanitization, MIME body extraction including the malformed-MIME battery (empty, binary, 50-deep nesting, declared-huge-missing-body) (`email_to_smol`), the accepted-recipient token gate including default-port normalization against a real fumi store (`email_to_smol`), per-recipient LMTP reply assembly, the accepted-recipient RCPT reject, command parsing and dot-unstuffing (`smtp_server`), the Postfix `tcp_table(5)` lookup protocol including the same gate and `%XX` quoting round-trips (`lookup_server`), and failure isolation plus the accepted-sender tier selection in the smolmail → email poll loop (`smol_to_email`).
 
 ### Stress test (`scripts/stress_test.py`, `devbox run stress`)
 
-Concurrent load against pneumatube directly, and through the real Postfix container rig.
+Concurrent load against pneumatube directly, and through the real Postfix container rig (`devbox run stress-postfix`). The throwaway server is `bunshin`; the load-driving sender and recipient accounts use the reference `smolmail.py` client in-process, which doubles as an interop check of the Python reference client against the Rust gateway and Rust server. Needs `bunshin` and `fumi` binaries on PATH and a `smolmail` checkout next to this repo.
 
-- **Healthy at low-to-moderate concurrency**: ~5 concurrent senders sustains ~140 msg/s with zero errors on both directions.
-- **Found and fixed**: `EmailToSmolBridge.user_exists` resolved a recipient's identity at RCPT time and discarded the result, so `forward()` resolved the same username again moments later — doubling the network round trips (each a full TCP + Noise handshake, smolmail.py has no connection pooling) for every message. Fixed by caching the resolved identity as a contact, the same trust-on-first-use pattern smolmail.py's own CLI already uses.
-- **A real scaling ceiling, not a pneumatube bug**: at concurrency ≈30, throughput collapses to mass client-side timeouts. The likely cause (via code inspection, not a clean isolated reproduction): smolmail's reference server (`smolmaild.py`) uses Python's `socketserver.ThreadingTCPServer` with its default backlog of 5, uncorrected, compounding with pneumatube's own single-worker-thread serialization (`worker.py`, by design, for sqlite thread-safety). Worth pinning down further before relying on sustained throughput much above a handful of messages/sec against the reference server.
-- Both targets pass cleanly at healthy concurrency: correctness verified (every expected message delivered exactly once, no loss or duplication) on the direct path and through the full Postfix → LMTP → pneumatube path.
+- **Direct target, default concurrency 20 (200 messages per phase)**: both phases correct (133 ok / 67 rejected exactly as constructed; 200/200 relayed to the outbound sink, no loss or duplication). Email → smolmail sustains ~88 msg/s — each accepted recipient paying a live connect + Noise handshake + RESOLVE at RCPT time — and smolmail → email SENDs land at ~335 msg/s.
+- **No serialization point left**: the Python implementation serialized every smolmail operation onto one worker thread (sqlite thread-affinity of the vendored client). fumi's store is `Send + Sync` (mutex + WAL) and every dial is its own Noise session, so the Rust port's concurrency is capped only by the smolmail server on the other end. The old concurrency ≈30 collapse against the reference `smolmaild.py` (ThreadingTCPServer backlog of 5) does not reproduce against `bunshin`: concurrency 30 completes both phases with zero errors and full drain.
+- The poll loop relays one message per SMTP conversation, sequentially by design; the harness's drain deadline is sized for that (~150 ms per relay against a local sink).
 
 ### Security test (`scripts/security_test.py`, `devbox run security`)
 
-11 checks: 8 PASS (defense verified), 3 RISK (confirmed, accepted design tradeoffs — not bugs, but worth knowing they're real), 0 BUG.
+Live checks against a throwaway rig (same fixtures as the stress test): 3 PASS (defense verified), 3 RISK (confirmed, accepted design tradeoffs — not bugs, but worth knowing they're real), 0 BUG. The unit-style checks the Python version ran in this script live in `cargo test` now (see Unit tests above).
 
 **PASS:**
-- Address validation (`smtp_client.validate`) rejects every CR/LF/NUL injection variant tried, not just the literal `\r\n` case.
-- The frontmatter sanitizer (`email_to_smol._single_line`) strips CR/LF from every payload tried.
-- Python's `EmailMessage` rejects embedded CR/LF in header values — confirmed as a real backstop, since `smtp_client.send()` has no explicit sanitizer of its own and otherwise relies entirely on smolmail's frontmatter format structurally preventing multi-line values.
-- `extract_text` survives a battery of malformed MIME (empty messages, non-UTF8 bytes, deeply nested multipart, multipart with no text part) without raising or stalling.
-- **Found and fixed**: the smolmail → email poll loop had no per-message failure isolation — unlike the email → smolmail side (`smtp_server.py`, already correct), one message raising an exception `_relay()`'s own except clause didn't cover would propagate out of `poll_once()` entirely. Since the poisoned message is only ever deleted on success, it would sort first on every subsequent poll and jam the entire direction forever. Not confirmed reachable via the documented frontmatter format today, but fixed regardless as defense-in-depth with a regression test proving later messages still get processed after an earlier one fails.
-- **Found and fixed**: the smolmail → email direction relayed *any* sender by default, including first-contact senders with no accept token — confirmed live (15/15 first-contact sends reached the outbound sink). Fixed with `PNEUMATUBE_REQUIRE_ACCEPTED_SENDER` (default `1`): `poll_once()` now polls only the main/accepted tier (SPEC.md §5.8), so a sender the gateway account hasn't explicitly accepted is left alone. Re-verified live: 0/15 first-contact sends reached the sink with the new default. Note the gateway never replies to smolmail senders, so accepting someone is a two-step process — `smolmail.py accept <address>` alone doesn't retroactively unblock a message already sitting in the local inbox, and the sender's *next* message won't carry a valid token either until the operator also sends them one message back (that's how the token actually reaches them, per the protocol).
-- **Found and fixed**: `Email-From` (the email → smolmail direction's frontmatter field carrying the original email's `MAIL FROM`) has been removed entirely. `MAIL FROM` is never authenticated, so forwarding it into a field that looks like gateway-attested metadata let any email sender forge an apparently-trustworthy "from" inside a signed smolmail message. Re-verified live with a forged `security-team@trusted-bank.example`: the relayed message now carries no field reflecting it at all.
-- **Found and fixed**: the email → smolmail direction relayed to *any* registered recipient by default, with no throttling and no way for a recipient to opt out. Fixed with `PNEUMATUBE_REQUIRE_ACCEPTED_RECIPIENT` (default `1`): reuses the real smolmail accept-token mechanism (SPEC.md §5.8) rather than inventing a new one — pneumatube already needs to know locally whether it holds a token for a recipient to attach a valid accept MAC on `cmd_send`, and that same fact is exactly "has this recipient accepted the gateway". A recipient who hasn't is now rejected live at RCPT TO (an all-or-nothing gate, not a demotion: without it, the message would still land in the recipient's own requests tier; with it, they get nothing at all). Checked in both `handle_RCPT` and the Postfix lookup service, so Postfix itself rejects live rather than generating a bounce later. Re-verified live: a registered-but-never-accepted recipient is now rejected with `outcome=rejected`, where it previously always succeeded.
+- The smolmail → email direction relays no first-contact sender by default: 0/15 never-before-seen senders with `Email-To` fields reached the outbound sink with `PNEUMATUBE_REQUIRE_ACCEPTED_SENDER` at its default.
+- The email → smolmail direction rejects a registered-but-never-accepted recipient live at RCPT TO (outcome=rejected, not queued-then-bounced) with `PNEUMATUBE_REQUIRE_ACCEPTED_RECIPIENT` at its default.
+- A forged `MAIL FROM` (`security-team@trusted-bank.example`) leaves no trace in the relayed message's frontmatter — the email → smolmail direction carries no `Email-From` field at all, since `MAIL FROM` is never authenticated.
 
 **RISK (confirmed, accepted tradeoffs):**
 - The Postfix lookup service unambiguously reveals which usernames exist, with no auth or rate limiting of its own — correct and necessary for `check_recipient_access` to work, but only safe because it's loopback-only. This requirement is load-bearing, not cosmetic: never expose `PNEUMATUBE_LOOKUP_HOST`/`PNEUMATUBE_LOOKUP_PORT` beyond Postfix.
-- **Open relay, smolmail → email, opt-in**: setting `PNEUMATUBE_REQUIRE_ACCEPTED_SENDER=0` deliberately restores the open-bridge behavior above (confirmed live: 15/15 first-contact sends relayed) — an accepted tradeoff for an operator who wants a public bridge, not a default.
-- **Open relay, email → smolmail, opt-in**: setting `PNEUMATUBE_REQUIRE_ACCEPTED_RECIPIENT=0` deliberately restores the "relay to anyone" behavior above (confirmed live: 15/15 distinct-sender messages relayed to an unaccepted recipient) — same tradeoff, operator's explicit choice. Note there is still no *per-sender* throttling in either mode: the gate is per-recipient (have they accepted the gateway at all), not per-sender, so an accepted recipient remains reachable by any external sender once opted in.
+- **Open relay, smolmail → email, opt-in**: setting `PNEUMATUBE_REQUIRE_ACCEPTED_SENDER=0` deliberately restores the open-bridge behavior (confirmed live: 15/15 first-contact sends relayed) — an accepted tradeoff for an operator who wants a public bridge, not a default.
+- **Open relay, email → smolmail, opt-in**: setting `PNEUMATUBE_REQUIRE_ACCEPTED_RECIPIENT=0` deliberately restores the "relay to anyone" behavior (confirmed live: 15/15 distinct-sender messages relayed to an unaccepted recipient) — same tradeoff, operator's explicit choice. Note there is still no *per-sender* throttling in either mode: the gate is per-recipient (have they accepted the gateway at all), not per-sender, so an accepted recipient remains reachable by any external sender once opted in.