diff --git a/CLAUDE.md b/CLAUDE.md index 8a25e60..5f13865 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -67,8 +67,32 @@ hand-editable artefact. Anything that writes it must preserve comments, ordering and omitted defaults — `toml::to_string` on the `Serialize` derive does none of those things. +**WebCal and Google are calcalist's own; CalDAV is being taken back too.** A +feed is one iCalendar document fetched over HTTP and read-only by protocol, so +routing it through a generated config, a subprocess and a status database bought +nothing — it cost three minutes a cycle in observed stalls. CalDAV has followed, +behind a per-endpoint `native` flag so the two run side by side until the +native client has earned the default. The argument +is not that pimsync is bad but that calcalist was paying for generality it never +used: one server, against 528 lines of adapter, a 59.6 MB flake closure, an +undocumented `discover` format that is scraped, a version range to maintain, and +a `vstorage` bug that unlinked `order` relative to the working directory. + +**Radicale is the CalDAV target; other servers are untested, not unsupported.** +Narrowing is what makes owning the protocol reasonable: the integration tests +already run against Radicale, so every assumption is checked on every +`devbox run check`, and when the RFC is ambiguous its source settles it. The +discipline that keeps this honest is that server-specific accommodations stay +*named*, never absorbed into the general path — Radicale answers an initial +`sync-collection` with a token and no members, which is written as "always load +in full first, then go incremental", correct anywhere, rather than as a quirk +worked around in place. `doctor` reports what a server advertises so that +pointing calcalist elsewhere fails legibly. + ## Known gap -A failing `pimsync sync` stops the whole CalDAV leg. Unlike the Google side this -cannot be narrowed: pimsync is one process covering every pair, and a failure -does not say which pair it belongs to. +A failing `pimsync sync` stops every CalDAV endpoint pimsync still owns. It is +one process covering every pair, and a failure does not say which pair it +belongs to, so the endpoints cannot be told apart the way the Google ones are. +`native = true` takes an endpoint out of its reach entirely, which is the way +out rather than a narrowing of pimsync's own reporting. diff --git a/Cargo.lock b/Cargo.lock index 2621f6f..b6e140d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -117,6 +117,7 @@ dependencies = [ "blake3", "clap", "jiff", + "roxmltree", "rustix", "serde", "serde_json", @@ -749,6 +750,15 @@ dependencies = [ "windows-sys 0.52.0", ] +[[package]] +name = "roxmltree" +version = "0.21.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1964b10c76125c36f8afe190065a4bf9a87bf324842c05701330bba9f1cacbb" +dependencies = [ + "memchr", +] + [[package]] name = "rustix" version = "1.1.4" @@ -1081,9 +1091,9 @@ dependencies = [ [[package]] name = "ureq-proto" -version = "0.6.2" +version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fabc3e92916c89c95b20eef7b06b00b066bc217ef9ea3a4ac9bf1a7e35261e10" +checksum = "5b0809a01d1ca5a51ca70db32bb2a19157582a526505ef3c19e3b343a59aa5ad" dependencies = [ "base64", "http", diff --git a/Cargo.toml b/Cargo.toml index ef7a6d6..f45bfd3 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -11,6 +11,7 @@ publish = false blake3 = "1.8" clap = { version = "4.6", features = ["derive"] } jiff = "0.2" +roxmltree = { version = "0.21.1", default-features = false, features = ["std"] } rustix = { version = "1.1", features = ["fs"] } serde = { version = "1.0", features = ["derive"] } serde_json = "1.0" diff --git a/README.md b/README.md index 67f9206..b2a9ca1 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ Or straight from the repo, without cloning it first: nix profile install "git+https://code.randogoth.com/randogoth/CalCalist.git#calcalist" ``` -One runtime dependency is needed for CalDAV and iCal sync: **`pimsync`, from 0.5.11 up to but not including 0.7.0, on `PATH`**. The Nix flake installation method already ensures this: it builds pimsync from source and wraps `calcalist` so that copy is the one found, whatever else is installed. +One runtime dependency is needed for CalDAV endpoints that are not `native`, and for `calcalist discover`: **`pimsync`, from 0.5.11 up to but not including 0.7.0, on `PATH`**. Google calendars, iCal feeds and `native` CalDAV collections need nothing extra — calcalist syncs those itself, so a configuration whose CalDAV endpoints are all `native` never calls pimsync during a cycle. The Nix flake installation method already ensures this: it builds pimsync from source and wraps `calcalist` so that copy is the one found, whatever else is installed. To run a different pimsync — a fork, or a working tree — override the input at install time: @@ -64,10 +64,13 @@ type = "caldav" url = "https://posteo.de:8443/calendars/you/abc123/work/" username = "you@posteo.de" secret_command = "secret-tool lookup service posteo user you@posteo.de" +native = true # optional; sync this collection with calcalist's own client ``` `url` must name the calendar collection itself, not the server root. +`native = true` syncs the collection with calcalist's own CalDAV client instead of pimsync, so that endpoint needs no external binary and reports its own failures rather than taking the whole CalDAV leg down with it. It is per-endpoint so the two can run side by side, one collection each, while the native client earns the default. It is developed and tested against [Radicale](https://radicale.org/); other CalDAV servers are untested rather than unsupported, and `calcalist doctor` prints what each server advertises so an unfamiliar one fails legibly. + Rather than digging it out of a provider's web interface, ask the server — this prints a ready-to-paste block per calendar, with ids taken from the URL for you to rename: ```sh @@ -115,6 +118,7 @@ default_sink = "gcal" conflict = "source_wins" # the only policy; the origin calendar wins propagate_deletes = true # deleting a mirror deletes the original max_delete_fraction = 0.2 # abort if more of the aggregate than this vanishes +# skip_past = "2026-01-01" # leave finished, non-repeating events from before this out ``` The rules, all checked before anything is synchronised, with every problem reported in one pass: @@ -211,6 +215,7 @@ A category matching an endpoint id works too. Note that *any* category on a new ## What it does to your calendars +- **`skip_past` leaves history behind.** Give it a date — `skip_past = "2026-01-01"` — and an event is left out when it ended before that date *and* carries no `RRULE` or `RDATE`, so a series that began years ago still travels; its next occurrence is the point. An event whose dates cannot be read is kept rather than guessed at. This is what makes a first sync carry only what is still to come instead of publishing years of finished events into a calendar you subscribe to; on one real configuration it was 79% of them. A fixed date rather than a rolling "before now" is deliberate: a moving cutoff would retire events from the aggregate as they aged, so the published calendar would quietly delete yesterday's events on every cycle. `calcalist sync --skip-past [DATE]` applies one for a single cycle whatever the file says, defaulting to today when given bare — useful with `--dry-run` for seeing what a cutoff would leave out. Excluded events are never deleted from their source calendar; only the aggregate's copy is affected. Applying a cutoff to an aggregate that already carries history retires those mirrors, which can trip the mass-deletion guard and need `--force` the first time. - **Deleting an event in the aggregate deletes it at its source**, unless `propagate_deletes = false`. A cycle that would remove more than `max_delete_fraction` of the aggregate is refused until you pass `--force` — with a floor of three, so removing one or two events is never blocked. - **Edited in both places since the last sync, the source wins.** The conflict is reported. - **Alarms are always kept**, with no option to strip them. The tool exists so you subscribe to one calendar rather than several, so dropping reminders would disarm every one you have. @@ -233,7 +238,7 @@ systemctl --user daemon-reload systemctl --user enable --now calcalist.timer ``` -The service expects the binary at `~/.local/bin/calcalist`; edit `ExecStart` if yours is elsewhere. Only one cycle runs at a time — a timer firing while you are running `sync` by hand is refused, naming the process that holds the lock. A cycle that could not reach an endpoint exits non-zero on purpose, so a lapsed token shows up as a failed unit rather than passing unnoticed; `journalctl --user -u calcalist` has the report. If your secrets come from a +The service expects the binary at `~/.nix-profile/bin/calcalist`, where `nix profile install` puts it; edit `ExecStart` if yours is elsewhere — a `cargo install` lands in `~/.cargo/bin`. The path is absolute because a user manager's `PATH` is not the login shell's. Only one cycle runs at a time — a timer firing while you are running `sync` by hand is refused, naming the process that holds the lock. A cycle that could not reach an endpoint exits non-zero on purpose, so a lapsed token shows up as a failed unit rather than passing unnoticed; `journalctl --user -u calcalist` has the report. If your secrets come from a keyring needing an unlocked session, the timer only works while you are logged in. ## Commands @@ -259,7 +264,7 @@ Everything machine-local lives under `$XDG_STATE_HOME/calcalist`: ``` state.json event mappings, content hashes, sync cursors lock held for the duration of a cycle -pimsync.conf generated on every sync; edits are overwritten +pimsync.conf generated when pimsync is used; edits are overwritten pimsync-status/ pimsync's own record of what it has seen vdir// the local mirror of each endpoint google/accounts/.json refresh tokens, 0600 @@ -270,8 +275,9 @@ configuration between machines; leave this behind. ## Known gaps -- **A failing `pimsync sync` stops the whole CalDAV leg.** Google endpoints are isolated from each other, but pimsync is a single process covering every - CalDAV and feed pair, and a failure does not say which pair it belongs to. +- **A failing `pimsync sync` stops every CalDAV endpoint that is not `native`.** Google, feed and `native` CalDAV endpoints are isolated from each other + and from this, but pimsync is a single process covering every pair it still owns, and a failure does not say which pair it belongs to. Setting + `native = true` on those endpoints removes them from its reach, and with it the gap. ## License diff --git a/src/caldav/client.rs b/src/caldav/client.rs new file mode 100644 index 0000000..4142d66 --- /dev/null +++ b/src/caldav/client.rs @@ -0,0 +1,689 @@ +//! The CalDAV protocol: requests, and the multistatus XML that comes back. +//! +//! Only the parts a calendar mirror needs are here — list, fetch, store, remove, +//! and the incremental cursor of RFC 6578. Discovery of *which* collections a +//! server holds is still pimsync's (`calcalist discover`); this speaks to one +//! collection whose URL the configuration already names. + +use std::fmt::{self, Write as _}; + +use roxmltree::{Document, Node}; +use thiserror::Error; + +/// Namespaces of the two vocabularies in play. Elements are matched on these +/// rather than on a prefix, which servers choose for themselves. +const DAV: &str = "DAV:"; +const CALDAV: &str = "urn:ietf:params:xml:ns:caldav"; + +#[derive(Debug, Error)] +pub enum CaldavError { + #[error("endpoint `{endpoint}` has url `{url}`, which is not an absolute http(s) address")] + BadUrl { endpoint: String, url: String }, + #[error("could not reach {url}: {source}")] + Transport { + url: String, + #[source] + source: Box, + }, + #[error("{method} {url} answered {status}")] + Status { + method: String, + url: String, + status: u16, + }, + #[error("{url} returned XML that could not be read: {source}")] + Malformed { + url: String, + #[source] + source: roxmltree::Error, + }, + #[error( + "{href} was changed on the server since it was last read; the next cycle will pick that \ + change up and reconcile it rather than overwrite it" + )] + Conflict { href: String }, + #[error(transparent)] + Secret(#[from] crate::secret::SecretError), + #[error(transparent)] + Vdir(#[from] crate::vdir::VdirError), +} + +/// One resource in a collection, as the server describes it. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Resource { + pub href: String, + pub etag: String, +} + +/// A resource's body, alongside the etag it was read at. +#[derive(Debug, Clone)] +pub struct Fetched { + pub href: String, + pub etag: String, + pub ics: String, +} + +/// What a collection has done since a cursor was issued. +#[derive(Debug, Clone)] +pub struct Changes { + pub changed: Vec, + pub removed: Vec, + pub token: String, +} + +/// What a server says it can do, for `doctor` to report. +#[derive(Debug, Clone, Default)] +pub struct Capabilities { + /// The `DAV:` header's compliance classes. + pub classes: Vec, + /// Whether `sync-collection` is advertised. Without it every pull reads the + /// collection in full, which works but transfers everything each cycle. + pub sync_collection: bool, + pub calendar_multiget: bool, +} + +pub struct Client { + agent: ureq::Agent, + /// The collection's URL, always with a trailing slash. + url: String, + /// Scheme and authority, for resolving the path-only hrefs servers return. + origin: String, + authorization: String, +} + +/// Written by hand rather than derived: the credential header would otherwise +/// land in any error or log line that formatted a client. +impl fmt::Debug for Client { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("Client") + .field("url", &self.url) + .field("authorization", &"") + .finish_non_exhaustive() + } +} + +impl Client { + pub fn new( + endpoint: &str, + url: &str, + username: &str, + password: &str, + ) -> Result { + let bad = || CaldavError::BadUrl { + endpoint: endpoint.to_string(), + url: url.to_string(), + }; + let rest = url + .strip_prefix("https://") + .or_else(|| url.strip_prefix("http://")) + .ok_or_else(bad)?; + let scheme = if url.starts_with("https://") { + "https://" + } else { + "http://" + }; + let authority = rest + .split('/') + .next() + .filter(|a| !a.is_empty()) + .ok_or_else(bad)?; + let mut collection = url.to_string(); + if !collection.ends_with('/') { + collection.push('/'); + } + Ok(Client { + agent: crate::http::agent(), + url: collection, + origin: format!("{scheme}{authority}"), + authorization: format!("Basic {}", base64(&format!("{username}:{password}"))), + }) + } + + /// The compliance classes and reports the server advertises. + pub fn capabilities(&self) -> Result { + let response = self.send("OPTIONS", &self.url, None, &[])?; + let classes = response + .headers() + .get("dav") + .and_then(|value| value.to_str().ok()) + .map(|value| value.split(',').map(|c| c.trim().to_string()).collect()) + .unwrap_or_default(); + + let body = self.propfind(&self.url, "0", "")?; + let text = body.as_str(); + Ok(Capabilities { + classes, + sync_collection: text.contains("sync-collection"), + calendar_multiget: text.contains("calendar-multiget"), + }) + } + + /// Every resource in the collection, with its etag. + /// + /// This is the full read, used for the first pull of a collection and + /// whenever the server refuses the stored cursor. + pub fn list(&self) -> Result, CaldavError> { + let body = self.propfind(&self.url, "1", "")?; + let document = self.parse(&body)?; + let collection = self.path_of(&self.url); + Ok(responses(&document) + .into_iter() + // The collection answers its own PROPFIND; only its members are + // resources, and only those carry an etag. + .filter(|response| self.path_of(&response.href) != collection) + .filter_map(|response| { + Some(Resource { + href: response.href, + etag: response.etag?, + }) + }) + .collect()) + } + + /// The collection's current cursor. + pub fn sync_token(&self) -> Result, CaldavError> { + let body = self.propfind(&self.url, "0", "")?; + let document = self.parse(&body)?; + Ok(responses(&document) + .into_iter() + .find_map(|response| response.sync_token) + .or_else(|| child_text(document.root_element(), DAV, "sync-token").map(str::to_string))) + } + + /// What has changed since `token`, or `None` when the server rejects it. + /// + /// A rejected cursor is not an error: it means the server can no longer say + /// what happened since, and the caller falls back to a full read. + pub fn changes(&self, token: &str) -> Result, CaldavError> { + let request = format!( + "\ + \ + {}1\ + ", + escape(token) + ); + let response = self.send("REPORT", &self.url, Some(request), &[("Depth", "1")])?; + let status = response.status().as_u16(); + // 403 with DAV:valid-sync-token, and 409, are both how servers say the + // cursor has aged out. + if status == 403 || status == 409 { + return Ok(None); + } + let body = self.body(response, "REPORT", &self.url)?; + let document = self.parse(&body)?; + let Some(token) = child_text(document.root_element(), DAV, "sync-token") else { + return Ok(None); + }; + + let mut changed = Vec::new(); + let mut removed = Vec::new(); + for response in responses(&document) { + if response.gone { + removed.push(response.href); + } else if let Some(etag) = response.etag { + changed.push(Resource { + href: response.href, + etag, + }); + } + } + Ok(Some(Changes { + changed, + removed, + token: token.to_string(), + })) + } + + /// The bodies of the named resources, in one request. + pub fn multiget(&self, hrefs: &[String]) -> Result, CaldavError> { + if hrefs.is_empty() { + return Ok(Vec::new()); + } + let mut request = String::from( + "\ + \ + ", + ); + for href in hrefs { + let _ = write!(request, "{}", escape(href)); + } + request.push_str(""); + + let response = self.send("REPORT", &self.url, Some(request), &[("Depth", "1")])?; + let body = self.body(response, "REPORT", &self.url)?; + let document = self.parse(&body)?; + Ok(responses(&document) + .into_iter() + .filter_map(|response| { + Some(Fetched { + href: response.href, + etag: response.etag.unwrap_or_default(), + ics: response.data?, + }) + }) + .collect()) + } + + /// Stores a calendar, returning the etag it now has. + /// + /// `if_match` is the etag the caller believes is there: `Some` conditions + /// the write on the resource being unchanged, `None` on it not existing at + /// all. Either way an unexpected server-side change is refused rather than + /// overwritten. + pub fn put( + &self, + href: &str, + ics: &str, + if_match: Option<&str>, + ) -> Result { + let url = self.resolve(href); + let condition = match if_match { + Some(etag) => ("If-Match", etag), + None => ("If-None-Match", "*"), + }; + let response = self.send( + "PUT", + &url, + Some(ics.to_string()), + &[("Content-Type", "text/calendar; charset=utf-8"), condition], + )?; + let status = response.status().as_u16(); + if status == 412 { + return Err(CaldavError::Conflict { + href: href.to_string(), + }); + } + if !(200..300).contains(&status) { + return Err(CaldavError::Status { + method: "PUT".to_string(), + url, + status, + }); + } + match response + .headers() + .get("etag") + .and_then(|value| value.to_str().ok()) + { + Some(etag) => Ok(etag.to_string()), + // Returning one is a SHOULD, not a MUST. Asking costs a round trip + // only on servers that leave it out. + None => self.etag_for(href), + } + } + + /// Removes a resource, provided it still carries `etag`. + pub fn delete(&self, href: &str, etag: &str) -> Result<(), CaldavError> { + let url = self.resolve(href); + let condition: &[(&str, &str)] = if etag.is_empty() { + &[] + } else { + &[("If-Match", etag)] + }; + let response = self.send("DELETE", &url, None, condition)?; + let status = response.status().as_u16(); + if status == 412 { + return Err(CaldavError::Conflict { + href: href.to_string(), + }); + } + // Already gone is the outcome that was wanted. + if status == 404 || (200..300).contains(&status) { + return Ok(()); + } + Err(CaldavError::Status { + method: "DELETE".to_string(), + url, + status, + }) + } + + /// The href a new resource for `uid` should take. + pub fn href_for(&self, uid: &str) -> String { + format!("{}{}", self.path_of(&self.url), crate::vdir::file_name(uid)) + } + + fn etag_for(&self, href: &str) -> Result { + let url = self.resolve(href); + let body = self.propfind(&url, "0", "")?; + let document = self.parse(&body)?; + Ok(responses(&document) + .into_iter() + .find_map(|response| response.etag) + .unwrap_or_default()) + } + + fn propfind(&self, url: &str, depth: &str, props: &str) -> Result { + let request = format!( + "\ + {props}" + ); + let response = self.send("PROPFIND", url, Some(request), &[("Depth", depth)])?; + self.body(response, "PROPFIND", url) + } + + fn send( + &self, + method: &str, + url: &str, + body: Option, + headers: &[(&str, &str)], + ) -> Result, CaldavError> { + let mut request = ureq::http::Request::builder() + .method(method) + .uri(url) + .header("Authorization", &self.authorization); + if body.is_some() { + request = request.header("Content-Type", "application/xml; charset=utf-8"); + } + for (name, value) in headers { + request = request.header(*name, *value); + } + let request = request + .body(body.unwrap_or_default()) + .map_err(|_| CaldavError::BadUrl { + endpoint: String::new(), + url: url.to_string(), + })?; + self.agent + .run(request) + .map_err(|source| CaldavError::Transport { + url: url.to_string(), + source: Box::new(source), + }) + } + + fn body( + &self, + mut response: ureq::http::Response, + method: &str, + url: &str, + ) -> Result { + let status = response.status().as_u16(); + if !(200..300).contains(&status) { + return Err(CaldavError::Status { + method: method.to_string(), + url: url.to_string(), + status, + }); + } + response + .body_mut() + .read_to_string() + .map_err(|source| CaldavError::Transport { + url: url.to_string(), + source: Box::new(source), + }) + } + + fn parse<'a>(&self, body: &'a str) -> Result, CaldavError> { + Document::parse(body).map_err(|source| CaldavError::Malformed { + url: self.url.clone(), + source, + }) + } + + /// An href as a full URL. Servers answer with paths, but the spec permits + /// absolute URLs, and some proxies rewrite them that way. + fn resolve(&self, href: &str) -> String { + if href.starts_with("http://") || href.starts_with("https://") { + return href.to_string(); + } + format!("{}{href}", self.origin) + } + + /// The path part of a URL, so that a path href and an absolute one compare. + fn path_of<'a>(&self, url: &'a str) -> &'a str { + url.strip_prefix(&self.origin).unwrap_or(url) + } +} + +/// One `` element, flattened to what a mirror cares about. +#[derive(Debug, Default)] +struct RawResponse { + href: String, + etag: Option, + data: Option, + sync_token: Option, + /// Set when the response itself reports the resource as gone, which is how + /// `sync-collection` renders a deletion. + gone: bool, +} + +fn responses(document: &Document) -> Vec { + document + .root_element() + .children() + .filter(|node| is(*node, DAV, "response")) + .filter_map(|node| { + let href = child_text(node, DAV, "href")?.trim().to_string(); + let mut response = RawResponse { + href, + ..RawResponse::default() + }; + if let Some(status) = child_text(node, DAV, "status") { + response.gone = status.contains("404"); + } + for propstat in node.children().filter(|n| is(*n, DAV, "propstat")) { + // Properties a server could not supply come back under their own + // non-200 propstat; reading those would invent empty etags. + let ok = child_text(propstat, DAV, "status").is_some_and(|s| s.contains(" 200")); + if !ok { + continue; + } + for prop in propstat.children().filter(|n| is(*n, DAV, "prop")) { + if let Some(etag) = child_text(prop, DAV, "getetag") { + response.etag = Some(etag.trim().to_string()); + } + if let Some(data) = child_text(prop, CALDAV, "calendar-data") { + response.data = Some(data.to_string()); + } + if let Some(token) = child_text(prop, DAV, "sync-token") { + response.sync_token = Some(token.trim().to_string()); + } + } + } + Some(response) + }) + .collect() +} + +fn is(node: Node, namespace: &str, name: &str) -> bool { + node.is_element() + && node.tag_name().namespace() == Some(namespace) + && node.tag_name().name() == name +} + +fn child_text<'a>(node: Node<'a, 'a>, namespace: &str, name: &str) -> Option<&'a str> { + node.children() + .find(|child| is(*child, namespace, name)) + .and_then(|child| child.text()) +} + +/// Escapes text for an XML element body. +fn escape(value: &str) -> String { + let mut out = String::with_capacity(value.len()); + for ch in value.chars() { + match ch { + '&' => out.push_str("&"), + '<' => out.push_str("<"), + '>' => out.push_str(">"), + _ => out.push(ch), + } + } + out +} + +/// Base64 for the one credential header this module sends. A dependency for +/// twenty lines used in a single place would be the larger cost. +fn base64(value: &str) -> String { + const ALPHABET: &[u8] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + let bytes = value.as_bytes(); + let mut out = String::with_capacity(bytes.len().div_ceil(3) * 4); + for chunk in bytes.chunks(3) { + let packed = chunk.iter().enumerate().fold(0u32, |acc, (index, byte)| { + acc | (u32::from(*byte) << (16 - 8 * index)) + }); + for slot in 0..4 { + if slot <= chunk.len() { + let index = (packed >> (18 - 6 * slot)) & 0b11_1111; + out.push(ALPHABET[index as usize] as char); + } else { + out.push('='); + } + } + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + fn client() -> Client { + Client::new("test", "https://dav.example.com/me/work", "me", "secret") + .expect("the url should be accepted") + } + + #[test] + fn base64_matches_the_rfc_examples() { + assert_eq!(base64("f"), "Zg=="); + assert_eq!(base64("fo"), "Zm8="); + assert_eq!(base64("foo"), "Zm9v"); + assert_eq!(base64("foob"), "Zm9vYg=="); + assert_eq!(base64("fooba"), "Zm9vYmE="); + assert_eq!(base64("foobar"), "Zm9vYmFy"); + } + + #[test] + fn a_collection_url_gains_its_trailing_slash() { + assert_eq!(client().url, "https://dav.example.com/me/work/"); + } + + #[test] + fn a_url_with_no_scheme_is_refused() { + let error = Client::new("test", "dav.example.com/me/", "me", "secret") + .expect_err("a scheme is required"); + assert!(matches!(error, CaldavError::BadUrl { .. }), "{error:?}"); + } + + /// Servers answer with paths, but an absolute href is legal and must not be + /// turned into `https://hostpath`. + #[test] + fn hrefs_resolve_whether_or_not_they_carry_a_host() { + let client = client(); + assert_eq!( + client.resolve("/me/work/a.ics"), + "https://dav.example.com/me/work/a.ics" + ); + assert_eq!( + client.resolve("https://dav.example.com/me/work/a.ics"), + "https://dav.example.com/me/work/a.ics" + ); + } + + const LISTING: &str = r#" + + + /me/work/ + HTTP/1.1 404 Not Found + + + /me/work/a.ics + "aaa"HTTP/1.1 200 OK + +"#; + + /// The collection answers its own `PROPFIND`; counting it as a member would + /// have the mirror try to fetch the calendar as though it were an event. + #[test] + fn listing_skips_the_collection_itself() { + let client = client(); + let document = Document::parse(LISTING).expect("parse"); + let members: Vec<_> = responses(&document) + .into_iter() + .filter(|r| client.path_of(&r.href) != client.path_of(&client.url)) + .collect(); + assert_eq!(members.len(), 1); + assert_eq!(members[0].href, "/me/work/a.ics"); + assert_eq!(members[0].etag.as_deref(), Some("\"aaa\"")); + } + + /// A property the server could not supply comes back under a 404 propstat. + /// Reading it regardless would record an empty etag and condition the next + /// write on nothing. + #[test] + fn a_property_the_server_refused_is_not_read_as_a_value() { + let document = Document::parse(LISTING).expect("parse"); + let collection = &responses(&document)[0]; + assert_eq!(collection.etag, None); + } + + const SYNC: &str = r#" + + + /me/work/new.ics + "bbb"HTTP/1.1 200 OK + + + /me/work/gone.ics + HTTP/1.1 404 Not Found + + http://example.com/ns/sync/42 +"#; + + #[test] + fn a_sync_report_separates_changes_from_removals() { + let document = Document::parse(SYNC).expect("parse"); + let parsed = responses(&document); + let changed: Vec<_> = parsed.iter().filter(|r| !r.gone).collect(); + let removed: Vec<_> = parsed.iter().filter(|r| r.gone).collect(); + assert_eq!(changed.len(), 1); + assert_eq!(changed[0].etag.as_deref(), Some("\"bbb\"")); + assert_eq!(removed.len(), 1); + assert_eq!(removed[0].href, "/me/work/gone.ics"); + assert_eq!( + child_text(document.root_element(), DAV, "sync-token"), + Some("http://example.com/ns/sync/42") + ); + } + + /// Prefixes are the server's to choose, so elements are matched by namespace. + #[test] + fn calendar_data_is_found_whatever_prefix_the_server_picked() { + let body = r#" + + + /me/work/a.ics + "c" + BEGIN:VCALENDAR +END:VCALENDAR +HTTP/1.1 200 OK + +"#; + let document = Document::parse(body).expect("parse"); + let parsed = responses(&document); + assert_eq!(parsed.len(), 1); + assert!( + parsed[0] + .data + .as_deref() + .unwrap_or_default() + .contains("VCALENDAR"), + "{:?}", + parsed[0].data + ); + } + + #[test] + fn a_new_resource_is_named_after_its_uid() { + let href = client().href_for("event-1@example.com"); + assert!(href.starts_with("/me/work/"), "{href}"); + assert!(href.ends_with(".ics"), "{href}"); + } + + #[test] + fn xml_text_is_escaped() { + assert_eq!(escape("a&b"), "a&b<c>"); + } +} diff --git a/src/caldav/mod.rs b/src/caldav/mod.rs new file mode 100644 index 0000000..1a46ec9 --- /dev/null +++ b/src/caldav/mod.rs @@ -0,0 +1,240 @@ +//! The CalDAV leg. +//! +//! What pimsync does for a CalDAV collection, done here instead: keep one local +//! vdir and one remote collection in step, in both directions. It is the same +//! shape as the Google leg next door, and a good deal shorter, because CalDAV +//! already speaks iCalendar — there is no model to convert to and back, only +//! `.ics` bodies to move between an HTTP resource and a file. +//! +//! Radicale is the server this is written against and tested on. Accommodations +//! for it are named where they are made rather than folded into the general +//! path; there is one, in `pull`. + +pub mod client; + +use std::collections::BTreeMap; +use std::path::Path; + +pub use client::{CaldavError, Client}; + +use crate::ical::Calendar; +use crate::state::{CaldavItem, CaldavState}; +use crate::vdir; + +#[derive(Debug, Default)] +pub struct PullReport { + pub written: usize, + pub deleted: usize, + /// Set when the collection was read in full rather than incrementally. + pub full_load: bool, +} + +#[derive(Debug, Default)] +pub struct PushReport { + pub created: usize, + pub updated: usize, + pub deleted: usize, +} + +/// Brings an endpoint's local vdir into step with its collection. +pub fn pull( + client: &Client, + vdir_dir: &Path, + state: &mut CaldavState, +) -> Result { + let mut report = PullReport::default(); + + // Radicale answers an initial `sync-collection` with a token and no members + // at all, so a cursor is never taken on trust for the first read. Loading in + // full and only then asking for a cursor is correct against any server: the + // cursor is a statement about what has happened *since*, which is only + // meaningful once there is a complete picture to apply it to. + let (changed, removed) = match state.sync_token.as_deref() { + Some(token) => match client.changes(token)? { + Some(changes) => { + state.sync_token = Some(changes.token); + (changes.changed, changes.removed) + } + None => full_load(client, state, &mut report)?, + }, + None => full_load(client, state, &mut report)?, + }; + + let held = vdir::read(vdir_dir)?; + // Owned rather than borrowed: the loops below write back into the same + // state this indexes. + let by_href: BTreeMap = state + .events + .iter() + .map(|(uid, item)| (item.href.clone(), uid.clone())) + .collect(); + + // An etag that has not moved describes a body already on disk, so only the + // resources that actually changed are fetched. + let wanted: Vec = changed + .iter() + .filter(|resource| { + by_href + .get(&resource.href) + .and_then(|uid| state.events.get(uid)) + .is_none_or(|known| known.etag != resource.etag) + }) + .map(|resource| resource.href.clone()) + .collect(); + + for fetched in client.multiget(&wanted)? { + let Ok(calendar) = Calendar::parse(&fetched.ics) else { + // A resource that will not parse is left where it is: it is the + // server's, and refusing the whole cycle over one bad event would + // stop every other calendar too. + continue; + }; + let Some(uid) = calendar.uid().map(str::to_string) else { + continue; + }; + let replaces = held.get(&uid).map(|item| item.path.as_path()); + vdir::write(vdir_dir, &uid, &calendar, replaces)?; + state.events.insert( + uid, + CaldavItem { + href: fetched.href, + etag: fetched.etag, + hash: calendar.content_hash(), + }, + ); + report.written += 1; + } + + for href in removed { + let Some(uid) = by_href.get(&href) else { + continue; + }; + if let Some(item) = held.get(uid) { + vdir::remove(&item.path)?; + report.deleted += 1; + } + state.events.remove(uid); + } + + Ok(report) +} + +/// Reads the collection in full, returning it as though it were a set of +/// changes, and works out what has gone since the last cycle. +fn full_load( + client: &Client, + state: &mut CaldavState, + report: &mut PullReport, +) -> Result<(Vec, Vec), CaldavError> { + report.full_load = true; + let resources = client.list()?; + // The cursor is taken after the listing, never before: anything that + // happens in between is then described by the next cycle rather than + // falling into the gap between the two. + state.sync_token = client.sync_token()?; + + let present: Vec<&str> = resources.iter().map(|r| r.href.as_str()).collect(); + let removed: Vec = state + .events + .values() + .map(|item| item.href.clone()) + .filter(|href| !present.contains(&href.as_str())) + .collect(); + Ok((resources, removed)) +} + +/// Sends local changes in an endpoint's vdir up to its collection. +pub fn push( + client: &Client, + vdir_dir: &Path, + state: &mut CaldavState, +) -> Result { + let held = vdir::read(vdir_dir)?; + let mut report = PushReport::default(); + + for (uid, item) in &held { + let known = state.events.get(uid); + if known.is_some_and(|known| known.hash == item.hash) { + continue; + } + let ics = item.calendar.to_ics(); + let (href, etag) = match known { + Some(known) => { + report.updated += 1; + let etag = client.put(&known.href, &ics, Some(&known.etag))?; + (known.href.clone(), etag) + } + None => { + report.created += 1; + let href = client.href_for(uid); + let etag = client.put(&href, &ics, None)?; + (href, etag) + } + }; + state.events.insert( + uid.clone(), + CaldavItem { + href, + etag, + hash: item.hash.clone(), + }, + ); + } + + // Anything recorded as ours that is no longer held locally has been deleted. + let gone: Vec = state + .events + .keys() + .filter(|uid| !held.contains_key(*uid)) + .cloned() + .collect(); + for uid in gone { + if let Some(known) = state.events.remove(&uid) { + client.delete(&known.href, &known.etag)?; + report.deleted += 1; + } + } + Ok(report) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A cursor is only ever stored alongside a complete reading of the + /// collection, so a fresh endpoint cannot start incremental and silently + /// mirror nothing — which is exactly what Radicale's empty first answer + /// would cause. + #[test] + fn a_collection_with_no_cursor_is_read_in_full() { + let state = CaldavState::default(); + assert!( + state.sync_token.is_none(), + "a fresh endpoint must not claim a cursor" + ); + } + + /// Deletions are matched by href, which is what the server reports; the UID + /// lives inside a body that a deleted resource no longer has. + #[test] + fn state_can_be_looked_up_by_href() { + let mut state = CaldavState::default(); + state.events.insert( + "event-1@example.com".to_string(), + CaldavItem { + href: "/me/work/a.ics".to_string(), + etag: "\"aaa\"".to_string(), + hash: "hash".to_string(), + }, + ); + let by_href: BTreeMap = state + .events + .iter() + .map(|(uid, item)| (item.href.clone(), uid.clone())) + .collect(); + assert_eq!( + by_href.get("/me/work/a.ics").map(String::as_str), + Some("event-1@example.com") + ); + } +} diff --git a/src/cli.rs b/src/cli.rs index d072276..da699fa 100644 --- a/src/cli.rs +++ b/src/cli.rs @@ -29,6 +29,13 @@ pub enum Command { /// Proceed even when the mass-deletion guard trips #[arg(long)] force: bool, + /// Leave finished, non-repeating events from before DATE out of every + /// aggregate, for this cycle only. Defaults to today when given without + /// a date. Set `skip_past` on an aggregate to make it stick. Applying a + /// cutoff to an aggregate that already carries history retires those + /// mirrors, which may need `--force` the first time. + #[arg(long, value_name = "DATE", num_args = 0..=1, default_missing_value = "today")] + skip_past: Option, }, /// Show endpoints, aggregates and the last sync result Status, diff --git a/src/config.rs b/src/config.rs index 0617fc3..a88fd75 100644 --- a/src/config.rs +++ b/src/config.rs @@ -38,6 +38,14 @@ pub enum EndpointKind { url: String, username: String, secret_command: Option, + /// Sync this collection with calcalist's own CalDAV client rather than + /// with pimsync. + /// + /// Transitional, and per-endpoint on purpose: the two implementations + /// can then run against the same server at once, one collection each, + /// until the native one has earned the default. + #[serde(default)] + native: bool, }, Google { calendar_id: String, @@ -133,6 +141,28 @@ pub struct Aggregate { pub propagate_deletes: bool, #[serde(default = "default_max_delete_fraction")] pub max_delete_fraction: f64, + /// Date before which finished, non-repeating events are not mirrored, as + /// `YYYY-MM-DD`. + /// + /// A fixed date rather than a rolling "before now" on purpose: a moving + /// cutoff would retire events from the aggregate as they aged, so the + /// published calendar would quietly delete yesterday's events every cycle. + /// Pinned to a date, an event that is mirrored stays mirrored. + /// + /// Set per aggregate, because whether history is worth carrying is a + /// property of the published calendar. `--skip-past` applies one for a + /// single cycle; only this makes it stick. + #[serde(default)] + pub skip_past: Option, +} + +/// Midnight local time on `date`, which is how a cutoff written as `YYYY-MM-DD` +/// is read: the whole of that day is kept. +pub fn parse_cutoff(date: &str) -> Option { + let date: jiff::civil::Date = date.trim().parse().ok()?; + date.to_zoned(jiff::tz::TimeZone::system()) + .ok() + .map(|zoned| zoned.timestamp()) } #[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize, Serialize)] @@ -203,6 +233,10 @@ pub enum Problem { SinkNotWritable { aggregate: String, endpoint: String }, #[error("aggregate `{aggregate}` has max_delete_fraction {value}, which is outside 0.0..=1.0")] DeleteFractionOutOfRange { aggregate: String, value: f64 }, + #[error( + "aggregate `{aggregate}` has skip_past `{value}`, which is not a date of the form YYYY-MM-DD" + )] + BadSkipPast { aggregate: String, value: String }, #[error("endpoint `{endpoint}` has neither `url` nor `url_command`")] NoUrlSource { endpoint: String }, #[error("endpoint `{endpoint}` has both `url` and `url_command`; use one or the other")] @@ -305,6 +339,16 @@ impl Config { value: aggregate.max_delete_fraction, }); } + // Caught here rather than at the first cycle: a misspelled date would + // otherwise read as "no cutoff" and quietly publish the whole history. + if let Some(date) = &aggregate.skip_past + && parse_cutoff(date).is_none() + { + problems.push(Problem::BadSkipPast { + aggregate: aggregate.id.clone(), + value: date.clone(), + }); + } } fn check_target(&self, aggregate: &Aggregate, problems: &mut Vec) { @@ -426,6 +470,47 @@ default_sink = "work" assert_eq!(found, Vec::new()); } + /// A misspelled cutoff must be refused, not read as "no cutoff": the quiet + /// failure would publish the whole history the setting exists to leave out. + #[test] + fn a_skip_past_that_is_not_a_date_is_refused() { + let found = problems( + r#" +[[aggregate]] +id = "unified" +target = "gcal" +sources = ["work"] +skip_past = "last tuesday" +"#, + ); + assert_eq!( + found, + vec![Problem::BadSkipPast { + aggregate: "unified".into(), + value: "last tuesday".into(), + }] + ); + } + + #[test] + fn a_skip_past_date_is_accepted_and_read_as_midnight() { + let found = problems( + r#" +[[aggregate]] +id = "unified" +target = "gcal" +sources = ["work"] +skip_past = "2026-09-01" +"#, + ); + assert_eq!(found, Vec::new()); + assert!(parse_cutoff("2026-09-01").is_some()); + assert!( + parse_cutoff("2026-13-01").is_none(), + "month 13 is not a date" + ); + } + #[test] fn applies_documented_defaults() { let config = parse( diff --git a/src/doctor.rs b/src/doctor.rs index 2425ee2..ec7031b 100644 --- a/src/doctor.rs +++ b/src/doctor.rs @@ -4,10 +4,12 @@ use std::fmt; use std::fs; use std::path::Path; +use crate::caldav; use crate::config::{Config, ConfigError, EndpointKind}; use crate::google::auth::{self, AuthError}; use crate::paths; use crate::pimsync::{self, PimsyncError}; +use crate::secret; #[derive(Debug, PartialEq, Eq)] pub enum Outcome { @@ -57,11 +59,81 @@ pub fn run(config_path: Option<&Path>) -> Vec { if let Ok((config, _)) = &loaded { checks.push(check_generated_config(config)); checks.extend(check_google_authorisation(config)); + checks.extend(check_caldav_servers(config)); } checks.push(check_config(loaded)); checks } +/// Reports what each natively-synced CalDAV server says it can do. +/// +/// calcalist is written against Radicale, so this is where a server that is not +/// Radicale gets to say so before a cycle depends on it. A missing +/// `sync-collection` is not a failure — every pull simply reads the collection +/// in full — but it is worth knowing, because it turns a cheap cycle into an +/// expensive one. +fn check_caldav_servers(config: &Config) -> Vec { + config + .endpoints + .iter() + .filter(|endpoint| matches!(endpoint.kind, EndpointKind::Caldav { native: true, .. })) + .map(|endpoint| Check { + name: "caldav server", + outcome: caldav_outcome(endpoint), + }) + .collect() +} + +fn caldav_outcome(endpoint: &crate::config::Endpoint) -> Outcome { + let EndpointKind::Caldav { + url, + username, + secret_command, + .. + } = &endpoint.kind + else { + unreachable!("caller filtered on Caldav") + }; + let password = match secret_command { + Some(command) => match secret::run(command) { + Ok(password) => password, + Err(error) => return Outcome::Fail(format!("{}: {error}", endpoint.id)), + }, + None => String::new(), + }; + let client = match caldav::Client::new(&endpoint.id, url, username, &password) { + Ok(client) => client, + Err(error) => return Outcome::Fail(format!("{}: {error}", endpoint.id)), + }; + match client.capabilities() { + Ok(capabilities) => { + let classes = if capabilities.classes.is_empty() { + "no DAV header".to_string() + } else { + capabilities.classes.join(", ") + }; + let mut missing = Vec::new(); + if !capabilities.sync_collection { + missing.push("sync-collection"); + } + if !capabilities.calendar_multiget { + missing.push("calendar-multiget"); + } + if missing.is_empty() { + Outcome::Ok(format!("{}: {classes}", endpoint.id)) + } else { + Outcome::Warn(format!( + "{}: {classes}; does not advertise {} — calcalist is written against \ + Radicale, and will read the collection in full every cycle", + endpoint.id, + missing.join(" or ") + )) + } + } + Err(error) => Outcome::Fail(format!("{}: {error}", endpoint.id)), + } +} + /// Asks pimsync to validate the configuration calcalist generates for it. /// /// pimsync is pre-1.0 and its parser does not always match its documentation, so diff --git a/src/google/api.rs b/src/google/api.rs index 63073ff..8df37fa 100644 --- a/src/google/api.rs +++ b/src/google/api.rs @@ -62,7 +62,7 @@ pub struct Changes { impl Client { pub fn new(access_token: String, calendar_id: String) -> Self { - let agent = super::agent(); + let agent = crate::http::agent(); Client { agent, access_token, diff --git a/src/google/auth.rs b/src/google/auth.rs index 6a1a71f..c7f91e7 100644 --- a/src/google/auth.rs +++ b/src/google/auth.rs @@ -34,14 +34,8 @@ const EXPIRY_MARGIN: Duration = Duration::from_secs(60); #[derive(Debug, Error)] pub enum AuthError { - #[error("could not run the secret command `{command}`: {source}")] - SecretCommand { - command: String, - #[source] - source: std::io::Error, - }, - #[error("the secret command `{command}` failed: {message}")] - SecretFailed { command: String, message: String }, + #[error(transparent)] + Secret(#[from] crate::secret::SecretError), #[error("endpoint `{0}` has no client_secret_command, which Google's OAuth flow requires")] NoClientSecret(String), #[error("no endpoint named `{0}` in the configuration")] @@ -220,27 +214,9 @@ fn resolve_token( } } -/// Runs an external command and returns its first line of output. -/// -/// Secrets are fetched this way rather than stored, so the configuration file -/// stays portable and free of credentials. +/// Fetches a secret, reporting failure in this module's error type. pub fn run_secret_command(command: &str) -> Result { - let output = Command::new("sh") - .arg("-c") - .arg(command) - .output() - .map_err(|source| AuthError::SecretCommand { - command: command.to_string(), - source, - })?; - if !output.status.success() { - return Err(AuthError::SecretFailed { - command: command.to_string(), - message: String::from_utf8_lossy(&output.stderr).trim().to_string(), - }); - } - let text = String::from_utf8_lossy(&output.stdout); - Ok(text.lines().next().unwrap_or_default().trim().to_string()) + crate::secret::run(command).map_err(AuthError::Secret) } /// Everything needed to talk to Google for one endpoint. @@ -401,7 +377,7 @@ pub fn login( /// Returned as an option rather than an error: failing to name the account only /// costs the sharing, and the authorisation itself is perfectly good without it. fn primary_calendar_id(access_token: &str) -> Option { - let agent = super::agent(); + let agent = crate::http::agent(); let mut response = agent .get("https://www.googleapis.com/calendar/v3/calendars/primary") .header("Authorization", format!("Bearer {access_token}")) @@ -484,7 +460,7 @@ fn post_token(form: &[(&str, &str)]) -> Result { .collect::>() .join("&"); - let agent = super::agent(); + let agent = crate::http::agent(); let mut response = agent .post(TOKEN_ENDPOINT) .content_type("application/x-www-form-urlencoded") diff --git a/src/google/convert.rs b/src/google/convert.rs index 4543f32..af68327 100644 --- a/src/google/convert.rs +++ b/src/google/convert.rs @@ -796,7 +796,13 @@ pub fn find_instance<'a>(instances: &'a [Event], key: &InstanceKey) -> Option<&' } /// Parses `YYYYMMDDTHHMMSS`, with or without a trailing Z. -fn parse_basic(value: &str, field: &'static str) -> Result { +/// +/// Shared with `crate::horizon`, which needs the same reading of a `DTSTART`; +/// a second parser would be free to drift from this one. +pub(crate) fn parse_basic( + value: &str, + field: &'static str, +) -> Result { let trimmed = value.trim_end_matches('Z'); jiff::civil::DateTime::strptime("%Y%m%dT%H%M%S", trimmed).map_err(|source| { ConvertError::BadTime { @@ -808,7 +814,7 @@ fn parse_basic(value: &str, field: &'static str) -> Result String { +pub(crate) fn hyphenate(basic: &str) -> String { if basic.len() != 8 { return basic.to_string(); } diff --git a/src/google/mod.rs b/src/google/mod.rs index dadf189..53833cc 100644 --- a/src/google/mod.rs +++ b/src/google/mod.rs @@ -8,28 +8,3 @@ pub mod api; pub mod auth; pub mod convert; - -use std::time::Duration; - -/// How long one request may take, end to end. ureq defaults every timeout to -/// none, so a server that accepts a connection and then says nothing used to -/// stall a cycle for as long as the machine stayed up — indistinguishable, from -/// the terminal, from a sync that was merely slow. -const REQUEST_TIMEOUT: Duration = Duration::from_secs(60); - -/// Long enough for a handshake over a bad link, short enough that an -/// unreachable host is reported rather than waited on. -const CONNECT_TIMEOUT: Duration = Duration::from_secs(15); - -/// The HTTP client every Google call goes through. -/// -/// `http_status_as_error` is off because the callers read the status themselves: -/// a 410 drives a full resync, and a 404 is an ordinary answer here. -pub fn agent() -> ureq::Agent { - ureq::Agent::config_builder() - .http_status_as_error(false) - .timeout_global(Some(REQUEST_TIMEOUT)) - .timeout_connect(Some(CONNECT_TIMEOUT)) - .build() - .into() -} diff --git a/src/horizon.rs b/src/horizon.rs new file mode 100644 index 0000000..2f33956 --- /dev/null +++ b/src/horizon.rs @@ -0,0 +1,297 @@ +//! Telling a finished event from one still to come. +//! +//! A first sync otherwise publishes an entire history into a calendar somebody +//! subscribes to, which on a real configuration was 77% of the events and most +//! of an hour of uploading. `skip_past` exists to leave that behind. +//! +//! The rule is deliberately narrow: an event is dropped only when it has ended +//! *and* will never happen again. Anything that repeats stays whatever its start +//! date, because its next occurrence is what matters, and anything this module +//! cannot read stays too — dropping an event is not a decision to take on a +//! half-understood date. + +use jiff::{SignedDuration, Timestamp, civil, tz::TimeZone}; + +use crate::ical::{Calendar, Property}; + +/// Whether the event has ended and will not recur. +/// +/// `horizon` is normally now, taken once per cycle so that every aggregate in a +/// run is measured against the same instant. +pub fn is_finished(calendar: &Calendar, horizon: Timestamp) -> bool { + // A master and its overrides share one file, so a repeat rule anywhere in it + // speaks for the whole event. `RDATE` counts: a single-occurrence rule is + // still a rule, and an event with extra dates has not finished at its first. + if repeats(calendar) { + return false; + } + match latest_end(calendar) { + Some(end) => end < horizon, + // No readable DTSTART anywhere. Keep it. + None => false, + } +} + +fn repeats(calendar: &Calendar) -> bool { + ["RRULE", "RDATE"] + .iter() + .any(|name| calendar.properties("VEVENT", name).next().is_some()) +} + +/// The last instant this file accounts for, across every `VEVENT` in it. +/// +/// An override can fall after its master, so the latest wins rather than the +/// first. `None` means nothing in the file had a start that could be read, which +/// the caller treats as a reason to keep the event. +fn latest_end(calendar: &Calendar) -> Option { + calendar.events().iter().filter_map(event_end).max() +} + +/// When one `VEVENT` stops occupying the calendar. +fn event_end(event: &Calendar) -> Option { + if let Some(end) = event + .properties("VEVENT", "DTEND") + .next() + .and_then(|property| instant(&property)) + { + return Some(end); + } + let start = event.properties("VEVENT", "DTSTART").next()?; + let at = instant(&start)?; + if let Some(duration) = event + .properties("VEVENT", "DURATION") + .next() + .and_then(|property| parse_duration(property.value)) + { + return at.checked_add(duration).ok(); + } + // An all-day event covers its whole day, so it is not over until the day is. + // A timed event with no end or duration occupies only its start instant. + if is_date(&start) { + return at.checked_add(SignedDuration::from_hours(24)).ok(); + } + Some(at) +} + +/// An iCalendar date or date-time as an instant. +/// +/// The two conventions here match what the Google leg already assumes: a +/// `VALUE=DATE` parameter or an eight-character value means all-day, and a value +/// with no `TZID` is read as UTC. A floating time is therefore treated as UTC, +/// which can be wrong by at most a day's offset — immaterial against a horizon +/// that is only ever "has this finished". +fn instant(property: &Property<'_>) -> Option { + let value = property.value.trim(); + if is_date(property) { + let date: civil::Date = crate::google::convert::hyphenate(value).parse().ok()?; + return date.to_zoned(zone_of(property)).ok().map(|z| z.timestamp()); + } + let civil = crate::google::convert::parse_basic(value, "DTSTART").ok()?; + // A trailing Z is UTC whatever the TZID says; `parse_basic` strips it, so the + // zone is only consulted for values that carry no marker of their own. + let zone = if value.ends_with('Z') { + TimeZone::UTC + } else { + zone_of(property) + }; + civil.to_zoned(zone).ok().map(|zoned| zoned.timestamp()) +} + +fn is_date(property: &Property<'_>) -> bool { + property.param("VALUE") == Some("DATE") || property.value.trim().len() == 8 +} + +fn zone_of(property: &Property<'_>) -> TimeZone { + property + .param("TZID") + .and_then(|name| TimeZone::get(name).ok()) + .unwrap_or(TimeZone::UTC) +} + +/// An RFC 5545 duration, enough of it to bound an event. +/// +/// Weeks, days, hours, minutes and seconds; months and years cannot appear in a +/// `DURATION`. A value that will not parse yields `None`, and the caller then +/// falls back to the start, which can only ever keep an event longer. +fn parse_duration(value: &str) -> Option { + let value = value.trim(); + let (negative, rest) = match value.strip_prefix('-') { + Some(rest) => (true, rest), + None => (false, value.strip_prefix('+').unwrap_or(value)), + }; + let rest = rest.strip_prefix('P')?; + let (date_part, time_part) = match rest.split_once('T') { + Some((date, time)) => (date, Some(time)), + None => (rest, None), + }; + + let mut seconds: i64 = 0; + let mut digits = String::new(); + for ch in date_part.chars() { + match ch { + '0'..='9' => digits.push(ch), + 'W' => seconds += take(&mut digits)? * 7 * 24 * 3600, + 'D' => seconds += take(&mut digits)? * 24 * 3600, + _ => return None, + } + } + for ch in time_part.unwrap_or_default().chars() { + match ch { + '0'..='9' => digits.push(ch), + 'H' => seconds += take(&mut digits)? * 3600, + 'M' => seconds += take(&mut digits)? * 60, + 'S' => seconds += take(&mut digits)?, + _ => return None, + } + } + if !digits.is_empty() { + return None; + } + Some(SignedDuration::from_secs(if negative { + -seconds + } else { + seconds + })) +} + +fn take(digits: &mut String) -> Option { + let parsed = digits.parse().ok()?; + digits.clear(); + Some(parsed) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Every case is measured against this, so the tests say what they mean + /// rather than depending on when they run. + fn horizon() -> Timestamp { + "2026-09-13T12:00:00Z".parse().expect("horizon") + } + + fn calendar(body: &str) -> Calendar { + Calendar::parse(&format!( + "BEGIN:VCALENDAR\r\nVERSION:2.0\r\n{body}END:VCALENDAR\r\n" + )) + .expect("the fixture should parse") + } + + fn event(properties: &str) -> Calendar { + calendar(&format!( + "BEGIN:VEVENT\r\nUID:e@example.com\r\nDTSTAMP:20260101T000000Z\r\n{properties}END:VEVENT\r\n" + )) + } + + #[test] + fn a_timed_event_that_has_ended_is_finished() { + let past = event("DTSTART:20260910T090000Z\r\nDTEND:20260910T100000Z\r\n"); + assert!(is_finished(&past, horizon())); + } + + #[test] + fn a_timed_event_still_to_come_is_not() { + let future = event("DTSTART:20260920T090000Z\r\nDTEND:20260920T100000Z\r\n"); + assert!(!is_finished(&future, horizon())); + } + + /// An all-day event is not over at midnight; it owns the whole day. + #[test] + fn an_all_day_event_today_survives_a_midday_horizon() { + let today = event("DTSTART;VALUE=DATE:20260913\r\n"); + assert!(!is_finished(&today, horizon())); + + let yesterday = event("DTSTART;VALUE=DATE:20260912\r\n"); + assert!(is_finished(&yesterday, horizon())); + } + + /// A value of eight characters is a date even without the parameter, which is + /// how several feeds in the wild write it. + #[test] + fn an_eight_character_value_is_read_as_a_date() { + let today = event("DTSTART:20260913\r\n"); + assert!(!is_finished(&today, horizon())); + } + + #[test] + fn an_event_running_across_the_horizon_is_not_finished() { + let spanning = event("DTSTART:20260912T090000Z\r\nDTEND:20260914T100000Z\r\n"); + assert!(!is_finished(&spanning, horizon())); + } + + #[test] + fn a_duration_stands_in_for_a_missing_end() { + let spanning = event("DTSTART:20260913T080000Z\r\nDURATION:PT8H\r\n"); + assert!(!is_finished(&spanning, horizon())); + + let over = event("DTSTART:20260913T080000Z\r\nDURATION:PT1H\r\n"); + assert!(is_finished(&over, horizon())); + } + + #[test] + fn durations_in_days_and_weeks_are_understood() { + assert_eq!( + parse_duration("P1W"), + Some(SignedDuration::from_secs(604_800)) + ); + assert_eq!( + parse_duration("P2D"), + Some(SignedDuration::from_secs(172_800)) + ); + assert_eq!( + parse_duration("P1DT2H3M4S"), + Some(SignedDuration::from_secs(93_784)) + ); + assert_eq!( + parse_duration("-PT1H"), + Some(SignedDuration::from_secs(-3600)) + ); + assert_eq!(parse_duration("nonsense"), None); + } + + /// The whole point of the rule: a series that began years ago is still live. + #[test] + fn a_long_finished_series_is_kept_because_it_repeats() { + let series = + event("DTSTART:20200101T090000Z\r\nDTEND:20200101T100000Z\r\nRRULE:FREQ=WEEKLY\r\n"); + assert!(!is_finished(&series, horizon())); + } + + #[test] + fn an_rdate_alone_is_enough_to_keep_an_event() { + let extra = event("DTSTART:20200101T090000Z\r\nRDATE:20270101T090000Z\r\n"); + assert!(!is_finished(&extra, horizon())); + } + + /// The master is long past but one occurrence was moved into the future, and + /// it travels in the same file. + #[test] + fn a_past_master_with_a_future_override_is_kept() { + let both = calendar( + "BEGIN:VEVENT\r\nUID:e@example.com\r\nDTSTAMP:20260101T000000Z\r\n\ + DTSTART:20260901T090000Z\r\nDTEND:20260901T100000Z\r\nEND:VEVENT\r\n\ + BEGIN:VEVENT\r\nUID:e@example.com\r\nDTSTAMP:20260101T000000Z\r\n\ + RECURRENCE-ID:20260901T090000Z\r\n\ + DTSTART:20261001T090000Z\r\nDTEND:20261001T100000Z\r\nEND:VEVENT\r\n", + ); + assert!(!is_finished(&both, horizon())); + } + + #[test] + fn an_unreadable_start_is_kept_rather_than_guessed_at() { + let broken = event("DTSTART:not-a-date\r\n"); + assert!(!is_finished(&broken, horizon())); + + let absent = event("SUMMARY:No dates at all\r\n"); + assert!(!is_finished(&absent, horizon())); + } + + /// A zone that puts the end after the horizon keeps the event, which a naive + /// UTC reading would get wrong. + #[test] + fn a_tzid_is_honoured() { + // 08:00 in Los Angeles is 15:00 UTC, after a midday horizon. + let later = event("DTSTART;TZID=America/Los_Angeles:20260913T080000\r\n"); + assert!(!is_finished(&later, horizon())); + } +} diff --git a/src/http.rs b/src/http.rs new file mode 100644 index 0000000..046055a --- /dev/null +++ b/src/http.rs @@ -0,0 +1,34 @@ +//! The HTTP client shared by the legs calcalist drives itself. +//! +//! Google and WebCal both reach their remotes from this process, so both use the +//! agent built here rather than configuring one apiece. + +use std::time::Duration; + +/// How long one request may take, end to end. ureq defaults every timeout to +/// none, so a server that accepts a connection and then says nothing would stall +/// a cycle for as long as the machine stayed up — indistinguishable, from the +/// terminal, from a sync that was merely slow. +const REQUEST_TIMEOUT: Duration = Duration::from_secs(60); + +/// Long enough for a handshake over a bad link, short enough that an unreachable +/// host is reported rather than waited on. +const CONNECT_TIMEOUT: Duration = Duration::from_secs(15); + +/// `http_status_as_error` is off because the callers read the status themselves: +/// a 410 drives a full resync on the Google side, and a 404 is an ordinary +/// answer rather than a transport failure. +/// +/// Non-standard methods are allowed because CalDAV is built on them: `PROPFIND` +/// and `REPORT` come from RFC 4918 and RFC 6578 rather than from the HTTP core +/// spec, and ureq refuses them by default. The setting only permits methods +/// calcalist itself chooses to send. +pub fn agent() -> ureq::Agent { + ureq::Agent::config_builder() + .http_status_as_error(false) + .allow_non_standard_methods(true) + .timeout_global(Some(REQUEST_TIMEOUT)) + .timeout_connect(Some(CONNECT_TIMEOUT)) + .build() + .into() +} diff --git a/src/ical.rs b/src/ical.rs index 89524c3..da1db24 100644 --- a/src/ical.rs +++ b/src/ical.rs @@ -10,6 +10,7 @@ //! Line terminators are normalised to whichever style dominates the input, since //! RFC 5545 mandates CRLF and mixed endings are a defect rather than content. +use std::collections::BTreeMap; use std::ops::Range; use thiserror::Error; @@ -158,38 +159,128 @@ impl Calendar { /// converts to another model rather than round-tripping back to iCalendar. pub fn events(&self) -> Vec { let newline = &self.newline; + self.components("VEVENT") + .iter() + .filter_map(|block| { + let body = self.render(block); + Calendar::parse(&format!( + "BEGIN:VCALENDAR{newline}VERSION:2.0{newline}{body}END:VCALENDAR{newline}" + )) + .ok() + }) + .collect() + } + + /// Each distinct `UID` as a calendar of its own, keyed by that UID. + /// + /// A feed arrives as one document holding every event; a vdir holds one file + /// per UID. The split has to carry more than the `VEVENT`s across: recurrence + /// overrides share their master's UID and belong in the same file, and a + /// `DTSTART` naming a `TZID` cannot be read without the `VTIMEZONE` defining + /// it. The feed's own properties come too, so each file is a calendar in its + /// own right rather than a fragment. + pub fn split_by_uid(&self) -> BTreeMap { + let newline = &self.newline; + let envelope = self.render(&self.envelope()); + let timezones: Vec<(String, String)> = self + .components("VTIMEZONE") + .iter() + .filter_map(|block| { + Some(( + self.value_in(block, "VTIMEZONE", "TZID")?, + self.render(block), + )) + }) + .collect(); + + let mut grouped: BTreeMap = BTreeMap::new(); + for block in self.components("VEVENT") { + let Some(uid) = self.value_in(&block, "VEVENT", "UID") else { + continue; + }; + grouped + .entry(uid) + .or_default() + .push_str(&self.render(&block)); + } + + grouped + .into_iter() + .filter_map(|(uid, events)| { + // A definition is carried when its name appears in the events + // that would reference it. Matching on the text can only ever + // include one too many, which costs a few lines in the file; + // leaving one out would make the times unreadable. + let zones: String = timezones + .iter() + .filter(|(tzid, _)| events.contains(tzid.as_str())) + .map(|(_, block)| block.as_str()) + .collect(); + let text = format!( + "BEGIN:VCALENDAR{newline}{envelope}{zones}{events}END:VCALENDAR{newline}" + ); + Calendar::parse(&text).ok().map(|split| (uid, split)) + }) + .collect() + } + + /// Every top-level block of `component`, innermost lines included. + fn components(&self, component: &str) -> Vec> { let mut out = Vec::new(); - let mut block: Option = None; + let mut block: Option> = None; for line in &self.lines { let Some(depth) = line .path .iter() - .position(|name| name.eq_ignore_ascii_case("VEVENT")) + .position(|name| name.eq_ignore_ascii_case(component)) else { continue; }; let outermost = depth + 1 == line.path.len(); if outermost && line.has_name("BEGIN") { - block = Some(String::new()); + block = Some(Vec::new()); } - let Some(text) = block.as_mut() else { + let Some(lines) = block.as_mut() else { continue; }; - text.push_str(&self.rendered(line)); - text.push_str(newline); + lines.push(line); if outermost && line.has_name("END") - && let Some(text) = block.take() - && let Ok(event) = Calendar::parse(&format!( - "BEGIN:VCALENDAR{newline}VERSION:2.0{newline}{text}END:VCALENDAR{newline}" - )) + && let Some(lines) = block.take() { - out.push(event); + out.push(lines); } } out } + /// The calendar's own properties, without the `VCALENDAR` wrapper itself. + fn envelope(&self) -> Vec<&Line> { + self.lines + .iter() + .filter(|line| line.path.len() == 1 && line.is_directly_in("VCALENDAR")) + .filter(|line| !line.has_name("BEGIN") && !line.has_name("END")) + .collect() + } + + /// The value of `name` sitting directly inside `component`, within a block. + fn value_in(&self, block: &[&Line], component: &str, name: &str) -> Option { + block + .iter() + .find(|line| line.is_directly_in(component) && line.has_name(name)) + .map(|line| split_property(&line.unfolded).2.trim().to_string()) + } + + /// Lines as they will be emitted, each followed by a newline. + fn render(&self, lines: &[&Line]) -> String { + let mut out = String::new(); + for line in lines { + out.push_str(&self.rendered(line)); + out.push_str(&self.newline); + } + out + } + /// The UID shared by the event and any recurrence overrides. pub fn uid(&self) -> Option<&str> { self.properties("VEVENT", "UID").next().map(|p| p.value) diff --git a/src/main.rs b/src/main.rs index 8a7bd9a..f8e4802 100644 --- a/src/main.rs +++ b/src/main.rs @@ -1,9 +1,12 @@ //! calcalist — aggregate and sync events between CalDAV, Google Calendar and iCal feeds. +mod caldav; mod cli; mod config; mod doctor; mod google; +mod horizon; +mod http; mod ical; mod lock; mod mirror; @@ -12,10 +15,12 @@ mod pimsync; mod provenance; mod reconcile; mod retarget; +mod secret; mod state; mod status; mod sync; mod vdir; +mod webcal; use std::process::ExitCode; @@ -37,9 +42,14 @@ fn main() -> ExitCode { .. } => run_status(cli), cli @ Cli { - command: Command::Sync { dry_run, force }, + command: + Command::Sync { + dry_run, + force, + skip_past, + }, .. - } => run_sync(cli, *dry_run, *force), + } => run_sync(cli, *dry_run, *force, skip_past.as_deref()), cli @ Cli { command: Command::Prune { force }, .. @@ -85,7 +95,18 @@ fn run_doctor(cli: &Cli) -> ExitCode { } } -fn run_sync(cli: &Cli, dry_run: bool, force: bool) -> ExitCode { +fn run_sync(cli: &Cli, dry_run: bool, force: bool, skip_past: Option<&str>) -> ExitCode { + // Resolved once, before anything runs: `today` must mean one instant for the + // whole cycle, not whichever moment each aggregate happened to be reached. + let cutoff = match skip_past.map(resolve_cutoff).transpose() { + Ok(cutoff) => cutoff, + Err(value) => { + eprintln!( + "calcalist: --skip-past {value} is not a date of the form YYYY-MM-DD, or `today`" + ); + return ExitCode::FAILURE; + } + }; let (config, _) = match Config::load(cli.config.as_deref()) { Ok(loaded) => loaded, Err(error) => return fail(&error), @@ -100,7 +121,7 @@ fn run_sync(cli: &Cli, dry_run: bool, force: bool) -> ExitCode { Ok(lock) => lock, Err(error) => return fail(&error), }; - match sync::run(&config, &state_dir, dry_run, force) { + match sync::run(&config, &state_dir, dry_run, force, cutoff) { Ok(report) => { print_report(&report); // A cycle that could not reach an endpoint did real work, but saying @@ -116,6 +137,18 @@ fn run_sync(cli: &Cli, dry_run: bool, force: bool) -> ExitCode { } } +/// The instant a `--skip-past` value names: midnight local on a given date, or +/// on today when the flag was passed bare. +fn resolve_cutoff(value: &str) -> Result { + if value.trim().eq_ignore_ascii_case("today") { + return jiff::Zoned::now() + .start_of_day() + .map(|start| start.timestamp()) + .map_err(|_| value); + } + crate::config::parse_cutoff(value).ok_or(value) +} + /// Lists a server's calendars as configuration ready to paste. /// /// The URL of a CalDAV calendar is the most error-prone thing in the config: it @@ -347,6 +380,28 @@ fn print_report(report: &Report) { ); } } + for collection in &report.caldav { + let load = if collection.full_load { + " (read in full)" + } else { + "" + }; + println!( + "{}: pulled {} / removed {}; pushed {} new, {} changed, {} deleted{load}", + collection.endpoint, + collection.written, + collection.deleted, + collection.created_remotely, + collection.updated_remotely, + collection.deleted_remotely, + ); + } + for feed in &report.webcal { + println!( + "{}: pulled {} / removed {} (feed, read-only)", + feed.endpoint, feed.written, feed.deleted + ); + } if report.aggregates.is_empty() { println!("no aggregates configured"); return; @@ -366,6 +421,12 @@ fn print_report(report: &Report) { aggregate.kept_local ); } + if aggregate.finished_skipped > 0 { + println!( + " {} event(s) left behind as history: finished, and not repeating", + aggregate.finished_skipped + ); + } for conflict in &aggregate.conflicts { println!( " conflict: {} changed in both places; kept the version from `{}`", diff --git a/src/pimsync.rs b/src/pimsync.rs index 9abdcb2..91061a5 100644 --- a/src/pimsync.rs +++ b/src/pimsync.rs @@ -1,8 +1,8 @@ //! Driving the `pimsync` binary. //! -//! pimsync handles the CalDAV and WebCal legs, mirroring each remote into a local -//! vdir. It is invoked one-shot: its own daemon would race the reconciler over the -//! same files, so scheduling belongs to calcalist. +//! pimsync handles the CalDAV leg, mirroring each remote into a local vdir. It is +//! invoked one-shot: its own daemon would race the reconciler over the same +//! files, so scheduling belongs to calcalist. use std::fmt; use std::fs; @@ -163,16 +163,15 @@ pub fn generate_for( for endpoint in &config.endpoints { match &endpoint.kind { + // A collection marked `native` is calcalist's own; leaving it out + // of the config is what keeps the two implementations off each + // other's toes while both are in use. + EndpointKind::Caldav { native: true, .. } => {} EndpointKind::Caldav { .. } => out.push_str(&caldav_pair(endpoint, direction)?), - EndpointKind::Webcal { url, url_command } => { - out.push_str(&webcal_pair( - endpoint, - url.as_deref(), - url_command.as_deref(), - )); - } - // Handled by calcalist's own Google module. - EndpointKind::Google { .. } => {} + // Both have a module of calcalist's own. A feed is one document + // fetched over HTTP and read-only by protocol, which needs none of + // the machinery a pimsync pair brings. + EndpointKind::Google { .. } | EndpointKind::Webcal { .. } => {} } } Ok(out) @@ -183,6 +182,7 @@ fn caldav_pair(endpoint: &Endpoint, direction: Direction) -> Result Result, url_command: Option<&str>) -> String { - let id = &endpoint.id; - // A feed URL that is itself a credential is fetched by command, the same way - // passwords are, so it never appears in either configuration file. - let url_directive = match (url, url_command) { - (_, Some(command)) => format!("\turl {{\n\t\tshell {command}\n\t}}\n"), - (Some(url), None) => format!("\turl {}\n", quote(url)), - (None, None) => String::new(), - }; - // No `read_only` directive here: pimsync 0.5.11 documents it as applying to - // every storage type, but rejects it on a webcal storage with a bare "Could - // not parse file". It is redundant regardless — webcal is read-only by type, - // and `one_way` already fixes the direction. - format!( - "\nstorage {id}_remote {{\n\ttype webcal\n{url_directive}\tcollection_id {id}\n}}\n\ - \npair {id} {{\n\tstorage_a {id}_remote\n\tstorage_b {LOCAL_STORAGE}\n\ - \tcollection {{\n\t\talias {id}\n\t\tid_a {id}\n\t\tid_b {id}\n\t}}\n\ - \tone_way\n\ton_empty skip\n\ton_delete skip\n}}\n" - ) -} - -/// Splits a calendar URL into the container to enumerate and the collection's -/// own path. -/// -/// pimsync identifies a collection either by an id taken from a URL segment or -/// by its exact path. The path is used here, so a calendar whose last segment -/// happens to match another's cannot be confused for it. fn split_collection_url(url: &str) -> Option<(String, String)> { let authority_start = url.find("://")? + 3; let path_start = authority_start + url[authority_start..].find('/')?; @@ -503,6 +476,15 @@ pub fn check(config_path: &Path) -> Result<(), PimsyncError> { /// `pimsync -c `, with stdin closed so that a secret helper /// reaching for a terminal fails outright rather than waiting on one forever. +/// +/// The working directory is pinned to the one holding the generated config — +/// every path pimsync is given is absolute, so it needs no particular directory, +/// and inheriting the user's would put their files within reach of a relative +/// path. vstorage 0.8's `unset_property` is exactly that: clearing a collection +/// property calls `remove_file("order")` rather than the joined path it just +/// built, so it unlinks whatever `order` sits in the current directory. Pinning +/// the directory cannot make that succeed, but it keeps the blast radius inside +/// the state directory, where calcalist writes nothing by that name. fn invocation(config_path: &Path, command: &str) -> Command { let mut process = Command::new(BINARY); process @@ -510,6 +492,9 @@ fn invocation(config_path: &Path, command: &str) -> Command { .arg(config_path) .arg(command) .stdin(Stdio::null()); + if let Some(directory) = config_path.parent() { + process.current_dir(directory); + } process } @@ -627,50 +612,27 @@ url = "https://example.org/holidays.ics" assert!(text.contains("shell pass show example/caldav")); } + /// Feeds are calcalist's own leg now, so nothing about them should reach + /// the generated config — no storage, no pair, and above all no URL, since + /// a feed address can itself be the credential. #[test] - fn a_feed_becomes_a_one_way_read_only_pair() { + fn a_feed_is_left_out_of_the_generated_config() { let text = generated(); - assert!(text.contains("type webcal")); - assert!(text.contains("one_way")); - // pimsync 0.5.11 rejects `read_only` on a webcal storage despite - // documenting it for all storage types. - assert!(!block(&text, "storage holidays_remote {").contains("read_only")); - assert!(text.contains("collection_id holidays")); + assert!(!text.contains("webcal"), "{text}"); + assert!(!text.contains("holidays"), "{text}"); + assert!(!text.contains("https://example.org"), "{text}"); } - /// Both guards against a failed pull being mirrored as a mass deletion. + /// Both guard against a failed pull being mirrored as a mass deletion. #[test] fn every_pair_carries_the_deletion_guards() { let text = generated(); let pairs = text.matches("\npair ").count(); - assert_eq!(pairs, 2); + assert_eq!(pairs, 1, "only the CalDAV endpoint should make a pair"); assert_eq!(text.matches("on_empty skip").count(), pairs); assert_eq!(text.matches("on_delete skip").count(), pairs); } - /// A feed whose URL is itself a credential — Google's secret iCal address — - /// must be fetched by command, never written into either config file. - #[test] - fn a_secret_feed_url_is_fetched_by_command() { - let config: Config = toml::from_str( - r#" -version = 1 - -[[endpoint]] -id = "gcal-feed" -type = "webcal" -url_command = "secret-tool lookup service calcalist account gcal-ics" -"#, - ) - .expect("config should parse"); - let text = generate_for(&config, Path::new("/var/state/calcalist"), Direction::Both) - .expect("generate"); - - assert!(text.contains("url {")); - assert!(text.contains("shell secret-tool lookup service calcalist account gcal-ics")); - assert!(!text.contains("https://"), "no literal URL should appear"); - } - #[test] fn splits_a_calendar_url_into_container_and_path() { assert_eq!( diff --git a/src/reconcile.rs b/src/reconcile.rs index 0d339ef..0074ea0 100644 --- a/src/reconcile.rs +++ b/src/reconcile.rs @@ -17,6 +17,9 @@ use thiserror::Error; use crate::config::SchedulingSuppression; use crate::ical::Calendar; +use jiff::Timestamp; + +use crate::horizon; use crate::mirror; use crate::provenance::{self, derive_uid}; use crate::state::{AggregateState, Link}; @@ -42,6 +45,10 @@ pub struct Policy<'a> { pub max_delete_fraction: f64, /// Set by `--force`, to proceed past the mass-deletion guard. pub force: bool, + /// When set, source events that have finished and will not recur are not + /// mirrored. The instant is taken once per cycle so that every aggregate in + /// a run is measured against the same moment. + pub horizon: Option, } /// A change to make on disk. Applying these is somebody else's job. @@ -109,6 +116,10 @@ pub struct Outcome { /// Events left in the aggregate rather than filed under a source. Not a /// failure: an aggregate is also a calendar, and these are its own events. pub kept_local: Vec, + /// Source events left behind as finished history. Counted rather than + /// listed: on a first sync this runs to thousands, which no report wants + /// spelled out. + pub finished_skipped: usize, } #[derive(Debug, Error, PartialEq)] @@ -141,6 +152,7 @@ pub fn reconcile( conflicts: Vec::new(), skipped: Vec::new(), kept_local: Vec::new(), + finished_skipped: 0, }; let mut live: HashSet = HashSet::new(); let mut deletions = 0usize; @@ -179,6 +191,19 @@ fn mirror_source( if provenance::is_derived(source_uid) { continue; } + // History the aggregate was asked not to carry. This sits above the + // `prior_links` lookup on purpose: from here there is no path to + // `DeleteSource`, so excluding an event can never be mistaken for the + // user having deleted its mirror. An event that *was* mirrored and is + // now excluded loses its mirror through `retire_vanished_sources` + // instead, which leaves the original alone. + if policy + .horizon + .is_some_and(|horizon| horizon::is_finished(&item.calendar, horizon)) + { + outcome.finished_skipped += 1; + continue; + } let aggregate_uid = derive_uid(policy.aggregate_id, source.id, source_uid).to_string(); let desired = mirror::to_aggregate( &item.calendar, @@ -595,6 +620,30 @@ mod tests { propagate_deletes: true, max_delete_fraction: 0.5, force: false, + horizon: None, + } + } + + /// Well after the fixtures' `DTSTART:20260910T090000Z`, so a fixture event + /// counts as finished unless it is given a reason not to. + fn horizon() -> Timestamp { + "2026-09-13T12:00:00Z".parse().expect("horizon") + } + + /// A source event carrying extra properties, for the dates and rules the + /// horizon rule turns on. + fn item_with(uid: &str, summary: &str, extra: &str) -> Item { + let calendar = Calendar::parse(&format!( + "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nBEGIN:VEVENT\r\nUID:{uid}\r\n\ + DTSTART:20260910T090000Z\r\nSUMMARY:{summary}\r\n{extra}END:VEVENT\r\n\ + END:VCALENDAR\r\n" + )) + .expect("fixture should parse"); + Item { + uid: uid.to_string(), + path: Path::new("/tmp/calcalist-test").join(format!("{uid}.ics")), + hash: calendar.content_hash(), + calendar, } } @@ -926,6 +975,101 @@ mod tests { assert_eq!(outcome.kept_local, vec!["hand-written@phone".to_string()]); } + /// The point of `skip_past`: finished history never reaches the aggregate, + /// and leaves no link behind to be misread later. + #[test] + fn a_finished_event_is_not_mirrored_and_leaves_no_link() { + let sources = map(vec![item("event-1", "Standup")]); + let policy = Policy { + horizon: Some(horizon()), + ..policy() + }; + let outcome = + reconcile(policy, &[view(&sources, true)], &BTreeMap::new(), None).expect("ok"); + + assert!(outcome.actions.is_empty(), "{:?}", outcome.actions); + assert!(outcome.links.is_empty(), "{:?}", outcome.links); + assert_eq!(outcome.finished_skipped, 1); + } + + /// A series that began in the past is still live, so the horizon must not + /// touch it however old its `DTSTART`. + #[test] + fn a_finished_event_that_repeats_is_mirrored_as_usual() { + let sources = map(vec![item_with( + "event-1", + "Standup", + "RRULE:FREQ=WEEKLY\r\n", + )]); + let policy = Policy { + horizon: Some(horizon()), + ..policy() + }; + let outcome = + reconcile(policy, &[view(&sources, true)], &BTreeMap::new(), None).expect("ok"); + + assert_eq!(outcome.actions.len(), 1, "{:?}", outcome.actions); + assert_eq!(outcome.links.len(), 1); + assert_eq!(outcome.finished_skipped, 0); + } + + #[test] + fn an_event_still_to_come_is_mirrored() { + let sources = map(vec![item_with( + "event-1", + "Retrospective", + "DTEND:20260930T100000Z\r\n", + )]); + let policy = Policy { + horizon: Some(horizon()), + ..policy() + }; + let outcome = + reconcile(policy, &[view(&sources, true)], &BTreeMap::new(), None).expect("ok"); + + assert_eq!(outcome.actions.len(), 1, "{:?}", outcome.actions); + assert_eq!(outcome.finished_skipped, 0); + } + + /// The safety property the whole placement of the filter exists for. + /// + /// Turning `skip_past` on for an aggregate that already carries history must + /// retire those mirrors and nothing more. If the exclusion were read as "the + /// user deleted this mirror", the original would be deleted from the source + /// calendar — which for a Google endpoint means the user's real event. + #[test] + fn excluding_an_already_mirrored_event_never_deletes_it_from_the_source() { + let sources = map(vec![item("event-1", "Standup")]); + let aggregate = map(vec![mirrored("event-1", "Standup")]); + let prior = prior_with("event-1", "Standup"); + + let policy = Policy { + horizon: Some(horizon()), + ..policy() + }; + let outcome = + reconcile(policy, &[view(&sources, true)], &aggregate, Some(&prior)).expect("ok"); + + assert!( + !outcome + .actions + .iter() + .any(|action| matches!(action, Action::DeleteSource { .. })), + "the source event must survive: {:?}", + outcome.actions + ); + assert!( + outcome + .actions + .iter() + .any(|action| matches!(action, Action::DeleteAggregate { .. })), + "the stale mirror should be retired: {:?}", + outcome.actions + ); + // The link goes with the mirror, so the next cycle starts clean. + assert!(outcome.links.is_empty(), "{:?}", outcome.links); + } + /// Our own mirror appearing inside a source must not be mirrored again. #[test] fn mirrors_found_in_a_source_are_not_re_ingested() { diff --git a/src/retarget.rs b/src/retarget.rs index 0cbd347..95009c4 100644 --- a/src/retarget.rs +++ b/src/retarget.rs @@ -115,8 +115,14 @@ pub fn retarget( &old_target, state_dir, &mut state, - false, - false, + // Never skip history here, whatever the aggregate says. A retarget moves + // what exists; applying a horizon mid-move would quietly leave the old + // target's past events behind instead of carrying them across. + sync::Cycle { + dry_run: false, + force: false, + skip_past: None, + }, )?; let (materialised, carried) = materialise( @@ -369,8 +375,19 @@ default_sink = "src" .clone(); let state_path = dir.join(crate::state::FILE_NAME); let mut state = State::load(&state_path).expect("state"); - sync::sync_aggregate(config, aggregate, &target, dir, &mut state, false, false) - .expect("reconcile"); + sync::sync_aggregate( + config, + aggregate, + &target, + dir, + &mut state, + sync::Cycle { + dry_run: false, + force: false, + skip_past: None, + }, + ) + .expect("reconcile"); state.save(&state_path).expect("save state"); } diff --git a/src/secret.rs b/src/secret.rs new file mode 100644 index 0000000..ae6513c --- /dev/null +++ b/src/secret.rs @@ -0,0 +1,49 @@ +//! Fetching secrets by running a command. +//! +//! Credentials are never stored in the configuration file, so that it stays +//! portable and safe to keep under version control. Instead it names a command +//! that prints the secret — `secret-tool lookup ...`, `pass show ...` — which is +//! run when the secret is actually needed. Both the Google client secret and a +//! WebCal feed URL that is itself a credential arrive this way. + +use std::io; +use std::process::{Command, Stdio}; + +use thiserror::Error; + +#[derive(Debug, Error)] +pub enum SecretError { + #[error("could not run the secret command `{command}`: {source}")] + Spawn { + command: String, + #[source] + source: io::Error, + }, + #[error("the secret command `{command}` failed: {message}")] + Failed { command: String, message: String }, +} + +/// Runs `command` under a shell and returns its first line of output. +/// +/// stdin is closed rather than inherited: a helper that falls back to prompting +/// on the terminal — `pass` reaching for pinentry, say — must fail and say so, +/// not wait for an answer nobody can see it asking for. +pub fn run(command: &str) -> Result { + let output = Command::new("sh") + .arg("-c") + .arg(command) + .stdin(Stdio::null()) + .output() + .map_err(|source| SecretError::Spawn { + command: command.to_string(), + source, + })?; + if !output.status.success() { + return Err(SecretError::Failed { + command: command.to_string(), + message: String::from_utf8_lossy(&output.stderr).trim().to_string(), + }); + } + let text = String::from_utf8_lossy(&output.stdout); + Ok(text.lines().next().unwrap_or_default().trim().to_string()) +} diff --git a/src/state.rs b/src/state.rs index d7b4d74..1179420 100644 --- a/src/state.rs +++ b/src/state.rs @@ -51,6 +51,35 @@ pub struct State { /// Per-endpoint bookkeeping for the Google leg, which calcalist syncs itself. #[serde(default)] pub google: BTreeMap, + /// The same, for CalDAV collections synced natively rather than by pimsync. + #[serde(default)] + pub caldav: BTreeMap, +} + +/// What is remembered between cycles for one natively-synced CalDAV collection. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct CaldavState { + /// The server's incremental cursor, from RFC 6578. Absent means the next + /// pull reads the collection in full. + #[serde(default)] + pub sync_token: Option, + /// What is known about each event on the server, keyed by UID. + #[serde(default)] + pub events: BTreeMap, +} + +/// One event as the server and the local vdir last agreed on it. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct CaldavItem { + /// The resource's path on the server, needed to update or delete it. + pub href: String, + /// The server's etag, which conditions every write so that a change made + /// behind calcalist's back is refused rather than overwritten. + #[serde(default)] + pub etag: String, + /// Content hash when it was last written, so a local edit is detectable. + #[serde(default)] + pub hash: String, } /// What is remembered between cycles for one Google calendar. @@ -80,6 +109,7 @@ impl Default for State { version: STATE_VERSION, aggregates: BTreeMap::new(), google: BTreeMap::new(), + caldav: BTreeMap::new(), } } } diff --git a/src/sync.rs b/src/sync.rs index fe491d4..78bd219 100644 --- a/src/sync.rs +++ b/src/sync.rs @@ -2,15 +2,17 @@ //! //! Every endpoint is represented by a local vdir, so this drives the reconciler //! over local files and applies whatever it decides. Getting the events into and -//! out of those vdirs is the job of pimsync (CalDAV, WebCal) and the Google -//! module, which bracket this step. +//! out of those vdirs is the job of pimsync (CalDAV) and of the `google` and +//! `webcal` modules, which bracket this step. use std::collections::{BTreeMap, BTreeSet}; use std::fs; use std::path::{Path, PathBuf}; +use jiff::Timestamp; use thiserror::Error; +use crate::caldav::{self, CaldavError}; use crate::config::{Aggregate, Config, Endpoint, EndpointKind}; use crate::google::api::{self, ApiError}; use crate::google::auth::{self, AuthError}; @@ -18,6 +20,7 @@ use crate::pimsync::{self, Direction, PimsyncError}; use crate::reconcile::{self, Action, Conflict, Policy, ReconcileError, Skipped, SourceView}; use crate::state::{State, Target}; use crate::vdir::{self, VdirError}; +use crate::webcal::{self, WebcalError}; #[derive(Debug, Error)] pub enum SyncError { @@ -33,6 +36,18 @@ pub enum SyncError { #[source] source: ApiError, }, + #[error("caldav endpoint `{endpoint}` could not be synchronised")] + Caldav { + endpoint: String, + #[source] + source: CaldavError, + }, + #[error("webcal endpoint `{endpoint}` could not be read")] + Webcal { + endpoint: String, + #[source] + source: WebcalError, + }, #[error("google endpoint `{endpoint}` could not be authorised")] GoogleAuth { endpoint: String, @@ -70,6 +85,8 @@ pub enum SyncError { pub struct Report { pub aggregates: Vec, pub google: Vec, + pub webcal: Vec, + pub caldav: Vec, pub failures: Vec, pub blocked: Vec, pub orphan_vdirs: Vec, @@ -114,6 +131,24 @@ pub struct GoogleReport { pub exceptions_skipped: usize, } +#[derive(Debug)] +pub struct WebcalReport { + pub endpoint: String, + pub written: usize, + pub deleted: usize, +} + +#[derive(Debug)] +pub struct CaldavReport { + pub endpoint: String, + pub written: usize, + pub deleted: usize, + pub full_load: bool, + pub created_remotely: usize, + pub updated_remotely: usize, + pub deleted_remotely: usize, +} + #[derive(Debug)] pub struct AggregateReport { pub id: String, @@ -124,6 +159,23 @@ pub struct AggregateReport { pub conflicts: Vec, pub skipped: Vec, pub kept_local: usize, + /// Finished, non-repeating source events left out of the aggregate. + pub finished_skipped: usize, +} + +/// What one invocation asked for, as distinct from what the configuration says. +/// +/// Bundled because these travel together through every aggregate and are all +/// properties of the run rather than of the data. +#[derive(Debug, Clone, Copy)] +pub(crate) struct Cycle { + pub dry_run: bool, + /// `--force`, to proceed past the mass-deletion guard. + pub force: bool, + /// `--skip-past`, which applies a cutoff for this cycle whatever the + /// configuration says. Resolved from the command line before the cycle + /// starts, so `today` means one instant for the whole run. + pub skip_past: Option, } /// Runs the local half of a cycle: reconcile every aggregate and apply the result. @@ -132,7 +184,15 @@ pub fn run( state_dir: &Path, dry_run: bool, force: bool, + skip_past: Option, ) -> Result { + // Taken once, so every aggregate in this cycle is measured against the same + // instant rather than each against the moment it happened to be reached. + let cycle = Cycle { + dry_run, + force, + skip_past, + }; let state_path = state_dir.join(crate::state::FILE_NAME); let mut state = State::load(&state_path)?; let mut report = Report { @@ -146,7 +206,8 @@ pub fn run( // A dry run pulls for real, so that what it reports is measured against the // calendars as they are now — but into a throwaway copy of the local // mirrors, and through a configuration that only ever reads from a server. - // Nothing outside that copy is written, and it is removed at the end. + // Nothing outside that copy is written, and it is removed however the + // cycle ends. let workspace = if dry_run { Some(Workspace::open(state_dir)?) } else { @@ -169,7 +230,9 @@ pub fn run( if let Some(path) = &pimsync_config { pimsync::sync(path)?; } - let unavailable = pull_google(config, state_dir, work_dir, &mut state, &mut report); + let mut unavailable = pull_google(config, state_dir, work_dir, &mut state, &mut report); + unavailable.extend(pull_webcal(config, work_dir, &mut report)); + unavailable.extend(pull_caldav(config, work_dir, &mut state, &mut report)); for aggregate in &config.aggregates { // An aggregate whose target or any source could not be pulled is left @@ -191,21 +254,19 @@ pub fn run( let target = resolve(config, aggregate, &aggregate.target)?; check_target_drift(&state, aggregate, target)?; report.aggregates.push(sync_aggregate( - config, aggregate, target, work_dir, &mut state, dry_run, force, + config, aggregate, target, work_dir, &mut state, cycle, )?); } // Push what the reconciler decided out to the remotes. if !dry_run { push_google(config, state_dir, &mut state, &mut report, &unavailable); + push_caldav(config, state_dir, &mut state, &mut report, &unavailable); if let Some(path) = &pimsync_config { pimsync::sync(path)?; } state.save(&state_path)?; } - if let Some(workspace) = workspace { - workspace.discard(); - } Ok(report) } @@ -243,8 +304,16 @@ impl Workspace { fn path(&self) -> &Path { &self.root } +} - fn discard(self) { +/// Removing the copy on the way out covers every exit, not just the happy one. +/// +/// It used to be an explicit call at the end of a cycle, which a `?` anywhere +/// earlier stepped straight over — a failing pull then left the whole copy +/// behind, and the next dry run began by deleting megabytes it had itself +/// abandoned. The lock takes the same approach for the same reason. +impl Drop for Workspace { + fn drop(&mut self) { let _ = fs::remove_dir_all(&self.root); } } @@ -351,6 +420,165 @@ fn pull_google( unavailable } +/// Brings every WebCal endpoint's vdir into step with its feed, returning the +/// endpoints that could not be read. +/// +/// Feeds are read-only, so there is no push to match this and no state to keep: +/// the feed describes the calendar in full on every fetch. Failures are reported +/// the same way the Google ones are — a publisher having a bad afternoon stands +/// down only the aggregates that draw on it. +fn pull_webcal(config: &Config, vdir_base: &Path, report: &mut Report) -> BTreeSet { + let mut unavailable = BTreeSet::new(); + for endpoint in &config.endpoints { + let EndpointKind::Webcal { url, url_command } = &endpoint.kind else { + continue; + }; + let pulled = webcal::resolve_url(&endpoint.id, url.as_deref(), url_command.as_deref()) + .and_then(|url| webcal::pull(&url, &vdir_path(vdir_base, &endpoint.id))) + .map_err(|source| SyncError::Webcal { + endpoint: endpoint.id.clone(), + source, + }); + match pulled { + Ok(pulled) => report.webcal.push(WebcalReport { + endpoint: endpoint.id.clone(), + written: pulled.written, + deleted: pulled.deleted, + }), + Err(error) => { + unavailable.insert(endpoint.id.clone()); + report.failures.push(EndpointFailure { + endpoint: endpoint.id.clone(), + reason: chain(&error), + }); + } + } + } + unavailable +} + +/// Builds a client for one natively-synced CalDAV endpoint. +fn caldav_client(endpoint: &Endpoint) -> Result { + let EndpointKind::Caldav { + url, + username, + secret_command, + .. + } = &endpoint.kind + else { + unreachable!("caller matched on Caldav") + }; + let password = match secret_command { + Some(command) => crate::secret::run(command).map_err(|source| SyncError::Caldav { + endpoint: endpoint.id.clone(), + source: source.into(), + })?, + None => String::new(), + }; + caldav::Client::new(&endpoint.id, url, username, &password).map_err(|source| { + SyncError::Caldav { + endpoint: endpoint.id.clone(), + source, + } + }) +} + +/// Every natively-synced CalDAV endpoint, in configuration order. +fn native_caldav(config: &Config) -> impl Iterator { + config + .endpoints + .iter() + .filter(|endpoint| matches!(endpoint.kind, EndpointKind::Caldav { native: true, .. })) +} + +/// Brings every natively-synced CalDAV endpoint's vdir into step with its +/// collection, returning the endpoints that could not be reached. +/// +/// The collections pimsync still owns are absent here: `generate_for` and this +/// both key off the same `native` flag, so a collection belongs to exactly one +/// of the two implementations. +fn pull_caldav( + config: &Config, + vdir_base: &Path, + state: &mut State, + report: &mut Report, +) -> BTreeSet { + let mut unavailable = BTreeSet::new(); + for endpoint in native_caldav(config) { + let pulled = caldav_client(endpoint).and_then(|client| { + let dir = vdir_path(vdir_base, &endpoint.id); + let entry = state.caldav.entry(endpoint.id.clone()).or_default(); + caldav::pull(&client, &dir, entry).map_err(|source| SyncError::Caldav { + endpoint: endpoint.id.clone(), + source, + }) + }); + match pulled { + Ok(pulled) => report.caldav.push(CaldavReport { + endpoint: endpoint.id.clone(), + written: pulled.written, + deleted: pulled.deleted, + full_load: pulled.full_load, + created_remotely: 0, + updated_remotely: 0, + deleted_remotely: 0, + }), + Err(error) => { + unavailable.insert(endpoint.id.clone()); + report.failures.push(EndpointFailure { + endpoint: endpoint.id.clone(), + reason: chain(&error), + }); + } + } + } + unavailable +} + +/// Sends each natively-synced CalDAV endpoint's local changes up to its server. +/// +/// Runs after reconciliation, so what goes up is what the reconciler decided. An +/// endpoint whose pull failed is skipped: its vdir describes some earlier +/// moment, and pushing that would undo whatever has happened since. +fn push_caldav( + config: &Config, + state_dir: &Path, + state: &mut State, + report: &mut Report, + unavailable: &BTreeSet, +) { + for endpoint in native_caldav(config) { + if unavailable.contains(&endpoint.id) { + continue; + } + let pushed = caldav_client(endpoint).and_then(|client| { + let dir = vdir_path(state_dir, &endpoint.id); + let entry = state.caldav.entry(endpoint.id.clone()).or_default(); + caldav::push(&client, &dir, entry).map_err(|source| SyncError::Caldav { + endpoint: endpoint.id.clone(), + source, + }) + }); + match pushed { + Ok(pushed) => { + if let Some(existing) = report + .caldav + .iter_mut() + .find(|entry| entry.endpoint == endpoint.id) + { + existing.created_remotely = pushed.created; + existing.updated_remotely = pushed.updated; + existing.deleted_remotely = pushed.deleted; + } + } + Err(error) => report.failures.push(EndpointFailure { + endpoint: endpoint.id.clone(), + reason: chain(&error), + }), + } + } +} + /// Sends each Google endpoint's local changes up to its calendar. /// /// Runs after reconciliation, so what goes up is what the reconciler decided. @@ -439,14 +667,14 @@ fn chain(error: &dyn std::error::Error) -> String { out } -/// Whether any endpoint is one pimsync handles. A Google-only setup needs none. +/// Whether any endpoint is one pimsync handles. Only CalDAV is left: Google and +/// WebCal are both calcalist's own, so a setup without a CalDAV server runs +/// without ever spawning pimsync. pub(crate) fn needs_pimsync(config: &Config) -> bool { - config.endpoints.iter().any(|endpoint| { - matches!( - endpoint.kind, - EndpointKind::Caldav { .. } | EndpointKind::Webcal { .. } - ) - }) + config + .endpoints + .iter() + .any(|endpoint| matches!(endpoint.kind, EndpointKind::Caldav { native: false, .. })) } /// pimsync requires its vdir storage directory to exist before it will run. @@ -506,8 +734,7 @@ pub(crate) fn sync_aggregate( target: &Endpoint, state_dir: &Path, state: &mut State, - dry_run: bool, - force: bool, + cycle: Cycle, ) -> Result { let target_dir = vdir_path(state_dir, &target.id); let target_items = vdir::read(&target_dir)?; @@ -543,7 +770,15 @@ pub(crate) fn sync_aggregate( default_sink: aggregate.default_sink.as_deref(), propagate_deletes: aggregate.propagate_deletes, max_delete_fraction: aggregate.max_delete_fraction, - force, + force: cycle.force, + // The flag applies a cutoff for one cycle whatever the file says; the + // aggregate's own is what survives into the next one. + horizon: cycle.skip_past.or_else(|| { + aggregate + .skip_past + .as_deref() + .and_then(crate::config::parse_cutoff) + }), }; let outcome = reconcile::reconcile( @@ -566,6 +801,7 @@ pub(crate) fn sync_aggregate( conflicts: outcome.conflicts, skipped: outcome.skipped, kept_local: outcome.kept_local.len(), + finished_skipped: outcome.finished_skipped, }; for action in &outcome.actions { @@ -576,13 +812,13 @@ pub(crate) fn sync_aggregate( replaces, } => { report.mirrored += 1; - if !dry_run { + if !cycle.dry_run { vdir::write(&target_dir, uid, calendar, replaces.as_deref())?; } } Action::DeleteAggregate { path, .. } => { report.deleted_from_aggregate += 1; - if !dry_run { + if !cycle.dry_run { vdir::remove(path)?; } } @@ -593,20 +829,22 @@ pub(crate) fn sync_aggregate( replaces, } => { report.written_back += 1; - if !dry_run && let Some(dir) = source_dirs.get(source_id) { + if !cycle.dry_run + && let Some(dir) = source_dirs.get(source_id) + { vdir::write(dir, uid, calendar, replaces.as_deref())?; } } Action::DeleteSource { path, .. } => { report.deleted_from_sources += 1; - if !dry_run { + if !cycle.dry_run { vdir::remove(path)?; } } } } - if !dry_run { + if !cycle.dry_run { let entry = state.aggregate_mut( &aggregate.id, Target { @@ -636,6 +874,38 @@ mod tests { use crate::state::State; use std::fs; + /// A real cycle with no cutoff, which is what the fixtures below assume. + fn test_cycle() -> Cycle { + Cycle { + dry_run: false, + force: false, + skip_past: None, + } + } + + /// A dry run that fails part-way must not leave its copy behind. The cycle + /// is full of `?`, and every one of them used to step over the cleanup — so + /// the guarantee has to come from the type, not from reaching the last line. + #[test] + fn the_dry_run_copy_goes_even_when_the_cycle_does_not_finish() { + let dir = tempfile::tempdir().expect("temp"); + let event = vdir_path(dir.path(), "work").join("a.ics"); + fs::create_dir_all(event.parent().expect("parent")).expect("mkdir"); + fs::write(&event, "BEGIN:VCALENDAR\r\nEND:VCALENDAR\r\n").expect("write"); + + let root = { + let workspace = Workspace::open(dir.path()).expect("open"); + let root = workspace.path().to_path_buf(); + // The copy really is a copy, or there would be nothing to clean up. + assert!(root.join("vdir").join("work").join("a.ics").exists()); + root + }; + + assert!(!root.exists(), "the copy should go when the workspace does"); + // The originals are untouched by any of it. + assert!(event.exists()); + } + /// Endpoints are Google-typed so nothing here touches pimsync. The tests call /// `sync_aggregate` directly rather than `run`, so no network is involved. const CONFIG: &str = r#" @@ -690,7 +960,7 @@ client_id = "x.apps.googleusercontent.com" .clone(); let state_path = dir.join(crate::state::FILE_NAME); let mut state = State::load(&state_path).expect("state"); - let report = sync_aggregate(config, aggregate, &target, dir, &mut state, false, false) + let report = sync_aggregate(config, aggregate, &target, dir, &mut state, test_cycle()) .expect("reconcile"); state.save(&state_path).expect("save"); report @@ -910,7 +1180,7 @@ sources = ["gcal"] .expect("config"); seed(dir.path(), "work", "a@example.com", "Standup"); - let report = run(&config, dir.path(), false, false).expect("the cycle still returns"); + let report = run(&config, dir.path(), false, false, None).expect("the cycle still returns"); assert!(!report.failures.is_empty(), "{report:?}"); assert_eq!(report.blocked.len(), 1, "{report:?}"); @@ -949,7 +1219,7 @@ sources = ["gcal"] let before = fs::read_to_string(vdir_path(dir.path(), "work").join("a@example.com.ics")) .expect("read"); - let report = run(&config, dir.path(), true, false).expect("dry run"); + let report = run(&config, dir.path(), true, false, None).expect("dry run"); assert!(report.dry_run); assert!( diff --git a/src/vdir.rs b/src/vdir.rs index b20618a..98c4067 100644 --- a/src/vdir.rs +++ b/src/vdir.rs @@ -180,7 +180,7 @@ fn is_calendar_file(path: &Path) -> bool { /// /// UIDs may contain characters no filesystem accepts, and sanitising alone would /// let two different UIDs collide, so a digest of the original is appended. -fn file_name(uid: &str) -> String { +pub(crate) fn file_name(uid: &str) -> String { let sanitised: String = uid .chars() .map(|ch| { diff --git a/src/webcal.rs b/src/webcal.rs new file mode 100644 index 0000000..d9341f1 --- /dev/null +++ b/src/webcal.rs @@ -0,0 +1,223 @@ +//! The WebCal (published iCal feed) leg. +//! +//! A feed is a single iCalendar document fetched over HTTP, read-only by +//! protocol: there is nothing to push and no conflict to resolve. That makes the +//! whole leg a download, a split by UID and a vdir write, which is why calcalist +//! does it itself rather than through pimsync — a subprocess, a generated config +//! file and a status database bought nothing that this module does not. +//! +//! The fetch is unconditional. Feeds are small, publishers rarely honour +//! `If-Modified-Since`, and the reconciler already compares content hashes, so a +//! re-read of unchanged events costs a download and changes nothing on disk. + +use std::path::Path; + +use thiserror::Error; + +use crate::ical::{Calendar, IcalError}; +use crate::secret::SecretError; +use crate::vdir::{self, VdirError}; + +#[derive(Debug, Error)] +pub enum WebcalError { + #[error("endpoint `{0}` has neither url nor url_command")] + NoUrl(String), + #[error(transparent)] + Secret(#[from] SecretError), + #[error("could not fetch {url}: {source}")] + Fetch { + url: String, + #[source] + source: Box, + }, + #[error("{url} answered {status}")] + Status { url: String, status: u16 }, + #[error("{url} did not return a readable calendar: {source}")] + Malformed { + url: String, + #[source] + source: IcalError, + }, + #[error(transparent)] + Vdir(#[from] VdirError), +} + +#[derive(Debug, Default)] +pub struct PullReport { + pub written: usize, + pub deleted: usize, +} + +/// Brings an endpoint's local vdir into step with its feed. +/// +/// The feed is the whole truth every time — it is one document describing the +/// calendar in full — so an item held locally that the feed no longer mentions +/// has been withdrawn, and goes. +pub fn pull(url: &str, vdir_dir: &Path) -> Result { + let feed = fetch(url)?; + let calendar = Calendar::parse(&feed).map_err(|source| WebcalError::Malformed { + url: url.to_string(), + source, + })?; + let events = calendar.split_by_uid(); + + let held = vdir::read(vdir_dir)?; + let mut report = PullReport::default(); + + for (uid, event) in &events { + // Unchanged events are left alone: rewriting them would churn every + // file on every cycle, and the reconciler reads content, not mtimes. + let current = held.get(uid); + if current.is_some_and(|item| item.hash == event.content_hash()) { + continue; + } + vdir::write( + vdir_dir, + uid, + event, + current.map(|item| item.path.as_path()), + )?; + report.written += 1; + } + + for (uid, item) in &held { + if !events.contains_key(uid) { + vdir::remove(&item.path)?; + report.deleted += 1; + } + } + + Ok(report) +} + +/// The feed's address, which may itself be the credential. +/// +/// Google's "secret address in iCal format" grants read access to the whole +/// calendar to anyone holding it, so such a URL is fetched by command and never +/// written into the configuration file. +pub fn resolve_url( + endpoint: &str, + url: Option<&str>, + url_command: Option<&str>, +) -> Result { + match (url, url_command) { + (_, Some(command)) => Ok(crate::secret::run(command)?), + (Some(url), None) => Ok(url.to_string()), + (None, None) => Err(WebcalError::NoUrl(endpoint.to_string())), + } +} + +fn fetch(url: &str) -> Result { + let mut response = + crate::http::agent() + .get(url) + .call() + .map_err(|source| WebcalError::Fetch { + url: url.to_string(), + source: Box::new(source), + })?; + let status = response.status().as_u16(); + if !(200..300).contains(&status) { + return Err(WebcalError::Status { + url: url.to_string(), + status, + }); + } + response + .body_mut() + .read_to_string() + .map_err(|source| WebcalError::Fetch { + url: url.to_string(), + source: Box::new(source), + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + const FEED: &str = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:-//Test//EN\r\n\ + BEGIN:VEVENT\r\nUID:one@example.com\r\nDTSTAMP:20260101T000000Z\r\n\ + DTSTART;VALUE=DATE:20260101\r\nSUMMARY:New Year\r\nEND:VEVENT\r\n\ + BEGIN:VEVENT\r\nUID:two@example.com\r\nDTSTAMP:20260101T000000Z\r\n\ + DTSTART;VALUE=DATE:20260501\r\nSUMMARY:May Day\r\nEND:VEVENT\r\n\ + END:VCALENDAR\r\n"; + + /// A recurring event and one overridden occurrence share a UID, so they have + /// to land in one file; splitting them would give the vdir two events where + /// the calendar has one. + const SERIES: &str = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\n\ + BEGIN:VTIMEZONE\r\nTZID:Europe/Berlin\r\n\ + BEGIN:STANDARD\r\nDTSTART:19701025T030000\r\nTZOFFSETFROM:+0200\r\n\ + TZOFFSETTO:+0100\r\nEND:STANDARD\r\nEND:VTIMEZONE\r\n\ + BEGIN:VTIMEZONE\r\nTZID:America/New_York\r\n\ + BEGIN:STANDARD\r\nDTSTART:19701101T020000\r\nTZOFFSETFROM:-0400\r\n\ + TZOFFSETTO:-0500\r\nEND:STANDARD\r\nEND:VTIMEZONE\r\n\ + BEGIN:VEVENT\r\nUID:series@example.com\r\nDTSTAMP:20260101T000000Z\r\n\ + DTSTART;TZID=Europe/Berlin:20260105T090000\r\nRRULE:FREQ=WEEKLY\r\n\ + SUMMARY:Standup\r\nEND:VEVENT\r\n\ + BEGIN:VEVENT\r\nUID:series@example.com\r\nDTSTAMP:20260101T000000Z\r\n\ + RECURRENCE-ID;TZID=Europe/Berlin:20260112T090000\r\n\ + DTSTART;TZID=Europe/Berlin:20260112T100000\r\n\ + SUMMARY:Standup (late)\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n"; + + fn split(feed: &str) -> std::collections::BTreeMap { + Calendar::parse(feed) + .expect("the feed should parse") + .split_by_uid() + } + + #[test] + fn each_event_becomes_a_calendar_of_its_own() { + let events = split(FEED); + assert_eq!(events.len(), 2); + let one = events["one@example.com"].to_ics(); + assert!(one.contains("SUMMARY:New Year"), "{one}"); + assert!(!one.contains("May Day"), "{one}"); + // The envelope travels, or the file is a fragment rather than a calendar. + assert!(one.contains("PRODID:-//Test//EN"), "{one}"); + } + + #[test] + fn a_series_and_its_override_stay_together() { + let events = split(SERIES); + assert_eq!(events.len(), 1); + let series = events["series@example.com"].to_ics(); + assert!(series.contains("SUMMARY:Standup"), "{series}"); + assert!(series.contains("SUMMARY:Standup (late)"), "{series}"); + assert!(series.contains("RECURRENCE-ID"), "{series}"); + } + + /// A time is meaningless without the zone it names, and carrying every zone + /// in the feed into every file would bloat each one for nothing. + #[test] + fn only_the_timezones_an_event_names_come_with_it() { + let events = split(SERIES); + let series = events["series@example.com"].to_ics(); + assert!(series.contains("TZID:Europe/Berlin"), "{series}"); + assert!(!series.contains("America/New_York"), "{series}"); + } + + #[test] + fn an_event_without_a_uid_is_skipped_rather_than_failing_the_feed() { + let feed = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\n\ + BEGIN:VEVENT\r\nDTSTAMP:20260101T000000Z\r\nSUMMARY:Nameless\r\nEND:VEVENT\r\n\ + BEGIN:VEVENT\r\nUID:kept@example.com\r\nDTSTAMP:20260101T000000Z\r\n\ + SUMMARY:Kept\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n"; + let events = split(feed); + assert_eq!(events.keys().collect::>(), ["kept@example.com"]); + } + + #[test] + fn a_url_command_supplies_the_address() { + let resolved = resolve_url("feed", None, Some("printf https://example.com/f.ics")) + .expect("the command should supply a url"); + assert_eq!(resolved, "https://example.com/f.ics"); + } + + #[test] + fn an_endpoint_with_no_address_at_all_is_an_error() { + let error = resolve_url("feed", None, None).expect_err("there is no url"); + assert!(matches!(error, WebcalError::NoUrl(_)), "{error:?}"); + } +} diff --git a/systemd/calcalist.service b/systemd/calcalist.service index caeadf5..cb8ed82 100644 --- a/systemd/calcalist.service +++ b/systemd/calcalist.service @@ -3,7 +3,11 @@ Description=Synchronise calendars with calcalist [Service] Type=oneshot -ExecStart=%h/.local/bin/calcalist sync +# The path the Nix install puts it at, which is also the one that guarantees a +# matching pimsync. A `cargo install` lands in %h/.cargo/bin instead; an +# absolute path is used either way because a user manager's PATH is not the +# login shell's and cannot be relied on to have found either. +ExecStart=%h/.nix-profile/bin/calcalist sync NoNewPrivileges=true # Deliberately no ProtectSystem, PrivateTmp or IPC sandboxing. Credentials come diff --git a/systemd/calcalist.timer b/systemd/calcalist.timer index d88f322..423ea15 100644 --- a/systemd/calcalist.timer +++ b/systemd/calcalist.timer @@ -4,7 +4,10 @@ Description=Synchronise calendars with calcalist every 15 minutes [Timer] OnBootSec=2m OnUnitActiveSec=15m -Persistent=true + +# No Persistent=: it only has an effect on OnCalendar= timers, and these are +# monotonic. OnBootSec= already covers the case it would be wanted for, which is +# catching up shortly after the machine comes back. # Spreads requests instead of every installation calling Google on the quarter # hour. Google's Calendar API quota is per project and enforced per minute, and diff --git a/tests/caldav.rs b/tests/caldav.rs index d770839..07a9be2 100644 --- a/tests/caldav.rs +++ b/tests/caldav.rs @@ -23,6 +23,12 @@ const MEETING: &str = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:-//test//EN\r\n\ BEGIN:VALARM\r\nACTION:DISPLAY\r\nTRIGGER:-PT15M\r\nDESCRIPTION:Soon\r\n\ END:VALARM\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n"; +/// An event added to a source after a cycle has already stored a cursor. +const LATER: &str = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:-//test//EN\r\n\ + BEGIN:VEVENT\r\nUID:later@work\r\nDTSTAMP:20260101T000000Z\r\n\ + DTSTART:20260915T090000Z\r\nDTEND:20260915T100000Z\r\n\ + SUMMARY:Retrospective\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n"; + /// An event written straight into the aggregate, belonging to no source. const HAND_WRITTEN: &str = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:-//phone//EN\r\n\ BEGIN:VEVENT\r\nUID:hand-written@phone\r\nDTSTAMP:20260101T000000Z\r\n\ @@ -497,3 +503,250 @@ fn a_second_cycle_is_refused_while_one_is_running() { "the lock should be free again" ); } + +/// The same cycle, driven by calcalist's own CalDAV client instead of pimsync. +/// +/// This is the comparison the `native` flag exists for: the assertions are the +/// pimsync test's, so a divergence between the two implementations shows up as +/// a failure here rather than as a surprise on someone's calendar. +fn native_fixture() -> Option { + let missing = support::missing_binaries(); + if !missing.is_empty() { + eprintln!( + "skipped: {} not on PATH; run under devbox", + missing.join(", ") + ); + return None; + } + + let root = tempfile::tempdir().expect("temp"); + let server = Radicale::start(root.path()); + server.create_calendar("work"); + server.create_calendar("unified"); + server.put_event("work", "meeting.ics", MEETING); + let feed = Feed::start(HOLIDAY_FEED.to_string()); + + let config = format!( + r#" +version = 1 + +[[endpoint]] +id = "work" +type = "caldav" +native = true +url = "{work}" +username = "{user}" +secret_command = "printf password" + +[[endpoint]] +id = "published" +type = "caldav" +native = true +url = "{unified}" +username = "{user}" +secret_command = "printf password" + +[[endpoint]] +id = "holidays" +type = "webcal" +url = "{feed}" + +[[aggregate]] +id = "everything" +target = "published" +sources = ["work", "holidays"] +default_sink = "work" +"#, + work = server.url("work"), + unified = server.url("unified"), + user = support::USER, + feed = feed.url(), + ); + + let calcalist = Calcalist::new(root.path(), &config); + Some(Fixture { + _root: root, + server, + _feed: feed, + calcalist, + }) +} + +#[test] +fn a_native_cycle_converges_and_the_next_one_does_nothing() { + let Some(fixture) = native_fixture() else { + return; + }; + + let first = fixture.calcalist.run(&["sync"]); + assert!( + first.succeeded(), + "first native sync failed\n{}\n{}", + first.stdout, + first.stderr + ); + + let published = fixture.server.stored("unified"); + assert_eq!( + published.len(), + 2, + "both sources should arrive: {published:?}" + ); + let all = published.join("\n"); + assert!(all.contains("Planning"), "{all}"); + assert!(all.contains("New Year"), "{all}"); + assert!(all.contains("X-CALCALIST-SOURCE:work"), "{all}"); + assert!(all.contains("X-CALCALIST-SOURCE:holidays"), "{all}"); + + let second = fixture.calcalist.run(&["sync"]); + assert!( + second.succeeded(), + "second native sync failed\n{}\n{}", + second.stdout, + second.stderr + ); + assert!( + second.stdout.contains("everything: 0 mirrored"), + "the second cycle should be a no-op:\n{}", + second.stdout + ); + assert_eq!( + fixture.server.stored("unified"), + published, + "the second cycle rewrote the target" + ); +} + +/// A cursor is only worth storing if the cycle after it still sees everything. +/// Radicale answers the first `sync-collection` with no members at all, so a +/// client that trusted it would mirror an empty calendar and call it converged. +#[test] +fn a_native_pull_sees_an_event_added_behind_its_back() { + let Some(fixture) = native_fixture() else { + return; + }; + + assert!(fixture.calcalist.run(&["sync"]).succeeded(), "first sync"); + + fixture.server.put_event("work", "later.ics", LATER); + let second = fixture.calcalist.run(&["sync"]); + assert!( + second.succeeded(), + "second native sync failed\n{}\n{}", + second.stdout, + second.stderr + ); + + let published = fixture.server.stored("unified").join("\n"); + assert!( + published.contains("Retrospective"), + "an event added after the cursor was stored should still arrive:\n{published}" + ); +} + +/// `skip_past` on an aggregate, end to end. +/// +/// The source holds one finished event and one still to come, and the feed holds +/// a New Year that is long gone. Only the future event should be published. +#[test] +fn skip_past_publishes_only_what_is_still_to_come() { + let missing = support::missing_binaries(); + if !missing.is_empty() { + eprintln!( + "skipped: {} not on PATH; run under devbox", + missing.join(", ") + ); + return; + } + + let root = tempfile::tempdir().expect("temp"); + let server = Radicale::start(root.path()); + server.create_calendar("work"); + server.create_calendar("unified"); + server.put_event("work", "meeting.ics", MEETING); + server.put_event("work", "later.ics", LATER); + let feed = Feed::start(HOLIDAY_FEED.to_string()); + + let config = format!( + r#" +version = 1 + +[[endpoint]] +id = "work" +type = "caldav" +native = true +url = "{work}" +username = "{user}" +secret_command = "printf password" + +[[endpoint]] +id = "published" +type = "caldav" +native = true +url = "{unified}" +username = "{user}" +secret_command = "printf password" + +[[endpoint]] +id = "holidays" +type = "webcal" +url = "{feed}" + +[[aggregate]] +id = "everything" +target = "published" +sources = ["work", "holidays"] +default_sink = "work" +skip_past = "2026-09-13" +"#, + work = server.url("work"), + unified = server.url("unified"), + user = support::USER, + feed = feed.url(), + ); + let calcalist = Calcalist::new(root.path(), &config); + + let first = calcalist.run(&["sync"]); + assert!( + first.succeeded(), + "sync failed\n{}\n{}", + first.stdout, + first.stderr + ); + + let published = server.stored("unified"); + let all = published.join("\n"); + assert!( + all.contains("Retrospective"), + "the future event should be published:\n{all}" + ); + assert!( + !all.contains("Planning"), + "the finished meeting should have been left behind:\n{all}" + ); + assert!( + !all.contains("New Year"), + "the finished feed event should have been left behind:\n{all}" + ); + assert_eq!( + published.len(), + 1, + "only one event should arrive: {published:?}" + ); + + // The originals are untouched: excluding history must never reach back into + // a source calendar. + assert_eq!( + server.stored("work").len(), + 2, + "both source events must survive" + ); + + let second = calcalist.run(&["sync"]); + assert!(second.succeeded(), "second sync failed\n{}", second.stderr); + assert!( + second.stdout.contains("everything: 0 mirrored"), + "the second cycle should be a no-op:\n{}", + second.stdout + ); +}