CalCalist/TODO.md

127 lines
7.1 KiB
Markdown
Raw Normal View History

# Roadmap
Milestones from the implementation plan. See SPECS.md for the architecture and the
sync semantics these items implement.
## M0 — skeleton (done)
- [x] Initialise jj colocated with git; `.gitignore` written before the first build
- [x] `devbox.json` pinning Rust 1.97.1, pimsync 0.5.11, jujutsu 0.44.0, radicale 3.7.8
- [x] Configuration model with referential validation, reporting every problem in one pass
- [x] `calcalist doctor` — pimsync presence and version series, state directory, configuration
- [x] Define the full CLI surface; unimplemented commands exit 2 rather than pretend
- [x] SPECS.md: Rust naming conventions, `devbox run check` gate, recorded sync semantics
## M1 — bidirectional sync
Functionally complete. Everything below is done except the two automated
integration tests, which are listed with the remaining gaps at the end.
### Core modules
Add the aggregation engine and a local sync cycle The core of M1: everything needed to reconcile source calendars against an aggregate, short of getting events in and out over the network. - ical: surgical line-level editing. Each logical line keeps a byte range into the original, so untouched lines are emitted verbatim and only edited ones are rebuilt. Parsing and re-serialising would drop every property we do not model. - ical: content hashing excludes DTSTAMP and LAST-MODIFIED. Servers rewrite them on every store, so hashing them would report a change on every cycle forever. - provenance: aggregate UIDs derived as blake3(aggregate, source, source_uid), length-prefixed so field boundaries cannot collide. Deriving rather than recording makes the state file a cache, and makes our own mirrors recognisable, which is what stops writes echoing back around. - mirror: the transforms. An aggregate copy must be scheduling inert, so writing it never mails invitations for a meeting already invited from its source. Google can suppress notification and keeps real attendees; CalDAV cannot, so the guest list is demoted to inert data and a declined meeting is marked TRANSP:TRANSPARENT. Writing an edit back uses the source as donor for what the demotion removed, so editing a time cannot silently drop the guests. - reconcile: pure decision engine. Only the source changed updates the mirror, only the aggregate changed writes back, both changed keeps the source and logs a conflict. - Mass-deletion guard takes an absolute floor as well as a fraction: a share alone is meaningless at small counts, where deleting the only event is 100%. - sync refuses to run when an aggregate's configured target differs from the recorded one, before reconciling. Otherwise the new empty target would read as an aggregate whose every event was deleted, and delete propagation would then remove them from every source. Found by end-to-end testing: writing an item already present under a different filename created a duplicate rather than replacing it, because filenames are derived from the UID while pimsync picks its own. Writes now carry the path they supersede. Covered by a regression test. 79 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 09:55:06 +03:00
- [x] `state.rs` — JSON sidecar, atomic temp + fsync + rename; records each aggregate's
resolved target endpoint id **and** backend type, plus per-Google-endpoint cursors
- [x] `vdir.rs` — read and write vdir directories, keyed by the UID inside each file
rather than by filename, since pimsync names files its own way
Add the aggregation engine and a local sync cycle The core of M1: everything needed to reconcile source calendars against an aggregate, short of getting events in and out over the network. - ical: surgical line-level editing. Each logical line keeps a byte range into the original, so untouched lines are emitted verbatim and only edited ones are rebuilt. Parsing and re-serialising would drop every property we do not model. - ical: content hashing excludes DTSTAMP and LAST-MODIFIED. Servers rewrite them on every store, so hashing them would report a change on every cycle forever. - provenance: aggregate UIDs derived as blake3(aggregate, source, source_uid), length-prefixed so field boundaries cannot collide. Deriving rather than recording makes the state file a cache, and makes our own mirrors recognisable, which is what stops writes echoing back around. - mirror: the transforms. An aggregate copy must be scheduling inert, so writing it never mails invitations for a meeting already invited from its source. Google can suppress notification and keeps real attendees; CalDAV cannot, so the guest list is demoted to inert data and a declined meeting is marked TRANSP:TRANSPARENT. Writing an edit back uses the source as donor for what the demotion removed, so editing a time cannot silently drop the guests. - reconcile: pure decision engine. Only the source changed updates the mirror, only the aggregate changed writes back, both changed keeps the source and logs a conflict. - Mass-deletion guard takes an absolute floor as well as a fraction: a share alone is meaningless at small counts, where deleting the only event is 100%. - sync refuses to run when an aggregate's configured target differs from the recorded one, before reconciling. Otherwise the new empty target would read as an aggregate whose every event was deleted, and delete propagation would then remove them from every source. Found by end-to-end testing: writing an item already present under a different filename created a duplicate rather than replacing it, because filenames are derived from the UID while pimsync picks its own. Writes now carry the path they supersede. Covered by a regression test. 79 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 09:55:06 +03:00
- [x] `ical.rs` — surgical line-level `.ics` editing (UID rewrite, property injection),
respecting RFC 5545 folding; no parse-and-reserialize
Add the aggregation engine and a local sync cycle The core of M1: everything needed to reconcile source calendars against an aggregate, short of getting events in and out over the network. - ical: surgical line-level editing. Each logical line keeps a byte range into the original, so untouched lines are emitted verbatim and only edited ones are rebuilt. Parsing and re-serialising would drop every property we do not model. - ical: content hashing excludes DTSTAMP and LAST-MODIFIED. Servers rewrite them on every store, so hashing them would report a change on every cycle forever. - provenance: aggregate UIDs derived as blake3(aggregate, source, source_uid), length-prefixed so field boundaries cannot collide. Deriving rather than recording makes the state file a cache, and makes our own mirrors recognisable, which is what stops writes echoing back around. - mirror: the transforms. An aggregate copy must be scheduling inert, so writing it never mails invitations for a meeting already invited from its source. Google can suppress notification and keeps real attendees; CalDAV cannot, so the guest list is demoted to inert data and a declined meeting is marked TRANSP:TRANSPARENT. Writing an edit back uses the source as donor for what the demotion removed, so editing a time cannot silently drop the guests. - reconcile: pure decision engine. Only the source changed updates the mirror, only the aggregate changed writes back, both changed keeps the source and logs a conflict. - Mass-deletion guard takes an absolute floor as well as a fraction: a share alone is meaningless at small counts, where deleting the only event is 100%. - sync refuses to run when an aggregate's configured target differs from the recorded one, before reconciling. Otherwise the new empty target would read as an aggregate whose every event was deleted, and delete propagation would then remove them from every source. Found by end-to-end testing: writing an item already present under a different filename created a duplicate rather than replacing it, because filenames are derived from the UID while pimsync picks its own. Writes now carry the path they supersede. Covered by a regression test. 79 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 09:55:06 +03:00
- [x] `provenance.rs` — deterministic `blake3(aggregate_id, source_id, source_uid)` UIDs
- [x] `mirror.rs` — the to-aggregate and to-source transforms (added; not in the
original plan, which folded these into `reconcile`)
- [x] `reconcile.rs` — the aggregation engine; pure, no I/O
- [x] `sync.rs` — one cycle: pull, reconcile, push
- [x] `retarget.rs` — moving an aggregate to a different target, deliberately
Drive pimsync for the CalDAV and WebCal legs Generates pimsync's configuration from calcalist.toml and invokes it one-shot either side of the reconcile step: pull first so the reconciler sees one consistent snapshot, push afterwards to publish what it decided. Its daemon mode stays unused because it would write the same vdirs the reconciler reads. - Endpoints are mapped with pimsync's collection blocks, matching a local vdir subdirectory to an exact remote path. Matching by collection id instead would have tied our directory names to whatever the server's URL segments happen to be, and two calendars ending in the same segment would collide. - The remote is storage_a with conflict_resolution keep a, so a concurrent server-side change wins over a local one calcalist has not yet reconciled. Nothing is lost: the next cycle sees the server's version and decides properly. - on_empty skip and on_delete skip on every pair, so a failed pull is not mirrored as a mass deletion. - Secrets stay out of the portable config: pimsync is given the same command calcalist would run to fetch them. - Google endpoints are omitted entirely, since pimsync can reach neither its REST API nor its CalDAV endpoint. doctor now asks pimsync to validate the generated config. pimsync is pre-1.0 and its parser does not always match its documentation, so asking it is the only reliable check — a parse rejection is reported as a failure, an unreachable server only as a warning. Found while testing against the real binary: pimsync 0.5.11 documents read_only as applying to every storage type but rejects it on a webcal storage, with only "Could not parse file" to go on. Dropped; it was redundant beside one_way. Verified end to end against a live Radicale server and an HTTP-served feed: the feed synced in, mirrored with a derived UID and provenance, and pushed to the server; repeated runs are no-ops and adding one feed event mirrors exactly one. 87 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 10:25:38 +03:00
- [x] `pimsync.rs` — generate `pimsync.conf` (with `on_empty skip` and `on_delete skip`),
drive one-shot `pimsync sync` bracketing the reconcile step
Add Google OAuth, and let feed URLs come from a secret command Google requires OAuth for calendar access; app passwords stopped working for CalDAV, CardDAV and IMAP in March 2025, so there is no simpler path to offer. - Authorisation code flow over a loopback redirect, which is what Google supports for desktop clients now the copy-paste flow is gone, with PKCE so an intercepted code is useless without the verifier. Only the refresh token is persisted, 0600, in the state directory. - An expired grant is reported as itself: a consent screen still in Testing has its refresh tokens expired after 7 days, and "run calcalist google login" is more use than Google's bare invalid_grant. - doctor reports whether each Google endpoint is still authorised, since an installation that worked last week can stop with nothing having changed here. A webcal URL may now come from a command instead of the config. Google's secret iCal address grants read access to a whole calendar to anyone holding it, so writing it into a file described as portable and secret-free was a contradiction. Fixed a serious defect in the first draft of this module: random_token used fs::read on /dev/urandom, which reads to end of file. /dev/urandom has no end, so it allocated until the machine ran out of memory — it took the editor down with it. It now reads exactly 32 bytes, and a randomness failure is fatal rather than falling back to the clock, since a guessable state or PKCE verifier defeats the point of having them. Verified end to end against a live Posteo CalDAV calendar: pimsync validated the generated config against the real server, 58 events from a public feed were mirrored and pushed, and a second run was a no-op. 97 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 12:40:47 +03:00
- [x] `google/auth.rs` — OAuth loopback flow with PKCE, refresh, keyring-sourced secrets
- [x] `google/convert.rs` — iCalendar both ways, including recurrence, timezones and alarms
- [x] `google/api.rs` — incremental pull by syncToken; push by import / update / delete
with notification suppressed
### Behaviour
- [x] **`events.import` gate** — settled from Google's own API discovery document:
`events.import` accepts no `sendUpdates` parameter at all, while `insert`,
`update` and `delete` all do, and it is documented as adding "a private copy of
an existing event". Confirmed live that attendees and alarms survive an import.
Creation goes through `import`; update and delete pass `sendUpdates=none`.
Add the aggregation engine and a local sync cycle The core of M1: everything needed to reconcile source calendars against an aggregate, short of getting events in and out over the network. - ical: surgical line-level editing. Each logical line keeps a byte range into the original, so untouched lines are emitted verbatim and only edited ones are rebuilt. Parsing and re-serialising would drop every property we do not model. - ical: content hashing excludes DTSTAMP and LAST-MODIFIED. Servers rewrite them on every store, so hashing them would report a change on every cycle forever. - provenance: aggregate UIDs derived as blake3(aggregate, source, source_uid), length-prefixed so field boundaries cannot collide. Deriving rather than recording makes the state file a cache, and makes our own mirrors recognisable, which is what stops writes echoing back around. - mirror: the transforms. An aggregate copy must be scheduling inert, so writing it never mails invitations for a meeting already invited from its source. Google can suppress notification and keeps real attendees; CalDAV cannot, so the guest list is demoted to inert data and a declined meeting is marked TRANSP:TRANSPARENT. Writing an edit back uses the source as donor for what the demotion removed, so editing a time cannot silently drop the guests. - reconcile: pure decision engine. Only the source changed updates the mirror, only the aggregate changed writes back, both changed keeps the source and logs a conflict. - Mass-deletion guard takes an absolute floor as well as a fraction: a share alone is meaningless at small counts, where deleting the only event is 100%. - sync refuses to run when an aggregate's configured target differs from the recorded one, before reconciling. Otherwise the new empty target would read as an aggregate whose every event was deleted, and delete propagation would then remove them from every source. Found by end-to-end testing: writing an item already present under a different filename created a duplicate rather than replacing it, because filenames are derived from the UID while pimsync picks its own. Writes now carry the path they supersede. Covered by a regression test. 79 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 09:55:06 +03:00
- [x] `sync` refuses to run on aggregate target drift, before reconciliation
Add aggregate retarget Moving an aggregate to a different endpoint is the sharpest failure mode in the design, so it is a command rather than something a config edit triggers: sync already refuses on target drift and points here. The move settles against the old target first. An event a user created in the aggregate and that has not yet reached a sink exists only there, and would be lost the moment we stop looking at that calendar. Only then is every mapped event rebuilt on the new target — a rewrite rather than a copy, since the rendered content differs between backends that can and cannot suppress scheduling. Orphans on the old target are kept unless --purge-old, and a purge is bounded by the derivation, so events the user keeps in that calendar themselves survive. Also fixed prepare_vdirs, which created a directory per endpoint. That told pimsync a local collection existed before its remote counterpart had been seen, so it tried to create the counterpart: unsupported for a read-only feed, and it would have invented calendars on a CalDAV server. Only the vdir root is created now; collections are pimsync's to make from what it discovers. One test premise was wrong rather than the code: a foreign event in an aggregate that has a sink is a user-created event and gets adopted into that sink, so it never reaches a purge as an outsider. Both behaviours are now covered. Verified against the live Posteo calendar after the change: still a clean no-op. 106 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 13:04:41 +03:00
- [x] `aggregate retarget` — flush unrouted creations against the old target, then
re-materialise; keep old orphans by default, purge bounded by the derivation
Add the aggregation engine and a local sync cycle The core of M1: everything needed to reconcile source calendars against an aggregate, short of getting events in and out over the network. - ical: surgical line-level editing. Each logical line keeps a byte range into the original, so untouched lines are emitted verbatim and only edited ones are rebuilt. Parsing and re-serialising would drop every property we do not model. - ical: content hashing excludes DTSTAMP and LAST-MODIFIED. Servers rewrite them on every store, so hashing them would report a change on every cycle forever. - provenance: aggregate UIDs derived as blake3(aggregate, source, source_uid), length-prefixed so field boundaries cannot collide. Deriving rather than recording makes the state file a cache, and makes our own mirrors recognisable, which is what stops writes echoing back around. - mirror: the transforms. An aggregate copy must be scheduling inert, so writing it never mails invitations for a meeting already invited from its source. Google can suppress notification and keeps real attendees; CalDAV cannot, so the guest list is demoted to inert data and a declined meeting is marked TRANSP:TRANSPARENT. Writing an edit back uses the source as donor for what the demotion removed, so editing a time cannot silently drop the guests. - reconcile: pure decision engine. Only the source changed updates the mirror, only the aggregate changed writes back, both changed keeps the source and logs a conflict. - Mass-deletion guard takes an absolute floor as well as a fraction: a share alone is meaningless at small counts, where deleting the only event is 100%. - sync refuses to run when an aggregate's configured target differs from the recorded one, before reconciling. Otherwise the new empty target would read as an aggregate whose every event was deleted, and delete propagation would then remove them from every source. Found by end-to-end testing: writing an item already present under a different filename created a duplicate rather than replacing it, because filenames are derived from the UID while pimsync picks its own. Writes now carry the path they supersede. Covered by a regression test. 79 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 09:55:06 +03:00
- [x] Mass-deletion guard (`max_delete_fraction`), overridable with `--force`, with an
absolute floor so deleting a couple of events is never refused
- [x] Echo suppression: derived UIDs are never re-ingested as source events
- [x] Routing: a `@endpoint-id` line in the description, or a matching category, picks
which source a new aggregate event is filed under; a hint naming an invalid sink
is refused rather than redirected to the default
- [x] Alarms map both ways, including Google's minutes-before reminder model
- [x] `webcal` URLs may come from a command, for feeds whose address is itself a
credential — Google's secret iCal address being the case in point
- [x] `doctor` validates the generated pimsync config and each Google authorisation
### Tests
- [x] `reconcile` table-driven cases: create/update/delete each direction,
both-sides-changed, routing, echo suppression, mass-delete abort
Add the aggregation engine and a local sync cycle The core of M1: everything needed to reconcile source calendars against an aggregate, short of getting events in and out over the network. - ical: surgical line-level editing. Each logical line keeps a byte range into the original, so untouched lines are emitted verbatim and only edited ones are rebuilt. Parsing and re-serialising would drop every property we do not model. - ical: content hashing excludes DTSTAMP and LAST-MODIFIED. Servers rewrite them on every store, so hashing them would report a change on every cycle forever. - provenance: aggregate UIDs derived as blake3(aggregate, source, source_uid), length-prefixed so field boundaries cannot collide. Deriving rather than recording makes the state file a cache, and makes our own mirrors recognisable, which is what stops writes echoing back around. - mirror: the transforms. An aggregate copy must be scheduling inert, so writing it never mails invitations for a meeting already invited from its source. Google can suppress notification and keeps real attendees; CalDAV cannot, so the guest list is demoted to inert data and a declined meeting is marked TRANSP:TRANSPARENT. Writing an edit back uses the source as donor for what the demotion removed, so editing a time cannot silently drop the guests. - reconcile: pure decision engine. Only the source changed updates the mirror, only the aggregate changed writes back, both changed keeps the source and logs a conflict. - Mass-deletion guard takes an absolute floor as well as a fraction: a share alone is meaningless at small counts, where deleting the only event is 100%. - sync refuses to run when an aggregate's configured target differs from the recorded one, before reconciling. Otherwise the new empty target would read as an aggregate whose every event was deleted, and delete propagation would then remove them from every source. Found by end-to-end testing: writing an item already present under a different filename created a duplicate rather than replacing it, because filenames are derived from the UID while pimsync picks its own. Writes now carry the path they supersede. Covered by a regression test. 79 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 09:55:06 +03:00
- [x] `ical` round-trip fixtures: recurring with overrides, all-day, TZID, unknown `X-` props
- [x] `convert` against real captured API payloads, including the case where Google's
offset and its `timeZone` disagree
- [x] Topologies: a source feeding several aggregates, several aggregates sharing one
target, a cycle between two aggregates, a delete cascading across aggregates,
and competing edits arriving through two aggregates at once
- [x] Retarget: drift makes `sync` exit non-zero having written nothing and losing no
source event; an unrouted creation reaches a sink first
- [x] Safety (unit): no live `ATTENDEE`/`ORGANIZER` on a CalDAV-targeted mirror,
`VALARM` intact, `PARTSTAT: DECLINED` maps to `TRANSP: TRANSPARENT`
Let an event choose which source it is filed under An event created in an aggregate went to the configured default_sink and nowhere else, so with several writable sources there was no way to say which calendar a new event belonged in. A line reading @endpoint-id in the description now picks the source, and a matching CATEGORIES value does too. The description rather than the title because every calendar client exposes a notes field and editing it does not disfigure the event's name; CATEGORIES as well because that is the field iCalendar intends, even though many mobile clients hide it. The marker is stripped before the event reaches the calendar, being calcalist's bookkeeping rather than content. A marker naming something that is not a writable source of that aggregate is refused and reported, not redirected to the default: a typo should not quietly file an event in the wrong calendar. A bare address in prose is not a marker either, since a marker must be a line of its own. Also covers the shapes beyond many-into-one: a source feeding several aggregates, several aggregates sharing one target, a cycle between two aggregates, a delete cascading across aggregates, and competing edits arriving through two aggregates at once — the last being caught by the existing conflict detection rather than silently overwriting. Error display no longer repeats itself; thiserror already prints the cause chain. Verified live against real accounts: two Google calendars aggregating into a Posteo calendar, an edit in the aggregate reaching the originating Google calendar, an event routed to a chosen source by its description marker, and a deletion propagating from the aggregate through to Google. 129 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 13:29:02 +03:00
### Verified live, end to end
Against real accounts — two Google calendars aggregating into a Posteo CalDAV
calendar:
- Fan-in from both Google sources, each mirror tagged with its origin
- A public iCal feed synced through pimsync and pushed to CalDAV; idempotent across
repeated runs and correctly incremental
- An edit made in the aggregate reaching the originating Google calendar, and the
revert travelling the same way
- An event created in the aggregate routed to a chosen source by its description
marker, with the marker stripped before it arrived
- A deletion in the aggregate propagating through to Google
- The mass-deletion guard refusing a 100% removal until `--force`
- `pimsync check` validating the generated config against a live CalDAV server
## Remaining
### Tests not yet automated
- [ ] Integration against Radicale plus a WebCal fixture, asserting convergence and
idempotence. Done by hand twice; not yet a test that runs in CI.
- [ ] Safety integration: a real CalDAV server with an SMTP sink, proving no mail is
emitted on mirror writes or mirror deletions.
### Known gaps
- [ ] A recurring series' *exceptions* are not pushed to Google. Pulling them works.
Google models them as separate events against an already existing series, so
pushing needs `events.instances` plus a patch per exception. Reported per sync
rather than dropped silently.
- [ ] Each Google endpoint needs its own `google login`, even for the same account and
OAuth client, because tokens are keyed by endpoint id. Two endpoints on one
account should share a credential.
- [ ] A failed Google pull aborts the whole cycle, including the CalDAV side. Safe —
reconciling against a stale snapshot could read as mass deletion — but a lapsed
token stops everything. Skipping only the affected aggregates would be better.
- [ ] Removing an endpoint from the config leaves its vdir behind, holding events
nothing manages any more.
- [ ] `--dry-run` skips the pull entirely, so it reports against whatever the last
real cycle left behind rather than against current remote state.
- [ ] A `TZID` is emitted without an accompanying `VTIMEZONE`. Tolerated by the servers
tested so far, and confined to recurring events, but not strictly conformant.
## M2 — interface and packaging
- [ ] axum configuration UI, bound to 127.0.0.1
- [ ] Trigger `google login` from the UI (the loopback handler itself already exists
in `google/auth.rs`)
- [ ] systemd user units: `calcalist.service` (oneshot) and `calcalist.timer`