2026-09-10 09:34:42 +03:00
|
|
|
# Roadmap
|
|
|
|
|
|
|
|
|
|
Milestones from the implementation plan. See SPECS.md for the architecture and the
|
|
|
|
|
sync semantics these items implement.
|
|
|
|
|
|
|
|
|
|
## M0 — skeleton (done)
|
|
|
|
|
|
|
|
|
|
- [x] Initialise jj colocated with git; `.gitignore` written before the first build
|
|
|
|
|
- [x] `devbox.json` pinning Rust 1.97.1, pimsync 0.5.11, jujutsu 0.44.0, radicale 3.7.8
|
|
|
|
|
- [x] Configuration model with referential validation, reporting every problem in one pass
|
|
|
|
|
- [x] `calcalist doctor` — pimsync presence and version series, state directory, configuration
|
|
|
|
|
- [x] Define the full CLI surface; unimplemented commands exit 2 rather than pretend
|
|
|
|
|
- [x] SPECS.md: Rust naming conventions, `devbox run check` gate, recorded sync semantics
|
|
|
|
|
|
|
|
|
|
## M1 — bidirectional sync
|
|
|
|
|
|
Close the remaining M1 gaps
Eight items were still open at the end of M1: two integration tests that had
only been run by hand, and six known gaps.
Recurrence overrides now reach Google. Google addresses an exception through
the series rather than as an event of its own, so the master is sent first and
each override is then matched to its instance by original start time and
patched. Matching needs the two sides' spellings reduced to one key: iCalendar
writes a zoned local time, Google an absolute offset. An override matching no
occurrence is counted rather than forced — that means a stale RECURRENCE-ID
left behind by an edited RRULE, and inventing an event for it would put
something in the calendar the series does not contain.
Reading one component apart from another needed a view `properties` cannot
give: it flattens every VEVENT together, which is right for the UID a series
shares and wrong for an override, whose SUMMARY and the master's are then
indistinguishable. `Calendar::events` splits them.
A TZID now travels with the VTIMEZONE that defines it, derived from the zone's
own transition table as the yearly rule it implies. This changes the content
hash of every zoned recurring event, so the first cycle after this re-pushes
them.
A Google authorisation is filed under the account it was granted for rather
than the endpoint that asked for it, so two endpoints on one account no longer
need a login each. The account is read from the primary calendar's id, which
needs no scope beyond the calendar one already granted. Authorisations written
by the previous scheme are still honoured, and move across at the next login.
An unreachable Google endpoint no longer ends the cycle — one lapsed token used
to stop the CalDAV side too. It is named, only the aggregates depending on it
stand down, and the run exits non-zero so a partial cycle cannot pass for
success. The CalDAV leg cannot be narrowed the same way: pimsync is one process
covering every pair, so a failure does not say which pair it belongs to.
`--dry-run` now pulls for real, into a throwaway copy of the local mirrors and
through a pimsync configuration that only ever reads from a server. What it
reports is measured against the calendars as they are now rather than against
whatever the last real cycle left behind.
`calcalist prune` reports local mirrors of endpoints the configuration no
longer names, and removes them under --force.
The integration tests run against a real Radicale server and a real iCal feed:
convergence and idempotence, the dry run, prune, and the scheduling rule
asserted on the bytes that actually reached the server. The plan asked for an
SMTP sink for that last one; Radicale implements no RFC 6638 scheduling, so a
quiet SMTP port would have proved nothing about the transform.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 13:49:38 +03:00
|
|
|
Complete. Every item from the plan is built, tested and verified live.
|
2026-09-10 13:39:54 +03:00
|
|
|
|
|
|
|
|
### Core modules
|
2026-09-10 09:34:42 +03:00
|
|
|
|
Add the aggregation engine and a local sync cycle
The core of M1: everything needed to reconcile source calendars against an
aggregate, short of getting events in and out over the network.
- ical: surgical line-level editing. Each logical line keeps a byte range into
the original, so untouched lines are emitted verbatim and only edited ones are
rebuilt. Parsing and re-serialising would drop every property we do not model.
- ical: content hashing excludes DTSTAMP and LAST-MODIFIED. Servers rewrite them
on every store, so hashing them would report a change on every cycle forever.
- provenance: aggregate UIDs derived as blake3(aggregate, source, source_uid),
length-prefixed so field boundaries cannot collide. Deriving rather than
recording makes the state file a cache, and makes our own mirrors recognisable,
which is what stops writes echoing back around.
- mirror: the transforms. An aggregate copy must be scheduling inert, so writing
it never mails invitations for a meeting already invited from its source.
Google can suppress notification and keeps real attendees; CalDAV cannot, so
the guest list is demoted to inert data and a declined meeting is marked
TRANSP:TRANSPARENT. Writing an edit back uses the source as donor for what the
demotion removed, so editing a time cannot silently drop the guests.
- reconcile: pure decision engine. Only the source changed updates the mirror,
only the aggregate changed writes back, both changed keeps the source and logs
a conflict.
- Mass-deletion guard takes an absolute floor as well as a fraction: a share
alone is meaningless at small counts, where deleting the only event is 100%.
- sync refuses to run when an aggregate's configured target differs from the
recorded one, before reconciling. Otherwise the new empty target would read as
an aggregate whose every event was deleted, and delete propagation would then
remove them from every source.
Found by end-to-end testing: writing an item already present under a different
filename created a duplicate rather than replacing it, because filenames are
derived from the UID while pimsync picks its own. Writes now carry the path they
supersede. Covered by a regression test.
79 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 09:55:06 +03:00
|
|
|
- [x] `state.rs` — JSON sidecar, atomic temp + fsync + rename; records each aggregate's
|
2026-09-10 13:39:54 +03:00
|
|
|
resolved target endpoint id **and** backend type, plus per-Google-endpoint cursors
|
|
|
|
|
- [x] `vdir.rs` — read and write vdir directories, keyed by the UID inside each file
|
|
|
|
|
rather than by filename, since pimsync names files its own way
|
Add the aggregation engine and a local sync cycle
The core of M1: everything needed to reconcile source calendars against an
aggregate, short of getting events in and out over the network.
- ical: surgical line-level editing. Each logical line keeps a byte range into
the original, so untouched lines are emitted verbatim and only edited ones are
rebuilt. Parsing and re-serialising would drop every property we do not model.
- ical: content hashing excludes DTSTAMP and LAST-MODIFIED. Servers rewrite them
on every store, so hashing them would report a change on every cycle forever.
- provenance: aggregate UIDs derived as blake3(aggregate, source, source_uid),
length-prefixed so field boundaries cannot collide. Deriving rather than
recording makes the state file a cache, and makes our own mirrors recognisable,
which is what stops writes echoing back around.
- mirror: the transforms. An aggregate copy must be scheduling inert, so writing
it never mails invitations for a meeting already invited from its source.
Google can suppress notification and keeps real attendees; CalDAV cannot, so
the guest list is demoted to inert data and a declined meeting is marked
TRANSP:TRANSPARENT. Writing an edit back uses the source as donor for what the
demotion removed, so editing a time cannot silently drop the guests.
- reconcile: pure decision engine. Only the source changed updates the mirror,
only the aggregate changed writes back, both changed keeps the source and logs
a conflict.
- Mass-deletion guard takes an absolute floor as well as a fraction: a share
alone is meaningless at small counts, where deleting the only event is 100%.
- sync refuses to run when an aggregate's configured target differs from the
recorded one, before reconciling. Otherwise the new empty target would read as
an aggregate whose every event was deleted, and delete propagation would then
remove them from every source.
Found by end-to-end testing: writing an item already present under a different
filename created a duplicate rather than replacing it, because filenames are
derived from the UID while pimsync picks its own. Writes now carry the path they
supersede. Covered by a regression test.
79 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 09:55:06 +03:00
|
|
|
- [x] `ical.rs` — surgical line-level `.ics` editing (UID rewrite, property injection),
|
2026-09-10 09:34:42 +03:00
|
|
|
respecting RFC 5545 folding; no parse-and-reserialize
|
Add the aggregation engine and a local sync cycle
The core of M1: everything needed to reconcile source calendars against an
aggregate, short of getting events in and out over the network.
- ical: surgical line-level editing. Each logical line keeps a byte range into
the original, so untouched lines are emitted verbatim and only edited ones are
rebuilt. Parsing and re-serialising would drop every property we do not model.
- ical: content hashing excludes DTSTAMP and LAST-MODIFIED. Servers rewrite them
on every store, so hashing them would report a change on every cycle forever.
- provenance: aggregate UIDs derived as blake3(aggregate, source, source_uid),
length-prefixed so field boundaries cannot collide. Deriving rather than
recording makes the state file a cache, and makes our own mirrors recognisable,
which is what stops writes echoing back around.
- mirror: the transforms. An aggregate copy must be scheduling inert, so writing
it never mails invitations for a meeting already invited from its source.
Google can suppress notification and keeps real attendees; CalDAV cannot, so
the guest list is demoted to inert data and a declined meeting is marked
TRANSP:TRANSPARENT. Writing an edit back uses the source as donor for what the
demotion removed, so editing a time cannot silently drop the guests.
- reconcile: pure decision engine. Only the source changed updates the mirror,
only the aggregate changed writes back, both changed keeps the source and logs
a conflict.
- Mass-deletion guard takes an absolute floor as well as a fraction: a share
alone is meaningless at small counts, where deleting the only event is 100%.
- sync refuses to run when an aggregate's configured target differs from the
recorded one, before reconciling. Otherwise the new empty target would read as
an aggregate whose every event was deleted, and delete propagation would then
remove them from every source.
Found by end-to-end testing: writing an item already present under a different
filename created a duplicate rather than replacing it, because filenames are
derived from the UID while pimsync picks its own. Writes now carry the path they
supersede. Covered by a regression test.
79 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 09:55:06 +03:00
|
|
|
- [x] `provenance.rs` — deterministic `blake3(aggregate_id, source_id, source_uid)` UIDs
|
|
|
|
|
- [x] `mirror.rs` — the to-aggregate and to-source transforms (added; not in the
|
|
|
|
|
original plan, which folded these into `reconcile`)
|
|
|
|
|
- [x] `reconcile.rs` — the aggregation engine; pure, no I/O
|
2026-09-10 13:39:54 +03:00
|
|
|
- [x] `sync.rs` — one cycle: pull, reconcile, push
|
|
|
|
|
- [x] `retarget.rs` — moving an aggregate to a different target, deliberately
|
Drive pimsync for the CalDAV and WebCal legs
Generates pimsync's configuration from calcalist.toml and invokes it one-shot
either side of the reconcile step: pull first so the reconciler sees one
consistent snapshot, push afterwards to publish what it decided. Its daemon
mode stays unused because it would write the same vdirs the reconciler reads.
- Endpoints are mapped with pimsync's collection blocks, matching a local vdir
subdirectory to an exact remote path. Matching by collection id instead would
have tied our directory names to whatever the server's URL segments happen to
be, and two calendars ending in the same segment would collide.
- The remote is storage_a with conflict_resolution keep a, so a concurrent
server-side change wins over a local one calcalist has not yet reconciled.
Nothing is lost: the next cycle sees the server's version and decides properly.
- on_empty skip and on_delete skip on every pair, so a failed pull is not
mirrored as a mass deletion.
- Secrets stay out of the portable config: pimsync is given the same command
calcalist would run to fetch them.
- Google endpoints are omitted entirely, since pimsync can reach neither its REST
API nor its CalDAV endpoint.
doctor now asks pimsync to validate the generated config. pimsync is pre-1.0 and
its parser does not always match its documentation, so asking it is the only
reliable check — a parse rejection is reported as a failure, an unreachable
server only as a warning.
Found while testing against the real binary: pimsync 0.5.11 documents read_only
as applying to every storage type but rejects it on a webcal storage, with only
"Could not parse file" to go on. Dropped; it was redundant beside one_way.
Verified end to end against a live Radicale server and an HTTP-served feed: the
feed synced in, mirrored with a derived UID and provenance, and pushed to the
server; repeated runs are no-ops and adding one feed event mirrors exactly one.
87 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 10:25:38 +03:00
|
|
|
- [x] `pimsync.rs` — generate `pimsync.conf` (with `on_empty skip` and `on_delete skip`),
|
|
|
|
|
drive one-shot `pimsync sync` bracketing the reconcile step
|
Add Google OAuth, and let feed URLs come from a secret command
Google requires OAuth for calendar access; app passwords stopped working for
CalDAV, CardDAV and IMAP in March 2025, so there is no simpler path to offer.
- Authorisation code flow over a loopback redirect, which is what Google
supports for desktop clients now the copy-paste flow is gone, with PKCE so an
intercepted code is useless without the verifier. Only the refresh token is
persisted, 0600, in the state directory.
- An expired grant is reported as itself: a consent screen still in Testing has
its refresh tokens expired after 7 days, and "run calcalist google login" is
more use than Google's bare invalid_grant.
- doctor reports whether each Google endpoint is still authorised, since an
installation that worked last week can stop with nothing having changed here.
A webcal URL may now come from a command instead of the config. Google's secret
iCal address grants read access to a whole calendar to anyone holding it, so
writing it into a file described as portable and secret-free was a contradiction.
Fixed a serious defect in the first draft of this module: random_token used
fs::read on /dev/urandom, which reads to end of file. /dev/urandom has no end,
so it allocated until the machine ran out of memory — it took the editor down
with it. It now reads exactly 32 bytes, and a randomness failure is fatal rather
than falling back to the clock, since a guessable state or PKCE verifier defeats
the point of having them.
Verified end to end against a live Posteo CalDAV calendar: pimsync validated the
generated config against the real server, 58 events from a public feed were
mirrored and pushed, and a second run was a no-op.
97 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 12:40:47 +03:00
|
|
|
- [x] `google/auth.rs` — OAuth loopback flow with PKCE, refresh, keyring-sourced secrets
|
2026-09-10 13:39:54 +03:00
|
|
|
- [x] `google/convert.rs` — iCalendar both ways, including recurrence, timezones and alarms
|
|
|
|
|
- [x] `google/api.rs` — incremental pull by syncToken; push by import / update / delete
|
|
|
|
|
with notification suppressed
|
2026-09-10 09:34:42 +03:00
|
|
|
|
2026-09-10 13:39:54 +03:00
|
|
|
### Behaviour
|
2026-09-10 09:34:42 +03:00
|
|
|
|
Push local changes up to Google, completing the Google leg
Mirrors are created with events.import, updated and deleted with
sendUpdates=none, so a write to an aggregate cannot mail anyone about a meeting
that was already invited from its source.
The import question the plan flagged is settled from Google's own API discovery
document rather than by guesswork: events.import accepts no sendUpdates parameter
at all, while insert, update and delete all do, and it is documented as adding "a
private copy of an existing event". A method with notification behaviour would
need that control. Confirmed live that attendees survive an import with their
response statuses intact.
Testing against the real API caught a defect that unit tests could not: the first
push dropped VALARM entirely and Google substituted the calendar's default
reminders. Alarms are never stripped by decision, so they now map to and from
Google's reminder overrides in both directions, with relative TRIGGER durations
converted to whole minutes. A trigger Google cannot express — absolute, or after
the start — is dropped rather than guessed at.
Known gaps recorded in TODO.md rather than papered over: a series' exceptions are
not pushed, since Google models those as separate events against an existing
series; and a failed Google pull still aborts the cycle, which is safe but stops
the CalDAV side too.
121 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 13:21:01 +03:00
|
|
|
- [x] **`events.import` gate** — settled from Google's own API discovery document:
|
|
|
|
|
`events.import` accepts no `sendUpdates` parameter at all, while `insert`,
|
|
|
|
|
`update` and `delete` all do, and it is documented as adding "a private copy of
|
|
|
|
|
an existing event". Confirmed live that attendees and alarms survive an import.
|
|
|
|
|
Creation goes through `import`; update and delete pass `sendUpdates=none`.
|
Add the aggregation engine and a local sync cycle
The core of M1: everything needed to reconcile source calendars against an
aggregate, short of getting events in and out over the network.
- ical: surgical line-level editing. Each logical line keeps a byte range into
the original, so untouched lines are emitted verbatim and only edited ones are
rebuilt. Parsing and re-serialising would drop every property we do not model.
- ical: content hashing excludes DTSTAMP and LAST-MODIFIED. Servers rewrite them
on every store, so hashing them would report a change on every cycle forever.
- provenance: aggregate UIDs derived as blake3(aggregate, source, source_uid),
length-prefixed so field boundaries cannot collide. Deriving rather than
recording makes the state file a cache, and makes our own mirrors recognisable,
which is what stops writes echoing back around.
- mirror: the transforms. An aggregate copy must be scheduling inert, so writing
it never mails invitations for a meeting already invited from its source.
Google can suppress notification and keeps real attendees; CalDAV cannot, so
the guest list is demoted to inert data and a declined meeting is marked
TRANSP:TRANSPARENT. Writing an edit back uses the source as donor for what the
demotion removed, so editing a time cannot silently drop the guests.
- reconcile: pure decision engine. Only the source changed updates the mirror,
only the aggregate changed writes back, both changed keeps the source and logs
a conflict.
- Mass-deletion guard takes an absolute floor as well as a fraction: a share
alone is meaningless at small counts, where deleting the only event is 100%.
- sync refuses to run when an aggregate's configured target differs from the
recorded one, before reconciling. Otherwise the new empty target would read as
an aggregate whose every event was deleted, and delete propagation would then
remove them from every source.
Found by end-to-end testing: writing an item already present under a different
filename created a duplicate rather than replacing it, because filenames are
derived from the UID while pimsync picks its own. Writes now carry the path they
supersede. Covered by a regression test.
79 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 09:55:06 +03:00
|
|
|
- [x] `sync` refuses to run on aggregate target drift, before reconciliation
|
2026-09-10 13:04:41 +03:00
|
|
|
- [x] `aggregate retarget` — flush unrouted creations against the old target, then
|
2026-09-10 13:39:54 +03:00
|
|
|
re-materialise; keep old orphans by default, purge bounded by the derivation
|
Add the aggregation engine and a local sync cycle
The core of M1: everything needed to reconcile source calendars against an
aggregate, short of getting events in and out over the network.
- ical: surgical line-level editing. Each logical line keeps a byte range into
the original, so untouched lines are emitted verbatim and only edited ones are
rebuilt. Parsing and re-serialising would drop every property we do not model.
- ical: content hashing excludes DTSTAMP and LAST-MODIFIED. Servers rewrite them
on every store, so hashing them would report a change on every cycle forever.
- provenance: aggregate UIDs derived as blake3(aggregate, source, source_uid),
length-prefixed so field boundaries cannot collide. Deriving rather than
recording makes the state file a cache, and makes our own mirrors recognisable,
which is what stops writes echoing back around.
- mirror: the transforms. An aggregate copy must be scheduling inert, so writing
it never mails invitations for a meeting already invited from its source.
Google can suppress notification and keeps real attendees; CalDAV cannot, so
the guest list is demoted to inert data and a declined meeting is marked
TRANSP:TRANSPARENT. Writing an edit back uses the source as donor for what the
demotion removed, so editing a time cannot silently drop the guests.
- reconcile: pure decision engine. Only the source changed updates the mirror,
only the aggregate changed writes back, both changed keeps the source and logs
a conflict.
- Mass-deletion guard takes an absolute floor as well as a fraction: a share
alone is meaningless at small counts, where deleting the only event is 100%.
- sync refuses to run when an aggregate's configured target differs from the
recorded one, before reconciling. Otherwise the new empty target would read as
an aggregate whose every event was deleted, and delete propagation would then
remove them from every source.
Found by end-to-end testing: writing an item already present under a different
filename created a duplicate rather than replacing it, because filenames are
derived from the UID while pimsync picks its own. Writes now carry the path they
supersede. Covered by a regression test.
79 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 09:55:06 +03:00
|
|
|
- [x] Mass-deletion guard (`max_delete_fraction`), overridable with `--force`, with an
|
|
|
|
|
absolute floor so deleting a couple of events is never refused
|
|
|
|
|
- [x] Echo suppression: derived UIDs are never re-ingested as source events
|
2026-09-10 13:39:54 +03:00
|
|
|
- [x] Routing: a `@endpoint-id` line in the description, or a matching category, picks
|
|
|
|
|
which source a new aggregate event is filed under; a hint naming an invalid sink
|
|
|
|
|
is refused rather than redirected to the default
|
|
|
|
|
- [x] Alarms map both ways, including Google's minutes-before reminder model
|
|
|
|
|
- [x] `webcal` URLs may come from a command, for feeds whose address is itself a
|
|
|
|
|
credential — Google's secret iCal address being the case in point
|
|
|
|
|
- [x] `doctor` validates the generated pimsync config and each Google authorisation
|
2026-09-10 13:40:27 +03:00
|
|
|
- [x] `calcalist status` — endpoints, aggregates, how many events are mirrored, and the
|
|
|
|
|
exact `@marker` names an event can carry, so routing is discoverable without
|
|
|
|
|
opening the config file
|
Close the remaining M1 gaps
Eight items were still open at the end of M1: two integration tests that had
only been run by hand, and six known gaps.
Recurrence overrides now reach Google. Google addresses an exception through
the series rather than as an event of its own, so the master is sent first and
each override is then matched to its instance by original start time and
patched. Matching needs the two sides' spellings reduced to one key: iCalendar
writes a zoned local time, Google an absolute offset. An override matching no
occurrence is counted rather than forced — that means a stale RECURRENCE-ID
left behind by an edited RRULE, and inventing an event for it would put
something in the calendar the series does not contain.
Reading one component apart from another needed a view `properties` cannot
give: it flattens every VEVENT together, which is right for the UID a series
shares and wrong for an override, whose SUMMARY and the master's are then
indistinguishable. `Calendar::events` splits them.
A TZID now travels with the VTIMEZONE that defines it, derived from the zone's
own transition table as the yearly rule it implies. This changes the content
hash of every zoned recurring event, so the first cycle after this re-pushes
them.
A Google authorisation is filed under the account it was granted for rather
than the endpoint that asked for it, so two endpoints on one account no longer
need a login each. The account is read from the primary calendar's id, which
needs no scope beyond the calendar one already granted. Authorisations written
by the previous scheme are still honoured, and move across at the next login.
An unreachable Google endpoint no longer ends the cycle — one lapsed token used
to stop the CalDAV side too. It is named, only the aggregates depending on it
stand down, and the run exits non-zero so a partial cycle cannot pass for
success. The CalDAV leg cannot be narrowed the same way: pimsync is one process
covering every pair, so a failure does not say which pair it belongs to.
`--dry-run` now pulls for real, into a throwaway copy of the local mirrors and
through a pimsync configuration that only ever reads from a server. What it
reports is measured against the calendars as they are now rather than against
whatever the last real cycle left behind.
`calcalist prune` reports local mirrors of endpoints the configuration no
longer names, and removes them under --force.
The integration tests run against a real Radicale server and a real iCal feed:
convergence and idempotence, the dry run, prune, and the scheduling rule
asserted on the bytes that actually reached the server. The plan asked for an
SMTP sink for that last one; Radicale implements no RFC 6638 scheduling, so a
quiet SMTP port would have proved nothing about the transform.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 13:49:38 +03:00
|
|
|
- [x] Recurrence overrides are pushed to Google. Google addresses an exception
|
|
|
|
|
through the series rather than as an event of its own, so the master goes up
|
|
|
|
|
first and each override is then matched to its instance by original start
|
|
|
|
|
time and patched. An override matching no occurrence is counted, not forced.
|
|
|
|
|
- [x] A `TZID` now travels with the `VTIMEZONE` that defines it, derived from the
|
|
|
|
|
zone's own transition table as the yearly rule it implies. A zone that does
|
|
|
|
|
not shift gets one fixed observance.
|
|
|
|
|
- [x] One Google authorisation covers every endpoint on that account. Tokens are
|
|
|
|
|
keyed by the account, discovered at login from the primary calendar's id —
|
|
|
|
|
which needs no scope beyond the calendar one already granted. An endpoint
|
|
|
|
|
names its `account` only when calcalist is logged in to more than one.
|
|
|
|
|
- [x] A Google endpoint that cannot be reached no longer ends the cycle. It is
|
|
|
|
|
named, only the aggregates depending on it stand down, and the run exits
|
|
|
|
|
non-zero so a lapsed token cannot pass for success.
|
|
|
|
|
- [x] `calcalist prune` — reports local mirrors of endpoints the configuration no
|
|
|
|
|
longer names, and removes them under `--force`
|
|
|
|
|
- [x] `--dry-run` pulls for real, into a throwaway copy of the local mirrors and
|
|
|
|
|
through a pimsync configuration that only ever reads from a server. What it
|
|
|
|
|
reports is measured against the calendars as they are now, and nothing
|
|
|
|
|
outside the copy is written.
|
2026-09-10 13:39:54 +03:00
|
|
|
|
|
|
|
|
### Tests
|
|
|
|
|
|
|
|
|
|
- [x] `reconcile` table-driven cases: create/update/delete each direction,
|
|
|
|
|
both-sides-changed, routing, echo suppression, mass-delete abort
|
Add the aggregation engine and a local sync cycle
The core of M1: everything needed to reconcile source calendars against an
aggregate, short of getting events in and out over the network.
- ical: surgical line-level editing. Each logical line keeps a byte range into
the original, so untouched lines are emitted verbatim and only edited ones are
rebuilt. Parsing and re-serialising would drop every property we do not model.
- ical: content hashing excludes DTSTAMP and LAST-MODIFIED. Servers rewrite them
on every store, so hashing them would report a change on every cycle forever.
- provenance: aggregate UIDs derived as blake3(aggregate, source, source_uid),
length-prefixed so field boundaries cannot collide. Deriving rather than
recording makes the state file a cache, and makes our own mirrors recognisable,
which is what stops writes echoing back around.
- mirror: the transforms. An aggregate copy must be scheduling inert, so writing
it never mails invitations for a meeting already invited from its source.
Google can suppress notification and keeps real attendees; CalDAV cannot, so
the guest list is demoted to inert data and a declined meeting is marked
TRANSP:TRANSPARENT. Writing an edit back uses the source as donor for what the
demotion removed, so editing a time cannot silently drop the guests.
- reconcile: pure decision engine. Only the source changed updates the mirror,
only the aggregate changed writes back, both changed keeps the source and logs
a conflict.
- Mass-deletion guard takes an absolute floor as well as a fraction: a share
alone is meaningless at small counts, where deleting the only event is 100%.
- sync refuses to run when an aggregate's configured target differs from the
recorded one, before reconciling. Otherwise the new empty target would read as
an aggregate whose every event was deleted, and delete propagation would then
remove them from every source.
Found by end-to-end testing: writing an item already present under a different
filename created a duplicate rather than replacing it, because filenames are
derived from the UID while pimsync picks its own. Writes now carry the path they
supersede. Covered by a regression test.
79 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 09:55:06 +03:00
|
|
|
- [x] `ical` round-trip fixtures: recurring with overrides, all-day, TZID, unknown `X-` props
|
2026-09-10 13:39:54 +03:00
|
|
|
- [x] `convert` against real captured API payloads, including the case where Google's
|
|
|
|
|
offset and its `timeZone` disagree
|
|
|
|
|
- [x] Topologies: a source feeding several aggregates, several aggregates sharing one
|
|
|
|
|
target, a cycle between two aggregates, a delete cascading across aggregates,
|
|
|
|
|
and competing edits arriving through two aggregates at once
|
|
|
|
|
- [x] Retarget: drift makes `sync` exit non-zero having written nothing and losing no
|
|
|
|
|
source event; an unrouted creation reaches a sink first
|
|
|
|
|
- [x] Safety (unit): no live `ATTENDEE`/`ORGANIZER` on a CalDAV-targeted mirror,
|
|
|
|
|
`VALARM` intact, `PARTSTAT: DECLINED` maps to `TRANSP: TRANSPARENT`
|
2026-09-10 09:34:42 +03:00
|
|
|
|
Let an event choose which source it is filed under
An event created in an aggregate went to the configured default_sink and nowhere
else, so with several writable sources there was no way to say which calendar a
new event belonged in.
A line reading @endpoint-id in the description now picks the source, and a
matching CATEGORIES value does too. The description rather than the title because
every calendar client exposes a notes field and editing it does not disfigure the
event's name; CATEGORIES as well because that is the field iCalendar intends,
even though many mobile clients hide it. The marker is stripped before the event
reaches the calendar, being calcalist's bookkeeping rather than content.
A marker naming something that is not a writable source of that aggregate is
refused and reported, not redirected to the default: a typo should not quietly
file an event in the wrong calendar. A bare address in prose is not a marker
either, since a marker must be a line of its own.
Also covers the shapes beyond many-into-one: a source feeding several aggregates,
several aggregates sharing one target, a cycle between two aggregates, a delete
cascading across aggregates, and competing edits arriving through two aggregates
at once — the last being caught by the existing conflict detection rather than
silently overwriting.
Error display no longer repeats itself; thiserror already prints the cause chain.
Verified live against real accounts: two Google calendars aggregating into a
Posteo calendar, an edit in the aggregate reaching the originating Google
calendar, an event routed to a chosen source by its description marker, and a
deletion propagating from the aggregate through to Google.
129 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 13:29:02 +03:00
|
|
|
### Verified live, end to end
|
|
|
|
|
|
2026-09-10 13:39:54 +03:00
|
|
|
Against real accounts — two Google calendars aggregating into a Posteo CalDAV
|
|
|
|
|
calendar:
|
|
|
|
|
|
|
|
|
|
- Fan-in from both Google sources, each mirror tagged with its origin
|
|
|
|
|
- A public iCal feed synced through pimsync and pushed to CalDAV; idempotent across
|
|
|
|
|
repeated runs and correctly incremental
|
|
|
|
|
- An edit made in the aggregate reaching the originating Google calendar, and the
|
|
|
|
|
revert travelling the same way
|
|
|
|
|
- An event created in the aggregate routed to a chosen source by its description
|
|
|
|
|
marker, with the marker stripped before it arrived
|
|
|
|
|
- A deletion in the aggregate propagating through to Google
|
|
|
|
|
- The mass-deletion guard refusing a 100% removal until `--force`
|
|
|
|
|
- `pimsync check` validating the generated config against a live CalDAV server
|
|
|
|
|
|
Close the remaining M1 gaps
Eight items were still open at the end of M1: two integration tests that had
only been run by hand, and six known gaps.
Recurrence overrides now reach Google. Google addresses an exception through
the series rather than as an event of its own, so the master is sent first and
each override is then matched to its instance by original start time and
patched. Matching needs the two sides' spellings reduced to one key: iCalendar
writes a zoned local time, Google an absolute offset. An override matching no
occurrence is counted rather than forced — that means a stale RECURRENCE-ID
left behind by an edited RRULE, and inventing an event for it would put
something in the calendar the series does not contain.
Reading one component apart from another needed a view `properties` cannot
give: it flattens every VEVENT together, which is right for the UID a series
shares and wrong for an override, whose SUMMARY and the master's are then
indistinguishable. `Calendar::events` splits them.
A TZID now travels with the VTIMEZONE that defines it, derived from the zone's
own transition table as the yearly rule it implies. This changes the content
hash of every zoned recurring event, so the first cycle after this re-pushes
them.
A Google authorisation is filed under the account it was granted for rather
than the endpoint that asked for it, so two endpoints on one account no longer
need a login each. The account is read from the primary calendar's id, which
needs no scope beyond the calendar one already granted. Authorisations written
by the previous scheme are still honoured, and move across at the next login.
An unreachable Google endpoint no longer ends the cycle — one lapsed token used
to stop the CalDAV side too. It is named, only the aggregates depending on it
stand down, and the run exits non-zero so a partial cycle cannot pass for
success. The CalDAV leg cannot be narrowed the same way: pimsync is one process
covering every pair, so a failure does not say which pair it belongs to.
`--dry-run` now pulls for real, into a throwaway copy of the local mirrors and
through a pimsync configuration that only ever reads from a server. What it
reports is measured against the calendars as they are now rather than against
whatever the last real cycle left behind.
`calcalist prune` reports local mirrors of endpoints the configuration no
longer names, and removes them under --force.
The integration tests run against a real Radicale server and a real iCal feed:
convergence and idempotence, the dry run, prune, and the scheduling rule
asserted on the bytes that actually reached the server. The plan asked for an
SMTP sink for that last one; Radicale implements no RFC 6638 scheduling, so a
quiet SMTP port would have proved nothing about the transform.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 13:49:38 +03:00
|
|
|
### Integration
|
|
|
|
|
|
|
|
|
|
- [x] Against a real Radicale server and a real iCal feed, over pimsync: two
|
|
|
|
|
sources converge on a CalDAV target, and the next cycle is a no-op.
|
|
|
|
|
- [x] Safety, against the same server: no live `ATTENDEE` or `ORGANIZER` reaches
|
|
|
|
|
the aggregate, the guest list survives as inert data, the alarm survives
|
|
|
|
|
intact, and the source keeps its scheduling properties.
|
|
|
|
|
|
|
|
|
|
The plan asked for an SMTP sink here. Radicale implements no RFC 6638
|
|
|
|
|
scheduling, so a quiet SMTP port would have proved nothing about the
|
|
|
|
|
transform — no server in reach of a test sends calendar mail at all. What
|
|
|
|
|
is asserted instead is the bytes that reached the server, which is the
|
|
|
|
|
thing the transform is actually responsible for. Proving the Google half
|
|
|
|
|
was settled separately, from the API discovery document.
|
|
|
|
|
- [x] `--dry-run` reaches the servers, reports what it found, and leaves both the
|
|
|
|
|
target and the local mirrors untouched.
|
|
|
|
|
- [x] A retired endpoint's mirror is reported by `prune` and removed under
|
|
|
|
|
`--force`.
|
|
|
|
|
|
Finish M2: run lock, systemd units, discover; drop the web UI
Designing the configuration UI in full made the case against building it. Its
audience would be people who find TOML hard, but with bring-your-own OAuth
client settled, every user must first create a Google Cloud project, configure
a consent screen and put a secret in a keyring — a far higher bar than editing
thirty lines of config. Anyone who clears it can edit the file; anyone who
cannot never reaches the file. Against that stood three dependencies, five
modules, an auth.rs refactor and a security surface guarding something that
reads the config and touches the keyring. `doctor` and `status` had already
absorbed most of what it was for. The reasoning is recorded in TODO.md and
SPECS.md rather than left as an apparent oversight.
The run lock is not a UI feature and closes a gap that already existed: nothing
stopped a timer firing into a hand-run cycle, and two cycles interleaving
writes over the same vdirs is what the design otherwise avoids. flock is used
rather than a pid file because the kernel releases it however the process ends,
so a crash cannot leave a lock to clear by hand — which also means a lock we
failed to take is held by a live process, so the pid in it is worth reporting.
The one idea worth keeping from the UI design was collection discovery, which
needed no web layer. `calcalist discover` prints a ready-to-paste endpoint
block per calendar a server offers, removing the most error-prone field in the
config. pimsync's discovery output is undocumented, so the format was
established against a real server first. Two things it teaches: everything
arrives on stdout including failures, and a pair has two storages, so pimsync
reports the scratch vdir's contents too — parsing anchors on the heading naming
the server, or a probe directory's leftovers would be offered as the user's
calendars.
Verified against Posteo as well as Radicale: all four calendars found, the
first matching the URL already configured.
Also fixes a real defect in the test harness rather than its symptom. Ports were
chosen by binding one and letting go, so two tests could pick the same number —
and the loser's readiness check then succeeded against the winner's server,
silently sharing it. Startup now confirms the child we spawned is the one alive,
retries on another port if not, and waits for a real HTTP response rather than
an open socket.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 16:17:34 +03:00
|
|
|
## M2 — packaging (done)
|
|
|
|
|
|
|
|
|
|
- [x] Run lock (`lock.rs`) — `flock` on the state directory, held by `sync`,
|
|
|
|
|
`aggregate retarget` and `prune --force`. Not a UI feature: nothing
|
|
|
|
|
previously stopped a timer firing into a hand-run cycle, and two cycles
|
|
|
|
|
interleaving writes over the same vdirs is what the design otherwise
|
|
|
|
|
avoids. The kernel releases it however the process ends, so a crash
|
|
|
|
|
cannot leave a lock to clear by hand; the refusal names the holding pid.
|
|
|
|
|
- [x] systemd user units in `systemd/` — `calcalist.service` (oneshot) and
|
|
|
|
|
`calcalist.timer`, with `RandomizedDelaySec` so installations do not all
|
|
|
|
|
call Google on the quarter hour, and no filesystem or IPC sandboxing
|
|
|
|
|
because the secret commands need the session keyring over D-Bus.
|
|
|
|
|
- [x] `calcalist discover` — asks a CalDAV server which calendars it has and
|
|
|
|
|
prints a ready-to-paste `[[endpoint]]` block for each. The `url` field is
|
|
|
|
|
the most error-prone thing in the config, and providers rarely show it.
|
|
|
|
|
- [x] `serve` removed from the CLI, and with it the last `unimplemented`
|
|
|
|
|
command, so every command the binary advertises now does something.
|
|
|
|
|
|
|
|
|
|
### The web interface, dropped
|
|
|
|
|
|
|
|
|
|
Specified from the start and designed in full before being dropped. The
|
|
|
|
|
reasoning, so it is not rediscovered as an oversight:
|
|
|
|
|
|
|
|
|
|
- Its audience would be people who find TOML hard. But with bring-your-own
|
|
|
|
|
OAuth client settled, every user must first create a Google Cloud project,
|
|
|
|
|
configure a consent screen and put a secret in a keyring — a far higher bar
|
|
|
|
|
than editing thirty lines of config. Anyone who clears it can edit the file;
|
|
|
|
|
anyone who cannot never reaches the file.
|
|
|
|
|
- The cost was three dependencies, five modules, an `auth.rs` refactor and a
|
|
|
|
|
security surface (token, `Host` validation, CSRF, secrets through subprocess
|
|
|
|
|
stdin) guarding something that reads the config and touches the keyring —
|
|
|
|
|
against SPECS.md's own "no speculative features or dependencies".
|
|
|
|
|
- `doctor`, `status` and `discover` had already absorbed what it was for.
|
|
|
|
|
|
|
|
|
|
The one idea worth keeping from the design was collection discovery, which
|
|
|
|
|
needed no web layer at all.
|
|
|
|
|
|
Close the remaining M1 gaps
Eight items were still open at the end of M1: two integration tests that had
only been run by hand, and six known gaps.
Recurrence overrides now reach Google. Google addresses an exception through
the series rather than as an event of its own, so the master is sent first and
each override is then matched to its instance by original start time and
patched. Matching needs the two sides' spellings reduced to one key: iCalendar
writes a zoned local time, Google an absolute offset. An override matching no
occurrence is counted rather than forced — that means a stale RECURRENCE-ID
left behind by an edited RRULE, and inventing an event for it would put
something in the calendar the series does not contain.
Reading one component apart from another needed a view `properties` cannot
give: it flattens every VEVENT together, which is right for the UID a series
shares and wrong for an override, whose SUMMARY and the master's are then
indistinguishable. `Calendar::events` splits them.
A TZID now travels with the VTIMEZONE that defines it, derived from the zone's
own transition table as the yearly rule it implies. This changes the content
hash of every zoned recurring event, so the first cycle after this re-pushes
them.
A Google authorisation is filed under the account it was granted for rather
than the endpoint that asked for it, so two endpoints on one account no longer
need a login each. The account is read from the primary calendar's id, which
needs no scope beyond the calendar one already granted. Authorisations written
by the previous scheme are still honoured, and move across at the next login.
An unreachable Google endpoint no longer ends the cycle — one lapsed token used
to stop the CalDAV side too. It is named, only the aggregates depending on it
stand down, and the run exits non-zero so a partial cycle cannot pass for
success. The CalDAV leg cannot be narrowed the same way: pimsync is one process
covering every pair, so a failure does not say which pair it belongs to.
`--dry-run` now pulls for real, into a throwaway copy of the local mirrors and
through a pimsync configuration that only ever reads from a server. What it
reports is measured against the calendars as they are now rather than against
whatever the last real cycle left behind.
`calcalist prune` reports local mirrors of endpoints the configuration no
longer names, and removes them under --force.
The integration tests run against a real Radicale server and a real iCal feed:
convergence and idempotence, the dry run, prune, and the scheduling rule
asserted on the bytes that actually reached the server. The plan asked for an
SMTP sink for that last one; Radicale implements no RFC 6638 scheduling, so a
quiet SMTP port would have proved nothing about the transform.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 13:49:38 +03:00
|
|
|
## Known gaps
|
|
|
|
|
|
|
|
|
|
- [ ] A `pimsync sync` that fails takes the whole CalDAV leg with it. Unlike the
|
|
|
|
|
Google side this cannot be narrowed: pimsync is one process covering every
|
|
|
|
|
pair, so a failure does not say which pair it belongs to.
|