Cloud storage and synchronization design

Status: steps 1-4 are implemented as of 2026-09-17; step 5 has a reproducible baseline. Durable outbox/inbox receipts, dirty markers, a process-local sync session, and a provider-neutral runner with MemoryTransport fault injection are implemented in crates/core/src/sync.rs. OneDrive/Google Drive file adapters, token refresh and CLI cloud sync are implemented and tested with local fixtures. Live provider accounts have not been validated. Mobile AppAuth login and foreground sync controls are implemented; native consent requires platform registrations. See cloud-authorization.md for the current authorization scope. This document extends the OneDrive/Google Drive request with the requirement that concurrent devices must not corrupt or lock each other's database files.

Decision

Retain SQLite as a device-local transactional store. Add a provider-independent sync protocol that exchanges immutable, authenticated encrypted objects through cloud APIs. Each device owns its own database, identity, password, and local writer. Cloud storage is an untrusted object transport, not a mounted database.

Never put the live memories.db, journal, WAL, SHM, or lock file in a cloud-managed sync directory. Do not open the database through a remote filesystem. Keep the default OS application-data location. Detection/rejection of known OneDrive/Drive roots is a future configuration safeguard, not currently enforced; arbitrary sync clients and mount points cannot be detected reliably.

flowchart LR
    A[Device A local SQLite] --> QA[Durable outgoing queue]
    QA --> C[OneDrive or Google Drive encrypted objects]
    C --> VB[Download and verify]
    VB --> B[Device B local SQLite]
    B --> QB[Durable outgoing queue]
    QB --> C
    C --> VA[Download and verify]
    VA --> A

Existing behavior and limits

• crates/core/src/store.rs: an OS exclusive lock rejects a second local store owner. SQLite uses journal_mode=DELETE, synchronous=FULL, and transactional updates of encrypted state plus signed checkpoints. This protects cooperating local processes; it is not a distributed lock across independent file replicas.
• Every save serializes and encrypts the complete State; unlocking checks the complete checkpoint history. Larger libraries will increase write cost and startup time. This is a scaling concern, not evidence that SQLite itself must be replaced. No workload benchmark has been performed for this assessment.
• Database initialization holds the destination's OS lock while building a private staging database in the same directory. It commits the initial encrypted state and checkpoint, closes SQLite, syncs the file, and publishes without replacing an existing destination. A pre-publication crash leaves the final path available for retry. Abandoned .memories-init-* files may remain after a crash; they are never loaded as the live database and are not automatically deleted.
• Service processes mutations on cloned state and commits before accepting them. Vault::merge validates a candidate before replacing the current vault. Signed revision histories already preserve concurrent edits and tombstones.
• Desktop IPC accepts up to 16 concurrent requests; state dispatch holds the service mutex. Mobile uses individual service locks and a short registry lookup. A cloud HTTP request made directly inside these critical sections would delay other operations, including locking. This is blocking, not a demonstrated circular deadlock. Cloud workers must release service locks before network I/O; desktop request handling now uses bounded concurrent connections.
• Existing sync exports are full-vault, recipient-encrypted envelopes. A newly cloud-authenticated device still needs device approval before it can decrypt or import a vault. A cloud login must not replace cryptographic device pairing.

Database alternatives

Switching SQLite to WAL is not a cloud-sync solution. WAL may later help local read concurrency after measurement, but requires same-host coordination and does not remove the need for a single local writer.

Cloud object contract

Start with the existing full-vault envelope format; avoid changing storage, cryptography, and synchronization granularity simultaneously. Add incremental encrypted revisions/chunks only after the immutable envelope path is proven.

1. Generate each envelope once and persist its exact bytes in the local outbox before uploading. Use SHA-256 of the complete ciphertext object as its logical ID. Retries reuse those bytes; re-encrypting on every retry creates a new object.
2. Objects and publication descriptors are immutable. There is no shared writable latest.json, shared lock file, global cloud sequence, or database overwrite. Any future mutable head/cache is advisory and recoverable by listing objects.
3. Publish data objects first, then an immutable signed descriptor binding the format version, opaque vault reference, sender, recipient, object hashes and lengths, and per-device predecessor/sequence. Use ciphertext identifiers, not plaintext titles, account names, or password hashes, in filenames.
4. A descriptor can become visible before its dependencies because cloud listing and delivery can be delayed or reordered. Keep it pending until every object is present and validates. Never partially import it or advance its applied state while dependencies are missing.
5. Verify length, ciphertext hash, signature, recipient, vault scope, current authorization, and existing history invariants before committing. Provider ETags and hashes are transport metadata, not cryptographic authority.
6. Replayed objects and duplicate provider files are harmless: deduplicate by verified logical object ID, not filename. Missing cloud files never imply application-level deletion; deletion is a signed tombstone.

A cloud provider can delete or withhold objects. Local known heads/checkpoints can detect some rollback, but a fresh device without a trusted newer checkpoint cannot prove it has the newest state. This design provides integrity and explicit conflicts, not guaranteed availability or freshness against a malicious provider.

Transactions and worker lifecycle

The local writer owns application state, checkpoint updates, outbox/inbox state, and synchronization acknowledgements. Use two short critical sections around network activity:

1. Prepare locally: authorize, capture a consistent export, and commit the exact encrypted upload job and generation/session binding. A user edit and its dirty/export marker must commit atomically so a crash cannot lose work.
2. Transfer unlocked: a separate worker performs OAuth refresh, upload, download, pagination, and retries without holding the service mutex or a database write transaction. Bound concurrency, response sizes, retries, and connection/read deadlines. Honor provider throttling and Retry-After.
3. Commit locally: reacquire the writer, verify authorization and session generation, merge into the current state rather than an old captured copy, then commit state, checkpoint, applied IDs, and inbox status together. A provider listing cursor may advance only when the discovered work is durable.

Lock/logout cancels jobs and invalidates late results. No new decrypt/import may complete in an invalidated session. Cancellation cannot recall a ciphertext upload already accepted by the provider. Keep incomplete jobs resumable; upload success means cloud persistence, not acknowledgement by another device.

OAuth tokens remain encrypted and device-local, excluded from cloud objects, device approval packages, exported backups, search, and MCP. Refresh-token rotation must be persisted before subsequent operations use the new token; concurrent refresh is serialized per account, without holding the service lock during the HTTP exchange. Never add a public token-read operation for workers.

Provider-specific behavior

Both adapters must implement pagination, resumable transfers, bounded downloads, lost-response recovery, and provider-specific error handling. Do not assume Drive supports filesystem rename/locking or the same precondition semantics as Graph. Redirected download URLs and resumable-upload URLs require explicit credential-forwarding rules so OAuth bearer tokens are not sent to arbitrary hosts.

The initial application-folder design serves a user's paired devices connected to the same provider account and application. Google app-data storage does not support sharing files between different accounts. Cross-user vault sharing needs an explicit transport/sharing design; it must not silently broaden OAuth scopes. OneDrive and Google Drive are independent replicas if both are enabled, not mutually synchronized folders. Track pending and successful transfers separately.

OAuth client registrations and provider consent configuration are required before real-account verification. CLI authorization should use browser OAuth with PKCE, state validation, and a loopback callback. Mobile requires its supported native authorization flow and lifecycle handling; desktop callback assumptions must not be copied into mobile unchanged.

Conflicts, recovery, and retention

• Preserve both concurrent credential/card revisions. Require explicit selection or resolution before use; do not pick the newest wall-clock timestamp or merge password strings. Reuse existing policy-fork and old-epoch quarantine rules.
• A checksum/signature failure rejects the incoming object and keeps the previous local state. Distinguish corrupt input, missing dependencies, expired login, quota, and content conflict in status instead of reporting all as lock errors. A response whose bytes do not match its hash remains retryable with backoff. Hash-verified but invalid objects have separate rejection receipts; they never count as successfully applied dependencies. Rejections prevent reporting a successful sync and remain visible by provider in cloud status. Disconnecting a provider clears its rejection receipts so a subsequent connection can rescan. Legacy receipts that mixed rejections with successful imports are cleared once; the existing signed-packet receipts still prevent duplicate imports.
• Directory discovery has a 4096-entry inbox budget plus 16 reserved dependency slots. Verified descriptors schedule missing objects directly by hash, ahead of ordinary downloads. Unreferenced envelope buffers may be discarded under pressure and fetched again when their descriptor arrives. A partial directory page never advances the cursor past IDs that were not admitted.
• Every state commit removes queued exports to recipients whose current vault access has been revoked. Workers validate their session between object operations, including after provider ID allocation. Google file IDs are saved in the encrypted outbox before the first upload, so lost acknowledgements and restarts reuse the same immutable provider object.
• Keep old cloud objects initially. Safe garbage collection needs per-device acknowledgements, offline-device policy, revocation handling, and a recovery retention policy. Listing absence and elapsed time alone are insufficient.
• Keep independent encrypted backups. Multi-device sync also distributes valid deletions and is not a substitute for point-in-time recovery.
• Add a recovery-aware migration path before a schema migration: lock locally, retain a verified backup, migrate transactionally, verify, then activate. A failed migration must leave the previous store usable.

Implementation sequence

1. Done. Bounded writer requests (CloudSyncEnqueue / CloudSyncWork / CloudSyncAck) and durable outbox/inbox receipts inside encrypted State. SQLite encryption, journal_mode=DELETE, and whole-state saves are unchanged. No WAL or database-engine migration.
2. Done, local transport only. Provider-neutral immutable-object runner (sync::execute) with MemoryTransport fault injection. Tests cover retry of exact bytes, reopen after commit, duplicate put verification, descriptor before data, corrupt objects, lock/logout invalidation, and two paired devices. This test transport does not use live cloud accounts.
3. Done, local fixtures only. Token refresh, OneDrive/Google Drive immutable object adapters, daemon CloudSyncRun (HTTP outside the writer lock), and memories cloud sync. Live Microsoft/Google accounts were not authorized.
4. Implemented, live native consent pending. Mobile login uses AppAuth; foreground sync shares the core runner. Backgrounding locks all handles and cancels late results. Widget and native FFI regressions cover cancellation; real-provider/real-device consent remains a release gate. Background scheduling is intentionally not enabled.
5. Baseline available. See performance for the disposable release benchmark of library size, save/unlock/search latency and sync traffic. If needed, migrate the local whole-state blob to encrypted per-vault/revision/key rows with transactional receipts, then introduce incremental cloud objects. Preserve signed IDs, revision history, ciphertext privacy, and backup recovery.

Validation evidence and remaining gates

On Windows, crates/core/tests/storage_resilience.rs exercises disposable stores and subprocesses. Three scenarios passed during this assessment:

• A separate process is rejected while another owns the store; killing the owner releases the OS lock and the original encrypted state can be reopened.
• A child starts a SQLite write transaction, forces dirty pages into a rollback journal, and is killed before commit. Store::unlock validates the original state/checkpoint and PRAGMA integrity_check returns ok.
• Two authorized devices edit the same asset independently. Both merge orders converge, replay is idempotent, both heads remain, and an incoming snapshot with a missing parent is rejected without mutating the good state.

The deliberately ignored helper is explicitly launched by parent tests. Run:

cargo test -p memories-core --test storage_resilience --locked -j 1 -- --test-threads=1
cargo test -p memories-core --lib store::tests --locked -j 1 -- --test-threads=1

Initialization regressions in store.rs kill a subprocess after staging but before publication, then recreate/unlock the final store. They also verify the destination lock remains held and that an existing or newly appeared destination is never overwritten. OAuth tests cover invalidation of late results after lock.

These are local process-crash and model tests, not proof against actual power loss, disk/controller faults, cloud-client file replacement, or real provider behavior. Disk-full failures, large libraries, power loss at publication, and cloud retry/publication boundaries still need tests. No live cloud account was authorized or written during this assessment.

References

Official documentation checked on 2026-09-09:

• SQLite over a network
• SQLite WAL constraints
• SQLite corruption causes
• OneDrive application folders
• Graph upload sessions and conflict behavior
• Google Drive application data
• Google Drive uploads and pre-generated IDs
• Google native-app OAuth
• redb capabilities