# Delta Sync: Implemented Auction Delta Sync (Issue #4207) **URL:** \ **Category:** Technical\ **Created:** [April 7, 2026, 12:29am UTC](https://forum.cow.fi/t/delta-sync-implemented-auction-delta-sync-issue-4207/3413 "2026-04-07T00:29:47Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![ashleychandy](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/ashleychandy/32/1717_2.png) [@ashleychandy](https://forum.cow.fi/u/ashleychandy)\ **Post date:** [April 7, 2026, 12:29am UTC](https://forum.cow.fi/t/delta-sync-implemented-auction-delta-sync-issue-4207/3413/1 "2026-04-07T00:29:47Z")
## Background Every auction cycle, the autopilot was broadcasting a full orderbook snapshot to every connected solver. On mainnet, over 90% of that payload was identical between consecutive auctions — mostly static limit orders sitting unchanged. As we scale toward a unified mega-auction across chains, this approach doesn’t hold up. The fix: replace full snapshots with incremental updates. * * * ## What Changed ### Autopilot Side **`SolvableOrdersCache` now emits structured change events.** The core idea is that instead of rebuilding the full auction from scratch and shipping it wholesale, the cache now tracks what actually changed between cycles and emits a `DeltaEnvelope` containing only those changes: ```auto DeltaEvent::OrderAdded(order) DeltaEvent::OrderRemoved(uid) DeltaEvent::OrderUpdated(order) DeltaEvent::PriceChanged { token, price } DeltaEvent::BlockChanged { block } DeltaEvent::JitOwnersChanged { ... } DeltaEvent::AuctionChanged { new_auction_id } ``` Each envelope carries a monotonically increasing `delta_sequence` (global, never resets) and an `auction_sequence` (per-auction counter). This separation matters: `auction_sequence` resets at auction boundaries, but `delta_sequence` is the authoritative counter for replay and gap detection. **Three new HTTP endpoints** on the autopilot API: - `GET /delta/snapshot` — full state for bootstrapping a new connection - `GET /delta/stream?after_sequence=N` — SSE stream of incremental envelopes - `GET /delta/checksum` — SHA-256 hash of current order UIDs and prices for integrity verification The stream uses Server-Sent Events with a dedicated unbounded control channel for critical `resync_required` events so they’re never dropped when the main bounded buffer fills up. **Incremental cache rebuild (opt-in).** An `IndexedAuctionState` tracks which orders are currently filtered (in-flight, no balance, no price, invalid signature) so that on each cycle we only re-evaluate orders that actually changed. This runs as a shadow comparison first (`AUTOPILOT_DELTA_SYNC_SHADOW_COMPARE=true`) and can be promoted to primary (`AUTOPILOT_DELTA_SYNC_INCREMENTAL_PRIMARY=true`). **Delta history retention** is configurable with `MIN_DELTA_HISTORY_RETAINED`, `DEFAULT_DELTA_HISTORY_MAX_AGE`, and `MAX_DELTA_HISTORY` constants, with age-based pruning using the monotonic `created_at_instant` timestamp (not the wire `published_at`, which can drift under load). * * * ### Driver Side **The driver maintains a local `Replica`** that applies snapshot + delta envelopes to keep a local copy of the order set and prices: rust ```auto replica.apply_snapshot(snapshot)?; replica.apply_delta(envelope)?; ``` The replica validates sequence continuity, rejects stale envelopes, and requests a resnapshot on any gap or protocol version mismatch. It also validates a `boot_id` UUID (generated fresh on each autopilot start) so a restart triggers a clean resnapshot rather than applying deltas across sessions. A background `delta_sync` task runs the fetch-snapshot → follow-stream loop with automatic retry and exponential backoff: ```auto Syncing → apply snapshot → follow SSE stream ↑ | └──── Resyncing ←────────┘ (on resync_required or gap) ``` **Thin solve requests.** Solvers that set `supports_thin_solve_request: true` in their config receive a solve request with an empty `orders` array. The driver reconstructs the full auction from its local replica before passing it to the solver. This is gated behind `DRIVER_DELTA_SYNC_USE_REPLICA=true` and falls back gracefully to the full body if the replica isn’t ready. New HTTP headers carry metadata for thin requests so the driver can skip deserializing the body entirely when the replica is already warm: ```auto X-Auction-Body-Mode: thin | full X-Auction-Id: 42 X-Auction-Deadline: 2025-01-02T03:04:05Z X-Auction-Tokens: 0x...:1,0x...:0 X-Auction-Jit-Order-Owners: 0x...,0x... ``` * * * ## Correctness Guarantees The trickiest part was ensuring the incremental projection always matches the canonical full rebuild. After computing delta events, the code applies them back to the previous auction state and compares the result against the canonical rebuild: - **Match** → use the reconstructed state (preserving non-delta fields like `block` from the full rebuild) - **Mismatch** → fall back to canonical auction, recompute events from scratch, increment a metric A randomized property test runs 120 iterations of random order/price mutations and verifies that `apply_delta_events_to_auction(base, compute_delta_events(base, next)) == next` holds throughout. Checksum verification compares SHA-256 hashes of sorted order UIDs and token prices between the driver replica and the autopilot at health check time, surfacing any divergence without blocking the hot path. * * * ## Rollout Everything is feature-flagged: | Flag | Default | Purpose | | --- | --- | --- | | `AUTOPILOT_DELTA_SYNC_ENABLED` | `false` | Enable delta API endpoints | | `AUTOPILOT_DELTA_SYNC_SHADOW_COMPARE` | `false` | Run incremental path alongside full, compare results | | `AUTOPILOT_DELTA_SYNC_INCREMENTAL_PRIMARY` | `false` | Promote incremental path to primary | | `DRIVER_DELTA_SYNC_ENABLED` | `true` | Enable driver replica background task | | `DRIVER_DELTA_SYNC_USE_REPLICA` | `false` | Use replica for solve request preprocessing | | `supports_thin_solve_request` | `false` | Per-solver config to opt into thin mode | The plan is to enable shadow comparison first, watch the `delta_shadow_compare{result="mismatch"}` metric for a week, then flip incremental primary, then gradually roll out thin mode solver by solver. **References** Issue: [https://github.com/cowprotocol/services/issues/4207](https://github.com/cowprotocol/services/issues/4207) Implementation Branch: [https://github.com/ashleychandy/services/tree/delta-sync](https://github.com/ashleychandy/services/tree/delta-sync) ---
**Author:** ![moo\_keeper](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/moo_keeper/32/1634_2.png) [@moo\_keeper](https://forum.cow.fi/u/moo_keeper)\ **Post date:** [April 7, 2026, 12:27pm UTC](https://forum.cow.fi/t/delta-sync-implemented-auction-delta-sync-issue-4207/3413/2 "2026-04-07T12:27:06Z")
This is a great write-up — thanks for putting it together and sharing the implementation details as well. We’ve flagged it internally with the team. They’ll take a closer look, and we’ll circle back here if there’s feedback or discussion to bring in 👍 ---
**Author:** ![ashleychandy](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/ashleychandy/32/1717_2.png) [@ashleychandy](https://forum.cow.fi/u/ashleychandy)\ **Post date:** [April 7, 2026, 10:11pm UTC](https://forum.cow.fi/t/delta-sync-implemented-auction-delta-sync-issue-4207/3413/3 "2026-04-07T22:11:42Z")
Thanks for flagging this internally. I’ve updated the write-up, the previous version was incorrect, so I’ve replaced it entirely with the correct issue and implementation details. Happy to clarify or expand on anything if needed. ---
**Author:** ![moo\_keeper](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/moo_keeper/32/1634_2.png) [@moo\_keeper](https://forum.cow.fi/u/moo_keeper)\ **Post date:** [May 27, 2026, 12:43pm UTC](https://forum.cow.fi/t/delta-sync-implemented-auction-delta-sync-issue-4207/3413/4 "2026-05-27T12:43:40Z")
Apologies for the delay. This is still something we’d like to see happen and it remains on the roadmap, though it’s not our highest priority at the moment. Since it involves a fair amount of changes and coordination with the core team, we don’t expect external contributors to carry the full effort on their own, but we’re definitely open to incremental improvements along the way.