Hi all, we run kaisersolver on Arbitrum and Base.
Something we kept running into as a small solver team: there is no way to test a solver against real auctions without going live. Shadow mode needs onboarding and runs on the live stream, and the playground is a fork, so routing and config changes mostly get judged in production. We ended up building an offline harness for ourselves and figured it might be useful to others, so we cleaned it up and open-sourced it:
It replays archived auctions from the public solver-instances bucket against any /solve endpoint, reconstructs the winning set from on-chain settlement data (grouped per auction for CIP-67, cross-checkable against the v2 competition API), validates responses roughly the way the driver would (fill accounting, fill or kill, fee-adjusted limit feasibility), and scores both sides on the same basis. With two endpoints, it runs an A/B on identical auctions. There is a bundled mock solver so you can see the output without wiring anything up.
To be upfront about the limits: replay quotes against live liquidity rather than the historical block, the bucket only retains about a month, and the scoring basis differs from the official score by the network fee component. All of this is spelled out in the README, along with how the winner reconstruction works.
The first run against our own engine was humbling; it showed we were consistently a bit behind one competitor on one specific pair, something that had taken us a long time to piece together from logs. That convinced us it was worth sharing.
MIT licensed. If anyone spots a place where the accounting diverges from what the protocol actually does, that is the most useful feedback we could get.