Grant Application: Solver Score Conformance Test Set

Overview

In thread 3546, mfw78 named cowprotocol/services as the source of truth for this kind of work. This proposal is scoped that way. A solver must predict two steps that it cannot run: the driver applies protocol fees to the solver’s proposed amounts, and winner-selection scores the result. Core has written that copies of this arithmetic can diverge, and copies did in production (#4669, and #5010 on 2026-10-02). I will publish a test set that any solver, in any language, can run to check its fee and score model against services at a pinned commit. In the finished set, services code generates or checks every expected value. KaiserSolver will use the test set in cow-certify. The core costs 12,000 USDC over 7 weeks. The source map, the schema and the first 10 vectors, with a CI check against winner-selection, are public today, so you can inspect the work before you vote.

Author(s)

ZeroNine on this forum. GitHub: C-09-07, with the work under ZeroNineSolver. ZeroNine is also my solver, which runs against the CoW Protocol shadow competition on Plasma.

Experiences and Qualifications

I rebuilt the fee and score pipeline outside services and checked it against services itself. Three public artefacts show the work:

  1. The first 10 vectors of the test set and a Python replayer of the fee and score pipeline. A public CI job links winner-selection at the pinned commit and scores every vector with it; all 10 match to the wei. GitHub - ZeroNineSolver/cow-score-test-set: Solver score conformance test set for CoW Protocol: fee and score vectors generated from services at a pinned commit · GitHub
  2. A public-data audit of the fee policies on 57 settled Plasma trades, from the competition record and GET /api/v1/trades only. Every trade carried a surplus-family policy (surplus or price improvement) at factor 0.5 with maxVolumeFactor 0.0098. 50 of the 57 also carried a volume policy at factor 0.0002. cow-score-test-set/audits/plasma-fee-policies at main · ZeroNineSolver/cow-score-test-set · GitHub
  3. PR #1 to KaiserSolver/cow-backtester, merged 2026-09-23. It fixed a guard that could not see a cluster of related artefacts. I wrote the failing test first, then the fix.

Grant Description

Category: developer tooling for solvers, under CIP-82’s first priority (developer enablement). File and line numbers below refer to services at commit 67cd3a16 (2026-09-23). Milestone 1 re-pins to main on the day work starts.

The pipeline a solver must predict. A solver proposes executed amounts. The driver applies each protocol fee policy to them, in order (with_protocol_fees, crates/driver/src/domain/competition/solution/fee.rs:42-48). The autopilot then scores the resulting amounts with the winner-selection crate (score, crates/winner-selection/src/arbitrator.rs:267-272). The score is surplus plus protocol fees, in the native token (CIP-38). A Rust solver can link winner-selection for the score stage. The fee stage sits inside the driver, a GPL-licensed service. A solver in another language cannot call it, and a Rust solver can call it only by building the driver.

Two more copies of the arithmetic exist for accounting. The settlement observer (crates/autopilot/src/domain/settlement/trade/math.rs) produces the executedProtocolFees that /api/v1/trades publishes. The solver-rewards SQL, outside services, computes fees again. #2465 notes that it “needs to make a weird adjustment for sell orders (so that it’s aligned with driver)”.

Core’s own record. The maintainers have written down each item below.

  • #3978 moved ranking to autopilot scores only: “If the scores computed by the driver are different from what the autopilot computes it can result in situations where not enough rewards are paid out”.
  • #4669: “26 executions on prod xdai were persisted with executed_fee = 0”. One product needed more than 256 bits in the observer.
  • #5010, merged 2026-10-02: a mainnet ETH buy settled with only WETH in the uniform prices. The observer read the wrong price, the fee came out negative, and it was stored as zero with no protocol fees. A replay of one day of dfusion’s ETH-buy trades, 7,299 of them, found two stored that way.
  • #2857 fixed floor and ceiling rounding of surplus, protocol fee and score. #2531 followed surplus calculation errors that happened in production.
  • #3340: “eventually we should have 1 code shared across autopilot and driver”.

What services already tests. The driver has 39 protocol-fee integration cases. They run the real driver against a mock solver (crates/driver/src/tests/cases/protocol_fees.rs). The autopilot has 10 winner-selection scenarios, and winner-selection has 6 tests of its own. These tests check services against itself. None of them is published in a form that another implementation can run. The test set closes that gap, with a schema compatible with the inline JSON TestCase format the scenarios use (crates/autopilot/src/domain/competition/winner_selection.rs:967-975).

Where the copies differ. Most differences show only at overflow or at a boundary. They are design choices and boundary behaviour, not bugs. Each is a vector class, because that is where a solver’s own model can drift without notice, and the solver learns about it from its reward.

  1. Native conversion. The driver’s in_eth does not check for overflow (crates/driver/src/domain/competition/auction.rs:248-250). winner-selection saturates to the maximum value (crates/chain-types/src/lib.rs:43-47).
  2. Sum of trade scores. The driver’s sum does not check for overflow (crates/eth-domain-types/src/ether.rs:62-66). winner-selection saturates (crates/winner-selection/src/arbitrator.rs:282-305).
  3. Surplus multiply-then-divide. The observer widens to 512 bits (crates/autopilot/src/domain/settlement/trade/math.rs:34-72). The scorer does not (crates/chain-types/src/evm.rs:63-75, called from crates/winner-selection/src/arbitrator.rs:444-473).
  4. Forward application, then inverse recovery. The driver applies a fee to pre-fee amounts. The scorer recovers the same fee from post-fee prices with the adjusted factor f / (1 - f) (crates/winner-selection/src/arbitrator.rs:536-548). This is by design, and it rounds: in 7 of the first 10 vectors the recovered fee differs from the applied one by 1 to 20 wei. The vectors record the difference as a fact.
  5. Zero score. Core changed how zero-score orders are treated twice in 2026 (#4408, then #4753). #4753 allows orders with zero score, because tokens with zero decimals could not show surplus after rounding.

The driver score still matters, although the autopilot ranks. The driver drops a solution when its own score fails, and it proposes only its best-scored solutions, one by default (crates/driver/src/domain/competition/mod.rs:492-537, crates/driver/src/infra/config/file/mod.rs:248-250). winner-selection also drops a solution when its score fails (crates/winner-selection/src/arbitrator.rs:139-172). Its own comment says that “fairness guarantees heavily depend on these scores being accurate”. So a difference between the two can remove a valid solution before it is ranked.

What I will build. A public test set of JSON vectors. Each vector holds one order (side, limits, partial-fill state, fee policies with factors, quote if any), the native prices, and one solution: executed amounts, solver fee, and uniform clearing prices (one price per token for the whole settlement). Each vector holds the expected outputs: each protocol fee in the surplus token and in the sell token, the custom clearing prices (the per-trade prices after fees), the user surplus, and the score in the native token. The surplus token is the buy token for a sell order and the sell token for a buy order. Intermediate values sit beside the final score, so a mismatch shows the step where it happened. Every vector carries the services commit, the fee policy kind with its CIP, and units on every field. When Milestone 1 re-pins, the schema gains the contributes_to_score flag that #4900 added to volume policies on 2026-09-25 (crates/driver/src/domain/competition/order/fees.rs on main), two days after the map was built. The input grid includes cases where fee caps bind: max_volume_factor and, in O1, the partner-fee cap.

How services generates the expected values. No upstream change is needed:

  • Fee stage: the harness runs the driver binary as a black box with a mock solver, the same way core’s integration tests do (crates/driver/src/tests/setup/solver.rs). It reads the executed amounts from the driver’s response.
  • Score stage: the harness links winner-selection and calls score().
  • Intermediate values: a fork pinned at the same commit, with a visibility patch under 50 lines. The patch is published as a diff. It makes private functions callable and does not touch any arithmetic.

Licences. The driver and autopilot crates are GPL-3.0-or-later. The harness links only winner-selection and runs the driver as a separate process. The vector data is CC0. The two replayers and the harness are MIT. The harness links winner-selection, which has no licence field in its Cargo.toml today; if core names one, I will follow it.

Consumers. Three public consumers show the “any language” claim holds: a Python reference replayer (standard library only), a TypeScript replayer (no dependencies), and a pull request to cow-certify that adds a check which consumes the test set.

KaiserSolver (cow-certify, cow-backtester, a live solver on Arbitrum and Base) agreed on 2026-09-26 to be named as a second user, on three conditions. The cow-certify pull request gets their normal review, and its merge is not a milestone. No grant funds go to KaiserSolver or to cow-certify. The test set lets them say that “cow-certify’s scorer conforms to services at version X”. It does not confirm the score of a live settlement. Their README, in the section “What it cannot check, honestly”, names one of their own checks C5 and says: “C5 does not confirm the competition score” (cow-certify/README.md at 4586f2743d76de8e00a132c9e01429b7140a4d13 · KaiserSolver/cow-certify · GitHub).

Public-data check. Milestone 3 runs two checks on four chains: Ethereum, Arbitrum One, Base and Plasma. GET /api/v1/trades gives each applied policy and the charged amount in executedProtocolFees. The v2 competition endpoint gives native prices, orders, and each solution’s executed amounts and published score. S3 holds the auction bodies for about 30 days, so the bodies I use are committed as fixtures.

  1. Fee check: recompute each protocol fee from its policy and compare it with the published amount. This tests the settlement observer. The zero-fee cases in #4669 and #5010 are the kind of mismatch it reports.
  2. Score check: recompute the score from executed amounts and native prices and compare it with the published score. This tests the deployed winner-selection against the pinned commit.

The inputs exist. The Plasma audit linked above reads every applied policy, with its factors, from /api/v1/trades alone. The recompute, its extension to buy orders, multi-order solutions, partial fills and non-winning solutions, and the cross-chain mismatch report are Milestone 3’s work.

Every mismatch goes into one closed class: reverted winner, JIT owner, fast path, upcoming_fee_policies switch, zero-score, version skew, or other. Each row also records penaltyCapNative from /trades. That is the CIP-87 per-order penalty cap, not a fee cap. No fee or score function reads it.

Sustainability. The score code changes often. In the 90 days to 2026-09-23, 15 commits touched the fee, score, winner-selection, number, chain-types and penalty-cap files, including #4702 and #4773. In the 12 days since, three merged changes altered fee behaviour: #4900 rewrote the driver’s haircut as a volume policy with a new contributes_to_score flag, #4981 made the fast path net the quote’s gas cost before it applies volume fees, so the settled amount matches the signed quote, and #5010 fixed the zero fee above. A weekly CI job builds the harness against services main and regenerates every vector. It has two outcomes. If values move, it opens an issue with the old and new values and the upstream commit range. If the build breaks, it opens a separate issue, and the test set stays valid at the last pinned commit. Until the grant ends, I repair the harness within 14 days of a break. After it ends, I will give up to 4 hours a month, unpaid, to keep the weekly job green. Upkeep beyond that would need a maintenance grant, and I would apply for one if the test set is in use. Today’s score check builds from clean in 42 s on a free GitHub runner. The Milestone 2 harness also builds the driver, so I will measure it again when it exists.

Demand. Core has asked for one reference twice (#3978, #3340). Inside services the arithmetic lives in four places: the driver’s fee step, the driver’s score, winner-selection and the observer. #2465 shows a fifth implementation, outside services, that had to be aligned by hand. The BYOS grant (thread 3476) opens the auction to permissionless sub-solvers that bid on single orders through one bonded solver, and it ranks their proposals with a score that, in its own words, “mirrors the driver’s scoring”. A sub-solver that wants to predict that score faces the same gap. This is a statement about structure, not an endorsement from the BYOS team. The community Rust SDK thread (3406) is a natural consumer.

Benefit. Any solver can check its fee and score model before it competes. Boundary behaviour becomes documented fact with a vector attached, a form maintainers can act on.

Type of Grant

Milestone-based.

Milestones

M Deliverable Due USDC
1 Source map, schema, 50 checked vectors week 2 3,500
2 Generator, two replayers, cow-certify PR, drift job week 5 5,000
3 Public-data check on four chains, v1.0 week 7 3,500
Core total 12,000
O1 Partner-fee and app-data vectors week 8 1,500
O2 Property-based divergence harness week 10 2,500
Optional total 4,000

Effort. The weeks are full-time: about 10, 15 and 10 working days for M1, M2 and M3.

Delivered at posting (links below). The services fee and score map, every branch cited to file and line. The draft vector schema. 10 hand-built vectors. A small check in public CI that links winner-selection and runs those 10 vectors. The Plasma fee-policy audit.

M1: Source map, schema, 50 checked vectors (week 2, 3,500 USDC). 50 hand-built vectors that cover: sell and buy; in-market and out-of-market; full and partial fills; each fee policy kind alone; two policies in both orders; zero fee; floor and ceiling boundaries; boundary classes 1 to 5 above. Each vector cites the file and line that justifies it. The CI check feeds every vector to the pinned code: the final score through score(), the intermediate values through the published visibility patch. Definition of done: tag v0.1; 50 vectors; a coverage table with one row per class and no empty row; the CI check green in public.

M2: Generator, two replayers, cow-certify PR, drift job (week 5, 5,000 USDC). The harness that runs the driver as a process and links winner-selection, and emits vectors from a committed input grid. The Python and TypeScript replayers. The cow-certify pull request. The weekly drift job. Definition of done: tag v0.2; at least 500 generated vectors; both replayers pass 100 % of vectors in CI; the cow-certify pull request is open and passes the test set in my CI; the drift job is live. Merge into cow-certify is at KaiserSolver’s discretion and is not a condition.

M3: Public-data check on four chains, v1.0 (week 7, 3,500 USDC). The two checks above on Ethereum, Arbitrum One, Base and Plasma, at least 200 settled auctions per chain. A per-chain table with match count and mismatch count per check. A quickstart that runs a replayer on the full test set in a clean container. A milestone report on the forum. Definition of done: tag v1.0; every mismatch assigned to a class; the match rate reported, with no rate promised; any class above 1 % of a chain’s rows investigated, with a vector attached; the drift job has run at least twice; the quickstart timed in CI, with the time in the report.

O1: Partner-fee and app-data vectors (1,500 USDC). Vectors for the partner-fee path, where factors come from app data (crates/autopilot/src/domain/fee/mod.rs:106-202) and are capped against a compounding accumulator (crates/shared/src/fee.rs:235-239). M3 does not need O1, because M3 reads applied policies from /trades, not from app data.

O2: Property-based divergence harness (2,500 USDC). A property-based harness that feeds the same random inputs to each copy of the arithmetic and reports every class of difference. Issues go to my repository first. An issue goes upstream only for a class that changes a score or a winner.

Each optional item is accepted or declined separately.

Length

Seven weeks for the core. Ten weeks with both optional items. Work starts when the Snapshot vote passes.

Funding Request

12,000 USDC for the core. Up to 4,000 USDC for the optional items, each accepted or declined separately, paid in USDC on Ethereum mainnet. No COW requested.

Payment Address (Ethereum mainnet)

0x8851B7950238201cF1180F147dB7B891F998122F

Other Information

Terms and Conditions

By submitting this grant application, I acknowledge and agree to be bound by the CoW DAO Participation Agreement and the CoW Grant Terms and Conditions.

Hi @ZeroNine, thanks for the application.

Before we take this further, the main thing we’d like to understand is who would use it.

The pull requests you cite show the fee and score arithmetic has diverged before, but we haven’t seen a services maintainer or a solver team say that a published vector set is something they would run. Has anyone told you they would? You mention KaiserSolver. If @kaisersolver plans to use it, a note here would help.

On scope: the source map, schema, and first 10 vectors are already public. Could you say which parts of Milestone 1 are new work?

Two small template items while you’re here: the terms section needs the links to the Participation Agreement and the Grant Agreement Terms, and the note to the committee is missing.

Thanks again for your interest,

Hi @Sov, confirming: yes, we plan to use it. KaiserSolver would run the vectors in cow-certify’s CI, so that its scorer conforms to services at the pinned version, and in our own engine’s tests, since our bids depend on predicting the driver’s protocol fees and the winner-selection score.

Hiya @Sov, thanks for the great questions. I’ll take them in order with adequate detail.

Who would use it. To your direct question: yes. KaiserSolver has answered above, and said more than the application claims: the vectors in cow-certify’s CI and in their own engine’s tests, because their bids depend on predicting the same two steps. Thanks to KaiserSolver for taking the time to post that, I really appreciate it. I use it myself: ZeroNine runs on the Plasma shadow feed, and even there I had to rebuild the fee and score arithmetic from services to understand where my solutions stood; the ten vectors are what came out of that work. Before a solver is on the feed, a Rust team can link winner-selection for the final score, but nobody can check the driver’s fee step without building the driver, and a team in any other language can’t check either stage. Once on the feed, a solver learns mostly from where it’s ranked, one submitted solution at a time, never step by step. The test set runs locally and sends nothing, so a team can test without showing anyone how its solver works. That gap isn’t hypothetical. There are teams waiting for access right now, and until the autopilot calls them, the only way to check what they’ve built is to rebuild the checks themselves, each from their own reading of the code; KaiserSolver and I have each done that rebuild once already. The set is that rebuild done once, checked against services, so every team measures against the same reference instead of its own. It’s the gap I’d like to fill with this grant, and it’s honestly something I look forward to building.

The maintainers have a stake in this too, even though they wouldn’t be the ones running it: a solver whose fee and score model matches the driver’s has fewer solutions dropped before ranking, and the cheapest place to keep that model matching is the solver’s own CI. The offer in the application stands: after M3, a subset of the vectors as a pull request to services, at the maintainers’ discretion and not as a milestone. The ten vectors are public and stay public whatever happens with the vote.

What in Milestone 1 is new. Fair question! None of what’s public counts toward M1’s ten days: the source map, the draft schema, the ten vectors and the CI check were all done before applying. The milestone’s title lists everything v0.1 will contain, including those. The new work starts from a limit the README states openly: CI currently checks only the final score, score_native, against winner-selection, because every fee and surplus helper in that crate is private. The fee per policy, the custom clearing prices and the surplus in every vector come from my Python transcription of the cited lines. M1 replaces that transcription for the score stage: a visibility patch on a pinned fork, published as a diff with the arithmetic untouched, and the harness extended so each vector’s per-policy fees, custom clearing prices and surplus are checked by winner-selection itself. The driver’s fee stage moves onto the driver binary in M2, as the application says. Then forty more hand-built vectors, each citing the line that justifies it, so every class in the coverage table has a row: sell and buy; in-market and out-of-market; full and partial fills; each policy kind alone; two policies in both orders; zero fee; floor and ceiling boundaries; the five boundary classes in the application. The forty are not the ten again: the ten each sit on the happy path of one class, while most of the forty have to land on a boundary (overflow, saturation, widening, round trip, zero score), and each needs an input built to hit the line it cites. The map and schema are then re-pinned to main, which already means a schema change: #4900 added a contributes_to_score flag to volume policies on 2026-09-25, two days after the pinned commit. The coverage table and the v0.1 tag close it out. As in the application, that’s about ten working days, most of them on the patch, the harness and the boundary vectors.

Template items. Apologies for missing those. The terms section should read:

By submitting this grant application, I acknowledge and agree to be bound by the CoW DAO Participation Agreement and the CoW DAO Grant Agreement Terms.

Note to committee: Please notify the Grantee of their reviewer and their steward in the thread and latest upon successful approval of the Grant on Snapshot.

I can’t seem to edit the original, so the corrected terms section lives here; please lift it into the Snapshot copy as needed. Happy to answer anything else, and thanks again! :slightly_smiling_face:

Hi,
Thanks for the work you’ve done thus far on this.

For something that is integral to the solver competition, such as solver score conformance tests, having golden vectors written by the core team is significantly more appropriate than having 3rd party assembled work - for example, new solvers may come to rely on something that may indeed contain errors or omissions, and cause consternation later should solvers follow the 3rd party tooling.

Solvers are incentivised by CoW Protocol, and any work that is done that addresses actions taken within the solver competition requires very careful consideration from the core team / solver team to ensure appropriate competition dynamics. I remain severely hesitant to enact any grant, retrospective or otherwise, that may influence the solver competition without this being thoroughly vetted and understood by the core / solver team (if not actually requested specifically by them).

mfw

Appologies, I deleted the response in attempt to edit it for replying..

Hi @mfw78, thanks for reading it closely, and for saying plainly where the hesitation is,

One thing I’d like to clarify, because it bears on the first concern. The design goal is that no expected value comes from my own transcription. For the ten that exist today, public CI already checks each final score against winner-selection itself at a pinned commit; the fee-stage values still come from my transcription, as the README says, and putting the score-stage intermediates under the same check is the first job in Milestone 1, with the driver’s fee stage following in Milestone 2. In the finished set, services code generates or checks every expected value, and a vector that disagrees with services fails CI. Today a Rust solver can link winner-selection for the final score, but its check on the driver’s fee step is one it wrote itself, a team in another language writes both, and the first time either learns whether that was right is in the ranking. I wrote one for our solver; KaiserSolver has said above that its bids depend on predicting the same two steps, and cow-certify carries its own scorer. That’s the state this grant serves to replace, with answers that come from services rather than from each team’s reading of it.

On vetting, I agree. If this goes ahead, the design (the source map, the schema, the harness) should be reviewed by the core or solver team before any work starts, and which cases go in, the input grid and the coverage table, should be theirs to approve, since choosing the cases is the real authorship. If core would rather the repository sat under cowdao-grants, it can move there before any work starts; every change to the vectors can go through core’s review, and core’s definitions take precedence over any draft values if there are discrepancies. The vectors are CC0 and the harness MIT, so ownership isn’t the question; the grant would fund the labour of building the rest under core’s direction. On competition dynamics, that judgement is core’s, and I’d want it made before any work starts. The intent is narrow: a way for a solver to check its baseline arithmetic before onboarding, the same way for everyone.

I didn’t want to impose on the core or solver team uninvited over work they haven’t asked for, which is why I came through the grants process rather than to them directly. If you think the source map in docs/ and the ten vectors are worth a brief look by the solver team, to say whether this is tooling they’d welcome under their review, I’d be grateful for the steer; if not, I understand. If the outcome is that core would rather own the whole of it, the gap is closed and I’ll say so here. Either way the ten stay public, and thank you for the time on it. :slightly_smiling_face:

Z9

Hi @ZeroNine, thanks for the follow-up and for taking the time to respond to mfw’s points.

After discussing this internally with @mfw78, we’ve decided not to move forward with this one.

Our reasoning: getting fee and score calculations right is something every solver already has to do for themselves, and if the competition needs a shared set of test cases, we think that should come from the core team rather than through a grant.

Appreciate the effort you put into this and wish you the best with your other work.