Exactly. I would end losing while someone else would get the “extra surplus”. Anyway, I just mentioned this because people seem to forget how Cowswap works. There’s much more than just the competition between Solvers.
It would be nice if we keep improving, meaning more CoWs, more trades in each batch and so on but I don’t see that atm.
Conclusion: If people really want to redesign the reward structure we need to end the way rewards linearly scale with trades. Rewards should scale considering the “value” each batch adds to the protocol.
This absolute value of -6 was just a heuristic number I came up with by observing many settlements. It just feels that pennying more than 6-7 dollars would end up being unprofitable for a solver (on an average week) so it feels that if you hit this interval multiple times, then you are most likely pennying; at least this the case now.
To clarify again, I am not saying this should be the only rule, but I am saying that we could forbid pennying in general but also give a non-complete list of rules that will automatically flag a solver as a pennying solver.
Regarding the percent, I just feel that since rewards are absolute (i.e., no volume dependent), the easiest way to penny is simply decide on an absolute amount of it, and this is why I think that this absolute slippage accounting is what would reveal pennying.
In any case, and since these numbers might look arbitrary, we could also skip them; in any case, I believe we all have a very clear understanding of what consistutes pennying, so ultimately we could let the DAO decide on whether certain solvers are pennying or not.
One less heuristic way to do this is to look at the cumulative density function of the slippage distribution, and then evaluate if |CDF[d](0.0) - 0.5| < epsilon.
Here is a google collab with an algorithm we can run for detecting pennying. The CDF approach mentioned above is perhaps too simplistic, I believe a kernel density estimate is required to properly estimate the mode.
@tbosman I believe you can access the data you were asking from this link (if you prefer I can also send you the data via other means).
Here is a DRAFT CIP that proposes to ban pennying.
Regarding concrete comments about the text of the CIP, I suggest that we move the discussion there. For brainstorming about other ways of fighting pennying and general comments etc, we can continue here.
In discussions with Felix and Haris, I realized that there’s sort of “two ways” to penny. Assuming the solver provides their own interactions, they can either accurately report the token inputs and outputs of the interaction or not. If they accurately report, it’s very easy to detect the pennying because the token amounts provided do not sum to zero. If they do not, it’s harder to detect but there is already a rule against misreporting these in CIP-11: CIP-11: Rules of the Solver Competition - status quo and an update proposal (rule 4 under update 1). Apparently most pennying is in the former case right now. So yeah just wanted to put that out there for everyone.
I like this. Nice and simple but if a solver pennies even a small amount on a significant fraction of batches, it is probable to show up here by substantially increasing the cdf at 0 (i.e. the number of batches with negative slippage). Pretty similar to Haris’ proposal.
Side note: is the slippage in percentage or dollar terms in that graph?
I do think that using the median is the cleanest, most sensible way to do it actually. It quite accurately captures the pennying solvers. One small issue is that it also captures three of the largest gnosis solvers (1inch, 0x, Paraswap). All of these lose money in over 60% of cases.
I think the easiest way to fix it would be to tighten up the slippage limits, which incidentally would also save a lot of money (eg for this week the paraswap, 0x and 1inch solvers have racked up 1300, 1500, 2700 worth of slippage respectively, which is more than their reward for the former two and wipes out most if the reward of the latter).
For example, setting the slippage tolerance at the 20th percentile of the current distribution should be enough to push the median close to 0 (as anything in excess will revert, hence not contribute to the negative slippage counter at all). Based on the query below the 20th percentile would a -16$ for 1nch, and even lower 0x and paraswap.
I see an issue here about setting the slippage tol for 1inch to 0.1 ETH:
Perhaps setting it to 0.01ETH and implementing it for all three would be a good start?
Thanks for the collab marco. I actually meant data on the solutions that entered the competition (in particular their objective function, utility, gas, fees and so on), to be able to analyze using a second price auction format. Like the information returned by api.cow.fi/mainnet/api/v1/solver_competition/
Yes, as you observed, three of the gnosis solvers, due to a very skewed distribution to the negative side, have a quite significative negative median. This is why I believe it is necessary to use the mode instead as shown in the linked collab:
Unfortunately skweness also gives false positives for the CDF approach too (sorry I only realized afterwards). On the other hand, using the mode seems to work quite well:
I agree that if we could ban solvers collecting all positive slippage then this would make the median approach also work. But I guess that is a different problem we will have to solve.
I actually meant data on the solutions that entered the competition (in particular their objective function, utility, gas, fees and so on)
I’ll try to collect this data soon and will send it to you. Thanks for looking into this.
Sorry I don’t quite follow. Can you elaborate what you mean by this?
Just to clarify, what I am saying is that we could make those three gnosis solvers more efficient while also moving their median by just reducing the slippage tolerance on them. Whether or not they get to keep positive slippage doesn’t really affect this
I meant that ideally we should be able to solve the problem of verifying if a solver is pennying or not independently of solving the problem of not being able to force how solvers set their slippage tolerance.
In line with previous posts, we (1kx) see pennying as a function of the current rewards scheme as well. We are proponents of applying incentives for the desired outcome, and this behavior exemplifies how the current incentives could use an overhaul. We are convinced that the approaches presented above (e.g. rewards based on differences to second best solutions, random sampling between top k solutions) will lead to a more effective incentivization of solvers vs. the flat rewards currently in use.
While we don’t believe pennying is a negative to the user as elaborated by others, we understand the rationale behind removing this practice in the short-term. However, it isn’t hard to envision that this can become a game of whack-a-mole, e.g. once pennying is banned, solvers might leverage their private liquidity pools to adjust their prices (eating the marginal loss there in the same way as pennying is done today) to win the auction. For this reason, we view this ban as a stop-gap measure but strongly recommend work on a redesign of the reward mechanism to avoid further distractions and efforts on such issues that are merely an outcome of its shortcomings.
We will be voting FOR this proposal to stop the practice in the interim while a longer-term reward scheme solution is worked out.
