Open Sourcing ZeroMEV and Funding Its Maintenance

CoW Grants Program
1

Grant Title: Open Sourcing ZeroMEV and Funding Its Maintenance

Author: Pmcgoohan (zeromev.org)

About You:
Pmcgoohan has a 20-year career in developing automated trading systems and has been involved in Ethereum since before the pre-sale. His work includes the EthInclude project and contributions to discussions on MEV and builder centralization. He founded ZeroMEV in 2021 with an Ethereum Foundation grant.

Additional Links:

Grant Description:
ZeroMEV seeks funding to continue providing essential Miner Extractable Value (MEV) data services to the Ethereum community until the end of 2024. This funding will cover infrastructure and maintenance costs, ensuring the continued operation of the ZeroMEV API, website, and Ethereum nodes. The project will also be open-sourced by the end of 2024.

Key components of the project include:

  1. Maintaining the ZeroMEV API providing MEV summary data for each relevant transaction.
  2. Ensuring the ZeroMEV website remains operational and accessible.
  3. Continuing to run Ethereum nodes that collect transaction data for real-time MEV classification.
  4. Open-sourcing the ZeroMEV project code.

Grant Goals and Impact:
The goals of this grant are to:

  1. Maintain a critical source of MEV data for the Ethereum community.
  2. Promote transparency and awareness of MEV issues.
  3. Support research and development in MEV mitigation.
  4. Provide reliable data sources for the Ethereum community and related projects.
  5. Open-source the ZeroMEV codebase to enable community contributions and potential optimizations.

The impact of this grant will be significant for the CoW Protocol ecosystem and the broader Ethereum community. ZeroMEV plays a vital role in enhancing transparency around MEV on Ethereum. Many MEV research papers and community discussions on ethresear.ch have cited ZeroMEV, making it important to keep available for the completeness of those citations and for future research.

Milestones:

Milestone Due Date Payment
Milestone 1 31.12.2024 6,000 xDAI
Milestone 2 31.12.2024 4,000 xDAI

Milestone 1
Maintain continuous operation of the ZeroMEV API, website, and Ethereum nodes.

Milestone 2
By the end of 2024, ZeroMEV will:

  • Open-source the ZeroMEV project code, making it available on a public repository.
  • Documentation
  • Code examples for deploying and running the services

Funding Request:
We are requesting 10,000 xDAI from CoW Grants to support the continuation of ZeroMEV services until the end of 2024.

Budget Breakdown:
6,000 xDAI will be allocated to infrastructure and maintenance costs, and payed upon approval of the grant:

  • Server costs for running multiple nodes across the globe
  • Data collection and processing
  • API and website hosting
  • Ongoing maintenance and updates

4,000 xDAO will be payed upon completion of milestone 2

Gnosis Chain Address (to receive the grant):
0x022a0D82f00Ed885f3707B92279Aa8528dc1b0A0

Terms and Conditions:
By submitting this grant application, I acknowledge and agree to be bound by the CoW DAO Participation Agreement and the CoW Grant Terms and Conditions.

3 Likes
2

I support this proposal ensuring a critical source of MEV data is available for the Ethereum community.

2 Likes
3

I’ve updated funding and milestones to reflect full funding (as additional funding plans didn’t materialize).
I’d love to get @Pmcgoohan involved in this thread before proceeding to vote on the proposal.

4

Thank you for your work on this proposal, @middleway.eth. I fully concur with it (including the updated milestones).

Thank you all for your continued support of what I also consider to be a vital project for the Ethereum community.

1 Like
5

In regards to “code examples for running the services”, I would sooner see this revised to something along the lines of:

“Deployment code / artifacts for using a cloud-infrastructure orchestration language/framework such as Pulumi, Ansible, Helm charts or similar that allows for 1:1 replica deployment of the services / data provided by ZeroMev”.

This should then reduce the barrier of entry for decentralised deployment / replica of ZeroMev, and in the event that CoW DAO needs to step in, reduces many of the infrastructure devops deployment tasks that would otherwise be required.

1 Like
6

I will provide code examples of how I deploy and run services as well as documentation on the various projects and modules, as per the grant proposal.

I am not familiar with cloud orchestration languages such as Pulumi, Ansidy and Helm and did not use them to deploy zeromev.

7

I support this grant and believe doing the work to open source and contribute ZeroMEV to the community is an important effort after all that has already been put into it.

@Pmcgoohan going from mfw78’s comment, it would be helpful to understand how you’ve deployed the services up until now and how complex/manual the setup was (rough number of services, scaling considerations, etc). It would also be helpful to have a rough breakdown for the ongoing infrastructure costs of operating the service.

8

The server costs for last year were 9062 GBP so approx 12000 USD. Beyond my work in open sourcing the code, I don’t currently have the time for further development.

The site and API use 13 instances of varying spec:

extractor (ie: transaction timing data)
x3 full geth nodes (each run my extractor service)
x1 AWS Infura node (also runs my extractor service)
x2 replicating postgres dbs (extractor and site data)

mev classification
x1 full archive node (source for mev-inspect, also runs my classifier service)
x1 flashbots mev-inspect k8 node
x1 k8 master node
x2 replicating postgres dbs (hold MEV data)

zeromev site
x2 webnodes
free cloudflare hosting of static pages

Maintenance/installation is as you would expect for the geth and full archive nodes.
I hired a k8 specialist to get Flashbots mev-inspect working on K8s, the maintenance of which is well documented.
The extractor and classifier services are released manually via scp.
The website is updated automatically from github.
The API runs off postgrest and a node.js script.

If it ever came to me handing over the maintenance of the site to CoW, I think the simplest and lowest cost option would be to transfer the servers into your name rather than deploying from scratch.

9

I feel the need to add some additional context to this discussion, as I’m not sure whether all of you are fully aware of the situation as it stands.

I have been self funding the zeromev site and ZM/CoW API since April when the prior EF and CoW grants lapsed, and can no longer afford to do so.

As detailed above, the proposed grant barely covers costs, not my support and maintenance which I continue to give freely to the project.

I would gratefully welcome the opportunity to keep the site going and to share the source code with the community, as is proposed, as I think it is a vital and unique resource and one that could benefit greatly from community contributions.

However, I was scheduled to power down the servers this week in the abscence of further funding.

I cannot afford to be delayed further on this and, without agreement on funding, I will be forced to shut down all servers except the website and website databases next Monday 23rd.

This will irrevocably halt the extraction of new MEV data, reduce the website to published historical data only, and will result in the loss of the API.

My apologies if this feels pressurizing, but sadly this is the reality of the situation right now, and I feel it’s only fair to make you aware of it.

I sincerely hope we can proceed with the proposed grant to avoid this and so continue to service the Ethereum community with what is a unique resource in the MEV space.

2 Likes
10

Hey! A quick progress update for you all.

I have refactored the code to prepare it for open source and I am most of the way through the documentation.

I have opted for a step-by-step walkthrough of installing and running every component in the system (extractor, mev-inspect-py, classifier, web server/client and api) on a single low spec VM, with explanations along the way.

Anyone able to use a Unix prompt should be able to follow it and get an instance of the system up and running, including the website and API.

The biggest obstruction has been the Flashbots mev-inspect-py dependency, the latest version of which does not run without error. We are working on a fork of our own to get around this.

If this goes well, I hope to have the project completed by the end of next week (29 Nov).

1 Like
11

Thanks for the update and the commitment on open sourcing!

12

Hey @Pmcgoohan thanks for the update!

In order to actually start reviewing, would be great to get a github repo, once it is ready (:

I hope we can align everything towards the end of the year, so there’s a community discussion about the evolving MEV data toolings, for example the Sorella dashboard.

How does ZeroMEV fits in? What are the main advantages and unique info that it collects and exposes?
I hope this is clear enough for contributors to have the motivation (financial or other) for starting to contribute and maintain Zero MEV.

13

I’m happy to announce that the milestone 2 grant work is now complete!

I have documented and open sourced every component of the zeromev site and systems, including the extractor, classifier, web server, web client and Zeromev API.

The main repo and readme is here:

I’ve gone above and beyond to ensure that the readme guide is accurate and easy to follow. Anyone with technical experience should be able to setup a local instance of the entire system locally using it.

Those following the guide will also gain a good understanding of each component of the system and how they interact. Copious code and commit comments will support anyone wishing to dig deeper.

I’m grateful for the opportunity of open sourcing this project. It represents a lot of work for me over the last couple of years, and it feels good for it to be out there!

Once you’ve had a chance to review it, I’ll promote the release publically.

1 Like