Grant Application Data Pipeline for Solver Team

CoW Grants Program Core-Contributor-Grant
1

Grant Application Data Pipeline for Solver Team - Core Contributor

Grant Title:

Grant Application Data Pipeline for Solver Team - Core Contributor

Author:

About You:

I am a data engineer and data analyst with a solid background in building data pipelines.

Grant Category:

Other/Miscellaneous: designing and developing data pipelines

Grant Description:

Assist with the development and migration of the CoW data infrastructure.

In an effort to improve the current data infrastructure, this grant would properly advise the core team of CoW protocol in redeveloping these data pipelines and the way that this data is processed. Additionally, it would provide assistance to the core team to implement and develop these pipelines.

Grant Goals and Impact:

Upon successful completion of this grant the core Solver team will have a detailed overview of the new infrastructure, and relevant completed data pipelines that follow this infrastructure.

I will devote at least 10 hours each week to carrying out the Milestones specified in Milestones.

I shall provide the Services to CoW DAO from 28.10.2024 to 31.01.2024, to be actively renewed after the initial period through proposal and voting.

Milestones:

Milestone 1

Providing a detailed overview of an improved infrastructure for extracting, processing, and uploading data for CoW Protocol based on the current infrastructure and future plans.

Milestone 2

Redeveloping and implementing (together with the solver team) the solver-rewards repository and dune-sync repository following the new design. The two repositories (GitHub - cowprotocol/solver-rewards: Data Aggregation for Solver Reimbursement & Rewards Distributor and GitHub - cowprotocol/dune-sync: Components for syncing off-chain data with Dune Community Sources) contain all necessary code for executing the solver rewards payouts on mainnet, as well as syncing relevant data on Dune in order to verify these payouts, and are crucial components of the protocol mechanism.

Milestone 3:

Redeveloping and implementing (together with the solver team) the tokens-imbalances repository following this new design. For context, the token-imbalances repository (GitHub - cowprotocol/token-imbalances) is currently in its testing phase and is intended to provide almost real-time processing of all transactions that happen onchain with a focus on (1) tracking buffer changes in the settlement contract, (2) computing fees accumulated by solvers in it, and (3) fetching several price feeds in real-time in order to estimate the value of these buffer changes. This is seen as a crucial step towards more robust accounting and monitoring of buffers.

Funding Request:

My fee is 60 EUR per hour inclusive of taxes.

Gnosis Chain Address (to receive the grant):

0xb64da7096c50948721154036eC63665E9Db1C7c7

Other Information:

Past grant on which I worked: Grant Application - Leakage Analysis

Result of that grant: GitHub - ferrau10/cowswap

Referral:

The grant idea was discussed with Felix Leupold, Haris Angelidakis and his team.

Terms and Conditions:

Include the following line in your grant application (with careful attention to ensure the hyperlinks to the relevant documents are included):

5 Likes
2

Hey @butter_fly,

It’s great to see another grant proposal coming from you !

In light of the discussions with Felix and Haris, and previous grant execution, I’m in full support of this grant.

I would like to ask if you have any time estimates breakdown to milestone 1, and perhaps even to milestones 2 and 3?

Since the compensation is hourly-based, I think it is not crucial.
And yet still, I’d imagine it might be better to separate milestone 1 and come up with a more detailed system design spec, which then can be evaluated for execution.
I think this will allow for better review of the work and progress, even though might add slight overhead for submitting a followup grant.
Either way happy to support this proposal.

One tiny comment, the starting date is in October, are you expecting some retroactive compensation for work that has already been done? Or maybe that’s a mistake?

3

I agree with @middleway.eth. This is an interesting proposal and will provide value to the DAO.

4

Hi @middleway.eth glad to be back!
It’s really hard to breakdown the task yet since I need first to understand the context here - and this is what I have been doing since end of October. We decided to put three months for this grant because it’s at least certain it will take minimum that amount of time, and we talked about the possibility of a follow up grant if three months is not enough. It will also completely depend on how much development help I get from the team.

2 Likes
5

I am definitely in support of this, as the current data pipelines have several issues and the contributor here has significant experience in the domain in order to provide very meaningful contributions and improvements on this.

Timelines are indeed tricky to estimate here, and in my head i think that the possibility of a 3-month extension is likely, but also I don’t really see any issue with this. I would definitely hope that by end of January, Milestone 1 is done and Milestone 2 is well underway, but indeed another 2-3 months might be needed for the completion of everything described here.

1 Like
6

I can not modify this grant to add this to Milestone 1:
As a outcome of Milestone 1, I will post the proposed system design on this forum.

1 Like
7

Thanks @butter_fly
I think you should move your proposal to Snapshot for the committee to vote on it.
Just take into account that voting is live for three days, so it is advised to not have it during weekends when committee members are less attentive.

8

Thanks @middleway.eth for the suggestion. Here is the link to the snapshot: Snapshot

1 Like
9

:clap: already 6 votes “For”, so assuming this passes in a couple of days, I wanted to ask here on the accrued fees for and including November (the monthly Committee payout meeting is happening early next week). Would be great if you could update by Monday (cc @harisang for approval).

Thanks!

2 Likes
10

Hi Grant Committee,

For the period of 28.10 to 30.11 (included) I have worked 36.25 hours.
Multiplying this by my hourly rate, I request 2175 EUR for this period.

@harisang can you please confirm, thanks!

3 Likes
11

Confirming that @butter_fly has indeed worked the reported hours and has already provided significant insights and contributions towards Milestone 1

2 Likes
12

heya - your requested compensation is processed this week.

The next payout meeting is scheduled for 27th of December, could you please update here before then the time accrued / fund requested and for Haris to approve in time? (I’ll be off and won’t ping again) :slight_smile: // cc @middleway.eth

1 Like
13

Super thank you! And will do it earlier in December then!

1 Like