# Core Contributor: Data pipeline engineer for Solver Team **URL:** \ **Category:** Core-Contributor-Grant\ **Tags:** complete, grant-funded, core-contributor\ **Created:** [November 4, 2024, 1:27pm UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626 "2024-11-04T13:27:55Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![butter\_fly](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/butter_fly/32/1240_2.png) [@butter\_fly](https://forum.cow.fi/u/butter_fly)\ **Post date:** [November 4, 2024, 1:27pm UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/1 "2024-11-04T13:27:55Z")
# _Grant Application Data Pipeline for Solver Team_ - Core Contributor * * * **Grant Title:** _Grant Application Data Pipeline for Solver Team_ - Core Contributor * * * **Author:** - [ferrau10@gmail.com](mailto:ferrau10@gmail.com) * * * **About You:** I am a data engineer and data analyst with a solid background in building data pipelines. * * * **Grant Category:** Other/Miscellaneous: designing and developing data pipelines * * * **Grant Description:** Assist with the development and migration of the CoW data infrastructure. In an effort to improve the current data infrastructure, this grant would properly advise the core team of CoW protocol in redeveloping these data pipelines and the way that this data is processed. Additionally, it would provide assistance to the core team to implement and develop these pipelines. * * * **Grant Goals and Impact:** Upon successful completion of this grant the core Solver team will have a detailed overview of the new infrastructure, and relevant completed data pipelines that follow this infrastructure. I will devote at least 10 hours each week to carrying out the Milestones specified in **Milestones**. I shall provide the Services to CoW DAO from 28.10.2024 to 31.01.2024, to be actively renewed after the initial period through proposal and voting. * * * **Milestones:** **Milestone 1** Providing a detailed overview of an improved infrastructure for extracting, processing, and uploading data for CoW Protocol based on the current infrastructure and future plans. **Milestone 2** Redeveloping and implementing (together with the solver team) the solver-rewards repository and dune-sync repository following the new design. The two repositories ([GitHub - cowprotocol/solver-rewards: Data Aggregation for Solver Reimbursement & Rewards Distributor](https://github.com/cowprotocol/solver-rewards) and [GitHub - cowprotocol/dune-sync: Components for syncing off-chain data with Dune Community Sources](https://github.com/cowprotocol/dune-sync)) contain all necessary code for executing the solver rewards payouts on mainnet, as well as syncing relevant data on Dune in order to verify these payouts, and are crucial components of the protocol mechanism. **Milestone 3:** Redeveloping and implementing (together with the solver team) the tokens-imbalances repository following this new design. For context, the token-imbalances repository ([GitHub - cowprotocol/token-imbalances](https://github.com/cowprotocol/token-imbalances/)) is currently in its testing phase and is intended to provide almost real-time processing of all transactions that happen onchain with a focus on (1) tracking buffer changes in the settlement contract, (2) computing fees accumulated by solvers in it, and (3) fetching several price feeds in real-time in order to estimate the value of these buffer changes. This is seen as a crucial step towards more robust accounting and monitoring of buffers. * * * **Funding Request:** My fee is 60 EUR per hour inclusive of taxes. * * * **Gnosis Chain Address (to receive the grant):** 0xb64da7096c50948721154036eC63665E9Db1C7c7 * * * **Other Information:** Past grant on which I worked: [Grant Application: Leakage Analysis](https://forum.cow.fi/t/grant-application-leakage-analysis/2413) Result of that grant: [GitHub - ferrau10/cowswap](https://github.com/ferrau10/cowswap) * * * **Referral:** The grant idea was discussed with Felix Leupold, Haris Angelidakis and his team. * * * **Terms and Conditions:** _Include the following line in your grant application (with careful attention to ensure the hyperlinks to the relevant documents are included):_ - By submitting this grant application, I acknowledge and agree to be bound by the [CoW DAO Participation Agreement](https://cloudflare-ipfs.com/ipfs/Qmf9MYhcG2pFrDoVy13p6FWeVF4nG9HbJvRfYYbhazTCFe) and the [CoW Grant Terms and Conditions](https://cloudflare-ipfs.com/ipfs/Qmag8BoNWhBnJ7TFyv2hzWwtoH17Xo7m9468NY6bN8sR6U). - By submitting this grant application, I acknowledge and agree to be bound by the applicable [CoW DAO Core Contributor Grant Terms](https://cloudflare-ipfs.com/ipfs/QmSUsd8yFrzbwQ5rBTqGv4uFDFSVStCFDcDwmNw9cx4vGT). ---
**Author:** ![middleway.eth](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/middleway.eth/32/12_2.png) [@middleway.eth](https://forum.cow.fi/u/middleway.eth)\ **Post date:** [November 8, 2024, 9:26am UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/2 "2024-11-08T09:26:35Z")
Hey @butter_fly, It’s great to see another grant proposal coming from you ! In light of the discussions with Felix and Haris, and previous grant execution, I’m in full support of this grant. I would like to ask if you have any time estimates breakdown to milestone 1, and perhaps even to milestones 2 and 3? Since the compensation is hourly-based, I think it is not crucial. And yet still, I’d imagine it might be better to separate milestone 1 and come up with a more detailed system design spec, which then can be evaluated for execution. I think this will allow for better review of the work and progress, even though might add slight overhead for submitting a followup grant. Either way happy to support this proposal. One tiny comment, the starting date is in October, are you expecting some retroactive compensation for work that has already been done? Or maybe that’s a mistake? ---
**Author:** ![netrunner.eth](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/netrunner.eth/32/252_2.png) [@netrunner.eth](https://forum.cow.fi/u/netrunner.eth)\ **Post date:** [November 10, 2024, 8:55am UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/3 "2024-11-10T08:55:05Z")
I agree with @middleway.eth. This is an interesting proposal and will provide value to the DAO. ---
**Author:** ![butter\_fly](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/butter_fly/32/1240_2.png) [@butter\_fly](https://forum.cow.fi/u/butter_fly)\ **Post date:** [November 11, 2024, 10:23am UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/4 "2024-11-11T10:23:21Z")
Hi @middleway.eth glad to be back! It’s really hard to breakdown the task yet since I need first to understand the context here - and this is what I have been doing since end of October. We decided to put three months for this grant because it’s at least certain it will take minimum that amount of time, and we talked about the possibility of a follow up grant if three months is not enough. It will also completely depend on how much development help I get from the team. ---
**Author:** ![harisang](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/harisang/32/524_2.png) [@harisang](https://forum.cow.fi/u/harisang)\ **Post date:** [November 21, 2024, 9:23am UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/5 "2024-11-21T09:23:49Z")
I am definitely in support of this, as the current data pipelines have several issues and the contributor here has significant experience in the domain in order to provide very meaningful contributions and improvements on this. Timelines are indeed tricky to estimate here, and in my head i think that the possibility of a 3-month extension is likely, but also I don’t really see any issue with this. I would definitely hope that by end of January, Milestone 1 is done and Milestone 2 is well underway, but indeed another 2-3 months might be needed for the completion of everything described here. ---
**Author:** ![butter\_fly](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/butter_fly/32/1240_2.png) [@butter\_fly](https://forum.cow.fi/u/butter_fly)\ **Post date:** [November 27, 2024, 3:59pm UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/6 "2024-11-27T15:59:01Z")
I can not modify this grant to add this to Milestone 1: As a outcome of Milestone 1, I will post the proposed system design on this forum. ---
**Author:** ![middleway.eth](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/middleway.eth/32/12_2.png) [@middleway.eth](https://forum.cow.fi/u/middleway.eth)\ **Post date:** [November 28, 2024, 10:27am UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/7 "2024-11-28T10:27:31Z")
Thanks @butter_fly I think you should move your proposal to Snapshot for the committee to vote on it. Just take into account that voting is live for three days, so it is advised to not have it during weekends when committee members are less attentive. ---
**Author:** ![butter\_fly](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/butter_fly/32/1240_2.png) [@butter\_fly](https://forum.cow.fi/u/butter_fly)\ **Post date:** [November 28, 2024, 4:17pm UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/8 "2024-11-28T16:17:58Z")
Thanks @middleway.eth for the suggestion. Here is the link to the snapshot: [Snapshot](https://snapshot.org/#/cowgrants.eth/proposal/0x747d34743f675f5625d15e262904c13251d20db86209ebd32906f8c7f9427b8a) ---
**Author:** ![c3rnst](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/c3rnst/32/219_2.png) [@c3rnst](https://forum.cow.fi/u/c3rnst)\ **Post date:** [November 29, 2024, 2:21pm UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/9 "2024-11-29T14:21:20Z")
👏 already 6 votes “For”, so assuming this passes in a couple of days, I wanted to ask here on the accrued fees for and including November (the monthly Committee payout meeting is happening early next week). Would be great if you could update by Monday (cc @harisang for approval). Thanks! ---
**Author:** ![butter\_fly](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/butter_fly/32/1240_2.png) [@butter\_fly](https://forum.cow.fi/u/butter_fly)\ **Post date:** [December 2, 2024, 4:14pm UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/10 "2024-12-02T16:14:14Z")
Hi Grant Committee, For the period of 28.10 to 30.11 (included) I have worked 36.25 hours. Multiplying this by my hourly rate, I request 2175 EUR for this period. @harisang can you please confirm, thanks! ---
**Author:** ![harisang](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/harisang/32/524_2.png) [@harisang](https://forum.cow.fi/u/harisang)\ **Post date:** [December 2, 2024, 4:19pm UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/11 "2024-12-02T16:19:27Z")
Confirming that @butter_fly has indeed worked the reported hours and has already provided significant insights and contributions towards Milestone 1 ---
**Author:** ![c3rnst](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/c3rnst/32/219_2.png) [@c3rnst](https://forum.cow.fi/u/c3rnst)\ **Post date:** [December 3, 2024, 8:52am UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/12 "2024-12-03T08:52:34Z")
heya - your requested compensation is processed this week. The next payout meeting is scheduled for 27th of December, could you please update here **before** then the time accrued / fund requested and for Haris to approve in time? (I’ll be off and won’t ping again) 🙂 // cc @middleway.eth ---
**Author:** ![butter\_fly](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/butter_fly/32/1240_2.png) [@butter\_fly](https://forum.cow.fi/u/butter_fly)\ **Post date:** [December 3, 2024, 9:21am UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/13 "2024-12-03T09:21:39Z")
Super thank you! And will do it earlier in December then! ---
**Author:** ![middleway.eth](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/middleway.eth/32/12_2.png) [@middleway.eth](https://forum.cow.fi/u/middleway.eth)\ **Post date:** [December 27, 2024, 11:12am UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/14 "2024-12-27T11:12:44Z")
Dear @butter_fly and @harisang, Would love to get the summary of hours for December so we can get it payed in this payment cycle 🙌 Happy new year 🎄 ---
**Author:** ![butter\_fly](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/butter_fly/32/1240_2.png) [@butter\_fly](https://forum.cow.fi/u/butter_fly)\ **Post date:** [December 27, 2024, 11:34am UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/15 "2024-12-27T11:34:12Z")
Hi @middleway.eth For the period of 01.12 to 25.12 (included) I have worked 1610 min. Multiplying this by my hourly rate, I request 1610 EUR for this period. @harisang can you please confirm, thanks! Happy Holidays! ---
**Author:** ![butter\_fly](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/butter_fly/32/1240_2.png) [@butter\_fly](https://forum.cow.fi/u/butter_fly)\ **Post date:** [January 24, 2025, 5:51pm UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/16 "2025-01-24T17:51:01Z")
Hi @middleway.eth Posting already the data for January before you need to ask 🙂 For the period of 27.12 to 24.01 (included) I have worked 1930 min. Multiplying this by my hourly rate, I request 1930 EUR for this period. @harisang can you please confirm, thanks! ---
**Author:** ![harisang](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/harisang/32/524_2.png) [@harisang](https://forum.cow.fi/u/harisang)\ **Post date:** [January 27, 2025, 10:08am UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/17 "2025-01-27T10:08:04Z")
Confirming that @butter_fly’ continues to contribute and is working actively towards Milestone 2. Hours reported look accurate, and the core team has been consistently in touch with @butter_fly over the past several months (on a weekly basis) so that these numbers can be verified. ---
**Author:** ![butter\_fly](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/butter_fly/32/1240_2.png) [@butter\_fly](https://forum.cow.fi/u/butter_fly)\ **Post date:** [February 7, 2025, 11:44am UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/18 "2025-02-07T11:44:15Z")
Hi Grant Commitee, Here is the overview of the data pipeline architecture I have been working on: ![Screenshot 2025-02-07 at 12.39.04](https://europe1.discourse-cdn.com/flex013/uploads/cow/original/2X/9/9ed9ffd9403ad0ccbd387ef09fb6226ee091ec93.png) In the dbt part, I have implemented the data transformations for the solver rewards (part of Milestone 2). The automation of the data pipeline is still to do, and we are working on it with the solver team. Because I have not yet finished this project, I will submit another grant proposal. ---
**Author:** ![butter\_fly](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/butter_fly/32/1240_2.png) [@butter\_fly](https://forum.cow.fi/u/butter_fly)\ **Post date:** [March 10, 2025, 7:51pm UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/19 "2025-03-10T19:51:34Z")
Hi @middleway.eth Posting the data for February only now that I am back from vacation. For the period of 28.01 to 21.02 (included) I have worked 2150 min. Multiplying this by my hourly rate, I request 2150 EUR for this period. @harisang can you please confirm, thanks! ---
**Author:** ![harisang](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.cow.fi/harisang/32/524_2.png) [@harisang](https://forum.cow.fi/u/harisang)\ **Post date:** [March 13, 2025, 10:01am UTC](https://forum.cow.fi/t/core-contributor-data-pipeline-engineer-for-solver-team/2626/20 "2025-03-13T10:01:53Z")
Confirming that this is accurate and in favor of proceeding with the payment