Grant Application: Dune Sync V2


Grant Title:
General Purpose Bidirectional Data Synchronization: Dune ↔ Local DB


Author:
The implementation of this project will be carried out by

Github: @bh2smith & @mooster531


About the Authors:

  • bh2smith is an experienced blockchain engineer and former contributor to CoW Protocol. He was the original author and primary maintainer of Dune Sync V1.

  • mooster531 is a seasoned python developer eager to dip their toes into blockchain via data engineering.


Grant Category:

Other (Operation Automation)


Grant Description:

Dune Sync V1 is narrowly scoped and uni-directional. Version 2 aims to improve on by implementing bidirectional data “syncing” and furthermore separating the fetch-insert logic from the Source-Destination configuration. This would allow anyone to run the service from a plain configuration file and preexisting binary image.

The general-purpose tool will enable syncing:

  1. Dune to Local:
    Archives data from Dune (e.g., historical prices or arbitrary aggregated data) via their API. This allows users to save on API costs and provides quick, easy, and local access to well-structured, readily available.

  2. Local to Dune:
    Uploads off-chain data collected by an application-specific server (such as CoWSwap’s APP Data, Solver Competition, or ERC-4337 UserOperation requests) to Dune for the sake of transparency, availability, application insight, and overall protocol-data completion.

An outline for Version 2 is sketched in Dune-Sync v2 · Issue #112 · cowprotocol/dune-sync · GitHub. It shows how the final result will use a configuration file to specify the Source and Destination (along with the credentials for read/write access), allowing the tool to implement fetch and insert logic for both DuneAPI and Postgres.


Milestones:

For the first two milestones Local DB refers to Postgres

Milestone 1: Dune to Local DB

This milestone will archive data from Dune via its API, inserting it into Postgres. For example, CoW Protocol may want to archive some of the CoW AMM Data. The docker image delivered will:

• Parse a configuration file containing Dune queries
• Fetch query results from Dune
• Insert records into Postgres, mindful of column type inference
• Avoid duplication and support scheduled syncing (e.g., daily or hourly)

A proof of concept script has already been prepared at GitHub - bh2smith/dune-sync (where the project development will take place).

Milestone 2: Local to Dune

Uploads off-chain data to Dune via its Upload CSV endpoint. This generalizes the functionality implemented in Version 1, allowing users to upload data from a local DB to Dune using only a configuration file.

The container image should be capable of the following flow:

  • User provides read access to a database and a raw SQL statement
  • Data retrieved from Postgres will be transformed into CSV and uploaded to Dune via the upload_csv endpoint

Milestone 3 [Optional]: Additional Features

This milestone is reserved for the purpose of further maintenance, improvements, extra features or any unforeseen requirements that couldn’t fit into the first two.

This could include

  • support for other databases
  • asyncronous query executions (performance)
  • more robust insertion conflict resolution

but is ultimately left open to the desires of CoW DAO after delivery of Milestones 1 & 2.


Timeline:

Milestones 1 & 2 aim to ship by early November 2024 (in time for DevCon VII).


Funding Request:

Funding request summary: 10000 COW + (optional 5000 COW for Milestone 3) paid as 5000 COW per milestone.


Gnosis Chain Address (to receive the grant):

gno:0x1ac70B075D431379c3eaF18C130764B2f609C503


Referral:

@fleupold


Additional References & Resources:


Terms and conditions:

By applying for this grant, I agree to be bound by the CowDAO Participation Agreement and the COWDAO Grant Terms and Conditions.

6 Likes

Grant is well-scoped, and the result would definitely be useful for more efficiently pushing and fetching raw competition data and CoW AMM data, so that the dbs maintained by the core team are kept in sync with Dune. Looking forward to it!

1 Like

Glad you like it, I have created the snapshot voting here:

https://snapshot.org/#/cowgrants.eth/proposal/0x11be3f17f867719563898f9c36e36fe9b1423c56105155f3e203e504d6ca851e

4 Likes

Milestones 1 & 2 have been completed and candidate release can be found here:

cc @fleupold, @harisang

If considered satisfactory, we would commence the development of Milestone 3.
Issues for milestone 3 have been recorded here: Issues · bh2smith/dune-sync · GitHub

3 Likes

Dang this looks AWESOME! Can’t wait to try this out!

Thanks a lot for the update! Will make sure to try this out in the next few days and post an update here soon!

2 Likes

Milestone 3 released under v0.3.0 at

Kindly requesting payment. Mooooooo!

3 Likes

Hello everybody!

Following the release of Milestone 3, we’d hereby like to apply for a grant for future development.

Grant Title:
General Purpose Bidirectional Data Synchronization: Dune ↔ Local DB


Author:
The implementation of this project will be carried out by

Github: @bh2smith & @mooster531


About the Authors:

bh2smith is an experienced blockchain engineer and former contributor to CoW Protocol. He was the original author and primary maintainer of Dune Sync V1.

mooster531 is a seasoned python developer eager to dip their toes into blockchain via data engineering.


Grant Category:

Other (Operation Automation)


Grant Description:

Dune Sync V2 aims to be a versatile, user-configurable and fast utility to synchronize one or multiple Dune tables with local Postgres databases, or vice versa.
Following the implementation and successful release of the first three milestones we’re ready to continue working on implementing the features agreed upon for Milestone 4:

  1. Utilize the Dune API to create/update tables in addition to allowing data upload via CSV. The advantage of this approach is the more straightforward and robust implementation of a data upload on Dune side.
  2. Ship the default Dune Sync V2 application container with a local database which can be used a temporary/staging area for data, so that multiple results from various queries can be concatenated and shipped to Dune in one Job.
  3. Support appending Dune tables from a local Postgres instance
  4. Support Amazon S3 Buckets as Source and Destination targets

The issues mentioned above have already been filed in the Github repo and can be seen here:


Timeline:

Milestone 4 can be delivered by end of March 2025. Sooner is possible, should there be an urgent need for those features.


Funding Request:

Funding request summary: 5000 COW for Milestone 4


Gnosis Chain Address (to receive the grant):

gno:0x1ac70B075D431379c3eaF18C130764B2f609C503


Additional References & Resources:

3 Likes

Thanks for the update, I’m generally supportive of the new scope. However, could you clarify the following please:

Shouldn’t this already be possible once support for the new dune api is completed.

I think from our current point of view, we are less interested in the S3 feature (seems more like a feature that other user’s of the system might want to use), but I’m not against it as it makes a lot of sense.

However, one other thing I’d like to add is some “technical marketing” (ie. a blog post, tutorial, etc.) to make sure more people in the space are being made a ware of this awesome public good that was developed with support of CoW Grants.

Wdyt?

2 Likes

Right, yes I had started preparing this write up for a knowledge sharing session:

We could scrape some things out of there.

As for

Support appending Dune tables from a local Postgres instance

I believe @mooster was referring to this issue:

Perhaps they can clarify…

I would also like to include support for SQLite at some point as well.

1 Like

Hey @fleupold and @bh2smith ,

About this:

Support appending Dune tables from a local Postgres instance

I was referring to the issue currently on the board: Dune Append Seems Possible · Issue #69 · bh2smith/dune-sync · GitHub
However, if the currently open PR 129 handles that then point 3 in my proposal can be considered “almost resolved”.
I hope this clears that up :slight_smile:

+1 on the local SQLite, despite the differences in dialects between that and Postgres, I think we should consider it.

2 Likes