Introducing Gadus: CLI for doing geospatial data analysis using Compute over data pipeline

Report on our work @ EthParis hackathon working with Bacalhau and Filecoin to build tool for users to generate 3D maps rendered on DePIN infrastructure. this is the followup article to the one published by bacalhau here.

Intro

It was an indeed thrilling experience to participate in Ethglobal Paris hackathon, where we hacked on our project for 2 continuous days. We also get to meet with representatives of great web3 solution providers, especially Protocol Labs.

Our team consisted of 2 members :

• Dhruv Malik (working as developer and web3 researcher at Extra labs)
• Charlie Durand (founder of Extra labs along with expert in tokenomics and product manager).

With the aim of building a simple developer tool, we want to allow anyone to generate maps from our algorithms by providing inputs such as map coordinates. All the while, the entire pipeline would be running on the Bacalhau framework.

At this point, we had been working together for a few months, building the foundations at Extra Labs, a company that focuses on building developer tools and frameworks to host large-scale, near-real-time 3D maps in the world using decentralized physical infrastructure network (DePIN) infrastructure.

Problem statement

Extra labs , working with various projects interested in creating 3D maps has encountered many challenges when trying to deploy 3D maps. We observed that existing solutions for 3D maps were lacking in several dimensions, namely level of detail, recency, and accessibility. In terms of maps, the ecosystem suffers from a steep monopoly held by Google Maps.

Their massive investments in the early 2000s and since have build impressive products like google earth , maps API rendering etc., but also has seen huge price increases. In 2018 alone, the price for map-related Google APIs increased by an average of 1,400% (ref) along with the limitation of the rendering 3D tiles to be under 10k , a strict limitation for projects working on detailed tilings(ref).

As 3D maps become the next frontier for solving pressing challenges, such as sustainable town planning, protecting heritage monuments, and mapping during natural disasters, we need to build a platform as a public good that helps various stakeholders make sense of various data sources (such as satellite images and sensors like LiDAR or natural photos) and apply data transformations to make high-quality maps available to all at a reasonable cost.

To accomplish this, we require an efficient, scalable, and inexpensive compute framework. Prior to the hackathon, we built a pipeline that connects different open-source algorithms for 3D surface reconstruction (which can be found here) and benchmarked some decentralized computing services. We discovered compute-over-data and Bacalhau, which met our needs very well. During EthGlobal Paris, we explored a demonstration project that would enable us to deploy our pipeline on Bacalhau.

Project Gadus: CLI for running geospatial compute jobs:

We named our project Gadus (after the Latin word for fish, similar to Bacalhau). The inspiration of our project was Plex, developed by LabDAO for running BioML tools on private hosted instances of Bacalhau.

Similarly our CLI tools enable users to orchestrate 3D surface reconstruction operations on Bacalhau and to store them on decentralized storage, simplifying the various intermediate steps in the process : definition of the job specs, querying the requestor node to run the given geospatial transformation container algorithm and then reliably tracking the status and finally storing the result on local instance/IPFS with verifier.

We choose lilypad as the framework on top of Bacalhau, to create a marketplace of the compute resources provider, which host the compute jobs of the clients in lieu of clients paying the cost along with the commission for giving the access to our bacalhau cluster.

Along with that we also tried on developing :

• The wallet client to allow users to build a data wallet (using DFNS)
• A role based access for clients (based on their on-chain authentication using Worldcoin).
• A way to define compute jobs and manage payments via on-chain contract adapter with lilypad.

Here is the workflow that we had envisioned (still WIP with first release in coming weeks) using Gadus:

1. User has to create their wallet, which will create an EOA for the user and drops some lil-ETH token (which is the payment token for the lilypad testnet deployment).
2. For the data engineering part, we store the input data model of point cloud images together with the pipeline template information on the storage provider.
3. Next, the user must define the corresponding Docker image in which the job will be executed. To prevent sybil and cyber attacks, access to the compute infrastructure will be restricted to specific compute options.
4. These parameters are passed via CLI or hosted serverless application, which then relays them to the requester node. The user can then approve the result, which will be stored in their personal space on the decentralized storage system, using web3.storage.

submission of job and progress

We were able to demonstrate the minimal workflow showing the capabilities of the tool. Many thanks to the Protocol Lab/Filecoin team members that were present during the hackathon and helped us troubleshoot issues. Various detailed examples written in the documentation were also significantly helpful for new entrant to start building E2E compute pipelines of various categories of data.

Challenges and way forward:

We appreciate the significant progress that Bacalhau has made since its inception, such as vertical scaling with support for various compute runtimes and inputs from different storage platforms. After the proposed v1 launch, there will be progress in supporting real-time processing. However, Bacalhau has yet to address the significant potential for horizontal scaling, such as reducing latency, increasing bandwidth by running compute jobs in parallel across various compute nodes, and scaling resources based on the needs of client jobs.

In the end, we believe Bacalhau and other compute-over-data projects represents a promising layer to support large-scale, near-real time 3D maps. We’ll be happy to keep working with the people involved to solve the existing challenges and benefit the DePIN ecosystem.

In coming weeks, we will launch the MVP of Gadus, along with technical blog series on system design for compute over data platforms, so follow our medium, linkedin and github. we also just published a long article detailing the 101 about the potential benefits of using COD framework on geospatial data analysis.

Finally, I would like to express my appreciation for the help provided by the dev-rep team of Bacalhau/Lilypad (Allison and Luke), as well as the members of the Bacalhau Slack community. Kudos to this active community that assists users in overcoming production challenges related to Bacalhau.

If you have any questions or feedbacks, we’ll love to hear about it, you can reach out on our discord here.

Cheers and keep BUILDL!