Quary Core: Open Source Rust data transformation engine

TLDR: Human communication is the main blocker in analytics/data-driven decisions, and Quary’s core is our first step to improve that.

Wednesday marked a significant milestone for Quary - we announced the launch of our open-source data transformation core and I thought we’d share some of our thinking.

Problem

In surveys, 90% of respondents identify communication and cultural issues as the primary barriers to deriving value from data (source). Only 10% attribute the challenges to technological limitations. The key concerns include the inability to recognize bad data, the lack of self-service tools for end-users, the exclusion of end-users from the development process, and a general mistrust of data provided by others.

Solution

Enhancing communication with advanced tools that leverage Rust

Currently, the gold standard in data analytics involves sophisticated frontends, dashboards, and reports. For more in-depth analysis, users typically rely on data transformation tools that can only run on a local machine or a paid server.

At Quary, we're revolutionizing this process with our new core. By leveraging Rust's performance and portability, we've developed a tool that operates directly within your browser. This means you can access and perform data transformations in the browser and share your work through a simple, shareable link, enhancing communication and collaboration in data analytics.

Why open source?

• Grow the community/ecosystem: We're inviting developers from around the world to join our mission because we can’t solve every data challenge alone. Building an ecosystem is essential.
• Ensure trust: Our users can now see exactly how our platform operates, building trust and confidence in our product.
• We want to build an open-source company, we believe it’s the future.

Why start from scratch?

For those in the know, there is another core out there that most of the world is building on top of. It’s not good enough.

It lacks portability. Nowadays, our databases, including Postgres, DuckDB, and others, can operate directly within browsers—a concept we find incredibly forward-thinking. Yet, it's surprising that a SQL templating engine hasn't embraced this adaptability. Our goal is to bring this transformative experience into the browser, widening the reach and understanding of current technologies to a broader audience, thus enhancing communication. For our initial client, this approach has already democratized data transformation, making it accessible to those who previously found it out of reach.

It’s hard to build on top of. This is how one of a current prospect interacts with the current core

# initialize
competitor = competitorRunner()
 
# create CLI args as a list of strings
cli_args = ["run", "--select", "tag:my_tag"]
 
# run the command
res: competitorRunnerResult = competitor.invoke(cli_args)

It’s too slow. This is how long it takes the same person to generate their docs.

time competitor docs generate
real    49 seconds

Foundation for a great web experience

We are just about to launch our web experience, but we thought we’d show off some glimmers of what the above enables. And remember, all this runs in the browser. No need to install anything or run data through Quary’s servers.