Should LadybugDB be rewritten in Rust?

[adsharma]

This is a difficult question and frankly a very divisive issue in a lot of other projects. Before it becomes a thing, here's an attempt to describe the current thinking behind sticking to an incremental path.

• Rust is a leading memory-safe language with excellent performance characteristics.
• Rust also has a steep learning curve comparable to C++
• Rewriting a large code base from any language A to language B is expensive, risky and should be considered carefully

While there is no question that if we can do it safely and at low cost, we should do it. But...

• LadybugDB has a packed schedule to deliver functionality (see Release schedule and Roadmap #122)
• There is an incremental path available. C++20 -> C++26 -> Carbon -> Eventually Rust
• New generation of coding tools are changing how software is developed.

The last part is the biggest wildcard. We have to wait a bit to figure out where things land when the dust settles. Perhaps different pieces of software get developed at different levels of abstraction and then lowered gradually to things like Rust, MLIR and then machine code.

[prrao87]

While I don't have the real world systems engineering experience with C++ that some others who comment here may have (and only a basic knowledge of Rust), I've worked with enough people in Kuzu and outside, who deeply understand the tradeoffs involved. From a long-term project maintainability standpoint (with the goal of drawing more OSS contributors who will remain committed to the project), Rust would tend to have the upper hand.

During my time at Kuzu, having seen numerous build issues, segfaults and data race bugs (many of which could have potentially been avoided by Rust's thread and memory safety), and most importantly imo, lack of asyncio support for Python users due to interop issues with pybind11, I've always been in support of database/data infrastructure projects being maintained long-term in Rust. PyO3 makes the bridge between Rust and Python an absolute joy, and imo is the reason why so many Python projects built on top of Rust today are so successful.

Although LadybugDB is its own binary and is meant to cater to all language clients, there's no doubt that Python users will be its biggest community, so if I had to point to valid reasons for my thinking, my vote for a Rust rewrite is based on the above factors.

[adsharma]

Make life better for a majority of end users - yes, that's a worthy goal and 100% behind it. The main question is the timing.

If we don't ship a leading storage engine that people embed to maintain their agent state, something else will take the market. And then it's not worth rewriting anything.

[prrao87]

That's a very fair point too, and rewriting anything is not a light undertaking. It's a slow burner process.

[aheev]

From a long-term project maintainability standpoint (with the goal of drawing more OSS contributors who will remain committed to the project), Rust would tend to have the upper hand.
During my time at Kuzu, having seen numerous build issues, segfaults and data race bugs (many of which could have potentially been avoided by Rust's thread and memory safety), and most importantly imo, lack of asyncio support for Python users due to pybind/pybind11#2658, I've always been in support of database/data infrastructure projects being maintained long-term in Rust. pybind/pybind11#2658 makes the bridge between Rust and Python an absolute joy, and imo is the reason why so many Python projects built on top of Rust today are so successful

+1

[aheev]

There is an incremental path available. C++20 -> C++26 -> Carbon -> Eventually Rust

Carbon would take atleast 2 more years to be prod ready

Rewriting a large code base from any language A to language B is expensive, risky and should be considered carefully

I agree that rewriting a large codebase needs to be considered carefully. In this case, though, the codebase isn’t especially big and, with the help of coding assistants, I think we could move fairly quickly

There are also some notable design issues in the current codebase which is making it harder to implement new features. I’m happy to have a go at an initial rewrite to get a clearer view of the risks involved

[adsharma]

It's small relative to other large projects such as Linux kernel or Postgres. But we don't have 100s of engineers working on it.

Suggest fixing any problems in the C++ code base first. Once ladybug has a significant user and developer community (I'd say 5k github stars and a healthy industry adoption), we can consider a more gradual approach towards that north star.

More pressing problem: Ladybug doesn't have a stable storage format (#35 ). An inadvertent code edit will break users of older versions.

Duckdb's solution: https://github.com/duckdb/duckdb/blob/main/scripts/generate_serialization.py
Potential ladybug solution: https://github.com/adsharma/typespec-py

Instead of writing JSON files like this: https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/storage/serialization/extra_drop_info.json

We would write typespec as specified by: typespec.info

Switching these things from C++ to Rust would be trivial.

My belief is that in the long term, much rust code will be developed at a higher level of abstraction and not hand written.

[aheev]

Makes sense. Thanks!!

[ritwizsinha]

Here are my observations as a person who is trying to establish familarity with the codebase and the design.
(Context: I have production grade experience with cpp and go but only surface level understanding of rust)

The community around ladybugdb is not mature enough to create a healthy feeback/review cycle yet. This is an essential quality required for rewrites.

Someone who is rewriting from language A to B should be ideally proficient in both of them to understand why something was written the way it is and how to actually translate it. I see @adsharma being the only active contributor currently, so they must be a good enough judge of their expertise in both.

Coming to the topic of coding agents being helpful for a quick rewrite. I highly doubt its efficacy in actual production code. At the end of the day which product would you like to market and have confidence in, a well tested piece of software written with intention in a slightly annoying language or a completely new vibe coded high flyer written in Rust.

[adsharma]

Similar to typespec -> compiled languages for things like WAL records, I suspect there are some opportunities where a high level algorithm can be expressed with design-by-contract clauses and then lowered to compiled languages. My past writings on the topic:

https://adsharma.github.io/agentic-transpilers/
https://adsharma.github.io/diffusion-model-for-programming/

For some tasks, for a small number of devs proficient in them, direct coding in the compiled language makes sense.

I'll hopefully get to write mcp-server-py2many one of these days.

[vkozio]

I'm into deep research of code2code and developing some unique technology for that.

[adsharma]

What is code2code? Do you have a url?

[vkozio]

Not public sorry. Concept IR lowering and a semantic substrate. Open to discussing compute sponsorship for a Ladybug-to-Rust effort.

[barracuda156]

This is an upsetting development, if rewrite will happen :(

[prrao87]

Curious, what part is upsetting? Did you (or did you not) want it rewritten in Rust? As far as I can see, Ladybug will continue on its C++ roadmap and interface with Rust on an as-needed basis.

[adsharma]

I imagine this to be the default way to access ladybug via a wire protocol when compatibility is not required:
https://github.com/LadybugDB/ladybug-adbc-rs

Distributed Actors based on Ladybug:
https://github.com/orgs/Latentpedia/discussions/4