Hi guys, Hi @mmzeeman
Is it possible to use a RAFT implemtation (this one or this one) to distribute ESQLite (NIF driver) between 3 nodes?
If not, what can be done to make ESQLite distributed as Khepri or Mnesia?
Thanks
Z.
You mean something similar to Canocials dqlite? But then with Erlang? I don’t see why it shouldn’t be possible with Erlang. What would your intended use case be? It sounds like a fun project for sure.
Maas
Btw, dqlite also seems to have the single concurrent writer limitation sqlite has. So I’m not sure if this will be your intended use-case.
Maas
@mmzeeman yes, exactly as dqlite but in pure Erlang.
The idea is to be able to access ESQLite database from any node for High Availability.
@mmzeeman how much effort is needed to extend ESQLite to support dqlite? The downside with dqlite is that only runs on Linux and has many deps (io_uring, libraft, libsqlite3, etc.).
Indeed. Concurrent writes will be a plus. But my original idea is to replicate the database on multiple nodes for HA.
My expectation is that dqlite is pretty slow… It only supports single writers, meaning you have to wait for (network) consensus before every write you do. It really depends on your use case if this is worth it.
There are a couple of approaches you could think of:
- When you want to use dqlite as-is, you could create a dqlite nif. That is not really hard to do.
- If you would like erlang to handle the network bits… then it would be possible to study dqlite a bit more, and come up with a proper raft state machine and use esqlite as a storage layer. This approach takes a bit more intellectual effort, but has the benefit that it is more Erlang/Beamy.
There are also other approaches like what litestream does. It replicates the WAL log, so it is more a disaster recovery tool than a distributed database. I think it is possible to already use litestream with esqlite.
Both are valid approaches, but it depends on your use-case.
Maas
1 Like
Thanks for sharing @mmzeeman. Didn’t know litestream.
How could you imagine using it with esqlite?
Supposedly it works externally from your sqlite application. It runs as an external application, which opens the same database file as your own application. It then somehow captures WAL updates and replicates this to a different location.
FYI, dqlite also has a wire protocol. This makes it possible to integrate it into a pure erlang implementation without using nifs.
1 Like
While checking the doc and playing with Litestream, I couldn’t find a way to perform replication the DB on another machine. What’s supported is S3 (AWS), SFTP, etc.
Linux only supports is a real blocker.
1 Like
I have no experience with it but rqlite is an alternative RAFT on SQLite that does support Windows and Linux.
1 Like
I found this post comparing litestream, dqlite and rqlite.
1 Like
Hmm, so you need yet another service.
@mmzeeman I wish I can achieve this in pure Erlang with minimum external (ideally 0) dependencies.
I’ve been glancing through the litestream implementation. It is entirely build in with the standard sqlite api. There is some parsing/recognizing of the WAL log file, but that is it.
The only thing missing in the current esqlite api is are a call to run checkpoints (sqlite3_wal_checkpoint_v2) and a hook which could trigger code on the Erlang side when something is written to the WAL log (sqlite3_wal_hook).
When those are in place, you can build a litestream like service, in erlang only.
1 Like
@mmzeeman This idea is simply fantastic, and the Erlang community can greatly benefit from it: a distributed esqlite !!!
You are a great salesman of my development time. 
At the moment I have some other priorities (putting webrtc support into zotonic), but I would gladly look into this in more detail later.
4 Likes
@mmzeeman I’ve talked to my line manager. The company I work for is OK to sponsor this work as long as the changes stay open source, and anyone can benefit from them. We can take this discussion offline if you’re interested.
4 Likes