Hi guys, Hi @mmzeeman
Is it possible to use a RAFT implemtation (this one or this one) to distribute ESQLite (NIF driver) between 3 nodes?
If not, what can be done to make ESQLite
distributed as Khepri
or Mnesia
?
Thanks
Z.
Hi guys, Hi @mmzeeman
Is it possible to use a RAFT implemtation (this one or this one) to distribute ESQLite (NIF driver) between 3 nodes?
If not, what can be done to make ESQLite
distributed as Khepri
or Mnesia
?
Thanks
Z.
You mean something similar to Canocials dqlite? But then with Erlang? I don’t see why it shouldn’t be possible with Erlang. What would your intended use case be? It sounds like a fun project for sure.
Maas
Btw, dqlite also seems to have the single concurrent writer limitation sqlite has. So I’m not sure if this will be your intended use-case.
Maas
@mmzeeman yes, exactly as dqlite but in pure Erlang.
The idea is to be able to access ESQLite
database from any node for High Availability
.
@mmzeeman how much effort is needed to extend ESQLite
to support dqlite
? The downside with dqlite is that only runs on Linux and has many deps (io_uring, libraft, libsqlite3, etc.).
Indeed. Concurrent writes will be a plus. But my original idea is to replicate the database on multiple nodes for HA.
My expectation is that dqlite is pretty slow… It only supports single writers, meaning you have to wait for (network) consensus before every write you do. It really depends on your use case if this is worth it.
There are a couple of approaches you could think of:
There are also other approaches like what litestream does. It replicates the WAL log, so it is more a disaster recovery tool than a distributed database. I think it is possible to already use litestream with esqlite.
Both are valid approaches, but it depends on your use-case.
Maas
Thanks for sharing @mmzeeman. Didn’t know litestream.
How could you imagine using it with esqlite
?
Supposedly it works externally from your sqlite application. It runs as an external application, which opens the same database file as your own application. It then somehow captures WAL updates and replicates this to a different location.
FYI, dqlite also has a wire protocol. This makes it possible to integrate it into a pure erlang implementation without using nifs.
While checking the doc and playing with Litestream, I couldn’t find a way to perform replication the DB on another machine. What’s supported is S3 (AWS), SFTP, etc.
Linux only supports is a real blocker.
I have no experience with it but rqlite is an alternative RAFT on SQLite that does support Windows and Linux.
I found this post comparing litestream
, dqlite
and rqlite
.
Hmm, so you need yet another service.
@mmzeeman I wish I can achieve this in pure Erlang with minimum external (ideally 0) dependencies.
I’ve been glancing through the litestream implementation. It is entirely build in with the standard sqlite api. There is some parsing/recognizing of the WAL log file, but that is it.
The only thing missing in the current esqlite api is are a call to run checkpoints (sqlite3_wal_checkpoint_v2
) and a hook which could trigger code on the Erlang side when something is written to the WAL log (sqlite3_wal_hook
).
When those are in place, you can build a litestream like service, in erlang only.
@mmzeeman This idea is simply fantastic, and the Erlang community can greatly benefit from it: a distributed esqlite
!!!
You are a great salesman of my development time. At the moment I have some other priorities (putting webrtc support into zotonic), but I would gladly look into this in more detail later.
@mmzeeman I’ve talked to my line manager. The company I work for is OK to sponsor this work as long as the changes stay open source, and anyone can benefit from them. We can take this discussion offline if you’re interested.