Barrel - an embeddable, local-first database for AI applications (Erlang/OTP)

Your Libraries & OS Mentoring Libraries
, , , ,
1

Barrel 1.0 is out.

Some of you will remember Barrel as a CouchDB-inspired distributed database I worked on years ago. It went quiet for a long time. This release brings it back, rebuilt and refocused as an embeddable database for AI applications and agents, still pure Erlang/OTP on RocksDB.

The core is still the same: a document, its attachments (blobs), and now its vector are one record, under one id, written once. Not a document store next to a vector store next to a blob store stitched together in your app, but a single thing you read, search, and version. And it runs inside the Erlang BEAM, so there is no network hop and no separate database process.

Barrel is an umbrella of standalone OTP applications, each usable on its own and hex-publishable:

  • barrel : the embeddable facade, composes the layers under one id space
  • barrel_docdb : documents, HLC version-vector MVCC, changes feed, replication, attachments, TTL, retained history, BQL
  • barrel_vectordb : vector search (HNSW, DiskANN, optional FAISS), BM25, hybrid, quantization
  • barrel_embed : embedding generation (local Python, Ollama, OpenAI)
  • barrel_rerank : cross-encoder reranking
  • barrel_crypto : encryption-at-rest primitives
  • barrel_spaces : the agent layer (spaces, capability tokens, sessions, handoffs)
  • barrel_server : REST/JSON and MCP over livery, opt-in

A few things more things :wink: :

  • *HLC version vectors replace rev-trees. The old CouchDB rev-tree read-path baggage is gone; conflicts are last-write-wins by hybrid logical clock (causality-respecting), losing versions are superseded and stay queryable, and there is a merge hook. Multi-master, not last-push-wins.
  • Timeline. Branch a database via RocksDB checkpoints (hard-linked SSTs, near-instant), rewind to a point in time, and merge a branch back. Merge is just replication since the fork, so it reuses the sync engine.
  • BQL, a PartiQL dialect over documents, vectors, and full-text, parsed with leex/yecc and lowered onto the existing path index. Live SUBSCRIBE queries are Erlang processes pushing deltas.
  • Encryption at rest with per-database keys: RocksDB EncryptedEnv plus a sector cipher for the mmap’d flat files EncryptedEnv cannot cover.
  • MCP\ served through barrel_mcp on livery, databases and live queries exposed as MCP resources with subscriptions.
  • barrel-lite, a small TypeScript client that runs the same protocol in the browser (OPFS, offline-first, byte-exact codecs, a local BQL subset). Local data is a cache, sync is the durability. (Work in progress)

A taste of the embedded API:


%% runs in the BEAM, no external database process
{ok, Db} = barrel:open(<<"notes">>, #{
    embedding => #{ fields => \[<<"title">>, <<"text">>\], mode => async }
}),

{ok, \_} = barrel:put_doc(Db, #{
    <<"id">>    => <<"a">>,
    <<"title">> => <<"Foxes">>,
    <<"text">>  => <<"the quick brown fox">>
}),

%% hybrid vector + keyword search over the same records
{ok, Hits} = barrel:search_hybrid(Db, <<"fox">>, #{k => 5}),
%% BQL, a PartiQL dialect
{ok, Rows, \_} = barrel:query(Db, <<"SELECT id, title FROM notes LIMIT 10">>).

Build (you need Erlang/OTP, rebar3, CMake and a C/C++ compiler for the vectordb NIF):

rebar3 compile         # embeddable stack
rebar3 as server shell # add the REST/MCP server

It is built on livery, my web framework on the BEAM (HTTP/1.1, 2, 3, plus the MCP side), which I posted about here recently.

Apache 2.0, and it stays that way; you can run the whole thing yourself. There is a commercial side, Enki Bridge, for managed hosting and support, but that is optional and it is the same code.

Still early in places and moving fast, but 1.0 is real and in use. Feedback, questions, and criticism all welcome.

9 Likes