Strata - Schema migrations for Erlang

Your Libraries & OS Mentoring Libraries
,
1

Hi fellow Erlangers!

2 years ago I made this post, asking how people were handling schema migrations with their Erlang apps. (Actually I caused confusion by not being clear I was talking about schema migrations not database migrations, but luckily people understood my point!).

Since then I’ve been busy off and on iterating on a solution that I liked. My goals were to make it native to Erlang and easy enough to use where it felt competitive or similar to other language ecosystems.

After a long wait, I am happy to announce the release of strata.

Relevant repos:

  • strata - Generic schema migration library
  • rebar3_strata - Opinionated rebar3 plugin to make creating and managing schema migration modules more convenient. Optional.
  • strata_sqlite3 - Sqlite3 implementation of the strata protocol
  • strata_postgres - Postgres implementation of the strata protocol
  • strata_mysql - MySQL implementation of the strata protocol

Example app using strata, strata_sqlite3, and rebar3_strata.

Walkthrough:
A developer creates an Erlang module for each schema migration. The module contains up/0 and down/0. The module is associated with a timestamp. The timestamps are used to determine the order the migrations will need to run in. This configuration of erlang modules and timestamps is given to strata. As part of strata’s migration protocol, the config is used by the database specific “backend”. These backends contain the actual implementation of the migration handling. The backend does not have to be for a SQL database. It is possible to make one for other databases like Mnesia for example. The final product being an Erlang native schema migration process in only a few lines of code.

15 Likes
2

That’s nice!
I am currently playing with Erlang for a toy project using your nine library.

My current solution for db migrations involves an external tool, having something similar in Erlang is great!

I really love that uer is able to write migrations in SQL and not a DSL (I am part of the minority I suppose).
I think with ~b or """ the SQL reads even better.

Keep going and kudos for the effort!

2 Likes
3

This is great!

We have a similar app internally for Postgres, Cassandra and ElasticSearch.

I’m just adding a few notes about how we do things as they may be things you want to consider adding:

  1. migrations are named with a timestamp prefix so they’re logically ordered on disk. Individual migration records are not added to the schema_migrations table in the database until after they are successfully applied.
  2. new migration templates can be generated using emigrate:new(AppName, Backend, Name) or emigrate:new_file(AppName, Backend, Name) generates a new migration module in the AppName/Backend-specific migrate_path as {migrate_path}/{timestamp}_{name}.erl and;
  • if emigrate:new_file is used it generates {migrate_path}/{timestamp}_{name}_(up|down).sql files and the generated module loads the up/down sql from those, rather than having to edit SQL within the erlang module. Having external files does make editing/writing SQL easier
  1. application of migrations is done via our Makefile, supporting make {AppName}-migrate-{Backend} and {AppName}-migrate-{Backend}-transaction. The transaction version applies all pending migrations as a single transaction (this could probably be improved by having a ‘group’ tag for migrations to have all pending migrations with the same group name applied as a single transaction. Migrations can also be applied by calling enmigrate:migrate(AppName, Backend, [MigrationNames]). or enmigrate:migrate_all(AppName, Backend).. All migrations happen in a single transaction in the former.
  2. All configurations for backends are configured in the including applications sys.config eg
[
...
{emigrate, [
  {AppName, [
    {BackendName, [
      {migrate_path, "path/to/migrations"},
      {connection, ConnectionSettings}
    ]},
    ....
  ]}
]},
...
].
3 Likes
4

Thanks for the encouragement! I am with you in preferring plain SQL over a DSL. Generally, I prefer less abstraction. The triple quoted string pairs really well with this library so I’m happy that is a feature in Erlang now. When I first started this project it wasn’t available yet.

Previously, I had been using diesel, similarly to how you are using dbmate. Using an external tool totally works, the only downside is you have to either ship the tool with your Erlang release which is kind of awkward. Or figure out some other way to actually run your migrations. With strata you just need to ship the Erlang release and nothing else. Then the migrations can be run with the shell or on app startup.

5

Very cool! I’m not surprised to find out someone has a similar tool. I really feel like strata was a natural solution to a very common problem.

This is a good list, and I think strata already has most of these things or could be easily modified to do so.

  1. strata doesn’t make an opinion about name ordering, but it does assume that name has an associated timestamp. rebar3_strata makes modules with a timestamp suffix and uses that suffix as the timestamp. So very similar to your approach but the suffix instead of the prefix. Double checked since it’s been a while since I made some of the original code, but strata does make sure to only add the migration after the migration has completed. You can check out this function here. So every database backend will function this way since it is implemented at this level.

  2. rebar3_strata has a command new that does something similar.

  3. rebar3_strata could be modified to do something similar and then in the up/0 & down/0 methods they could read that sql file and return the string. Would the sql files be stored in priv? That’s the only way I could think they would still get shipped as part of the Erlang release.

  4. While strata can support multiple database backend types in a single app, rebar3_strata doesn’t handle this scenario yet. This shows me I really need to treat it as a config problem. So I’ll keep pondering on this, but should be able to come up with something quickly once I decide on the direction.

1 Like
6

Thank you so much for releasing this, I’m using erlang-pure-migrations in a hobby project, but was dying to see a more up-to-date solution for migrations in Erlang (with better tooling). One the issues I’m having with my current lib is that it requires some hacks to work in a recent version of postgres.

I will play with this during the weekend

1 Like