This week I’ve been experimenting with using Google Sheets as a data store for a small Gleam web app.
Querying is obviously very limited, but as a zero cost place to store a few thousand rows of data it seems pretty good so far. And it comes with a built in admin and reporting interface! The oauth2 dance to get authenticated is a hassle though.
I was originally inspired by some people using Airtable to do the same, but I was put off by the very low API rate limit.
You might find this from Joe Armstrong interesting @lpil
Databases are basically rectangular tables of cells, where the cells contain very simple types like strings and integers - every time you access a row of an external database this list of cells has to be converted to beam internal data structures - this conversion is extremely expensive.
The best way to persist data is in a process - then no conversion is needed, but this is not fault tolerant - so you need to keep a trail of updates to the data and store this on disk.
Often you don’t need a database for example you might like to have a system where you store all the user data as in the file system with “one file per user” this will scale very nicely - just move the files to a new machine if you need more capacity.
Erlang has two primitives term_to_binary and the inverse binary_to_term that serialise any term and reconstruct it - so storing complex terms on disk is really easy.
I have mixed feelings about databases, they are great for aggregate operations (for example, find all users that have these attributes) but terrible for operations on individual users (where a single file per user is far better).
If I were designing a new system I’d go for ‘one file per user’ as much as possible and try to limit databases for operations over all users.
I let it print
In my old company, each of our team made another cup, with logos from tech that got forbidden by the management. On my cup is RabbitMQ and Erlang, on another cup was kube and gRPC…
And i am still trying to think of the best way to cross compile erlang and embed it in a release in the Nixpkgs ecosystem because i want to be able to produce release artefact for both Graviton2 and x86 for AWS. No good solution yet.
Reading Erlang syntax. Even if Elixir is enough in its own, maybe one day i may be stuck with an Elixir dependency that has sprinkles of Erl syntax. Wouldn’t hurt to learn it sooner than later.
Also, do we have Erlang version of Dave Thomas’ Exexif Elixir library for extracting exif data from an image?
The main Erlang/OTP project I’m involved with would be ejabberd, but lately I got sidetracked to work on a TURN server called eturnal. I like how the actual protocol code is less than 2k lines of straightforward Erlang and yet offers everything modern A/V clients need. It’s being used in production by various sites and seems to scale well. I’m actually pondering with supporting proper release upgrades given that those would be especially useful for TURN, where service restarts kill the running calls.
helping to maintain all Erlang stuff under Miniclip · GitHub, with a few other colleagues
Internal - company (@Miniclip)
previously, creating/maintaining several game backends (distributed Erlang) + backoffices (HTTP API using cowboy)
maintaining a consents’ manager + privacy framework (HTTP API using cowboy)
maintaining a (legacy) mass push notifications’ system
creating/maintaining the main CI-specific libraries used by all others
previously, creating/maintaining over 80 libraries that support the previous (e.g. Datadog client, JWT wrapper, Redis wrapper, Facebook API wrapper, Google API wrapper, Apple API wrapper, AWS+erlcloud stateful wrapper, pub/sub, …)
right now i’m working on a library to do something similar to what pathom does in clojure. it started as an elixir project, but since i didn’t use anything particular to elixir, i rewrote it in erlang. i’m just finishing documenting it.
In the commercial realm, occasional freelance work, as long as it is interesting. (And as @juhlig said elsewhere, no company that values its pride should think of hiring me )
In the realm of FOSS, I mostly like to contribute to OTP and whateverother (Erlang-related) project sparks my interest. Generally, I like to find ways to poke holes in shiny things If I love them enough (like Erlang ), I also help in closing them up.
Tested a lot of packages which were entered for the SpawnFest 2022 contest as a judge. It’s a really good opportunity for me to learn what’s up in the world or Erlang, Elixir, and BEAM.
Just finished reading Gene Sher’s book “Handbook of Neuroevolution Through Erlang”. Good book, very interesting. Wish the code base was updated as it is a decade old! But still would recommend the book for those interested in using Erlang for ML. It definitely has some advantages.
I have worked with distributed systems and familiarized myself with various problems that occur in distributed systems. Attempting to solve them using Erlang.
Trying to use more what otp has to offer by using other behaviours such as gen_statem and gen_event which are not used so often.
We’re building an Ethereum Virtual Machine indexer using Erlang with a CQRS architecture. Devs can subscribe to events for a particular smart contract (like an ERC-20’s transfer events) and keep track of state of things like wallet balances. Using the distributed capabilities built into Erlang, the dev can have their entire app model running locally rather than on the remote server so they can hit the service as hard as they want and it won’t effect other users.
Wondering how to improve the performance of the scenario: Why is there no ordered_bag type ETS table?. Thinking whether it is feasible is add update_spec to ETS, similar to match_spec but for in place updating.