Your point of view on DevOps in pure Erlang/OTP world

I’m very curious if anyone here developed highly scalable web applications/APIs in pure Erlang/OTP and if their DevOp portion of their work is different from running dockerized applications within Kubernetes. My newbie point of view is that, Erlang/OTP provides distribution, and networking out of the box, unless there is major security issues, I wouldn’t complicated it with docker and k8s. Could someone shine some light into my confusion here?

3 Likes

Hello,
I have no professional experience but the book Adopting Erlang suggests it in it’s production section.

2 Likes

From mentioned link I gave my thinking here, again I might be wrong as Im still a newbie in Erlang/OTP.

Kubernetes provides a scheduler for efficiently packing your programs, 
as containers, on physical or virtual machines.

I disagree as a positive point: we got releases for packaging and could use versioning for consistency across multiple machines, and it is unfair to k8s to say ‘on physical or virtual machine’ the whole idea is to abstract from that; and… Erlang/OTP could run in various env as well.

Using containers eases deployment by bundling all dependencies (think OpenSSL in an Erlang 
release’s case) into a single image and isolation that protects against dependency conflicts between
services running on a machine.

Again: I could have multiple clusters on one machine, and why do I need to use multiple version of the same tool/lib/languages on one machine… doesn’t make sense in Erlang or at least I could avoid that. Yes if I had legacy Python 2.7 and Python 3.7 codebase maybe… but still not a good practice to follow anyway.

Additionally, deployment and management becomes consistent across the services in your production 
environment no matter the language or runtime they are built with. When you integrate your Erlang
service with common monitoring and tracing services, you will also ensure it’s no longer the oddball
your coworkers dread having to deal with.

I want to use Erlang/OTP for all the backend so that doesn’t apply. I like being an odball, if Erlang is one :smiley: .

2 Likes

I am going to repeat again that I don’t have professional experience with Erlang, Kubernetes nor Docker. Only home experience with Erlang.

I think Kubernets offers a second layer for manage the behavior of your nodes and allow you to automate the deployments, which is something that we don’ t have in Erlang.

Docker is Docker, I mean, It offers to Erlang the same that it offers to another language. For example, if you have an heterogeneous cluster, you need to make the release in each machine, but with docker you don’t need it, or if you have to remap ports, it is easy with Docker.

And I “think” there are some C libraries that don’t come with the release so you need to depend on the host.

Maybe if you need to do a machine learning application, you will need a python port or a python service so using docker to manage the enviroment is not that bad.

2 Likes

There certainly is a little overlap with OTP and k8s, but there’s other things you’ll also want in your production system. How do you roll out new code versions? How do you get those updates onto servers? How do you monitor your applications and collect logs? Do you need to perform load balancing and handle traffic over HTTPS?

OTP largely doesn’t help with these, so something else is required. k8s is one of many tools you could use to tackle these problems, depending on your team and what your constraints and needs are it could be a good option.

4 Likes

Lets assume everything is written in Erlang/OTP and related Erlang-based frameworks (my intention really), then adding k8s to the mix will create unnecessary complexity in my opinion. With a load balancer in front (Nginx), and adding Grafana to it, I can get visitation logs, then I can run Erlang clusters for serving those requests. As it goes for logs, I can tab to the resources similar what Observer is doing. For pushing new versions, Rebar3 config file comes with versions, every merge to master I can then push and run in the target server.

3 Likes

For sure. Grafana + nginx + some other mechanism for moving deployment artefacts to servers + some way of performing upgrades would be an alternative to k8s. In terms of simplicity I find both approaches to be about the same, though there may be more setup work to do if you take a modular approach like that rather than an all-in-one system like k8s, swarm-mode, or dokku.

I think it comes down to what your team is more comfortable with. There’s not any clear advantage to one or the other outside of that.

4 Likes

This is true for any language/runtime :). Using Erlang doesn’t make the complexity of k8s any more or less worth dealing with. Reasons to be using k8s are separate from the language used and more often than not it is not the right tool for the job.

4 Likes

What sort of complexity k8s completely solves that then we don’t need to address with Erlang? K8s is for container orchestration, why do I even need to add Erlang app inside a container, so when the container fails in a pod, I need k8s to restart the pod. I don’t see a benefit a container engine or k8s for that matter to manage my Erlang cluster just because Erlang itself could handle it.

Other languages need k8s because their VM cant handle distributed and concurrent computing at the production level.

One more point here, to me it is very hard to run a k8s as a distributed cluster in 3+ location points. However it is a one liner to link and connect two Erlang clusters in two different machines.

Maybe Im naive and yes Im new to Erlang, but I still don’t see a point on running k8s.

2 Likes

Container orchestration with k8s will also let you do things like choose placement options (how spread out across AZs do you want to be, can containers for given apps be co-located or should they not be), CPU and memory consumption limits, image caching for faster scaling, DNS registration handling, role management in cloud environments like AWS, limits around RBAC, secret handling and setup, configuration of (some) healthcheck/liveness probes, managing scaling and replacement policies, and to do so in a consistent manner across multiple languages if you need them, to name a few.

Erlang can do some things, and containers are helpful for a few others (if you don’t have a container, then you need a series of scripts or tools to properly set up the OS that is going to run your Erlang nodes), and container orchestration can be useful to deal with the allocation and setup of these containers.

I’m not a fan of kubernetes, but I nevertheless use it in prod because there are things that container orchestration does that have zero overlap with what Erlang lets you do, the same way there are things Erlang covers that have zero overlap with what containers or k8s let you do.

They’re not even operating on the same level for most features/cases, and so they shouldn’t be considered to be mutually exclusive as far as I’m concerned.

6 Likes

In production you still have something, like systemd, that restarts the Erlang node if it crashes. Yes, there is heart also, but that is still a separate executable and part of my point that the Erlang node doesn’t supervise itself. I discuss this in Adopting Erlang :slight_smile:

You are talking about two different levels of the network. The Erlang clustering only works if the networking between the different machines already works, which is what is done (though virtual networks) when you setup k8s.

3 Likes

This has another important point in it: You won’t be running k8s in a one person shop. You may technically be running on k8s even in a one person shop, like if you deploy to fly.io, but where the complexity of k8s becomes most useful is in larger deployment, more than likely containing more than just Erlang nodes. In which case you use it because it is what the company is using :). The decision to use it or not doesn’t factor in your runtime.

3 Likes

First of all thanks for the feedback. Could you elaborate more about Erlang node and its self supervision and the problem it creates in production? Any link would be appreciated. Thanks.

I remember the time where k8s was not as a service, and it required expertise to set the master and workers, by using VMs (e.g., EC2s). Now it is one click away and yes companies have a DevOp teams that set things up by config files (best case scenario btw). I still however believe the complexity bite people at the end just because the apps that deployed to k8s are originally written in a single threaded languages (Python, Java, etc.) and having instances of them and let k8s handle multi threaded issues is an artificial way of solving a problem that provides unnecessary complexity and wasting money. I know companies that their bill of k8s are very high, something is not right here.

At least as a learning aim I would like to drop k8s & docker from my lingo and focus on Erlang/OTP that can handle all these issues on a failover manner across multiple servers and handle requests. I bet I can reduce the complexity and cost significantly.

1 Like

For “self supervision” I take it that refers to my mention of heart? Erlang -- heart

Note that Erlang’s distributed applications (not simply clustered nodes, but use of stuff like failover of applications) were not made for a dynamic world. If you have a use case where this isn’t an issue and you don’t have nodes joinng/leaving (either on purpose or through a netsplit) then yes, distributed applications may be the way to go, but it is not a common setup.

Either way you more than likely have no need for k8s, the point isn’t that anyone needs k8s it is just that k8s and Erlang don’t work at the same level and don’t change whether you’d benefit from using one or the other.

2 Likes