I’m very curious if anyone here developed highly scalable web applications/APIs in pure Erlang/OTP and if their DevOp portion of their work is different from running dockerized applications within Kubernetes. My newbie point of view is that, Erlang/OTP provides distribution, and networking out of the box, unless there is major security issues, I wouldn’t complicated it with docker and k8s. Could someone shine some light into my confusion here?
4 Likes
Hello,
I have no professional experience but the book Adopting Erlang suggests it in it’s production section.
3 Likes
From mentioned link I gave my thinking here, again I might be wrong as Im still a newbie in Erlang/OTP.
Kubernetes provides a scheduler for efficiently packing your programs,
as containers, on physical or virtual machines.
I disagree as a positive point: we got releases for packaging and could use versioning for consistency across multiple machines, and it is unfair to k8s to say ‘on physical or virtual machine’ the whole idea is to abstract from that; and… Erlang/OTP could run in various env as well.
Using containers eases deployment by bundling all dependencies (think OpenSSL in an Erlang
release’s case) into a single image and isolation that protects against dependency conflicts between
services running on a machine.
Again: I could have multiple clusters on one machine, and why do I need to use multiple version of the same tool/lib/languages on one machine… doesn’t make sense in Erlang or at least I could avoid that. Yes if I had legacy Python 2.7 and Python 3.7 codebase maybe… but still not a good practice to follow anyway.
Additionally, deployment and management becomes consistent across the services in your production
environment no matter the language or runtime they are built with. When you integrate your Erlang
service with common monitoring and tracing services, you will also ensure it’s no longer the oddball
your coworkers dread having to deal with.
I want to use Erlang/OTP for all the backend so that doesn’t apply. I like being an odball, if Erlang is one 
.
3 Likes
I am going to repeat again that I don’t have professional experience with Erlang, Kubernetes nor Docker. Only home experience with Erlang.
I think Kubernets offers a second layer for manage the behavior of your nodes and allow you to automate the deployments, which is something that we don’ t have in Erlang.
Docker is Docker, I mean, It offers to Erlang the same that it offers to another language. For example, if you have an heterogeneous cluster, you need to make the release in each machine, but with docker you don’t need it, or if you have to remap ports, it is easy with Docker.
And I “think” there are some C libraries that don’t come with the release so you need to depend on the host.
Maybe if you need to do a machine learning application, you will need a python port or a python service so using docker to manage the enviroment is not that bad.
2 Likes
There certainly is a little overlap with OTP and k8s, but there’s other things you’ll also want in your production system. How do you roll out new code versions? How do you get those updates onto servers? How do you monitor your applications and collect logs? Do you need to perform load balancing and handle traffic over HTTPS?
OTP largely doesn’t help with these, so something else is required. k8s is one of many tools you could use to tackle these problems, depending on your team and what your constraints and needs are it could be a good option.
5 Likes
Lets assume everything is written in Erlang/OTP and related Erlang-based frameworks (my intention really), then adding k8s to the mix will create unnecessary complexity in my opinion. With a load balancer in front (Nginx), and adding Grafana to it, I can get visitation logs, then I can run Erlang clusters for serving those requests. As it goes for logs, I can tab to the resources similar what Observer is doing. For pushing new versions, Rebar3 config file comes with versions, every merge to master I can then push and run in the target server.
4 Likes
For sure. Grafana + nginx + some other mechanism for moving deployment artefacts to servers + some way of performing upgrades would be an alternative to k8s. In terms of simplicity I find both approaches to be about the same, though there may be more setup work to do if you take a modular approach like that rather than an all-in-one system like k8s, swarm-mode, or dokku.
I think it comes down to what your team is more comfortable with. There’s not any clear advantage to one or the other outside of that.
5 Likes
This is true for any language/runtime :). Using Erlang doesn’t make the complexity of k8s any more or less worth dealing with. Reasons to be using k8s are separate from the language used and more often than not it is not the right tool for the job.
5 Likes
What sort of complexity k8s completely solves that then we don’t need to address with Erlang? K8s is for container orchestration, why do I even need to add Erlang app inside a container, so when the container fails in a pod, I need k8s to restart the pod. I don’t see a benefit a container engine or k8s for that matter to manage my Erlang cluster just because Erlang itself could handle it.
Other languages need k8s because their VM cant handle distributed and concurrent computing at the production level.
One more point here, to me it is very hard to run a k8s as a distributed cluster in 3+ location points. However it is a one liner to link and connect two Erlang clusters in two different machines.
Maybe Im naive and yes Im new to Erlang, but I still don’t see a point on running k8s.
3 Likes
Container orchestration with k8s will also let you do things like choose placement options (how spread out across AZs do you want to be, can containers for given apps be co-located or should they not be), CPU and memory consumption limits, image caching for faster scaling, DNS registration handling, role management in cloud environments like AWS, limits around RBAC, secret handling and setup, configuration of (some) healthcheck/liveness probes, managing scaling and replacement policies, and to do so in a consistent manner across multiple languages if you need them, to name a few.
Erlang can do some things, and containers are helpful for a few others (if you don’t have a container, then you need a series of scripts or tools to properly set up the OS that is going to run your Erlang nodes), and container orchestration can be useful to deal with the allocation and setup of these containers.
I’m not a fan of kubernetes, but I nevertheless use it in prod because there are things that container orchestration does that have zero overlap with what Erlang lets you do, the same way there are things Erlang covers that have zero overlap with what containers or k8s let you do.
They’re not even operating on the same level for most features/cases, and so they shouldn’t be considered to be mutually exclusive as far as I’m concerned.
6 Likes
In production you still have something, like systemd, that restarts the Erlang node if it crashes. Yes, there is heart also, but that is still a separate executable and part of my point that the Erlang node doesn’t supervise itself. I discuss this in Adopting Erlang 
You are talking about two different levels of the network. The Erlang clustering only works if the networking between the different machines already works, which is what is done (though virtual networks) when you setup k8s.
4 Likes
This has another important point in it: You won’t be running k8s in a one person shop. You may technically be running on k8s even in a one person shop, like if you deploy to fly.io, but where the complexity of k8s becomes most useful is in larger deployment, more than likely containing more than just Erlang nodes. In which case you use it because it is what the company is using :). The decision to use it or not doesn’t factor in your runtime.
3 Likes
First of all thanks for the feedback. Could you elaborate more about Erlang node and its self supervision and the problem it creates in production? Any link would be appreciated. Thanks.
I remember the time where k8s was not as a service, and it required expertise to set the master and workers, by using VMs (e.g., EC2s). Now it is one click away and yes companies have a DevOp teams that set things up by config files (best case scenario btw). I still however believe the complexity bite people at the end just because the apps that deployed to k8s are originally written in a single threaded languages (Python, Java, etc.) and having instances of them and let k8s handle multi threaded issues is an artificial way of solving a problem that provides unnecessary complexity and wasting money. I know companies that their bill of k8s are very high, something is not right here.
At least as a learning aim I would like to drop k8s & docker from my lingo and focus on Erlang/OTP that can handle all these issues on a failover manner across multiple servers and handle requests. I bet I can reduce the complexity and cost significantly.
2 Likes
For “self supervision” I take it that refers to my mention of heart? Erlang -- heart
Note that Erlang’s distributed applications (not simply clustered nodes, but use of stuff like failover of applications) were not made for a dynamic world. If you have a use case where this isn’t an issue and you don’t have nodes joinng/leaving (either on purpose or through a netsplit) then yes, distributed applications may be the way to go, but it is not a common setup.
Either way you more than likely have no need for k8s, the point isn’t that anyone needs k8s it is just that k8s and Erlang don’t work at the same level and don’t change whether you’d benefit from using one or the other.
3 Likes
I think the biggest benefit of using k8 is that there are some well-defined best practices for workflows and security around containers which “assist” in the division of work between devops and developers (this division has always been a problem for me in the corporate world though).
Generally, the common software development practices (agile and waterfall) will result in teams targeting a CI based workflow with testing and deployable assets across multiple environments (dev/test/uat/prod). This sits nicely within the deployment profiles of such a system that targets blue-green deployments etc. Now i’d imagine deploying a beam application through this same methodology would work just as well as any other language/runtime (which is the beauty of this ecosystem) but as noted the topic speaks to a pure erlang/OTP world… and because of that I think perhaps reviewing the role of devops needs to be reviewed as I’d imagine Joe and co. didn’t have the role segregation which we have today.
Perhaps in a pure erlang world the devops function comes down to just provisioning a beam vm runtime with appropriate networking and security configured? (I’m sure there is more to it than that and as pointed out this can get complicated with respect to AZ’s and scaling etc. which I’m sure there is an art too much like all cloud offerings)
I don’t know enough about what’s provided by OTP or out there as a library which would satisfy something like the k8 ‘service-mesh’ capabilities, so the blue-green deployment scenario isn’t the ideal path for an otp deployment to my knowledge, but I’m sure it can be satisfied in other ways (since the production runtime can be a checked out git branch and things can simply be switched over and (re)loaded at runtime (how these paths can be conditionally executed while leaving existing inline with a blue-green deployment is an exercise left to the reader) but I think the power is there just exposed differently and requires perhaps some process around how these things should be changed in a production environment (I can’t imagine anyone in $JOB would allow me to jump on a production system and execute code like this… although it happens more than never)
I guess a lot of this stuff comes back to process and how much you trust our team. In larger teams i’m very much an advocate to the ‘defacto’ process which leans against attempting to deploy well defined and tested artifacts and having little/no user interaction on production system deploys but when the team is smaller (trust is higher) I think having reloadable production code (based off a source of truth ie git) is more than enough and the production deployment automation can be sidelined if so desired.
1 Like