Personally, I think of FreeBSD as a box of Legos for making your own operating system & user space. You can build the entire OS from scratch very easily and end up with an ISO or USB image ready for deployment, with the precise packages and config files already embedded, much more easily than an equivalent Linux approach. It’s ideal for appliances, but obviously also as a general-purpose desktop and laptop OS (like right now).
Most of the time what I need is already in FreeBSD ports or easy enough to add things. Like lang/gleam Adding new ports is not too difficult, and like most things the more you do the easier it gets. Adding go/python/perl/rust/erlang apps is an hour or two of massaging the build process to match. Most of this is encouraging various apps to behave like a proper UNIX process, and occasionally doing small upstream patches to ensure various things compile correctly on FreeBSD systems. Often, we can borrow these from NetBSD and OpenBSD if they already have the port.
But yes, for a jail there is limited set of pre-configured stuff, using packages. I think that BSD users tend towards the artisanal hand-crafted automation end of the “devops spectrum”. There is work on OCI compliant containers, but again these will be FreeBSD based containers and not linux ones. I don’t think the FreeBSD community wants to be a Linux Clone
For infrastructure, google has up to date cloud images, as does AWS also in bare metal.
As I’m broadly interested in low-level IP networking, I tend towards bare metal myself (all hail my tinfoil hat), at https://netactuate.com/ or https://metal.equinix.com/ . At the cheaper end Server auction - Hetzner Online GmbH is also fine, and https://digitalocean.com/ too.
For VPS hosting, https://vultr.com/ is great.
As my network connection in Austria sucks, I run FreeBSD on almost everything from a Raspberry Pi to a 32-core arm64 server in my cellar.
Elixir & erlang apps are built as boring releases, including erts at time of build, as a native FreeBSD package. A small shell script does the rebar/mix build, and massages erlang releases to behave like a proper UNIX app (read-only binaries, data in
/var/db/app/ and runtime stuff in
/var/run/app/*, configs in
/usr/local/etc/app/, use syslog for logging). Over time this has largely been refactored into a single script that’s almost identical across Elixir and Erlang apps. The end result is a standard package that is installed in the usual
pkg add ... way.
Each app is in its own jail, usually across multiple servers. Each jail has a private mesh VPN that links jails together across servers, it’s set up as IPv6 only, and erlang distribution works perfectly across that. ZeroTier has an inbuilt firewall so additional network restrictions are placed across these nodes to restrict traffic.
Installing and configuring our apps is as easy as:
### install the package
# pkg install -r private-repo my-app
### add some config files like sys.config or similar
# cp configs* /usr/local/etc/my-app/
### go go go
# service my-app start
These are done in ephemeral jails, usually with some form of network clustering (like global anycast, BGP + ECMP networking) through to haproxy in front, and then your usual erlang/elixir/phoenix distribution spreading state across nodes.
Services like CouchDB have a jailed zfs dataset that holds the data but the container itself is entirely reproducible.
Patching and Upgrades
Upgrades and patching are a simple ansible playlist that does this:
- select a single FreeBSD node (jail host)
- turn off BGP (so this node is taken out of clusters)
- turn off haproxy (so jails drain their traffic)
- destroy all jails
- rebuild all jails
- restart jails
- wait until jailed services are up
- turn on haproxy & then BGP
- loop to next node
For building images, I have a small CI system that runs a simple build.sh script in a container after every git push, and after every git tag we push to production.
I don’t yet have the need for automating this at 1000s of servers scale, but I’ve worked with places that do this. Ansible isn’t quite the right tool for that anymore, but there are quite a few that are available that do help with this, like Nomad. All the common FreeBSD jail tools have support to plug into things like Nomad. This is not the polished Kubernetes experience you would have, though.
I like @sg2342’s approach with signed datasets and so forth (please tell us more) but I am not sure this would introduce much more simplicity.
I started out using packaged erts (because back in 2015-2016 OpenSSL security patches were coming out almost every week) but these days its less Ops work to include erts and just re-tag / re-build / re-deploy automatically if OTP patches are required.
Jails and Linux Containers are quite different in how I use them. On FreeBSD, the firewall, native command line tools, the filesystem, and network stacks are very very tightly integrated with jails. While you can choose to build a jail with any combo of FS+Network+Process restrictions, a jailed filesystem is not accessible from other jails, and a jail can have a full “virtual” network stack of its own.
I have 2 ways for using jails - the production side, as above (jailed storage, jailed network stack, jailed apps), and the local side.
The local side is a small fish shell script which spins up a container using a simple template, and gives me a tmux session in it.
I’ll typically have 3-4 of these at any one time, with a nullfs mounted “loopback” filesystem from my elixir/erlang app repo, to build and test in isolation. Sometimes I have these jailed filesystems over NFS to a remote system.