Nerves.System.Nix - seeking guidance in how the Nerves platform works

Questions / Help
1

TL;DR: Personal toy project idea, asking for some guidance in how the existing Nerves platform works.

I have been toying with the idea of using Nix/NixOS to create a Nerves system, instead of using Buildroot. (As opposed to compiling Buildroot on NixOS, which is possible and I’m doing that for a project I’m working on. Required a couple of changes along the lines of file isn’t in /usr/bin/, nor is install, etc.) This should also give me some opportunity to get to understand how Nerves works more

I’m just putting this here in case people have ideas on it, though I can’t guarantee I’ll be spending a lot of time on it so it might not happen too fast.

Usual terminology when discussing Nix/NixOS/Nixpkgs:

  • Nix :: a package manager, can be installed on Linux and MacOS (also WSL2). Takes a more formalized approach to package management which may make things like cross-compilation easier. Has the same problem as other things that take a formal approach though – may make some simpler things sounds more difficult.
  • NixOS :: a Linux distro written as a Nix package. A Linux distro is after all (more or less), the kernel, bootloader, system services (systemd in this case) and a set of installed packages. (Usually people talk about it in terms of the OS they are using, but I have used it as an OS I’m cross-compiling, that needn’t be done from NixOS, any OS will do as long as it has Nix installed. You can absolutely use Nix without NixOS to build NixOS (e.g. build VM images, or a particularly fun case is called NixOS lustrate – where NixOS takes over the existing OS :p).
  • Nixpkgs :: A huge set of packages (just look at https://repology.org/). Not all of them will cross-compile (looking at you, spidermonkey! – though maybe that’s now fixed, and polkit now supports duktape I think).

I have some prior experience with maintaining Nix packages, and with using Nix to cross-compile a NixOS for a Raspberry Pi.

Pros & Cons of Nix

  • + Individual, isolated compilation of separate packages, including good tracking of when things don’t get changed. Currently my experience of buildroot is that the whole system gets rebuilt, especially in a CI context where you want a clean build each time. With Nix each package is built in isolation (as a clean build) so this shouldn’t be such a problem.
  • + Declarative description of the system. Not sure if it’ll make sense to reuse NixOS or make something lighter (NixOS depends on systemd/is a bunch of systemd packages, plus uses Perl for “system activation”, kind of a step akin to install/upgrade in other Linux distributions). To be honest, I haven’t used Buildroot in this sense, only used someone else’s Buildroot config, so I don’t know how good/bad Buildroot is from this point.
  • + It’s own management of artifacts/caching. I mean Nerves has a good go at this, though it doesn’t build things hermetically, i.e. things not in nerves_package under checksum can still end up in the resulting image. Also I’m not sure the URL of the artifact (or at least the root of the path) should be stored in the artifact itself, this makes mirrors a pain.
  • Not really a plus, more of a comment, but it can also do the toolchain not just the cross-compilation using the toolchain.
  • - Nixpkgs not primarily being targeted at cross-compilation means package updates are more likely to break cross-compilation. Perhaps this needs a branch/tags of nixpkgs that build a usable system.

Here is my understanding of how things work:

Here is what I plan on doing:

  • Getting a simple system to cross-compile for a Raspberry Pi
  • Minimising that system to be a similar size to the existing Nerves Raspberry Pi system (~25MiB)
  • Getting Nerves to use that system
5 Likes
2

Might be worth posting this on the Nerves Forum on EF Robert?

Unless @fhunleth spots it and has any thoughts…

2 Likes
3

As far as I know, Erlang within Nixpkgs also isn’t ready for cross-compilation.

@DianaOlympos once explored the topic.

In general there doesn’t currently seems to be a great focus for cross-compilation within the Nix ecosystem. Even if it would be a real great opportunity. At least for Erlang it would be great.

As I’m not a Nerves user, I can’t help really.

2 Likes
4

I can concur, cross compiling OTP itself is not ready on nixpkgs. I need it and i would love if someone tried to get it to work. Last time i tried, i got bogged down pretty fast and could not understand why.

I searched for help, but it seems noone in the nixpkgs community really look at cross-compiling anymore. That said, i would love to help if someone that knows better take a crack at it or is ready to support.

4 Likes
5

Hi Rob! Good to bump into you again! :smiley:

Good luck in your nerves quest

3 Likes
6

Hello @robert.kovacsics, maybe it is out of topic but are you claiming you have a working buildroot setup on NixOS that works with Nerves? Would you mind sharing it? :smiley:

1 Like
7

Sure it’s an aside but I’m happy to share. Be warned it’s more like a bunch of hacks to get something working rather than a proper solution, because buildroot has a bunch of hardcoded expectations about paths. I should look at using FHS user envs but because this works and it’s only for me for developing I haven’t yet.

This is my shell.nix, though it could be simpler:

The most important bit is probably the hardeningDisable = [ "format" ]; because that caused buildroot to fail to compile because it had format warnings it was treating as errors.

<shell.nix>=

{ pkgs ? import <nixpkgs> { } }:

with pkgs;

let
  inherit (lib) optional optionals;

  # We need to use the same Erlang/OTP as we do for the Nerves cross-compile,
  # there is a refactor between the Nerves Erlang/OTP 24.0.5 and the Nixpkgs
  # Erlang/OTP 24.2 which causes a problem of `-record(cert,` already defined.
  # It is defined in the file
  # "${nixpkgs.erlangR24}/lib/erlang/lib/public_key-1.11.3/include/public_key.hrl"
  # which the Nerves cross-compile Erlang uses, because `erlc` uses the OTP it
  # was compiled with, and Erlang when cross-compiling doesn't generate a
  # bootstrap `erlc` binary (which should then use the OTP that is being
  # compiled, which has the right public_key.hrl not the one in the nix store)
  erlangR24_0_5 = callPackage ./nix/erlangR24_0_5.nix { };
  erlangR24_0_5-packages = beam.packagesWith erlangR24_0_5;
  elixir_1_12 = erlangR24_0_5-packages.elixir_1_12;
in

mkShell {
  buildInputs = [
    elixir_1_12
    erlangR24_0_5
    fwup
    perl
    nodejs-14_x
    python3

    # Programs for nerves_system_br
    which
    gnused
    file
    patch
    perl
    gnutar
    wget
    cpio
    unzip
    rsync
    bc
    # Using the pkg-config-unwrapped ensures we use the libnl from the nerves
    # system
    pkg-config-unwrapped
    squashfsTools
  ]
  ++ optional stdenv.isLinux inotify-tools;

  hardeningDisable = [ "format" ];

  LOCALE_ARCHIVE =
    if pkgs.stdenv.isLinux then
      "${pkgs.glibcLocales}/lib/locale/locale-archive" else "";

  ERL_AFLAGS = "-kernel shell_history enabled";

  shellHook =
    ''
      # ERL_LIBS causes a load of compile warnings (warning: this clause cannot
      # match because a previous clause at line 1 always matches) in the standard
      # library. It appears to be because things are evaluated twice.
      # An actual export -n isn't inherited properly so we just set it blank.

      export ERL_LIBS=""
    '';
}

<nix/erlangR24_0_5.nix>=

{ erlangR24, fetchFromGitHub }:

erlangR24.overrideAttrs (drv: rec {
  version = "24.0.5";
  name = "erlang-${version}";
  src = fetchFromGitHub {
    rev = "OTP-${version}";
    sha256 = "153kg6351yrkilr4gwg1jh7ifxpz9ar664mz7vdax9sy31q9i771";
    owner = "erlang";
    repo = "otp";
  };
})

Then there is a couple of buildroot patches. For you it’s probably going to be different depending on the version of nerves/buildroot you are using, so I’ll first say how I arrived at those patches.

First, make sure you have this change artifact.ex: Print string errors instead of &inspect/1 · nerves-project/nerves@84e57ba · GitHub e.g. after doing MIX_ENV=prod MIX_TARGET=... mix deps.get edit the deps/nerves/lib/nerves/artifact.ex file – it will make it far more obvious what’s going wrong.

Then once you try to do MIX_ENV=prod MIX_TARGET=... mix firmware it will fail (due to no /usr/bin/file), but create deps/nerves_system_br/buildroot, and you can apply this patch`

diff --git a/package/autoconf/autoconf.mk b/package/autoconf/autoconf.mk
index 336ac59..ec1034f 100644
--- a/deps/nerves_system_br/buildroot/package/autoconf/autoconf.mk
+++ b/deps/nerves_system_br/buildroot/package/autoconf/autoconf.mk
@@ -23,4 +23,4 @@ $(eval $(host-autotools-package))
 # variables used by other packages
 AUTOCONF = $(HOST_DIR)/bin/autoconf -I "$(ACLOCAL_DIR)" -I "$(ACLOCAL_HOST_DIR)"
 AUTOHEADER = $(HOST_DIR)/bin/autoheader -I "$(ACLOCAL_DIR)" -I "$(ACLOCAL_HOST_DIR)"
-AUTORECONF = $(HOST_CONFIGURE_OPTS) ACLOCAL="$(ACLOCAL)" AUTOCONF="$(AUTOCONF)" AUTOHEADER="$(AUTOHEADER)" AUTOMAKE="$(AUTOMAKE)" AUTOPOINT=/bin/true GTKDOCIZE=/bin/true $(HOST_DIR)/bin/autoreconf -f -i
+AUTORECONF = $(HOST_CONFIGURE_OPTS) ACLOCAL="$(ACLOCAL)" AUTOCONF="$(AUTOCONF)" AUTOHEADER="$(AUTOHEADER)" AUTOMAKE="$(AUTOMAKE)" AUTOPOINT=true GTKDOCIZE=true $(HOST_DIR)/bin/autoreconf -f -i
diff --git a/package/fakedate/fakedate b/package/fakedate/fakedate
index a64d9b9..05f3767 100755
--- a/package/fakedate/fakedate
+++ b/package/fakedate/fakedate
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/bin/sh
 # vim: set sw=4 expandtab:
 #
 # This program is free software; you can redistribute it and/or modify
diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
index 1954f03..be1e044 100755
--- a/deps/nerves_system_br/buildroot/support/dependencies/dependencies.sh
+++ b/deps/nerves_system_br/buildroot/support/dependencies/dependencies.sh
@@ -72,7 +72,7 @@ check_prog_host "sed"

 # 'file' must be present and must be exactly /usr/bin/file,
 # otherwise libtool fails in incomprehensible ways.
-check_prog_host "/usr/bin/file"
+# check_prog_host "/usr/bin/file"

 # Check make
 MAKE=$(which make 2> /dev/null)

Then basically as you continue, it might have one or two FHS expectations, e.g. dtc/Makefile at main · dgibson/dtc · GitHub but those should be easy to spot and fix.

As I said at this point it is a hack for personal use, so your mileage may vary, but if it doesn’t scare you away then perhaps it might help. And probably creating a buildFHSUserEnv would allow all of the patch steps to be not required, I might give this a try now

3 Likes
8

Amazing, thank you @robert.kovacsics ! I was not aware of the buildFHSUserEnv function, I’m giving it a try as well.

2 Likes
9

Hi @robert.kovacsics,

Sorry for the much delayed reply. I don’t log into the Erlang forum as much as I probably should.

I feel like your Nix idea is really interesting.

There are two places to integrate it with Nerves:

  1. As a replacement for Buildroot and nerves_system_br
  2. Augment Mix releases to pull Nix packages

You’re going down option one which is probably the more obvious one. We started down that route as well. Historically Buildroot was used out of convenience to build the Linux kernel, bootloader and Erlang. We could have rolled our own shell scripts, but there are many gotchas. Buildroot is just outstanding, imho, in its hardware support and attention to detail in getting cross-compilation right. As a bonus, if you want an extra C library or two, you can get those.

The goal with Nerves has always been to minimize the time spent on the non-Erlang/Elixir side. I.e., you port to a device, get any C/C++ libraries and programs you need for your project, build the Nerves System and then try to spend as much time as possible in Erlang and Elixir and use mix to build everything. If I had my way, the Nerves systems would be much smaller. I’ve tried to trim them down more, but it’s makes supporting new users hard especially when so little is available.

If I could suggest exploring Nix integration with mix releases, I think that would work better with Nerves. For one, you could let Buildroot handle the Linux kernel, bootloader, and other low level compilation which I’d expect will be quite time consuming to maintain long term. I also don’t think those pieces are as interesting from a Nix perspective.

I think that it would be really cool if I could add Nix packages to a dependency list kept in the mix.exs and then they’d be included in the OTP release. Obviously, lots of details would need to be figured out here. However, it just seems way easier to list the packages here and have them downloaded automatically similar to what we already do with Erlang and Elixir from hex.pm rather than having to do something special with the Nerves System.

Hope this makes sense and good luck,

Frank

5 Likes
10

No worries about reply delay, my time is also split between many things so this isn’t going to be a fast project.

I haven’t thought about option 2 – though I should point out that if we do that we will end up with some duplication on the target system, e.g. the C library and similar low-level dependencies. Nix cannot* use the buildroot one because that would be a global mutable state kind of thing (i.e. a package input which is not in the control of Nix which can change the behaviour of the package) , which is what Nix tries to avoid.

*Well it probably can be made to, but not sure it should

Some of the heavy lifting of cross-compiling the kernel and other low-level bits is already taken care of because to some extent Nix does support cross-compilation, and some people do send patches to make that work better too. One of the things I do mean to have a look at, is the OTP cross-compilation, as @DianaOlympos pointed out. But I have previously used NixOS to cross-compile an image for a Raspberry Pi that ran Octoprint and mjpg-streamer, for my 3D printer, most of the things worked out of the box actually – one thing that didn’t is spidermonkey (required by policykit), I think this might now be fixed, and also the policykit duktape patches have also landed so that’s hopefully good news either way. And also that’s a far too heavy Linux system for Nerves – this is going to be the trickier bit I think, reducing the system size.

2 Likes
11

I too have been interested in running Nerves with NixOS instead of Buildroot. Glad to see there’s some discussion about it here (although I would have expected it in the Elixir Slack or Forum instead; no worries).

I have managed to run NixOS on a Raspberry Pi 4 along with some non-OTP packages for a separate purpose, but never committed the time to looking at cross-compiling OTP. Sounds like that will be one of the major hurdles here.

I think that it would be really cool if I could add Nix packages to a dependency list kept in the mix.exs and then they’d be included in the OTP release.

Yes! This would be fantastic.

I’m interested to see where this goes and help out if I’m able to. After spending a bit of time with both Nerves and NixOS, I feel like there’s some great potential gains to be made in terms of developer ergonomics by leveraging the declarative configuration options provided by Nix.

3 Likes
12

I found Cooking and Baking Linux Distributions in Nix helpful in understanding how this could be approached.

Having the ability to compile a package and pull in all of its dependency through nix has a lot of benefits. In my mind there are two approaches that are complementary:

  1. Enhance the rootfs with nix packages using only the cook portion.
  2. Create a minimal nerves system with buildroot use and bake in the majority of the system using Nix.

Being able to push a large build like QtWebEngine to CI and taking advantage of reproducible builds and cache-able artifacts would be ideal.

2 Likes
13

Why keep buildroot in #2 at all if you’re going to use nix?

1 Like
14

Why keep buildroot in #2 at all if you’re going to use nix?

Good question, my knowledge is somewhat limited but I think there would be challenges with compiling dependencies. Besides package management, nerves_system_br is responsible for setting up environment variables such as PKG_CONFIG_SYSROOT_DIR used for cross compiling dependencies in your application.

I’m assuming that with a nix shell, that wouldn’t be too problematic but might be difficult for a non nix users to use the system that I produce.

1 Like
15

Been contemplating this myself.

I realized that in addition to not needing Buildroot if using nixos, I also don’t really need nerves. Could just use a simpler erlang or elixir agent that is run by systemd, and could communicate with a simpler server. Nixos would take care of the rest on each machine.

16

There is also the option to do 1 and 2.

Where @robert.kovacsics points out

I understand that there are issues with with both nix cross-compilation for dependencies (an evolving state of affairs over time) and nix cross-compilation of OTP.

I do wonder why native compiling isn’t seen as an option? When I was tasked with building an OS for a RaspberryPi-like device, I just used an arm64 build machine.