Horus - extract an anonymous function as a standalone module

Your Libraries & OS Mentoring Libraries
, , ,
1

Hi!

On behalf of the RabbitMQ team, I’m proud to announce the Horus library!

Horus is a library that extracts an anonymous function’s code and creates a standalone version of it in a new module at runtime. The goal is to get a storable and transferable function which does not depend on the availability of the module that defined it.

This library is born from the khepri_fun module which was moved out of the Khepri library to make an independent library, easier to maintain and test. We need this in Khepri to extract transactions and store/transfer standalone functions as part of Ra/Raft cluster work.

Example

%% Here is the function we want to transfer or store.
Fun = fun() ->
          do_something_fancy()
      end,

%% Extract the function and create the standalone module.
StandaloneFun = horus:to_standalone_fun(Fun),

%% Later, execute the extracted standalone function.
horus:exec(StandaloneFun, []).

How does it work?

To achieve the goal, Horus extracts the assembly code of the anonymous function and creates a standalone Erlang module based on it. This module can be stored, transfered to another Erlang node and executed anywhere without the presence of the initial anonymous function’s module.

One caveat is that the created module will only work with the same version of Erlang/OTP as the one used to compile the original module, or a later version.

Documentation

Documentation is already available. It describes the concepts and the API in greater detail, though it is far from complete.

Feedback welcome!

The implementation is alpha at this stage. The internal design should not change drastically now but the API and ABI might.

Release 0.1.0 is now available from Hex.pm. This should make it easier for anyone to test it.

I know this is a very narrow and specific use case, but I would still be very happy to get some feedback on this :slight_smile: What do you think of it?

Thank you!

12 Likes
2

Should the example use horus:exec(StandaloneFun, []). instead of horus:exec(Fun, []).?

1 Like
3

Indeed, I made a mistake there. I can’t edit the message anymore ; perhaps @AstonJ, could you please fix the example? :slight_smile:

1 Like
4

Done …also changed the settings to allow unlimited edits for Library threads :023:

2 Likes
5

Thank you!

1 Like
6

Wouldn’t it be enough to just send the full original module around? That’s what I do for ierl to run things remotely.

2 Likes
7

That’s a good question. I should have been clearer about our initial usecase and needs.

Horus will not extract the anonymous function code only, but all the functions it calls, whether they sit in the same module or another module. At the same time, it will “analyse” the code and give the caller (through callbacks) the opportunity to deny specific operations or function calls.

In Khepri, we need to do that as part of the transaction functions feature. In case the reader doesn’t know what Khepri is, it is a key/value store where keys are organized in a tree. The replication of data relies on the Raft algorithm. Raft is based on state machines where a leader state machine sends a journal of commands to followers. The leader and follower state machines modify their state after applying comands and they must all reach the exact same state. They also must reach the same state again if the journal of commands needs to be replayed. You can learn more about Khepri in the Erlang forums thread about it.

For transaction functions to fullfill this “reach same state” constraint no matter the node running the transaction function, no matter the time or the number of times the function is executed, we need to deny any operations with side effects or taking input from or having output to something external to the state machine. For example:

  • the transaction function can’t send or receive messages to/from other processes
  • it can’t perform any disk I/O
  • it can’t use e.g. persistent_term, self(), nodes(), the current time, etc.

We also need to ensure that the transaction function remains the same if it is executed again in the future, even after an upgrade.

This is where Horus comes into play. Its role is to collect the entire code of the transaction function even if it is split into multiple Erlang functions, accross multiple Erlang modules. This is to prevent that an upgrade changes the behavior of a transaction function.

While Horus collects the code, it uses callbacks provided by the caller to let it say if an operation is allowed or denied. Khepri will deny messages being sent or received, calls to functions such as self() or node() and calls to any functions Khepri doesn’t approve (i.e. we, the author of Khepri, know these functions have no side effects).

By default, Horus will stop following calls (for code extraction) when the code calls a module provided by the erts, kernel or stdlib applications. The collected code will keep a regular external function call in this case. This is to avoid the extraction of code we know have no side effects and the behavior will not change between upgrades. Also because some functions can’t be extracted because they are simple placeholders replaced by NIFs at runtime.

@filmor, did I answer your question clearly?

1 Like
8

Yep, that answers it (and a lot more :wink:), thanks :slight_smile:

2 Likes