RFC: Erlang Dist Security Filtering Prototype

Erlang/OTP Proposals Proposals: RFC
, , ,
1

I wanted to “Request for Comments” on the ideas behind a prototype I put together recently for Erlang Distribution Protocol node-specific control command filtering.

The full description and code for the prototype is available here: RFC: Erlang Dist Security Filtering Prototype by potatosalad · Pull Request #1 · potatosalad/otp · GitHub

Summary

In short: it’s a Firewall for Erlang Dist.

Example:

  • Node B fully trusts Node A. All control commands sent over Erlang Dist from Node A to Node B will be accepted.
  • Node A partially trusts Node B, but will only accept control commands sent to a registered process node_b_gateway. All other control commands will be rejected.

Motivation

Consider the following scenario where 3 nodes are halted as a result of 1 line entered into an initially unconnected node:

erl -name foo@127.0.0.1
erl -name bar@127.0.0.1
erl -name baz@127.0.0.1

On bar@127.0.0.1:

1> net_adm:ping('baz@127.0.0.1').
pong

On foo@127.0.0.1:

1> erpc:cast('bar@127.0.0.1', fun Halt() -> erpc:multicast(nodes(), Halt), erlang:halt(1) end).
% all 3 nodes are halted

What can be done to prevent foo@127.0.0.1 from recursively halting all other nodes?

Guide-level explanation

NOTE: The API below is not finalized and will likely change in the future. The concepts related to coarse-grained filters, fine-grained filters, combination of filters, and spawn_request handlers on top of Erlang Dist are the main focus of this proposal.

Filters (or rules) are configured per-node connection using either coarse-grained or fine-grained BIF calls that modify the dist_entry table within dist.c for the given connection.

Coarse-grained filters may be configured with either accept or reject (with accept being the default):

net_kernel:set_filter(Node, link, reject).
net_kernel:set_filter(Node, reg_send, reject).
net_kernel:set_filter(Node, group_leader, reject).
net_kernel:set_filter(Node, monitor, reject).
net_kernel:set_filter(Node, send, reject).
net_kernel:set_filter(Node, spawn_request, reject).
net_kernel:set_filter(Node, alias_send, reject).

Fine-grained filters may be configured for reg_send and spawn_request command types:

net_kernel:add_filter(Node, reg_send, my_secret_process, reject).
net_kernel:add_filter(Node, reg_send, my_public_process, accept).

net_kernel:add_filter(Node, spawn_request, {my_mod, my_fun, 0}, accept).
net_kernel:add_filter(Node, spawn_request, {my_mod, my_fun, 1}, reject).

Fine-grained filters may also be removed for reg_send and spawn_request command types:

net_kernel:del_filter(Node, reg_send, my_secret_process).
net_kernel:del_filter(Node, reg_send, my_public_process).

net_kernel:del_filter(Node, spawn_request, {my_mod, my_fun, 0}).
net_kernel:del_filter(Node, spawn_request, {my_mod, my_fun, 1}).

Combined filtering may be tested to see whether a given command from a node will result in an accept or reject:

1> net_kernel:test_filter(Node, reg_send, my_secret_process).
accept
2> net_kernel:add_filter(Node, reg_send, my_secret_process, reject).
accept
3> net_kernel:test_filter(Node, reg_send, my_secret_process).
reject

In addition, there is an option to enable Erlang-based filtering for spawn_request commands:

-module(my_spawn_request_handler).

-export([dist_spawn_init/4]).

-spec dist_spawn_init(Node, Module, Function, Arguments) -> Result when
      Node :: node(),
      Module :: module(),
      Function :: atom(),
      Arguments :: [term()],
      Result :: any().
dist_spawn_init(Node, Module, Function, Arguments) ->
    % Perform any extra filtering based on `Node' here...
    erlang:apply(Module, Function, Arguments).

Our my_spawn_request_handler module can be enabled per-node with:

4> net_kernel:set_handler(Node, spawn_request, my_spawn_request_handler).
undefined

Reference-level explanation

See the following for more details: RFC: Erlang Dist Security Filtering Prototype by potatosalad · Pull Request #1 · potatosalad/otp · GitHub

Drawbacks

  • Why should we not do this?
    • Does this actually improve security at all? Node-level control command filtering may not be fine-grained enough in practice to sufficiently increase security between nodes within an Erlang Dist cluster.
    • Extra memory used as part of the Erlang Dist entry table in order to store filters may prevent larger clusters.
    • Silently rejecting messages on the receiving side is a bad idea and something like the capability system proposed in SafeErlang sent over Erlang Dist would be better here.
    • What’s the point? Just use something other than Erlang Dist between nodes of different service types.

Rationale and alternatives

  • Why is this design the best in the space of possible designs?
    • I think this is a pragmatic solution that solves a real problem today where Erlang Dist is used between fully trusted and partially trusted nodes.
    • The filtering table uses the same hash table technique used by the dist entries themselves.
    • Preliminary performance shows that checking the filters in the hash table has a negligible effect on the throughput and latency of Erlang Dist communication between nodes with filtering enabled.
    • This actually provides a solution to prevent the problem mentioned above where all 3 nodes were halted remotely.
  • What other designs have been considered and what is the rationale for not choosing them?
    • Don’t use Erlang Distribution
      • This is probably the most common solution I have seen implemented.
      • For nodes of different service types, a different protocol is used: gRPC, Thrift, REST, GraphQL, BERT-RPC, etc.
      • Problems: Support for process-level message routing is lost or must be manually implemented (for example: RemotePid ! foo). Other dist features, like monitoring, linking, etc, are no longer available out-of-the-box.
    • SafeErlang: Access control with capabilities, resources, gates, rights, etc
      • See prior art below for more information about the SafeErlang papers.
      • Problems: To quote Rickard Green, this approach “require(s) quite a lot of work.”
    • Pure Erlang proto_dist implementation
      • This would allow a filtering solution per-node with roughly equivalent functionality without requiring a change to Erlang/OTP itself.
      • Problems: I initially started with this approach, but it required having duplicate decoding, encoding, and state tracking both internally by dist.c and by the proto_dist pure Erlang module. Keeping dist.c and the proto_dist implementation in-sync in the future would likely be expensive from a maintenance perspective and the resulting performance over the distribution protocol was sub-optimal.
    • Client-Side Restricted Shell
      • See shell: Restricted Shell for more information.
      • This is roughly the idea that inspired the prototype, but focusing on server-side instead of client-side.
        The spawn_request handler has similar functionality to the non_local_allowed/3 callback used by shell:start_restricted/1, but it is instead executed on the server-side.
      • Problems: It’s client-side and not server-side. Manually starting a non-restricted shell is fairly trivial where all restrictions may be bypassed.
    • Filter at the network layer instead
      • Why not just use existing firewall solutions to filter the traffic a the network layer?
      • Problems: No support for encryption (like TLS). The atom cache table will need to be maintained and updated for every node connection pair. Difficult to keep in-sync with upstream dist.c changes in the future.
  • What is the impact of not doing this?
    • Certain use-cases for Erlang Dist in the future will continue to be impossible due to security constraints.
    • I will be sad :frowning:

Prior art

Unresolved questions

  • How to configure node-specific filters for a given connection before the connection is established?
  • Do we want to support pid, port, and alias-level filtering? For example: right now, we can disable all sending to processes, but what if we want to allow selective sending? What use-cases exist for this kind of setup?
  • Does this actually improve security? If not, is it possible to build on top of this idea?
  • Does this increase the probability that Erlang Dist may be used in future services versus replacing it with another protocol between service types?

Future possibilities

  • Pid, port, and alias-level filtering.
  • spawn_request pid filtering. For example: Node A can only send messages to processes on Node B that it directly spawned with spawn_request.
  • Tracing integration and counters for profiling and statistics related to accepted and rejected control commands.
  • Preflight checking to find out whether a remote node is going to reject a request before it is sent.
  • Integration with the restricted shell so that allowed code execution may be defined remotely.
  • Possibily more integration of ideas from SafeErlang related to capabilities that might enable a form of “sandbox checking” prior to evaluation of remote commands.
18 Likes
2

I have added support for 4 additional handler types to enable Erlang-based filtering for reg_send, group_leader, send, spawn_request, and alias_send commands:

-module(my_dist_handler).

-export([
    reg_send/3,
    group_leader/3,
    send/3,
    dist_spawn_init/4, % spawn_request handler function
    alias_send/3
]).

-spec reg_send(Node, From, ToRegName) -> no_return() when
    Node :: node(),
    From :: pid() | '',
    ToRegName :: atom().
reg_send(Node, From, ToRegName) ->
    % Perform any pre-message-filtering here...
    receive
        Message ->
            % Perform any post-message-filtering here...
            catch ToRegName ! Message,
            exit(normal)
    end.

-spec group_leader(Node, GroupLeader, Pid) -> no_return() when
    Node :: node(),
    GroupLeader :: pid(),
    Pid :: pid().
group_leader(Node, GroupLeader, Pid) ->
    % Perform any filtering here...
    true = erlang:group_leader(GroupLeader, Pid),
    exit(normal).

-spec send(Node, From, ToPid) -> no_return() when
    Node :: node(),
    From :: pid() | '',
    ToPid :: pid().
send(Node, From, ToPid) ->
    % Perform any pre-message-filtering here...
    receive
        Message ->
            % Perform any post-message-filtering here...
            catch ToPid ! Message,
            exit(normal)
    end.

-spec dist_spawn_init(Node, Module, Function, Arguments) -> Result when
      Node :: node(),
      Module :: module(),
      Function :: atom(),
      Arguments :: [term()],
      Result :: any().
dist_spawn_init(Node, Module, Function, Arguments) ->
    % Perform any filtering here...
    erlang:apply(Module, Function, Arguments).

-spec alias_send(Node, From, ToAlias) -> no_return() when
    Node :: node(),
    From :: pid() | '',
    ToAlias :: reference().
alias_send(Node, From, ToAlias) ->
    % Perform any pre-message-filtering here...
    receive
        Message ->
            % Perform any post-message-filtering here...
            catch ToAlias ! Message,
            exit(normal)
    end.

The my_dist_handler module can be enabled per-node with:

4> net_kernel:set_handler(Node, reg_send, my_dist_handler).
undefined
5> net_kernel:set_handler(Node, group_leader, my_dist_handler).
undefined
6> net_kernel:set_handler(Node, send, my_dist_handler).
undefined
7> net_kernel:set_handler(Node, spawn_request, my_dist_handler).
undefined
8> net_kernel:set_handler(Node, alias_send, my_dist_handler).
undefined

Any inbound control commands for these types, if accepted by the coarse-grained and fine-grained filters, will spawn a new process and execute the above callbacks based on the command type.

4 Likes
3

Writing down the technical points discussed yesterday in the EEF Security WG Meeting:

Naming

Propose to use deny instead of reject to conform with other firewall-like tools.

Default Settings

It would be great to be able to specify default filter settings that can be overridden for each node.

One example use case would be in a homogenous cluster all needed capabilities can be whitelisted and everything else is rejected.

Register Settings before Node connect

Right now permissions can only be set after the node is connected. In the small timeframe between connect & the filter settings application, everything is allowed.

4 Likes
4

I’m planning on breaking down the prototype into smaller, more easily reviewable pieces.

Part one PR is available here: Add support for erlang:statistics(dist) counters by potatosalad · Pull Request #5838 · erlang/otp · GitHub

Usage looks like this:

(foo@0.0.0.0)1> erlang:statistics(dist).
[{alias_send,[{accept,0}]},
 {demonitor,[{accept,0}]},
 {exit,[{accept,0}]},
 {exit2,[{accept,0}]},
 {group_leader,[{accept,0}]},
 {link,[{accept,0}]},
 {monitor,[{accept,0}]},
 {monitor_exit,[{accept,0}]},
 {reg_send,[{accept,0}]},
 {send,[{accept,0}]},
 {spawn_reply,[{accept,0}]},
 {spawn_request,[{accept,0}]},
 {unlink,[{accept,0}]},
 {unlink_ack,[{accept,0}]}]
3 Likes
5

I really like this proposal and it could be useful to us in Livebook where we have a server that needs to connect to multiple clients (but we don’t necessarily want the client to talk about to the server). We have so far thought that we would need to move away from the Erlang Distribution.

I have a question: have you considered making the whole filtering be based on Erlang module?

net_kernel:filter(Node, some_module).

some_module has a function that receives filter(Node, Type), where the type is link, {reg_send, Process}, and so on. This would reduce the API surface and perhaps give more flexibility.

Another question is: imagine you have a cluster with nodes A, B, and C. Now node A connects to node Z and sets up the proper filters. What is stopping Z from connecting to B and C without said filters? I guess we could make A, B, and C have a specific cookie and use a different cookie between A and Z? Any other ideas?

4 Likes
6

Yeah, I’ve definitely considered that, especially as the number of supported filters has grown (I initially only planned to have reg_send and spawn_request). I think the final version will probably look closer to this than all-the-hard-to-remember-callbacks we have right now.

We’re accomplishing things like this very roughly using TLS and a custom verify_fun which can decide whether a connection is allowed or not based on the peer’s certificate information. It can also be accomplished today using a custom erl -proto_dist ... module that implements the select/1 callback, but the developer UX here in both cases has room for improvement.

Alternatively, as @maennchen mentioned, I think having global defaults for filters could help to accomplish effectively the same thing:

By having default filters ('_' here represents “all command types” or “all values”):

net_kernel:set_default_filter('_', '_', drop),
net_kernel:set_default_filter(spawn_request, {my_dist_auth_module, authenticate, 1}, accept).

Nodes A, B, and C could override the default filters to adopt a more “trusted” setup amongst themselves. Node Z could still connect, but if Node B and C don’t have any filters defined, it would default to only allowing calls to “login” or request more access on the connection. This could be especially useful as part of an emergency “break-glass procedure” where a human may need to fix something in-flight on production servers, while still requiring that the human authenticate themselves as part of an audit trail. After a period of time, the temporary higher access could fallback to the lower default as a safety net.

Separate cookies are also a possibility, I think, but I haven’t really looked very deeply into what might be involved there change-wise to make that all work.

Another idea I’ve toyed with (but haven’t written any code for) would involve having a separate resource type for the filtering table itself, instead of having it attached to every dist entry. When a new dist connection is setup, this erts_internal:new_connection/1 BIF gets called, which returns a erlang:dist_handle() reference which points to a dist entry resource type. A similar thing could be done for filters and they could then be attached to multiple dist entries which would each only store a pointer to the filter table. An update to the filter table could propagate to all of the attached dist entries, versus having to walk 1-by-1 to update each dist entry. This would allow Nodes A1, A2, A3, … to have their own filter group for how they communicate with each other separate from Nodes B1, B2, B3, …, etc. Depending on the quantity of filters, this could help to reduce memory consumption in very large clusters at the cost of losing more granular per-node specific filtering.

3 Likes
7

The cookies already work today. You can specify a node cookie before you connect to it and that would make it impossible to connect to the rest of the cluster. However, today you can just rpc and get the cookie between the other nodes, which is a gap this proposal would close.

I like the TLS with a custom verify_fun approach too, thanks!

5 Likes
8

A very likely scenario is that you will have to reject all communication except for specific communication with your specific application. This also implies that you can only use hidden nodes.

No parts of OTP have been written to be prepared for communication using Erlang signaling with potentially hostile processes. Before being able to allow potentially hostile processes to communicate with an OTP service (using Erlang signaling; implied from here on), one would have to thoroughly go through the code of the service and most likely make quite substantial changes.

After having rewritten such a service one would also have to write filters that restricts signaling to just the right extent in order to prevent harmful side effects. I imagine that this can be quite a large task as well even for quite a small service. Over time the protocols used by these services also change. This in order to improve the service or fix bugs. When these changes are introduced, the filters used for communication with a service would most likely also have to be changed. We can take rpc as an example. As of OTP 23 an rpc call request changed from being an ordinary message signal sent to the rex server on a node to become a spawn-request signal sent to the node. Just looking back a few releases there have been quite a lot of changes in OTP internal protocols like this.

Going through all parts of OTP, rewriting the code and making filters for different communication scenarios would be an enormous task. I cannot see that we at OTP would ever have the resources for such a rewrite. Just maintaining already pre-existing filters would create quite a lot of work. In the rpc example above, it would perhaps not be have been a huge task since it more or less is a one to one mapping between two different types of signals, but depending on the change it might not be that easy.

Even if we magically could transform OTP code into a state where it would be secure to communicate with hostile processes I am not sure we would want the extra cost it would imply to maintain and improve the system going forward from there. Each change regardless of whether it is new functionality, an improvement or a bugfix would require a lot more extra work compared to today when we know that we only communicate with trusted processes.

That is, I don’t see how it realistically would be possible to allow other communication than specific signals to your specifically designed application and reject all other traffic. The difference between using such a heavily restricted Erlang distribution over TLS, and letting your application communicate between the Erlang nodes directly over an ordinary TLS channel is quite small. To me it seems like more or less all benefits of communicating over the Erlang distribution have disappeared. I could be wrong, but to me it also seems like it would be easier to ensure that security is enforced by taking care of all of it at the input/output of the TLS connection.

Another issue that also would have to be addressed. Before Erlang signaling is enabled on a channel between two nodes, the two nodes performs handshaking. This handshake could very well be exploited in order to trick a node into performing unwanted things. This handshake would also have to be secured.

6 Likes