Best way to send files between nodes. Which to choose?

Questions / Help
,
1

Hello all!

What is the best way to send files between nodes for modern Erlang:

  • Shell command + rsync and notify Erlang with erl_call
  • Send process-to-process between nodes in Erlang itself
  • Something else

Any examples?

By myself found this example GitHub - innaky/transfer_file: Transfer or copy file between erlang nodes. but 6 years no activities. Need something very fast.

1 Like
2

“Best” depends on your requirements. Large files vs small files, many all the time or occasional, do the receivers want their contents as values in the VMs or not?

rsync and os:cmd would be a good start, unless the requirements clearly say otherwise.

3

And what about sendfile for point-to-point transfers or, if having a network of nodes, peer-to-peer protocols? (for example apparently several Erlang BitTorrent clients can be found)

4

Just one anecdotal warning - we’ve seen issues on the receiving end of such a file transfer (using file:sendfile). At high concurrency and rate of operations we’ve seen the binary allocator struggle with deallocating all the binaries received from socket. A stop-gap solution of a simple NIF for receiving and writing to a file with a pool of buffers worked well for us.

1 Like
5

Alternatives:

  • Shared filesystem where the file implicitly exists on all nodes?
  • Shared S3-like where the file exists elsewhere, but is accessible from all nodes?
  • Outbox, inbox pattern: you write to local filesystem, something else picks it up and writes it to the remote filesystem? Remote end polls for newly-arrived files.

Like @mikpe says: “best” depends on your requirements.