Erlang inotify missing events on Docker

Questions / Help
,
1

Hi Everyone

I’m using fs to get notification when files get deposited on an FTP directory.

All works fine on my macOS (DEV env) and when testing on Ubuntu 22.04. All inotify events get received by my Erlang app. Not a single miss.

But when deploying on Docker, in average 15% to 20% events were missed.

In general, I get around 450 files copied to the FTP at the same time. Then nothing for 1h, then 450 files more, etc.

I’ve tried increasing these OS’s settings (according to this link) without success:

sudo sysctl fs.inotify.max_user_instances=1000000
sudo sysctl fs.inotify.max_user_watches=1000000
sudo sysctl fs.inotify.max_queued_events=1000000
sudo sysctl -p

Many events are still missing.

My second attempt was to reduce the number of events to be triggered. Got a slightly better result, but still missing events.

  1. Is this a problem with the port_driver design, or is it related to Docker itself?
  2. Anyone has experienced the same issue?
  3. Any other idea to reliably solve on Docker?

Thanks
Z.

2

Do you see the Docker problems on macOS?
On macOS I stay away from Docker as its file system implementation is horrible.

For the Zotonic filewatcher we use fswatch on macOS. Works splendid.

1 Like
3

@mworrell Nope. I never tried Docker on macOS, only Linux.

Zotonic’s fswatcher is also based on inotifywait just like fs. No doubt it works, but pretty sure it gonna drop events under load (i.e many files created/modifed/deleted at once) on Docker.

My config:

$ uname -a
Linux xxxxx.default 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

$ docker --version
Docker version 27.2.0, build 3ab4256

$ erl
Erlang/OTP 27 [erts-15.0.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit:ns]
4

Dropping events is bad, very bad.

Guess a kind of fallback/workaround is needed, where after a quick succession of many events a full directory scan is done to find any changes. Which is a pain… not scanning is exactly the reason to use inotify in the first place.

1 Like
5

@mworrell exactly. I’d like to avoid polling (as much as I avoid mnesia) :slight_smile:

6

Inspired by @mworrell Zotonic fswatcher, I’ve manage to fix the issue. My fs fork is available here.

1 Like
7

How did you fix the issue?

8

@mworrell a three steps process:

  1. switched from list to binary parsing (as you did)
  2. listen to less events. It is now configurable via application:get_env/3
  3. finally, I pre-compiled the line split regexp for speed.
9

I am thinking of copying your limited set of events. Wondering if both close_write and modified are needed? Is there a difference?

1 Like
10

In my case, i avoid getting duplicate events and only use one of them.