This started as a proposal for solutions and after writing that out I decided it might be better to start with just describing the issue an seeing where that goes.
The latest use case I have for limiting concurrency: A process that is called with the pid of another process to monitor and has to do some work when any process it is monitoring dies. This work does not need to block the handling of the next dead process.
Since there is no limit on how many processes die at the same time I can’t simply spawn a process to handle each death. So a pool or a semaphore are two options to deal with this situation, neither with support in OTP.
The other thing about this process is there is no reason it has to be only 1, it only needs to be at least one, but there is no builtin ability to choose from a set of children of say a simple_one_for_one. In this particular case the chance of the process being overloaded is one not great enough to warrant additional engineering or dependencies until someone actually hits the issue, but if it was simply an option builtin to OTP that would be a different story.
Curious what others think. Are there other options I’m missing? Do you think it would make sense to include something in OTP for such cases and for those working on OTP, would such an addition be welcome?
Sounds like a pretty clear case for the usual hierarchy of supervisors. Why wouldn’t you just set some max qty of children per supervisor and spawn a new supervisor that takes on the new processes each time you fill a “batch”? It’s not clear from your description why this standard OTP pattern wouldn’t work for you. Am I missing something?
Correct me if I got it wrong: you have a set of processes (“S”), and you need to perform some action upon termination of any process from that set. That action may be somewhat slow, so you cannot just have one process (“P”) monitor all processes from “S”. Hence you’re thinking "what if I have several instances of “P”, and looking to know how to make that happen?
One thing that I might question is, why do you want to run the “post-termination” code in the context of a different process? Meaning, could you change the code of the “process being terminated” to run the “post-termination code” in the context of that process? E.g.:
The operation could be slow, yes. It runs user defined Processors on the Span when it is ended. Usually this is just two writes to two separate ETS tables, so for most (if not all) users this isn’t an issue.
As for what it needs to do, It acts as a backstop and for times you don’t control the process code or the lifecycle. One case is apparently LiveView where this is needed when dealing with spans.
I’m not sure what you mean by “max qty of children” and there is no batch.
Are you suggesting using a supervisor with N children that can do the work? Problem is selecting one of those children to send the work to. Obviously one with plenty of solutions, just nothing builtin.
I’m curious what do you think an abstraction like this would look like?
A pool makes sense to me with a no knobs queuing discipline (e.g., codel, fq codel, etc.), but are you thinking a plain ole app that has to be configured like any other or possibly an augmentation of supervisors?
I’d spawn one monitor process per monitored processs (the monitoree). The monitor will sit idle in a non-runnable state until the monitoree terminates, so it’ll consume minimal resources.
A single monitor with a worker pool should work too, but I don’t see the need for that complexity here ATM.
I think Tristan’s case is a case where this could be very problematic. Specifically, there’s no way for otel to know how many processes might be monitored. Processes are cheap, but they are not for free, not to mention that it’s not just a matter of resources consumed by virtue of the process itself, there’s also the cost of observing those processes. Other operations become more expensive as well (literal area collector comes to mind) simply because you have more.
It would be pretty rude for otel to double the amount of processes in a given area of an application
I constantly try to remind myself of something Joe Armstrong said : You need a lagom amount of processes
I currently don’t have the time to grasp the whole thing right now, but it made me think of this project we did for fun a while ago. It is a queue service, running as a separate process, which can be accessed from other processes.
In this use case, it could be used like this:
Have 1 or several (producer) processes to monitor all the processes whose death you are interested in
When it receives a 'DOWN' message, just publish whatever is needed for the post processing to such a queue
On the other end, you can have a number of (consumer) processes listening at the queue. If such a process gets an item from the queue, it does the required post processing, while other processes keep listening. When it has finished, it goes back to listening.
For one thing, you can thus turn the number of consumer threads up and down as needed without the need to reshuffle anything else.
For another, if one of your consumer processes crashes while performing post processing, you will only lose this one item it had polled from the queue. If you monitor and post-process in the same process, you will lose all the messages that arrived in the message queue while it was doing work.
If the rate of process deaths is temporarily higher than what the consumers can handle, there will be a buildup of items in the queue, but as long as it is lower on average it should work out in the long run, otherwise you need to add more consumers.
In the case of bursts of process deaths, there might also be a buildup when inserting into the queue, especially if you decide to put a limit on it. This will push through to the message queue of the producer process since it can retrieve from its message queue only as fast as it can insert into the queue.
Keep in mind however that, as I said, this was done as a fun project. I don’t know if anybody has ever been using it, so there may be bugs, and overall performance has never been seriously tested, either. You may also find that it emphasizes aspects that are irrelevant to you and lacks others that you want or need. tl;dr, just try it out if you think it might be useful
So, I know this case is somewhat more involved here, but FWIW the Elixir’s Task.async_stream works great at doing a parallel map over a collection with limited concurrency and is generally my goto for situations where I need that (and by default it limits it to the number of schedulers). Perhaps this could be some inspiration?
