So we got a ton of profiling tools which are great, but I wonder if it would be possible to track which function calls produces lots of minor short lived garbage, this usually happens in hot paths. I’ve spend the better part of four weeks barking up of the wrong tres while trying to decipher perf reports, tprofs and benchmarks.
None of those pointed out the exact function producing garbage other then that gc minor was hot.
There is no way to profile how long data is alive. To do it either the compiler and/or the run-time would have to annotate each allocated element with a location (and/or timestamp) which could later be inspected by the GC when they are freed.
The best you have right now is tprof in call_memory mode which at least lets you see which functions allocates a lot of data.
Interesting, sounds like that could be beneficial for the compiler to have that information. Is it something the team have explored before?
We’ve talked about it, but (AFAIK) never implemented anything. It would be a very disruptive change to the runtime and so far the gains have not been large enough to warrant it.
The compiler has a lot of information about the liveness of things, but as soon as a term is passed into another module it looses all information about that term. So its job would be to tell the runtime that it might be a good idea to check for liveness of a term in more places than when a GC check is done.
I believe that GHC heap profiling for Haskell can get close to what you want.
5.4. Profiling memory usage
GHC, of course, compiles the whole program, not just a module at a time, and
certainly doesn’t support hot loading, so I wouldn’t expect the techniques to
transfer to Erlang.
Thats great, although I think there might be a little difference on holding on to memory and creating a huge amount of small garbage that needs to be cleanup and increase your latency.
Although, more insight into memory and allocation behavior for the compiler seams to be a good idea and could potentially make it easer to debug issues or perhaps generate better code and GC behavior.