Setting up tracing for a future process: how?

Questions / Help
, , , ,
1

Hi!

How does one sets up tracing for a future short-lived process pid, that doesn’t exist yet? Ideally, on a live production system, without direct code modification & in-memory module reloading.

For context, I am looking for a way to collect some debug info (function calls, arguments passed & values returned) when processing an HTTP request coming from a particular customer.

I am naively thinking something like this should be possible by OTP tracing capabilities:

  1. when process handling an HTTP request reaches a function X such that returned an ID of a client made the HTTP request,
  2. stop execution of a process,
  3. run my code which will create a [trace session](trace — kernel v10.5) for just current pid and enable extensive (and expensive) tracing on all functions that belong to my application,
  4. resume execution of a process,
  5. info is collected, I can proceed with analysis - offline.

Can the above steps be expressed by just calling a bunch functions in a remote shell, while connecting to a production app?

I think I could probably add a bit of extra no-op-like code to achieve exactly this:

  1. store customer ID globally (for example, using ets or persistent_term),
  2. somewhere in the HTTP request processing path, have code that would run an if to determine if a tracing session should be created, and if so, create an enable desired tracing,

But I was curious if there’s a way to achieve this without permanently adding any code upfront.

Purpose of this is first and foremost to aid my understanding about tracing capabilities & limitations of of OTP. But I am also considering creating an internal tool in my company’s project, such that would help us debug problems that occur during HTTP interaction faster.

P.S. For context, my app is written in Elixir & uses Bandit web server.

1 Like
2

That part is not possible with tracing. Trace messages come async to receivers (it’s message passing) and do not block the process the traces are about - that‘s an explicit feature of tracing.

You can create traces, which e.g. on process start or some specific function call of new processes setup additional traces for that process specifically though.

If you need to block the observed process you need other options.

1 Like
3

Perhaps you could instrument your code to start edbg:fpid/1 at a certain point. Then tracing will continue until you call edbg:fstop/0 and trace info is stored on file for later analysis.
Have a look at: Tracing · etnt/edbg Wiki · GitHub

–T.