Has anyone come up with or come across a general solution for self-modifying programs in Erlang or any BEAM language (maybe LFE is best suited for this)? Let me explain.
It is possible to modify a running program in some languages/environments to different extents. For example, in some late binding environments (such as LISPs) one can re-define a function using macros, re-compile and reload them. One can accomplish similar results in interpreted environments.
BEAM has hot code loading, so is it possible to have a general self-modifying pattern for a module in a running application? Something that would
Get called from a module that wants to modify itself or another module with some info about what to change (e.g. replace a function with the new version)
Read the module source code*
Apply the change and generate new module code
Compile and load the new module
Store the new module source code for future reference
It seems like each of those steps are possible individually, but I’m curious if a general solution exists: perhaps an OTP application that can be added as a dependency, or a set of macros that can be imported to desired modules? Or maybe there’s a better way…
*presumably in LISPs you don’t need access to source because it is already available as AST at runtime (but I’m not a LISP expert)
I wonder, what would be the use for something like this? I’m not a LISPer and I don’t know how they go about it, but IMO this would be a perfect recipe for disaster and a road to debugging hell.
Say you have a bug somewhere, some module does not behave as it should do. So you check out the project from github or whatever, look at the buggy modules’ source, and what is written there is something entirely different from what the module running in your system obviously does.
So now you have to get the modified code somehow (which may be different from running system to running system and may be modified again only seconds after you obtained the now-running source), and yes, there is the bug you were looking for, all right.
But how did it get there? So you have to find and debug the module that modified the now-buggy one, which in turn may have been modified by a third one, etcetc.
Some of the things you are asking for look like release upgrades (which are tricky already), some don’t. Self-modifying and other-modifying, as part of your programming routine anyway, sounds like a bad idea to me, as @Maria-12648430 already said.
The steps you are listing are performed by the cover tool when it takes compiled modules and add the required instrumentation for tracking code coverage. The AST is taken from the .beam file and is then converted to a new AST which is compiled and loaded.
What would be the reason to have it? In the past we assumed that self-compiling programs will be much faster, but found that it wasn’t really the case. There is quite good talk from Basho proving that it might not necessarily be faster. Maybe now, with JIT, self-compiling programs are finally beneficial, but…
You’d need a compiler on every production host. That may not be acceptable from security point of view.
Testing becomes somewhat… intriguing. Because there might have different compiler/OTP versions on your machine, in CI, and in production.
So here I’m with @juhlig - it sounds like a bad idea.
As for cover tool mentioned by @kennethL, we ended up not using it, for it turned to be very expensive process working with large codebases, and implemented an alternative that used parse_transform to inject counter bumps.
The OP asked for self-modifying programs. So your post is a bit off-point, but then again the modified parts of the program need to be compiled, of course, which brings the points you raised back in scope again
Then again, it might just be possible to do that via compile:file/1,2 and friends… I never used that, so I don’t really know.
The cover tool mentioned by @kennethL is a good example where IMO code modification makes absolute sense (though it is not self-modification): Triggered manually, when definitely nothing of what you are modifying is running, for a specific purpose that may or may not be production-related.
Anyway, the (crucial) point is, once the code is running, it should stay that way for as long as it is running.
(Release upgrades are kind of an exception here, yes.)
However, what the OP asked for, if I understood him correctly, is modifying the code of a running program, from within itself, as part of what the program routinely does in its normal everyday workflow. Aside from the technical difficulties involved to pull this off one way or another, this simply can’t be a good thing
Thank you all for replies; and sorry I didn’t mention any use cases in my original post.
That’s partially because I’m not planning on this being a strategy to ship applications to production or anything like that. I merely wanted to see if any pattern at all existed for a self-modifying program and then abuse it for fun (I have some half-baked ideas).
Just want to confirm that the cover refers to this code (lots LoC to read). I’ll play around with the idea some more in free time and may even report back if I have something to show, so you can all tell me how horrible of an idea that is.
That tool has a lot of code, if you just want to know how to read AST from debug chunk and recompile the module, you might want to look at beam_lib:chunks(Binary, [abstract_code]). It may also be used that way for additional resilience.
I’m not a LISPer and I don’t know how they go about it, but IMO this would be a perfect recipe for disaster and a road to debugging hell.
