Gen_agent - behavior module to simplify resource fetching (request for comments)

gen_agent is a behavior to make resource fetching with backoff and retry logic easy to implement. In a general context, it can be used to implement any task that may require retrying something with a backoff between attempts.


This project grew out of an attempt to address a recurring supervisor feature request: delaying the restart of failing children, namely when the children in question depend on an external resource which may be temporarily down, ie where retrying would help, but not frantic retrying.
The subject has been discussed over and over again, and while it would help in the case mentioned, it is a bad idea generally (and we (@juhlig and I) know, as we tried a general solution earlier this year).
There are known better ways to address this, but they are also leagues away from a simple knob on the supervisor, usually boiling down to “roll your own state machine with custom backoff and retry logic”.


gen_agent is an attempt to make this common task easier, ie to let an implementor focus on how to obtain a resource and not have to worry too much about implementing the retry logic and then bake it all into the component that will be using it.

It is still in its early stages, and so the documentation is messy and consist only of the README and two examples, no tests, and there are probably a number of bugs.
However, we think it has by now evolved to a point where it would be nice to have some feedback from the community.

Generally, we put focus mainly on simplicity, not on flexibility. The more flexibility you want, the more it becomes just like a gen_statem.

So if you are interested, you are welcome to take a look and try it out.
We would like to know if you think this project would be useful (or not), for yourself and/or others, even if not in the state it is currently in.
What is missing, what is strange, what is dangerous, what is downright nonsense?

Convenience features are currently not on the top of our list, we want to build a minimal, solid, generally useful “core” first.

9 Likes

Great work! I see that for this lib was used erlang.mk, this is mean that - this lib cannot be used as dependency in rebar3 projects. Could you please update it? And also - can I ask published it into hex.pm? Also it would be great to have GitHub Actions with tests for this project.

4 Likes

I think this is great. It literally scratches an itch I had recently. I’ll give it a whirl myself soon. Cheers!

4 Likes

Look great @Maria-12648430 (and @juhlig!) I can see how useful something like this would be :023:

3 Likes

Happy that you all like it so far :blush:

I’m sure we can do all that, but we first want it to settle down and solidify a bit more :slight_smile:

Ah, as to that… I forgot to mention that, given it stays general enough and receives enough interest, we hope that it may be adapted as a standard behavior in OTPs stdlib. In that case, it would be problematic, at least under the same name, as a project using the 3rd-party gen_agent from us as a dependency or sub-dependency would cause the stdlib module to be overridden I think.

So I guess the question goes out to the OTP people: Are you interested? No problem if you’re not :wink:
(Since I don’t know how to address an entire group, I’ll just mention a few people who I think may have an opinion on the subject: @kennethL @garazdawi @raimo @bjorng @rickard @ingela @hansn :sweat_smile:)

3 Likes

I was just reading the docs there and was confused. Why does it go into the sleeping state when the instruction isperform? Wouldn’t it execute?

2 Likes

Yes, there is definitely more work to do on the docs :wink:

No, sleeping is indeed correct here.

  • The first thing it does when entering the sleeping state is call the callback module’s sleep_time/2 function with the attempt number as first argument. Coming from idle via the perform instruction, the attempt number is always 0.
  • It will then set a state_timeout with the time returned by sleep_time/2.
  • When the state timeout happens it will enter the executing state and start performing its task.

In most cases that I can think of, the sleep time for the 0th attempt would be 0 (ie, immediately), but there may still be other cases. The callback module may decide differently even for the same attempt number, eg based on the current state (the second argument to sleep_time/2) or some external condition.

2 Likes

Hm, in fact it can be used with rebar3 projects even though it uses erlang.mk. The app.src file was missing, though, I think that was why rebar3 may have complained. That and the fact that the branch was named main as GitHub suggested, but rebar3 seems to expect master, so we renamed it to master now for that reason.

4 Likes

Yep. Thanks for updates.

2 Likes

Sorry for the late reply, life has been crazy busy lately. Finding a general solution to the problem you described sounds interesting. Have not had time to evaluate your suggestion yet, hope to have time to take a look “soonish”.

2 Likes

I like it but I have a problem with the name gen_agent: I get it mixed up with closure’s agents which are different things. :smile: I will have to learn :wink:

3 Likes

I like it too, and I would have needed it some years ago!

But like rvirding I have a problem with the name but from other reasons.
An “agent” is a very general concept used in very many different contexts basically meaning “one who acts”. Your gen_agent is more specialized than that, although quite general.

Maybe “gen_fetcher” or something else that describes it better?

3 Likes

Hi @ingela :slight_smile: We are not in a hurry, take your time :wink:

Hi @rvirding :slight_smile: We are not nailed to a name (yet), as it is still more of a draft than anything else. Open to suggestions :wink:

And hi @hansn :slight_smile: gen_fetcher seems to be too restricting IMO. The concept is not limited to fetching things, although this is probably what it may mostly be used for. However, you can also imagine using it to post something on Twitter, retrying if it happens to be down or something.

4 Likes

…gen_fetcher seems to be too restricting IMO. The concept is not limited to fetching things, …

You are right! I was too restricted in my thinking.

4 Likes

gen_resource perhaps?

2 Likes

Nice.

But yea, the name has problems. One problem with the name agent is also that Elixir has Agent and this acts differently and would definitely be confusing, and limit their ability to make an idiomatic Elixir wrapper like GenServer.

As for rebar3, as you note you just need the .app.src file since there are no dependencies. For anyone wondering, rebar3 will attempt to build anything it can tell is an OTP application – which the .app.src in the src directory lets it know – whether there is a rebar.config file or not. So when no dependencies are required, no rebar.config is required either.

As for publishing to hex.pm, unless there is already agreement with the OTP team to add it to OTP I would publish to Hex. And even if there isn’t agreement it would be nice to have it added since it is usable from older OTP’s and won’t be backported into older OTPs, so users would benefit from it being on Hex.

Lastly, rebar3’s hard-coding to master is indeed a pain that we want to change. It is difficult because of how locks work. I was hoping to not to just force the change to be hardcoding to main but it may be what we have to do.

2 Likes

That doesn’t sound nearly as bad-ass as “agent” :grinning_face_with_smiling_eyes:

But seriously, I don’t think that cuts it either. It really is not a resource, like an IODevice or network connection would be…

1 Like

Ok, so we agree that the name is problematic :wink: Still, it is the best I can come up with to describe what the thing does =^^= I consulted a thesaurus, and possible alternatives that seem feasible are “broker”, “delegate”, “intermediary”, “negotiator”… They all sound a bit cumbersome to me, but what do you think?

There is no agreement yet, neither pro nor con =^^=

If agreement turns out to not have it in OTP, there is no problem with publishing on Hex, but if it should be adopted into OTP, I wonder what would happen if some library used the gen_… uh… _thing Hex package as a dependency with gen_thing also present in OTP? (You’re associated with relx, right? Enlighten me if you know :blush:)

1 Like

In rebar3/relx the release building will error out with an error message about a duplicate module. This ensures the user is not caught off guard. They can then either exclude the dependency itself or the module from the stdlib.

1 Like

Those all seem too generic to me and doesn’t fit the description of the behavior:

I’d expect something more to the point like gen_fetch, gen_backoff or gen_retry perhaps.

2 Likes