Logstasher - is Erlang Logger formatter for Logstash

Hi folks! Together with @mworrell for @zotonic_core_team we are made new Erlang/OTP library logstasher :metal:

This library is provide easy way to format and send logs from Logger to Logstash by TCP and UDP. Or you can send output logs directly into console if you want :wink:.

For add this library into your rebar3 project, just update your rebar.config:

{deps, [
    {logstasher, "~> 1.0.0"}
]}.

Configure logstasher and Logger with using new handler in sys.config like:

[
     {kernel, [
         {logger, [
             {handler, logstash, logstasher_h,
                 #{
                     level => info
                 }
             }
         ]}
     ]},

     {logstasher, [
         {transport, udp},     % tcp | udp | console
         {host, "localhost"},  % inet:hostname()
         {port, 5000}          % inet:port_number()
     ]}
 ].

Also don’t forget add the logstasher application to your .app.src file:

{applications, [
   ....
   logstasher,
   ....
]},

The Logstash configuration can looks like:

input {
  udp {
    codec => json
    port => 5000
    queue_size => 10000
    workers => 10
    type => default_log_type
  }
}
output {
  stdout {}
  elasticsearch {
    protocol => http
  }
}

And of course this library is available on hex.pm :upside_down_face:

Enjoy :metal: and let us know what you think about this library and what we can improve for you!

6 Likes

Works great in testing now, and the console option is nice for systems that log the (container) console output to external systems.

4 Likes

From my experience (migration of 1.5M LOC project from io:format-style to structured logging), automatic conversion of iolists to strings not always do what developers might expect:

if someone would do

Tags = ["foo", "bar", "baz"],
logger:info(#{label => something_happened, tags => Tags})

they most likely want it to be logged as

{"label": "something_happened", "tags": ["foo", "bar", "baz"]}

not

{"label": "something_happened", "tags": "foobarbaz"}

Another point:

I see you support TCP transport as well. Did you consider handling TCP backpressures / overload protection somehow? Right now it seems if write to socket blocks for more than 5s this handler will crash and will be removed by logger AFAIK

5 Likes

Hi @seriyps, thanks for your review!
Could you add your findings as issues at the repo?
Then we can have a look at them and make a fix.

Thanks! Marc

3 Likes

Hi @seriyps! Many thanks for your review this is very helpful! Please keep going to investigation and provide your any opinions about it. Yes, all those points what you mention so far is important and will be fixed/improved. Thanks again.

3 Likes

Created 2 tickets on GitHub @mworrell @vkatsuba :+1:

5 Likes

Thank you Sergey!

4 Likes