Shigoto - Background job processing

Your Libraries & OS Mentoring Libraries
,
1

Shigoto is a PostgreSQL-backed background job processing library for Erlang, built on OTP.

Features:

  • Named queues with configurable concurrency
  • Retries with backoff
  • Cron scheduling
  • Job batches with completion callbacks
  • Circuit breakers for failure-heavy queues
  • Telemetry integration

Quick example:

%% Enqueue a job
shigoto:insert(#{
worker => my_email_worker,
args => #{<<“to”>> => <<“user@example.com”>>},
queue => <<“emails”>>
}).



%% Define a worker
-module(my_email_worker).
-behaviour(shigoto_worker).

-export([perform/1]).

perform(#{<<“to”>> := To}) → 
send_email(To),
ok.

Demo app:

We put together shigoto_demo - a small Nova app that exercises the main features: multiple queues, cron jobs, batches, webhook retries with circuit breaking. It comes with a Python traffic simulator to generate load.

The demo also includes two live dashboards built with Arizona (real-time UI framework for Erlang):

  • shigoto_board (/shigoto) - real-time job dashboard with queue stats, job failures, batch progress, and cron entries
  • nova_liveboard (/liveboard) - BEAM VM dashboard with process explorer, ETS tables, supervision trees, and system metrics

Both dashboards are still in active development - expect rough edges and missing features. Feedback and ideas welcome.

Links:

shigoto:

7 Likes
2

Thank you for building this @Taure. Shigoto is exactly the kind of library the Erlang ecosystem has been missing. This is something Elixir developers have long taken for granted with libraries like Oban, and it is wonderful to see this gap finally being closed on the Erlang side.

This is a meaningful contribution to the community. Looking forward to following its development.

1 Like
3

Thank you.

I have been working on Nova and tooling around for some time and now felt it is time that we have the same tooling as Elixir.

Erlang have the same things, maybe a bit hidden but you can build web apps the same way as Phoenix and ecto in Erlang. Maybe not as mature but we need to start somewhere. :grin:

4 Likes
4

Quick update for anyone following the thread. Shigoto is now at v1.2.2 with some notable additions since the original post.

Fanout queues (v1.2.0)

Broadcast delivery mode where a single enqueue fans out to multiple consumers. Useful for webhook delivery, cache invalidation, and any pub/sub shaped workload where you want the durability of a job queue. There is a guide for it in the repo.

Repo config (v1.1.0)

You can now configure Shigoto via a repo module instead of a raw pgo pool name. Plays nicely with Kura if you are already using it.

Auto-migration on startup (v1.2.2)

No need to call shigoto_migration:up/1 manually anymore. Tables are created on application start.

Also worth flagging some things that were already in v1.0.0 but did not make it into the original post:

  • Job dependencies with cycle detection (depends_on chains)
  • Job snoozing, workers can return {snooze, Seconds} without burning a retry attempt
  • Unique jobs with debounce, time windows, and field replacement on conflict
  • Full seki integration: rate limiting (token bucket, sliding window, GCRA, leaky bucket), per-worker and global bulkheads, CoDel load shedding
  • AES-256-GCM encryption at rest with key rotation
  • Stale job rescue via heartbeats
  • Bulk operations: insert_all/1, cancel_by/2, retry_by/2
  • Synchronous drain_queue/1 for deterministic tests
  • OpenTelemetry instrumentation as a separate package, opentelemetry_shigoto

Feedback and issues welcome on the repo.

1 Like