Kafine - Kafka client for Erlang

Your Libraries & OS Mentoring Libraries
,
1

Kafine is a Kafka client for Erlang.

We’re (Happening, the technology stack behind Superbet) pleased to announce the initial public release of Kafine, our new Kafka client for Erlang.

While Kafine is all-new, and still a bit rough around the edges, it’s based on years of experience using Kafka in production with our existing internal Kafka libraries.

The secret sauce in kafine is the automatically-generated message codecs in the kafcod project, which aims to achieve better performance and compatibility than existing Kafka client libraries in the BEAM ecosystem.

We’re in the process of publishing packages to hex.pm. Until then, you can take a look at the repositories on GitHub:

Note that this release is currently considered unstable. We’re using it in production, but only for a small number of non-critical workloads. We’re working towards being production ready, and we’d appreciate feedback from the community.

18 Likes
2

Kafine v0.6.0

We’ve pushed the 0.6.0 release of kafine. Change notes are also at that link, but for completeness:

  • First pass implementation of process-per-partition. The consumer
    callback is invoked from an isolated process (one per partition).
    Right now, this doesn’t do much. In future, it will be used to improve
    concurrency and reliability.
  • Fix: docker/get-host-ip.sh doesn’t work on brand-new Ubuntu
    installation (discovered while attempting a demo at CodeBEAM Lite London).
  • Improvements to generated documentation, including main README.
  • Added/improved unit tests.
  • Bump dependencies.
  • Various documentation, comment tidying up tasks.
3 Likes
3

Kafine 0.12.0

We’ve pushed the 0.12.0 releases of kafine, kafcod and kamock.

Here’s what’s changed:

kafine

  • Implement kcat-style negative offsets.
  • This replaces the ‘last’ offset reset policy, and makes offset reset
    policy more consistent with other Kafka client libraries.
  • Add some notes about use of the telemetry library.
  • Fix ‘eqwalize’ target. It now runs in the ‘eqwalizer’ rebar profile,
    so as to include the type hints for OTP.
  • Fix invocation of ‘git-vsn’ script.
  • Pin kafcod dependency to 0.12.0.
  • Update notes about running integation tests.
  • Report consumer ref in logs.
  • Implement ‘acks’ option in producer.
  • Implement retries in producer.
  • Implement compression in producer.
    • Currently only ‘none’, ‘gzip’ and ‘snappy’.
    • ‘zstd’ support will come with OTP-28, once it’s built-in.

Bugs:

  • We think there’s a bug to do with negative offsets and resuming paused
    consumers. We’ve not found a reliable replication yet.

kafcod

  • Rebuild codecs against Kafka 3.8.1
    • Add workaround for tagged fields specified with strings instead
      of integers.
  • Fix ‘eqwalize’ target. It now runs in the ‘eqwalizer’ rebar profile,
    so as to include the type hints for OTP.
  • Fix invocation of ‘git-vsn’ script.
  • Update ‘reassign’ example:
    • Use the ‘kafcod_connection’ module.
    • Optionally make existing leader sticky.
    • Optionally make reassign rack-aware.
    • Add ‘elect-leaders’ feature.
    • Add ‘validate-replication’ feature.

kamock

  • Add kamock_coordinator:info/1 and kamock_coordinator:rebalance/1.
  • Implement DESCRIBE_CLUSTER.
2 Likes