glazer started as a blazing-fast NIF JSON codec (single-pass, straight to/from native terms - no AST). In 0.3.0, the same engine now parses and encodes YAML and CSV too.
The numbers:
JSON: faster encoding than everything I tried - jason, thoas, euneus, torque, jiffy, simdjsone, OTP json- and neck-and-neck with torque (Rust sonic-rs) on decoding
YAML: 10x faster than yaml_rustler/fast_yaml, up to 100x faster than yamerl/ymlr
CSV: 2-20x faster than nimble_csv, and csv/erl_csv don’t even finish on large files (timeout) while glazer is done
Plus: streaming decode for JSON/CSV, bignums support, configurable null, zero external C++ deps.
Really nice work on glazer. The CSV throughput looks impressive.
Quick question so I’m reading the API right before I run a few numbers of my
own (against our proprietary CSV lib): when decoding CSV, does glazer always hand back every field as a binary?
i.e. csv_decode/1,2 returns [[binary()]], and any actual type interpretation : integers, floats, booleans, dates … is left to the caller?
Just want to make sure I’m comparing like-for-like.
Currently yes, though it would likely be a good addition to perform a type conversion on the C++ side if the {col_types, [integer|float|boolean|date|{date, Format::binary()}|binary|charlist]} option is provided. You can submit an issue, and I’ll try to get to it soon.
This weekend I made a few non-backward compatible changes needed to normalize API across JSON/YAML/CSV codecs, but as of now it should be pretty stable.
Announcing the first stable release 1.0.0 of glazer battle-tested in production under continuous 1M req/s rate for two weeks handling JSON 2k request payload without errors.
We switched from jiffy to glazer (1.0.x) a few weeks ago and put it straight to
work: decode JSON, transform, re-encode as CSV, on a stream of roughly
25 GiB per minute, continuously, for about a month. It ran extraordinarily
well. No surprises, nothing to go back and fix, and after a while we
stopped thinking about it entirely, which is the highest compliment I can
pay to a dependency.
Author of erl_csv here, dunno when did you compare against it but a couple of months ago I ran a big bunch of optimisations on it based on nimble_csv, so now it should be fairly equivalent to the latter for all kinds of input. Which is still an order of magnitude worse than glazer but at least it’s not that terrible anymore
That not to say, super super cool work. I noticed at a quick glance you’re also using SIMD magic (love it!). For big inputs I saw you’re offloading to dirty schedulers. Have you considered splitting the work with enif_schedule_nif? It has advantages and disadvantages. Obviously a single continuous uninterrupted work is much faster (cache locality and branch prediction and whatnot), but also dirty schedulers are a limited pool of workers and there’s also a price paid on scheduler transfers (that obviously fades away with very big inputs, but with the middle-sized inputs… gets shadier to measure). No strong opinions really, it’s all tradeoffs, just curious about what you think
There are definitely still some micro-optimizations we could squeeze out down the line, but honestly, I’m super happy with its current performance and stability - it’s been running solidly in production and already saved us about 15% on our AWS bill!
Did my best to improve the situation for erl_csv, one thing was that your measurement was probably taken a long time ago before I gave some love to that library, and also, line termination wasn’t configured. Line termination fixed at Call erl_csv with a CRLF delimiter in the CSV benchmark by NelsonVides · Pull Request #15 · saleyn/glazer · GitHub and with yesterday’s version of erl_csv, glazer is 6-10x faster for both decoding and encoding. So I ran some much deeper optimisations on erl_csv, and now glazer is 2x faster for decoding and 4-5x faster for encoding.
I doubt it can get any bit faster with pure erlang but for more casual use cases might be worth the simplicity of pure erlang code — for hot code indeed, no chance to beat native SIMD here, a 15% cloud bill difference is insane
Anyway, can I ask you to update the comparison in your repo so it doesn’t come up thaaaat terrible?
