Numerl - matrices in Erlang

For my master thesis, I’m currently writing the numerl library.

You can find it at: https://github.com/tanguyl/numerl

It aims to bring, via the power of NIFs, fast vector and matrix operations to Erlang!
So far, a couple of those (as well as some BLAS) have been added.

I’d really love to get any advice/feedback :slight_smile:

11 Likes

Sounds exciting! Try take a look at https://www.rebar3.org this project can help you build your library. Here is a documentation about - how build Nif’s - Building C/C++ | Rebar3.

5 Likes

Thank you for the feedback! I’ll get a look into it, it does look more elegant then a selfmade makefile.

6 Likes

Of course :upside_down_face: because this project is beside of Erlang Open Source GitHub - erlang/rebar3: Erlang build tool that makes it easy to compile and test Erlang applications and releases.. And used in a lot of projects. If you like Makefile you can also take a look to other tool https://erlang.mk/ - here is a repo GitHub - ninenines/erlang.mk: A build tool for Erlang that just works.

5 Likes

I see you are discussing how to build nif, maybe I have a good suggestion. When I was learning nif, I wanted to build nif easily and conveniently. I didn’t want to use makefile. I thought that both win and linux should be available for compiling and platform-free, so I went to study it. I saw GitHub - davisp/jiffy: JSON NIFs for Erlang and I learned it and made some small changes. Finally you can take a look at eUtils/c_src at master · ErlGameWorld/eUtils · GitHub GitHub - ErlGameWorld/eNpc: Erlang Nif or port_driver generic compilation scripts for Windows Linux Mac

5 Likes

Thank you for your answer! Being able to compile platform-free is great, and your project sells a lot. However, as most explanations/examples are done into Chinese, I do have quite some difficulties understanding how it works… It would be fun to someday explore it, but for this project I will stick to either rebar3 or erlang.mk.

5 Likes

I used a makefile to keep a tool I’m used to; Rebar3 looks like the strongest tool for Erlang. I’ll adapt shortly my project to it.

6 Likes

I wrote this a couple of years ago, the project was dropped so it never used and
don’t have any experience with blas so don’t know how useful it could be or
if the api was on correct level.

Maybe some ideas are useful and can be reused.

6 Likes

The readme mentioned dirty schedulers but a cursory view over the code I couldn’t discover dirty scheduling.

We want to use a low-level (non dirty scheduled) BLAS on small matrices and build a concurrency friendly larger matrix lib on top of it (using the small matrices as block matrices) in the next project.

2 Likes

I couldn’t get everything to compile on my m1 (I need to upgrade the OS I believe), but this may help : GitHub - starbelly/numerl at rebar-up

It uses the pc plugin for compiling, but you could easily adjust to use a Makefile instead. Here’s an old project of mine as an example that’s setup using a Makefile : hydro instead of using pc.

It also includes rebar3_hex and rebar3_ex_doc in project_plugins for when you’re ready to put this up on hex :wink:

You’ll notice some files have been moved around to be idomatic as well.

One suggestion I could make that’s aesthetic would be to use words instead of symbols for functions like +/2, ‘-/2’, '==/2`, (e.g., add/2, sub/2, eq/2, etc. ) instead.

I hope this helps! :slight_smile: Good luck :rocket:

3 Likes

Continue to read the sentence, it needs more work :slight_smile:
It was a prototype but it was never used so this is as far as I came.

2 Likes

Well in our case this might be a feature. Using dirty schedulers for something like that is always a crutch and we have plans to avoid it altogether to allow highly concurrent linear algebra operations.

2 Likes

I think we (the community) need to find one standard and well supported way to build NIFs. So you say pc is not working well, can it be fixed? How?

Makefiles called from rebar3 are a easy way but have their own issues with portability.

A special case which is not unimportant is cross compilation of NIFs and/or statically linked NIFs. For GRiSP we are patching the NIFs into OTP so they will be built and automatically statically linked on cross build. This currently requires a custom approach but if there would be a standard way of building NIFs in rebar3 (or a plugin like pc) we could maybe automate this.

3 Likes

The main issue with pc is that someone wrote that feature inside rebar 2.x more than half a decade ago and when spinning up Rebar3, we took it out because neither Tristan nor myself wanted to actually maintain C builds as a core feature. People turned the old code into a Rebar3 plugin that nobody has willingly maintained since then.

The whole maintenance model is essentially “someone patches a thing when they must” and otherwise everyone avoids it.

It’s still the most portable way to build C code around, but it’s been bankrupt forever in terms of maintainership.

4 Likes

I think pc works fine IMHO, possibly room for improvements, but easily doable. I also agree a standard would be great. One great thing about standardizing on pc would be being able to support other languages needs. For example mix calls the rebar3 bare compiler with an environment variable telling it where to output artifacts. With the recently released version of pc that is now supported, so if you’re using pc, your project gets that support without you having to do anything.

Maybe a group of us could take on maintaining it.

2 Likes

Ok that really looks like a job for EEF, will think about how to achieve it

3 Likes

Hello there!

TLDR: hex and M1 on the way, the rest is done.

Firstly, I apologize for answering this late. Communication is not one of my strong points, and I’ll do my best to be more reactive here.

Secondly, your ‘rebar3’ rewrite is awesome! I’ve done some tweaks on your work: and now numerl is available as a rebar3 dependency with work for hex on the way.

Currently, I left behind the makefile and use pc instead, which works without issues.

Your idea of switching up names makes a lot of sense. I was using symbols for legacy reasons. This project can be summarized in porting an Erlang library made by another student to NIFs, and the original library used those symbols. I switched to a mixture of your suggestions/numpy names.

Finally, other people are having issues with this library on M1 mac. The good news is that I have temporarily access to such a computer; hopefully I can make it work in the next couple of days.

Overall, thank you a lot for your ideas and contributions! For the next couple of days, work on hex and M1 support will be done.

3 Likes

That work is impressive. I regret not stumbling on it when I started working on numerl; it would have saved me quite a lot of time.

It is fun to see the different choices we made, such as using Resources instead of Binaries allowing your library the modification “in place” of matrices.

However, I do have ‘stupid’ question; for which reason are all your NIFs declared static?

3 Likes

I don’t know, it might have been my first time using nifs and I followed some example
for the function declarations.

3 Likes

But I divided the interface in parts, if I remember correctly, one high level with functional api,
and one low level where you access the data more raw so that can manipulate or chain
operations without converting the to/from erlang terms.

PS: I’m saying this without looking nor knowing if it was actually implemented but the idea was that anyway :smiley:

2 Likes