Erlang shall power the Metaverse

Wondersye · January 15, 2022, 4:48pm

Hi,

A slightly less controversial title could have been: any Erlangers interested in video game programming?

After all, even for standalone games, there used to be a SDL binding, there is GL/GLU support through wx now, various interfacing solutions could target appropriate libraries.

And Erlang could shine in the context of networked, possibly MMORPG-like games, for the server-side (in charge of the “world simulation and update”), whereas client-side could be delegated to Godot, Unity3D, Unreal or any other, the communication between the two relying on data formats such as Protobuf, etc.

Anyway, regardless of the nature and specificity of the “interactive multimedia applications” that would be targeted, I was wondering whether there was some interest in (attempts of) video game (probably hobbyist) programming in Erlang, so that likely-minded folks could possibly share in some sub-community thoughts, advice, code, projects?

Best regards,

Olivier.

vkatsuba · January 15, 2022, 5:04pm

Here is some info:

Also looks like was similar discussion in Reddit - Dive into anything

vkatsuba · January 15, 2022, 5:21pm

Also if you want play with Lua in Erlang - here is amaizing project GitHub - rvirding/luerl: Lua in Erlang which created by @rvirding.

kartheek · January 15, 2022, 6:11pm

I have built game servers in elixir and other languages. Most of the game servers I built are for casual turn based games.

I have seen posts of erlang powering game servers. Most of them are quite old posts.

I attempted to use elixir game servers for physics based games like 8 ball pool, 9 ball pool, air hockey and carroms. There was lag and jitter in object movement and a lot of problems. Realised i need to put in a lot of effort and I was reinventing wheel as most of it is available in other networking libraries like Mirror. I scrapped and rebuilt them in Mirror.

Theoretically Unity3d clients and erlang/elixir server look good. In practice, using them together is very difficult - unless its a huge corporation with plenty of resources. It does not make sense for indies and smaller teams to mix them (except for casual games).

There are mature game networking frameworks out there which handle most of the corner cases. Its easy to use them rather build a game server in erlang/elixir (Unet, Mirror, Dark Rift for Unity)
Communication between client and server is a huge issue - Unity side SDK needs to be written to integrate server into game loop.
Talent pool for building a game server in elixir/erlang is very small.
Documents / knowledge is scarce - except for some presentations and GitHub sample projects.
Verifying collisions, simulating physics, importing models(OBJ files) and game environment is very difficult in erlang/elixir if you are building authoritative game servers.
Ecosystem of libraries and frameworks relating to games are very less.
Erlang/elixir are more suitable for building relay game servers. Even in this case - expect a lot of work on Unity side.
Game servers for casual turn based games can be easily built on erlang/elixir.
Very easy to achieve all of the above using networking libraries of Unity or other game engines. A server build of the game can be made on click of a button and can be deployed on to Playfab, etc.

I have answered recently about a similar topic in elixir forum

Dom’s post embeds a tweet - it is gives Information about various types of multiplayers and availability of developers who can actually build them.

I am sharing my experience - might not be the same for everyone.

I want hear more from other developers about their experience of developing erlang/elixir game servers .

Wondersye · January 15, 2022, 10:38pm

@vkatsuba : thanks for your pointers, which are very relevant indeed; here is my take on them:

unfortunately the Reddit thread is probably still right about the few solutions available for graphical output in Erlang (and, apart possibly an OpenAL binding, audio output is maybe even more problematic)
Mark Allen’s talk is interesting regarding its use of the gen_server-based architecture, the design of his game API and its simple, direct, UI-less demo
the earth-defender project seems to be a good example of browser-based game client; that they mention so early the CAP theorem was a surprise
the Eonblast game stack raises key problems, including the serialization of the game state; as for the scripting (here through LUA), I suppose that its purpose is to make at least part of the game logic significantly quicker/simpler to write than having it done directly in Erlang… is it so true? (unless some game elements were to be implemented by non-programmers, which may, or may not, be desirable)
the ErlangFactory talk of QP mentions this Demonware presentation about Erlang, with offers a rich Erlang-related operational feedback: https://www.erlang-factory.com/upload/presentations/395/ErlangandFirst-PersonShooters.pdf

@kartheek:

thanks for the much valuable information that you shared, including for pointing out libraries like Mirror (1, 2), DarkRift or UNet (becoming Netcode), some of them being even free and/or open source; indeed overcoming as much as possible lag and jitter must require a lot of effort, so these libraries are very welcome, yet, as I understand it, they could not integrate easily to an Erlang server logic? Maybe then their client-side (ex: Unity-based) could be re-used as it is, and, at least for some well-chosen low-level transports, a simple Erlang version of their server-side could be devised? (yes, it still sounds like a large chunk of work unfortunately)
the scability, fault-tolerance and hot code upgrade brought by Erlang should be very welcome for game servers, and may compensate for some difficulties; as for the latency (thinking to a real-time game as opposed to a turn-based one), I would expect Erlang to fare well, at least on average; I suppose that the network-induced latency (that has to be overcome by all approaches anyway) would be greater than the one induced by the game logic / Erlang itself
on the server-side, the concurrency allowed by Erlang is certainly a key asset in order to evaluate “in parallel”, on multiple cores/CPUs, a large number of players and NPCs, even though I believe that in a concurrent setting these instances should not interact directly (by means of direct messages), but should be properly synchronised by the game logic instead (notably in order to avoid causality/consistency issues); I wonder to which extent the commercial (non-Erlang) MMORPGs are truly concurrent, and what the order of magnitude in the number of players interacting for real (not through sharding) one of their servers can support
verifying collisions, simulating physics: indeed a problem; depending on the game, needs might be kept low enough so that they can be fulfilled in a simple, Erlang-based ad hoc form (rather than having to integrate a full-blown open source physics engine, for example as a NIF)
still on the server side: possibly that importing 3D models might be manageable by converting first (ex: thanks to Blender) FBX/OBJ content in glTF 2.0 format (natively read by Godot on the client side), and then reading it (ex: like Wings3D does), especially if the game logic is not interested so much in features like textures, materials or even animations

Clearly many elements will need to be (over)simplified first so that the whole remains manageable for hobbyistic endeavors.

Well, much food for thoughts and experiments, many thanks to both of you!

kartheek · January 16, 2022, 6:24am

Thank you @Wondersye for you thoughts:

If we write a transport for Mirror, then erlang server will be a relay server. Like you pointed - huge chunk of work. For some of the network protocols like kcp, litenet, etc - there are no libraries which are well maintained. One has to stick to tcp for good quality libraries.
If server side libraries are different from Unity libraries for physics and collision - there might be a chance of different results between client and server.
NIFs have their own problems (particularly if they are complex third party libraries).
Game environments (say the objects, obstacles, trees, etc) should be same in both Client(unity) and Server(erlang) for consistency. If there is a way to import scene data into erlang for object positions, etc it would be ideal. I don’t know if anyone does this using erlang.
A game server need not achieve huge concurrency levels like application servers. Horizontal scaling is good enough. Containers have simplified this problem. New servers will be spawned based on demand. Only players who are playing together need to connect to the same server. Playfab, Google Agones, Unity Multiplay and AWS offer game server hosting service which auto scale based on player demand.

A lot effort is needed to build game servers in erlang which are not relay servers. This needs exponential resources and time when compared to writing a performant application server.

I would not use erlang for anything other than relay server or casual turn based games. One can go live say within 3-4 months using Mirror or other libraries - same cannot be guaranteed using erlang(speaking for developer like me).

I feel Rust is better placed when it comes to building game servers compared to erlang. There are many game frameworks being developed in Rust like bevy, etc. Rust libraries like tokio + bevy, etc are promising (i have not built one yet).

SisMaker · January 16, 2022, 4:15pm

I have seen many game servers are written in erlang, some of them have made a lot of money, and these game types include mmo, slg, arpg, card, web games, etc. I am also an erlang game Server developer, then I also wrote some erlang game server related projects
I have used c++, c++/lua, go, erlang to write game servers, but I think the best way is to use erlang, which is inherently concurrent and distributed, can easily simulate many player processes, can easily expand many servers, and The hot update code can easily hot update game logic and bugs, which are not absent in other languages.
But there are also disadvantages. The computing performance of erlang is not good, so some game servers that require computing performance are not suitable for writing in erlang. But most types of servers can be written in erlang.
If you are interested, you can take a look herehttps://github.com/ErlGameWorld. I started with configuration, protocol, logger, network library, database driver, and wrote some other tools. I want to build a perfect mmo server from scratch based on erlang. Recently I in a webserver suitable for game services.

Wondersye · January 16, 2022, 9:37pm

Thanks kartheek for your added points; here are the corresponding plans that I have (before any actual experimentation - so they must be quite naive):

for the carrier, I thought that only a mix of UDP (ex: for disposable continuous world updates) and TCP (ex: for content streaming) could do the trick; it is very fortunate that Mirror is open-source, moreover with a permissive licence: plenty of learning/forking opportunities!
I would favor an authoritative server (simpler, defensive against cheaters), leaving any client-side physics and collision to be done (for example by Unity) only for local rendering purpose; the player would have to live with any transient inconsistencies (that happen anyway even with high-end game clients, when they make update predictions in an attempt to mask network latency)
NIFs would be used only as last resort (expensive to integrate+jeopardizing the VM); especially with JIT, sticking to Erlang may suffice for light-enough needs (another choice aggravating the server-side burden!)
hopefully, thanks to glTF, on either client/server side, the geometry of world elements would derive from the same reference
in an ideal setting, there would be a unique world state (no horizontal scaling) that would be shared between all players (typically, for spatial-based interactions, based on octrees); despite Erlang concurrency and scalability, this would certainly be hard on the server (knowing that moreover instance interactions would have to be properly synchronized by the game scheduling logic, inducing a large extra cost in processing and latency); however, even considering a “large” number of simultaneous players, I have elements that make me think that Erlang could do that - albeit at an unknown overall resulting speed; luckily one may advocate that a slow game pace is actually a gameplay feature
Bevy looks quite impressive indeed, and may be a source of inspiration; as an hobbyist, I feel no real productivity pressure, so exploring the topic in Erlang and mostly from scratch has my preference, as a learning opportunity - even if it is a lot longer or does not yield much result!

Thanks again,

Olivier.

Wondersye · January 16, 2022, 9:51pm

Thanks SisMaker for the sharing!

I also made a few steps forward, some of them concentrated here, a toolbox to gather some general-purpose facilities but also some that are more related to such multimedia applications (a bit of: linear algebra, protocols and data formats including first glTF elements, integration of the generation of speech synthesis, some GUI and OpenGL efforts, etc.); everything being rather in early stage, yet meant to progress as much as needed.

I also started collecting information about 3D management here, since it is a rich theme.

As for the aforementioned server-side game scheduling logic to evaluate instance interactions concurrently, I think open-source (Erlang) solutions may already exist.

So indeed many Erlang-related elements could - and should - be shared!
(yet any resulting game could probably not, at least because of the assets - i.e. 3D models, music, etc.).

LincolnBryant · January 20, 2022, 5:14pm

I’m really excited for this thread. I have been working in the game server space for a few years now as a solo hobbyist.

I’ve been building my own sort-of-general library for integrating Erlang and Godot called Gremlin. I have a pretty simple system setup where I use websockets (via Cowboy) and protobuf (via GPB) with a client written in Godot. Godot has native support for Websockets and I use a nice library called Godobuf for decoding/encoding protobuf messages. This means of course that all traffic happens over TCP, but I’ve been trying to be careful about encapsulating things such that the Websocket/TCP layer could be swapped out for a more traditional UDP-based solution.

The Gremlin core supplies routing for packets from your game client to your game application on the server side, a very basic account system, plus some common utilities like functions for working with vectors, tile grids, etc. Gremlin’s core library is a perpetual work in progress as I identify more useful things to include All of the stateful stuff that needs to be persisted to disk just uses Mnesia.

Gremlin also defines some behaviours for setting up your own callbacks to your game application. The RPC design is intentionally old school - you can specify an RPC by defining an OpCode, a Protobuf message associated with that RPC, a client function and a server function. Then I’ve built a (very ugly ) code generator in Erlang that will build a client library in GDScript that handles all of the WebSocket communication and message decoding in Godot. In Godot, you simply drop this into place and add it as an autoload. In your game, the Gremlin client library emits these messages as signals.

Over the summer last year I worked in the other direction as well. I stared at the Luerl sources for a long time and wrote my own interpreter in Erlang for a subset of GDScript that I call “GDMinus”. It’s intended to be a strict subset of GDScript to facilitate server-side scripting with a syntax familiar to Godot developers. It’s a simple tree-walking interpreter and as such is probably 10x slower than the “native” GDScript, but has most of the basic GDScript constructs in place but it doesn’t support classes.

I’m not 100% ready to share any code just yet - I’m waist-deep in writing a game using Gremlin and it’s subject to massive overhauls and code cleanups at this point - but I’d to hear any feedback on the direction I’m going! I plan to release this project as Free Software.

I think Erlang is absolutely a killer language for writing game servers! Especially if you want to do stuff in the MMO or so-called Live Service game space. For the game I’m writing now, I really like the old school design of games like Everquest or Neverwinter Nights where every area of play is a separate board or “zone”. While seamless worlds are very fashionable and modern, I think there’s a lot of charm in the older designs. For the game I’m building now, I model each zone as a gen_server, with NPCs and whatnot being gen_statems that interact with the zone they’re spawned in. We’ll see how it turns out

Wondersye · January 20, 2022, 9:29pm

Gremlin looks very relevant! If it could, as much as possible, abstract-out the actual carrier (direct TCP, a Websocket, UDP) this would be ideal indeed; the message-based mode of operation of Protobuf relates well (GPB being a very solid technological choice) and leads naturally to RPC… and having a client-side automatically prepared in Godot for that would be certainly a huge advantage, especially if it is generated from Erlang.

I do not know enough Godot and Unity to have a clear opinion on the best choice for the context at hand (free software vs more services/higher quality rendering/plenty of assets readily available). Should no killer rendering feature exist in Unity but not in Godot, and should, even for high-end content, an actual asset conversion be doable to target the latter (materials, prefabs and advanced animation-related features may be a problem?), many (like me) could prefer Godot; the perspective that it would then neatly integrate to an Erlang server-side thanks to Gremlin would be awesome!

The other direction, for server-side (Erlang-powered) scripting in a GDScript dialect, is interesting as well, and surely another technical feat! Yet on the server I would still probably prefer remaining in pure Erlang as much as possible and having the slimmest dependency onto the client-side software (seen mostly as a dumb view and a basic controller, in MVC parlance), thinking that the libraries and languages there might be less interesting in their own right / more prone to change.

As the functional services seem to me relatively close, Gremlin could end up as an Erlang-based Mirror-like library for Godot, isn’t it? Have you considered taking inspiration from Mirror (or the other technologies Kartheek mentioned), or even sought some level of compliance with it, at least on an API level? Switching more easily solutions on the client and/or server side would offer much welcome flexibility.

Overall, there are so many needs besides rendering and middleware (persistence, world simulation, resource management, live upgrade, etc. - and of course the game logic itself) that, especially for hobbyists, it is clearly overwhelming. Please share pointers here to your code when you are ready to open-source it! IMHO, the biggest threat to such ambitious game-related projects is not a premature sharing of unpolished work-in-progress, but the exhaustion of motivation because the task is too big for a solo developer, or even for a few of them; this is why I started this thread at the first place

A parallel topic could consist in exploring the actual game that each Erlanger interested by this thread would like to create (the great absentee in these discussions), as this may explain many of the design decisions. Moreover the second biggest threat to these projects is starting the creation of a game and finishing by devising some library instead; another enjoyable, yet different, goal…

AstonJ · January 21, 2022, 2:02am

I’m glad you said that because I was going to ask what games you’re all working on but didn’t want to take the thread off-topic - so please, start one

If you include a link to the first post of this thread in the new thread, it will automatically become a linked topic - or alternatively you can click on the little time-stamp in the top right of the post and then click on ‘new topic’ (and then you can edit the post as you see fit):