# Reduce memory usage of large maps **URL:** \ **Category:** Questions / Help\ **Tags:** erlang\ **Created:** [August 4, 2026, 11:30am UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843 "2026-08-04T11:30:15Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![systream](https://erlangforums.com/user_avatar/erlangforums.com/systream/32/1514_2.png) [@systream](https://erlangforums.com/u/systream)\ **Post date:** [August 4, 2026, 11:30am UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/1 "2026-08-04T11:30:15Z")
We have a service that uses graphQL. This graphQL file is parsed into an AST. It is a huge map (~7 Mb) that contains atoms, small binaries, lists, and maps. The problem is that we have to analyse the schema every time a request comes to our server. So the AST gets copied to the process heap. For example, if we have 100 concurrent requests, it takes 700 MB of memory. We expect the schema to grow so it will be worse. My question is, is there a better way to store the AST or any trick what we can use, to optimize the memory usage? ---
**Author:** ![jhogberg](https://erlangforums.com/user_avatar/erlangforums.com/jhogberg/32/179_2.png) [@jhogberg](https://erlangforums.com/u/jhogberg)\ **Post date:** [August 4, 2026, 11:57am UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/2 "2026-08-04T11:57:45Z")
[persistent\_term — OTP 29.0.5 (erts 17.0.5)](https://www.erlang.org/doc/apps/erts/persistent_term.html) ? ---
**Author:** ![systream](https://erlangforums.com/user_avatar/erlangforums.com/systream/32/1514_2.png) [@systream](https://erlangforums.com/u/systream)\ **Post date:** [August 4, 2026, 12:57pm UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/3 "2026-08-04T12:57:10Z")
Thanks, we might write it back. We used to use the persistent\_term function, but the schema changes from time to time, so it drops the performance for seconds. ---
**Author:** ![jhogberg](https://erlangforums.com/user_avatar/erlangforums.com/jhogberg/32/179_2.png) [@jhogberg](https://erlangforums.com/u/jhogberg)\ **Post date:** [August 4, 2026, 1:13pm UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/4 "2026-08-04T13:13:53Z")
If the updates are relatively few and you’re okay with ”leaking” memory for a while, consider placing the schema under a new `persistent_term` key for each version (index it by an `atomic` or whatever). That way you can shift the cost of garbage collecting the old schemas until off-peak hours. ---
**Author:** ![mikpe](https://erlangforums.com/letter_avatar_proxy/v4/letter/m/b487fb/32.png) [@mikpe](https://erlangforums.com/u/mikpe)\ **Post date:** [August 5, 2026, 11:56am UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/5 "2026-08-05T11:56:27Z")
Instead of having fixed code interpreting occasionally-changing data, you could consider generating code specialized for a given version of the data. ---
**Author:** ![systream](https://erlangforums.com/user_avatar/erlangforums.com/systream/32/1514_2.png) [@systream](https://erlangforums.com/u/systream)\ **Post date:** [August 6, 2026, 7:49am UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/6 "2026-08-06T07:49:48Z")
The way it works is that a request came to the server, we do different kind of traversal, find operations on the AST based on the graphQL request. Usually the same requests have been made to the server, so the result can be cached. We used to use persistent\_term but, when a schema changes we need to erase multiple items which kicks multiple gc on the vm. We already using the schema versioning because we need to deal the request during the schema change. Now I just need to figure out the off-peak hours. 🙂 ---
**Author:** ![lpil](https://erlangforums.com/user_avatar/erlangforums.com/lpil/32/222_2.png) [@lpil](https://erlangforums.com/u/lpil)\ **Post date:** [August 10, 2026, 9:16am UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/7 "2026-08-10T09:16:29Z")
What does each process do with this large data structure when handling a request? ---
**Author:** ![systream](https://erlangforums.com/user_avatar/erlangforums.com/systream/32/1514_2.png) [@systream](https://erlangforums.com/u/systream)\ **Post date:** [August 10, 2026, 9:59am UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/8 "2026-08-10T09:59:24Z")
We do Graphql schema federation. In simple terms, basically different parts of the schema belong to a different services and we have to figure out how to execute based on the query and the given schema. ---
**Author:** ![lpil](https://erlangforums.com/user_avatar/erlangforums.com/lpil/32/222_2.png) [@lpil](https://erlangforums.com/u/lpil)\ **Post date:** [August 10, 2026, 10:17am UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/9 "2026-08-10T10:17:43Z")
Could it be broken up into those sub-schemas and only the needed parts be loaded by each process? A technique that could be a good fit (though I’ve not seen it used in some time) is to compile the data into a module, taking advantage of the literal pool. Updating a module is fairly expensive, but it doesn’t stop-the-world like persistent term does. The [mochiglobal](https://github.com/mochi/mochiweb/blob/main/src/mochiglobal.erl) and [fastglobal](https://github.com/discord/fastglobal) libraries are built around this idea. ---
**Author:** ![jhogberg](https://erlangforums.com/user_avatar/erlangforums.com/jhogberg/32/179_2.png) [@jhogberg](https://erlangforums.com/u/jhogberg)\ **Post date:** [August 10, 2026, 10:45am UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/10 "2026-08-10T10:45:54Z")
> [@lpil](#): > > A technique that could be a good fit (though I’ve not seen it used in some time) is to compile the data into a module, taking advantage of the literal pool. Updating a module is fairly expensive, but it doesn’t stop-the-world like persistent term does. The [mochiglobal](https://github.com/mochi/mochiweb/blob/main/src/mochiglobal.erl) and [fastglobal](https://github.com/discord/fastglobal) libraries are built around this idea. Unfortunately, it does global GC like `persistent_term` does :-/ ---
**Author:** ![lpil](https://erlangforums.com/user_avatar/erlangforums.com/lpil/32/222_2.png) [@lpil](https://erlangforums.com/u/lpil)\ **Post date:** [August 10, 2026, 11:26am UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/11 "2026-08-10T11:26:12Z")
Oh! I thought it did the code version check more slowly but asynchronously prior to purging any processes. My bad. ---
**Author:** ![jhogberg](https://erlangforums.com/user_avatar/erlangforums.com/jhogberg/32/179_2.png) [@jhogberg](https://erlangforums.com/u/jhogberg)\ **Post date:** [August 10, 2026, 11:50am UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/12 "2026-08-10T11:50:12Z")
It does, but it also needs to GC the literals associated with the module, just like `persistent_term`. ---
**Author:** ![lpil](https://erlangforums.com/user_avatar/erlangforums.com/lpil/32/222_2.png) [@lpil](https://erlangforums.com/u/lpil)\ **Post date:** [August 10, 2026, 12:17pm UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/13 "2026-08-10T12:17:05Z")
ah of course! I didn’t think of that. Thank you ---
**Author:** ![systream](https://erlangforums.com/user_avatar/erlangforums.com/systream/32/1514_2.png) [@systream](https://erlangforums.com/u/systream)\ **Post date:** [August 31, 2026, 1:40pm UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/14 "2026-08-31T13:40:42Z")
Usually when the schema changes, multiple persistent terms are deleted (even if it is delayed). I thought that if we knew all the keys to be removed, maybe the erlang virtual machine could do it’s magic and do the deletion in an optimised way? I mean we could have more than 100 persistent terms each version of a schema. ---
**Author:** ![garazdawi](https://erlangforums.com/user_avatar/erlangforums.com/garazdawi/32/175_2.png) [@garazdawi](https://erlangforums.com/u/garazdawi)\ **Post date:** [August 31, 2026, 2:26pm UTC](https://erlangforums.com/t/reduce-memory-usage-of-large-maps/5843/15 "2026-08-31T14:26:51Z")
> [@systream](#): > > maybe the erlang virtual machine could do it’s magic and do the deletion in an optimised way? It could, but it doesn’t, mostly because no one has gotten around to implement it yet and that your usage scenario of deleting keys from persistent term is never going to be cheap nor recommended no matter what you do. However, what you could do is store a map/tuple into a single persistent\_term key and then just lookup your values in there. Then you just need to delete one key and should not have as large an issue?