Need help trying to parse a complex binary protocol, getting: Warning: BINARY CREATED: binary is returned from the function

Questions / Help
,
1

Hi guys

I’m trying to parse a complex binary protocol but for the sake of simplicity, this is my problem:

01. -module(protocol).
02. -export([ parse1/1, parse2/1 ]).
03. 
04. parse1(<<$a, Rest/binary>>) ->
05.    %% call: command(a),
06.    parse1(Rest);
07. parse1(<<$b, Rest/binary>>) ->
08.    %% call: command(b),
09.    parse1(Rest);
10. parse1(<<Rest/binary>>) ->
11.    %% means end of protocol, and simply return Rest
12.     Rest.
13. 
14. parse2(<<Char, Rest/binary>>) ->
15.     case Char of
16.         $a -> %% call command(a);
17.            parse2(Rest);
18.         $b -> %% call command(b);
19.            parse2(Rest);
20.         _ -> %% means end of protocol, and simply return Rest
21.             Rest
22.     end.

parse1/1 and parse2/1 are equivalent.

When compiling this code, I got this warnings:

src/protocol.erl:4: Warning: BINARY CREATED: binary is returned from the function
%    4| parse1(<<$a, Rest/binary>>) ->
--
src/protocol.erl:14: Warning: BINARY CREATED: binary is returned from the function
%   14| parse2(<<Char, Rest/binary>>) ->

What needs to be done to avoid the binary creation?

Also, the last clause is important for me:

20.         _ -> %% means end of protocol, and simply return Rest
21.             Rest

I want to convince the compiler to NOT create a new binary in this case because I know that the parsing is done.

Thanks

2

Are you actually interested in returning Rest ?

3

Yes I’m. Rest needs to be returned to be saved in some process’s state.

4

What is your environment? I compiled your code with
erl23, 25 and 26 with -Wall
and it compiled without any warnings…

5

Since you return Rest a binary has to be created.

The returned binary will be optimized to be a sub-binary (i.e. pointer into the original binary) by the compiler so there will be no copy of the data.

Edit: This exact scenario is described in the docs here: Erlang -- Constructing and Matching Binaries

6

Try with:

$ erlc +bin_opt_info protocol.erl

I’m using Erlang 26.0.2 on macOS and Linux.

7

Since you return Rest a binary has to be created.

ok.

The returned binary will be optimized to be a sub-binary (i.e. pointer into the original binary) by the compiler so there will be no copy of the data.

If i get it well, I can ignore any warning about returned Rest binary in this case:

20.         _ -> %% means end of protocol, and simply return Rest
21.             Rest

@garazdawi is there a way to see what the compiler is doing (i.e creating a sub-binary and not performing a binary copy)?

8

Not that I know of… also it isn’t always creating a sub-binary. If Rest is very small, it may create a new binary with copied content in order to be able to GC the parsed binary.

Maybe @bjorng knows of a way? or is it something we can/should add?

1 Like
9

No, but it always creates a sub-binary unless the number of bytes in the binary would be less than 64 bytes, in which case it will create a heap binary and copy the bytes into that.

It is recommended to not keep the bin_opt_info option permanently turned on, because you will never be able to eliminate all of the “warnings” that are emitted.

1 Like
10

Thanks for the clarification.