Can't get rocksdb to build with compression support

Questions / Help
,
1

Trying to get rebar.config in my main project to pass down the configuration options to get rocksdb to build with support for snappy, bz2, and lz4 compression support. Here’s my rebar.config and below is the output I get which shows the necessary environment parameters are still not getting set. As you can see I’m trying a number of different ways to get this stuff set but clearly I’m missing the mark.

{erl_opts, [debug_info]}.
{deps, [
    {rocksdb, {git, "https://gitlab.com/barrel-db/erlang-rocksdb.git", {branch, "master"}}},
    {jiffy,"1.1.1"}
]}.


{port_env, [
    {"ERLANG_ROCKSDB_OPTS", "-D WITH_SNAPPY=ON -D WITH_LZ4=ON -DWITH_BZ2=ON"},
    {"CFLAGS", "-D WITH_SNAPPY=ON -D WITH_LZ4=ON -D WITH_BZ2=ON"},
    {"LDFLAGS", ""}
]}.

{overrides, [
    {override, rocksdb, [
        {port_env, [
            {"USE_SNAPPY", "ON"},
            {"USE_ZLIB", "ON"},
            {"USE_LZ4", "ON"},
            {"CFLAGS", "-D WITH_SNAPPY=ON -D WITH_LZ4=ON -D WITH_BZ2=ON"},
            {"LDFLAGS", ""}
        ]}
    ]}
]}.

{shell, [
  % {config, "config/sys.config"},
    {apps, [abstract_parse]}
]}.

rm -rf _build; rebar3 compile

scherrey@pixies:~/develop/abstracts/abstract_parse
$ rm -rf _build; rebar3 compile
*DEBUG* Current CONFIG: [{erl_opts,[debug_info]},
                         {deps,
                             [{rocksdb,
                                  {git,
                                      "https://gitlab.com/barrel-db/erlang-rocksdb.git",
                                      {branch,"master"}}},
                              {jiffy,"1.1.1"}]},
                         {port_env,
                             [{"ERLANG_ROCKSDB_OPTS",
                               "-D WITH_SNAPPY=ON -D WITH_LZ4=ON -DWITH_BZ2=ON"},
                              {"CFLAGS",
                               "-D WITH_SNAPPY=ON -D WITH_LZ4=ON -D WITH_BZ2=ON"},
                              {"LDFLAGS",[]}]},
                         {overrides,
                             [{override,rocksdb,
                                  [{port_env,
                                       [{"USE_SNAPPY","ON"},
                                        {"USE_ZLIB","ON"},
                                        {"USE_LZ4","ON"},
                                        {"CFLAGS",
                                         "-D WITH_SNAPPY=ON -D WITH_LZ4=ON -D WITH_BZ2=ON"},
                                        {"LDFLAGS",[]}]}]}]},
                         {shell,[{apps,[abstract_parse]}]}]

===> Verifying dependencies...
===> Fetching jiffy v1.1.1                                                                                                                                                        
===> Fetching pc v1.14.0                                                                                                                                                          
===> Analyzing applications...                                                                                                                                                    
===> Compiling pc                                                                                                                                                                 
===> Fetching rocksdb (from {git,"https://gitlab.com/barrel-db/erlang-rocksdb.git",                                                                                               
                   {branch,"master"}})                                                                                                                                            
+ mkdir -p _build/cmake
+ cd _build/cmake
+ type cmake3
+ CMAKE=cmake
+ cmake ../../c_src
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- cmake args -DCMAKE_POSITION_INDEPENDENT_CODE=ON;-DCMAKE_CXX_COMPILER=/usr/bin/c++;-DCMAKE_AR=/usr/bin/ar;-DCMAKE_BUILD_TYPE=Release;-DFAIL_ON_WARNINGS=OFF;-DPORTABLE=ON;-DUSE_RTTI=1;-DWITH_TOOLS=OFF;-DWITH_TESTS=OFF;-DWITH_GFLAGS=OFF;-DWITH_JEMALLOC=OFF;-DWITH_TBB=;-DWITH_SNAPPY=;-DWITH_LZ4=;-DWITH_ZLIB=;-DWITH_BZ2=;-DWITH_ZSTD=
-- Found RocksDB library: /home/scherrey/develop/abstracts/abstract_parse/_build/default/lib/rocksdb/_build/cmake/rocksdb-prefix/src/rocksdb-build/librocksdb.a
-- Found RocksDB includes: /home/scherrey/develop/abstracts/abstract_parse/_build/default/lib/rocksdb/c_src/../deps/rocksdb/include
-- Using OTP lib: /opt/erlang/25.3.2.2/lib - found
1 Like
2

I think your output was so long it got cut off.

1 Like
3

I cut it off just past the text where it shows the parameters passed to the C++ compiler. You’ll note that the - DWITH_SNAPPY= has nothing on the right side which means it didn’t get any of my attempts to set those values. If that’s not set it won’t build with compression.

1 Like
4

I see, thanks. It looks like the issue here is that erlang rockdb doesn’t use the port compiler (pc) anymore, thus you’ll have to set your environment variable ERLANG_ROCKSDB_OPTS in your environment.

1 Like