What is the best way of providing test data from outside the SUITE_data folder?

Questions / Help
1

I’m quite new using common test and I am trying to get rid of some absolute paths we have inside a test suite. The tests were working before because they were eunit and we run them in a custom way.

Imagine I have the following project structure to illustrate the situation:

.
├── apps
│   └── example
│       ├── src
│       └── test
│           ├── example_SUITE_data
│           └── example_SUITE.erl
└── extlibs
    └── test_data
        └── data.txt

The test data is in a git submodule (extlibs/test_data/) and the hardcoded paths we currently have point to them.

I tried to create a relative symbolic link pointing to data.txt inside the example_SUITE_data/ so I can later use ?config(data_dir, Config). Unfortunately, when I run the tests, the symbolic file is copied to instead of dereferencing the content of it. I’m also using rebar3 ct, so the content is copied to the _build/tests... and the symbolic link won’t work anymore because the relative path has changed. I suspect the result will be the same with ct_run so ?config(data_dir, Config) won’t work.

I’m looking for a way of telling common test to dereference the content of the symbolic links (similar to cp -L) or specify a different data folder that will be used globally for all the tests (extlibs/test_data in this example). I couldn’t find anything similar in the documentation. Does anyone know if there is a way of doing it? Maybe common test is not able to do that right now, should it have and option of dereferencing the symbolic links ?

Also, what is the best way of handling the situation when two test suites need to access the same test data but it is not wanted to copy the files twice to the different ..._SUITE_data/ folders? I’m thinking in the case where the files are a few MB and I don’t want to commit them twice.

2 Likes
2

Could you maybe have your submodule checkout targeting the priv folder of your application?
That way your path would just be code:priv_dir(App) ++ "exlibs/test_data/data.txt"

2 Likes
3

Thank you, I think this might work if I create a new app and I checkout the git submodule as you said. After, I just need to avoid to copy the app to the production environment. Do you know if using priv folder for this use case can cause any issue (slower compilation etc,)?

1 Like
4

Since it’s not compiling anything in your submodule it should not affect compile time.

I’m not sure if it’s possible, but maybe you can handle the checkout or removal of the submodule in your Makefile or rebar config.

2 Likes
5

The files are copied when I run rebar3 compile but it is so far the best solution. Thank you :slight_smile:

1 Like
6

It might be a good idea to move your ‘test_data’ app into ‘test_apps’ directory (same level as apps) and then
add {project_app_dirs, ["apps/*","test_apps/*"]} in test profile
or add it as a dependency in test profile with path plugin GitHub - benoitc/rebar3_path_deps: A rebar plugin to specify path dependencies.

This way it should not by copied with rebar3 compile but rebar3 ct and it will only exist in ‘_build/test/lib’.

2 Likes
7

I realised I never replied how I’ve got around this problem when I saw a similar post. I added some symbolic links inside the _SUITE_data/ and made some changes to rebar3 so the content is dereferenced and copied into the _build/test/..._SUITE_data/ folder