Rebar v4: Which Rebar plugins should we prioritize for compatibility testing?

Questions / Help
1

We’re working on Rebar v4, and one of the things we need to do is make sure the changes we’re making continue to work well with the plugins people actually use in real projects.

@bchassoul is leading this part of the work, and we’ve put together a first ranking of Rebar plugins that we think would be most useful to include in compatibility testing.

How we ranked the plugins

Rather than trying to test every plugin, we ranked them using five criteria:

  • 25% – How widely the plugin is used
  • 25% – How much of Rebar it exercises
  • 20% – Whether it is maintained and current
  • 20% – How useful the workflow is to cover
  • 10% – Whether it can run reliably in CI

We started with the recommended plugins listed on the Rebar3 website, but didn’t use that list on its own. A plugin can be listed there and still be a poor fit for regular testing. We also considered community usage, which parts of Rebar the plugin relies on and whether we can test it reliably.

Our current top 10

  1. rebar3_hex – 93: Package build and publishing workflows
  2. pc – 91: Native-code builds, hooks and clean behaviour
  3. rebar3_ex_doc – 87: Documentation, project metadata and dependencies
  4. rebar3_lint – 84: A familiar CI and provider workflow
  5. covertool – 82: Test output and coverage reports
  6. rebar3_gpb_plugin – 80: Code generation and pre-compile hooks
  7. rebar3_format – 74: Formatting in CI
  8. rebar3_proper – 73: Property-testing workflows
  9. rebar3_appup_plugin – 70: Release upgrades and appup generation
  10. rebar3_path_deps – 68: Local and path-based dependencies

We have another 10 plugins in the ranking, with priorities ranging from nightly or targeted testing to lower-priority legacy smoke tests.

A couple of things we decided along the way: aliases belong in Rebar’s own tests because alias support is built in, and rebar3_vendor has been superseded by rebar3_path_deps, so we don’t think testing both would add much.

We’d like your input

Does this top 10 reflect the plugins and workflows you actually depend on?

Are we missing a plugin that should have a much higher priority?

When thinking about it, we’re especially interested in the workflows you rely on: publishing packages, compiling native code, generating code, testing, making releases… or something else entirely.

We’re not trying to defend the scores. The goal is to build a test suite that catches the kinds of regressions that would actually get in your way.

@bchassoul has written up the full ranking, including all 20 plugins and how we arrived at it, here:

Which Rebar Plugins Should We Test First? :eyes: → Which Rebar Plugins Should We Test First? 👀 | GRiSP

Thanks in advance for any feedback, especially from people maintaining or regularly using these plugins.

And thank you to everyone supporting the Rebar v4 Kickstarter work. Backers will also receive a separate poll so they can weigh in directly.