[ADR-0018] Choose a testing framework

Switch to Playwright for our one testing framework for e2e tests.

Status: accepted
Deciders: Development team

Technical Story: Harmonize our tests tooling so ideally we use just one framework of writing tests whenever that’s possible.

Context and Problem Statement

Currently we rely heavily on Codeception for most of our tests. But we use two different syntaxes. The “old” one and Gherkin. During the WYSIWYG Epic we also wrote some tests in JS. Ideally we should use one way of writing all of these tests.

Considered Options

Decision Outcome

Chosen option: Switch to Playwright as our one testing framework for e2e tests.

Pros and Cons of the Options

Cypress

Good, because easy to start using it (one binary, simple interface)
Good, because assertion libraries are included
Good, because still in active development
Good, because large documentation, numerous examples and tutorials
Good, because it has a watch mode integrated that allows for easier development
Bad, because dropped by WordPress in favor of Puppeteer, then Playwright
Bad, because some complex cases require a succession of promises, which make them less readable

Playwright

Good, because easy to use: install/setup is quick and async/await notation + clear function names (using the expect library for assertions) make the tests easy to write and understand
Good, because still in active development
Good, because it’s the tool that WordPress will soon use too (and they’re working on a npm package with various utilities that we might be able to use in the future?) (+ best practices)
Good, because it has a useful VSCode package to easily run and debug tests
Good, because it’s faster by default, using concurrent workers
Bad, because since Playwright is fairly new, the support from the community is limited and the documentation is sometimes lacking