Implements feature requested in
https://github.com/microsoft/playwright/issues/30457
The test runner treats flaky tests as failures when the flag is enabled,
but still reports flaky tests as flaky in the reporting interface. It
feels like something worth discussing as this behaviour makes sense to
me, but looked a bit odd to @BJSS-russell-pollock when I ran this past
him.
Closes#30457.
A summary about the attempts we tried in
https://github.com/microsoft/playwright/pull/26931 for
https://github.com/microsoft/playwright/issues/26859 and
https://github.com/microsoft/playwright/issues/24591.
1. attempt: collect all the stdio from the worker host.
- this made it hard to capture the live console logs for the live trace
2. attempt: collect all the stdio from the worker host, but keep
proxying `process.{stdout,stderr}.write`.
- this solved the live console logs problem, since we could still listen
on them
- we found out about a race that the stdout/stderr streams can still
buffer data while the test already had finished
- we created a best effort flush implementation but didn't like the
complexity it introduced to the project
3. attempt: keep things like they are but also listen to stdio from the
worker host
- this solves the race of console.log in the test (things stay like they
are)
- this keeps live trace working (things stay like they are)
- a known pitfall is that it could result in logs which don't end up in
the reporter onStdOut/onStdErr, but it is already a significant
improvement over how it was before.
- before: they were not getting displayed
- after: they are getting displayed and there is a low likelihood that
they don't end up in the reporter API if the write happens slightly
before a test finished.
Closes#26931Fixes#26859Fixes#24591
This makes `test.fail` tests considered as passing when they actually fail:
- Stop restarting the worker.
- Retry when it passes instead of a fail.
- Behaves similar to regular tests in a `describe.serial` suite.
- Source now lives at `src/test`.
- Former folio tests live at `tests/playwright-test`.
- We use `src/test/internal.ts` that exposes base test without
Playwright fixtures for most tests (to avoid modifications for now).
- Test types live in `types/testFoo.d.ts`.
- Stable test runner is installed to `tests/config/test-runner` during `npm install`.
- All deps including test-only are now listed in `package.json`.
Non-test deps must also be listed in `build_package.js` to get included.
Joe-Hendley
