kyfx
/
playwright
mirror of https://github.com/microsoft/playwright.git
4
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

chore: allow storing aria snapshots in files (#33919)

This commit is contained in:
Pavel Feldman 2024-12-10 11:45:16 -08:00 committed by GitHub
parent 200e868b63
commit a25bda6950
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
12 changed files with 370 additions and 26 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/src/api/class-locator.md
View File

@ -155,7 +155,7 @@ Additional locator to match.
- returns: <[string]>
Captures the aria snapshot of the given element.
Read more about [aria snapshots](../aria-snapshots.md) and [`method: LocatorAssertions.toMatchAriaSnapshot`] for the corresponding assertion.
Read more about [aria snapshots](../aria-snapshots.md) and [`method: LocatorAssertions.toMatchAriaSnapshot#2`] for the corresponding assertion.
**Usage**

61
docs/src/api/class-locatorassertions.md
View File

@ -446,7 +446,7 @@ Expected options currently selected.
* since: v1.49
* langs: python
The opposite of [`method: LocatorAssertions.toMatchAriaSnapshot`].
The opposite of [`method: LocatorAssertions.toMatchAriaSnapshot#2`].
### param: LocatorAssertions.NotToMatchAriaSnapshot.expected
* since: v1.49
@ -2121,7 +2121,58 @@ Expected options currently selected.
* since: v1.23
## async method: LocatorAssertions.toMatchAriaSnapshot
## async method: LocatorAssertions.toMatchAriaSnapshot#1
* since: v1.50
* langs:
- alias-java: matchesAriaSnapshot
Asserts that the target element matches the given [accessibility snapshot](../aria-snapshots.md).
**Usage**
```js
await expect(page.locator('body')).toMatchAriaSnapshot();
await expect(page.locator('body')).toMatchAriaSnapshot({ name: 'snapshot' });
await expect(page.locator('body')).toMatchAriaSnapshot({ path: '/path/to/snapshot.yml' });
```
```python async
await expect(page.locator('body')).to_match_aria_snapshot(path='/path/to/snapshot.yml')
```
```python sync
expect(page.locator('body')).to_match_aria_snapshot(path='/path/to/snapshot.yml')
```
```csharp
await Expect(page.Locator("body")).ToMatchAriaSnapshotAsync(new { Path = "/path/to/snapshot.yml" });
```
```java
assertThat(page.locator("body")).matchesAriaSnapshot(new LocatorAssertions.MatchesAriaSnapshotOptions().setPath("/path/to/snapshot.yml"));
```
### option: LocatorAssertions.toMatchAriaSnapshot#1.name
* since: v1.50
* langs: js
- `name` <[string]>
Name of the snapshot to store in the snapshot folder corresponding to this test. Generates ordinal name if not specified.
### option: LocatorAssertions.toMatchAriaSnapshot#1.path
* since: v1.50
- `path` <[string]>
Path to the YAML snapshot file.
### option: LocatorAssertions.toMatchAriaSnapshot#1.timeout = %%-js-assertions-timeout-%%
* since: v1.50
### option: LocatorAssertions.toMatchAriaSnapshot#1.timeout = %%-csharp-java-python-assertions-timeout-%%
* since: v1.50
## async method: LocatorAssertions.toMatchAriaSnapshot#2
* since: v1.49
* langs:
- alias-java: matchesAriaSnapshot
@ -2170,12 +2221,12 @@ assertThat(page.locator("body")).matchesAriaSnapshot("""
""");
```
### param: LocatorAssertions.toMatchAriaSnapshot.expected
### param: LocatorAssertions.toMatchAriaSnapshot#2.expected
* since: v1.49
- `expected` <string>
### option: LocatorAssertions.toMatchAriaSnapshot.timeout = %%-js-assertions-timeout-%%
### option: LocatorAssertions.toMatchAriaSnapshot#2.timeout = %%-js-assertions-timeout-%%
* since: v1.49
### option: LocatorAssertions.toMatchAriaSnapshot.timeout = %%-csharp-java-python-assertions-timeout-%%
### option: LocatorAssertions.toMatchAriaSnapshot#2.timeout = %%-csharp-java-python-assertions-timeout-%%
* since: v1.49

2
docs/src/aria-snapshots.md
View File

@ -154,7 +154,7 @@ structure of a page, use the [Chrome DevTools Accessibility Pane](https://develo
## Snapshot matching
The [`method: LocatorAssertions.toMatchAriaSnapshot`] assertion method in Playwright compares the accessible
The [`method: LocatorAssertions.toMatchAriaSnapshot#2`] assertion method in Playwright compares the accessible
structure of the locator scope with a predefined aria snapshot template, helping validate the page's state against
testing requirements.

2
docs/src/release-notes-csharp.md
View File

@ -9,7 +9,7 @@ toc_max_heading_level: 2
### Aria snapshots
New assertion [`method: LocatorAssertions.toMatchAriaSnapshot`] verifies page structure by comparing to an expected accessibility tree, represented as YAML.
New assertion [`method: LocatorAssertions.toMatchAriaSnapshot#2`] verifies page structure by comparing to an expected accessibility tree, represented as YAML.
```csharp
await page.GotoAsync("https://playwright.dev");

2
docs/src/release-notes-java.md
View File

@ -8,7 +8,7 @@ toc_max_heading_level: 2
### Aria snapshots
New assertion [`method: LocatorAssertions.toMatchAriaSnapshot`] verifies page structure by comparing to an expected accessibility tree, represented as YAML.
New assertion [`method: LocatorAssertions.toMatchAriaSnapshot#2`] verifies page structure by comparing to an expected accessibility tree, represented as YAML.
```java
page.navigate("https://playwright.dev");

2
docs/src/release-notes-js.md
View File

@ -15,7 +15,7 @@ import LiteYouTube from '@site/src/components/LiteYouTube';
### Aria snapshots
New assertion [`method: LocatorAssertions.toMatchAriaSnapshot`] verifies page structure by comparing to an expected accessibility tree, represented as YAML.
New assertion [`method: LocatorAssertions.toMatchAriaSnapshot#2`] verifies page structure by comparing to an expected accessibility tree, represented as YAML.
```js
await page.goto('https://playwright.dev');

2
docs/src/release-notes-python.md
View File

@ -8,7 +8,7 @@ toc_max_heading_level: 2
### Aria snapshots
New assertion [`method: LocatorAssertions.toMatchAriaSnapshot`] verifies page structure by comparing to an expected accessibility tree, represented as YAML.
New assertion [`method: LocatorAssertions.toMatchAriaSnapshot#2`] verifies page structure by comparing to an expected accessibility tree, represented as YAML.
```python
page.goto("https://playwright.dev")

2
packages/playwright-core/types/types.d.ts vendored
View File

@ -12428,7 +12428,7 @@ export interface Locator {
/**
* Captures the aria snapshot of the given element. Read more about [aria snapshots](https://playwright.dev/docs/aria-snapshots) and
* [expect(locator).toMatchAriaSnapshot(expected[, options])](https://playwright.dev/docs/api/class-locatorassertions#locator-assertions-to-match-aria-snapshot)
* [expect(locator).toMatchAriaSnapshot(expected[, options])](https://playwright.dev/docs/api/class-locatorassertions#locator-assertions-to-match-aria-snapshot-2)
* for the corresponding assertion.
*
* **Usage**

69
packages/playwright/src/matchers/toMatchAriaSnapshot.ts
View File

@ -18,19 +18,25 @@
import type { LocatorEx } from './matchers';
import type { ExpectMatcherState } from '../../types/test';
import { kNoElementsFoundError, matcherHint, type MatcherResult } from './matcherHint';
import { colors } from 'playwright-core/lib/utilsBundle';
import { EXPECTED_COLOR } from '../common/expectBundle';
import { callLogText } from '../util';
import { callLogText, sanitizeFilePathBeforeExtension, trimLongString } from '../util';
import { printReceivedStringContainExpectedSubstring } from './expect';
import { currentTestInfo } from '../common/globals';
import type { MatcherReceived } from '@injected/ariaSnapshot';
import { escapeTemplateString } from 'playwright-core/lib/utils';
import { escapeTemplateString, isString, sanitizeForFilePath } from 'playwright-core/lib/utils';
import fs from 'fs';
import path from 'path';
type ToMatchAriaSnapshotExpected = {
name?: string;
path?: string;
} | string;
export async function toMatchAriaSnapshot(
this: ExpectMatcherState,
receiver: LocatorEx,
expected: string,
options: { timeout?: number, matchSubstring?: boolean } = {},
expectedParam: ToMatchAriaSnapshotExpected,
options: { timeout?: number } = {},
): Promise<MatcherResult<string | RegExp, string>> {
const matcherName = 'toMatchAriaSnapshot';
@ -39,7 +45,7 @@ export async function toMatchAriaSnapshot(
throw new Error(`toMatchAriaSnapshot() must be called during the test`);
if (testInfo._projectInternal.ignoreSnapshots)
return { pass: !this.isNot, message: () => '', name: 'toMatchAriaSnapshot', expected };
return { pass: !this.isNot, message: () => '', name: 'toMatchAriaSnapshot', expected: '' };
const updateSnapshots = testInfo.config.updateSnapshots;
@ -48,12 +54,25 @@ export async function toMatchAriaSnapshot(
promise: this.promise,
};
if (typeof expected !== 'string') {
throw new Error([
matcherHint(this, receiver, matcherName, receiver, expected, matcherOptions),
`${colors.bold('Matcher error')}: ${EXPECTED_COLOR('expected',)} value must be a string`,
this.utils.printWithType('Expected', expected, this.utils.printExpected)
].join('\n\n'));
let expected: string;
let expectedPath: string | undefined;
if (isString(expectedParam)) {
expected = expectedParam;
} else {
if (expectedParam?.path) {
expectedPath = expectedParam.path;
} else if (expectedParam?.name) {
expectedPath = testInfo.snapshotPath(sanitizeFilePathBeforeExtension(expectedParam.name));
} else {
let snapshotNames = (testInfo as any)[snapshotNamesSymbol] as SnapshotNames;
if (!snapshotNames) {
snapshotNames = { anonymousSnapshotIndex: 0 };
(testInfo as any)[snapshotNamesSymbol] = snapshotNames;
}
const fullTitleWithoutSpec = [...testInfo.titlePath.slice(1), ++snapshotNames.anonymousSnapshotIndex].join(' ');
expectedPath = testInfo.snapshotPath(sanitizeForFilePath(trimLongString(fullTitleWithoutSpec)) + '.yml');
}
expected = await fs.promises.readFile(expectedPath, 'utf8').catch(() => '');
}
const generateMissingBaseline = updateSnapshots === 'missing' && !expected;
@ -102,8 +121,24 @@ export async function toMatchAriaSnapshot(
if ((updateSnapshots === 'all') ||
(updateSnapshots === 'changed' && pass === this.isNot) ||
generateMissingBaseline) {
const suggestedRebaseline = `toMatchAriaSnapshot(\`\n${escapeTemplateString(indent(typedReceived.regex, '{indent} '))}\n{indent}\`)`;
return { pass: this.isNot, message: () => '', name: 'toMatchAriaSnapshot', suggestedRebaseline };
if (expectedPath) {
await fs.promises.mkdir(path.dirname(expectedPath), { recursive: true });
await fs.promises.writeFile(expectedPath, typedReceived.regex, 'utf8');
const relativePath = path.relative(process.cwd(), expectedPath);
if (updateSnapshots === 'missing') {
const message = `A snapshot doesn't exist at ${relativePath}, writing actual.`;
testInfo._hasNonRetriableError = true;
testInfo._failWithError(new Error(message));
} else {
const message = `A snapshot is generated at ${relativePath}.`;
/* eslint-disable no-console */
console.log(message);
}
return { pass: true, message: () => '', name: 'toMatchAriaSnapshot' };
} else {
const suggestedRebaseline = `toMatchAriaSnapshot(\`\n${escapeTemplateString(indent(typedReceived.regex, '{indent} '))}\n{indent}\`)`;
return { pass: false, message: () => '', name: 'toMatchAriaSnapshot', suggestedRebaseline };
}
}
}
@ -134,3 +169,9 @@ function unshift(snapshot: string): string {
function indent(snapshot: string, indent: string): string {
return snapshot.split('\n').map(line => indent + line).join('\n');
}
const snapshotNamesSymbol = Symbol('snapshotNames');
type SnapshotNames = {
anonymousSnapshotIndex: number;
};

31
packages/playwright/types/test.d.ts vendored
View File

@ -8430,6 +8430,37 @@ interface LocatorAssertions {
timeout?: number;
}): Promise<void>;
/**
* Asserts that the target element matches the given [accessibility snapshot](https://playwright.dev/docs/aria-snapshots).
*
* **Usage**
*
* ```js
* await expect(page.locator('body')).toMatchAriaSnapshot();
* await expect(page.locator('body')).toMatchAriaSnapshot({ name: 'snapshot' });
* await expect(page.locator('body')).toMatchAriaSnapshot({ path: '/path/to/snapshot.yml' });
* ```
*
* @param options
*/
toMatchAriaSnapshot(options?: {
/**
* Name of the snapshot to store in the snapshot folder corresponding to this test. Generates ordinal name if not
* specified.
*/
name?: string;
/**
* Path to the YAML snapshot file.
*/
path?: string;
/**
* Time to retry the assertion for in milliseconds. Defaults to `timeout` in `TestConfig.expect`.
*/
timeout?: number;
}): Promise<void>;
/**
* Asserts that the target element matches the given [accessibility snapshot](https://playwright.dev/docs/aria-snapshots).
*

172
tests/playwright-test/aria-snapshot-file.spec.ts Normal file
View File

@ -0,0 +1,172 @@
/**
* Copyright Microsoft Corporation. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { test, expect } from './playwright-test-fixtures';
import fs from 'fs';
import path from 'path';
test.describe.configure({ mode: 'parallel' });
test('should match snapshot with name', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
'playwright.config.ts': `
export default {
snapshotPathTemplate: '__snapshots__/{testFilePath}/{arg}{ext}',
};
`,
'__snapshots__/a.spec.ts/test.yml': `
- heading "hello world"
`,
'a.spec.ts': `
import { test, expect } from '@playwright/test';
test('test', async ({ page }) => {
await page.setContent(\`<h1>hello world</h1>\`);
await expect(page.locator('body')).toMatchAriaSnapshot({ name: 'test.yml' });
});
`
});
expect(result.exitCode).toBe(0);
});
test('should match snapshot with path', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
'test.yml': `
- heading "hello world"
`,
'a.spec.ts': `
import { test, expect } from '@playwright/test';
import path from 'path';
test('test', async ({ page }) => {
await page.setContent(\`<h1>hello world</h1>\`);
await expect(page.locator('body')).toMatchAriaSnapshot({ path: path.resolve(__dirname, 'test.yml') });
});
`
});
expect(result.exitCode).toBe(0);
});
test('should generate multiple missing', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
'playwright.config.ts': `
export default {
snapshotPathTemplate: '__snapshots__/{testFilePath}/{arg}{ext}',
};
`,
'a.spec.ts': `
import { test, expect } from '@playwright/test';
test('test', async ({ page }) => {
await page.setContent(\`<h1>hello world</h1>\`);
await expect(page.locator('body')).toMatchAriaSnapshot({ name: 'test-1.yml' });
await page.setContent(\`<h1>hello world 2</h1>\`);
await expect(page.locator('body')).toMatchAriaSnapshot({ name: 'test-2.yml' });
});
`
});
expect(result.exitCode).toBe(1);
expect(result.output).toContain(`A snapshot doesn't exist at __snapshots__${path.sep}a.spec.ts${path.sep}test-1.yml, writing actual`);
expect(result.output).toContain(`A snapshot doesn't exist at __snapshots__${path.sep}a.spec.ts${path.sep}test-2.yml, writing actual`);
const snapshot1 = await fs.promises.readFile(testInfo.outputPath('__snapshots__/a.spec.ts/test-1.yml'), 'utf8');
expect(snapshot1).toBe('- heading "hello world" [level=1]');
const snapshot2 = await fs.promises.readFile(testInfo.outputPath('__snapshots__/a.spec.ts/test-2.yml'), 'utf8');
expect(snapshot2).toBe('- heading "hello world 2" [level=1]');
});
test('should rebaseline all', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
'playwright.config.ts': `
export default {
snapshotPathTemplate: '__snapshots__/{testFilePath}/{arg}{ext}',
};
`,
'__snapshots__/a.spec.ts/test-1.yml': `
- heading "foo"
`,
'__snapshots__/a.spec.ts/test-2.yml': `
- heading "bar"
`,
'a.spec.ts': `
import { test, expect } from '@playwright/test';
test('test', async ({ page }) => {
await page.setContent(\`<h1>hello world</h1>\`);
await expect(page.locator('body')).toMatchAriaSnapshot({ name: 'test-1.yml' });
await page.setContent(\`<h1>hello world 2</h1>\`);
await expect(page.locator('body')).toMatchAriaSnapshot({ name: 'test-2.yml' });
});
`
}, { 'update-snapshots': 'all' });
expect(result.exitCode).toBe(0);
expect(result.output).toContain(`A snapshot is generated at __snapshots__${path.sep}a.spec.ts${path.sep}test-1.yml`);
expect(result.output).toContain(`A snapshot is generated at __snapshots__${path.sep}a.spec.ts${path.sep}test-2.yml`);
const snapshot1 = await fs.promises.readFile(testInfo.outputPath('__snapshots__/a.spec.ts/test-1.yml'), 'utf8');
expect(snapshot1).toBe('- heading "hello world" [level=1]');
const snapshot2 = await fs.promises.readFile(testInfo.outputPath('__snapshots__/a.spec.ts/test-2.yml'), 'utf8');
expect(snapshot2).toBe('- heading "hello world 2" [level=1]');
});
test('should not rebaseline matching', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
'playwright.config.ts': `
export default {
snapshotPathTemplate: '__snapshots__/{testFilePath}/{arg}{ext}',
};
`,
'__snapshots__/a.spec.ts/test.yml': `
- heading "hello world"
`,
'a.spec.ts': `
import { test, expect } from '@playwright/test';
test('test', async ({ page }) => {
await page.setContent(\`<h1>hello world</h1>\`);
await expect(page.locator('body')).toMatchAriaSnapshot({ name: 'test.yml' });
});
`
}, { 'update-snapshots': 'changed' });
expect(result.exitCode).toBe(0);
const snapshot1 = await fs.promises.readFile(testInfo.outputPath('__snapshots__/a.spec.ts/test.yml'), 'utf8');
expect(snapshot1.trim()).toBe('- heading "hello world"');
});
test('should generate snapshot name', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
'playwright.config.ts': `
export default {
snapshotPathTemplate: '__snapshots__/{testFilePath}/{arg}{ext}',
};
`,
'a.spec.ts': `
import { test, expect } from '@playwright/test';
test('test name', async ({ page }) => {
await page.setContent(\`<h1>hello world</h1>\`);
await expect(page.locator('body')).toMatchAriaSnapshot();
await page.setContent(\`<h1>hello world 2</h1>\`);
await expect(page.locator('body')).toMatchAriaSnapshot();
});
`
});
expect(result.exitCode).toBe(1);
expect(result.output).toContain(`A snapshot doesn't exist at __snapshots__${path.sep}a.spec.ts${path.sep}test-name-1.yml, writing actual`);
expect(result.output).toContain(`A snapshot doesn't exist at __snapshots__${path.sep}a.spec.ts${path.sep}test-name-2.yml, writing actual`);
const snapshot1 = await fs.promises.readFile(testInfo.outputPath('__snapshots__/a.spec.ts/test-name-1.yml'), 'utf8');
expect(snapshot1).toBe('- heading "hello world" [level=1]');
const snapshot2 = await fs.promises.readFile(testInfo.outputPath('__snapshots__/a.spec.ts/test-name-2.yml'), 'utf8');
expect(snapshot2).toBe('- heading "hello world 2" [level=1]');
});

49
tests/playwright-test/update-aria-snapshot.spec.ts
View File

@ -116,6 +116,55 @@ test('should update missing snapshots', async ({ runInlineTest }, testInfo) => {
expect(result2.exitCode).toBe(0);
});
test('should update multiple missing snapshots', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
'.git/marker': '',
'a.spec.ts': `
import { test, expect } from '@playwright/test';
test('test', async ({ page }) => {
await page.setContent(\`<h1>hello</h1>\`);
await expect(page.locator('body')).toMatchAriaSnapshot(\`\`);
await expect(page.locator('body')).toMatchAriaSnapshot(\`\`);
});
`
});
expect(result.exitCode).toBe(0);
expect(stripAnsi(result.output).replace(/\\/g, '/')).toContain(`New baselines created for:
a.spec.ts
git apply test-results/rebaselines.patch
`);
const patchPath = testInfo.outputPath('test-results/rebaselines.patch');
const data = fs.readFileSync(patchPath, 'utf-8');
expect(trimPatch(data)).toBe(`diff --git a/a.spec.ts b/a.spec.ts
--- a/a.spec.ts
+++ b/a.spec.ts
@@ -2,7 +2,11 @@
import { test, expect } from '@playwright/test';
test('test', async ({ page }) => {
await page.setContent(\`<h1>hello</h1>\`);
- await expect(page.locator('body')).toMatchAriaSnapshot(\`\`);
- await expect(page.locator('body')).toMatchAriaSnapshot(\`\`);
+ await expect(page.locator('body')).toMatchAriaSnapshot(\`
+ - heading "hello" [level=1]
+ \`);
+ await expect(page.locator('body')).toMatchAriaSnapshot(\`
+ - heading "hello" [level=1]
+ \`);
});
\\ No newline at end of file
`);
execSync(`patch -p1 < ${patchPath}`, { cwd: testInfo.outputPath() });
const result2 = await runInlineTest({});
expect(result2.exitCode).toBe(0);
});
test('should generate baseline with regex', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
'.git/marker': '',