Skip to content

Visual Regression Testing

Dataface uses golden snapshot testing to catch unintended visual changes to chart and dashboard rendering. Every fenced dataface-example block in the documentation doubles as a visual test case.

How It Works

The documentation pages under docs/docs/ use fenced dataface-example blocks to show live-rendered dashboards inline. The visual test suite discovers those blocks, renders each one through the full Dataface pipeline (compile → execute → render), and compares the SVG output against a stored golden file.

docs/docs/**/*.md          → source of truth for which examples exist
dataface/tests/visual/conftest.py   → discovers examples, renders to SVG
dataface/tests/visual/snapshots/    → golden SVG files

The flow:

  1. Discovery — conftest.py scans all markdown files for fenced dataface-example blocks
  2. Render — Each example is compiled and rendered to SVG using the CSV adapter (queries reference examples/_doc_examples.yaml)
  3. Normalize — Non-deterministic content is stripped (timestamps, auto-generated IDs, excessive float precision)
  4. Compare — The normalized SVG is compared byte-for-byte against the golden file

Two kinds of examples

Pattern Example
Inline YAML ```dataface-example ... ```
File reference ```dataface-example {file=examples/charts/file.yml} ```

Inline examples are the most common. File references point to YAML files under examples/.

Where Things Live

Path What
docs/docs/**/*.md Documentation pages with fenced dataface-example blocks
examples/_doc_examples.yaml Shared query definitions used by doc examples
dataface/tests/visual/conftest.py Example discovery, SVG rendering, normalization
dataface/tests/visual/test_visual_snapshots.py The test that compares against goldens
dataface/tests/visual/snapshots/ Golden SVG files, organized by doc section

Golden files are named {page}_{index}_{content_hash}.svg. The content hash means renaming or reordering examples within a page produces new filenames (old ones should be deleted).

Running the Tests

# Check for regressions (what CI runs)
just dataface visual-check

# Check a specific example
just dataface visual-check -k "types_2"

# See a text diff of changed snapshots
just dataface visual-diff

Local review is authoritative

Visual snapshots are reviewed and updated locally, not enforced in GitHub CI. Approved snapshot updates should be generated on macOS. If you need to run the snapshot tests elsewhere, set DATAFACE_VISUAL_FORCE=1 explicitly and treat the result as diagnostic rather than canonical.

Approving Changes

When you change rendering code, chart styling, or documentation examples, the golden files may need updating. Visual changes still need review before approval, but that review is no longer human-exclusive.

What happens when visuals change

When a visual test fails, the test writes the new render to a .actual.svg file next to the golden and fails with a prominent banner:

╔══════════════════════════════════════════════════════════════╗
║  VISUAL SNAPSHOT CHANGED — Review before approval.          ║
║  Inspect the .actual.svg diff, then update if intended.     ║
╚══════════════════════════════════════════════════════════════╝

Example: charts/types_2
Actual render saved to: dataface/tests/visual/snapshots/charts/types_2_0d1fe066.actual.svg
Golden file: dataface/tests/visual/snapshots/charts/types_2_0d1fe066.svg

Local approval workflow

Run this locally on macOS after verifying the changes look correct:

# Approve all changed examples
just dataface visual-approve

# Approve only specific examples
just dataface visual-approve -k "types_2"

Then commit the updated golden files alongside your code changes.

Working with an AI agent

If you're working with an AI agent and visual tests fail:

  1. Review the .actual.svg files (open them in a browser or SVG viewer)
  2. If the changes look correct, the agent can run just dataface visual-approve and commit the results
  3. If the changes are wrong, fix the rendering issue instead of approving the new goldens

The workflow

  1. Make your changes and run just dataface visual-check locally
  2. Review the diffs — compare .actual.svg against the golden, or use just dataface visual-diff
  3. Approve locally with just dataface visual-approve
  4. Commit the updated golden files alongside your code changes

Review snapshot changes before approving them

GitHub CI does not act as the approval gate. Approval should follow an intentional review of the .actual.svg output, whether that review is done by a person directly or delegated through an agent workflow.

Adding New Examples

Every fenced dataface-example block in the docs automatically becomes a visual test. To add a new golden test case:

  1. Add a fenced dataface-example block to a documentation page under docs/docs/
  2. Ensure the query data exists — inline examples typically reference queries from examples/_doc_examples.yaml
  3. Run the tests — on first run, the golden file is created automatically:
just dataface visual-check

The test will skip with a message like "Golden file created: types_8_abc12345.svg. Human review required before commit."

  1. Review the generated SVG in dataface/tests/visual/snapshots/ and commit it

What makes a good golden example

The golden set is not a random collection — it's the documentation. Each example exists because it teaches users something:

  • Chart types (charts/types.md) — one example per chart type showing the basic usage
  • More types (charts/more-types.md) — specialized charts (heatmap, histogram) and the all-types gallery
  • Layouts (boards/layouts.md) — rows, columns, grid, tabs
  • Board features (boards/content.md, boards/sizing.md) — content blocks, responsive sizing
  • Quick guide (quick-guide.md) — progressive examples building up complexity

If you're adding a new chart type or layout feature, add a documentation example. The golden test comes for free.

SVG Normalization

Raw SVG output contains non-deterministic elements that would cause false failures. The normalizer (conftest.py:normalize_svg) strips:

  • data-rendered-at timestamps
  • id="dataface-svg-..." attributes
  • Timestamp <text> elements
  • Embedded <script> tags
  • Auto-generated Vega-Lite IDs (gradient_N, clip_N)
  • Excessive float precision (rounded to 2 decimal places)
  • Blank lines

This means golden files are stable across renders as long as the actual visual output doesn't change.

Troubleshooting

Test skipped off macOS — Expected unless you set DATAFACE_VISUAL_FORCE=1. Approved snapshot updates should come from macOS-local review.

Golden file hash mismatch after editing YAML — If you change the YAML content of a fenced dataface-example block, the content hash in the filename changes. The old golden file becomes orphaned. Delete it and let the test create a new one.

Lots of golden files changed after a theme/rendering update — This is normal for broad rendering changes. Review the diffs, then just dataface visual-approve to update them all at once.

Visual Change History

When a PR changes visual snapshots, the committed golden files are the source of truth. The task serve UI reads snapshot-changing commits directly from git history.

PR workflow

  1. After changing rendering code, review the updated golden snapshots locally
  2. Commit the approved snapshot files on the PR branch
  3. The PR template includes checkboxes for visual review attestation
  4. Reviewers can inspect the snapshot diff directly or use /viz-changes

Viewing in task serve

The task serve UI has a Viz Changes tab in the left nav. It shows a chronological timeline of visual change records found in git history based on committed snapshot updates.

just tasks serve    # then visit http://localhost:8005/viz-changes