Skip to content

Core Engine

This document describes the internal architecture of dft-core, the engine that powers Dataface.

If you're new to the codebase, start with the Architecture Overview and Platform Overview, then read this page for engine internals.

For the chart-specific compile-to-render contract, ownership rules, and Vega-Lite boundary, read Chart Rendering Architecture.

Core modules

The engine lives in dataface/core/ and is organized around a small set of subsystems:

Module Path Role
Compile dataface/core/compile/ Parse YAML, validate it, and normalize it into typed compiled objects
Execute dataface/core/execute/ Execute compiled queries against sources and adapters
Render dataface/core/render/ Turn compiled dashboards plus data into SVG, HTML, PNG, PDF, or terminal output
Inspect dataface/core/inspect/ Inspect schemas, model shapes, and profiling metadata
Serve dataface/core/serve/ HTTP server layer for serving dashboards

Pipeline

Dataface transforms YAML dashboard definitions into rendered output through a mostly linear pipeline. We are not redefining what "render" means in the product sense: anything that turns a compiled dashboard and its data into pixels or files still lives under Render (dataface/core/render/).

The only subtlety is order of operations in the code. Queries do not all run up front in one batch. Render walks the layout and calls into Execute whenever a chart (or nested face) needs a dataset—so Execute is both a real subsystem (dataface/core/execute/) and a step that repeats inside the render pass. The diagram below keeps Execute as its own box so you can see the module boundary and the usual mental model (definitions → data → picture → export), even though the runtime is "render drives execute," not "execute finishes, then render starts."

flowchart LR
    YAML -->|parse, validate,\nnormalize| COMPILE
    COMPILE -->|CompiledFace| EXECUTE
    EXECUTE -->|query results,\nper chart as needed| RENDER
    RENDER -->|native chart output| CONVERT
    CONVERT --> Output
Stage Module What it does
Compile dataface/core/compile/ Produce a guaranteed CompiledFace
Execute dataface/core/execute/ Run queries against sources; render pulls results as it needs them
Render dataface/core/render/ Layout, chart semantics from YAML + data, and the actual chart picture (Vega-Lite / SVG path, etc.)
Convert dataface/core/render/converters/ Turn native output into PNG, PDF, terminal, or other formats

So in code, render orchestrates execute; in docs, we still list the work as:

  1. compile definitions
  2. fetch query results (on demand from render)
  3. finish chart meaning where YAML left things automatic ("auto" type, formats, …)
  4. build chart graphics
  5. convert to the requested output format

Render Internals (one chart, in order)

Inside render/, a single chart is best read as numbered steps. The diagram uses short labels ①–⑨; the list under it is the designer-friendly reading order. Engineers: each step names the main implementation hook in italics where it helps.

flowchart LR
    A["① Place"] --> B["② Prep"]
    B --> C["③ Data"]
    C --> D["④ Intent"]
    D --> E["⑤ Auto"]
    E --> F["⑥ Lock in"]
    F --> G["⑦ Engine"]
    G --> H["⑧ Draw"]
    H --> I["⑨ Slot"]
  1. ① Place — The chart sits in the board layout; we are still walking the same tree of rows, columns, and chart slots.
  2. ② Prep — Tidy the authored chart before data work: title templates, matching YAML field names to the query’s column names.
  3. ③ Data — Run the chart’s query and bind the result table. This is the moment the “picture” can become honest about the numbers (calls into execute/).
  4. ④ Intent — Turn the YAML chart into a single internal “intent” object: keep what the author set, and mark what is still “to be decided” (e.g. type: auto).
  5. ⑤ Auto — Use the live data to fill only what was left open—chart type, axes, formats—without overriding explicit author choices (enrichment / chart decisions).
  6. ⑥ Lock in — Freeze the chart’s meaning: encodings, type, and presentation rules are now fully specified (resolved chart).
  7. ⑦ Engine — Choose the drawing backend for that resolved type (standard Vega family, geo, or a small SVG-special case family).
  8. ⑧ Draw — Emit the actual graphic instruction (mostly a Vega-Lite spec that becomes SVG, or a direct SVG path).
  9. ⑨ Slot — Hand the finished chart output back to the layout pass so the board can size and place it.

Implementation breadcrumbs: ② preprocessing in rendering.py; ④–⑥ compute_enrichments() (type detection + enrich_chart() decisions), then resolve_chart() assembles ResolvedChart in the chart pipeline; ⑦–⑧ renderer selection and spec assembly in renderers.py / standard_renderer.py.

Why it feels sequential, not a loop: the dataset arrives at ③; after that, enrichment only completes open fields, then mechanics (⑦–⑧) never invent new chart meaning. Older “loop” pictures implied re-deciding semantics after every micro-step; the real control flow is closer to a straight line with one data gate in the middle.

Important invariants

  • Trust the normalizer. Downstream code should treat compiled objects as already-resolved typed structures.
  • Keep compile as the foundation. Other subsystems may depend on compile types, but compile should not depend on render or execute internals.
  • Keep app concerns out of core. Django, editor UX, and product workflow belong in apps/, not here.
  • Prefer clear failures over silent fixes. Core should validate aggressively and surface bad input early.

Read-only enforcement

Read-only enforcement is in-process for DuckDB only. SqlAdapter opens file-based DuckDB connections with read_only=True (the driver refuses all writes) and forces enable_external_access=False. For every other warehouse — Postgres, MySQL/MariaDB, Snowflake, BigQuery, Databricks, Redshift, SQL Server — the dbt-adapter exposes no native read-only flag, so Dataface does not enforce one. The connection-level read-only posture for those warehouses lives entirely on the operator side: bind a SELECT-only credential to the dbt profile.

sql_guard (the SQL allowlist, see dataface/core/execute/sql_guard.py) is the unconditional in-process defense for all warehouses. SELECT-only credentials are the connection-level complement, not a replacement, for that guard.

See Sources for the per-warehouse posture table and credential setup guidance.

Data Shape Boundary

Dataface treats query execution and chart rendering as separate responsibilities:

  • The query layer owns dataset meaning, grain, and transformation.
  • The render layer owns visual encoding and presentation.
  • If a chart needs different grouping, aggregation, bucketing, or semantic ordering, change the query rather than reshaping data in the viz layer.

This boundary keeps chart behavior predictable:

  • Queries define what the data means.
  • Charts define how that already-shaped data is encoded and displayed.
  • Render code should avoid hidden data rewrites that make the visual output depend on chart-local transformation logic.

For internal architecture discussions, a useful chart vocabulary is:

  • data: the bound dataset or query result
  • encoding: per-chart authored mappings and intent such as x, y, color, and title
  • style preset: reusable chart-scaffold defaults such as axis side, legend placement, and whether structural elements exist (lives in dataface/core/defaults/style_presets/)
  • theme: reusable visual styling such as palette, typography, and surfaces (lives in dataface/core/defaults/chart_themes/)

The implementation-backed assignment of authored chart properties to chart, style preset, theme, or intentional none now lives in the Single-Chart Property Catalog.

The canonical contributor doc for chart architecture, default ownership, and the intended compile/enrichment/render split now lives in Chart Rendering Architecture.

Theme YAML Anchors

Built-in themes use native YAML anchors to deduplicate repeated values within a file:

style:
  font:
    color: &color_text_primary '#222222'   # anchor definition
  charts:
    rule:
      stroke: *color_text_primary           # anchor reference

Rules:

  • Anchors are file-local: inheriting themes see resolved values, not anchors.
  • Define an anchor at the first (semantically cleanest) occurrence in the file.
  • Use prefixed names: color_*, font_*, size_*, weight_*.
  • Anchor definitions must come before their references in file order.
  • Apply to any value appearing 2+ times in the file. Skip single-occurrence values.

The metaphor is:

  • theme is like CSS — the painting of the scaffold
  • style preset is like HTML — the scaffold itself

This is internal terminology for reasoning about responsibilities. Authored YAML uses style.extends to select a named preset and style patches for overrides.

We also use a context spectrum for chart reasoning:

  • data type -> field semantics -> observed values -> comparative context -> human context

This spectrum is about how much information is available to support chart selection and enrichment, not about moving data-shaping responsibility into the render layer. The canonical definition lives in dataface/core/render/chart/DESIGN.md.

Render Sub-Stages Inside The Boundary

For a reader-first walkthrough of the same path, use the numbered Render Internals (one chart, in order) steps ①–⑨ above. This table is the module-oriented view for people editing code:

Within render/, the chart path is intentionally split into smaller stages. That matters because the render stage is not one undifferentiated step:

Sub-stage Main code Responsibility
Authored chart preprocessing dataface/core/render/chart/rendering.py Resolve title templates and align authored field names to actual result columns before chart semantics are resolved
Semantic enrichment / chart decisions dataface/core/render/chart/pipeline.py, dataface/core/render/chart/decisions.py compute_enrichments(chart, data, …) returns a sparse dict of inferred fields (type, axes, format, zero) from observed data and column metadata, keeping authored values authoritative
Resolution dataface/core/render/chart/pipeline.py resolve_chart() merges authored fields with the inferred dict to produce a ResolvedChart with concrete chart semantics ready for renderer selection
Renderer selection dataface/core/render/chart/renderers.py Choose the renderer family (vega, geo, svg) based on resolved chart type
Mechanical spec assembly dataface/core/render/chart/standard_renderer.py and related files Build the final Vega-Lite spec or SVG artifact from resolved semantics plus config/structure/theme defaults
Output conversion dataface/core/render/converters/ Convert native chart output into SVG, PNG, PDF, JSON, or other transport formats

The important boundary is between:

  • semantic enrichment, which may fill unresolved chart meaning
  • mechanical rendering, which should translate resolved chart meaning into output without inventing new semantics

The Stage That Populates Auto Properties

The section of the renderer that populates chart properties from data is the chart enrichment/resolution pipeline, especially the chart decisions stage.

The top-level orchestration path is:

  • rendering._render_chart_item_inner() executes the query and calls resolve_chart()
  • pipeline.resolve_chart() computes effective_style (board + chart-local merge), extracts inference = effective_style.charts.inference, then runs compute_enrichments(chart, data, …, inference=) → sparse dict → constructs ResolvedChart with authored-wins per field
  • compute_enrichments() combines chart-type detection with the decisions helpers in decisions.py, gated by the CompiledInferenceStyle flags

When a chart uses type: auto, or leaves semantic fields unresolved, the functions that populate those properties are:

Function File What it fills
detect_chart_type_full() dataface/core/compile/chart_type_detection.py Chooses chart type for type: auto, and may also propose x, y, color, metric, or theta
enrich_chart() dataface/core/render/chart/decisions.py Runs data-aware chart enrichment over the chart-like object used by the pipeline
_profile_columns() dataface/core/render/chart/decisions.py Builds per-column profiles from observed values and DB metadata
_pick_fields() dataface/core/render/chart/decisions.py Chooses missing x/y fields for axis-based charts
_pick_axis_settings() dataface/core/render/chart/decisions.py Chooses missing axis-related settings for the resolved measure field
_pick_format() dataface/core/render/chart/decisions.py Infers a numeric/percent/currency format string
_pick_scale() dataface/core/render/chart/decisions.py Infers scale behavior such as zero: false when appropriate

In current code, that stage is best described as:

  • chart intent, enrichment, and resolution pipeline at the pipeline level
  • chart decisions at the data-aware heuristic sub-stage inside it

That is the area to read or modify when the question is "what code filled in this chart property from the data?" It is not primarily the Vega-Lite spec builder in standard_renderer.py; by the time that file runs, the semantic choices should already be resolved.

Dependency direction

flowchart BT
    compile["compile/"]
    execute["execute/"] --> compile
    render["render/"] --> compile
    render --> execute
    inspect["inspect/"] --> compile
    serve["serve/"] --> render

When to read which part

  • Read compile/ when changing schema, normalization, or validation behavior.
  • Read execute/ when changing adapter behavior, query execution, or caching.
  • Read render/ when changing chart semantics, layout behavior, or output formats.
  • Read inspect/ when changing schema inspection, profiling, or metadata generation.
  • Read serve/ when changing the HTTP serving layer around the engine.