Weavori
⌘K

How Weavori Works

A conceptual overview of the synthetic data generation pipeline.

Last updated October 20, 2018

Overview

Weavori takes a PostgreSQL database connection string and produces a referentially intact synthetic replica. The process follows a pipeline with three distinct phases: introspection, dependency resolution, and generation.

Weavori also supports DDL paste mode — generating data from CREATE TABLE statements without connecting to a live database.

For a deeper dive into what makes synthetic data realistic — referential integrity, distribution sampling, and cross-column coherence — read How to generate realistic test data for PostgreSQL.

Phase 1: Schema Introspection

Weavori connects to your database and reads its schema — extracting every table, column, data type, default value, primary key, and foreign key constraint. This produces an in-memory model of your database structure. No data is read during this phase.

$ weavori generate "postgres://user:pass@localhost:5432/mydb"
Connecting to source database...
Introspecting schema...
Connected successfully.
Found 2 schema(s).
"public": 5 table(s)
- users (6 cols, 0 FK(s))
- orders (4 cols, 1 FK(s))

Introspected schemas are cached by default for faster repeat runs, and are re-introspected automatically when your schema changes. Use --no-cache on weavori generate to force fresh introspection, or weavori cache clear to purge all cached schemas.

Phase 2: FK Dependency Resolution

Once the schema is known, Weavori builds a dependency graph from the foreign key relationships. Tables with no FK dependencies (parents) are generated first. Tables that reference others (children) wait until their dependencies are ready, ensuring every generated foreign key has a matching primary key.

If circular dependencies are detected, Weavori warns you and continues — common patterns are handled automatically.

Generation order (parents before children):
  1. public.users
  2. public.products
  3. public.orders
  4. public.order_items

Weavori also detects ON INSERT triggers during introspection. Columns populated by triggers are automatically skipped during generation, and the generation order respects trigger dependencies.

Phase 3: Data Generation

For each table, Weavori samples the source column distributions to understand value patterns, then generates synthetic data that mirrors those patterns. Four sampling modes are available:

ModeFlagWhen to use
Auto(default)Automatically preserves realistic data distributions when possible.
Fast--fastPrioritizes speed over distribution accuracy.
Fixed sample--sample-rows NUses a user-specified sample size to balance accuracy and performance.
Random--no-samplingGenerates purely random values without reading source data.

Export Strategies

Generated rows are written to the target database using optimized bulk write operations. The appropriate strategy is selected automatically based on the target environment.

For very large datasets, see Large-scale Generation — generation streams rows in constant memory, and the --fast, --sample-rows, and --no-sampling flags trade distribution accuracy for throughput.

DDL Paste Mode

Weavori can generate synthetic data without a live source database using DDL paste mode:

$weavori generate --input schema.sql --input-format ddl --rows 100

DDL paste mode accepts common PostgreSQL CREATE TABLE statements and automatically interprets schema relationships when generating data.

Formulas

Computed columns can be defined with repeatable --formula flags:

$weavori generate ... --formula 'full_name=concat(first_name, " ", last_name)'

Formulas can reference any other generated column in the same row.

Custom Datasets

CSV files can be used as lookup sources for column values:

$weavori generate ... \ --dataset customers=customers.csv \ --map orders.customer_id=customers.id

This enables enum-like value generation from reference data without connecting to a source database.

Next steps