Large-scale Generation

Generate millions of synthetic rows with sampling, streaming, and background jobs.

Overview

When generating large datasets (millions of rows per table), three factors affect performance: distribution sampling, batch size, and memory management.

Sampling strategies

By default, Weavori automatically preserves realistic data distributions when generating synthetic data. For very large tables, you have four options:

Fast mode

Use --fast to prioritize speed over distribution accuracy. Best for large tables or quick test datasets.

weavori generate "postgres://user:pass@localhost:5432/source" \
    --target "postgres://user:pass@localhost:5432/target" \
    --fast --rows 1000000

Fixed sample

Use --sample-rows to draw a fixed-size sample of rows for distribution analysis when you want explicit control over the sample size.

weavori generate "postgres://user:pass@localhost:5432/source" \
    --target "postgres://user:pass@localhost:5432/target" \
    --sample-rows 10000 --rows 1000000

No sampling

Use --no-sampling to skip distribution sampling entirely. Fastest possible generation, but output values will be purely random.

weavori generate "postgres://user:pass@localhost:5432/source" \
    --target "postgres://user:pass@localhost:5432/target" \
    --no-sampling --rows 5000000

Schema caching

Schema introspection results are cached per source database. On repeat runs against the same database, Weavori skips the introspection phase and uses the cached schema.

Default: Cache enabled. Schema is re-introspected automatically when the fingerprint changes.
Bypass: Use --no-cache on weavori generate to force fresh introspection.
Purge all: Run weavori cache clear to delete all cached schemas.

Streaming and batching

Weavori generates and exports data in a streaming fashion — rows are produced, batched, and written incrementally. Memory usage stays constant regardless of total row count.

The default batch size is 1,000 rows. Adjust it with --batch on the sync command:

weavori sync "postgres://user:pass@localhost:5432/source" \
    --output "postgres://user:pass@localhost:5432/target" \
    --batch 5000

Larger batches reduce round trips but increase memory per batch. Start with the default and adjust based on throughput.

Output modes for monitoring

During long runs, choose the output format that best suits your monitoring needs:

--output pretty — Human-readable with colors (default for terminals)
--output plain — Timestamped log lines, ideal for CI/CD log capture
--output json — Machine-readable JSONL for programmatic processing

weavori generate ... --rows 10000000 --output plain

Sync compatibility mode

If the target database requires compatibility mode, use --mode insert:

weavori sync "postgres://user:pass@localhost:5432/source" \
    --output "postgres://user:pass@localhost:5432/target" \
    --mode insert --batch 500

The --mode insert flag applies to weavori sync.

Best practices

Start with estimate — verify connectivity and schema before running a large generation
Use --fast for wide schemas — prioritizes speed over distribution accuracy
Use --async for very large jobs — offload to server and track with weavori job watch
Use --no-cache for CI — ensure fresh introspection on each pipeline run
Monitor per-table timing — identify slow tables and adjust their sampling strategy
No-sampling for speed — if distribution accuracy doesn't matter, --no-sampling is fastest