Large-scale Generation

Generate millions of synthetic rows with sampling, streaming, and batch tuning.

Last updated October 20, 2018

Overview

When generating large datasets (millions of rows per table), three factors affect performance: distribution sampling, batch size, and memory management.

Sampling strategies

By default, Weavori automatically preserves realistic data distributions when generating synthetic data. For very large tables, you have four options:

Fast mode

Use --fast to prioritize speed over distribution accuracy. Best for large tables or quick test datasets.

$weavori generate "postgres://user:pass@localhost:5432/source" \ --target "postgres://user:pass@localhost:5432/target" \ --fast --rows 1000000

Fixed sample

Use --sample-rows to draw a fixed-size sample of rows for distribution analysis when you want explicit control over the sample size.

$weavori generate "postgres://user:pass@localhost:5432/source" \ --target "postgres://user:pass@localhost:5432/target" \ --sample-rows 10000 --rows 1000000

No sampling

Use --no-sampling to skip distribution sampling entirely. Fastest possible generation, but output values will be purely random.

$weavori generate "postgres://user:pass@localhost:5432/source" \ --target "postgres://user:pass@localhost:5432/target" \ --no-sampling --rows 5000000

Schema caching

Schema introspection results are cached per source database. On repeat runs against the same database, Weavori skips the introspection phase and uses the cached schema.

Default: Cache enabled. Schema is re-introspected automatically when it changes.
Bypass: Use --no-cache on weavori generate to force fresh introspection.
Purge all: Run weavori cache clear to delete all cached schemas.

For details on how the cache works, automatic invalidation, and environment-specific best practices, see Schema & Caching Management.

Streaming and batching

Weavori generates and exports data in a streaming fashion — rows are produced, batched, and written incrementally. Memory usage stays constant regardless of total row count.

The sync command batches rows with --batch (default 5,000 rows). Larger batches reduce round trips but increase memory per batch. Start with the default and adjust based on throughput:

$weavori sync "postgres://user:pass@localhost:5432/source" \ --target "postgres://user:pass@localhost:5432/target" \ --batch 10000

Output modes for monitoring

During long runs, choose the output format that best suits your monitoring needs:

--output pretty — Human-readable with colors (default for terminals)
--output plain — Timestamped log lines, ideal for CI/CD log capture
--output json — Machine-readable JSONL for programmatic processing

$weavori generate ... --rows 10000000 --output plain

Sync compatibility mode

If the target database requires compatibility mode, use --mode insert:

$weavori sync "postgres://user:pass@localhost:5432/source" \ --target "postgres://user:pass@localhost:5432/target" \ --mode insert --batch 500

The --mode insert flag applies to weavori sync.

Best practices

Start with estimate — verify connectivity and schema before running a large generation
Use --fast for wide schemas — prioritizes speed over distribution accuracy
Use --sample-rows for huge tables — draw a fixed sample when the source has over a billion rows
Use --no-cache for CI — ensure fresh introspection on each pipeline run
Monitor per-table timing — identify slow tables and adjust their sampling strategy
No-sampling for speed — if distribution accuracy doesn't matter, --no-sampling is fastest