CI/CD Integration

Integrate Weavori into your CI/CD pipelines for automated synthetic data generation.

Last updated October 20, 2018

Weavori integrates into any CI/CD pipeline that supports running npx commands. This guide covers GitHub Actions setup with best practices for production use.

Quick Start (GitHub Actions)

Add a job to generate synthetic data and run your test suite against it:

name: Integration Tests
 
on:
  push:
    branches: [main]
  pull_request:
 
env:
  WEAVORI_VERSION: 0.1.6
  WEAVORI_API_KEY: ${{ secrets.WEAVORI_API_KEY }}
  SOURCE_DB: ${{ secrets.SOURCE_DB }}
  TARGET_DB: ${{ secrets.TARGET_DB }}
 
jobs:
  generate-and-test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
          POSTGRES_DB: test
        ports:
          - 5432:5432
        options: >-
          --health-cmd="pg_isready -U test"
          --health-interval=5s
          --health-timeout=3s
          --health-retries=5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
 
      - name: Cache npm packages
        uses: actions/cache@v4
        with:
          path: ~/.npm
          key: npm-${{ runner.os }}-${{ env.WEAVORI_VERSION }}
 
      - name: Generate synthetic data
        run: >
          npx --yes @weavori/cli@$WEAVORI_VERSION generate "$SOURCE_DB"
          --target "$TARGET_DB"
          --output plain
          --yes
 
      - name: Run tests
        run: go test ./...

Key Patterns

1. Version Pinning

Always pin the Weavori version in CI. This ensures deterministic builds and protects against unexpected changes from new releases.

env:
  WEAVORI_VERSION: 0.1.6

Use the version variable in the npx invocation:

$npx --yes @weavori/cli@$WEAVORI_VERSION generate ...

Automating version updates: If you use Renovate, add a regex manager to your Renovate config:

{
  "regexManagers": [
    {
      "fileMatch": ["^\\.github/workflows/.+\\.ya?ml$"],
      "matchStrings": ["WEAVORI_VERSION: (?<currentValue>\\d+\\.\\d+\\.\\d+)"],
      "depNameTemplate": "@weavori/cli",
      "datasourceTemplate": "npm",
      "versioningTemplate": "npm"
    }
  ]
}

This creates automated PRs when new versions of @weavori/cli are published.

2. npm Cache Persistence

The Weavori CLI ships as a platform-specific npm package (@weavori/cli-linux-x64, @weavori/cli-darwin-arm64, and so on) installed automatically through optionalDependencies. Cache npm's package cache between CI runs to avoid re-resolving the package on every invocation:

- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-${{ runner.os }}-${{ env.WEAVORI_VERSION }}

The cache key is scoped to the OS and version. npm verifies package integrity from its own cache, and npx --yes keeps installation non-interactive.

3. Secrets and Configuration

Setting	Recommended approach
API key	`WEAVORI_API_KEY` environment variable (from secrets)
Source DSN	Environment variable, passed as the first argument
Target DSN	Environment variable, referenced by `--target` flag
Output format	`--output` CLI flag
Non-interactive mode	`--yes` CLI flag

Keep secrets out of CLI arguments — use environment variables sourced from your CI provider's secret store:

env:
  WEAVORI_API_KEY: ${{ secrets.WEAVORI_API_KEY }}
  SOURCE_DB: ${{ secrets.SOURCE_DB }}
  TARGET_DB: ${{ secrets.TARGET_DB }}

Advanced CI Patterns

Schema Caching

By default, Weavori caches introspected schemas and re-introspects automatically when they change. In CI, you may want fresh introspection each run:

- name: Generate synthetic data (no cache)
  run: >
    npx --yes @weavori/cli@$WEAVORI_VERSION generate "$SOURCE_DB"
    --target "$TARGET_DB"
    --output plain
    --yes
    --no-cache

To clear all cached schemas before a run:

- name: Clear schema cache
  run: npx --yes @weavori/cli@$WEAVORI_VERSION cache clear

CI Smoke Test (Pre-flight Validation)

Use estimate to verify database connectivity and schema introspection without writing any data:

- name: Validate schema compatibility
  run: npx --yes @weavori/cli@$WEAVORI_VERSION estimate "$DB_URL" --output plain

This is a safe, read-only operation suitable for pre-flight checks in PR pipelines. It reports the table count, approximate row counts, and an estimated generation time.

Working with output formats

Set the output format via environment variable instead of passing it on every command:

env:
  WEAVORI_OUTPUT: plain

This is equivalent to passing --output plain on every invocation.

Exit codes

The CLI returns the following standardized exit codes for CI/CD scripting:

Code	Meaning
0	Success
1	General error
2	Authentication error
3	Database error
4	Quota exceeded
5	Cancelled
10	Internal error

Use these in CI scripts for conditional logic based on the result.

Diagnostic Checks

Use doctor to diagnose common issues with the Weavori setup in CI:

- name: Diagnose Weavori setup
  run: npx --yes @weavori/cli@$WEAVORI_VERSION doctor --output plain

Matrix Testing

Run generation across multiple PostgreSQL versions by combining the service matrix with the cache pattern:

strategy:
  matrix:
    pg-version: ["16", "15", "14"]
 
services:
  postgres:
    image: postgres:${{ matrix.pg-version }}
    # ...
 
steps:
  - uses: actions/cache@v4
    with:
      path: ~/.npm
      key: npm-${{ runner.os }}-${{ env.WEAVORI_VERSION }}

The cache key is shared across matrix jobs (same OS, same version), so the package is fetched only once per workflow run.

Other CI Providers

The same best practices — version pinning, npm caching, and secrets via environment variables — apply across all CI platforms.

GitLab CI

Full .gitlab-ci.yml job with npm cache and version pinning:

image: node:22
 
variables:
  WEAVORI_VERSION: "0.1.6"
  WEAVORI_API_KEY: $WEAVORI_API_KEY  # set in CI/CD settings
  SOURCE_DB: $SOURCE_DB_CICD         # set in CI/CD settings
  TARGET_DB: $TARGET_DB_CICD         # set in CI/CD settings
 
cache:
  key: npm-$CI_RUNNER_EXECUTABLE_ARCH-$WEAVORI_VERSION
  paths:
    - ~/.npm/
 
generate-synthetic-data:
  stage: test
  before_script:
    - apt-get update && apt-get install -y postgresql-client
  script:
    - npx --yes @weavori/cli@$WEAVORI_VERSION generate "$SOURCE_DB"
        --target "$TARGET_DB"
        --output plain
        --yes
  services:
    - postgres:16
  variables:
    POSTGRES_DB: test
    POSTGRES_USER: test
    POSTGRES_PASSWORD: test

Set WEAVORI_API_KEY, SOURCE_DB_CICD, and TARGET_DB_CICD as CI/CD variables in your project settings.

CircleCI

Full .circleci/config.yml job with Node, cache, and generation:

version: 2.1
 
jobs:
  generate-and-test:
    docker:
      - image: cimg/node:22
      - image: postgres:16
        environment:
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
          POSTGRES_DB: test
    environment:
      WEAVORI_VERSION: "0.1.6"
    steps:
      - checkout
      - run:
          name: Install PostgreSQL client
          command: sudo apt-get update && sudo apt-get install -y postgresql-client
      - restore_cache:
          keys:
            - npm-{{ arch }}-{{ .Environment.WEAVORI_VERSION }}
      - run:
          name: Generate synthetic data
          command: |
            npx --yes @weavori/cli@$WEAVORI_VERSION generate "$SOURCE_DB" \
              --target "$TARGET_DB" \
              --output plain \
              --yes
      - save_cache:
          key: npm-{{ arch }}-{{ .Environment.WEAVORI_VERSION }}
          paths:
            - ~/.npm
 
workflows:
  version: 2
  ci:
    jobs:
      - generate-and-test

Set WEAVORI_API_KEY, SOURCE_DB, and TARGET_DB as CircleCI environment variables in your project settings.

Jenkins (Declarative Pipeline)

Full Jenkinsfile stage with Node setup, cache, and generation:

pipeline {
    agent any
 
    environment {
        WEAVORI_VERSION = '0.1.6'
        WEAVORI_API_KEY = credentials('weavori-api-key')
        SOURCE_DB = credentials('weavori-source-db')
        TARGET_DB = credentials('weavori-target-db')
    }
 
    stages {
        stage('Generate Synthetic Data') {
            steps {
                sh '''
                    npx --yes @weavori/cli@$WEAVORI_VERSION generate "$SOURCE_DB" \
                        --target "$TARGET_DB" \
                        --output plain \
                        --yes
                '''
            }
        }
    }
 
    post {
        success {
            echo 'Synthetic data generation complete.'
        }
    }
}

Use Jenkins Pipeline Utility Steps or the cache plugin to persist ~/.npm across builds for faster execution.

Weavori's --output plain flag produces log-friendly output suitable for all CI environments.