CI/CD Integration

Integrate Weavori into your CI/CD pipelines for automated synthetic data generation.

Weavori integrates into any CI/CD pipeline that supports running npx commands. This guide covers GitHub Actions setup with best practices for production use.

Quick Start (GitHub Actions)

Add a job to generate synthetic data and run your test suite against it:

name: Integration Tests

on:
  push:
    branches: [main]
  pull_request:

env:
  WEAVORI_VERSION: 1.0.0
  WEAVORI_API_KEY: ${{ secrets.WEAVORI_API_KEY }}
  SOURCE_DB: ${{ secrets.SOURCE_DB }}
  TARGET_DB: ${{ secrets.TARGET_DB }}

jobs:
  generate-and-test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
          POSTGRES_DB: test
        ports:
          - 5432:5432
        options: >-
          --health-cmd="pg_isready -U test"
          --health-interval=5s
          --health-timeout=3s
          --health-retries=5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"

      - name: Cache Weavori binary
        uses: actions/cache@v4
        with:
          path: ~/.cache/weavori/
          key: weavori-${{ runner.os }}-${{ env.WEAVORI_VERSION }}

      - name: Generate synthetic data
        run: >
          npx @weavori/cli@$WEAVORI_VERSION generate
          --url "$SOURCE_DB"
          --target "$TARGET_DB"
          --output plain
          --yes

      - name: Run tests
        run: go test ./...

Key Patterns

1. Version Pinning

Always pin the Weavori version in CI. This ensures deterministic builds and protects against unexpected changes from new releases.

env:
  WEAVORI_VERSION: 1.0.0

Use the version variable in the npx invocation:

npx @weavori/cli@$WEAVORI_VERSION generate ...

Automating version updates: If you use Renovate, add a regex manager to your Renovate config:

{
  "regexManagers": [
    {
      "fileMatch": ["^\\.github/workflows/.+\\.ya?ml$"],
      "matchStrings": ["WEAVORI_VERSION: (?<currentValue>\\d+\\.\\d+\\.\\d+)"],
      "depNameTemplate": "@weavori/cli",
      "datasourceTemplate": "npm",
      "versioningTemplate": "npm"
    }
  ]
}

This creates automated PRs when new versions of @weavori/cli are published.

2. Binary Cache Persistence

The Weavori npm launcher stores downloaded Go binaries at ~/.cache/weavori/. Cache this directory between CI runs to avoid re-downloading on every invocation:

- uses: actions/cache@v4
  with:
    path: ~/.cache/weavori/
    key: weavori-${{ runner.os }}-${{ env.WEAVORI_VERSION }}

The cache key is scoped to the OS and version. The launcher's built-in checksum verification ensures cached binaries remain valid.

3. Secrets and Configuration

Setting	Recommended approach
API key	`WEAVORI_API_KEY` environment variable (from secrets)
Source DSN	Environment variable, referenced by `--url` flag
Target DSN	Environment variable, referenced by `--target` flag
Output format	`--output` CLI flag
Non-interactive mode	`--yes` CLI flag

Keep secrets out of CLI arguments — use environment variables sourced from your CI provider's secret store:

env:
  WEAVORI_API_KEY: ${{ secrets.WEAVORI_API_KEY }}
  SOURCE_DB: ${{ secrets.SOURCE_DB }}
  TARGET_DB: ${{ secrets.TARGET_DB }}

Advanced CI Patterns

Schema Caching

By default, Weavori caches introspected schemas by DSN fingerprint. In CI, you may want fresh introspection each run:

- name: Generate synthetic data (no cache)
  run: >
    npx @weavori/cli@$WEAVORI_VERSION generate
    --url "$SOURCE_DB"
    --target "$TARGET_DB"
    --output plain
    --yes
    --no-cache

To clear all cached schemas before a run:

- name: Clear schema cache
  run: npx @weavori/cli@$WEAVORI_VERSION cache clear

CI Smoke Test (Pre-flight Validation)

Use estimate to verify database connectivity and schema introspection without writing any data:

- name: Validate schema compatibility
  run: npx @weavori/cli@$WEAVORI_VERSION estimate --url "$DB_URL" --output plain

This is a safe, read-only operation suitable for pre-flight checks in PR pipelines.

Use --output json for programmatic consumption:

- name: Validate schema compatibility (JSON)
  run: npx @weavori/cli@$WEAVORI_VERSION estimate --url "$DB_URL" --output json

Working with output formats

Set the output format via environment variable instead of passing it on every command:

env:
  WEAVORI_OUTPUT: plain

This is equivalent to passing --output plain on every invocation.

Exit codes

The CLI returns the following standardized exit codes for CI/CD scripting:

Code	Meaning
0	Success
1	General error
2	Authentication error
3	Database error
4	Quota exceeded
5	Cancelled
10	Internal error

Use these in CI scripts for conditional logic based on the result.

Diagnostic Checks

Use doctor to diagnose common issues with the Weavori setup in CI:

- name: Diagnose Weavori setup
  run: npx @weavori/cli@$WEAVORI_VERSION doctor --output plain

Matrix Testing

Run generation across multiple PostgreSQL versions by combining the service matrix with the cache pattern:

strategy:
  matrix:
    pg-version: ["16", "15", "14"]

services:
  postgres:
    image: postgres:${{ matrix.pg-version }}
    # ...

steps:
  - uses: actions/cache@v4
    with:
      path: ~/.cache/weavori/
      key: weavori-${{ runner.os }}-${{ env.WEAVORI_VERSION }}

The cache key is shared across matrix jobs (same OS, same version), so the binary is downloaded only once per workflow run.

Other CI Providers

The same best practices — version pinning, binary caching, and secrets via environment variables — apply across all CI platforms.

GitLab CI

Full .gitlab-ci.yml job with binary cache and version pinning:

image: node:22

variables:
  WEAVORI_VERSION: "1.0.0"
  WEAVORI_API_KEY: $WEAVORI_API_KEY  # set in CI/CD settings
  SOURCE_DB: $SOURCE_DB_CICD         # set in CI/CD settings
  TARGET_DB: $TARGET_DB_CICD         # set in CI/CD settings

cache:
  key: weavori-$CI_RUNNER_EXECUTABLE_ARCH-$WEAVORI_VERSION
  paths:
    - ~/.cache/weavori/

generate-synthetic-data:
  stage: test
  before_script:
    - apt-get update && apt-get install -y postgresql-client
  script:
    - npx @weavori/cli@$WEAVORI_VERSION generate
        --url "$SOURCE_DB"
        --target "$TARGET_DB"
        --output plain
        --yes
  services:
    - postgres:16
  variables:
    POSTGRES_DB: test
    POSTGRES_USER: test
    POSTGRES_PASSWORD: test

Set WEAVORI_API_KEY, SOURCE_DB_CICD, and TARGET_DB_CICD as CI/CD variables in your project settings.

CircleCI

Full .circleci/config.yml job with Node, cache, and generation:

version: 2.1

jobs:
  generate-and-test:
    docker:
      - image: cimg/node:22
      - image: postgres:16
        environment:
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
          POSTGRES_DB: test
    environment:
      WEAVORI_VERSION: "1.0.0"
    steps:
      - checkout
      - run:
          name: Install PostgreSQL client
          command: sudo apt-get update && sudo apt-get install -y postgresql-client
      - restore_cache:
          keys:
            - weavori-{{ arch }}-{{ .Environment.WEAVORI_VERSION }}
      - run:
          name: Generate synthetic data
          command: |
            npx @weavori/cli@$WEAVORI_VERSION generate \
              --url "$SOURCE_DB" \
              --target "$TARGET_DB" \
              --output plain \
              --yes
      - save_cache:
          key: weavori-{{ arch }}-{{ .Environment.WEAVORI_VERSION }}
          paths:
            - ~/.cache/weavori/

workflows:
  version: 2
  ci:
    jobs:
      - generate-and-test

Set WEAVORI_API_KEY, SOURCE_DB, and TARGET_DB as CircleCI environment variables in your project settings.

Jenkins (Declarative Pipeline)

Full Jenkinsfile stage with Node setup, cache, and generation:

pipeline {
    agent any

    environment {
        WEAVORI_VERSION = '1.0.0'
        WEAVORI_API_KEY = credentials('weavori-api-key')
        SOURCE_DB = credentials('weavori-source-db')
        TARGET_DB = credentials('weavori-target-db')
    }

    stages {
        stage('Setup') {
            steps {
                script {
                    // Use a tool-agnostic cache path
                    def cacheDir = "${env.HOME}/.cache/weavori"
                    env.WEAVORI_CACHE_DIR = cacheDir
                }
            }
        }

        stage('Generate Synthetic Data') {
            steps {
                sh '''
                    npx @weavori/cli@$WEAVORI_VERSION generate \
                        --url "$SOURCE_DB" \
                        --target "$TARGET_DB" \
                        --output plain \
                        --yes
                '''
            }
        }
    }

    post {
        success {
            echo 'Synthetic data generation complete.'
        }
    }
}

Use Jenkins Pipeline Utility Steps or the cache plugin to persist ~/.cache/weavori/ across builds for faster execution.

Weavori's --output plain flag produces log-friendly output suitable for all CI environments.