Get started

This walkthrough creates one repo-owned DuckDB from one checked-in SQL Server query. You will validate the complete contract before Skiff reads and publishes data.

Before you begin

You need:

• a SQL Server hostname, database, and login;
• SELECT permission for the exact source data you intend to copy;
• a repository where the manifest, queries, and checks will live.

Install a prebuilt release for your platform:

Release binaries support macOS 13 Ventura or newer on Apple Silicon and x86-64 Linux with glibc 2.35 or newer (Ubuntu 22.04, Debian 12, or comparable).

Skiff's repository is private, so the installer needs a GitHub token that can read it. Create a fine-grained personal access token with access to only the TheSkurvied/skiff repository and read-only Contents permission. The installer reads it from SKIFF_GITHUB_TOKEN.

export SKIFF_GITHUB_TOKEN=github_pat_...
curl --proto '=https' --tlsv1.2 -LsSf -H "Authorization: Bearer $SKIFF_GITHUB_TOKEN" https://github.com/TheSkurvied/skiff/releases/latest/download/skiff-installer.sh | sh

To update, run the same command again. Confirm the installed version:

skiff --version

These docs describe Skiff 0.7.0. Contributors can instead build from source with Rust.

Create the project files

From the repository that will own the DuckDB:

mkdir -p queries checks data

Create skiff.toml:

version = 1

[[sources]]
name = "reporting"
host = "sql.example.com"
database = "Reporting"
user = "skiff_reader"
password_env = "REPORTING_DB_PASS"

[output]
file = "data/reporting.duckdb"

[limits]
max_rows = 500000
timeout_secs = 60

[[queries]]
source = "reporting"
file = "queries/incidents.sql"
table = "incidents"

[[checks]]
name = "incident ids are unique"
file = "checks/incident_ids.sql"

Create queries/incidents.sql:

SELECT
    id,
    opened_at,
    priority
FROM dbo.incidents;

Create checks/incident_ids.sql:

SELECT id
FROM incidents
GROUP BY id
HAVING count(*) > 1;

A check passes when it returns zero rows. Checks run against the completed private DuckDB before publication.

Load the password

Skiff reads the variable named by password_env. It automatically loads missing variables from .env beside the discovered skiff.toml; values already exported by the launching process take precedence.

For one shell session:

export REPORTING_DB_PASS='your-password'

For a local .env workflow, keep the file out of source control:

cp .env.example .env

Validate before loading data

Run check from the repository or any child directory:

skiff check

Skiff validates the manifest and file paths, parses each query, requires every password variable, and tests every source connection. A successful check names the source, query, and check counts plus the resolved output path.

Publish the DuckDB

skiff refresh

Progress is written to stderr as each table finishes. The final stdout line reports the number of tables, rows loaded during this run, total rows in the published file, and its path.

Open the destination read-only in downstream tools. The database also contains _skiff_refresh, a single metadata row for the current generation.

If the refresh fails

Read the Action: line in the error. Query, source, limit, or check failures occur before publication, so the prior DuckDB is left untouched. Correct the reported cause and run skiff check or skiff refresh again.

Next, learn how refresh publication works or configure incremental loads.