Refresh and publish

A refresh is a complete publication operation. Skiff loads into private staging files, assembles and checks the next DuckDB, then atomically replaces the destination only after the new generation is ready.

Run a normal refresh

skiff refresh

Skiff discovers the nearest skiff.toml and freezes its plan for this process. It then:

1. locks the destination against another Skiff refresh;
2. removes abandoned private generations from earlier interrupted runs;
3. loads different sources in parallel into private staging files;
4. assembles every configured table into the new DuckDB;
5. runs each checked-in validation query;
6. writes one _skiff_refresh metadata row and checkpoints the database;
7. compares the new schema with the published file;
8. sets the new file to owner-only mode and atomically renames it over the destination.

Queries for the same source run in manifest order. Sources with snapshot = true wrap their queries in one SQL Server snapshot transaction.

Read progress and completion output

Table progress goes to stderr:

[2/4] reporting.incidents: 1842 rows in 731ms
[check] incident ids are unique: ok

The final stdout line distinguishes rows loaded during this run from rows retained in the published file:

refreshed 4 table(s), 2103 row(s) loaded, 98214 total → /repo/data/reporting.duckdb

This distinction matters for incremental tables, where a normal refresh can load only new rows while keeping earlier rows.

Treat checks as publication gates

A check passes when its DuckDB query returns zero rows. If any check returns rows or errors, Skiff discards the private generation and keeps the existing destination unchanged.

Checks can validate relationships across sources because they run after all configured tables have been assembled.

Review schema drift

When a destination already exists, Skiff compares its user-table schema with the private replacement and prints advisory changes:

[drift] incidents.closed_at added (TIMESTAMP)
[drift] units.status changed VARCHAR -> BIGINT

The refresh still publishes when drift is detected. Use the report to review downstream compatibility. If comparison itself is unavailable, Skiff reports that as an advisory schema change rather than failing an otherwise valid build.

MCP results include the same information in schema_changes.

Understand failure behavior

Source, query, limit, conversion, assembly, and check failures happen before publication. Their error includes:

Nothing was changed.
Action: The existing DuckDB is untouched; correct the source, query, or check and retry.

An error while syncing the parent directory after rename is different: the new DuckDB was already published, and Skiff reports that changes were made. Verify the destination before retrying.

Coordinate readers and writers

Skiff's .skiff.lock file prevents two Skiff refreshes from running at once and identifies the active PID, start time, and output. It does not coordinate arbitrary DuckDB writers.

Skiff must be the only writer. Readers should open the published file read-only and be prepared to reopen it after a generation changes.