# Agent Skills
Source: https://xata.io/docs/ai-agents/agent-skills

Guided workflows that teach AI coding agents how to operate Xata.

# Agent Skills

Agent skills extend AI coding assistants with Xata-specific knowledge: branch workflows, CLI commands, API conventions, and PostgreSQL troubleshooting.

## What are agent skills?

Agent skills follow the [Agent Skills](https://agentskills.io) open format. Each skill is a markdown file (`SKILL.md`) with metadata, step-by-step instructions, and reference files for common tasks.

When you ask an agent to create a branch, run a migration, or diagnose a slow query, it can load the matching skill and follow Xata-specific procedures instead of guessing.

### Supported tools

Any MCP client connected to the [Xata MCP server](/docs/platform/mcp) can list and read skills through the `list_skills` and `get_skill` tools, including Claude Code, Cursor, Codex, and any other client documented on the MCP setup page.

## Available skills

| Skill | Use when |
| - | - |
| `using-xata-cli` | Working with projects, branches, auth, migrations, clone, keys, or organizations from the terminal |
| `using-xata-api` | Calling the Xata HTTP API directly, scripting against `api.xata.tech`, or building apps that manage Xata programmatically |
| `branching-postgresql-for-agents` | Fixing a bug or building a feature that needs database queries or tests, without reading or writing the source or production database |
| `managing-postgresql` | Diagnosing slow queries, high CPU, connection issues, locks, vacuum, indexes, backups, replication, or other PostgreSQL problems on a branch |

## Access skills through MCP

After connecting to the Xata MCP server, ask your agent:

> List the available Xata skills.

The agent calls `list_skills` and returns the skill catalog. To load a skill:

> Read the `using-xata-cli` skill and create a development branch for this task.

The agent calls `get_skill` with the skill name and follows the instructions.

> [!TIP]
>
> Skills are served by the hosted MCP server. There is nothing to install locally. Connect once in your agent client and the skills are available automatically.

## Related docs

* [Xata for Agents](/docs/ai-agents/overview)
* [MCP Server](/docs/platform/mcp)
* [CLI Reference](/docs/cli)


# Claude Code
Source: https://xata.io/docs/ai-agents/claude-code

Use Claude Code with isolated Xata branches and branch-scoped Postgres connection strings.

# Claude Code with Xata

Claude Code is a terminal-based coding agent. With Xata, it can work against a dedicated Postgres branch while it edits files, runs tests, and checks schema behavior.

## What Claude Code is good for with Xata

* Creating feature branches that include code and database changes.
* Running migrations against a temporary Postgres branch before review.
* Inspecting schema, connection errors, and test failures from the terminal.
* Producing a summary of database changes before a pull request is merged.

## Quick start

1. Install and authenticate the Xata CLI.

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
xata auth login
xata init
```

2. Create a branch for Claude Code.

```bash theme={null}
xata branch create --name claude-task --parent-branch `xata branch get id`
```

3. Store the branch connection string in your local environment.

```bash theme={null}
echo "DATABASE_URL=$(xata branch url claude-task)" >> .env
```

4. Start Claude Code from the project directory and describe the task.

```bash theme={null}
claude
```

## Recommended Xata setup

Use a branch name that matches the task or pull request. If the task needs realistic data, create the branch from an anonymized clone rather than the production branch. Give Claude Code only the connection string for the branch it should use.

For schema work, ask Claude Code to run migrations against the branch and report the migration files, SQL statements, and test output before you apply anything to production.

## Example prompts

* "Use the Xata branch in `DATABASE_URL` for all database work. Do not connect to production."
* "Create a migration for this feature, run it against the current Xata branch, and show me the schema diff."
* "Run the test suite against this branch and fix any application code that assumes the old schema."
* "Inspect the failing query and explain whether the issue is SQL, connection configuration, or missing data."
* "Summarize every database change you made before I open a pull request."

## Common workflows

### Branch-scoped feature work

1. Create a Xata branch for the task.
2. Store `xata branch url <branch>` in `.env`.
3. Ask Claude Code to implement the feature and run tests.
4. Review code, migration files, and branch state.
5. Delete the branch after merge or when the task is abandoned.

### Migration review

1. Ask Claude Code to generate or update the migration.
2. Run the migration against the task branch.
3. Ask Claude Code to verify reads and writes against the new schema.
4. Review the SQL and test output before applying the migration elsewhere.

## Tips

* Keep production credentials out of files the agent can edit.
* Prefer one Xata branch per task so failures are easy to reset.
* Use [scale to zero](/docs/core-concepts/scale-to-zero) for idle agent branches.
* Link Claude Code to existing docs by mentioning [branch commands](/docs/cli/branch) and [schema changes](/docs/core-concepts/schema-changes) in prompts.

## Troubleshooting

| Issue | Fix |
| - | - |
| Claude Code connects to the wrong database | Replace `DATABASE_URL` with `xata branch url <branch>` and ask it to print the active database host before running writes. |
| Migrations pass locally but fail in CI | Create the CI branch from the same parent and run the same migration command there. |
| The agent changed production-like data | Stop the task, rotate exposed credentials if needed, and repeat the work on an isolated branch. |

## Related Xata docs

* [Quickstart](/docs/quickstart)
* [CLI branch commands](/docs/cli/branch)
* [Instant Branching](/docs/core-concepts/branching)
* [Schema Changes](/docs/core-concepts/schema-changes)
* [API Keys](/docs/platform/api-key)


# Cline
Source: https://xata.io/docs/ai-agents/cline

Use Cline with approval-based Xata branch workflows in VS Code.

# Cline with Xata

Cline is useful when you want an agent to propose file edits and terminal commands, then wait for approval. That approval model pairs well with Xata branches because each database action can be checked before it runs.

## What Cline is good for with Xata

* Supervised database setup in VS Code.
* Reviewing terminal commands before they touch a branch.
* Browser- or terminal-assisted debugging with a safe database target.
* Step-by-step migration work where every command is approved.

## Quick start

1. Create a Xata branch for the task.

```bash theme={null}
xata branch create --name cline-task --parent-branch `xata branch get id`
```

2. Add the branch URL to local environment configuration.

```bash theme={null}
echo "DATABASE_URL=$(xata branch url cline-task)" >> .env.local
```

3. Open the project in VS Code.
4. Ask Cline to propose commands before running migrations or writes.

## Recommended Xata setup

Use Cline's approval step for every database command. Give it a branch URL, not production credentials. For schema changes, approve the migration generation separately from the migration execution.

## Example prompts

* "Before running any database command, show me the exact command and wait for approval."
* "Use only the Xata branch in `.env.local`."
* "Generate the migration first, then wait before applying it."
* "Inspect this query failure and suggest a fix without changing production data."
* "After the task, list every Xata command you ran."

## Common workflows

### Supervised migration

1. Ask Cline to generate a migration.
2. Review the file diff.
3. Approve applying it to the Xata branch.
4. Approve running tests.
5. Review the summary before merging.

### Branch-scoped debugging

1. Reproduce the issue against the task branch.
2. Ask Cline to inspect logs, SQL, and schema.
3. Approve only the smallest proposed change.
4. Re-run the test against the same branch.

## Tips

* Approve reads freely, but review writes carefully.
* Keep production secrets out of VS Code workspace files.
* Ask Cline to use [branch commands](/docs/cli/branch) for branch state.
* Delete the task branch after the pull request is merged.

## Troubleshooting

| Issue | Fix |
| - | - |
| Cline proposes a command against production | Reject it and provide the branch URL explicitly. |
| Too many commands are proposed at once | Ask Cline to split the task into setup, migration, test, and cleanup phases. |
| A generated migration includes unrelated changes | Ask Cline to regenerate it from the branch state and explain each operation. |

## Related Xata docs

* [Quickstart](/docs/quickstart)
* [Branch commands](/docs/cli/branch)
* [Schema Changes](/docs/core-concepts/schema-changes)
* [API Keys](/docs/platform/api-key)


# Codex
Source: https://xata.io/docs/ai-agents/codex

Use Codex with Xata branches for sandboxed coding sessions and database-aware tests.

# Codex with Xata

Codex can inspect a repository, edit files, run shell commands, and verify changes. Xata gives Codex a safe Postgres branch to use while it implements features or fixes tests.

## What Codex is good for with Xata

* Implementing code changes that need a real Postgres database.
* Running test suites against a branch-scoped connection string.
* Updating SQL, ORM models, or migrations and validating them in one session.
* Cleaning up generated changes before a pull request.

## Quick start

1. Install and authenticate the Xata CLI.

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
xata auth login
xata init
```

2. Create a task branch.

```bash theme={null}
xata branch create --name codex-task --parent-branch `xata branch get id`
```

3. Export the branch connection string for the Codex session.

```bash theme={null}
export DATABASE_URL="$(xata branch url codex-task)"
```

4. Start Codex in the repository and include the database boundary in the prompt.

```bash theme={null}
codex
```

## Recommended Xata setup

Use a branch per Codex task. For risky changes, start from anonymized production data so tests cover realistic relationships without exposing sensitive values. If the task needs API access, create a scoped key and remove it after the session.

## Example prompts

* "Use only the Postgres database in `DATABASE_URL`. Do not ask for or use production credentials."
* "Add the feature, update the database migration, and run tests against the Xata branch."
* "Use the Xata CLI docs in `/cli/branch` if you need to inspect branch state."
* "Before finishing, summarize schema changes, test commands, and any rows you inserted."
* "Clean up temporary data in this branch before reporting completion."

## Common workflows

### Branch-per-task implementation

1. Create a Xata branch named for the task.
2. Export `DATABASE_URL` for that branch.
3. Ask Codex to implement the code change.
4. Ask Codex to run migrations and tests.
5. Review the diff and branch state before merging.

### Test failure investigation

1. Reproduce the failing test against the task branch.
2. Ask Codex to inspect the query, schema, and fixtures.
3. Let Codex change application code or test setup.
4. Keep data changes inside the temporary branch.

## Tips

* Put database boundaries in the first prompt.
* Prefer branch-scoped API keys for automation work.
* Ask Codex to show exact commands before destructive database operations.
* Use [branch cleanup automation](/docs/automations/ga-cleanup) for repeated team workflows.

## Troubleshooting

| Issue | Fix |
| - | - |
| Codex cannot connect to Postgres | Re-run `xata branch url codex-task` and confirm `DATABASE_URL` is exported in the same shell. |
| Codex proposes production writes | Reject the step and restate that all writes must target the Xata task branch. |
| The task creates noisy fixture data | Ask Codex to delete test rows or recreate the branch from its parent. |

## Related Xata docs

* [Quickstart](/docs/quickstart)
* [CLI Reference](/docs/cli)
* [Branch commands](/docs/cli/branch)
* [Data Anonymization](/docs/core-concepts/anonymization)
* [Create Branch automation](/docs/automations/ga-pr)


# Cursor
Source: https://xata.io/docs/ai-agents/cursor

Use Cursor with Xata branch connection strings for IDE and background-agent workflows.

# Cursor with Xata

Cursor works well for multi-file edits, background agents, and database-aware application changes. Use a Xata branch as the database target for Composer and agent sessions.

## What Cursor is good for with Xata

* Updating ORM models, queries, and migrations together.
* Testing a feature against an isolated branch from inside the IDE terminal.
* Reviewing generated code and SQL before it leaves your editor.
* Running background agents without giving them production database credentials.

## Quick start

1. Create a Xata branch for the task.

```bash theme={null}
xata branch create --name cursor-task --parent-branch `xata branch get id`
```

2. Put the branch URL in `.env.local`.

```bash theme={null}
echo "DATABASE_URL=$(xata branch url cursor-task)" >> .env.local
```

3. Open the project in Cursor.
4. Ask Composer or a background agent to use the connection string from `.env.local`.

## Recommended Xata setup

Keep a separate branch for each Cursor task or background agent. If Cursor needs to generate migrations, ask it to apply them only to the task branch and show the SQL or migration file before you merge.

## Example prompts

* "Use the Xata branch connection string in `.env.local` for all database work."
* "Update the Drizzle or Prisma schema and run the app against this branch."
* "Generate the migration, apply it to the branch, and show me the SQL before I merge."
* "Create a pull request summary that includes database changes and test output."
* "Do not edit production environment variables."

## Common workflows

### Composer with a branch URL

1. Create a branch for the feature.
2. Add the branch connection string to `.env.local`.
3. Ask Composer to implement the feature.
4. Run the app and tests from Cursor's terminal.
5. Review generated schema and data changes.

### Background agent for pull requests

1. Create a task branch in Xata.
2. Include the branch URL in the agent instructions.
3. Ask the agent to keep database changes isolated.
4. Review every migration and environment change before merging.

## Tips

* Keep `.env.local` out of commits.
* Prefer branch names that match the issue or pull request.
* Ask Cursor to explain generated SQL in plain language.
* Link Cursor to [Instant Branching](/docs/core-concepts/branching) when prompting for database safety.

## Troubleshooting

| Issue | Fix |
| - | - |
| Cursor edits the wrong environment file | Move production variables out of editable local files and keep only the branch URL in `.env.local`. |
| Background agent loses database context | Add the branch name and `DATABASE_URL` source to the task prompt. |
| Generated migrations are hard to review | Ask Cursor to summarize each operation and run it against the branch before merge. |

## Related Xata docs

* [Drizzle quickstart](/docs/quickstarts/drizzle)
* [Instant Branching](/docs/core-concepts/branching)
* [Schema Changes](/docs/core-concepts/schema-changes)
* [Branch commands](/docs/cli/branch)


# Devin
Source: https://xata.io/docs/ai-agents/devin

Use Devin with temporary Xata branches, scoped credentials, and explicit cleanup.

# Devin with Xata

Devin is designed for longer-running implementation tasks. When a task touches data or migrations, give Devin a temporary Xata branch and scoped credentials so it can work independently without access to production.

## What Devin is good for with Xata

* Longer implementation tasks that need a realistic database.
* Pull requests that include application, migration, and test changes.
* Repeated investigation work where the agent needs to run commands over time.
* Branch cleanup and handoff summaries after the task is complete.

## Quick start

1. Create a branch for the Devin task.

```bash theme={null}
xata branch create --name devin-task --parent-branch `xata branch get id`
```

2. Create or choose credentials scoped to that branch.
3. Give Devin the branch connection string and the task boundary.
4. Ask Devin to include database changes, test output, and cleanup steps in the handoff.

## Recommended Xata setup

Use a fresh branch for every Devin task. For production-like tasks, start from anonymized data. If Devin needs API access, create a branch-scoped key and revoke it after the task.

## Example prompts

* "Use this Xata branch for all database work and do not connect to production."
* "Implement the feature, run migrations against the branch, and include test output in the pull request."
* "If you need new data, create it only in the temporary branch."
* "Before finishing, summarize code changes, schema changes, credentials used, and cleanup steps."
* "After the task is merged, delete the temporary branch and remove scoped credentials."

## Common workflows

### Long-running feature task

1. Create a branch from the correct parent.
2. Give Devin the branch connection string.
3. Ask Devin to run migrations and tests against the branch.
4. Review the pull request and database summary.
5. Delete the branch after merge.

### Investigation task

1. Create a branch that reproduces the issue.
2. Ask Devin to inspect schema, queries, and logs.
3. Keep all inserts and updates inside the branch.
4. Ask Devin to produce a written finding before changing production workflows.

## Tips

* Use scoped API keys for cloud agents.
* Set a clear cleanup expectation in the first prompt.
* Prefer anonymized clones when the task needs realistic data.
* Ask Devin to report the branch name and database host in every handoff.

## Troubleshooting

| Issue | Fix |
| - | - |
| Devin needs long-lived access | Use a branch-scoped key and rotate or revoke it after the task. |
| The task modifies too much data | Recreate the branch from the parent and narrow the prompt. |
| The handoff omits database changes | Ask Devin to list migrations, SQL commands, branch name, and test output before review. |

## Related Xata docs

* [Data Anonymization](/docs/core-concepts/anonymization)
* [API Keys](/docs/platform/api-key)
* [Instant Branching](/docs/core-concepts/branching)
* [Branch commands](/docs/cli/branch)


# GitHub Copilot
Source: https://xata.io/docs/ai-agents/github-copilot

Use GitHub Copilot with Xata per-PR branches and CI-backed database workflows.

# GitHub Copilot with Xata

GitHub Copilot can help write application code, migrations, and CI configuration. Xata is a good fit for Copilot agent workflows when every pull request gets its own Postgres branch.

## What GitHub Copilot is good for with Xata

* Generating GitHub Actions workflows for database-backed pull requests.
* Updating code and migrations inside VS Code.
* Explaining test failures from CI logs.
* Producing pull request summaries that include database changes.

## Quick start

1. Create a Xata API key for automation.
2. Add the key to GitHub repository secrets as `XATA_API_KEY`.
3. Use the [Create Branch workflow](/docs/automations/ga-pr) as the starting point for per-PR branches.
4. Ask Copilot to adapt the workflow to your repository's package manager, test command, and migration command.

## Recommended Xata setup

Use branch-per-PR automation for team workflows. Copilot can update the workflow file, but keep secrets in GitHub and branch credentials scoped to the pull request. For local VS Code work, use a local `.env` or `.env.local` file with a branch URL.

## Example prompts

* "Create a GitHub Actions workflow based on Xata's per-PR branch docs."
* "Use `XATA_API_KEY` from GitHub secrets and do not hardcode credentials."
* "Add a CI step that runs migrations against the Xata PR branch before tests."
* "Explain this failed migration log and suggest the smallest code change."
* "Write a pull request summary that calls out schema and data changes."

## Common workflows

### Per-PR branch automation

1. Add `XATA_API_KEY` to repository secrets.
2. Add the Xata branch workflow from the docs.
3. Ask Copilot to wire the branch URL into your test command.
4. Run migrations and tests in CI.
5. Clean up PR branches after merge.

### Local VS Code changes

1. Create a Xata branch for the task.
2. Store the branch URL in `.env.local`.
3. Ask Copilot to update code and migrations.
4. Run the app and tests locally before pushing.

## Tips

* Ask Copilot to cite the exact Xata docs section it followed.
* Keep GitHub secrets out of generated files.
* Review generated workflow permissions carefully.
* Use [branch cleanup](/docs/automations/ga-cleanup) for repositories with many agent-created pull requests.

## Troubleshooting

| Issue | Fix |
| - | - |
| GitHub Actions cannot authenticate | Confirm `XATA_API_KEY` is present in repository secrets and the key has branch access. |
| Copilot generates a workflow with broad permissions | Reduce permissions to the minimum needed for the PR workflow before merging. |
| CI tests use production data | Make the branch URL an explicit CI environment variable and fail the job if it is missing. |

## Related Xata docs

* [Create Branch workflow](/docs/automations/ga-pr)
* [Branch cleanup](/docs/automations/ga-cleanup)
* [API Keys](/docs/platform/api-key)
* [CLI environment variables](/docs/cli)


# Agent onboarding
Source: https://xata.io/docs/ai-agents/onboarding

A guide for coding agents to help users get started with Xata, from installing the CLI to a working Postgres branch connected to their app.

This page contains instructions for how to help a user get started with Xata. It is written for coding agents; if you are a human, paste the prompt from the [Quickstart](/docs/quickstart) into your agent and it will follow this guide for you.

Walk through it interactively with the user, answer any questions, and make sure they understand what Xata is and what it can do. The end goal is a working Postgres branch connected to their project, plus hands-on experience with the branching workflow that makes Xata different. If they want to migrate an existing database, help them with that as well (see the migration appendix below).

Note that Xata is designed to work for both very technical and non-technical users. Establish their level of technical knowledge at the start and adjust the tone of your questions accordingly. If the user is non-technical, don't ask technical questions about Postgres versions, regions, or instance sizes. Pick good defaults and explain what you are doing in plain language.

Ground rules for this onboarding:

* Run the commands for the user, but explain why you are running them so they can follow along.
* The Xata CLI detects coding agents and disables interactive prompts. Always pass explicit flags, and add `--json` when you need to parse output. When a command fails because a value is missing or invalid, the error names the flag to pass — ask the user for that value rather than guessing.
* Before starting, look at the current directory. If there is an existing application, ask whether to connect it to Xata; if not, ask whether they want to scaffold a small app or just explore the database workflow. Don't scaffold a project without agreement.
* Before database access, read the project's existing agent instructions. If `branching-postgresql-for-agents` already applies, follow its complete isolated-branch workflow instead of this guide's `main`-based examples. Keep application runs, SQL, tests, and schema migrations on the verified child, and leave it available for review; do not switch the application back to `main` or delete that child. If a requested migration requires reading the source database, stop and explain the conflict before proceeding.
* Ask before installing software, creating billable cloud resources, deleting branches, or copying data from an existing database.
* Never print full connection strings or API keys into the conversation; write them to gitignored env files instead. Resource-creation, metadata, readiness, and deletion output can contain credentials too, including a credential-bearing `connectionString`. `--json` does not redact secrets. Capture sensitive output or filter it to explicitly allowed fields before it reaches tool output or the conversation; don't dump raw responses or enable shell tracing (`set -x`).

## What is Xata?

Xata is a PostgreSQL platform built around instant, copy-on-write branching. It is standard PostgreSQL, not a fork: existing drivers, ORMs, and Postgres tools work normally, and there is no SDK to install. (Available extensions are listed at [https://xata.io/docs/platform/extensions](https://xata.io/docs/platform/extensions).)

Key capabilities to explain to the user:

* **Instant branching** — a branch is a full Postgres database that starts as a copy-on-write copy of its parent, without copying the entire dataset. Copy-on-write avoids duplicating inherited data; additional storage and snapshots are billed according to your plan.
* **Scale to zero** — when scale-to-zero is enabled, idle branches release their compute and wake on the next connection, reducing compute costs for per-developer, per-PR, and per-agent branches. Manually hibernated branches without scale-to-zero enabled must be explicitly woken.
* **Data anonymization** — production data can be cloned into branches with PII masked, so development and agent work happens on realistic but safe data.

## Getting started

### Step 1: Install the CLI

The CLI is the main way you (the agent) and the user will interact with Xata. Confirm with the user, then run the install:

```bash theme={null}
# macOS / Linux
curl -fsSL https://xata.io/install.sh | bash

# Windows
powershell -c "irm https://xata.io/install.ps1 | iex"
```

The binary installs to `~/.config/xata/bin`. Make sure that directory is on the PATH, then verify with `xata version`.

#### Set up skills for future agent sessions

Xata skills teach coding agents how to use the CLI and follow Xata-specific workflows. Offer to install the CLI skill in the user's project so future sessions can discover it. This does not require Xata authentication or MCP setup.

Once the user has chosen a project directory, identify the coding agent running this session and its supported `--agent` value, such as `claude-code`, `codex`, `cursor`, or `amp`. Ask if uncertain; do not infer the agent from its model name or another client's configuration files. See [the skills command reference](/docs/cli/skill) for all supported agents. If the user is only exploring and has no project directory yet, defer installation.

Explain that this adds project-local instruction files and get approval, then check availability:

```bash theme={null}
xata skill install --help
xata skill list --json
```

If the installed CLI does not support these commands, offer to upgrade it with permission, or use [skills through MCP](/docs/ai-agents/agent-skills#access-skills-through-mcp). Do not report a local installation as complete if it was skipped or failed.

Install `using-xata-cli`, replacing both placeholders with the verified values:

```bash theme={null}
xata skill install using-xata-cli \
  --agent <agent-id> \
  --directory "<project-root>" \
  --json
```

Always supply the skill name and agent: without a name, noninteractive `install` only lists skills. `--directory` is the project root, not the final skills directory. Keep the installation project-local unless the user explicitly wants a global installation.

Verify the result and installed files, then read the installed `SKILL.md` now rather than assuming the running session has loaded it. Keep this guide's approval and credential-handling rules when following skill examples. An identical installation can report `unchanged`; if existing files differ, stop and ask before replacing them. Do not automatically add `--force`, which discards local edits. Installed skills are snapshots, not automatic updates.

With permission, append a short pointer to the project instruction file this client reads, such as an existing `AGENTS.md` or `CLAUDE.md`. Preserve existing guidance and replace the path below with the actual project-relative skill directory so the pointer works in other checkouts:

```markdown theme={null}
## Xata

Before running Xata CLI commands, read
<installed CLI skill path>/SKILL.md and relevant references.
If guidance is missing or conflicts with project rules,
stop and ask. Never expose credentials or bypass approval.
```

Offer to include the complete project-local skill directory, including references, and the instruction change in version control so they are available in future checkouts. Installation makes skills discoverable; it does not guarantee automatic use. A future session may need a restart or reload. Verify discovery in a fresh or reloaded session when the client supports it; otherwise tell the user that this check remains outstanding.

### Step 2: Log in

Check whether the user is already authenticated:

```bash theme={null}
xata auth status
```

If not, run `xata auth login`. It prints a URL and a code, then waits. Tell the user to open the URL in their browser, enter the code, and sign in (they can create a free account during this flow with GitHub, Google, or email). Keep the command running until it confirms success.

In headless or CI environments, authenticate with an API key from the environment instead: set `XATA_API_KEY` (keys are created at [https://console.xata.io](https://console.xata.io) or with `xata keys user create`). Prefer a scoped key stored in a secret manager over pasting keys into commands.

### Step 3: Create a project and branch

A **project** is the top-level container; a **branch** is a running Postgres database inside it. Before creating resources, check whether the folder already has a `.xata/` configuration. If so, inspect `xata status` and ask whether to reuse that setup or connect to a different project. `xata init` does not replace an existing project link just because new flags are supplied. Don't delete existing configuration or create another project without agreement.

Ask whether the user is setting up **development and branching** or a **production database**, then recommend settings in plain language:

* For development-only setups, use `--replicas 0`: one primary with no read replicas, reducing compute cost but providing no standby failover target. For production, default to `--replicas 1`: one primary plus one read replica, which provides a standby failover target and adds compute cost.
* For development-only setups, use `--scale-to-zero-base true` so the base branch can sleep when idle. For production, use `--scale-to-zero-base false` to keep it running. Use `--scale-to-zero-child true` for development branches in either case. When scale-to-zero is enabled, the default inactivity interval is **30 minutes**. Explain that sleeping branches save compute but incur wake-up latency.

Inform the user of the proposed region, instance size, replicas, and sleep behavior without requiring them to choose technical flags. For example: "I'll use one primary and one read replica for a standby failover target. The replica adds compute cost. Since this is production, I'll keep the main branch running and let development branches sleep when idle." Obtain approval for billable resources before provisioning.

For a new setup, find or create an organization, then create the project with an initial `main` branch. For development-only use, replace `<replicas>` with `0` and `<base-scale-to-zero>` with `true`; for production, use `1` and `false`, respectively. Because prompts are disabled for agents, pass explicit flags.

The filtered examples below use `jq` to show only resource IDs, names, and status. Check that it is installed (`command -v jq`); ask before installing it if needed. In Bash, enable `pipefail` so filtering does not hide a failed CLI command.

```bash theme={null}
set -o pipefail
xata organization list --json
# if the user has no organization yet:
xata organization create --name "<user or company name>" --json | jq '{id, name}'

xata project create --organization <org-id> --name "<project>" --branch-name main \
  --region <region> --replicas <replicas> --instance-type <instance-type> --postgres-version <version> \
  --scale-to-zero-base <base-scale-to-zero> --scale-to-zero-child true --json | jq '{id, name}'
```

Region, instance type, and Postgres version choices depend on the organization and plan. If a value is rejected, the error says so — ask the user what they want, or have them run `xata project create --name "<project>"` in their own terminal, where the CLI lists the valid options interactively. Explain any change to the proposed settings before proceeding.

Wait for the branch to be ready, then link the user's project folder so later commands don't need IDs. The explicit wait also supports CLI versions that return from `init` before an unhealthy branch is ready:

```bash theme={null}
set -o pipefail
xata branch wait-ready --organization <org-id> --project <project-id> --branch main --wake --json | jq '{id, name, status}'
xata init --organization <org-id> --project <project-id> --branch main --database postgres
```

`xata init` writes the configuration to `.xata/` in the current folder. `--database` selects the Postgres database on the branch; use the existing `postgres` database for onboarding. Verify with `xata status` that the organization, project, and branch match the intended target before changing application configuration or running migrations. A successful exit alone is not proof that the folder was linked to the new project.

CLI 1.14.2 does not show the database name in `xata status`. Inspect only the relevant fields in the local configuration, and check whether `XATA_DATABASE_NAME` overrides the configured database:

```bash theme={null}
jq '{branchName, databaseName}' .xata/branch.json
printf 'Database override: %s\n' "${XATA_DATABASE_NAME:-<not set>}"
```

Configuration identifies the intended target, not whether that database exists; this also applies if your CLI shows a database name in status output. Confirm the actual database with `SELECT current_database();` through the application's driver or, if `psql` is installed, the following command. It captures the URL without printing it and should return `postgres` for this setup. Resolve any mismatch before running migrations:

```bash theme={null}
psql "$(xata branch url)" -Atc 'SELECT current_database();'
```

### Step 4: Connect the user's application

Capture the branch connection string with `$(xata branch url)` and put it wherever the user's app reads its database URL (commonly `DATABASE_URL`). Don't run `xata branch url` on its own in agent tool output: it prints a standard `postgresql://` connection string containing credentials. Before writing it anywhere: check which env file the framework actually loads (`.env.local` for Next.js, `.env` otherwise), make sure that file is gitignored, don't duplicate an existing `DATABASE_URL` entry, and don't echo the URL into the conversation. A safe pattern:

```bash theme={null}
echo "DATABASE_URL=$(xata branch url)" >> .env
```

Because Xata is standard Postgres, the user's existing driver or ORM (Prisma, Drizzle, SQLAlchemy, ActiveRecord, ...) works without changes. Add `--type pooler` for serverless workloads that need pooled connections.

### Step 5: Create the schema and some data

Prefer the project's existing migration tooling (Prisma migrate, Drizzle Kit, etc.) if there is one — point it at the branch and run it. Otherwise apply SQL directly. Check that `psql` is installed (`command -v psql`) before using it; if it's missing, use the ORM or ask before installing anything:

```bash theme={null}
psql "$(xata branch url)" -c "CREATE TABLE items (id SERIAL PRIMARY KEY, name TEXT NOT NULL);"
psql "$(xata branch url)" -c "INSERT INTO items (name) SELECT 'item ' || i FROM generate_series(1, 10) i;"
```

Write the schema the user's project actually needs — don't implement persistence in memory when a real database is one command away. Keep schema migrations in the repo.

Before moving on, verify the connection end to end: run the user's app (or a one-line query through their driver) and confirm it reads from the branch.

### Step 6: Show off branching

This is the part that makes Xata click. Create a development branch as a copy-on-write copy of `main`, including its data. Child branches default to one primary and zero read replicas, even when the parent has replicas. Keep that zero-replica default for development to reduce compute cost; the child has no standby failover target:

```bash theme={null}
set -o pipefail
xata branch create --name dev --parent-branch main --json | jq '{id, name}'
xata checkout dev
xata branch wait-ready dev --wake --json | jq '{id, name, status}'
```

Now CLI commands target `dev`: `xata branch url` prints the `dev` connection string. Note that `xata checkout` changes only the CLI context — the application keeps using whatever `DATABASE_URL` is in its env file until you update it.

Prove the isolation to the user: query `dev` and show the data copied from `main` is already there, insert a row on `dev`, then query `main` and show it is unchanged. Explain the workflow this enables: a branch per feature, per pull request, per teammate, or per coding agent — each isolated, each disposable.

Switch back with `xata checkout main` when done. If the app was pointed at `dev`, update its env file to use `main` before deleting `dev`; checkout alone does not update the application. To clean up the experiment, first get the user's explicit approval to delete `dev`. The CLI cannot delete the currently checked-out branch, and agent/noninteractive mode requires `--yes`. Only after approval, run:

```bash theme={null}
set -o pipefail
xata checkout main
xata branch delete dev --yes --json | jq '{id, name}'
```

`--yes` confirms deletion to the CLI, not on behalf of the user — it does not replace their approval.

### Step 7: The console

Tell the user about [https://console.xata.io](https://console.xata.io), where they can see their branch tree, metrics, query insights, schema, and manage API keys and team members. The CLI and console are two views of the same platform: agents mostly use the CLI, humans often prefer the console.

## Ongoing agent access

Set the user up so their coding agents keep working well with Xata after onboarding:

* **MCP server** — Xata hosts an MCP server at `https://api.xata.tech/mcp` (Streamable HTTP, OAuth or API-key auth). Offer to configure it in the user's client now, for example `claude mcp add --transport http xata https://api.xata.tech/mcp` for Claude Code, or an entry in `.cursor/mcp.json` for Cursor. Per-client instructions: [https://xata.io/docs/platform/mcp](https://xata.io/docs/platform/mcp).
* **Agent authentication** — a machine-readable guide to obtaining Xata API credentials as an agent: [https://xata.io/auth.md](https://xata.io/auth.md).
* **Per-agent branches** — recommend giving every agent task its own branch created from `main` (or an anonymized clone of production) and reviewing changes before they reach `main`. Details: [https://xata.io/docs/ai-agents/overview](https://xata.io/docs/ai-agents/overview).

### Prepare future database tasks

After completing all requested onboarding and migration work, offer to install the isolated-branch workflow in the chosen project. If there is still no project directory, defer this step. If skill setup was deferred earlier, identify the supported agent, confirm the project root, and run the availability checks now. With the user's approval, use the verified agent and project root:

```bash theme={null}
xata skill install branching-postgresql-for-agents \
  --agent <agent-id> \
  --directory "<project-root>" \
  --json
```

This skill is for subsequent database-backed coding tasks. It requires creating and verifying an isolated child branch before database access, prohibits querying the source database even for reads, and does not allow reusing an inherited `DATABASE_URL`. Introduce it at this final handoff, not midway through the fresh-setup demonstration above, which deliberately queries and modifies `main`. If it already applies to the project, follow the ground rules above from the start.

Verify the installed files and read the skill. Preserve existing installations and ask before replacement, just as for the CLI skill. With permission, add a pointer using its actual project-relative skill directory to the project's agent instructions:

```markdown theme={null}
Before database-backed coding tasks, read
<installed branching skill path>/SKILL.md and follow its
isolated-branch workflow. Do not query the source database
or reuse an inherited DATABASE_URL.
```

Include this skill in the same version-control and discovery checks as the CLI skill. Offer `using-xata-api` for direct Xata management-API work and `managing-postgresql` for troubleshooting when relevant; do not install the entire catalog by default. At handoff, tell the user which skills were installed, where they live, and whether a fresh session has verified discovery or still needs a restart or reload.

## Appendix: migrating an existing database

If the user has an existing Postgres database (RDS, Aurora, Cloud SQL, Neon, Supabase, self-hosted, ...), `xata clone` snapshots it into a Xata branch. Treat this as a sensitive operation: get explicit approval before connecting to their database, and never paste its connection string into the conversation — have the user put it in the environment.

```bash theme={null}
export XATA_CLI_SOURCE_POSTGRES_URL="postgresql://user:pass@host:5432/db"   # user provides this out of band
xata clone config --mode auto --validation-mode strict
# Review .xata/clone.yaml and replace sensitive-column noop rules with masking rules.
# Get the user's approval for the reviewed rules and target branch before copying:
xata clone start --validation-mode strict
```

Strict validation requires every table and column to be covered by the rules in `.xata/clone.yaml`; it does **not** guarantee anonymization. Auto-generated `noop` rules copy values unchanged. Identify sensitive columns and replace their `noop` rules with appropriate masking transformations before copying any data. Only use `--validation-mode relaxed` if the user explicitly confirms the data contains nothing sensitive — relaxed mode copies uncovered columns unchanged. Double-check which branch is checked out before starting, so the clone lands where intended.

Provider-specific guides start at [https://xata.io/docs/migrations/aws-rds](https://xata.io/docs/migrations/aws-rds) (sibling pages cover Aurora, Cloud SQL, Azure, Neon, Supabase, and self-hosted). For continuous replication instead of a one-time snapshot, see `xata clone stream`.

## Notes

* It's the coding agent's job to run CLI commands and write schema migrations, not the human user. The human can use [https://console.xata.io](https://console.xata.io) to see everything you did.
* Every page on [https://xata.io](https://xata.io) is available as markdown by appending `.md` to its URL. The docs index for agents is at [https://xata.io/docs/llms.txt](https://xata.io/docs/llms.txt).
* If something fails, `xata status`, `xata auth status`, and `xata branch describe --json | jq '{id, name, status}'` are the fastest ways to diagnose state. Keep `pipefail` enabled and filter branch metadata rather than printing the raw response.


# OpenCode
Source: https://xata.io/docs/ai-agents/opencode

Use OpenCode with standard Postgres URLs and Xata CLI workflows.

# OpenCode with Xata

OpenCode is an open-source terminal coding agent. Because Xata exposes standard Postgres connection strings, OpenCode can work with Xata through the same environment variables, drivers, and CLI commands your application already uses.

## What OpenCode is good for with Xata

* Local terminal loops where you choose the model and provider.
* Database-aware code changes that use standard Postgres tooling.
* Reproducible branch setup in shell scripts.
* Provider-agnostic experimentation against temporary Xata branches.

## Quick start

1. Install and initialize the Xata CLI.

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
xata auth login
xata init
```

2. Create a branch for the OpenCode session.

```bash theme={null}
xata branch create --name opencode-task --parent-branch `xata branch get id`
```

3. Export the Postgres URL.

```bash theme={null}
export DATABASE_URL="$(xata branch url opencode-task)"
```

4. Start OpenCode from the project directory.

```bash theme={null}
opencode
```

## Recommended Xata setup

Keep OpenCode pointed at a single temporary branch. Use the branch connection string for application tests and standard tools like `psql`, migration CLIs, and ORM generators.

## Example prompts

* "Use the database in `DATABASE_URL`; it is a temporary Xata branch."
* "Run the migration and test commands against this branch only."
* "If you need a fresh database state, ask me before deleting or recreating the branch."
* "Use standard Postgres SQL and avoid Xata-specific assumptions unless the docs say otherwise."
* "Summarize all database writes before finishing."

## Common workflows

### Provider-agnostic database testing

1. Create a branch for the task.
2. Export `DATABASE_URL`.
3. Ask OpenCode to implement the change.
4. Run migrations and tests through the project scripts.
5. Delete the branch when the experiment is done.

### SQL inspection

1. Connect with `psql $(xata branch url opencode-task)`.
2. Ask OpenCode to inspect the schema and failing query.
3. Apply fixes in code or migrations.
4. Re-run the query and tests against the same branch.

## Tips

* Keep the session scoped to a single branch.
* Use `xata branch list` and `xata branch describe` when you need branch state.
* Prefer pooled URLs for high-concurrency local test runners.
* Use [API keys](/docs/platform/api-key) only when the agent needs Xata API access.

## Troubleshooting

| Issue | Fix |
| - | - |
| OpenCode cannot find Xata commands | Confirm the Xata CLI install path is in `PATH`. |
| Tests use a cached production URL | Print `DATABASE_URL` in the shell before starting the agent and restart the dev server. |
| The branch gets messy | Delete it with `xata branch delete <name>` and create a fresh branch from the parent. |

## Related Xata docs

* [CLI Reference](/docs/cli)
* [Branch commands](/docs/cli/branch)
* [Postgres extensions](/docs/platform/extensions)
* [Scale to Zero](/docs/core-concepts/scale-to-zero)


# Xata for Agents
Source: https://xata.io/docs/ai-agents/overview

Set up Xata for AI coding agents. Install the CLI, configure MCP, and use agent skills.

# Postgres for AI coding agents

Xata exposes a CLI, a hosted MCP server, and agent skills. AI coding agents can use any of them to create Postgres branches, run migrations, inspect schema, and work against realistic data without writing to production.

## Get set up

New to Xata? Hand the setup to your agent: the [agent onboarding guide](/docs/ai-agents/onboarding) walks it through installing the CLI, creating a project, connecting your app, and trying branching. The [Quickstart](/docs/quickstart) has a ready-made prompt.

### Or set up manually

If you do not have a Xata account yet, [sign up and create a project](/docs/quickstart) first.

Install the Xata CLI, authenticate, and link your local repository:

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
xata auth login
xata init
```

`xata init` writes your organization, project, branch, and database to `.xata/` in the current folder so later CLI commands know which project to use.

Connect the Xata MCP server in your agent client. Most clients only need the server URL; OAuth handles the rest:

```
https://api.xata.tech/mcp
```

Example for Claude Code:

```bash theme={null}
claude mcp add --transport http xata https://api.xata.tech/mcp
```

See [MCP Server](/docs/platform/mcp) for setup instructions for Cursor, VS Code, Codex, OpenCode, Windsurf, Cline, and other clients.

## Choose your integration

> [card] [Xata CLI](/docs/cli)
>
> Branches, auth, migrations, clone, stream, and project linking from the terminal.

> [card] [Xata MCP](/docs/platform/mcp)
>
> Connect through OAuth or an API key. Works with Cursor, Claude Code, VS Code, Codex, OpenCode, Windsurf, Cline, and more.

> [card] [Agent Skills](/docs/ai-agents/agent-skills)
>
> Guided workflows for the CLI, HTTP API, and PostgreSQL operations. Available through the MCP server.

> [card] [Agent guides](/docs/ai-agents/claude-code)
>
> Step-by-step setup for Claude Code, Cursor, Codex, Copilot, and other coding agents with isolated branches.

## When to use each

* **Xata MCP**: preferred for agent-native operations. Connect to `https://api.xata.tech/mcp` with OAuth or an API key. Exposes REST API operations, schema inspection, SQL, documentation search, and skills without shelling out to the CLI for every step.
* **Xata CLI**: preferred when the task depends on local machine state: linking the current directory with `xata init`, creating branches, running migrations with `xata roll`, cloning data with `xata clone`, and scripting in CI.
* **Agent Skills**: use alongside MCP so agents follow Xata-specific procedures for CLI commands, API calls, and PostgreSQL troubleshooting instead of guessing.
* **Agent guides**: use when you want a concrete workflow for a specific tool: branch connection strings, `.env.local` setup, example prompts, and cleanup steps for that agent.

## Working with sensitive data

> [card] [Anonymized production clones](/docs/core-concepts/anonymization)
>
> Give agents realistic schema and data for testing without exposing PII or other sensitive values. Create an anonymized clone as the parent branch before spinning up agent task branches.

## Agent directory

Use the guide for your coding agent to create an isolated branch, pass the branch connection string to the agent, and keep schema or data changes scoped to that branch until you review them.

> [card] [Claude Code](/docs/ai-agents/claude-code)
>
> Terminal-first workflows for branch-scoped migrations, tests, and database inspection.

> [card] [Codex](/docs/ai-agents/codex)
>
> Sandbox-friendly workflows for running tests and edits against a dedicated Xata branch.

> [card] [Cursor](/docs/ai-agents/cursor)
>
> IDE and background-agent setup with `.env.local` connection strings and reviewable schema changes.

> [card] [OpenCode](/docs/ai-agents/opencode)
>
> Provider-agnostic terminal agent workflows using standard Postgres and the Xata CLI.

> [card] [GitHub Copilot](/docs/ai-agents/github-copilot)
>
> VS Code and GitHub workflows for per-PR branches and CI-backed database checks.

> [card] [Windsurf](/docs/ai-agents/windsurf)
>
> Cascade and Flow patterns for repeatable branch creation, migration testing, and cleanup.

> [card] [Cline](/docs/ai-agents/cline)
>
> Approval-based VS Code workflows with safe terminal commands and branch-scoped credentials.

> [card] [Devin](/docs/ai-agents/devin)
>
> Long-running cloud-agent tasks with isolated branches, scoped credentials, and explicit cleanup.

## Recommended baseline

1. Create or choose a parent branch with the schema and data the agent should use. For sensitive data, start from an [anonymized production clone](/docs/core-concepts/anonymization).
2. Create a dedicated branch for the agent task.
3. Give the agent only the branch connection string or a [branch-scoped API key](/docs/platform/api-key).
4. Ask the agent to run migrations, tests, and data changes only against that branch.
5. Review code, SQL, schema diffs, and Xata branch state before merging.
6. Delete the temporary branch when the task is done.

```bash theme={null}
xata branch create --name agent-task --parent-branch `xata branch get id`
xata branch url agent-task
```

## See also

* [Data Anonymization](/docs/core-concepts/anonymization)
* [API Keys](/docs/platform/api-key)
* [Create a branch per pull request](/docs/automations/ga-pr)


# Windsurf
Source: https://xata.io/docs/ai-agents/windsurf

Use Windsurf Cascade and Flows with repeatable Xata branch workflows.

# Windsurf with Xata

Windsurf's Cascade and Flows can handle multi-step coding tasks. Pair them with Xata branches so every flow has a safe Postgres target for migrations, tests, and database inspection.

## What Windsurf is good for with Xata

* Turning repeated database setup into a Flow.
* Making multi-file application and schema changes in one IDE session.
* Running terminal commands while preserving project context.
* Debugging application behavior against a realistic branch.

## Quick start

1. Create a branch for the Windsurf task.

```bash theme={null}
xata branch create --name windsurf-task --parent-branch `xata branch get id`
```

2. Store the branch URL where the app reads local environment variables.

```bash theme={null}
echo "DATABASE_URL=$(xata branch url windsurf-task)" >> .env.local
```

3. Open the project in Windsurf.
4. Ask Cascade to use the local branch for all database commands.

## Recommended Xata setup

Create a Flow for the database setup you repeat often: create branch, print branch URL, run migrations, run tests, and report branch status. Keep destructive operations behind explicit approval.

## Example prompts

* "Create a Flow that prepares a Xata branch, runs migrations, runs tests, and reports results."
* "Use the Xata branch URL in `.env.local`; do not use production credentials."
* "Apply this schema change to the branch and explain the SQL."
* "Debug the failing test by inspecting the branch schema and application logs."
* "Add cleanup instructions for the temporary branch."

## Common workflows

### Flow-based branch setup

1. Create a Flow with Xata CLI setup commands.
2. Add project migration and test commands.
3. Ask Cascade to run the Flow before implementation.
4. Review the Flow output before merging changes.

### Migration testing

1. Create a branch from the current parent.
2. Ask Cascade to apply the migration.
3. Run application tests against the branch.
4. Ask Cascade to summarize changed tables and columns.

## Tips

* Keep the branch name visible in the Flow output.
* Ask Cascade to pause before deleting branches or changing credentials.
* Use anonymized clones for flows that need production-like data.
* Keep `.env.local` uncommitted.

## Troubleshooting

| Issue | Fix |
| - | - |
| Cascade forgets the database target | Put the branch name and environment file path in the Flow description. |
| A Flow runs against stale credentials | Regenerate the branch URL and restart the local dev server. |
| Generated SQL is too broad | Ask Cascade to split schema and data changes into separate reviewed steps. |

## Related Xata docs

* [Instant Branching](/docs/core-concepts/branching)
* [Data Anonymization](/docs/core-concepts/anonymization)
* [Schema Changes](/docs/core-concepts/schema-changes)
* [Branch commands](/docs/cli/branch)


# Bulk delete API Keys for an organization
Source: https://xata.io/docs/api-reference/api-keys/bulk-delete-api-keys-for-an-organization

https://api.xata.tech/openapi.json delete /organizations/{organizationID}/api-keys
Delete multiple API keys in a single request.

# Bulk delete API Keys for the authenticated user
Source: https://xata.io/docs/api-reference/api-keys/bulk-delete-api-keys-for-the-authenticated-user

https://api.xata.tech/openapi.json delete /api-keys
Delete multiple API keys in a single request.

# Create a User API Key
Source: https://xata.io/docs/api-reference/api-keys/create-a-user-api-key

https://api.xata.tech/openapi.json post /api-keys
Create a new API key for the authenticated user.

# Create an Organization API Key
Source: https://xata.io/docs/api-reference/api-keys/create-an-organization-api-key

https://api.xata.tech/openapi.json post /organizations/{organizationID}/api-keys
Create a new API key for an organization.

# List API Keys for an organization
Source: https://xata.io/docs/api-reference/api-keys/list-api-keys-for-an-organization

https://api.xata.tech/openapi.json get /organizations/{organizationID}/api-keys
Retrieve a list of API keys for a specific organization.

# List API Keys for the authenticated user
Source: https://xata.io/docs/api-reference/api-keys/list-api-keys-for-the-authenticated-user

https://api.xata.tech/openapi.json get /api-keys
Retrieve a list of API keys for the authenticated user.

# Create a new branch
Source: https://xata.io/docs/api-reference/branches/create-a-new-branch

https://api.xata.tech/openapi.json post /organizations/{organizationID}/projects/{projectID}/branches
Creates a new branch within the specified project. Branches can be created from scratch or derived from an existing parent branch.

# Delete a branch
Source: https://xata.io/docs/api-reference/branches/delete-a-branch

https://api.xata.tech/openapi.json delete /organizations/{organizationID}/projects/{projectID}/branches/{branchID}
Permanently deletes a specific branch by its ID and all associated data. This action cannot be undone.

# Get branch details
Source: https://xata.io/docs/api-reference/branches/get-branch-details

https://api.xata.tech/openapi.json get /organizations/{organizationID}/projects/{projectID}/branches/{branchID}
Retrieves detailed information about a specific branch by its ID, including status, connection string, and configuration.

# Get PostgreSQL configuration details
Source: https://xata.io/docs/api-reference/branches/get-postgresql-configuration-details

https://api.xata.tech/openapi.json get /organizations/{organizationID}/projects/{projectID}/branches/{branchID}/postgres-config
Retrieves detailed information about the current PostgreSQL configuration parameters for a branch, including parameter types, descriptions, acceptable ranges, default values, and current values.

# List all branches
Source: https://xata.io/docs/api-reference/branches/list-all-branches

https://api.xata.tech/openapi.json get /organizations/{organizationID}/projects/{projectID}/branches
Retrieves a list of all branches within the specified project, including their metadata, region, and access settings.

# Retrieve branch credentials
Source: https://xata.io/docs/api-reference/branches/retrieve-branch-credentials

https://api.xata.tech/openapi.json get /organizations/{organizationID}/projects/{projectID}/branches/{branchID}/credentials

# Retrieve branch logs
Source: https://xata.io/docs/api-reference/branches/retrieve-branch-logs

https://api.xata.tech/openapi.json post /organizations/{organizationID}/projects/{projectID}/branches/{branchID}/logs

# Retrieve branch metrics
Source: https://xata.io/docs/api-reference/branches/retrieve-branch-metrics

https://api.xata.tech/openapi.json post /organizations/{organizationID}/projects/{projectID}/branches/{branchID}/metrics
Returns time-series data for one or more metrics of a branch.

# Rotate branch credentials
Source: https://xata.io/docs/api-reference/branches/rotate-branch-credentials

https://api.xata.tech/openapi.json post /organizations/{organizationID}/projects/{projectID}/branches/{branchID}/credentials/rotate
Triggers a credential rotation for the branch database user. The new credentials can be retrieved via the credentials endpoint after rotation completes.

# Update branch details
Source: https://xata.io/docs/api-reference/branches/update-branch-details

https://api.xata.tech/openapi.json patch /organizations/{organizationID}/projects/{projectID}/branches/{branchID}
Updates the configuration of a specific branch by its ID, such as changing its name, description, or resource allocation.

# Execute SQL query
Source: https://xata.io/docs/api-reference/gateway/execute-sql-query

https://api.xata.tech/openapi.json post /sql
Execute a single SQL query or a batch of queries against a PostgreSQL branch.

**Authentication:** send the branch's PostgreSQL connection string in the
`Connection-String` header. The control-plane API key (Bearer token) is
**not** accepted on the gateway host.

**Routing:** the target branch, region, and endpoint type are taken from the
hostname embedded in the connection string, which must match the request host.
See the `Connection-String` security scheme for the host format.

**Single query:** provide `query` (and optional `params`) at the top level.

**Batch:** provide `queries` as an array of query objects, or send the request body
as a JSON array. Batch queries execute within a single transaction.

**Errors:** PostgreSQL errors return 400 with their SQLSTATE in `code` and their
diagnostic fields. A 503 means the gateway could not reach the database: `08001`
when a connection cannot be established and `08006` when an established connection
fails. A 503 does not guarantee that replaying a write is safe: a connection failure
can leave its outcome unknown.


# WebSocket wire protocol proxy
Source: https://xata.io/docs/api-reference/gateway/websocket-wire-protocol-proxy

https://api.xata.tech/openapi.json get /v2
Upgrade to a WebSocket connection that proxies the PostgreSQL wire protocol.
The client sends and receives PostgreSQL protocol messages over binary WebSocket frames.

**Authentication:** the same branch connection string credential as the
HTTP SQL endpoint is used, but it is supplied via the PostgreSQL startup
message inside the wire protocol rather than an HTTP header. The
control-plane API key (Bearer token) is not accepted.


# Send an MCP message
Source: https://xata.io/docs/api-reference/mcp/send-an-mcp-message

https://api.xata.tech/openapi.json post /mcp
Handle a single JSON-RPC 2.0 message from an MCP client, per the MCP
[Streamable HTTP transport](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports).

The server is stateless: every request is self-contained and carries the caller's
credential, so there is no session to establish or resume. `GET` and `DELETE` on this
path return `405 Method Not Allowed`.

**Requests** (messages with an `id`) are answered with the JSON-RPC response, either as
`application/json` or as a `text/event-stream` carrying the response and any notifications
emitted while it runs. Requests must accept both. **Notifications and responses**
(messages without an `id`) are answered with `202 Accepted` and an empty body.


# Check the DNS record for a claimed domain
Source: https://xata.io/docs/api-reference/organizations/check-the-dns-record-for-a-claimed-domain

https://api.xata.tech/openapi.json post /organizations/{organizationID}/sso/domains/{domain}/verify
Looks up the challenge TXT record. Verification is a poll, not a command: a missing or stale record is reported
as `verified: false` on a 200 rather than as an error, so a client can retry while DNS propagates. Verifying a
domain that lost its verification restores its provider, with enforcement off.


# Claim an email domain for SSO
Source: https://xata.io/docs/api-reference/organizations/claim-an-email-domain-for-sso

https://api.xata.tech/openapi.json post /organizations/{organizationID}/sso/domains
Claims an email domain and returns the DNS record that proves control of it. A claim is only a note to
ourselves: nothing is registered against the domain until verification succeeds, so claiming a domain neither
affects anyone's sign-in nor stops another organization claiming it first. A domain another organization has
already registered is refused with 409, before any DNS record is published.


# Create a new organization
Source: https://xata.io/docs/api-reference/organizations/create-a-new-organization

https://api.xata.tech/openapi.json post /organizations
Creates a new organization with the authenticated user as its single owner.

# Delete an invitation
Source: https://xata.io/docs/api-reference/organizations/delete-an-invitation

https://api.xata.tech/openapi.json delete /organizations/{organizationID}/invitations/{invitationID}
Permanently delete an invitation record.

# Delete an organization
Source: https://xata.io/docs/api-reference/organizations/delete-an-organization

https://api.xata.tech/openapi.json delete /organizations/{organizationID}
[Not implemented yet] Permanently delete an organization. This action cannot be undone.

# Get details of a specific invitation
Source: https://xata.io/docs/api-reference/organizations/get-details-of-a-specific-invitation

https://api.xata.tech/openapi.json get /organizations/{organizationID}/invitations/{invitationID}
Retrieve detailed information about a specific invitation by its ID.

# Get list of organizations
Source: https://xata.io/docs/api-reference/organizations/get-list-of-organizations

https://api.xata.tech/openapi.json get /organizations
Retrieve the list of all organizations the authenticated user belongs to, including their IDs and names. This endpoint allows users to view all organizations they have access to.

# Get organization details
Source: https://xata.io/docs/api-reference/organizations/get-organization-details

https://api.xata.tech/openapi.json get /organizations/{organizationID}
Retrieve detailed information about a specific organization by its ID.

# Get organization membership limits
Source: https://xata.io/docs/api-reference/organizations/get-organization-membership-limits

https://api.xata.tech/openapi.json get /organizations/{organizationID}/membership-limits
Retrieves the membership limits for the specified organization (maximum members and invitations).

# Get the organization's SSO configuration
Source: https://xata.io/docs/api-reference/organizations/get-the-organizations-sso-configuration

https://api.xata.tech/openapi.json get /organizations/{organizationID}/sso
Lists the organization's identity providers and the email domains claimed for them. An organization signs in
through one provider per verified domain, so both are collections. Client secrets are write-only and are never
returned. An organization that has not set anything up yet responds 200 with empty lists.


# List invitations for an organization
Source: https://xata.io/docs/api-reference/organizations/list-invitations-for-an-organization

https://api.xata.tech/openapi.json get /organizations/{organizationID}/invitations
Retrieve all invitations for an organization with optional filtering by status, email, and name.

# List members of an organization
Source: https://xata.io/docs/api-reference/organizations/list-members-of-an-organization

https://api.xata.tech/openapi.json get /organizations/{organizationID}/members

# Register an identity provider for a verified domain
Source: https://xata.io/docs/api-reference/organizations/register-an-identity-provider-for-a-verified-domain

https://api.xata.tech/openapi.json post /organizations/{organizationID}/sso/providers
Registers an identity provider for one verified domain. `google` needs only the credentials and pins the login
to that Google Workspace domain; `microsoft` pins it to the Entra tenant in the issuer; `oidc` is the fallback
for any other provider and discovers its endpoints from the issuer's `/.well-known/openid-configuration`. The
domain must already be verified, and registering does not redirect anyone on its own: that is enabled
separately.


# Release a claimed or verified domain
Source: https://xata.io/docs/api-reference/organizations/release-a-claimed-or-verified-domain

https://api.xata.tech/openapi.json delete /organizations/{organizationID}/sso/domains/{domain}
Drops a pending claim, or releases a verified domain so members on it sign in however they could before.
Refused with a 409 while the organization has a provider for the domain; remove the provider first. Idempotent.


# Remove a member from an organization
Source: https://xata.io/docs/api-reference/organizations/remove-a-member-from-an-organization

https://api.xata.tech/openapi.json delete /organizations/{organizationID}/members/{userID}

# Remove an identity provider
Source: https://xata.io/docs/api-reference/organizations/remove-an-identity-provider

https://api.xata.tech/openapi.json delete /organizations/{organizationID}/sso/providers/{providerAlias}
Deletes the provider, so members on its domain fall back to the sign-in methods they had before. The domain
stays verified and can be given another provider. Idempotent.


# Request organization deletion
Source: https://xata.io/docs/api-reference/organizations/request-organization-deletion

https://api.xata.tech/openapi.json post /organizations/{organizationID}/deletion-request
Flags the organization for deletion. The request is rejected if the organization still has active projects.
The subscription is cancelled immediately and any outstanding invoices are collected automatically after a grace period that prevents missing delayed usage reports.
Deletion completes asynchronously once all invoices are issued and settled.


# Require members on this provider's domain to sign in through it
Source: https://xata.io/docs/api-reference/organizations/require-members-on-this-providers-domain-to-sign-in-through-it

https://api.xata.tech/openapi.json put /organizations/{organizationID}/sso/providers/{providerAlias}/enforcement
Enabling sends every address on the provider's domain to it, leaving those members no other way in. Refused
with a 409 until the domain is verified. Disabling needs nothing, so the way out is always open.


# Resend an invitation
Source: https://xata.io/docs/api-reference/organizations/resend-an-invitation

https://api.xata.tech/openapi.json post /organizations/{organizationID}/invitations/{invitationID}/resend
Resend a pending invitation with a fresh expiration time.

# Send an invitation to join an organization
Source: https://xata.io/docs/api-reference/organizations/send-an-invitation-to-join-an-organization

https://api.xata.tech/openapi.json post /organizations/{organizationID}/invitations
Send an invitation email to a user to join an organization.

# Update an identity provider's credentials
Source: https://xata.io/docs/api-reference/organizations/update-an-identity-providers-credentials

https://api.xata.tech/openapi.json put /organizations/{organizationID}/sso/providers/{providerAlias}
Replaces the provider's credentials, and for `oidc` re-discovers its endpoints from the issuer. The domain it
serves and whether it is enforced are carried across unchanged, so rotating a client secret does not quietly
turn enforcement off. A provider whose domain stopped being verified is refused until the domain verifies again.


# Update organization details
Source: https://xata.io/docs/api-reference/organizations/update-organization-details

https://api.xata.tech/openapi.json put /organizations/{organizationID}
Update information for an existing organization, such as its name.

# Create a new project
Source: https://xata.io/docs/api-reference/projects/create-a-new-project

https://api.xata.tech/openapi.json post /organizations/{organizationID}/projects
Creates a new project within the specified organization. Projects are containers for database branches and resources.

# Delete a project
Source: https://xata.io/docs/api-reference/projects/delete-a-project

https://api.xata.tech/openapi.json delete /organizations/{organizationID}/projects/{projectID}
Permanently deletes a specific project by its ID and all associated resources including branches and databases. This action cannot be undone.

# Get available images
Source: https://xata.io/docs/api-reference/projects/get-available-images

https://api.xata.tech/openapi.json get /organizations/{organizationID}/images
Retrieves a list of all images for the specified organization and region

# Get available instance types
Source: https://xata.io/docs/api-reference/projects/get-available-instance-types

https://api.xata.tech/openapi.json get /organizations/{organizationID}/instanceTypes
Retrieves a list of all instance types for the specified organization and region

# Get available regions
Source: https://xata.io/docs/api-reference/projects/get-available-regions

https://api.xata.tech/openapi.json get /organizations/{organizationID}/regions
Retrieves a list of all regions where new branches can be deployed for the specified organization.

# Get organization resource limits
Source: https://xata.io/docs/api-reference/projects/get-organization-resource-limits

https://api.xata.tech/openapi.json get /organizations/{organizationID}/limits
Retrieves the effective org-level limits, including project creation limits and branch configuration defaults. Project-specific overrides are not applied here.

# Get project details
Source: https://xata.io/docs/api-reference/projects/get-project-details

https://api.xata.tech/openapi.json get /organizations/{organizationID}/projects/{projectID}
Retrieves detailed information about a specific project by its ID within the specified organization.

# Get project resource limits
Source: https://xata.io/docs/api-reference/projects/get-project-resource-limits

https://api.xata.tech/openapi.json get /organizations/{organizationID}/projects/{projectID}/limits
Retrieves the effective resource limits for the specified project, merging organization-level overrides with any project-specific overrides.

# List all projects
Source: https://xata.io/docs/api-reference/projects/list-all-projects

https://api.xata.tech/openapi.json get /organizations/{organizationID}/projects
Retrieves a list of all projects within the specified organization that the authenticated user has access to.

# Update project details
Source: https://xata.io/docs/api-reference/projects/update-project-details

https://api.xata.tech/openapi.json patch /organizations/{organizationID}/projects/{projectID}
Updates the details of a specific project by its ID, such as changing its name.

# Merge Readiness
Source: https://xata.io/docs/automations/ga-check

GitHub Actions workflow for validating main branch status and migration readiness

This workflow validates that the main branch is in a ready state for merging pull requests. It checks the migration status and ensures no pending migrations are running that could affect the merge process.

## Workflow configuration

```yaml theme={null}
name: Check merge readiness

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  check:
    runs-on: ubuntu-latest
    name: Xata dev branch
    steps:
      - uses: actions/checkout@v3

      - name: Install Xata CLI
        run: |
          curl -fsSL https://xata.io/install.sh | bash

      - name: Check Xata status
        run: |
          xata status
          xata roll status
          xata branch view

      - name: Check Xata main branch pgroll status
        id: xata-main-branch-roll-status
        run: |
          xata roll status | jq -r '.status'
          echo XATA_MAIN_BRANCH_ROLL_STATUS=`xata roll status | jq -r '.status'` >> $GITHUB_OUTPUT

      - name: Complete any pending migrations (inherited from main)
        if: ${{ steps.xata-main-branch-roll-status.outputs.XATA_MAIN_BRANCH_ROLL_STATUS == 'In progress' }}
        run: |
          echo "Warning: A migration is currently running on the main branch. Please wait for it to complete before merging this PR."
          exit 1
```

## Authentication and environment variables

The workflow requires several environment variables to be set:

* `XATA_API_KEY`: Your Xata API key (set as a GitHub secret)
* `XATA_ORGANIZATION_ID`: Your Xata organization ID
* `XATA_PROJECT_ID`: Your Xata project ID
* `XATA_BRANCH_ID`: The ID of the main branch
* `XATA_BRANCH_NAME`: The name of the main branch
* `XATA_DATABASE_NAME`: Your database name

If you have the Xata CLI configured with your project locally, you can obtain these values with the following commands:

```sh theme={null}
# XATA_API_KEY
xata keys organization create

# XATA_ORGANIZATION_ID
xata organization get id

# XATA_PROJECT_ID
xata project get id

# XATA_BRANCH_ID
xata branch get id
```

## Workflow steps

### 1. Install Xata CLI

```yaml theme={null}
- name: Install Xata CLI
  run: |
    curl -fsSL https://xata.io/install.sh | bash
```

This step installs the Xata CLI using the official installer script and adds it to the GitHub Actions PATH.

### 2. Check Xata status

```yaml theme={null}
- name: Check Xata status
  run: |
    xata status
    xata roll status
    xata branch view
```

This step performs three checks:

* Verifies CLI and connection status
* Checks for any pending migrations
* Views the current branch status

## Concurrency control

The workflow uses GitHub Actions concurrency to prevent multiple runs from interfering with each other:

```yaml theme={null}
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
```

This ensures that:

* Only one workflow run per branch is active at a time
* New runs cancel any in-progress runs
* Prevents race conditions during status checks

## Best practices

1. **Set appropriate timeout**
   * The workflow has a 10-minute timeout
   * Adjust based on your maximum migration duration

2. **Use environment variables**
   * Store sensitive information in GitHub secrets
   * Use environment variables for configuration

3. **Handle migration states**
   * Check for in-progress migrations
   * Prevent merging during active migrations
   * Provide clear error messages

## Related workflows

* [Clone workflow](./ga-clone) - For cloning databases in CI/CD
* [PR workflow](./ga-pr) - For managing pull request environments


# Bulk Delete Branches
Source: https://xata.io/docs/automations/ga-cleanup

GitHub Actions workflow for bulk deleting Xata branches with the Xata CLI, jq, and bash

This workflow shows a simple bulk-delete pattern using `xata branch list --json`, `jq`, and `bash`. The example deletes stale `gh-readonly-queue*` branches older than 1 hour, but the same pattern works for any bulk-delete use case — for example, cleaning up branches by naming convention, age, region, or any other attribute exposed by the Xata CLI.

The pattern in this guide is intentionally minimal. It does not require TypeScript, the Xata SDK, or any custom scripts — only the Xata CLI, `jq`, and a short bash loop in a scheduled GitHub Actions workflow.

## Workflow configuration

```yaml theme={null}
name: Bulk delete Xata branches

on:
  schedule:
    - cron: "0 * * * *" # Run every hour
  workflow_dispatch:

permissions:
  contents: read

jobs:
  cleanup:
    name: Delete stale Xata branches
    runs-on: ubuntu-latest
    env:
      XATA_API_KEY: ${{ secrets.XATA_API_KEY }}
      XATA_ORGANIZATION_ID: ${{ vars.XATA_ORGANIZATION_ID }}
      XATA_PROJECT_ID: ${{ vars.XATA_PROJECT_ID }}

    steps:
      - name: Install Xata CLI
        shell: bash
        run: |
          curl -fsSL https://xata.io/install.sh | bash
          echo "/home/runner/.config/xata/bin" >> "$GITHUB_PATH"

      - name: Delete stale Xata branches
        shell: bash
        run: |
          set -euo pipefail

          xata branch list \
            --organization "$XATA_ORGANIZATION_ID" \
            --project "$XATA_PROJECT_ID" \
            --json \
            | jq -r '
                .[]
                | select(.name | startswith("gh-readonly-queue"))
                | select((.createdAt | fromdateiso8601) < (now - 3600))
                | .name
              ' \
            | while IFS= read -r branch; do
                [ -z "$branch" ] && continue
                echo "Deleting Xata branch: $branch"
                xata branch delete \
                  --organization "$XATA_ORGANIZATION_ID" \
                  --project "$XATA_PROJECT_ID" \
                  --yes \
                  "$branch"
              done
```

## Authentication and environment variables

The workflow requires the following environment variables:

* `XATA_API_KEY`: Your Xata API key (set as a GitHub secret).
* `XATA_ORGANIZATION_ID`: Your Xata organization ID.
* `XATA_PROJECT_ID`: Your Xata project ID.

If you have the Xata CLI configured with your project locally, you can obtain these values with:

```sh theme={null}
# XATA_API_KEY
xata keys organization create

# XATA_ORGANIZATION_ID
xata organization get id

# XATA_PROJECT_ID
xata project get id
```

## Workflow steps

### 1. Install the Xata CLI

```yaml theme={null}
- name: Install Xata CLI
  shell: bash
  run: |
    curl -fsSL https://xata.io/install.sh | bash
    echo "/home/runner/.config/xata/bin" >> "$GITHUB_PATH"
```

This step installs the Xata CLI using the official installer script and adds it to the GitHub Actions `PATH`. `jq` is preinstalled on `ubuntu-latest` runners and does not require a separate install step.

### 2. List branches as JSON

```bash theme={null}
xata branch list \
  --organization "$XATA_ORGANIZATION_ID" \
  --project "$XATA_PROJECT_ID" \
  --json
```

The `--json` flag returns the full branch list as a JSON array, including `name`, `id`, `createdAt`, and other metadata. This output is the input to the filter pipeline.

### 3. Filter with jq

```jq theme={null}
.[]
| select(.name | startswith("gh-readonly-queue"))
| select((.createdAt | fromdateiso8601) < (now - 3600))
| .name
```

This filter keeps only branches that:

* Have a name starting with `gh-readonly-queue`.
* Were created more than 1 hour ago (`now - 3600` seconds).

It then emits just the branch name on stdout, one per line.

### 4. Delete with a bash loop

```bash theme={null}
while IFS= read -r branch; do
  [ -z "$branch" ] && continue
  echo "Deleting Xata branch: $branch"
  xata branch delete \
    --organization "$XATA_ORGANIZATION_ID" \
    --project "$XATA_PROJECT_ID" \
    --yes \
    "$branch"
done
```

The loop reads each branch name from stdin and runs `xata branch delete --yes` against it. The `--yes` flag is required in non-interactive environments such as GitHub Actions to skip the confirmation prompt.

## Customizing the filter

The same pipeline works for any bulk-delete rule. Replace the `jq` filter with whatever conditions match your use case:

```bash theme={null}
xata branch list --json \
  | jq -r '
      .[]
      | select(<your jq conditions>)
      | .name
    ' \
  | while IFS= read -r branch; do
      [ -z "$branch" ] && continue
      xata branch delete --yes "$branch"
    done
```

Common variations:

* **Match a different naming convention.** Replace `startswith("gh-readonly-queue")` with `startswith("preview-")`, `test("^pr-[0-9]+$")`, or any regex that matches your branch names.
* **Change the age threshold.** Replace `(now - 3600)` with `(now - 86400)` for 1 day, `(now - 604800)` for 1 week, and so on.
* **Exclude protected branches.** Add `select(.name != "main" and .name != "staging")` to make sure important branches are never matched.
* **Combine conditions.** Chain multiple `select(...)` filters to express more specific rules.

## Best practices

1. **Start with a narrow filter.** Begin with a specific name prefix and an age threshold, then broaden the rule once you are confident.
2. **Protect important branches explicitly.** Add `select(.name != "main" and .name != "staging")` to your filter when running broader cleanups.
3. **Dry-run before deleting.** Replace the `xata branch delete` call with `echo "$branch"` and run the workflow manually with `workflow_dispatch` to preview which branches would be deleted.
4. **Schedule conservatively.** Hourly is appropriate for short-lived branches such as merge-queue branches; daily or weekly is usually enough for most cleanup rules.

## When to use

Use this workflow when you want to:

* Automatically remove short-lived branches such as merge-queue or preview branches.
* Enforce branch retention policies based on age or naming.
* Bulk-delete branches matching any rule expressible with the Xata CLI output and `jq`.

For more advanced cleanup rules, you can extend this pattern by fetching external state (for example, the list of open GitHub branches via `gh api`) and deleting Xata branches that no longer have a corresponding upstream branch.

## Related workflows

* [Merge Readiness](./ga-check) — For validating main branch status.
* [Create Branch](./ga-pr) — For managing pull request branches.
* [Branch CLI reference](../cli/branch) — Full reference for `xata branch list` and `xata branch delete`.


# Scheduled Clone
Source: https://xata.io/docs/automations/ga-clone

GitHub Actions workflow for cloning databases on a schedule or on-demand

This workflow helps you clone your database on a schedule or manually trigger it. It's useful for creating regular backups or setting up test environments.

## Workflow configuration

```yaml theme={null}
name: Xata clone (daily, on-demand)

on:
  schedule:
    - cron: "0 0 * * *" # Run daily at midnight
  workflow_dispatch: # Allow manual triggers

env:
  XATA_API_KEY: ${{ secrets.XATA_API_KEY }}
  XATA_ORGANIZATION_ID: ${{ secrets.XATA_ORGANIZATION_ID }}
  XATA_PROJECT_ID: ${{ secrets.XATA_PROJECT_ID }}
  XATA_BRANCH_ID: ${{ secrets.XATA_BRANCH_ID }}
  XATA_BRANCH_NAME: ${{ secrets.XATA_BRANCH_NAME }}
  XATA_DATABASE_NAME: ${{ secrets.XATA_DATABASE_NAME }}
  XATA_CLI_SOURCE_POSTGRES_URL: ${{ secrets.XATA_CLI_SOURCE_POSTGRES_URL }}

jobs:
  clone:
    name: Xata Clone
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Xata CLI
        run: |
          curl -fsSL https://xata.io/install.sh | bash
          echo "/home/runner/.config/xata/bin" >> $GITHUB_PATH

      - name: Check Xata status
        run: |
          xata status
          xata branch describe

      - name: Xata clone
        run: |
          xata clone start --source-url ${{ env.XATA_CLI_SOURCE_POSTGRES_URL }}
```

## Authentication and environment variables

The workflow requires several environment variables to be set:

* `XATA_API_KEY`: Your Xata API key (set as a GitHub secret)
* `XATA_ORGANIZATION_ID`: Your Xata organization ID
* `XATA_PROJECT_ID`: Your Xata project ID
* `XATA_BRANCH_ID`: The ID of the main branch
* `XATA_BRANCH_NAME`: The name of the main branch
* `XATA_DATABASE_NAME`: Your database name
* `XATA_CLI_SOURCE_POSTGRES_URL`: The source PostgreSQL database URL (set as a GitHub secret)

If you have the Xata CLI configured with your project locally, you can obtain these values with the following commands:

```sh theme={null}
# XATA_API_KEY
xata keys organization create

# XATA_ORGANIZATION_ID
xata organization get id

# XATA_PROJECT_ID
xata project get id

# XATA_BRANCH_ID
xata branch get id
```

## Workflow steps

### 1. Checkout code

```yaml theme={null}
- uses: actions/checkout@v3
  with:
    fetch-depth: 0
```

This step checks out your repository code with full history and proper credentials configuration.

### 2. Install Xata CLI

```yaml theme={null}
- name: Install Xata CLI
  run: |
    curl -fsSL https://xata.io/install.sh | bash
```

This step installs the Xata CLI using the official installer script and adds it to the GitHub Actions PATH.

### 3. Check Xata status

```yaml theme={null}
- name: Check Xata status
  run: |
    xata status
    xata branch describe
```

This step verifies that your Xata CLI is properly configured and can connect to your database.

For more details, see:

* [status command documentation](../cli/status)
* [branch describe command documentation](../cli/branch#describe)

### 4. Checkout clone branch

```yaml theme={null}
- name: Checkout Xata dev branch
  run: |
    echo XATA_BRANCH_ID=`xata branch get clone_demo id` >> $GITHUB_ENV
```

This step:

* Checks out the clone\_demo branch
* Extracts the branch ID using `jq`
* Stores it in an environment variable

### 5. Start clone process

```yaml theme={null}
- name: Xata clone
  run: |
    xata clone start --source-url ${{ env.XATA_CLI_SOURCE_POSTGRES_URL }}
```

This step initiates the cloning process from your source PostgreSQL database.

For more details, see the [clone start command documentation](../cli/clone#start).

## Triggering the workflow

The workflow can be triggered in two ways:

1. **Scheduled run**

   ```yaml theme={null}
   schedule:
     - cron: "0 8 * * *"
   ```

   Runs daily at 8:00 AM UTC

2. **Manual trigger**
   ```yaml theme={null}
   workflow_dispatch:
   ```
   Can be triggered manually from the GitHub Actions UI

## Concurrency control

The workflow uses GitHub Actions concurrency to prevent multiple runs from interfering with each other:

```yaml theme={null}
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
```

This ensures that:

* Only one workflow run is active at a time
* New runs cancel any in-progress runs
* Prevents race conditions during cloning

## Best practices

1. **Set appropriate timeout**

   * The workflow has a 10-minute timeout
   * Adjust based on your database size and cloning duration

2. **Use environment variables**

   * Store sensitive information in GitHub secrets
   * Use environment variables for configuration

3. **Schedule regular clones**
   * Set up a schedule that matches your backup needs
   * Consider your database size and update frequency

## Related workflows

* [Check workflow](./ga-check) - For validating main branch status
* [PR workflow](./ga-pr) - For managing pull request environments


# Create Branch
Source: https://xata.io/docs/automations/ga-pr

GitHub Actions workflow for creating and managing development branches for pull requests

This workflow creates a dedicated Xata branch for each pull request, allowing you to test database changes in isolation. It also manages PR comments to provide branch information and status updates.

## Workflow Configuration

```yaml theme={null}
name: Create Xata dev branch

on:
  pull_request:
    ignore-branches:
      - main

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

env:
  FORCE_COLOR: 3
  XATA_API_KEY: ${{ secrets.XATA_API_KEY }}
  XATA_ORGANIZATION_ID: 123xyz
  XATA_PROJECT_ID: prj_i3qsq56h3p3d3crnk7htnc75d0
  XATA_BRANCH_ID: s99l2grl6d3nr1cqcbaaa5piic
  XATA_BRANCH_NAME: main
  XATA_DATABASE_NAME: app

permissions:
  id-token: write
  contents: write
  packages: write
  pages: write
  pull-requests: write
```

## Authentication and Environment Variables

The workflow requires several environment variables to be set:

* `XATA_API_KEY`: Your Xata API key (set as a GitHub secret)
* `XATA_ORGANIZATION_ID`: Your Xata organization ID
* `XATA_PROJECT_ID`: Your Xata project ID
* `XATA_BRANCH_ID`: The ID of the main branch
* `XATA_BRANCH_NAME`: The name of the main branch
* `XATA_DATABASE_NAME`: Your database name

If you have the Xata CLI configured with your project locally, you can obtain these values with the following commands:

```sh theme={null}
# XATA_API_KEY
xata keys organization create

# XATA_ORGANIZATION_ID
xata organization get id

# XATA_PROJECT_ID
xata project get id

# XATA_BRANCH_ID
xata branch get id
```

## Required Permissions

The workflow requires the following GitHub permissions:

```yaml theme={null}
permissions:
  id-token: write
  contents: write
  packages: write
  pages: write
  pull-requests: write
```

## Workflow Steps

### 1. Find Previous Comment

```yaml theme={null}
- name: Find Previous Comment
  id: find-comment
  uses: peter-evans/find-comment@v1
  with:
    token: ${{ secrets.GITHUB_TOKEN }}
    issue-number: ${{ github.event.pull_request.number }}
    comment-author: "github-actions[bot]"
    body-includes: "Xata Dev Branch"
```

This step searches for any existing comments from the workflow to update them later.

### 2. Checkout Code

```yaml theme={null}
- uses: actions/checkout@v4
  with:
    fetch-depth: 0
    persist-credentials: false
```

This step checks out your repository code with full history and proper credentials configuration.

### 3. Install Dependencies

```yaml theme={null}
- name: Install pnpm
  uses: pnpm/action-setup@v2
  with:
    version: 10

- name: Install Xata CLI
  run: |
    curl -fsSL https://xata.io/install.sh | bash
    echo "/home/runner/.config/xata/bin" >> $GITHUB_PATH
```

These steps install the required dependencies:

* pnpm package manager
* Xata CLI

### 4. Check Status and Migration State

```yaml theme={null}
- name: Check Xata Status
  run: |
    xata status
    xata roll status
    xata branch view

- name: Check Xata main branch pgroll status
  id: xata-main-branch-roll-status
  run: |
    echo XATA_MAIN_BRANCH_ROLL_STATUS=`xata roll status | jq -r '.status'` >> $GITHUB_OUTPUT
```

These steps verify the current state of your Xata environment and check for any pending migrations.

### 5. Create or Update Comment

```yaml theme={null}
- name: Create or Update Comment
  id: create_comment
  uses: peter-evans/create-or-update-comment@v1
  with:
    token: ${{ secrets.GITHUB_TOKEN }}
    comment-id: ${{ steps.find-comment.outputs.comment-id }}
    issue-number: ${{ github.event.pull_request.number }}
    edit-mode: replace
    body: |
      ### Xata Dev Branch

      Creating Xata Dev Branch for commit: ${{ github.sha }}
      Branch name: ${{ github.head_ref }}_${{ github.event.pull_request.number }}

      *Please wait for the comment to be updated with branch details.*
```

### 6. Branch Management

```yaml theme={null}
- name: Delete Any Existing Xata Dev Branch
  run: xata branch delete ${{ github.head_ref }}_${{ github.event.pull_request.number }} --yes
  continue-on-error: true

- name: Create Xata Dev Branch
  run: xata branch create --name ${{ github.head_ref }}_${{ github.event.pull_request.number }}

- name: Checkout Xata Dev Branch
  run: |
    echo XATA_BRANCH_ID=`xata branch get ${{ github.head_ref }}_${{ github.event.pull_request.number }} id` >> $GITHUB_ENV
```

### 7. Migration Handling

```yaml theme={null}
- name: Complete Any Pending Migrations
  if: ${{ steps.xata-main-branch-roll-status.outputs.XATA_MAIN_BRANCH_ROLL_STATUS == 'In progress' }}
  run: |
    xata roll complete

- name: Apply Xata Roll Migrations
  run: xata roll migrate
```

### 8. Environment Setup

```yaml theme={null}
- name: Get Latest Schema
  id: xata-roll-get-latest-schema
  run: |
    echo LATEST_SCHEMA=`xata roll latest --with-schema=true` >> $GITHUB_OUTPUT

- name: Setup New Preview Environment
  run: |
    echo DATABASE_URL=`xata branch url` >> $GITHUB_ENV
    echo SEARCH_PATH=${{ steps.xata-roll-get-latest-schema.outputs.LATEST_SCHEMA }} >> $GITHUB_ENV
```

### 9. Final Comment Update

```yaml theme={null}
- name: Update Comment with Branch Details
  uses: peter-evans/create-or-update-comment@v1
  with:
    token: ${{ secrets.GITHUB_TOKEN }}
    comment-id: ${{ steps.find-comment.outputs.comment-id }}
    issue-number: ${{ github.event.pull_request.number }}
    edit-mode: replace
    body: |
      ### Xata Dev Branch

      Created Xata Dev Branch for commit: ${{ github.sha }}
      Branch name: ${{ github.head_ref }}_${{ github.event.pull_request.number }}

      Set the following schemas as the search_path:
          ${{ steps.xata-roll-get-latest-schema.outputs.LATEST_SCHEMA }}

      You can drop into the psql shell for this branch with:
          xata branch checkout ${{ github.head_ref }}_${{ github.event.pull_request.number }}
          psql `xata branch url`

      Branch URL: [View in Console](https://console.xata.io/organizations/123xyz/projects/prj_i3qsq56h3p3d3crnk7htnc75d0/branches/s99l2grl6d3nr1cqcbaaa5piic)
```

## Concurrency Control

The workflow uses GitHub Actions concurrency to prevent multiple runs from interfering with each other:

```yaml theme={null}
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
```

This ensures that:

* Only one workflow run per PR is active at a time
* New runs cancel any in-progress runs
* Prevents race conditions during branch creation

## When to Use

Use this workflow when you want to:

* Create isolated database environments for PRs
* Test database changes in isolation
* Provide preview environments for PR reviewers
* Automate database branch management

## Best Practices

1. **Set Appropriate Timeout**

   * The workflow has a 10-minute timeout
   * Adjust based on your typical migration duration

2. **Use Environment Variables**

   * Store sensitive information in GitHub secrets
   * Use environment variables for configuration

3. **Handle Migration States**

   * Check for in-progress migrations
   * Complete pending migrations
   * Apply new migrations

4. **PR Communication**
   * Keep PR comments updated
   * Provide clear connection instructions
   * Include relevant warnings

## Related Workflows

* [Check Workflow](./ga-check) - For validating main branch status
* [Clone Workflow](./ga-clone) - For cloning databases in CI/CD


# GitHub App
Source: https://xata.io/docs/automations/github-app

Create and clean up Xata preview branches from GitHub pull requests

The Xata GitHub App links a GitHub repository to a Xata project. After it is linked, pull requests in that repository can create Xata preview branches automatically, without storing a Xata API key in GitHub Actions.

Use the GitHub App when you want Xata to manage the preview branch lifecycle for a repository. If you need a fully custom CI workflow, use the [GitHub Actions pull request workflow](/docs/automations/ga-pr) instead.

## Demo

[Xata GitHub App demo](https://www.youtube.com/embed/xoR9WdzvTYQ)

## How it works

1. Install the Xata GitHub App in a GitHub user account or organization.
2. Link that GitHub App installation to a Xata organization.
3. In the Xata project settings, choose the GitHub repository and the Xata root branch.
4. When a pull request is opened or reopened, Xata creates a child branch from the selected root branch.
5. When the pull request is closed, Xata deletes the preview branch if it is still a child of the selected root branch.

Xata posts or updates a pull request comment when it creates, finds, or deletes the preview branch.

## Prerequisites

* Access to the Xata organization and project that you want to link.
* Permission to install or request GitHub Apps for the GitHub account or organization that owns the repository.
* The Xata GitHub App must be granted access to the repository you want to link.

> [!NOTE]
>
> If you are not a GitHub organization owner, GitHub may submit an installation request instead of installing the app immediately. A GitHub organization owner must approve the request before you can link a repository in Xata.

## Install the GitHub App

1. Open the Xata Console.
2. Go to the project you want to connect.
3. Open **Project Settings**.
4. In **GitHub Repository**, click **Install GitHub App**.
5. In GitHub, choose the account or organization where the app should be installed.
6. Choose whether Xata can access all repositories or only selected repositories.
7. Complete the GitHub installation flow.

![Project settings page showing the GitHub Repository section and Install GitHub App button](https://mintcdn.com/xata/fA73syCjCLsHzvxD/images/automations/github-app-install-button.png?fit=max\&auto=format\&n=fA73syCjCLsHzvxD\&q=85\&s=6fa0ddac1b07d30facc16c04f9311a37)

GitHub lets you grant access to all repositories or only selected repositories. Select only the repositories that should be able to create Xata preview branches.

![GitHub App installation page showing repository selection and requested permissions](https://mintcdn.com/xata/fA73syCjCLsHzvxD/images/automations/github-app-install-page.png?fit=max\&auto=format\&n=fA73syCjCLsHzvxD\&q=85\&s=7b2b3a76b0cc7f99a1da42a369532e61)

After installation, Xata links the GitHub App installation to your Xata organization and returns you to the project settings page. If you installed the app directly from GitHub, Xata asks which Xata organization should own the installation before you continue.

## Link a repository to a project

After the GitHub App is installed, configure the repository mapping from the project settings page:

1. Open **Project Settings**.
2. In **GitHub Repository**, choose the repository to link.
3. Choose the **Root branch**.
4. Click **Save settings**.

![GitHub Repository settings showing repository and root branch selectors](https://mintcdn.com/xata/fA73syCjCLsHzvxD/images/automations/github-app-repository-mapping.png?fit=max\&auto=format\&n=fA73syCjCLsHzvxD\&q=85\&s=68ff047e3e3e71206a2d83cbb437b7c3)

The root branch is the base Xata branch from which GitHub App preview branches are created. For example, if you choose `main`, each preview branch starts as a child of `main` and contains the schema and data from `main` at the moment Xata creates the branch.

> [!NOTE]
>
> A Xata project can have one linked GitHub repository. A GitHub repository can be linked to one Xata project at a time.

## Pull request behavior

When a pull request is opened or reopened in the linked repository, Xata:

* Finds the repository mapping for the GitHub repository.
* Creates a child branch from the configured root branch.
* Names the Xata branch after the pull request's source branch.
* Posts or updates a comment on the pull request with the preview branch status.

If a branch with that name already exists, Xata does not overwrite it. It comments on the pull request that the branch already exists.

When a pull request is closed, Xata:

* Finds the Xata branch with the same name as the pull request's source branch.
* Deletes it only if it is a child of the configured root branch.
* Posts or updates a comment on the pull request when the preview branch is deleted.

Xata checks the parent branch before deletion so that closing a pull request does not delete an unrelated branch that happens to have the same name.

The GitHub App handles pull request open, reopen, and close events. It does not recreate or refresh the preview branch on every push to an already-open pull request.

## Manage repository access

If a repository does not appear in the repository selector, the GitHub App probably does not have access to it. From the project settings page, use the **manage repository access** or **manage installations** link to open the GitHub App installation settings.

In GitHub, grant the Xata GitHub App access to the repository, then return to Xata and refresh the project settings page.

## Manage installations

Use the **GitHub App installations** page in Xata to view the GitHub App installations linked to your Xata organizations. From there, you can:

* Open GitHub to manage repository access for an installation.
* Install the Xata GitHub App in another GitHub account.
* Uninstall a GitHub App installation.

Uninstalling the GitHub App stops GitHub App updates for linked repositories that depend on that installation. The uninstall is synchronized through a GitHub webhook, so it may take a short time for Xata to finish updating its installation records. Existing preview branches are not deleted automatically when you uninstall the app.

## Remove a repository mapping

To stop creating preview branches for a project without uninstalling the GitHub App:

1. Open **Project Settings**.
2. Find **GitHub Repository**.
3. Click **Remove**.
4. Confirm the removal.

Removing the mapping unlinks the GitHub repository from the Xata project. The GitHub App installation remains available for other projects or repositories. Existing preview branches are not deleted automatically when you remove the mapping.

## GitHub Actions comparison

The GitHub App and the GitHub Actions workflows solve related but different problems:

| Use case | Recommended approach |
| - | - |
| Automatically create and clean up one preview branch per pull request | Xata GitHub App |
| Run custom commands, migrations, tests, deployment steps, or comments in CI | [GitHub Actions workflow](/docs/automations/ga-pr) |
| Bulk cleanup of branches by naming convention or age | [Bulk delete workflow](/docs/automations/ga-cleanup) |

You can still run GitHub Actions alongside the GitHub App. For example, let the GitHub App create the Xata preview branch, then use your own CI workflow to run tests or deploy a preview environment with that branch's connection string.

To connect each Vercel preview deployment to its matching Xata branch, see [Vercel preview deployments](/docs/automations/vercel-github-app).

## Troubleshooting

### The installation request is pending

If GitHub shows that the installation was requested, ask a GitHub organization owner to approve the request. Xata cannot link repositories from the installation until GitHub approves it.

### The repository is not listed

Open the GitHub App installation settings and grant Xata access to the repository. If you have multiple GitHub App installations, make sure you are managing the installation for the GitHub account or organization that owns the repository.

### No preview branch is created

Check that the repository is linked in the Xata project settings and that the selected root branch still exists. Xata only reacts to pull request events for repositories that have a saved mapping.

### A branch already exists

Xata does not replace an existing branch with the same name as the pull request's source branch. Delete or rename the existing branch if you want Xata to create a new preview branch for that pull request.

### A closed pull request did not delete a branch

Xata deletes a branch on pull request close only when the branch is a child of the configured root branch. This protects manually created or unrelated branches that happen to share the pull request source branch name.


# Scheduled Clone
Source: https://xata.io/docs/automations/kubernetes-clone

Sample Kubernetes cron job to periodically copy the data from production to your production clone.

This contains a sample Kubernetes manifest file for setting up a periodic task to run Xata Clone.

## Kubernetes spec

```yaml theme={null}
apiVersion: batch/v1
kind: CronJob
metadata:
  name: xata-clone
  namespace: xata-clone          # change if needed
spec:
  schedule: "0 0 * * *"   # change if needed
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: xata-clone
              image: ghcr.io/xataio/xata:latest-pg18
              env:
                - name: XATA_CLI_SOURCE_POSTGRES_URL
                  valueFrom:
                    secretKeyRef:
                      name: xata-secrets
                      key: source-postgres-url
                - name: XATA_API_KEY
                  valueFrom:
                    secretKeyRef:
                      name: xata-secrets
                      key: XATA_API_KEY
                - name: XATA_ORGANIZATION_ID
                  valueFrom:
                    secretKeyRef:
                      name: xata-secrets
                      key: XATA_ORGANIZATION_ID
                - name: XATA_PROJECT_ID
                  valueFrom:
                    secretKeyRef:
                      name: xata-secrets
                      key: XATA_PROJECT_ID
                - name: XATA_DATABASE_NAME
                  valueFrom:
                    secretKeyRef:
                      name: xata-secrets
                      key: XATA_DATABASE_NAME
                - name: XATA_BRANCH_ID
                  valueFrom:
                    secretKeyRef:
                      name: xata-secrets
                      key: XATA_BRANCH_ID
              command:
                - /bin/bash
                - -c
                - |
                  set -e

                  xata init \
                    --organization $XATA_ORGANIZATION_ID \
                    --project $XATA_PROJECT_ID \
                    --database $XATA_DATABASE_NAME \
                    --branch $XATA_BRANCH_ID

                  xata status --json

                  echo "Starting clone..."
                  xata clone start --source-url "$XATA_CLI_SOURCE_POSTGRES_URL" --validation-mode=relaxed
          restartPolicy: OnFailure
```

## Setting up the tasks

Create a dedicated namespace for the task (if you'd like, you can also use an existing one):

```sh theme={null}
kubectl create namespace xata-clone
```

### Step 1: Create secret

The task requires several env variables. Assuming you have the xata CLI already configured on your computer, you can create a secret with all the relevant information like this:

```sh theme={null}
kubectl create secret generic xata-secrets \
  --namespace xata-clone \
  --from-literal=source-postgres-url="$RDS_URI" \
  --from-literal=XATA_API_KEY="$(xata keys organization create)" \
  --from-literal=XATA_ORGANIZATION_ID="$(xata organization get id)" \
  --from-literal=XATA_PROJECT_ID="$(xata project get id)" \
  --from-literal=XATA_BRANCH_ID="$(xata branch get id)" \
  --from-literal=XATA_DATABASE_NAME="postgres"
```

The above assumes branch name is `main` and the database name is `postgres`, but you should adjust to your case.

### Step 2: Create task

Assuming you saved the spec file above as `xata-clone-crontask.yaml`, you can create it with:

```sh theme={null}
kubectl apply -f xata-clone-crontask.yaml
```

### Step 3: Test the task

Create a test job manually based on the task:

```sh theme={null}
kubectl create job -n xata-clone --from=cronjob/xata-clone xata-clone-manual
```

And watch the logs:

```sh theme={null}
kubectl logs -l job-name=xata-clone-manual -f -n xata-clone
```

If there are any errors, you can delete the whole task (`kubectl delete -f xata-clone-crontask.yaml`), fix the issue, and repeat Step 2.


# Vercel preview deployments
Source: https://xata.io/docs/automations/vercel-github-app

Connect Vercel preview deployments to Xata branches created by the GitHub App

Vercel creates a preview deployment for each Git branch. The Xata GitHub App creates a Xata branch for each pull request and gives it the same name as the pull request's source branch.

You can use that shared branch name to connect every Vercel preview deployment to its matching Xata branch. Vercel exposes the Git branch as `VERCEL_GIT_COMMIT_REF`, and a custom build script passes it to `xata branch url` to retrieve the connection string.

This guide uses Next.js as an example. If you use another framework or package manager, adapt the environment file and build commands to your application.

## How it works

1. You push a Git branch and open a pull request.
2. The Xata GitHub App creates a Xata branch named after the pull request's source branch.
3. Subsequent pushes reuse that Xata branch. The GitHub App does not recreate or refresh it on every push.
4. During each preview build, Vercel sets `VERCEL_GIT_COMMIT_REF` to that same branch name.
5. The build script retrieves the Xata branch's connection string and makes it available to the application as `DATABASE_URL`.
6. When the pull request closes, the Xata GitHub App deletes the Xata branch.

## Prerequisites

* [Install the Xata GitHub App and link the repository to a Xata project](/docs/automations/github-app).
* Connect the same GitHub repository to a Vercel project.
* Configure your application to read its PostgreSQL connection string from `DATABASE_URL`.

> [!NOTE]
>
> This guide uses the Xata GitHub App to manage the branch lifecycle. You can use the [GitHub Actions pull request workflow](/docs/automations/ga-pr) instead, but its example names branches `<source-branch>_<pull-request-number>`. Adjust the branch name in the Vercel script to match your workflow.

## Create an API key for Vercel

The Vercel build uses the Xata CLI to find the preview branch and retrieve its connection string. Create a dedicated API key for this automation:

1. Open [User API keys](https://console.xata.io/account/api-keys) in the Xata Console.
2. Click **New API Key**.
3. Give the key a descriptive name, such as `Vercel preview deployments`, and choose an appropriate expiration.
4. Under **Projects & Branches**, select the Xata project linked to the repository. Select the project rather than an individual branch so the key can access preview branches created later.
5. Enable **Access control** and select the **CI/CD deploy bot** preset.
6. Create the key and copy it. The full value is shown only once.

![Xata user API key form scoped to one project with the CI/CD deploy bot preset selected](https://mintcdn.com/xata/39VoIPEG0avOsZrO/images/automations/vercel-api-key.png?fit=max\&auto=format\&n=39VoIPEG0avOsZrO\&q=85\&s=e3a76b822f2cf055c092d030edfe693c)

See [API keys](/docs/platform/api-key) for more information about project restrictions, scopes, and expiration.

> [!WARNING]
>
> Treat the API key like a password. Do not commit it to the repository or print it in build logs.

## Configure Vercel environment variables

In the Vercel project, open **Settings** > **Environment Variables** and add these variables for the **Preview** environment:

| Variable | Value |
| - | - |
| `XATA_API_KEY` | The API key created in the previous section. Mark it as **Sensitive**. |
| `XATA_ORGANIZATION_ID` | The organization ID from the Xata Console project URL, or from `xata organization get id`. |
| `XATA_PROJECT_ID` | The project ID from the Xata Console project URL, or from `xata project get id`. |

The Xata CLI automatically uses these standard environment variables for authentication and project selection. You do not need to install the CLI locally if you copy the organization and project IDs from a Console URL such as `https://console.xata.io/organizations/<organization-id>/projects/<project-id>`.

> [!NOTE]
>
> Add `XATA_API_KEY` to Vercel because the connection-string lookup runs in the Vercel build. If you use GitHub Actions to manage the branch lifecycle, also add the key as a [GitHub Actions secret](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets) named `XATA_API_KEY` for that workflow.

For production, configure `DATABASE_URL` separately in Vercel's **Production** environment. For example, get the primary connection string for the Xata production branch with:

```bash theme={null}
xata branch url main --type primary
```

The build script below only derives `DATABASE_URL` dynamically for preview deployments. Production deployments continue to use the value configured in Vercel.

### Enable Vercel system environment variables

The script requires `VERCEL_ENV` and `VERCEL_GIT_COMMIT_REF`. Vercel provides both as [system environment variables](https://vercel.com/docs/environment-variables/system-environment-variables).

In the Vercel project's **Environment Variables** settings, select **Enable access to System Environment Variables**. `VERCEL_GIT_COMMIT_REF` contains the Git branch that triggered the deployment.

## Add the build script

Create `scripts/vercel-build.sh` in your application repository:

```bash theme={null}
#!/usr/bin/env bash
# Vercel build entry point. For preview deploys, use the Xata branch created
# by the GitHub App and point DATABASE_URL at it. Production builds use the
# DATABASE_URL configured in the Vercel project.

# Do not enable `set -x`: it can print the connection string to build logs.
set -euo pipefail

if [[ "${VERCEL_ENV:-}" == "preview" && -n "${VERCEL_GIT_COMMIT_REF:-}" ]]; then
  # Install the CLI if the binary is not found.
  if ! command -v xata >/dev/null 2>&1; then
    curl -fsSL https://xata.io/install.sh | bash
    export PATH="$HOME/.config/xata/bin:$PATH"
  fi

  # Sanity check that the CLI was installed successfully.
  xata version

  # With the GitHub App, the Xata branch name matches the Git branch name.
  # If GitHub Actions uses another naming pattern, change this assignment.
  BRANCH_NAME="${VERCEL_GIT_COMMIT_REF}"

  # The GitHub App creates the branch once when the PR opens. Preview builds
  # reuse it instead of deleting and recreating it on every push.
  xata branch wait-ready "$BRANCH_NAME"

  DSN=$(xata branch url "$BRANCH_NAME" --type primary)
  echo "DATABASE_URL=$DSN" >> .env.production

  DATABASE_URL="$DSN" npm run db:migrate
fi

npm run build
```

Shell tracing (`set -x`) is intentionally disabled so the connection string is not written to Vercel build logs.

The `db:migrate` command applies the application's migrations to the preview branch. Remove that line if your application does not have a `db:migrate` script.

The GitHub App does not recreate the Xata branch on every push. Keeping the branch for the lifetime of the pull request preserves its branch ID and connection URL. Xata does not currently provide a branch-reset operation; deleting and recreating the branch would give it a new ID and URL.

The lookup fails if the matching Xata branch cannot be found. Do not add a fallback to the production connection string: a failed preview build is safer than a preview deployment that writes to production.

> [!NOTE]
>
> The `.env.production` line is specific to Next.js. If you use another framework, use its supported build-time mechanism to include `DATABASE_URL` in server-side deployment output. Never expose the connection string through a public or client-prefixed environment variable.

Add a script entry to `package.json` while keeping the existing `build` script unchanged:

```json theme={null}
{
  "scripts": {
    "build": "next build",
    "vercel-build": "bash scripts/vercel-build.sh"
  }
}
```

## Override the Vercel build command

In Vercel:

1. Open the project and go to **Settings** > [**Build and Deployment**](https://vercel.com/docs/builds/configure-a-build#build-command).
2. Under **Framework Settings**, enable **Override** for **Build Command**.
3. Set the command to `pnpm vercel-build`.
4. Save the settings and redeploy.

If your project uses npm, use `npm run vercel-build` instead.

> [!NOTE]
>
> Vercel may detect a `vercel-build` script automatically for some frameworks, but this behavior is not documented and may change. Set the Build Command explicitly in the Vercel project settings.

![Vercel Framework Settings with the Build Command override set to pnpm vercel-build](https://mintcdn.com/xata/39VoIPEG0avOsZrO/images/automations/vercel-build-command.png?fit=max\&auto=format\&n=39VoIPEG0avOsZrO\&q=85\&s=ebe25a3b6d2f8169f0729f016d72a90f)

## Verify the integration

1. Open a pull request in the linked GitHub repository.
2. Wait for the Xata GitHub App's pull request comment confirming that the preview branch was created.
3. Redeploy the Vercel preview if its first build started before the Xata branch was created.
4. Check the Vercel build log for the Xata branch name and a successful application build. The connection string itself should not appear in the log.
5. Open the preview deployment and verify that it reads from or writes to the preview branch, not the production branch.

## Troubleshooting

### `VERCEL_GIT_COMMIT_REF` is not set

Select **Enable access to System Environment Variables** in the Vercel project's environment variable settings, then redeploy.

### The Xata branch cannot be found

The GitHub App creates a branch when a pull request is opened or reopened, while Vercel can build a Git branch before a pull request exists. Confirm that the GitHub App posted a successful branch-creation comment, then redeploy the Vercel preview.

Also check that the same repository is linked to both projects and that your branch-lifecycle automation uses the expected naming convention.

### The Xata CLI cannot authenticate or select a project

Confirm that `XATA_API_KEY`, `XATA_ORGANIZATION_ID`, and `XATA_PROJECT_ID` are set for Vercel's **Preview** environment. Make sure the API key is scoped to the linked Xata project, not only to its root branch.

### The preview still connects to the production database

Confirm that your server-side application reads `DATABASE_URL` and that your framework includes the dynamically generated value in its deployment output. For Next.js, keep the `.env.production` step before `next build`.

### A preview stops working after the pull request closes

This is expected. Closing the pull request causes the Xata GitHub App to delete its preview branch, so existing Vercel deployments for that pull request can no longer connect to it.

### A pull request comes from a fork

Vercel protects environment variables for deployments from forks. An authorized project member must approve the deployment before it can access the Xata API key. Do not disable this protection for a project that contains secrets.


# Changelog
Source: https://xata.io/docs/changelog

Stay up to date with the latest Xata features, fixes and changes

**Oct 22, 2026**

The Queries page is now the SQL Editor, and it opens queries in tabs.

## SQL Editor with tabs

The branch **Queries** page is now called **SQL Editor** and lives at `/sql-editor`. Old `/queries` links redirect to the new page, and saved queries and editor settings are unchanged.

The editor now opens queries in tabs:

* Each tab keeps its own edits, undo history, cursor position, and results, and a running query keeps running when you switch tabs.
* Reorder tabs by dragging or with `Alt` + `Shift` + `←` / `→`. Right-click a tab to close it, close others, close to the right, or close all.
* New queries open as drafts, listed under **Drafts** in the sidebar until they are saved.
* Open tabs are restored after a reload, and you are asked before closing tabs or leaving the page with unsaved changes.

Learn more in the [SQL Editor](/docs/platform/sql-editor) documentation.

**Sep 29, 2026**

Organizations now have Admin and Editor roles, so you can decide who manages members, billing, API keys and single sign-on.

## Organization roles

Every organization member is now an **Admin** or an **Editor**. Admins manage members, billing, API keys and single sign-on; Editors work with projects and branches.

Existing members are Admins, so nobody lost access. New invitations default to Editor. Admins change roles from the **Members** page, with `xata organization members set-role`, or through the API. See [Organization roles](/docs/platform/organization-roles).

**Aug 25, 2026**

A hosted MCP server at api.xata.tech/mcp gives coding agents the Xata API, SQL against a branch, and the docs, behind 11 tools.

## MCP server

A hosted MCP server that lets AI assistants and coding agents work with your Xata organizations, projects, and branches.

There is nothing to install. Point an MCP client at `https://api.xata.tech/mcp` over Streamable HTTP and sign in with OAuth, or send a [Xata API key](/docs/platform/api-key) in an `Authorization` header for CI and headless agents.

The server exposes 11 tools:

* **REST API:** `search_operations` finds an operation by intent, `describe_operation` returns its schemas, and `call_read_operation`, `call_write_operation`, and `call_destructive_operation` invoke it. 63 API operations sit behind those five tools, so an agent pulls in a schema only when it needs one, and you can allow reads in your client without allowing deletes.
* **SQL:** `run_sql` runs a statement against a branch, and `describe_schema` lists its tables and columns. `run_sql` is read-only by default and runs inside a read-only batch the gateway enforces. Statements that modify data or schema require both `write=true` and `confirm=true`. `call_destructive_operation` requires `confirm=true` as well.
* **Skills and docs:** `list_skills` and `get_skill` read Xata's guided workflows for common multi-step tasks, and `search_xata` and `query_docs_filesystem_xata` search and read the documentation.

Credentials stay server-side. `run_sql` takes organization, project, and branch ids and resolves the branch gateway and its password itself, and connection strings and passwords are stripped out of responses before the model sees them.

Xata is listed in the MCP registry as `io.github.xataio/mcp`. Setup steps for Cursor, Claude Code, VS Code, Claude, ChatGPT, Codex CLI, Antigravity, OpenCode, Amp, Windsurf, Zed, and Cline are in the [MCP server](/docs/platform/mcp) documentation.

**Jul 30, 2026**

Query Insights is live in the console for every branch, and xata branch metrics and query-insights join xata branch logs in the CLI.

## Branch observability

Postgres logs, instance metrics, and `pg_stat_statements` statistics for any branch, in the console and from the CLI.

**Query Insights** is now in the console for every branch. It groups statements by their normalized form, highlights any statement whose slowest execution passed one second, and opens an inline detail panel with the full SQL, call count, and cache hit ratio. Filter by statement type, database, and role.

Two commands join `xata branch logs` in the CLI:

* **`xata branch metrics`** reports CPU, memory, disk, connections, IOPS, throughput, latency, and replication lag across the primary and its replicas. Add `-w` for a live view, which renders a dashboard in a terminal and NDJSON in a pipe.
* **`xata branch query-insights`** (alias `qi`) reads `pg_stat_statements` for historical query statistics. `qi active` reads `pg_stat_activity` for the queries running right now, with their state and wait events. There is no console page for `active`.

`xata branch logs`, `xata branch metrics`, and `xata branch qi list` all take `-o json` or `-o ndjson`. The remaining `qi` subcommands take a plain `--json`. Everything here is scoped to a single branch.

One upgrade note: as of CLI 1.5.0, `xata branch query-insights` needs the `credentials:read` scope on your API key, because it opens a direct Postgres connection to the branch. `xata branch logs` needs `logs:read`. Add the scope if your key predates 1.5.0.

See the [CLI branch reference](/docs/cli/branch), [Query Insights](/docs/platform/query-insights), [Logs](/docs/platform/logs), and [Metrics](/docs/platform/metrics), or follow the [Find slow queries](/docs/tutorials/find-slow-queries) tutorial.

**Jul 8, 2026**

Link a GitHub repository to a Xata project and get a preview branch per pull request, created and cleaned up automatically, with no API key in CI.

## GitHub App

The Xata GitHub App links a GitHub repository to a Xata project and manages preview branches for you. Once it is linked, opening a pull request creates a Xata branch from your root branch, and closing the pull request deletes it. No Xata API key in GitHub Actions, and no cleanup workflow to maintain.

* **Branch per pull request:** when a PR is opened or reopened, Xata creates a child branch from the root branch you choose (for example `main`), named after the PR's source branch, with the schema and data from the root branch at the moment the branch is created.
* **Automatic cleanup:** when the PR closes, Xata deletes the preview branch, but only if it is still a child of the configured root branch, so an unrelated branch that happens to share the name is left alone.
* **Status in the pull request:** Xata posts and updates a comment on the PR when it creates, finds, or deletes the preview branch.
* **Install from project settings:** open **Project Settings > GitHub Repository**, install the app, then pick the repository and root branch. One repository links to one project.

For custom migrations, tests, deploy steps, or comments in CI, use the [GitHub Actions pull request workflow](/docs/automations/ga-pr) instead, or run it alongside the app: let the app create the branch, then use your own workflow with that branch's connection string. See the [GitHub App](/docs/automations/github-app) documentation.

**Jun 30, 2026**

Run Xata on Google Cloud in us-central1: a Postgres branch in 3 seconds, or a new database in a second, for every job on a second cloud.

## GCP us-central1 region support

You can now run Xata on Google Cloud in us-central1 (Iowa), alongside AWS us-east-1, us-west-2, and eu-central-1. Google Cloud is Xata's second cloud.

If you already run Postgres on Google Cloud, you can now create a Postgres branch from your production data in 3 seconds, or provision a new database in a second, in-region on the same cloud as the compute that uses it. Branches use copy-on-write storage, so a thousand short-lived branches cost about as much as one.

* **Coding agents:** give every coding agent its own isolated Postgres branch in 3 seconds to build, test, and validate against production-like data.
* **Deployment previews and CI:** create a fresh branch in 3 seconds for every pull request and test run, with no shared staging.
* **AI application builders:** provision a new Postgres database in a second for every generated app, each with its own dedicated database from day one.

You can also run your production database on Google Cloud, not just branches. Talk to our team about production hosting on GCP.

**Jun 26, 2026**

You can now create Xata databases in AWS us-west-2, alongside us-east-1.

## AWS us-west-2 region support

You can now create Xata databases in AWS us-west-2, alongside us-east-1.

If your agents and CI jobs run on the West Coast, the Postgres branches they spin up can now run in the same region as the compute creating them. When every agent run and CI job gets its own database, and you spin up, run, and discard thousands of times an hour, keeping those branches in-region with your compute matters.

And if Xata runs your production database, not just your branches, you can now host it in us-west-2 as well.

Everything works the way it does in us-east-1 from day one: copy-on-write branching in under a second, snapshots, scale-to-zero, and subsecond wake-up, all on Xatastor. Pricing in us-west-2 is the same as us-east-1, so there is no regional premium.

Pick your region when you create a database. Existing databases stay where they are.

us-west-2 is the next step in running Xata where you already build: us-east-1 and us-west-2 on AWS today, GCP next, and BYOC across AWS, Azure, GCP, and Hetzner if you would rather keep data in your own cloud account.

**Jun 9, 2026**

Branch logs are now generally available in the dashboard for every branch.

## Branch logs GA

The **Logs** tab on every branch is now available to all users without an opt-in. Browse PostgreSQL server output across the primary and replicas, filter by level, instance, and process, and search messages with substring or regex matching.

* New Logs page in the branch sidebar with a searchable, paginated viewer.
* Level, instance, and process filters plus free-text search on the log body.
* Custom and preset time ranges, with details panel per entry.
* The `branchLogs` API remains available for programmatic access — see the [Logs](/docs/platform/logs) documentation.

**May 28, 2026**

Query editor refresh with draft saves, multi-statement results, and export options.

## Query editor improvements

The queries page has been reworked for a faster, more flexible SQL workflow.

* Resizable sidebar and editor/results panels, with responsive sidebar collapse on smaller screens.
* New draft flow: **New Query** opens an unsaved draft that you can persist with **Save as...** (manual mode) or automatically with **Auto-save**.
* Toolbar actions for save, rename, delete, and Ask AI, plus a save-status indicator and keyboard shortcuts (`⌘S`, `⌘K`, `⌘+Enter`).
* **Ask AI** is now an anchored popover in the editor instead of a modal.
* Results panel now supports multi-statement tabs, `EXPLAIN` plan rendering, row-count and status display, client-side row limits, compact density, striped rows, copy-as-JSON, CSV download, and full-screen expansion.
* **Run selected** runs only the highlighted portion of the editor.
* Performance warnings are now less intrusive and dismissible, with dismissal remembered locally.
* Editor preferences (auto-save, performance warnings, striped rows, compact mode, row limit) persist across sessions.

Learn more in the [SQL Editor](/docs/platform/sql-editor) documentation.

**Apr 27, 2026**

Serverless connections now route through the connection pooler by default.

## Serverless Pooler Routing

Serverless connections (HTTP and WebSocket) now route through the PgBouncer connection pooler by default when no endpoint suffix is specified in the hostname. This reduces the number of PostgreSQL connections opened by serverless and edge functions, improving stability under high concurrency.

* Hostnames without a suffix (e.g., `{branch-id}.<region>.xata.sh`) are routed through PgBouncer
* Explicit endpoint suffixes (`-rw`, `-ro`, `-r`, `-pooler`) continue to take precedence
* To bypass the pooler and connect directly to the primary, use the `-rw` suffix

Learn more about [connection endpoints](/docs/core-concepts/serverless-proxy#connection-endpoints).

**Apr 24, 2026**

Connection pooler endpoints now available from the console and CLI.

## Connection Pooler Endpoint

You can now connect to your branch through a built-in connection pooler. Pooled endpoints multiplex many client connections onto fewer server connections, making them ideal for serverless and high-concurrency workloads.

Select **Pooler** when building a connection string from the branch overview page, or use the CLI:

```bash theme={null}
xata branch url main --type pooler
```

The `xata branch url` command adds a new `--type` flag that accepts `primary`, `primary-or-replica`, `replica`, or `pooler`. Learn more about [endpoint types](/docs/platform/branch#endpoint-types).

**Apr 22, 2026**

Rotate database passwords for enhanced security.

## Password Rotation

You can now rotate the database password for the `xata` PostgreSQL user directly from the Console or CLI. This security feature allows you to generate new credentials without disrupting existing connections.

Key features:

* Rotate passwords through the branch settings page in the Console
* Use the new `xata branch rotate-password` CLI command
* Existing database connections remain active during rotation
* PgBouncer automatically picks up the new password for connection pooling

Learn more about [rotating database credentials](/docs/platform/branch#rotate-password).

**Apr 17, 2026**

PostgreSQL minor version upgrades now supported for branches.

## PostgreSQL Minor Version Upgrades

You can now upgrade your branches to newer minor versions of PostgreSQL through the Console, CLI, and API. This allows you to apply bug fixes and performance improvements from newer PostgreSQL releases while maintaining the same major version.

When upgrading:

* Only minor version upgrades are supported (e.g., 17.5 → 17.7)
* The PostgreSQL offering type (postgres or analytics) must remain the same
* Downgrades and major version changes are not permitted
* Upgrades trigger a rolling update of your branch instances

Learn more about [upgrading PostgreSQL versions](/docs/platform/branch#upgrading-postgresql-versions).

**Apr 7, 2026**

Connection pooler endpoint support for efficient connection management.

## Connection Pooler Endpoint

You can now connect to your Xata database through a connection pooler endpoint using PgBouncer. This is particularly useful for applications that create many short-lived connections, as the pooler maintains a pool of persistent connections to the database and reuses them across client requests.

When enabled, you can connect to your branch using the `-pooler` suffix in your connection string (e.g., `{branch-id}-pooler`). This routes your connections through PgBouncer for efficient connection pooling.

Learn more about [connection endpoints](/docs/core-concepts/serverless-proxy#connection-endpoints).

**Mar 12, 2026**

CLI improvements for binary downloads and authentication debugging.

## CLI Download Progress

The Xata CLI now displays download progress when upgrading or downloading pgroll and pgstream binaries. You'll see a progress bar with percentage completion, making it easier to track the status of binary downloads during `xata upgrade` and `xata stream/roll download` commands.

## Enhanced Authentication Debugging

When using the `XATA_API_KEY` environment variable for authentication, the CLI now provides helpful debug logging. If a command fails and you're using an API key from the environment, you'll see a note indicating this, making it easier to troubleshoot authentication issues.

**Jan 8, 2026**

Multiple PostgreSQL image sources and analytics image support.

## Multiple PostgreSQL Image Sources

You can now choose from multiple PostgreSQL image sources when creating or configuring branches. In addition to the standard `postgres` image, we now support:

* **Analytics**: Optimized PostgreSQL images for analytical workloads with specialized extensions and configurations

**Sep 26, 2025**

PG18 support, choosing a default database, preloaded libraries, new metrics and simplified onboarding.

## Support for Postgres 18!

Postgres 18 was officially [released yesterday](https://www.postgresql.org/about/news/postgresql-18-released-3142/) 🥳 We've been testing the alpha, beta and release candidates as they've come out to have PG18 support on day one. We won't be defaulting to PG18 likely until 18.1, but it's now available for everyone in our private beta.

![Choose Postgres 18 for your main branch](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-26/pg18-support.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=c78dd9259a97a93163f95e3202e1d0ff)

## Project-level Default Database

There are many views within a project that use the database selected. If there's more than one database, sometimes it's not the one you expected. We've had a [few requests](https://xata.canny.io/xata-feature-requests/p/specify-default-database) to introduce the ability to define a default database to make things like copying connection strings easier. You can now set a project level default database that all views and child branches will default to within the console.

![Select a default database for your project](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-26/default-database.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=ca1e7cf90addd12bac9fe184aad0eb9e)

## Preloaded libraries

You can now configure extensions and libraries be preloaded with your branches. These will be loaded into shared memory and available for all databases on the branch.

![Preload libraries to a branch](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-26/preload-libraries.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=b5104a3fcf7a3d5c931e9a000e7e22a2)

## Streamlined Onboarding

Previously there were many steps to onboarding -- create an organization, create a project and create your branch. This was unnecessary friction and we've simplified this initial experience into a single screen.

![Simplified onboarding](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-26/streamlined-onboarding.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=1457b286f106ce8e80aeff5b82ddc159)

## New Metrics

We've added two more metrics for WAL sync time and replication lag time. These new metrics require additional Postgres settings to be enabled. If they aren't enabled, you will now be prompted to enable to view the metrics.

![New metrics available on your Postgres instance](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-26/new-metrics.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=c625f2d56be5ff7b348a17654d836c61)

## Branch Settings Menu

It's the little things in life. We had too many branch settings and the page was getting bloated. As a result, we've added a new menu and search to help you get to the setting you need.

![New metrics available on your Postgres instance](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-26/branch-settings.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=69985aa1d4392788593c36027988d43e)

**Sep 10, 2025**

Scale to zero, a new region and much more.

## Scale to Zero

You can now configure a branch to scale to zero after a duration of inactivity. Set project defaults or branch specific settings for the duration of inactivity.  You can also manually hibernate a branch. Learn more about scale to zero [here](/docs/platform/compute).

![Configure compute and scale to zero for your branch](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-10/scale-to-zero.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=176ecf915818bf38f6cd635a4734b816)

## New Frankfurt Region

AWS Frankfurt is now available as a region 🎉

![Frankfurt now available as a region](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-10/frankfurt.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=58553e16e702113a82d5dcd86bb5144b)

## Endpoint Routing

When building your connection string, you can now choose to route traffic to your primary instance only, primary and read replicas or read replicas only.

![Choose from read-only and read / write endpoints](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-10/endpoint.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=66037612c3ea0457f413f180a86c79f5)

## PostgreSQL Configuration

From your branch settings page, you can now configure PostgreSQL settings directly in the console. Some maximum values are set by the instance size in use.

![Update PostgreSQL config directly from console](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-10/postgres-config.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=ca84d8d576864e84d3871db2ef137a7d)

## General SQL ✨

Generate SQL queries through natural language and explore your data simply by asking a question.

![Generate SQL with AI](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-10/generate-sql.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=4b6ce0f9112093a9b0ed6019d314b67e)

## Extensions View

You can now view all supported and enabled extensions directly from console. You can see what extensions are currently available [here](/docs/platform/extensions).

![View available extensions in console](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-10/extensions.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=e8366c154d7a3511465b3aa28549e91b)

## Revamped Metrics View

We keep adding metrics for improved visibility. Rather than scrolling forever, we've introduced a nice two column view to consolidate our charts. You can see what metrics are currently available [here](/docs/platform/metrics).

![View available metrics in console](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/changelog/2025-09-10/metrics.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=b9719e86e4a767eec9daecf6be152cc0)


# auth
Source: https://xata.io/docs/cli/auth

Authenticate with Xata

Sessions are stored as profiles, so several accounts and environments can be used side by side. Commands run against the active profile unless `--profile` names another one.

Every command below also takes `-h, --help`.

* [`xata auth login`](#login) — Log in to a Xata account
* [`xata auth logout`](#logout) — Log out of the current account
* [`xata auth status`](#status) — Display active account and authentication state
* [`xata auth switch`](#switch) — Switch to a different Xata account profile
* [`xata auth list`](#list) — List all available Xata account profiles
* [`xata auth refresh`](#refresh) — Refresh the access token of the current session
* [`xata auth access-token`](#access-token) — Print the current access token
* [`xata auth refresh-token`](#refresh-token) — Print the stored refresh token, without refreshing the session

## login

Log in to a Xata account

Prints a URL and a code to authorize this machine, or stores an API key with `--api-key` for non-interactive use. The issuer, API URL, console URL and client flags log in against a deployment other than production, which is how Enterprise customers connect the CLI to a custom deployment in their own cloud. Omit them and the CLI uses the default production values.

```bash theme={null}
xata auth login [--force] [--api-key value] [--issuer value] [--api-url value] [--client-id value] [--client-secret value] [--console-url value] [--profile value] [--debug] [--json]
```

**-f, --force** `boolean` (default: false)

Force login even if already logged in, revoking the previous session

**--api-key** `string`

Log in non-interactively with an API key instead of the browser OAuth flow

**--issuer** `string`

Issuer URL for custom environment

**--api-url** `string`

API base URL for custom environment

**--client-id** `string`

Client ID for custom environment (defaults to "cli")

**--client-secret** `string`

Client secret for custom environment

**--console-url** `string`

Console URL for custom environment, used by AI features

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Examples:**

```bash theme={null}
# Log in from a script or CI
xata auth login --api-key xau_...
# Log in as another profile
xata auth login --profile staging
# Log in to another deployment
xata auth login --profile staging --api-url https://api.staging.example.com
```

## logout

Log out of the current account

Revokes the session of the profile with the identity provider and removes the stored credentials. Because the CLI is a single application to the identity provider, revoking may also sign out other CLI installations that were authorized from the same browser session. Use `--local` when the identity provider is unreachable to remove the stored credentials without revoking the session. Profiles that use an API key are only removed locally; the key itself stays valid.

```bash theme={null}
xata auth logout [--yes] [--local] [--profile value] [--debug] [--json]
```

**--yes** `boolean` (default: false)

Do not ask for confirmation, assume yes.

**--local** `boolean` (default: false)

Only remove the stored credentials, do not revoke the session with the identity provider

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## status

Display active account and authentication state

Reads the stored session without contacting the server, so it stays fast and works offline. It reports an expiry the stored session has already passed, but it cannot see a session revoked server-side. Use `xata auth refresh` to verify against the server.

```bash theme={null}
xata auth status [--profile value] [--debug] [--json]
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## switch

Switch to a different Xata account profile

```bash theme={null}
xata auth switch [--profile value] [--debug] [--json] [<arg1>]
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**value** `string`

The profile to switch to

## list

List all available Xata account profiles

```bash theme={null}
xata auth list [--profile value] [--debug] [--json]
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata auth ls`

## refresh

Refresh the access token of the current session

Refreshes the session even when the current access token is still valid, and stores the new one. Every command already refreshes on its own when the token is about to expire, so this is for scripts that want to fail early, or to rotate the token before a long job.

```bash theme={null}
xata auth refresh [--profile value] [--debug] [--json]
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## access-token

Print the current access token

```bash theme={null}
xata auth access-token [--profile value] [--debug] [--json]
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## refresh-token

Print the stored refresh token, without refreshing the session

```bash theme={null}
xata auth refresh-token [--profile value] [--debug] [--json]
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.


# branch
Source: https://xata.io/docs/cli/branch

Create, list, and manage Xata branches

A branch is a running Postgres database. Creating one copies the data of its parent, and the branch checked out in this folder is the one commands act on when no branch is given.

Every command below also takes `-h, --help`.

* [`xata branch list`](#list) — List all branches
* [`xata branch describe`](#describe) — Describe a branch
* [`xata branch create`](#create) — Create a new branch
* [`xata branch delete`](#delete) — Delete a branch
* [`xata branch logs`](#logs) — Retrieve the PostgreSQL logs of a branch
* [`xata branch url`](#url) — Print URL (connection string) for a branch
* [`xata branch checkout`](#checkout) — Check out a branch in this folder
* [`xata branch tree`](#tree) — List all branches as a tree
* [`xata branch get`](#get) — Get a field from a branch description
* [`xata branch metrics`](#metrics) — Show CPU, memory and disk usage for a branch
* [`xata branch set`](#set) — Set a field value for a branch
* [`xata branch rotate-password`](#rotate-password) — Rotate the database password for a branch
* [`xata branch wait-ready`](#wait-ready) — Wait for a branch to be ready
* [`xata branch query-insights`](#query-insights) — Inspect query statistics and active queries for a branch
  * [`xata branch query-insights list`](#query-insights-list) — List historical query statistics for a branch, by total execution time
  * [`xata branch query-insights show`](#query-insights-show) — Show full query statistics for a query ID
  * [`xata branch query-insights active`](#query-insights-active) — List currently running queries for a branch
  * [`xata branch query-insights enable`](#query-insights-enable) — Enable pg\_stat\_statements for query insights on a branch
  * [`xata branch query-insights reset`](#query-insights-reset) — Reset accumulated query statistics for a branch

## list

List all branches

```bash theme={null}
xata branch list [--organization value] [--project value] [--branch value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata branch ls`

## describe

Describe a branch

```bash theme={null}
xata branch describe [--organization value] [--project value] [--branch value] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to describe

**Aliases:** `xata branch view`, `xata branch show`

## create

Create a new branch

A branch is a running Postgres database that starts as a copy of its parent. It takes a moment to come up, so `xata branch wait-ready` is what to run before connecting to it. It is checked out afterwards when this folder already has an organization and a project to work from.

```bash theme={null}
xata branch create [--organization value] [--project value] [--parent-branch value] [--no-parent] [--name value] [--description value] [--instance-type value] [--replicas value] [--region value] [--postgres-version value] [--storage value] [--scale-to-zero true|false] [--inactivity-period value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--parent-branch** `string`

Parent branch ID or name to fork from. Cannot be combined with --no-parent.

**--no-parent** `boolean` (default: false)

Create a root branch with no parent, instead of forking one

**--name** `string`

Branch name

**--description** `string`

Short description of what the branch is for

**--instance-type** `string`

Instance type for the branch

**--replicas** `string`

Number of read replicas for the branch

**--region** `string`

Region to create the branch in

**--postgres-version** `string`

PostgreSQL version for the branch

**--storage** `string`

Storage in GB for the branch. Root branches only, as a fork inherits its parent

**--scale-to-zero** `true | false`

Scale to zero status for the branch

**--inactivity-period** `string`

Inactivity period for the branch in whole minutes (1–1440)

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Examples:**

```bash theme={null}
# Branch the current branch
xata branch create --name my-branch
# Branch another branch, by ID or by name
xata branch create --name my-branch --parent-branch main
# Create a root branch with no parent
xata branch create --name my-branch --no-parent
# Create a root branch with a 10 GB disk
xata branch create --name my-branch --no-parent --storage 10
# Describe what the branch is for
xata branch create --name my-branch --description "Nightly import"
# Size the branch and let it scale to zero
xata branch create --name my-branch --instance-type <type> --replicas 1 --scale-to-zero true
```

## delete

Delete a branch

The branch checked out in this folder cannot be deleted, and outside an interactive terminal the confirmation has to come from `--yes`.

> [!WARNING]
>
> Deleting a branch destroys its database and cannot be undone.

```bash theme={null}
xata branch delete [--organization value] [--project value] [--branch value] [--yes] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--yes** `boolean` (default: false)

Do not ask for confirmation, assume yes.

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to delete

**Examples:**

```bash theme={null}
# Delete a branch, asking for confirmation
xata branch delete my-branch
# Delete a branch from a script
xata branch delete my-branch --yes
```

## logs

Retrieve the PostgreSQL logs of a branch

Reads the logs of every instance of the branch, the primary and any replicas, which is where slow queries, connection issues and replication problems show up. Requires the `logs:read` scope on the API key. Of the output formats, `raw` prints `<timestamp> [<level> <instanceID> <process>] <message>` per line, `json` a single array, `ndjson` one object per line for streaming into another process, and `csv` the columns `timestamp,level,instanceID,process,message`. Follow mode polls every 2 seconds with a 5 second overlap and de-duplicates entries. Logs can contain connection strings and other credentials, see [https://xata.io/docs/platform/logs](https://xata.io/docs/platform/logs) for what is redacted.

```bash theme={null}
xata branch logs [--organization value] [--project value] [--branch value] [--level debug|info|warning|error] [--instance value]... [--process value]... [--search value] [--start value] [--end value] [--limit value] [--follow] [--output raw|json|ndjson|csv] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--level** `debug | info | warning | error`

Filter by log level. Can be repeated.

**--instance** `string`

Filter by branch instance ID. Can be repeated.

**--process** `string`

Filter by process name. Can be repeated.

**--search** `string`

Case-insensitive substring search in the log message body. For regex filtering, pipe raw output to rg/grep, e.g. xata branch logs --output raw | rg 'timeout|deadlock'

**--start** `string`

Start time as YYYY-MM-DDTHH:mm:ss.sssZ or relative duration, e.g. 15m, 1h, 7d. Defaults to 1h ago.

**--end** `string`

End time as YYYY-MM-DDTHH:mm:ss.sssZ or relative duration. Defaults to now.

**--limit** `string` (default: 100)

Maximum number of logs to fetch, up to 1000

**-f, --follow** `boolean` (default: false)

Poll for new logs continuously. Cannot be combined with --output json

**-o, --output** `raw | json | ndjson | csv`

Output format. Defaults to raw, or json when an AI agent runs the command.

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to retrieve logs for

**Examples:**

```bash theme={null}
# Errors from the last 15 minutes
xata branch logs my-branch --level error --start 15m
# Follow errors and pull out the message text
xata branch logs my-branch -f --output ndjson | jq -r 'select(.level=="error") | .message'
# Regex search, which --search does not do, by piping raw output
xata branch logs my-branch --output raw | rg 'timeout|deadlock'
# Export one replica's Postgres process logs to a file
xata branch logs my-branch --process postgres --instance <replica-id> --output csv > logs.csv
```

## url

Print URL (connection string) for a branch

Reads the connection details from the credentials endpoint, so an API key needs the `credentials:read` scope, see [https://xata.io/docs/cli#required-scopes](https://xata.io/docs/cli#required-scopes).

```bash theme={null}
xata branch url [--organization value] [--project value] [--branch value] [--database value] [--type primary|primary-or-replica|replica|pooler] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name

**--type** `primary | primary-or-replica | replica | pooler` (default: primary)

Connection type: primary (direct access to the primary), primary-or-replica (routed access to primary or replicas), replica (read-only access to replicas only, requires at least one replica), pooler (pooled access to the primary, recommended for serverless and high-concurrency workloads)

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to get URL for

**Examples:**

```bash theme={null}
# Print the primary connection string
xata branch url main
# Pooled connection string, for serverless workloads
xata branch url main --type pooler
# Read-only connection string that targets replicas
xata branch url main --type replica
```

**Aliases:** `xata branch connection-string`

## checkout

Check out a branch in this folder

Writes the branch to `.xata/` in this folder, so later commands run against it without being told which branch to use.

```bash theme={null}
xata branch checkout [--organization value] [--project value] [--branch value] [--database value] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to switch to

**Examples:**

```bash theme={null}
# Check out a branch of the current project
xata branch checkout main
# Check out a branch of another project
xata branch checkout feature-branch --organization org-123 --project proj-456
```

## tree

List all branches as a tree

```bash theme={null}
xata branch tree [--organization value] [--project value] [--branch value] [--show-id] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--show-id** `boolean` (default: false)

Show branch IDs in the tree

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata branch topology`

## get

Get a field from a branch description

Run it without a field to list the fields the description holds.

```bash theme={null}
xata branch get [--organization value] [--project value] [--branch value] [--profile value] [--debug] [--json] <[branch] field>...
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**\[branch] field** `string`

Branch name and/or field to get

**Examples:**

```bash theme={null}
# Read one field of the checked out branch
xata branch get status
# Read one field of another branch
xata branch get my-branch status
```

## metrics

Show CPU, memory and disk usage for a branch

Reports the metrics of every instance of the branch, the primary and any replicas, as a snapshot or continuously with --watch.

```bash theme={null}
xata branch metrics [--organization value] [--project value] [--branch value] [--since value] [--start value] [--end value] [--metrics value] [--instances value] [--aggregations value] [--aggregation avg|max|min] [--refresh value] [--output table|json|ndjson|tui] [--watch] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--since** `string`

Time range ending now, such as 1h, 24h, or 7d

**--start** `string`

Start time as an ISO timestamp

**--end** `string`

End time as an ISO timestamp

**--metrics** `string` (default: default)

Metrics to query: default, all, or a comma-separated list

**--instances** `string` (default: all)

Instances to query: all, primary, replicas, or comma-separated instance IDs

**--aggregations** `string` (default: avg,max,min)

Aggregations to query: comma-separated avg,max,min

**--aggregation** `avg | max | min` (default: avg)

Aggregation to render in table or TUI output

**--refresh** `string` (default: 10s)

Refresh interval for watch mode, such as 10s, 1m, or 500ms

**-o, --output** `table | json | ndjson | tui`

Output format. Defaults to table, or json when an AI agent runs the command.

**-w, --watch** `boolean` (default: false)

Refresh metrics continuously

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to show metrics for

**Examples:**

```bash theme={null}
# One-shot snapshot of the default metrics
xata branch metrics my-branch
# Continuously refresh metrics in the TUI
xata branch metrics my-branch --watch
# Stream NDJSON updates every 5 seconds for the primary instance
xata branch metrics my-branch -w --refresh 5s --instances primary --output ndjson
```

## set

Set a field value for a branch

The `postgres-version` field upgrades PostgreSQL, and only accepts compatible upgrades within the same major version and offering type, see [https://xata.io/docs/platform/branch#upgrading-postgresql-versions](https://xata.io/docs/platform/branch#upgrading-postgresql-versions).

```bash theme={null}
xata branch set [--organization value] [--project value] [--branch value] [--profile value] [--debug] [--json] [<field>] [<value>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**field** `string`

The field to set: name, description, replicas, instance-type, storage, hibernate, scale-to-zero, inactivity-period or postgres-version

**value** `string`

The value to set. Prompted for when omitted in an interactive terminal

**Examples:**

```bash theme={null}
# Set a field non-interactively
xata branch set replicas 2 my-branch
# Describe what the branch is for
xata branch set description "Nightly import"
# Clear the description
xata branch set description ""
# Select the target version interactively
xata branch set postgres-version
# Upgrade to a specific PostgreSQL version
xata branch set postgres-version postgres:17.7
```

## rotate-password

Rotate the database password for a branch

Reads the current username from the credentials endpoint, so an API key needs the `credentials:read` scope, see [https://xata.io/docs/cli#required-scopes](https://xata.io/docs/cli#required-scopes).

```bash theme={null}
xata branch rotate-password [--organization value] [--project value] [--branch value] [--yes] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--yes** `boolean` (default: false)

Do not ask for confirmation, assume yes.

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to rotate the password for

## wait-ready

Wait for a branch to be ready

Blocks until the branch is healthy, which is what a script needs after creating one or after a change that restarts it. A hibernated branch stays hibernated unless `--wake` is passed.

```bash theme={null}
xata branch wait-ready [--organization value] [--project value] [--branch value] [--wake] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--wake** `boolean`

Wake up the branch if it is hibernated

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to wait for

**Examples:**

```bash theme={null}
# Wait for a branch to come up
xata branch wait-ready my-branch
# Wake a hibernated branch and wait for it
xata branch wait-ready my-branch --wake
```

## query-insights

Inspect query statistics and active queries for a branch

Which statements are slow, which are expensive, and what is running right now. Historical statistics come from `pg_stat_statements`, and the queries running at this moment from `pg_stat_activity`. Running `xata branch qi` without a subcommand is the same as `xata branch qi list`. Except for `active`, which reads `pg_stat_activity`, these commands need `pg_stat_statements` loaded: run `xata branch query-insights enable <branch>` and then wait for the restart with `xata branch wait-ready <branch> --wake`. Unlike `logs` and `metrics`, which go through the API, these commands open a direct PostgreSQL connection: they cannot run against a branch that is hibernated or still provisioning, and an API key needs the `credentials:read` scope. The console reads the same `pg_stat_statements` data, see [https://xata.io/docs/platform/query-insights](https://xata.io/docs/platform/query-insights).

**Aliases:** `xata branch qi`

### query-insights list

List historical query statistics for a branch, by total execution time

In table output a leading `!` marks a row worth a look, a slow mean, spikes far above it, a low cache hit rate, heavy temporary file use or a very large row count, and the command prints the `show` command for each flagged query. Below the table it prints how many queries are being shown and, when more remain, the `list` command that fetches the next page. Both `json` and `ndjson` emit `{ "total", "limit", "offset", "queries" }`, with the statements under `queries`.

```bash theme={null}
xata branch query-insights list [--organization value] [--project value] [--branch value] [--search value] [--type value] [--performance value] [--db value] [--role value] [--sort total-time|mean-time|min-time|max-time|stddev-time|calls|rows|shared-hit|shared-read|shared-dirtied|shared-written|local-hit|local-read|local-dirtied|local-written|temp-read|temp-written|database|user] [--direction asc|desc] [--limit value] [--offset value] [--wide] [--output table|json|ndjson|tui] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--search** `string`

Search query text, database, or role

**--type** `string`

Comma-separated query types: SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, OTHER

**--performance** `string`

Comma-separated performance categories: fast, moderate, slow

**--db** `string`

Comma-separated database names to include

**--role** `string`

Comma-separated role names to include

**--sort** `total-time | mean-time | min-time | max-time | stddev-time | calls | rows | shared-hit | shared-read | shared-dirtied | shared-written | local-hit | local-read | local-dirtied | local-written | temp-read | temp-written | database | user` (default: total-time)

Sort by a query insight metric or dimension

**--direction** `asc | desc` (default: desc)

Sort direction

**--limit** `string`

Maximum number of rows to return. Defaults to 50 for human output and 1000 for --json.

**--offset** `string` (default: 0)

Number of rows to skip

**-w, --wide** `boolean` (default: false)

Show all pg\_stat\_statements metrics in human output

**-o, --output** `table | json | ndjson | tui`

Output format. Defaults to table, or json when an AI agent runs the command.

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to inspect

**Examples:**

```bash theme={null}
# The slowest statements by average execution time
xata branch query-insights list my-branch --performance slow --sort mean-time
# Feed the top 100 statements to another process
xata branch query-insights list my-branch --limit 100 --output ndjson | jq -r '.queries[].query'
```

**Aliases:** `xata branch query-insights ls`

### query-insights show

Show full query statistics for a query ID

A query ID is not unique on its own, the same normalized statement is recorded once per database and role that ran it. When the ID matches more than one row the command exits non-zero, re-run it with `--db` and `--role`. Human output ends with the potential issues found for the query.

```bash theme={null}
xata branch query-insights show [--organization value] [--project value] [--branch value] [--db value] [--role value] [--profile value] [--debug] [--json] <queryid> [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--db** `string`

Database name to disambiguate the query ID

**--role** `string`

Role name to disambiguate the query ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**queryid** `string` *required*

Query ID from pg\_stat\_statements

**branch** `string`

The branch to inspect

**Examples:**

```bash theme={null}
# Full detail for one query
xata branch query-insights show -6744440887696913970 --db postgres --role postgres my-branch
```

**Aliases:** `xata branch query-insights get`

### query-insights active

List currently running queries for a branch

Reads `pg_stat_activity`, so it works without `pg_stat_statements` being loaded. The columns are PID, Age, State, Wait, DB, User, Client and Query.

```bash theme={null}
xata branch query-insights active [--organization value] [--project value] [--branch value] [--watch value] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--watch** `string`

Refresh interval in seconds. Cannot be combined with --json

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to inspect

**Examples:**

```bash theme={null}
# What is running right now, refreshed every 5 seconds
xata branch query-insights active my-branch --watch 5
```

### query-insights enable

Enable pg\_stat\_statements for query insights on a branch

Adds `pg_stat_statements` to the preloaded libraries, which restarts the branch, and creates the extension. Run it, wait for the branch with `xata branch wait-ready <branch> --wake`, then run it again to create the extension. Operations that rebuild a branch, such as a migration into it, can drop the extension, so run this again if query insights stop returning rows. The same can be done from the console, by adding it to the preloaded libraries in the branch settings, see [https://xata.io/docs/platform/extensions](https://xata.io/docs/platform/extensions).

```bash theme={null}
xata branch query-insights enable [--organization value] [--project value] [--branch value] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to enable query insights for

### query-insights reset

Reset accumulated query statistics for a branch

Calls `pg_stat_statements_reset()`, which requires elevated privileges.

> [!WARNING]
>
> Resetting is branch-wide and cannot be undone, all historical counters are discarded and rebuild as new queries run.

```bash theme={null}
xata branch query-insights reset [--organization value] [--project value] [--branch value] [--yes] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--yes** `boolean` (default: false)

Do not ask for confirmation, assume yes.

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to reset query statistics for


# checkout
Source: https://xata.io/docs/cli/checkout

Check out a branch in this folder

Writes the branch to `.xata/` in this folder, so later commands run against it without being told which branch to use.

This command also takes `-h, --help`.

## checkout

Check out a branch in this folder

Writes the branch to `.xata/` in this folder, so later commands run against it without being told which branch to use.

```bash theme={null}
xata checkout [--organization value] [--project value] [--branch value] [--database value] [--profile value] [--debug] [--json] [<branch>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**branch** `string`

The branch to switch to

**Examples:**

```bash theme={null}
# Check out a branch of the current project
xata checkout main
# Check out a branch of another project
xata checkout feature-branch --organization org-123 --project proj-456
```


# clone
Source: https://xata.io/docs/cli/clone

Clone another PostgreSQL database with anonymization

Copies an external PostgreSQL database into a Xata branch, once with `start` or continuously with `stream`, and can anonymize columns on the way in. `config` writes the anonymization rules the other two read.

Every command below also takes `-h, --help`.

* [`xata clone start`](#start) — Snapshot a PostgreSQL database into a Xata branch
* [`xata clone config`](#config) — Write the anonymization rules that clone start and stream apply
* [`xata clone stream`](#stream) — Stream a PostgreSQL database into a Xata branch continuously

## start

Snapshot a PostgreSQL database into a Xata branch

Copies the source database into the branch once. The anonymization rules in `.xata/clone.yaml` are applied on the way in, and strict validation refuses to run until every table and column is covered by them.

```bash theme={null}
xata clone start [--config value] [--log-format value] [--log-level trace|debug|info|warn|error|fatal|panic] [--no-color] [--debug-profile] [--dump-file value] (--source-url value) [--organization value] [--project value] [--branch value] [--database value] [--filter-tables value] [--validation-mode strict|relaxed|prompt] [--role value] [--copy-roles] [--tune-target] [--profile value] [--debug] [--json]
```

**--config** `string`

.env or .yaml config file to use with pgstream if any

**--log-format** `string`

Log output format. One of console, json

**--log-level** `trace | debug | info | warn | error | fatal | panic` (default: info)

Log level for pgstream

**--no-color** `boolean`

Disable ANSI color codes in console log output (ignored when --log-format=json)

**--debug-profile** `boolean`

Whether to produce CPU and memory profile files, as well as exposing a /debug/pprof endpoint on localhost:6060

**--dump-file** `string`

File where the pg\_dump output will be written

**--source-url** `string` *required*

The source URL of the database to clone

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--filter-tables** `string` (default: *.*)

Tables to filter

**--validation-mode** `strict | relaxed | prompt` (default: prompt)

Anonymization validation mode, strict implies that all tables and columns should be specified

**--role** `string`

Postgres role to use for the clone

**--copy-roles** `boolean` (default: false)

Copy roles, owners, and privileges to the target

**--tune-target** `boolean` (default: false)

Temporarily tune the target branch for bulk loading, reverting the change when the clone stops. Raises max\_wal\_size on the branch and the maintenance settings on the index rebuild.

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## config

Write the anonymization rules that clone start and stream apply

Inspects the source database and writes `.xata/clone.yaml`, interactively or with AI, so the columns that carry personal data are transformed as they are copied.

```bash theme={null}
xata clone config (--source-url value) [--mode auto|prompt|web|ai] [--validation-mode strict|relaxed|prompt] [--organization value] [--project value] [--branch value] [--prompt value] [--model gemini-2.5-flash|claude-sonnet-4-6|claude-haiku-4-5-20251001] [--profile value] [--debug] [--json]
```

**--source-url** `string` *required*

The source URL of the database to clone

**--mode** `auto | prompt | web | ai` (default: prompt)

The assisting mode to help with the configuration generation

**--validation-mode** `strict | relaxed | prompt` (default: prompt)

Anonymization validation mode, strict implies that all tables and columns should be specified

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--prompt** `string`

Instructions for AI mode (e.g., which columns to anonymize, specific transformers to use)

**--model** `gemini-2.5-flash | claude-sonnet-4-6 | claude-haiku-4-5-20251001`

AI model, gemini-2.5-flash by default

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## stream

Stream a PostgreSQL database into a Xata branch continuously

Copies the tables `--filter-tables` selects first, then follows the source through logical replication, so the branch keeps up with it until the command is stopped.

```bash theme={null}
xata clone stream [--config value] [--log-format value] [--log-level trace|debug|info|warn|error|fatal|panic] [--no-color] [--debug-profile] [--dump-file value] (--source-url value) [--organization value] [--project value] [--branch value] [--database value] [--filter-tables value] [--validation-mode strict|relaxed|prompt] [--role value] [--copy-roles] [--skip-ddl-tracking] [--replication-slot value] [--profile value] [--debug] [--json]
```

**--config** `string`

.env or .yaml config file to use with pgstream if any

**--log-format** `string`

Log output format. One of console, json

**--log-level** `trace | debug | info | warn | error | fatal | panic` (default: info)

Log level for pgstream

**--no-color** `boolean`

Disable ANSI color codes in console log output (ignored when --log-format=json)

**--debug-profile** `boolean`

Whether to expose a /debug/pprof endpoint on localhost:6060

**--dump-file** `string`

File where the pg\_dump output will be written if initial snapshot is enabled

**--source-url** `string` *required*

The source URL of the database to stream from

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--filter-tables** `string` (default: *.*)

Tables to filter

**--validation-mode** `strict | relaxed | prompt` (default: prompt)

Anonymization validation mode, strict implies that all tables and columns should be specified

**--role** `string`

Postgres role to use for streaming (it should have at least REPLICATION privilege)

**--copy-roles** `boolean` (default: false)

Copy roles, owners, and privileges to the target

**--skip-ddl-tracking** `boolean` (default: false)

Skip DDL tracking during streaming. Useful for managed PostgreSQL services that do not support superuser access required for event triggers. Requires --replication-slot flag to be set with a pre-created replication slot on the source database.

**--replication-slot** `string`

Name of the replication slot on the source database. Overrides the default slot name

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.


# completions
Source: https://xata.io/docs/cli/completions

Install or uninstall shell completions for the Xata CLI

Only bash is supported, and the completions are picked up the next time the shell starts.

Every command below also takes `-h, --help`.

* [`xata completions install`](#install) — Installs bash autocomplete support for xata
* [`xata completions uninstall`](#uninstall) — Uninstalls bash autocomplete support for xata

## install

Installs bash autocomplete support for xata

```bash theme={null}
xata completions install [--profile value] [--debug] [--json]
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## uninstall

Uninstalls bash autocomplete support for xata

```bash theme={null}
xata completions uninstall [--profile value] [--debug] [--json]
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.


# console
Source: https://xata.io/docs/cli/console

Open an interactive branch console

This command also takes `-h, --help`.

## console

Open an interactive branch console

```bash theme={null}
xata console [--organization value] [--project value] [--branch value] [--database value] [--type primary|primary-or-replica|replica|pooler] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID

**--database** `string`

Database name

**--type** `primary | primary-or-replica | replica | pooler` (default: primary)

Connection type: primary (direct access to the primary), primary-or-replica (routed access to primary or replicas), replica (read-only access to replicas only), pooler (pooled access to the primary)

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.


# Command-line Interface
Source: https://xata.io/docs/cli/index

Complete reference for the Xata CLI

The Xata CLI is a powerful tool for managing your Xata databases, projects, and organizations from the command line. It provides commands for authentication, project management, database operations, and more.

## Installation

```bash theme={null}
# Install using the official install script
curl -fsSL https://xata.io/install.sh | bash
```

> [!TIP]
>
> The CLI binary will be installed to `~/.config/xata/bin/`. Make sure this directory is in your PATH.

## Installation (Docker)

```bash theme={null}
# Pull docker image
docker pull ghcr.io/xataio/xata:latest-pg18
```

> [!TIP]
>
> The container image supports the last 3 PostgreSQL major releases: 16, 17 and 18. To select one, edit the image suffix `-pg{majorversion}`.

## Installation (Windows)

```bash theme={null}
# Install using npm
powershell -c "irm https://xata.io/install.ps1 | iex"
```

> [!TIP]
>
> `xata clone` currently doesn't work with the native Windows installation. We currently recommend using WSL with the Linux binaries if you need the clone functionality on Windows. Track progress [here](https://github.com/xataio/pgstream/issues/409).

## Basic Usage

```bash theme={null}
xata [command] [subcommand] [options]
```

### Container image

```bash theme={null}
docker run -e XATA_API_KEY=yourkey ghcr.io/xataio/xata:latest-pg18 [command] [subcommand] [options]
```

## Available Commands

Every command page opens with an index of the commands on it.

* [`xata skill`](/docs/cli/skill) — Discover and install Xata agent skills
  * [`xata skill list`](/docs/cli/skill#list) — List available skills from xataio/skills
  * [`xata skill install`](/docs/cli/skill#install) — Install a skill; select interactively or list when no name is given
* [`xata init`](/docs/cli/init) — Link this folder to a project and branch
* [`xata auth`](/docs/cli/auth) — Authenticate with Xata
  * [`xata auth login`](/docs/cli/auth#login) — Log in to a Xata account
  * [`xata auth logout`](/docs/cli/auth#logout) — Log out of the current account
  * [`xata auth status`](/docs/cli/auth#status) — Display active account and authentication state
  * [`xata auth switch`](/docs/cli/auth#switch) — Switch to a different Xata account profile
  * [`xata auth list`](/docs/cli/auth#list) — List all available Xata account profiles
  * [`xata auth refresh`](/docs/cli/auth#refresh) — Refresh the access token of the current session
  * [`xata auth access-token`](/docs/cli/auth#access-token) — Print the current access token
  * [`xata auth refresh-token`](/docs/cli/auth#refresh-token) — Print the stored refresh token, without refreshing the session
* [`xata organization`](/docs/cli/organization) — Create, list, and manage organizations
  * [`xata organization list`](/docs/cli/organization#list) — List all organizations
  * [`xata organization describe`](/docs/cli/organization#describe) — Describe an organization
  * [`xata organization create`](/docs/cli/organization#create) — Create a new organization
  * [`xata organization delete`](/docs/cli/organization#delete) — Delete an organization
  * [`xata organization get`](/docs/cli/organization#get) — Get a field from an organization description
  * [`xata organization members`](/docs/cli/organization#members) — Manage organization members
    * [`xata organization members list`](/docs/cli/organization#members-list) — List all members of an organization
    * [`xata organization members invite`](/docs/cli/organization#members-invite) — Send an invitation to join an organization
    * [`xata organization members remove`](/docs/cli/organization#members-remove) — Remove a member from an organization
    * [`xata organization members set-role`](/docs/cli/organization#members-set-role) — Set the role of an organization member
  * [`xata organization invitations`](/docs/cli/organization#invitations) — Manage organization invitations
    * [`xata organization invitations list`](/docs/cli/organization#invitations-list) — List all invitations for an organization
    * [`xata organization invitations get`](/docs/cli/organization#invitations-get) — Get details of a specific invitation
    * [`xata organization invitations create`](/docs/cli/organization#invitations-create) — Create and send an invitation to join an organization
    * [`xata organization invitations delete`](/docs/cli/organization#invitations-delete) — Delete an invitation
    * [`xata organization invitations resend`](/docs/cli/organization#invitations-resend) — Resend an invitation
  * [`xata organization sso`](/docs/cli/organization#sso) — Configure single sign-on for an organization
    * [`xata organization sso show`](/docs/cli/organization#sso-show) — Show the domains and identity providers configured for an organization
    * [`xata organization sso domains`](/docs/cli/organization#sso-domains) — Claim and verify the email domains that sign in through your identity providers
      * [`xata organization sso domains add`](/docs/cli/organization#sso-domains-add) — Claim an email domain and print the DNS record that proves you own it
      * [`xata organization sso domains verify`](/docs/cli/organization#sso-domains-verify) — Check the DNS record for a claimed domain and mark it verified
      * [`xata organization sso domains remove`](/docs/cli/organization#sso-domains-remove) — Remove a claimed domain from an organization
    * [`xata organization sso providers`](/docs/cli/organization#sso-providers) — Connect the identity providers that sign in each verified domain
      * [`xata organization sso providers add`](/docs/cli/organization#sso-providers-add) — Connect an identity provider to a verified domain
      * [`xata organization sso providers remove`](/docs/cli/organization#sso-providers-remove) — Disconnect an identity provider from an organization
      * [`xata organization sso providers enforce`](/docs/cli/organization#sso-providers-enforce) — Require members on a domain to sign in through its identity provider
      * [`xata organization sso providers test`](/docs/cli/organization#sso-providers-test) — Print a link that tests sign-in through an identity provider
* [`xata project`](/docs/cli/project) — Create, list, and manage projects
  * [`xata project list`](/docs/cli/project#list) — List all projects
  * [`xata project describe`](/docs/cli/project#describe) — Describe a project
  * [`xata project create`](/docs/cli/project#create) — Create a new project
  * [`xata project delete`](/docs/cli/project#delete) — Delete a project
  * [`xata project init`](/docs/cli/project#init) — Link this folder to a project and branch
  * [`xata project get`](/docs/cli/project#get) — Get a field from a project description
  * [`xata project set`](/docs/cli/project#set) — Set a field value for a project
  * [`xata project ip-filter`](/docs/cli/project#ip-filter) — Manage IP filtering for a project
    * [`xata project ip-filter list`](/docs/cli/project#ip-filter-list) — Show IP filtering status and configured CIDR entries
    * [`xata project ip-filter enable`](/docs/cli/project#ip-filter-enable) — Enable IP filtering for a project
    * [`xata project ip-filter disable`](/docs/cli/project#ip-filter-disable) — Disable IP filtering for a project
    * [`xata project ip-filter add`](/docs/cli/project#ip-filter-add) — Add a CIDR entry to the IP filter allow list
    * [`xata project ip-filter remove`](/docs/cli/project#ip-filter-remove) — Remove a CIDR entry from the IP filter allow list
* [`xata branch`](/docs/cli/branch) — Create, list, and manage Xata branches
  * [`xata branch list`](/docs/cli/branch#list) — List all branches
  * [`xata branch describe`](/docs/cli/branch#describe) — Describe a branch
  * [`xata branch create`](/docs/cli/branch#create) — Create a new branch
  * [`xata branch delete`](/docs/cli/branch#delete) — Delete a branch
  * [`xata branch logs`](/docs/cli/branch#logs) — Retrieve the PostgreSQL logs of a branch
  * [`xata branch url`](/docs/cli/branch#url) — Print URL (connection string) for a branch
  * [`xata branch checkout`](/docs/cli/branch#checkout) — Check out a branch in this folder
  * [`xata branch tree`](/docs/cli/branch#tree) — List all branches as a tree
  * [`xata branch get`](/docs/cli/branch#get) — Get a field from a branch description
  * [`xata branch metrics`](/docs/cli/branch#metrics) — Show CPU, memory and disk usage for a branch
  * [`xata branch set`](/docs/cli/branch#set) — Set a field value for a branch
  * [`xata branch rotate-password`](/docs/cli/branch#rotate-password) — Rotate the database password for a branch
  * [`xata branch wait-ready`](/docs/cli/branch#wait-ready) — Wait for a branch to be ready
  * [`xata branch query-insights`](/docs/cli/branch#query-insights) — Inspect query statistics and active queries for a branch
    * [`xata branch query-insights list`](/docs/cli/branch#query-insights-list) — List historical query statistics for a branch, by total execution time
    * [`xata branch query-insights show`](/docs/cli/branch#query-insights-show) — Show full query statistics for a query ID
    * [`xata branch query-insights active`](/docs/cli/branch#query-insights-active) — List currently running queries for a branch
    * [`xata branch query-insights enable`](/docs/cli/branch#query-insights-enable) — Enable pg\_stat\_statements for query insights on a branch
    * [`xata branch query-insights reset`](/docs/cli/branch#query-insights-reset) — Reset accumulated query statistics for a branch
* [`xata keys`](/docs/cli/keys) — Create, list, and delete API keys
  * [`xata keys user`](/docs/cli/keys#user) — Manage API keys for the current user
    * [`xata keys user list`](/docs/cli/keys#user-list) — List all API keys of the current user
    * [`xata keys user create`](/docs/cli/keys#user-create) — Create a new API key
    * [`xata keys user delete`](/docs/cli/keys#user-delete) — Delete one or more API keys
  * [`xata keys organization`](/docs/cli/keys#organization) — Manage API keys for an organization
    * [`xata keys organization list`](/docs/cli/keys#organization-list) — List all API keys for an organization
    * [`xata keys organization create`](/docs/cli/keys#organization-create) — Create a new API key
    * [`xata keys organization delete`](/docs/cli/keys#organization-delete) — Delete one or more API keys
* [`xata roll`](/docs/cli/roll) — Run pgroll to manage PostgreSQL schema migrations
  * [`xata roll baseline`](/docs/cli/roll#baseline) — Create a baseline migration for an existing database schema
  * [`xata roll complete`](/docs/cli/roll#complete) — Complete an ongoing migration with the operations present in the given file
  * [`xata roll init`](/docs/cli/roll#init) — Initialize pgroll in the target database
  * [`xata roll latest`](/docs/cli/roll#latest) — Print the name of the latest schema version or migration
    * [`xata roll latest migration`](/docs/cli/roll#latest-migration) — Print the latest migration name
    * [`xata roll latest schema`](/docs/cli/roll#latest-schema) — Print the latest version schema name
  * [`xata roll migrate`](/docs/cli/roll#migrate) — Apply outstanding migrations from a directory to a database
  * [`xata roll update`](/docs/cli/roll#update) — Update outdated migrations in a directory
  * [`xata roll pull`](/docs/cli/roll#pull) — Pull migration history from the target database and write it to disk
  * [`xata roll rollback`](/docs/cli/roll#rollback) — Roll back an ongoing migration
  * [`xata roll start`](/docs/cli/roll#start) — Start a migration for the operations present in the given file
  * [`xata roll status`](/docs/cli/roll#status) — Show pgroll status
  * [`xata roll convert`](/docs/cli/roll#convert) — Convert SQL statements to a pgroll migration
* [`xata clone`](/docs/cli/clone) — Clone another PostgreSQL database with anonymization
  * [`xata clone start`](/docs/cli/clone#start) — Snapshot a PostgreSQL database into a Xata branch
  * [`xata clone config`](/docs/cli/clone#config) — Write the anonymization rules that clone start and stream apply
  * [`xata clone stream`](/docs/cli/clone#stream) — Stream a PostgreSQL database into a Xata branch continuously
* [`xata status`](/docs/cli/status) — Show the organization, project, branch, and database this folder uses
* [`xata version`](/docs/cli/version) — Get the version of the Xata CLI, pgroll and pgstream
* [`xata licenses`](/docs/cli/licenses) — Show third-party software notices and licenses bundled with the Xata CLI
* [`xata checkout`](/docs/cli/checkout) — Check out a branch in this folder
* [`xata scratch`](/docs/cli/scratch) — Run SQL or a Postgres client command against a temporary scratch branch
* [`xata console`](/docs/cli/console) — Open an interactive branch console
* [`xata upgrade`](/docs/cli/upgrade) — Upgrade the Xata CLI
* [`xata completions`](/docs/cli/completions) — Install or uninstall shell completions for the Xata CLI
  * [`xata completions install`](/docs/cli/completions#install) — Installs bash autocomplete support for xata
  * [`xata completions uninstall`](/docs/cli/completions#uninstall) — Uninstalls bash autocomplete support for xata

## Global Flags

**-h, --help** `boolean`

Print help information and exit

**-v, --version** `boolean`

Print version information and exit

**--json** `boolean`

Output in JSON format (where applicable)

## Required scopes

Most commands work with a key scoped to `branch:read`. These need more:

| Command | Scope | Why |
| - | - | - |
| [`branch logs`](/docs/cli/branch#logs) | `logs:read` | Reads server logs through the Xata API |
| [`branch url`](/docs/cli/branch#url) | `credentials:read` | Reads connection details from the credentials endpoint |
| [`branch rotate-password`](/docs/cli/branch#rotate-password) | `credentials:read` | Reads the current username before rotating |
| [`branch query-insights`](/docs/cli/branch#query-insights) | `credentials:read` | Opens a direct PostgreSQL connection to the branch |
| [`roll url`](/docs/cli/roll) | `credentials:read` | Reads connection details from the credentials endpoint |
| [`project init`](/docs/cli/project) | `credentials:read` | Reads connection details from the credentials endpoint |
| [`clone start`](/docs/cli/clone), [`stream`](/docs/cli/stream) | `credentials:read` | Connects to the source and target branches |

A key that only has `branch:read` returns a 401 on the commands above. Scopes are set when you create the key, so a key created before a scope existed does not have it. See [API keys](/docs/platform/api-key) for how to create and scope one.

Commands that manage members, invitations, roles, organization API keys or single sign-on also need an Admin of the organization. See [Organization roles](/docs/platform/organization-roles).

## Environment Variables

The Xata CLI can be configured using environment variables. These are useful for CI/CD pipelines, automation workflows, and advanced configuration.

### Authentication & Configuration

These variables apply globally to all CLI commands.

**XATA\_API\_KEY** `string`

API key used to authenticate with Xata. In CI/CD environments, set this as a secret. Applies to all commands that require authentication.

**XATA\_API\_ENVIRONMENT** `string`

Select the Xata API environment. Typically not needed unless instructed by Xata support. Applies to all commands.

**XATA\_CONFIG\_DIR** `string`

Override the directory where the CLI stores its configuration files. Applies to all commands.

### Project & Branch Configuration

These environment variables override the project and branch configuration normally stored in local config files. They apply to all commands that require project or branch context and are especially useful in CI/CD workflows where the CLI is not initialized interactively.

> [!NOTE]
>
> Earlier CLI versions used names without an underscore between words (for example `XATA_ORGANIZATIONID`, `XATA_PROJECTID`, `XATA_BRANCHID`, `XATA_BRANCHNAME`, `XATA_DATABASENAME`). These legacy names are still supported, but the snake\_case names below are preferred and take precedence when both are set.

**XATA\_ORGANIZATION\_ID** `string`

Your Xata organization ID.

**XATA\_PROJECT\_ID** `string`

Your Xata project ID.

**XATA\_BRANCH\_ID** `string`

The ID of the target branch.

**XATA\_BRANCH\_NAME** `string`

The name of the target branch.

**XATA\_DATABASE\_NAME** `string`

The database name (default: `xata`).

### Binary Version Overrides

The CLI ships with pinned versions of the `pgroll` and `pgstream` binaries. These environment variables allow you to override the pinned version, for example to test a newer release or to pin a specific version in your CI/CD pipeline. The CLI will automatically download the specified version if it is not already present locally.

**XATA\_PGROLL\_BINARY\_VERSION** `string`

Override the pinned `pgroll` binary version used by the CLI. If not set, the CLI uses its built-in default. Applies to all `xata roll` subcommands, `xata version`, and `xata upgrade`.

**XATA\_PGSTREAM\_BINARY\_VERSION** `string`

Override the pinned `pgstream` binary version used by the CLI. If not set, the CLI uses its built-in default. Applies to all `xata stream` subcommands, all `xata clone` subcommands, `xata version`, and `xata upgrade`.

### Clone & Stream

**XATA\_CLI\_SOURCE\_POSTGRES\_URL** `string`

Source PostgreSQL URL. Can be used instead of the `--source-url` flag. Applies to `xata clone start`, `xata clone stream`, and `xata stream destroy`.

### Networking

**XATA\_PRIVATE\_BRANCH\_TIMEOUT** `string`

Timeout in milliseconds for the private branch reachability check (default: `1000`). Set to `0` to disable the check entirely. Applies to all `xata roll` and `xata clone` subcommands.

### Examples

```bash theme={null}
# Override the pgroll binary version for a single command
XATA_PGROLL_BINARY_VERSION=0.16.2 xata roll status

# Set environment variables for a CI/CD pipeline
export XATA_API_KEY="your-api-key"
export XATA_ORGANIZATION_ID="your-org-id"
export XATA_PROJECT_ID="your-project-id"
export XATA_BRANCH_ID="your-branch-id"
xata roll migrate --complete
```


# init
Source: https://xata.io/docs/cli/init

Link this folder to a project and branch

Writes the organization, project, branch and database to `.xata/` in this folder, so the commands run here no longer need them passed in.

This command also takes `-h, --help`.

## init

Link this folder to a project and branch

Writes the organization, project, branch and database to `.xata/` in this folder, so the commands run here no longer need them passed in.

```bash theme={null}
xata init [--organization value] [--project value] [--branch value] [--database value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Examples:**

```bash theme={null}
# Link without prompts
xata init --organization <org-id> --project <project-id> --branch <branch-id>
```


# keys
Source: https://xata.io/docs/cli/keys

Create, list, and delete API keys

A user key belongs to the account that created it, an organization key to an organization. Both are printed once, when they are created.

Every command below also takes `-h, --help`.

* [`xata keys user`](#user) — Manage API keys for the current user
  * [`xata keys user list`](#user-list) — List all API keys of the current user
  * [`xata keys user create`](#user-create) — Create a new API key
  * [`xata keys user delete`](#user-delete) — Delete one or more API keys
* [`xata keys organization`](#organization) — Manage API keys for an organization
  * [`xata keys organization list`](#organization-list) — List all API keys for an organization
  * [`xata keys organization create`](#organization-create) — Create a new API key
  * [`xata keys organization delete`](#organization-delete) — Delete one or more API keys

## user

Manage API keys for the current user

### user list

List all API keys of the current user

```bash theme={null}
xata keys user list [--profile value] [--debug] [--json]
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata keys user ls`

### user create

Create a new API key

Creates a key for the account you are logged in with. It is printed once, when it is created, and cannot be read again.

```bash theme={null}
xata keys user create [--name value] [--expiry value] [--profile value] [--debug] [--json]
```

**--name** `string`

API key name

**--expiry** `string`

Expiry, as a date or a phrase such as 'in 1 week', or 'never'

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Examples:**

```bash theme={null}
# Create a key that does not expire
xata keys user create --name ci
# Create a key that expires
xata keys user create --name ci --expiry 2026-12-31
```

### user delete

Delete one or more API keys

```bash theme={null}
xata keys user delete [--profile value] [--debug] [--json] <keyId>...
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**keyId** `string`

IDs of the keys to delete

## organization

Manage API keys for an organization

**Aliases:** `xata keys org`

### organization list

List all API keys for an organization

```bash theme={null}
xata keys organization list [--organization value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata keys organization ls`

### organization create

Create a new API key

Creates a key for an organization rather than for an account. It is printed once, when it is created, and cannot be read again.

```bash theme={null}
xata keys organization create [--organization value] [--name value] [--expiry value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--name** `string`

API key name

**--expiry** `string`

Expiry, as a date or a phrase such as 'in 1 week', or 'never'

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Examples:**

```bash theme={null}
# Create a key for an organization
xata keys organization create --name deploy
```

### organization delete

Delete one or more API keys

```bash theme={null}
xata keys organization delete [--organization value] [--profile value] [--debug] [--json] <keyId>...
```

**--organization** `string`

Organization ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**keyId** `string`

IDs of the keys to delete


# licenses
Source: https://xata.io/docs/cli/licenses

Show third-party software notices and licenses bundled with the Xata CLI

Prints the NOTICE bundled with the CLI, which identifies the statically linked LGPL components (JavaScriptCore/WebKit, tinycc) embedded via the Bun runtime and how to obtain their corresponding source and relink. Use --full to also print the full license texts.

This command also takes `-h, --help`.

## licenses

Show third-party software notices and licenses bundled with the Xata CLI

Prints the NOTICE bundled with the CLI, which identifies the statically linked LGPL components (JavaScriptCore/WebKit, tinycc) embedded via the Bun runtime and how to obtain their corresponding source and relink. Use --full to also print the full license texts.

```bash theme={null}
xata licenses [--full] [--profile value] [--debug] [--json]
```

**--full** `boolean` (default: false)

Also print the full texts of the GNU LGPL-2.0, LGPL-2.1 and GPL-2.0 licenses

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.


# organization
Source: https://xata.io/docs/cli/organization

Create, list, and manage organizations

An organization owns projects and their billing, and the people who can reach them. Members belong to it, invitations bring them in.

Every command below also takes `-h, --help`.

* [`xata organization list`](#list) — List all organizations
* [`xata organization describe`](#describe) — Describe an organization
* [`xata organization create`](#create) — Create a new organization
* [`xata organization delete`](#delete) — Delete an organization
* [`xata organization get`](#get) — Get a field from an organization description
* [`xata organization members`](#members) — Manage organization members
  * [`xata organization members list`](#members-list) — List all members of an organization
  * [`xata organization members invite`](#members-invite) — Send an invitation to join an organization
  * [`xata organization members remove`](#members-remove) — Remove a member from an organization
  * [`xata organization members set-role`](#members-set-role) — Set the role of an organization member
* [`xata organization invitations`](#invitations) — Manage organization invitations
  * [`xata organization invitations list`](#invitations-list) — List all invitations for an organization
  * [`xata organization invitations get`](#invitations-get) — Get details of a specific invitation
  * [`xata organization invitations create`](#invitations-create) — Create and send an invitation to join an organization
  * [`xata organization invitations delete`](#invitations-delete) — Delete an invitation
  * [`xata organization invitations resend`](#invitations-resend) — Resend an invitation
* [`xata organization sso`](#sso) — Configure single sign-on for an organization
  * [`xata organization sso show`](#sso-show) — Show the domains and identity providers configured for an organization
  * [`xata organization sso domains`](#sso-domains) — Claim and verify the email domains that sign in through your identity providers
    * [`xata organization sso domains add`](#sso-domains-add) — Claim an email domain and print the DNS record that proves you own it
    * [`xata organization sso domains verify`](#sso-domains-verify) — Check the DNS record for a claimed domain and mark it verified
    * [`xata organization sso domains remove`](#sso-domains-remove) — Remove a claimed domain from an organization
  * [`xata organization sso providers`](#sso-providers) — Connect the identity providers that sign in each verified domain
    * [`xata organization sso providers add`](#sso-providers-add) — Connect an identity provider to a verified domain
    * [`xata organization sso providers remove`](#sso-providers-remove) — Disconnect an identity provider from an organization
    * [`xata organization sso providers enforce`](#sso-providers-enforce) — Require members on a domain to sign in through its identity provider
    * [`xata organization sso providers test`](#sso-providers-test) — Print a link that tests sign-in through an identity provider

## list

List all organizations

```bash theme={null}
xata organization list [--profile value] [--debug] [--json]
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata organization ls`

## describe

Describe an organization

```bash theme={null}
xata organization describe [--organization value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata organization view`, `xata organization show`

## create

Create a new organization

```bash theme={null}
xata organization create (--name value) [--profile value] [--debug] [--json]
```

**--name** `string` *required*

Organization Name

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## delete

Delete an organization

```bash theme={null}
xata organization delete [--organization value] [--yes] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--yes** `boolean` (default: false)

Do not ask for confirmation, assume yes.

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## get

Get a field from an organization description

```bash theme={null}
xata organization get [--organization value] [--profile value] [--debug] [--json] <[organization] field>...
```

**--organization** `string`

Organization ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**\[organization] field** `string`

Organization name and/or field to get

## members

Manage organization members

### members list

List all members of an organization

```bash theme={null}
xata organization members list [--organization value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata organization members ls`

### members invite

Send an invitation to join an organization

```bash theme={null}
xata organization members invite [--organization value] [--email value] [--role admin|editor] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--email** `string`

Email address to invite

**--role** `admin | editor`

Role the new member holds once they accept

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata organization members add`

### members remove

Remove a member from an organization

```bash theme={null}
xata organization members remove [--organization value] [--user-id value] [--force] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--user-id** `string`

ID of the user to remove

**--force** `boolean` (default: false)

Skip confirmation prompt

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata organization members delete`, `xata organization members rm`

### members set-role

Set the role of an organization member

```bash theme={null}
xata organization members set-role [--organization value] [--user-id value] [--role admin|editor] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--user-id** `string`

ID of the member

**--role** `admin | editor`

Role to grant

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## invitations

Manage organization invitations

### invitations list

List all invitations for an organization

```bash theme={null}
xata organization invitations list [--organization value] [--status value] [--email value] [--search value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--status** `string`

Filter by invitation status (pending or expired)

**--email** `string`

Filter by email address

**--search** `string`

Search invitations by email or name

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata organization invitations ls`

### invitations get

Get details of a specific invitation

```bash theme={null}
xata organization invitations get [--organization value] [--invitation-id value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--invitation-id** `string`

ID of the invitation to view

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata organization invitations show`

### invitations create

Create and send an invitation to join an organization

```bash theme={null}
xata organization invitations create [--organization value] [--email value] [--role admin|editor] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--email** `string`

Email address to invite

**--role** `admin | editor`

Role the new member holds once they accept

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata organization invitations add`, `xata organization invitations invite`

### invitations delete

Delete an invitation

```bash theme={null}
xata organization invitations delete [--organization value] [--invitation-id value] [--force] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--invitation-id** `string`

ID of the invitation to delete

**--force** `boolean` (default: false)

Skip confirmation prompt

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata organization invitations rm`, `xata organization invitations remove`

### invitations resend

Resend an invitation

```bash theme={null}
xata organization invitations resend [--organization value] [--invitation-id value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--invitation-id** `string`

ID of the invitation to resend

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## sso

Configure single sign-on for an organization

Claim an email domain, prove you own it with a DNS record, connect the identity provider its members sign in through, and then require it. Each verified domain has its own provider, so an organization can have several. See [https://xata.io/docs/platform/sso](https://xata.io/docs/platform/sso).

### sso show

Show the domains and identity providers configured for an organization

```bash theme={null}
xata organization sso show [--organization value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata organization sso get`

### sso domains

Claim and verify the email domains that sign in through your identity providers

#### sso domains add

Claim an email domain and print the DNS record that proves you own it

```bash theme={null}
xata organization sso domains add [--organization value] [--profile value] [--debug] [--json] [<domain>]
```

**--organization** `string`

Organization ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**domain** `string`

Domain to claim, such as acme.com

**Aliases:** `xata organization sso domains claim`

#### sso domains verify

Check the DNS record for a claimed domain and mark it verified

```bash theme={null}
xata organization sso domains verify [--organization value] [--profile value] [--debug] [--json] [<domain>]
```

**--organization** `string`

Organization ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**domain** `string`

Domain to verify, such as acme.com

#### sso domains remove

Remove a claimed domain from an organization

```bash theme={null}
xata organization sso domains remove [--organization value] [--yes] [--profile value] [--debug] [--json] [<domain>]
```

**--organization** `string`

Organization ID

**--yes** `boolean` (default: false)

Do not ask for confirmation, assume yes.

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**domain** `string`

Domain to remove, such as acme.com

**Aliases:** `xata organization sso domains delete`

### sso providers

Connect the identity providers that sign in each verified domain

#### sso providers add

Connect an identity provider to a verified domain

Reads the client secret from --client-secret, the XATA\_SSO\_CLIENT\_SECRET environment variable, or a prompt, so it need not appear in shell history. Provider types: google (Google Workspace), microsoft (Microsoft Entra ID), oidc (OpenID Connect). Every type except google needs --issuer-url.

```bash theme={null}
xata organization sso providers add [--organization value] [--type google|microsoft|oidc] [--domain value] [--issuer-url value] [--client-id value] [--client-secret value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--type** `google | microsoft | oidc`

Provider type

**--domain** `string`

Verified domain this provider signs in

**--issuer-url** `string`

Issuer URL, required for every type except google

**--client-id** `string`

Client ID issued by the identity provider

**--client-secret** `string`

Client secret issued by the identity provider

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata organization sso providers connect`

#### sso providers remove

Disconnect an identity provider from an organization

```bash theme={null}
xata organization sso providers remove [--organization value] [--yes] [--profile value] [--debug] [--json] [<alias>]
```

**--organization** `string`

Organization ID

**--yes** `boolean` (default: false)

Do not ask for confirmation, assume yes.

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**alias** `string`

Provider alias, as shown by xata organization sso show

**Aliases:** `xata organization sso providers delete`

#### sso providers enforce

Require members on a domain to sign in through its identity provider

Members on the domain lose the password form and the shared Google and GitHub buttons, so check the provider works with `xata organization sso providers test` before requiring it. Pass --disable to lift the requirement.

```bash theme={null}
xata organization sso providers enforce [--organization value] [--disable] [--yes] [--profile value] [--debug] [--json] [<alias>]
```

**--organization** `string`

Organization ID

**--disable** `boolean` (default: false)

Stop requiring SSO instead of requiring it

**--yes** `boolean` (default: false)

Do not ask for confirmation, assume yes.

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**alias** `string`

Provider alias, as shown by xata organization sso show

#### sso providers test

Print a link that tests sign-in through an identity provider

Members are only sent to a provider once SSO is required on its domain. The link signs in through the provider directly, so you can check it works before running `xata organization sso providers enforce`. Open it in a private window, since it signs that browser in as the account you test with.

```bash theme={null}
xata organization sso providers test [--organization value] [--profile value] [--debug] [--json] [<alias>]
```

**--organization** `string`

Organization ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**alias** `string`

Provider alias, as shown by xata organization sso show


# project
Source: https://xata.io/docs/cli/project

Create, list, and manage projects

A project holds branches and the defaults they inherit, such as whether they scale to zero and how long they stay idle first. IP filtering is set per project.

Every command below also takes `-h, --help`.

* [`xata project list`](#list) — List all projects
* [`xata project describe`](#describe) — Describe a project
* [`xata project create`](#create) — Create a new project
* [`xata project delete`](#delete) — Delete a project
* [`xata project init`](#init) — Link this folder to a project and branch
* [`xata project get`](#get) — Get a field from a project description
* [`xata project set`](#set) — Set a field value for a project
* [`xata project ip-filter`](#ip-filter) — Manage IP filtering for a project
  * [`xata project ip-filter list`](#ip-filter-list) — Show IP filtering status and configured CIDR entries
  * [`xata project ip-filter enable`](#ip-filter-enable) — Enable IP filtering for a project
  * [`xata project ip-filter disable`](#ip-filter-disable) — Disable IP filtering for a project
  * [`xata project ip-filter add`](#ip-filter-add) — Add a CIDR entry to the IP filter allow list
  * [`xata project ip-filter remove`](#ip-filter-remove) — Remove a CIDR entry from the IP filter allow list

## list

List all projects

```bash theme={null}
xata project list [--organization value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata project ls`

## describe

Describe a project

```bash theme={null}
xata project describe [--organization value] [--project value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Aliases:** `xata project view`, `xata project show`

## create

Create a new project

```bash theme={null}
xata project create [--organization value] (--name value) [--branch-name value] [--description value] [--instance-type value] [--replicas value] [--region value] [--storage value] [--postgres-version value] [--scale-to-zero-base true|false] [--scale-to-zero-child true|false] [--inactivity-period-base value] [--inactivity-period-child value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--name** `string` *required*

Project Name

**--branch-name** `string`

Branch Name

**--description** `string`

Short description of what the first branch is for

**--instance-type** `string`

Instance type for the first branch

**--replicas** `string`

Number of read replicas for the first branch

**--region** `string`

Region to create the project in

**--storage** `string`

Storage in GB for the first branch

**--postgres-version** `string`

Please select the PostgreSQL version for this branch

**--scale-to-zero-base** `true | false`

Default scale to zero status for base branches

**--scale-to-zero-child** `true | false`

Default scale to zero status for child branches

**--inactivity-period-base** `string`

Default inactivity period for base branches in whole minutes (1–1440)

**--inactivity-period-child** `string`

Default inactivity period for child branches in whole minutes (1–1440)

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## delete

Delete a project

```bash theme={null}
xata project delete [--organization value] [--project value] [--yes] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--yes** `boolean` (default: false)

Do not ask for confirmation, assume yes.

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## init

Link this folder to a project and branch

Writes the organization, project, branch and database to `.xata/` in this folder, so the commands run here no longer need them passed in.

```bash theme={null}
xata project init [--organization value] [--project value] [--branch value] [--database value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Examples:**

```bash theme={null}
# Link without prompts
xata project init --organization <org-id> --project <project-id> --branch <branch-id>
```

**Aliases:** `xata project connect`, `xata project switch`, `xata project link`

## get

Get a field from a project description

```bash theme={null}
xata project get [--organization value] [--project value] [--profile value] [--debug] [--json] <[project] field>...
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**\[project] field** `string`

Project name and/or field to get

## set

Set a field value for a project

The fields are name, scale-to-zero-base, scale-to-zero-child, inactivity-period-base and inactivity-period-child. The scale to zero and inactivity fields are the defaults new branches inherit, base for branches without a parent and child for the rest. Run it without a field to list them.

```bash theme={null}
xata project set [--organization value] [--project value] [--profile value] [--debug] [--json] [<field>] [<value>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**field** `string`

The field to set

**value** `string`

The value to set

**Examples:**

```bash theme={null}
# Let new child branches scale to zero
xata project set scale-to-zero-child true
```

## ip-filter

Manage IP filtering for a project

### ip-filter list

Show IP filtering status and configured CIDR entries

```bash theme={null}
xata project ip-filter list [--organization value] [--project value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

### ip-filter enable

Enable IP filtering for a project

```bash theme={null}
xata project ip-filter enable [--organization value] [--project value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

### ip-filter disable

Disable IP filtering for a project

```bash theme={null}
xata project ip-filter disable [--organization value] [--project value] [--profile value] [--debug] [--json]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

### ip-filter add

Add a CIDR entry to the IP filter allow list

```bash theme={null}
xata project ip-filter add [--organization value] [--project value] [--label value] [--profile value] [--debug] [--json] [<cidr>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--label** `string`

Label for the CIDR entry

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**cidr** `string`

IP address or CIDR block to add (e.g. 192.168.0.0/24 or 10.0.0.1)

**Examples:**

```bash theme={null}
# Allow a single address
xata project ip-filter add 203.0.113.4
# Allow a range and label it
xata project ip-filter add 203.0.113.0/24 --label office
```

### ip-filter remove

Remove a CIDR entry from the IP filter allow list

```bash theme={null}
xata project ip-filter remove [--organization value] [--project value] [--force] [--profile value] [--debug] [--json] [<cidr>]
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--force** `boolean` (default: false)

Skip confirmation prompts

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**cidr** `string`

CIDR block to remove


# roll
Source: https://xata.io/docs/cli/roll

Run pgroll to manage PostgreSQL schema migrations

> [!NOTE]
>
> The pgroll binary version is pinned by the CLI and can be overridden with the `XATA_PGROLL_BINARY_VERSION` environment variable.

Every command below also takes `-h, --help`.

* [`xata roll baseline`](#baseline) — Create a baseline migration for an existing database schema
* [`xata roll complete`](#complete) — Complete an ongoing migration with the operations present in the given file
* [`xata roll init`](#init) — Initialize pgroll in the target database
* [`xata roll latest`](#latest) — Print the name of the latest schema version or migration
  * [`xata roll latest migration`](#latest-migration) — Print the latest migration name
  * [`xata roll latest schema`](#latest-schema) — Print the latest version schema name
* [`xata roll migrate`](#migrate) — Apply outstanding migrations from a directory to a database
* [`xata roll update`](#update) — Update outdated migrations in a directory
* [`xata roll pull`](#pull) — Pull migration history from the target database and write it to disk
* [`xata roll rollback`](#rollback) — Roll back an ongoing migration
* [`xata roll start`](#start) — Start a migration for the operations present in the given file
* [`xata roll status`](#status) — Show pgroll status
* [`xata roll convert`](#convert) — Convert SQL statements to a pgroll migration

## baseline

Create a baseline migration for an existing database schema

```bash theme={null}
xata roll baseline [--lock-timeout value] [--pgroll-schema value] [--postgres-url value] [--role value] [--schema value] [--use-version-schema] [--verbose] [--organization value] [--project value] [--branch value] [--database value] [--json] [--yes] [--profile value] [--debug] [<version>] [<folder>]
```

**--lock-timeout** `string`

Postgres lock timeout in milliseconds for pgroll DDL operations

**--pgroll-schema** `string`

Postgres schema to use for pgroll internal state

**--postgres-url** `string`

Postgres URL

**--role** `string`

Optional postgres role to set when executing migrations

**--schema** `string`

Postgres schema to use for the migration

**--use-version-schema** `boolean`

Create version schemas for each migration

**--verbose** `boolean`

Enable verbose logging

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--json** `boolean`

Output in JSON format instead of YAML

**--yes** `boolean`

Skip confirmation prompt

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**version** `string`

Version name for the baseline, for example 01\_initial\_schema

**folder** `string`

The directory that contains the migrations

## complete

Complete an ongoing migration with the operations present in the given file

```bash theme={null}
xata roll complete [--lock-timeout value] [--pgroll-schema value] [--postgres-url value] [--role value] [--schema value] [--use-version-schema] [--verbose] [--organization value] [--project value] [--branch value] [--database value] [--profile value] [--debug] [--json]
```

**--lock-timeout** `string`

Postgres lock timeout in milliseconds for pgroll DDL operations

**--pgroll-schema** `string`

Postgres schema to use for pgroll internal state

**--postgres-url** `string`

Postgres URL

**--role** `string`

Optional postgres role to set when executing migrations

**--schema** `string`

Postgres schema to use for the migration

**--use-version-schema** `boolean`

Create version schemas for each migration

**--verbose** `boolean`

Enable verbose logging

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## init

Initialize pgroll in the target database

```bash theme={null}
xata roll init [--lock-timeout value] [--pgroll-schema value] [--postgres-url value] [--role value] [--schema value] [--use-version-schema] [--verbose] [--organization value] [--project value] [--branch value] [--database value] [--profile value] [--debug] [--json]
```

**--lock-timeout** `string`

Postgres lock timeout in milliseconds for pgroll DDL operations

**--pgroll-schema** `string`

Postgres schema to use for pgroll internal state

**--postgres-url** `string`

Postgres URL

**--role** `string`

Optional postgres role to set when executing migrations

**--schema** `string`

Postgres schema to use for the migration

**--use-version-schema** `boolean`

Create version schemas for each migration

**--verbose** `boolean`

Enable verbose logging

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## latest

Print the name of the latest schema version or migration

### latest migration

Print the latest migration name

```bash theme={null}
xata roll latest migration [--lock-timeout value] [--pgroll-schema value] [--postgres-url value] [--role value] [--schema value] [--use-version-schema] [--verbose] [--organization value] [--project value] [--branch value] [--database value] [--local value] [--profile value] [--debug] [--json]
```

**--lock-timeout** `string`

Postgres lock timeout in milliseconds for pgroll DDL operations

**--pgroll-schema** `string`

Postgres schema to use for pgroll internal state

**--postgres-url** `string`

Postgres URL

**--role** `string`

Optional postgres role to set when executing migrations

**--schema** `string`

Postgres schema to use for the migration

**--use-version-schema** `boolean`

Create version schemas for each migration

**--verbose** `boolean`

Enable verbose logging

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--local** `string`

Retrieve the latest migration from a local migration directory

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

### latest schema

Print the latest version schema name

```bash theme={null}
xata roll latest schema [--lock-timeout value] [--pgroll-schema value] [--postgres-url value] [--role value] [--schema value] [--use-version-schema] [--verbose] [--organization value] [--project value] [--branch value] [--database value] [--local value] [--profile value] [--debug] [--json]
```

**--lock-timeout** `string`

Postgres lock timeout in milliseconds for pgroll DDL operations

**--pgroll-schema** `string`

Postgres schema to use for pgroll internal state

**--postgres-url** `string`

Postgres URL

**--role** `string`

Optional postgres role to set when executing migrations

**--schema** `string`

Postgres schema to use for the migration

**--use-version-schema** `boolean`

Create version schemas for each migration

**--verbose** `boolean`

Enable verbose logging

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--local** `string`

Retrieve the latest version from a local migration directory

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## migrate

Apply outstanding migrations from a directory to a database

```bash theme={null}
xata roll migrate [--lock-timeout value] [--pgroll-schema value] [--postgres-url value] [--role value] [--schema value] [--use-version-schema] [--verbose] [--organization value] [--project value] [--branch value] [--database value] [--backfill-batch-delay value] [--backfill-batch-size value] [--complete] [--expect-one] [--profile value] [--debug] [--json] [<folder>]
```

**--lock-timeout** `string`

Postgres lock timeout in milliseconds for pgroll DDL operations

**--pgroll-schema** `string`

Postgres schema to use for pgroll internal state

**--postgres-url** `string`

Postgres URL

**--role** `string`

Optional postgres role to set when executing migrations

**--schema** `string`

Postgres schema to use for the migration

**--use-version-schema** `boolean`

Create version schemas for each migration

**--verbose** `boolean`

Enable verbose logging

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--backfill-batch-delay** `string`

Duration of delay between batch backfills (eg. 1s, 1000ms)

**--backfill-batch-size** `string`

Number of rows backfilled in each batch

**--complete** `boolean`

Complete the final migration rather than leaving it active

**--expect-one** `boolean`

Abort if there is more than one migration to be applied

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**folder** `string`

The directory that contains the migrations

## update

Update outdated migrations in a directory

```bash theme={null}
xata roll update [--lock-timeout value] [--pgroll-schema value] [--postgres-url value] [--role value] [--schema value] [--use-version-schema] [--verbose] [--organization value] [--project value] [--branch value] [--database value] [--json] [--profile value] [--debug] [<folder>]
```

**--lock-timeout** `string`

Postgres lock timeout in milliseconds for pgroll DDL operations

**--pgroll-schema** `string`

Postgres schema to use for pgroll internal state

**--postgres-url** `string`

Postgres URL

**--role** `string`

Optional postgres role to set when executing migrations

**--schema** `string`

Postgres schema to use for the migration

**--use-version-schema** `boolean`

Create version schemas for each migration

**--verbose** `boolean`

Enable verbose logging

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--json** `boolean`

Output migration file in JSON format instead of YAML

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**folder** `string`

The directory that contains the migrations

## pull

Pull migration history from the target database and write it to disk

```bash theme={null}
xata roll pull [--lock-timeout value] [--pgroll-schema value] [--postgres-url value] [--role value] [--schema value] [--use-version-schema] [--verbose] [--organization value] [--project value] [--branch value] [--database value] [--json] [--profile value] [--debug] [<folder>]
```

**--lock-timeout** `string`

Postgres lock timeout in milliseconds for pgroll DDL operations

**--pgroll-schema** `string`

Postgres schema to use for pgroll internal state

**--postgres-url** `string`

Postgres URL

**--role** `string`

Optional postgres role to set when executing migrations

**--schema** `string`

Postgres schema to use for the migration

**--use-version-schema** `boolean`

Create version schemas for each migration

**--verbose** `boolean`

Enable verbose logging

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--json** `boolean`

Output each migration in JSON format instead of YAML

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**folder** `string`

The target directory to pull migrations into

## rollback

Roll back an ongoing migration

```bash theme={null}
xata roll rollback [--lock-timeout value] [--pgroll-schema value] [--postgres-url value] [--role value] [--schema value] [--use-version-schema] [--verbose] [--organization value] [--project value] [--branch value] [--database value] [--profile value] [--debug] [--json]
```

**--lock-timeout** `string`

Postgres lock timeout in milliseconds for pgroll DDL operations

**--pgroll-schema** `string`

Postgres schema to use for pgroll internal state

**--postgres-url** `string`

Postgres URL

**--role** `string`

Optional postgres role to set when executing migrations

**--schema** `string`

Postgres schema to use for the migration

**--use-version-schema** `boolean`

Create version schemas for each migration

**--verbose** `boolean`

Enable verbose logging

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## start

Start a migration for the operations present in the given file

```bash theme={null}
xata roll start [--lock-timeout value] [--pgroll-schema value] [--postgres-url value] [--role value] [--schema value] [--use-version-schema] [--verbose] [--organization value] [--project value] [--branch value] [--database value] [--backfill-batch-delay value] [--backfill-batch-size value] [--complete] [--skip-validation] [--profile value] [--debug] [--json] <file>
```

**--lock-timeout** `string`

Postgres lock timeout in milliseconds for pgroll DDL operations

**--pgroll-schema** `string`

Postgres schema to use for pgroll internal state

**--postgres-url** `string`

Postgres URL

**--role** `string`

Optional postgres role to set when executing migrations

**--schema** `string`

Postgres schema to use for the migration

**--use-version-schema** `boolean`

Create version schemas for each migration

**--verbose** `boolean`

Enable verbose logging

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--backfill-batch-delay** `string`

Duration of delay between batch backfills (eg. 1s, 1000ms)

**--backfill-batch-size** `string`

Number of rows backfilled in each batch

**--complete** `boolean`

Mark the migration as complete

**--skip-validation** `boolean`

Skip migration validation

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**file** `string` *required*

The migration file to start

## status

Show pgroll status

```bash theme={null}
xata roll status [--lock-timeout value] [--pgroll-schema value] [--postgres-url value] [--role value] [--schema value] [--use-version-schema] [--verbose] [--organization value] [--project value] [--branch value] [--database value] [--profile value] [--debug] [--json]
```

**--lock-timeout** `string`

Postgres lock timeout in milliseconds for pgroll DDL operations

**--pgroll-schema** `string`

Postgres schema to use for pgroll internal state

**--postgres-url** `string`

Postgres URL

**--role** `string`

Optional postgres role to set when executing migrations

**--schema** `string`

Postgres schema to use for the migration

**--use-version-schema** `boolean`

Create version schemas for each migration

**--verbose** `boolean`

Enable verbose logging

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## convert

Convert SQL statements to a pgroll migration

```bash theme={null}
xata roll convert [--lock-timeout value] [--pgroll-schema value] [--postgres-url value] [--role value] [--schema value] [--use-version-schema] [--verbose] [--organization value] [--project value] [--branch value] [--database value] [--json] [--profile value] [--debug] <file>
```

**--lock-timeout** `string`

Postgres lock timeout in milliseconds for pgroll DDL operations

**--pgroll-schema** `string`

Postgres schema to use for pgroll internal state

**--postgres-url** `string`

Postgres URL

**--role** `string`

Optional postgres role to set when executing migrations

**--schema** `string`

Postgres schema to use for the migration

**--use-version-schema** `boolean`

Create version schemas for each migration

**--verbose** `boolean`

Enable verbose logging

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--branch** `string`

Branch ID or name

**--database** `string`

Database name on the branch

**--json** `boolean`

Output migration file in JSON format instead of YAML

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**file** `string` *required*

The migration file to convert


# scratch
Source: https://xata.io/docs/cli/scratch

Run SQL or a Postgres client command against a temporary scratch branch

Creates a branch from the parent given, runs what it is asked to, and deletes the branch afterwards, so a query or a migration can be tried against real data without touching an existing branch.

This command also takes `-h, --help`.

## scratch

Run SQL or a Postgres client command against a temporary scratch branch

Creates a branch from the parent given, runs what it is asked to, and deletes the branch afterwards, so a query or a migration can be tried against real data without touching an existing branch.

```bash theme={null}
xata scratch [--organization value] [--project value] [--parent-branch value] [--database value] [--execute value] [--profile value] [--debug] [--json] <command>...
```

**--organization** `string`

Organization ID

**--project** `string`

Project ID

**--parent-branch** `string`

Source branch ID or name for the scratch branch

**--database** `string`

Database name

**-x, --execute** `string`

SQL query to execute in the scratch branch

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**command** `string`

Binary command to run with scratch database environment variables; pass arguments to the binary after --

**Examples:**

```bash theme={null}
# Run SQL with the built-in client
xata scratch --execute "select count(*) from users"
# Run SQL using the short execute flag
xata scratch -x "select count(*) from users"
# Run psql with arguments
xata scratch -- psql -c "select count(*) from users"
# Run a database tool against the scratch branch
xata scratch -- npm run migrate
```


# skill
Source: https://xata.io/docs/cli/skill

Discover and install Xata agent skills

Every command below also takes `-h, --help`.

* [`xata skill list`](#list) — List available skills from xataio/skills
* [`xata skill install`](#install) — Install a skill; select interactively or list when no name is given

## list

List available skills from xataio/skills

```bash theme={null}
xata skill list [--profile value] [--debug] [--json]
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

## install

Install a skill; select interactively or list when no name is given

```bash theme={null}
xata skill install [--agent claude-code|codex|cursor|amp|github-copilot|antigravity-cli|opencode|windsurf|zed|cline] [--global] [--directory value] [--force] [--profile value] [--debug] [--json] [<name>]
```

**--agent** `claude-code | codex | cursor | amp | github-copilot | antigravity-cli | opencode | windsurf | zed | cline`

Target agent; repeat to select multiple agents

**--global** `boolean`

Install at user scope instead of project scope

**--directory** `string`

Project root (defaults to the current directory); not the final skill directory

**--force** `boolean`

Replace differing existing skills, discarding local edits

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**name** `string`

Skill name from xataio/skills


# status
Source: https://xata.io/docs/cli/status

Show the organization, project, branch, and database this folder uses

Reads the context from the `XATA_*` variables and the local config, so it is the quickest way to see which branch and database the commands run here will act on.

This command also takes `-h, --help`.

## status

Show the organization, project, branch, and database this folder uses

Reads the context from the `XATA_*` variables and the local config, so it is the quickest way to see which branch and database the commands run here will act on.

```bash theme={null}
xata status [--profile value] [--debug] [--json]
```

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.


# stream
Source: https://xata.io/docs/cli/stream

Run pgstream to stream PostgreSQL data to another destination

> [!NOTE]
>
> The pgstream binary version is pinned by the CLI and can be overridden with the `XATA_PGSTREAM_BINARY_VERSION` environment variable.

Every command below also takes `-h, --help`.

* [`xata stream destroy`](#destroy) — Remove the pgstream setup from a source database

## destroy

Remove the pgstream setup from a source database

Drops the replication slot along with the tables, functions and triggers pgstream created, and the pgstream schema itself.

```bash theme={null}
xata stream destroy [--config value] [--log-format value] [--log-level trace|debug|info|warn|error|fatal|panic] [--no-color] [--migrations-only] [--replication-slot value] [--slot-only] [--with-injector] (--source-url value) [--profile value] [--debug] [--json]
```

**--config** `string`

.env or .yaml config file to use with pgstream if any

**--log-format** `string`

Log output format. One of console, json

**--log-level** `trace | debug | info | warn | error | fatal | panic` (default: info)

Log level for pgstream

**--no-color** `boolean`

Disable ANSI color codes in console log output (ignored when --log-format=json)

**--migrations-only** `boolean`

Whether to only revert the database migrations

**--replication-slot** `string`

Name of the postgres replication slot to be deleted by pgstream from the source url

**--slot-only** `boolean`

Whether to only drop the replication slot, leaving the pgstream schema and migrations in place

**--with-injector** `boolean`

Whether to also destroy the injector related database objects

**--source-url** `string` *required*

The source PostgreSQL URL

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.


# upgrade
Source: https://xata.io/docs/cli/upgrade

Upgrade the Xata CLI

This command also takes `-h, --help`.

## upgrade

Upgrade the Xata CLI

```bash theme={null}
xata upgrade [--channel dev|next|latest] [--version value] [--profile value] [--debug] [--json]
```

**--channel** `dev | next | latest` (default: latest)

The channel to upgrade from

**--version** `string`

The version to upgrade to

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.

**Examples:**

```bash theme={null}
# Move to a specific version
xata upgrade --version 1.5.4
# Follow the pre-release channel
xata upgrade --channel next
```


# version
Source: https://xata.io/docs/cli/version

Get the version of the Xata CLI, pgroll and pgstream

This command also takes `-h, --help`.

## version

Get the version of the Xata CLI, pgroll and pgstream

```bash theme={null}
xata version [--skip-download] [--profile value] [--debug] [--json]
```

**--skip-download** `boolean` (default: false)

Skip downloading the pgroll/pgstream binaries

**--profile** `string`

The profile to use

**--debug** `boolean`

Print where each resolved value came from

**--json** `boolean`

Output in JSON format when the command supports it. Defaults to on when an AI agent runs the command.


# Data Anonymization
Source: https://xata.io/docs/core-concepts/anonymization

Sensitive data anonymization and masking for PostgreSQL

Data anonymization and masking provide significant benefits for ephemeral database environments, such as those created for development, testing, and staging. By transforming sensitive data into anonymized or masked formats, these techniques ensure that developers and testers work with realistic but secure datasets that do not expose actual personally identifiable information (PII) or confidential data.

This dramatically reduces the risk of data breaches and compliance violations, aligning with data protection regulations like GDPR and HIPAA. Furthermore, anonymization and masking simplify data handling in non-production environments by offering developers realistic data-sets based on the production data.

## How it works

`xata clone` is a tool for copying data efficiently between different PostgreSQL instances. Note that data cloning is different from branching. Branching is faster but only works within the Xata platform. `clone` can be used to copy data from other Postgres providers (e.g. AWS RDS) to Xata, as well as to copy data between different Xata instances.

`xata clone` offers the ability to anonymize and mask the data on the fly while it is being copied. The process for cloning is typically set up on a periodic schedule (for example, nightly).

## Transformers basics

The primary mechanism to configure anonymization are *transformers*. Transformers typically work at the column level and anonymize or mask the data depending on the column type and purpose.

For example, an "email" transformer can be used to anonymize email addresses, and is guaranteed to generate syntactically correct email addresses (`info@xata.io -> a2asd112@example.com`). Dedicated transformers are possible for many types of data, like names, addresses, phone numbers, IP addresses, and so on.

![Transformers](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/transformers.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=4ddd457c3b6a94098ab9d6d89d004681)

A transformer can have configuration options, for example, whether to keep the domain part of the email address (`info@xata.io -> a2asd112@xata.io`).

Transformers can be **deterministic** which means that the same input value will always generate the same output value. For example, `info@xata.io` always results `a2asd112@example.com` even if it shows up multiple times in the database. This is particular important for maintaining data integrity in relational databases.

Transformers can **mask** data. Masking means that parts of the input are hidden (`info@xata.io -> i***o@xata.io`).

Transformers can use **templating** to implement conditional logic (e.g. don't anonymize particular values) or to generate composed values (e.g. `LastName, FirstName`).

### Supported transformers

`xata clone` uses [pgstream](https://github.com/xataio/pgstream) under the hood, a project for PostgreSQL CDC and replication. `pgstream` implements it's own transformers library, but it can also import transformers from other open source projects, in particular: [Greenmask](https://www.greenmask.io/), [NeoSync](https://github.com/nucleuscloud/neosync) and [go-masker](https://github.com/ggwhite/go-masker).

The supported transformers are documented [here](/docs/opensource/pgstream/docs/transformers#supported-transformers).

It is also possible to write your own **custom transformers** in the Go programming language.

## Getting started

To create an anonymized production clone in Xata, follow these steps:

1. **Create and checkout the production clone branch:**

```bash theme={null}
xata branch create --name production-clone
xata checkout production-clone
```

2. **Configure anonymization transforms:**

```bash theme={null}
xata clone config --source-url postgres://user:pass@host:5432/prod_db --mode prompt
```

Follow the prompts to specify which columns should be transformed.

3. **Clone your production database into the production clone branch:**

```bash theme={null}
xata clone start --source-url postgres://user:pass@host:5432/prod_db 
```

This will copy your production data into the production clone branch, applying the anonymization rules you configured.


# Branching
Source: https://xata.io/docs/core-concepts/branching

Create instant, isolated PostgreSQL environments with Copy-on-Write database branches.

Branches on Xata are isolated PostgreSQL databases that give you separate environments for development, testing, and previews. Xata uses Copy-on-Write (CoW) at the storage layer to create branches instantly: a child branch contains the exact schema and data of its parent at the moment of creation, regardless of database size.

After creation, branches are independent PostgreSQL instances. Changes made in one branch — to the schema or the data — do not affect any other branch. There is no replication between branches, so writing to one branch never shows up in another.

Typical use cases include development branches for pull requests, preview databases, ephemeral CI environments, ad-hoc testing against production-like data, and "freezing" data at a point in time.

## Base and child branches

Every branch in a project is either a base branch or a child branch:

| Term | Meaning |
| - | - |
| **Base branch** | A branch without a parent. It starts as an empty PostgreSQL database, and you choose its region, instance type, and configuration at creation. |
| **Child branch** | A branch created from a parent branch. It starts with the parent's exact schema and data at the moment of creation, inherits the parent's configuration, and runs in the parent's region. |

When you create your first branch in a project, the console suggests the name `main`. This is a naming convention only — `main` is not a special or default branch, and any branch can be the parent of a child branch. Branches can be nested: a child branch can itself have children, forming a branch tree.

> [!NOTE]
>
> Each branch runs on its own compute instance, billed by instance type for the time it runs. [Scale-to-zero](/docs/core-concepts/scale-to-zero) pauses compute on idle branches, which makes development branches cheap to keep around. See [Costs and limits](#costs-and-limits).

> [!WARNING]
>
> **A branch's region is fixed at creation.** Child branches always run in the same region as their parent. To move to a different region, create a new base branch in the target region and copy your data over with [`xata clone`](/docs/cli/clone).

## Create a branch

### Create a child branch

Creating a child branch copies the parent's schema and data using a Copy-on-Write storage snapshot, so it completes in seconds even for terabyte-scale databases.

> [tab] Console
>
> **Step 1**
>
> Navigate to the **Branches** page in your project
>
> **Step 2**
>
> Click **New branch**
>
> **Step 3**
>
> Select the source (parent) branch
>
> **Step 4**
>
> Give the branch a name
>
> **Step 5**
>
> Optionally adjust the [scale-to-zero](/docs/core-concepts/scale-to-zero) settings
>
> **Step 6**
>
> Click **Create branch**

> [tab] CLI
>
> ```bash theme={null}
> xata branch create --name "my-dev-branch" --parent-branch "main"
> ```
>
> See the [`xata branch create`](/docs/cli/branch#create) reference for all options.

> [tab] API
>
> ```bash theme={null}
> curl -X POST "https://api.xata.tech/organizations/{organizationId}/projects/{projectId}/branches" \
>   -H "Authorization: Bearer YOUR_API_KEY" \
>   -H "Content-Type: application/json" \
>   -d '{
>     "name": "my-dev-branch",
>     "mode": "inherit",
>     "parentID": "PARENT_BRANCH_ID"
>   }'
> ```
>
> With `mode: "inherit"`, the new branch inherits the parent branch's configuration and data. See the API Reference for the full request schema.

A few things to know about child branches:

* The branch starts with a single primary instance and no read replicas, regardless of how many replicas the parent has. You can add replicas later in the branch settings.
* The branch is a point-in-time copy. Changes made to the parent after creation do not appear in the child, and vice versa.
* If the parent contains sensitive production data, so does the child. For development environments, consider branching from an [anonymized staging replica](/docs/tutorials/create-production-clone) instead of production.

### Create a base branch

A base branch has no parent and starts empty — no schema, no data. Use it for a new production database, a staging environment in a specific region, or as the root of a new branch tree.

> [tab] Console
>
> **Step 1**
>
> Navigate to the **Branches** page and click **New branch**
>
> **Step 2**
>
> Give the branch a name
>
> **Step 3**
>
> Select a region
>
> **Step 4**
>
> Choose the PostgreSQL image, instance type, and number of replicas
>
> **Step 5**
>
> Optionally adjust the [scale-to-zero](/docs/core-concepts/scale-to-zero) settings
>
> **Step 6**
>
> Click **Create branch**

> [tab] CLI
>
> ```bash theme={null}
> xata branch create --name "staging" --no-parent \
>   --region "us-east-1" --instance-type "xata.small" --replicas 0
> ```

> [!WARNING]
>
> **A branch with 0 replicas is not suitable for production workloads.** It has no automatic failover target. Configure at least 1 replica for production. Learn more in [High availability and replicas](/docs/core-concepts/deployment#high-availability-and-replicas).

## Connect to a branch

Each branch has its own connection string and credentials. When connecting, you choose an endpoint type that controls how traffic is routed within the branch:

| Endpoint type | Access | Use it for |
| - | - | - |
| **Primary** (`rw`) | Read/write on the primary | Default for most applications |
| **Primary or replica** (`r`) | Routed to primary or a replica | Read-only workloads that can use either; do not use for writes |
| **Replica only** (`ro`) | Read-only, replicas only | Keeping read load off the primary; requires at least one replica |
| **Pooler** (`pooled_rw`) | Pooled access to the primary | Serverless and high-concurrency workloads |

To get a connection string, open the branch's **Connect** panel in the console, or use the CLI:

```bash theme={null}
# Default (primary) connection string
xata branch url my-dev-branch

# Pooled connection for a serverless app
xata branch url my-dev-branch --type pooler
```

Store connection strings as secrets — each one contains the branch's credentials. For endpoint details and connection examples for `psql`, Prisma, Drizzle, and other clients, see the [branch reference](/docs/platform/branch#endpoint-types).

## View branches

The **Branches** page in the console shows all branches in your project, either as a list or as a graph visualizing the parent/child hierarchy, along with each branch's region and creation time.

From the CLI:

```bash theme={null}
# List all branches
xata branch list

# Show the branch hierarchy as a tree
xata branch tree

# Switch the CLI context to another branch
xata checkout my-dev-branch
```

## Manage a branch

Each branch has its own **Settings** page where you can rename the branch, change the instance type, add or remove read replicas, increase storage, upgrade the PostgreSQL minor version, tune PostgreSQL parameters, and rotate credentials. See the [branch reference](/docs/platform/branch) for details on each setting.

### Scale development branches to zero

Branches that only see occasional use — dev branches, PR previews, staging — don't need to run around the clock. With [scale-to-zero](/docs/core-concepts/scale-to-zero), a branch automatically hibernates after a configurable inactivity period (15 minutes to 3 hours) and wakes automatically when a new connection arrives. While hibernated, the branch's data is retained but no compute is billed.

Scale-to-zero is configured per branch, and your project defines separate defaults for base branches and child branches, so development branches can default to aggressive hibernation while production stays always-on.

### Delete a branch

Deleting a branch permanently removes the branch and its data. This cannot be undone.

> [tab] Console
>
> **Step 1**
>
> Navigate to the **Branches** page
>
> **Step 2**
>
> Open the actions menu ("...") next to the branch
>
> **Step 3**
>
> Select **Delete**
>
> **Step 4**
>
> Type the branch name to confirm the deletion

> [tab] CLI
>
> ```bash theme={null}
> xata branch delete my-dev-branch
> ```

Development branches are billed for the time they run, so delete branches you no longer need — or let [scale-to-zero](/docs/core-concepts/scale-to-zero) pause them automatically.

## Branching workflows

> [card] [Dev branches for pull requests](/docs/automations/github-app)
>
> Connect the Xata GitHub App to create a dedicated dev branch for every pull request, and clean it up when the PR closes.

> [card] [Branch automation in CI](/docs/automations/ga-pr)
>
> Use GitHub Actions to create branches, apply migrations, and wire preview environments to a branch connection string.

> [card] [Anonymized production clones](/docs/tutorials/create-production-clone)
>
> Clone production into a staging branch with PII anonymized on the fly, then create instant dev branches from it.

> [card] [Test schema changes safely](/docs/core-concepts/schema-changes)
>
> Try a migration on a child branch first, then apply the same migration to production with zero downtime using pgroll.

Branches do not merge or synchronize schema changes with each other. The workflow is: test the migration on a child branch, then apply the same migration separately to the target branch — manually or through CI. See [schema changes](/docs/core-concepts/schema-changes).

## How Copy-on-Write branching works

When a branching operation starts, Xata creates a Copy-on-Write snapshot of the underlying logical volume at the storage level, and starts a new PostgreSQL instance on a volume initialized from that snapshot.

Because the mechanism lives entirely below PostgreSQL, Xata doesn't require any modifications to the PostgreSQL source code — branches run vanilla Postgres.

In generic terms, [Copy-on-Write](https://en.wikipedia.org/wiki/Copy-on-write) is a mechanism used to share data efficiently between processes. Instead of copying data right away when multiple programs use it, the same data is shared between programs until one tries to modify it.

When stored on disk, the database volume is split into *blocks*. In the illustration below, the blocks are numbered from 1 to 8. Because we're using a distributed storage system, the blocks can be on different disks and nodes. A metadata index is used to keep track of where the data blocks are stored.

![Copy-on-Write branching — initial state](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/core-concepts/copy-on-write-diagram-1.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=76e697227752a113273067e3b5adf840)

When the Copy-on-Write volume snapshot is created, and the child database branch started from this snapshot, a new index is created. However, no data blocks are copied at this point; the index simply points to the original data blocks. This makes the snapshot instant, regardless of how large the database is.

![Copy-on-Write branching — new branch created](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/core-concepts/copy-on-write-branching-2.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=d8f94c67678a2a38fd5cde505220a85c)

After the branch is created, whenever writes are received by a data block — either by the child or the parent branch — that particular block is first copied. In the illustration below, this has happened for blocks 3 and 6. When a block diverges like this, the index from each branch references its own private copy of it.

![Copy-on-Write branching — blocks diverging after writes](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/core-concepts/copy-on-write-branching-3.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=6a4052b79ef9c9fdf3a63496721b1a30)

Only the blocks that are modified need to be copied, which results in significant I/O and storage savings compared to a full copy.

## Costs and limits

* Each running branch has its own compute instance, billed hourly by instance type. [Scale-to-zero](/docs/core-concepts/scale-to-zero) eliminates compute costs while a branch is idle.
* Copy-on-Write shares unchanged storage blocks between a parent and its children, so creating a branch does not duplicate the parent's storage.
* The number of branches per project and the branch creation rate depend on your plan. If you hit a limit, [contact support](https://xata.io/contact-support).

## Behavior and limitations

* **Do changes in the parent appear in a child branch?** No. A child is a point-in-time copy; there is no replication between branches after creation.
* **Can I merge branches?** No. Schema changes are promoted by applying the same [migration](/docs/core-concepts/schema-changes) to each branch, manually or via CI.
* **Can I reset a child branch from its parent?** Not in place. Delete the branch and create a fresh one from the parent to get a new point-in-time copy.
* **Is `main` special?** No. It's the conventional name for the first branch, but Xata has no default-branch or promotion concept — any branch can be a parent.
* **Can I change a branch's region?** No. Region is fixed at creation, and child branches always live in the parent's region. Create a new base branch and [clone](/docs/cli/clone) your data to move regions.


# Deployment Models
Source: https://xata.io/docs/core-concepts/deployment

The various ways you can deploy Xata, whether for staging/dev or production, hosted in our cloud or in your cloud account.

You can use Xata in two major ways:

* **For staging and development use cases**, where you use Xata to enhance other Postgres DBaaS offerings by adding features like instant database branching and data anonymization.
* **For production use cases**, where you run your production Postgres as well as any development branches on our platform.

In either case, you can you can deploy PostgreSQL instances on our cloud, or in your cloud account or even on-prem (BYOC).

## Production clone and development branches on Xata

In this deployment model, we create a "production clone" of your production Postgres on the Xata platform. This works with
any Postgres service (AWS RDS, Amazon Aurora, GCP Cloud SQL, Azure Database, etc.). The production clone is updated with
the production dataset periodically, for example every night.  We call this process “cloning,” and it is accomplished using our
open-source CLI (`xata clone`). You can automate the cloning process using GitHub Actions or any other CI/CD tooling.

Anonymization can be done during the cloning phase. Depending on your requirements, you can set anonymization in `strict`
or `relaxed` mode. In `strict` mode, any new table and column needs to be explicitly present in the configuration,
minimizing the chance of accidental leaks.

![Xata for staging and dev use-cases](https://mintcdn.com/xata/zunLvnbMMVhTIsZq/images/deployment-models/xata-deployment-staging.png?fit=max\&auto=format\&n=zunLvnbMMVhTIsZq\&q=85\&s=19ef06a16fb3d83a6d6bff838395b5b1)

Because `xata clone` is executed in the context of your environment (e.g. you cloud account), there are two important benefits:

* The tool has access to your production database without requiring changes to your external firewall rules.
* Sensitive data is anonymized before it ever leaves your cloud account.

From the "production clone", you can create instant branches with all the data, for your development previews, ephemeral environments,
or ad-hoc testing needs.

## Production and development branches on Xata

In this deployment model, you run both production and any staging or development branches you require on the Xata platform. Your production
branch is typically configured for high availability (with one or multiple read replicas). Development branches generally use a single
instance and are configured with lower CPU and memory resources to save costs.

If you'd like to set up anonymization between your production and development branches, as well as have stricter controls on who can access production, you can use two Xata organizations. In the production organization you run only the production branch. You can configure `xata clone` to copy and anonymize the data between prod and staging.

![Production and staging/dev branches on the Xata platform](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/deployment-models/xata-prod-branches-diagram.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=1e26d43a7395275eaa34781e6a0b4574)

### High availability and replicas

Each branch has one primary PostgreSQL instance and can have zero or more replicas. Replicas can serve read-only traffic and act as standby promotion targets.

Configure at least 1 replica for production workloads. If the primary becomes unavailable, Xata can automatically promote a healthy replica. During maintenance or an upgrade that requires replacing the primary, Xata can first switch over to a replica to minimize disruption. Failovers and planned switchovers can briefly interrupt existing connections, so applications must reconnect and retry transient failures as appropriate.

> [!WARNING]
>
> **A branch with 0 replicas is not suitable for production workloads.** It has no standby promotion target and cannot automatically fail over. A primary failure, or an update that requires the primary to restart or be replaced, makes the database unavailable until the primary is available again.

#### Connection endpoints

Choose an endpoint based on where each connection should go:

* **Primary only (`rw`)**: Connects to the primary for reads and writes.
* **Primary or replica (`r`)**: Connects to either the primary or a replica. Use it only for read-only workloads; writes are unreliable because the connection can land on a read-only replica.
* **Replica only (`ro`)**: Connects only to read-only replicas and requires at least 1 replica.
* **Pooler (`pooled_rw`)**: Provides pooled connections to the primary.

For connection string options, see [Branch Endpoint Types](/docs/platform/branch#endpoint-types).

#### Configuring replicas

* **During branch creation**: Select the replica count when creating a base branch in the Xata console, or pass `--replicas` to [`xata branch create`](/docs/cli/branch#create).
* **On existing branches**: Open **Settings** → **Instance Details** and adjust **Replicas**.

## Bring Your Own Cloud (BYOC)

In this deployment model, we install the whole dataplane part of Xata in a Kubernetes setup that you control. This can be in your
own cloud account (AWS, Azure, GCP, Hetzner, etc.) or on-premises. The minimal setup requires three Kubernetes nodes, and there are multiple
installation models (disaggregated, hyper-converged, or hybrid). Our engineers will help you choose the best setup and configure it for you.

This model brings the following benefits:

* **Security, compliance, and data governance.** This setup ensure that all Postgres databases are started in your usual production environment.
* **Better use of your cloud provider credits.** Since you pay for the Kubernetes nodes directly to your cloud provider, you can take advantage fully of your cloud credits, discounts, and negotiated private agreements.
* **Throughput and latency.** Depending on your hardware and networking setup, this deployment model can achieve levels of performance not possible when using a separate cloud account.

![BYOC deployment model](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/deployment-models/byoc-architecture.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=f89e157ddc85ef95ac6dda6b13526051)

The Xata control plane—responsible for managing organizations, users, regions, and instances—continues to run in our AWS account. Latency between
the control and data planes is not critical, as it involves only control messages (e.g., start/stop instance). By keeping the control plane in
our account (and optionally also the monitoring/observability data), we can provide you with a managed service similar to what you'd get
when deploying on our platform.

This setup is especially convenient if you want to offer an internal “Postgres as a service” or if you’re looking to white-label Xata for your customers.

If you have any question regarding BYOC, do not hesitate to write us [an email](mailto:info@xata.io).


# Scale to Zero
Source: https://xata.io/docs/core-concepts/scale-to-zero

Minimize costs by hibernating inactive database branches

## Overview

Xata's Scale to Zero feature allows you to minimize costs by hibernating database branches during periods of inactivity. This is particularly useful for development, testing and staging environments that don't need to be active 24/7.

When a branch is hibernated:

* Compute is paused and no active database connections are allowed
* Your data remains safe and accessible
* You only pay for storage, not compute costs
* Branches can be reactivated immediately

Xata offers two different ways of implementing scale to zero for your branches.

* Manual hibernation
* Automatic scale to zero

At the moment, you can enable either manual hibernation or scale to zero, but not both simultaneously.

## Manual Hibernation

Currently, you can manually hibernate branches through the Xata Console:

### How to Hibernate a Branch

1. Go to the Compute section under Branch Settings in the Xata Console
2. Toggle "Hibernate branch" to enabled
3. Click "Save compute settings"

### Hibernation Status

You can monitor hibernation status in the Xata Console:

* Active branches are running and accepting connections
* Hibernated branches are paused and not accepting connections

### Waking Up a Manually Hibernated Branch

To reactivate a hibernated branch:

1. Go to the Compute section under Branch Settings in the Xata Console
2. Toggle "Hibernate branch" to "off"
3. Click "Save compute settings"
4. The branch will immediately start accepting connections

Alternatively you can click the "Wake up branch" button in the branch overview to wake up a Hibernated Branch. You can also use the "Wake up branch" button available in the SQL Editor, Extensions and Schemas tabs.

## Automatic Scale to Zero

Automatic scale to zero allows you to hibernate a branch after a configurable period of inactivity and automatically reactivate it when you attempt to connect to it.

### How to Enable Scale to Zero for a Branch

You can configure your scale to zero settings at the project level:

1. Go to the Compute section under Project Settings in the Xata Console
2. Toggle "Enable for base branch" if you want to enable scale to zero for your base branch by default
3. Toggle "Enable for child branches" if you want to enable scale to zero for child branches by default
4. Configure the relevant inactivity period for base/child branches
5. Click "Save settings"

The project scale to zero settings will be the default when creating new branches within the project. However, you will also be prompted at branch creation time in case you want to update those settings on a per branch basis. By default, these project level settings are only applied to new branches. If you would like existing branches to also be affected by these settings, you can check the "Apply these settings to existing branches" box.

Alternatively, you can also update the scale to zero settings once the branch has been created:

1. Go to the Compute section under Branch Settings in the Xata Console
2. Toggle "Scale to Zero" to enabled
3. Configure the inactivity period for the branch (default to 30min)
4. Click "Save compute settings"
5. The branch will now start tracking the open connections, and will automatically scale to zero once the configured period of inactivity has passed.

### Waking Up a Scaled to Zero Branch

A branch that has been automatically scaled to zero can be reactivated:

* By opening a connection to the branch (ex. connecting to the branch via psql)
* Manually, by clicking on the "Wake up branch" icon that appears on the branch overview or any other branch tabs that allow it.

> [!NOTE]
>
> When attempting to connect to a hibernated branch via the Serverless Proxy (HTTP or WebSocket), you'll receive a `409 Conflict` error with code `BRANCH_HIBERNATED` and the message "branch is hibernated, reactivate it to continue". The branch will need to be manually reactivated before connections can be established.

## Branching off a Hibernated Branch

It is possible to create a child branch from a hibernated base branch (regardless of it being manually hibernated or scaled to zero). The base branch will not be woken up, keeping the costs to a minimum during the branching process.

## When to use Scale to Zero

### Development Branches

Perfect for feature branches that are only used during active development:

* You can hibernate branches when not actively developing
* Branches wake up instantly when you start working
* This reduces costs for long-running feature development

### Staging Environments

Ideal for staging databases that mirror production:

* You can hibernate staging environments when not testing
* Branches wake up for deployment testing
* This maintains production-like data without 24/7 costs

### Testing Environments

Great for automated testing databases:

* You can hibernate between test runs
* Branches wake up for CI/CD pipelines
* This optimizes costs for comprehensive testing


# Schema Changes
Source: https://xata.io/docs/core-concepts/schema-changes

Zero-downtime schema migrations with automatic rollback support

Schema changes are a critical part of maintaining a healthy and evolving database. Whether it's adding new columns, creating indexes for faster queries, or normalizing tables to reduce data duplication, schema changes keep the database in sync with the application's needs.

However, schema changes in PostgreSQL come with significant challenges. Applying them in a production environment without downtime can be tricky. Changes to large tables, such as adding not-null unique columns or modifying data types, can lock the table for an extended period of time, essentially causing downtime.

Orchestrating application and schema changes in PostgreSQL is tricky because both need to evolve together without breaking functionality. For example, renaming a column is a simple operation from the Postgres point of view, however it is difficult to accomplish in a backwards compatible way for the application.

You can use any schema changes or schema migrations tool with Xata, however we integrate `pgroll` to solve these challenges. It is accessible directly in the CLI with the [`xata roll`](/docs/cli/roll) command.

## pgroll - Zero downtime, reversible, schema changes for PostgreSQL

[pgroll](https://github.com/xataio/pgroll) is a schema migration tool for Postgres. It is designed for application developers working on applications that require frequent schema changes but also need to maintain zero downtime around those schema changes.

There are two aspects that characterize `pgroll`'s approach to migrations.

### Multi-version migrations

Making a schema change with `pgroll` results in two versions of the schema; the one before the change and the one after the change - this allows applications to select which version of the schema they want to work with and allows side-by-side rollout of applications that require the new schema changes with old applications that may be incompatible with it.

![Schema changes flow](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/schema-changes-flow.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=0fef866af4697da52e5b38f83f96f8d4)

### Lock-safe migrations

Migrations using `pgroll` are expressed declaratively, rather than using SQL directly. This allows `pgroll` to implement the steps required to perform the schema change in a safe manner, ensuring that any locks required on the affected objects are held for the shortest possible time.

***

For more details about how `pgroll` works, please visit its [documentation](https://pgroll.com/docs/latest/getting-started).


# Serverless Proxy
Source: https://xata.io/docs/core-concepts/serverless-proxy

Connect to your Xata database via HTTP and WebSocket connections for serverless and edge environments.

The Serverless Proxy enables you to connect to your Xata PostgreSQL database using HTTP and WebSocket protocols, in addition to the standard PostgreSQL wire protocol (TCP). This is ideal for serverless functions and edge environments where maintaining persistent TCP connections is not practical.

> [!NOTE]
>
> Xata implements the [Neon serverless driver protocol](https://github.com/neondatabase/serverless), allowing you to use the `@neondatabase/serverless` npm package to connect to your Xata database over HTTP or WebSocket. If you're migrating from Neon, and you are using the serverless driver, you don't need any code changes.

## Why use the Serverless Proxy?

Traditional PostgreSQL connections use TCP, which requires maintaining a persistent connection. This works well for long-running servers but presents challenges in serverless environments:

* **Cold starts**: Serverless functions spin up and down frequently, making persistent connections impractical
* **Edge locations**: Edge functions often have restrictions on outbound TCP connections
* **Connection limits**: Each serverless invocation could exhaust your database connections

The Serverless Proxy solves these problems by allowing stateless HTTP requests and efficient WebSocket connections to your database.

## Connection methods

Xata supports three ways to connect to your database:

| Method | Protocol | Best for |
| - | - | - |
| Standard | TCP (wire protocol) | Traditional servers, long-running applications |
| HTTP | HTTPS | Serverless functions, one-off queries |
| WebSocket | WSS | Edge functions, interactive applications |

## Connection endpoints

When connecting to your Xata database, you can use different endpoints to route your traffic:

| Endpoint | Description | Use case |
| - | - | - |
| `{branch-id}` (no suffix) | Default endpoint | Routes through PgBouncer for connection pooling |
| `{branch-id}-rw` | Read-write endpoint | Routes directly to the primary instance for read and write operations |
| `{branch-id}-ro` or `{branch-id}-r` | Read-only endpoint | Routes to read replicas for read operations only |
| `{branch-id}-pooler` | Connection pooler endpoint | Routes through PgBouncer for connection pooling |

> [!NOTE]
>
> When connecting through the Serverless Proxy (HTTP and WebSocket), hostnames without an explicit endpoint suffix are routed through the PgBouncer connection pooler by default. To bypass the pooler and connect directly to the primary instance, use the `-rw` suffix.

The connection pooler uses PgBouncer to manage database connections efficiently. This is particularly useful for applications that create many short-lived connections, as the pooler maintains a pool of persistent connections to the database and reuses them across client requests. Routing serverless traffic through the pooler by default keeps the number of connections to PostgreSQL low, even when many serverless functions invoke queries concurrently.

## Get started

Install the `@neondatabase/serverless` driver:

```bash theme={null}
npm install @neondatabase/serverless
```

Set the `DATABASE_URL` environment variable to your connection string. You can obtain it by running `xata branch url` or by copying the connection string from the Xata console:

```bash theme={null}
export DATABASE_URL=$(xata branch url)
```

## HTTP vs WebSocket

The driver supports two modes of communication, each suited to different use cases:

| Feature | HTTP | WebSocket |
| - | - | - |
| Connection overhead | None (stateless) | One-time handshake |
| Latency per query | Higher (new request each time) | Lower (reuses connection) |
| Transactions | Not supported | Supported |
| Session variables | Not supported | Supported |
| Prepared statements | Not supported | Supported |
| Multiple queries | One per request | Batched efficiently |
| Best for | Single, simple queries | Complex operations |

**Use HTTP when:**

* You only need to run a single query per request
* Your queries are simple SELECTs or single INSERTs
* You want the simplest possible setup with zero connection management

**Use WebSocket when:**

* You need to run transactions (`BEGIN`/`COMMIT`/`ROLLBACK`)
* You're executing multiple queries in a single request
* You need session-level features like `SET` commands or prepared statements
* You want lower per-query latency for multiple operations

## HTTP connections

Use HTTP for simple, stateless queries. Each query is a single HTTP request—no connection management required.

```typescript theme={null}
import { neon } from '@neondatabase/serverless';

const sql = neon(process.env.DATABASE_URL);

const users = await sql`SELECT * FROM users WHERE id = ${userId}`;
```

> [!NOTE]
>
> HTTP connections are stateless. Each query runs in its own transaction and cannot share state with other queries. Use WebSocket connections if you need transactions or session-level features.

### Error handling

#### Query timeout

HTTP queries have a 30-second timeout limit. If your query exceeds this limit, you'll receive a `504 Gateway Timeout` error with the code `QUERY_TIMEOUT`. For long-running queries, consider:

* Optimizing your query with indexes or query restructuring
* Breaking the operation into smaller chunks
* Using WebSocket connections for complex operations

#### Hibernated branches

If you attempt to connect to a hibernated branch, you'll receive a `409 Conflict` error with the code `BRANCH_HIBERNATED` and the message "branch is hibernated, reactivate it to continue". You'll need to manually reactivate the branch before connections can be established. Learn more about [Scale to Zero](/docs/core-concepts/scale-to-zero).

## WebSocket connections

Use WebSocket connections when you need transactions, session variables, or want to execute multiple queries efficiently. The WebSocket mode maintains a persistent connection, allowing for stateful interactions with the database.

> [!NOTE]
>
> If you are running in Node.js versions earlier than 21, you need to supply a WebSocket implementation. Install the `ws` package (`npm install ws`) and configure it before creating a pool:
>
> ```typescript theme={null}
> import { neonConfig } from '@neondatabase/serverless';
> import ws from 'ws';
>
> neonConfig.webSocketConstructor = ws;
> ```
>
> This is not needed in edge runtimes (Vercel Edge, Cloudflare Workers, Deno) or Node.js 21+, which have built-in WebSocket support.

```typescript theme={null}
import { Pool } from '@neondatabase/serverless';

const pool = new Pool({ connectionString: process.env.DATABASE_URL });

const client = await pool.connect();
try {
  await client.query('BEGIN');
  await client.query('INSERT INTO orders (user_id) VALUES ($1)', [userId]);
  await client.query('UPDATE inventory SET quantity = quantity - 1 WHERE product_id = $1', [productId]);
  await client.query('COMMIT');
} catch (e) {
  await client.query('ROLLBACK');
  throw e;
} finally {
  client.release();
}
```

## Framework examples

> [tab] Vercel
>
> ```typescript theme={null}
> import { neon } from '@neondatabase/serverless';
>
> export async function GET() {
>   const sql = neon(process.env.DATABASE_URL!);
>   const users = await sql`SELECT * FROM users LIMIT 10`;
>   return Response.json(users);
> }
> ```

> [tab] Cloudflare Workers
>
> ```typescript theme={null}
> import { neon } from '@neondatabase/serverless';
>
> export default {
>   async fetch(request: Request, env: Env) {
>     const sql = neon(env.DATABASE_URL);
>     const users = await sql`SELECT * FROM users LIMIT 10`;
>     return Response.json(users);
>   },
> };
> ```

> [tab] Netlify
>
> ```typescript theme={null}
> import { neon } from '@neondatabase/serverless';
> import type { Context } from '@netlify/functions';
>
> export default async (request: Request, context: Context) => {
>   const sql = neon(process.env.DATABASE_URL!);
>   const users = await sql`SELECT * FROM users LIMIT 10`;
>   return Response.json(users);
> };
>
> export const config = { path: '/api/users' };
> ```

> [tab] AWS Lambda
>
> ```typescript theme={null}
> import { neon } from '@neondatabase/serverless';
>
> export const handler = async (event) => {
>   const sql = neon(process.env.DATABASE_URL);
>   const users = await sql`SELECT * FROM users LIMIT 10`;
>   return {
>     statusCode: 200,
>     body: JSON.stringify(users),
>   };
> };
> ```


# AWS Aurora
Source: https://xata.io/docs/migrations/aws-aurora

Learn how to migrate your AWS Aurora PostgreSQL database to Xata using xata clone

## Prerequisites

* AWS Aurora PostgreSQL cluster
* Access to AWS Console
* Xata account and project setup with a running `main` branch
* Network access to your Aurora cluster

## Install and configure the Xata CLI

Install the Xata CLI:

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
```

Authenticate with your Xata account:

```bash theme={null}
xata auth login
```

## Find the Aurora connection string

Aurora connection strings typically look something like this:

```
postgresql://your_username:your_password@aurora-endpoint:5432/dbname
```

You can find the aurora-endpoint in the AWS RDS Console under "Connectivity & security".

> [!TIP]
>
> Don't have a live connection, or prefer to export a snapshot to Amazon S3? You can also migrate from [Parquet files](/docs/migrations/parquet) exported to S3.

## Initialize Xata project

Go to a local folder that you want to save the Xata configuration to and initialize it like this:

```bash theme={null}
xata init
```

Select your organization and project you want to use. For the database name you can either choose a new one (e.g `xata`) or use the same that you have in Aurora. If the database doesn't exist, it will be created.

In case you need to redo this step, you can reset the configuration by deleting the `.xata/project.json` file and running `xata init` again.

## Start the migration

Begin the data transfer:

```bash theme={null}
xata clone start --validation-mode relaxed --source-url '<<aurora-connection-string>>'
```

Depending on how large the source database is, this step might take a while. A progress bar will show you the progress in the CLI.

## Verification

After migration, verify your data:

1. **Connect to Xata Branch**:
   ```bash theme={null}
   psql `xata branch url`
   ```

2. **Check Data Integrity**:
   ```sql theme={null}
   -- List tables
   \dt

   -- Compare row counts
   SELECT COUNT(*) FROM your_table;

   -- Check sample data
   SELECT * FROM your_table LIMIT 10;
   ```


# AWS RDS
Source: https://xata.io/docs/migrations/aws-rds

Learn how to migrate your AWS RDS PostgreSQL database to Xata using xata clone

## Prerequisites

* AWS RDS PostgreSQL instance
* Access to AWS Console
* Xata account and project setup with a running `main` branch
* Network access to your RDS instance

## Install and Configure the Xata CLI

Install the Xata CLI:

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
```

Authenticate with your Xata account:

```bash theme={null}
xata auth login
```

## Find the RDS connection string

RDS connection strings typically look something like this:

```
postgresql://your_username:your_password@rds-endpoint:5432/dbname
```

You can find the rds-endpoint in the AWS RDS Console under "Connectivity & security".

> [!TIP]
>
> Don't have a live connection, or prefer to export a snapshot to Amazon S3? You can also migrate from [Parquet files](/docs/migrations/parquet) exported to S3.

## Initialize Xata project

Go to a local folder that you want to save the Xata configuration to and initialize it like this:

```bash theme={null}
xata init
```

Select your organization and project you want to use. For the database name you can either choose a new one (e.g `xata`) or use the same that you have in RDS. If the database doesn't exist, it will be created.

In case you need to redo this step, you can reset the configuration by deleting the `.xata/project.json` file and running `xata init` again.

## Start the migration

Begin the data transfer:

```bash theme={null}
xata clone start --validation-mode relaxed --source-url '<<rds-connection-string>>'
```

Depending on how large the source database is, this step might take a while. A progress bar will show you the progress in the CLI.

## Verification

After migration, verify your data:

1. **Connect to Xata Branch**:
   ```bash theme={null}
   psql `xata branch url`
   ```

2. **Check Data Integrity**:
   ```sql theme={null}
   -- List tables
   \dt

   -- Compare row counts
   SELECT COUNT(*) FROM your_table;

   -- Check sample data
   SELECT * FROM your_table LIMIT 10;
   ```


# Azure Database
Source: https://xata.io/docs/migrations/azure

Learn how to migrate your Azure Database for PostgreSQL to Xata using xata clone

## Prerequisites

* Azure Database for PostgreSQL instance
* Access to Azure Portal
* Xata account and project setup with a running `main` branch
* Network access to your Azure database

## Install and Configure the Xata CLI

Install the Xata CLI:

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
```

Authenticate with your Xata account:

```bash theme={null}
xata auth login
```

## Find the Azure Database connection string

Azure connection strings typically look like this:

```
postgresql://your_username:your_password@your-server.postgres.database.azure.com:5432/postgres?sslmode=require
```

You can find the database endpoint in the Azure console.

## Initialize Xata project

Go to a local folder that you want to save the Xata configuration to and initialize it like this:

```bash theme={null}
xata init
```

Select your organization and project you want to use. For the database name you can either choose a new one (e.g `xata`) or use the same that you have in Azure. If the database doesn't exist, it will be created.

In case you need to redo this step, you can reset the configuration by deleting the `.xata/project.json` file and running `xata init` again.

## Start the migration

Begin the data transfer:

```bash theme={null}
xata clone start --validation-mode relaxed --source-url '<<azure-connection-string>>'
```

Depending on how large the source database is, this step might take a while. A progress bar will show you the progress in the CLI.

## Verification

After migration, verify your data:

1. **Connect to Xata Branch**:
   ```bash theme={null}
   psql `xata branch url`
   ```

2. **Check Data Integrity**:
   ```sql theme={null}
   -- List tables
   \dt

   -- Compare row counts
   SELECT COUNT(*) FROM your_table;

   -- Check sample data
   SELECT * FROM your_table LIMIT 10;
   ```


# GCP Cloud SQL
Source: https://xata.io/docs/migrations/gcp-cloudsql

Learn how to migrate your GCP Cloud SQL PostgreSQL database to Xata using xata clone

## Prerequisites

* GCP Cloud SQL PostgreSQL instance
* Access to Google Cloud Console
* Xata account and project setup with a running `main` branch
* Network access to your Cloud SQL instance

## Install and configure the Xata CLI

Install the Xata CLI:

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
```

Authenticate with your Xata account:

```bash theme={null}
xata auth login
```

## Find the Cloud SQL connection string

Form the PostgreSQL connection string like this:

```
postgresql://your_username:your_password@cloudsql-ip-address:5432/dbname?sslmode=require
```

You can find the cloudsql-ip-address in the Cloud SQL console under "Connect to this instance".

## Initialize Xata project

Go to a local folder that you want to save the Xata configuration to and initialize it like this:

```bash theme={null}
xata init
```

Select your organization and project you want to use. For the database name you can either choose a new one (e.g `xata`) or use the same that you have in Cloud SQL. If the database doesn't exist, it will be created.

In case you need to redo this step, you can reset the configuration by deleting the `.xata/project.json` file and running `xata init` again.

## Start the migration

Begin the data transfer:

```bash theme={null}
xata clone start --validation-mode relaxed --source-url '<<cloud-connection-string>>'
```

Depending on how large the source database is, this step might take a while. A progress bar will show you the progress in the CLI.

## Verification

After migration, verify your data:

1. **Connect to Xata Branch**:
   ```bash theme={null}
   psql `xata branch url`
   ```

2. **Check Data Integrity**:
   ```sql theme={null}
   -- List tables
   \dt

   -- Compare row counts
   SELECT COUNT(*) FROM your_table;

   -- Check sample data
   SELECT * FROM your_table LIMIT 10;
   ```


# Greenmask (subsets)
Source: https://xata.io/docs/migrations/greenmask

Learn how to import a referentially-consistent subset of your PostgreSQL database to Xata using Greenmask.

## When to use this guide

Use Greenmask when you want to import a **representative slice** of your production database into a Xata branch. For example to evaluate Xata against your real schema and workload without copying 100% of your data. Greenmask follows your foreign-key graph from a chosen "entrypoint" table and exports only the rows that hang off it (and their dependencies), preserving referential integrity.

If you want a **full copy** of your database into Xata, use `xata clone` instead — see the provider-specific guides for [AWS RDS](/docs/migrations/aws-rds), [AWS Aurora](/docs/migrations/aws-aurora), [Azure](/docs/migrations/azure), [GCP Cloud SQL](/docs/migrations/gcp-cloudsql), [Neon](/docs/migrations/neon), [Supabase](/docs/migrations/supabase), and [self-hosted PostgreSQL](/docs/migrations/self-hosted). Greenmask is for the subset case.

## Prerequisites

* A PostgreSQL source you can reach over the network (public endpoint or via your own VPN/SSH tunnel)
* Xata account and project setup with a running `main` branch
* Knowledge of your schema's foreign-key structure — at minimum, which table you want as the subset "entrypoint" (typically `users`, `tenants`, `accounts`, or similar)

## Install and configure the Xata CLI

Install the Xata CLI:

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
```

Authenticate with your Xata account:

```bash theme={null}
xata auth login
```

## Install Greenmask

Download a pre-built binary from the [Greenmask releases page](https://github.com/GreenmaskIO/greenmask/releases). This guide is written against **v0.2.19**.

```bash theme={null}
curl -fsSL https://github.com/GreenmaskIO/greenmask/releases/download/v0.2.19/greenmask-linux-amd64.tar.gz \
  | tar -xz
./greenmask --version
```

Replace `linux-amd64` with `darwin-arm64` etc. as needed.

## Find your source connection string

The exact format depends on your provider (see the other guides in this folder for AWS RDS, Neon, etc.). A generic PostgreSQL connection string looks like:

```
postgresql://your_username:your_password@your-host:5432/your_database?sslmode=require
```

## Initialize Xata project

Go to a local folder that you want to save the Xata configuration to and initialize it like this:

```bash theme={null}
xata init
```

Select your organization and project you want to use. For the database name you can either choose a new one (e.g `xata`) or use the same that you have in your source database. If the database doesn't exist, it will be created.

In case you need to redo this step, you can reset the configuration by deleting the `.xata/project.json` file and running `xata init` again.

Once initialized, you can get the branch connection string Greenmask will restore into with:

```bash theme={null}
xata branch url
```

Treat it as sensitive: it grants full write access.

## Prepare the target branch

`pg_dump --schema=...` (which Greenmask uses under the hood) doesn't export PostgreSQL extension definitions from the `public` schema. Create any extensions your source schema uses on the target *before* the restore:

```bash theme={null}
psql "$(xata branch url)" <<'SQL'
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS unaccent;
-- add any others your source schema uses
SQL
```

To list the extensions your source uses:

```bash theme={null}
psql '<<source-connection-string>>' -c "\dx"
```

## Write your subset config

Create `greenmask.yml` in your working directory. Replace `your_schema`, `your_entrypoint_table`, and the predicate to match your schema. The example below assumes a multi-tenant SaaS shape where `tenants.id` is the subset root.

```yaml theme={null}
common:
  pg_bin_path: "/usr/lib/postgresql/16/bin"   # or wherever your local pg_dump lives
  tmp_dir: "/tmp"

log:
  level: "info"
  format: "json"

storage:
  type: "directory"
  directory:
    path: "./dump"

dump:
  pg_dump_options:
    dbname: "<<source-connection-string>>"
    jobs: 4
    schema: "your_schema"
    load-via-partition-root: true

  transformation:
    - schema: "your_schema"
      name: "tenants"
      subset_conds:
        - "your_schema.tenants.id <= 100"

restore:
  pg_restore_options:
    dbname: "<<xata-branch-connection-string>>"
    jobs: 4
    exit-on-error: true
    no-owner: true
    no-privileges: true
```

> [!TIP]
>
> There's no single "right" predicate. Start with a `subset_conds` value that returns a small slice (for example `tenants.id <= 10`), run the dump, and check the resulting row counts. Widen or narrow the condition and re-run until the subset is the size and shape you want — a tight predicate keeps the first few iterations fast while you dial it in.

## Run the dump

```bash theme={null}
./greenmask --config greenmask.yml dump
```

## Restore into Xata

```bash theme={null}
./greenmask --config greenmask.yml restore latest
```

If you want to retry, drop the schema on the target first:

```bash theme={null}
psql "$(xata branch url)" -c "DROP SCHEMA IF EXISTS your_schema CASCADE;"
```

then re-create the extensions (see the "Prepare the target branch" step) and re-run `restore`.

## Verify

After restore, verify the import:

```bash theme={null}
psql "$(xata branch url)"
```

```sql theme={null}
-- List tables
\dt your_schema.*

-- Check counts on key tables
SELECT count(*) FROM your_schema.your_table;

-- Sample some rows
SELECT * FROM your_schema.your_table LIMIT 10;
```

### Foreign-key integrity check

For a subset to be useful, every foreign-key reference in the target must point to a row that exists. To audit this, compare the foreign-key constraint count between source and target — they must match:

```bash theme={null}
psql '<<source-connection-string>>' -At -c "
  SELECT count(*) FROM pg_constraint c
  JOIN pg_namespace n ON n.oid = c.connamespace
  WHERE c.contype='f' AND n.nspname='your_schema';"

psql "$(xata branch url)" -At -c "
  SELECT count(*) FROM pg_constraint c
  JOIN pg_namespace n ON n.oid = c.connamespace
  WHERE c.contype='f' AND n.nspname='your_schema';"
```

### Limits

* Greenmask runs `pg_dump`/`pg_restore` under the hood, so the same version compatibility rules apply: your local `pg_dump` major version must be ≥ the source server's, and the same major version as your Xata branch (currently PostgreSQL 18).
* Sources with custom PostgreSQL extensions that Xata doesn't support will fail during restore. Verify with `psql -c "\dx"` against your source first.
* Greenmask's subset model works best when your data has a clear "root" entity (a tenant, a user, an organization). Schemas without one will need more configuration work.

## Going further

Greenmask supports rich PII transformations (anonymising emails, hashing names, redacting fields) on top of subsetting. Those are useful if you want to use your imported subset for dev/test as well as evaluation. See the [Greenmask transformer docs](https://docs.greenmask.io/latest/built_in_transformers/) for the full list.

For full data migration (not subsets), see the other guides in this folder.


# Neon
Source: https://xata.io/docs/migrations/neon

Learn how to migrate your Neon PostgreSQL database to Xata using xata clone

## Prerequisites

* Access to Neon Console
* Xata account and project setup with a running `main` branch

## Install and configure the Xata CLI

Install the Xata CLI:

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
```

Authenticate with your Xata account:

```bash theme={null}
xata auth login
```

## Find the Neon connection string

Neon connection strings typically look something like this:

```
postgresql://neondb_owner:your_password@hostname.c-3.us-east-1.aws.neon.tech/dbname?sslmode=require&channel_binding=require
```

To find yours, follow these steps:

1. Go to Neon Console → Your Project
2. Select the branch you want to copy from
3. Click "Connect" to get the "Connect to your database" modal
4. Unset the Connection pooling toggle
5. Select "Connection string" in the dropdown
6. Click "Copy snippet" to copy it, including the password

## Initialize Xata project

Go to a local folder that you want to save the Xata configuration to and initialize it like this:

```bash theme={null}
xata init
```

Select your organization and project you want to use. For the database name you can either choose a new one (e.g `xata`) or use the same that you have in Neon. If the database doesn't exist, it will be created.

In case you need to redo this step, you can reset the configuration by deleting the `.xata/project.json` file and running `xata init` again.

## Start the migration

Begin the data transfer:

```bash theme={null}
xata clone start --validation-mode relaxed --source-url '<<neon-connection-string>>'
```

Depending on how large the source database is, this step might take a while. A progress bar will show you the progress in the CLI.

## Verification

After migration, verify your data:

1. **Connect to Xata Branch**:
   ```bash theme={null}
   psql `xata branch url`
   ```

2. **Check Data Integrity**:
   ```sql theme={null}
   -- List tables
   \dt

   -- Compare row counts
   SELECT COUNT(*) FROM your_table;

   -- Check sample data
   SELECT * FROM your_table LIMIT 10;
   ```

## Update your application

If you're using the `@neondatabase/serverless` driver, you can continue using it with Xata—just update your connection string. Xata implements the same serverless protocol.

1. Get your Xata connection string:
   ```bash theme={null}
   xata branch url
   ```

2. Update your environment variable:
   ```bash theme={null}
   DATABASE_URL=<your-xata-connection-string>
   ```

Your existing code works without changes:

```typescript theme={null}
import { neon } from '@neondatabase/serverless';

const sql = neon(process.env.DATABASE_URL);
const users = await sql`SELECT * FROM users`;
```

> [!TIP]
>
> For more details on using the serverless driver with Xata, see the [Serverless Proxy](/docs/core-concepts/serverless-proxy) documentation.


# Overview
Source: https://xata.io/docs/migrations/overview

Choose a guide to migrate your PostgreSQL database or data to Xata.

## Migrate with `xata clone`

> [card] [AWS RDS](/docs/migrations/aws-rds)
>
> Migrate an AWS RDS for PostgreSQL instance.

> [card] [AWS Aurora](/docs/migrations/aws-aurora)
>
> Migrate an AWS Aurora PostgreSQL cluster.

> [card] [GCP Cloud SQL](/docs/migrations/gcp-cloudsql)
>
> Migrate a GCP Cloud SQL for PostgreSQL instance.

> [card] [Azure Database](/docs/migrations/azure)
>
> Migrate an Azure Database for PostgreSQL server.

> [card] [Neon](/docs/migrations/neon)
>
> Migrate a Neon PostgreSQL database.

> [card] [Supabase](/docs/migrations/supabase)
>
> Migrate a Supabase PostgreSQL database.

> [card] [Self-Hosted PostgreSQL](/docs/migrations/self-hosted)
>
> Migrate a PostgreSQL database you run yourself.

## Other import options

> [card] [Greenmask (subsets)](/docs/migrations/greenmask)
>
> Import a referentially consistent subset of your database instead of a full copy.

> [card] [Parquet files](/docs/migrations/parquet)
>
> Load data from Parquet files in AWS S3, such as RDS or Aurora snapshot exports.


# Parquet files
Source: https://xata.io/docs/migrations/parquet

Learn how to migrate your data to Xata from Parquet files exported to AWS S3

## Overview

If your data already lives in [Parquet](https://parquet.apache.org/) files, you can load it into Xata. This is a common path when your source database doesn't expose a live PostgreSQL connection but can export snapshots to object storage instead.

A typical flow is to export your database to Parquet files in an AWS S3 bucket, and then have Xata load those files into your branch.

## Export to Parquet in AWS S3

Both AWS Aurora and AWS RDS can export snapshots to Amazon S3 in Apache Parquet format. AWS writes the export as a set of Parquet files, one set per table, into the S3 bucket you choose.

* **AWS Aurora**: Use *Export to Amazon S3* on a snapshot or cluster. See the [AWS Aurora export documentation](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-export-snapshot.html).
* **AWS RDS**: Use *Export to Amazon S3* on a snapshot. See the [AWS RDS export documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ExportSnapshot.html).

In both cases the export lands in S3 as Parquet files that preserve your table schema and data.

> [!TIP]
>
> If you have a live PostgreSQL connection to your source database, you can migrate directly with `xata clone` instead. See the [AWS Aurora](/docs/migrations/aws-aurora) and [AWS RDS](/docs/migrations/aws-rds) guides.

## Import Parquet files into Xata

Importing Parquet files into Xata is currently handled on a case-by-case basis. If you'd like to import Parquet files — whether exported from AWS Aurora, AWS RDS, or another source — [reach out to us](mailto:support@xata.io) and we'll help you load them into your branch.

When you contact us, it helps to share:

* The location of your Parquet files (for example, the S3 bucket and prefix)
* The source database and how the export was produced (AWS Aurora, AWS RDS, or other)
* The approximate size and number of tables
* A schema-only `pg_dump` of your source database:

  ```bash theme={null}
  pg_dump --schema-only --no-owner --no-privileges '<<source-connection-string>>' > schema.sql
  ```

## Verification

After your data is loaded, verify it:

1. **Connect to Xata Branch**:
   ```bash theme={null}
   psql `xata branch url`
   ```

2. **Check Data Integrity**:
   ```sql theme={null}
   -- List tables
   \dt

   -- Compare row counts
   SELECT COUNT(*) FROM your_table;

   -- Check sample data
   SELECT * FROM your_table LIMIT 10;
   ```


# Self-Hosted PostgreSQL
Source: https://xata.io/docs/migrations/self-hosted

Learn how to migrate your self-hosted PostgreSQL database to Xata using xata clone.

## Prerequisites

* Self-hosted PostgreSQL instance
* Access to PostgreSQL configuration files
* Xata account and project setup with a running `main` branch

## Install and configure the Xata CLI

Install the Xata CLI:

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
```

Authenticate with your Xata account:

```bash theme={null}
xata auth login
```

## Find the connection string

Form your PostgreSQL connection string like this:

```
postgresql://your_username:your_password@your-server-ip:5432/your_database
```

If your PostgreSQL server requires SSL:

```
postgresql://your_username:your_password@your-server-ip:5432/your_database?sslmode=require
```

## Initialize Xata project

Go to a local folder that you want to save the Xata configuration to and initialize it like this:

```bash theme={null}
xata init
```

Select your organization and project you want to use. For the database name you can either choose a new one (e.g `xata`) or use the same that you have in the source PostgreSQL. If the database doesn't exist, it will be created.

In case you need to redo this step, you can reset the configuration by deleting the `.xata/project.json` file and running `xata init` again.

## Start the migration

Begin the data transfer:

```bash theme={null}
xata clone start --validation-mode relaxed --source-url '<<postgres-connection-string>>'
```

Depending on how large the source database is, this step might take a while. A progress bar will show you the progress in the CLI.

## Verification

After migration, verify your data:

1. **Connect to Xata Branch**:
   ```bash theme={null}
   psql `xata branch url`
   ```

2. **Check Data Integrity**:
   ```sql theme={null}
   -- List tables
   \dt

   -- Compare row counts
   SELECT COUNT(*) FROM your_table;

   -- Check sample data
   SELECT * FROM your_table LIMIT 10;
   ```


# Supabase
Source: https://xata.io/docs/migrations/supabase

Learn how to migrate your Supabase PostgreSQL database to Xata using xata clone

## Prerequisites

* Access to Supabase Dashboard
* Xata account and project setup with a running `main` branch

## Install and configure the Xata CLI

Install the Xata CLI:

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
```

Authenticate with your Xata account:

```bash theme={null}
xata auth login
```

## Find the Supabase connection string

Supabase connection strings typically look like this:

```
postgresql://postgres:[YOUR_PASSWORD]@db.hostnameid.supabase.co:5432/postgres
```

To find yours, follow these steps:

1. Go to Supabase Dashboard → Your Project
2. Click "Connect" to bring up the "Connect to your project"
3. Copy the connection string
4. The password is not shown in the Supabase dashboard for security reasons. You can copy it from you records, or you can reset it if you have to.

## Initialize Xata project

Go to a local folder that you want to save the Xata configuration to and initialize it like this:

```bash theme={null}
xata init
```

Select your organization and project you want to use. For the database name you can either choose a new one (e.g `xata`) or use the same that you have in Supabase. If the database doesn't exist, it will be created.

In case you need to redo this step, you can reset the configuration by deleting the `.xata/project.json` file and running `xata init` again.

## Start the migration

Begin the data transfer:

```bash theme={null}
PGSTREAM_POSTGRES_SNAPSHOT_TABLES='public.*' xata clone start --validation-mode relaxed --source-url '<<supabase-connection-string>>'
```

Adjust the `PGSTREAM_POSTGRES_SNAPSHOT_TABLES` env variable with the schema names you want to transfer. It is needed here to avoid copying Supabase specific schemas like `storage` and `realtime`.

Depending on how large the source database is, this step might take a while. A progress bar will show you the progress in the CLI.

## Verification

After migration, verify your data:

1. **Connect to Xata Branch**:
   ```bash theme={null}
   psql `xata branch url`
   ```

2. **Check Data Integrity**:
   ```sql theme={null}
   -- List tables
   \dt

   -- Compare row counts
   SELECT COUNT(*) FROM your_table;

   -- Check sample data
   SELECT * FROM your_table LIMIT 10;
   ```


# Observability
Source: https://xata.io/docs/opensource/pgstream/docs/Observability

pgstream provides comprehensive observability through [OpenTelemetry](https://opentelemetry.io/) metrics and distributed tracing. This document outlines the available metrics and how to use them for monitoring your data streaming pipeline.

## Overview

pgstream instruments all major components of the data streaming pipeline with metrics and traces using OpenTelemetry. The instrumentation covers:

* WAL replication and processing
* Snapshot generation
* Kafka read/write operations
* PostgreSQL query operations
* Search indexing operations
* Data transformations
* [Go runtime metrics](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/runtime) (memory, GC, goroutines, etc.)
* [Go profiling](https://go.dev/blog/pprof) (CPU, memory, goroutines, etc.)

## Quick Start with Local Setup

pgstream includes a local observability setup using [SigNoz](https://signoz.io/) with a pre-configured [dashboard](https://github.com/xataio/pgstream/blob/main/build/docker/signoz/dashboards/pgstream.json) that visualizes all the metrics described below.

To get started:

```bash theme={null}
# Start the observability stack
docker-compose -f build/docker/docker-compose-signoz.yml --profile instrumentation up

# To start it alongside a different profile
docker-compose -f build/docker/docker-compose.yml -f build/docker/docker-compose-signoz.yml --profile pg2pg --profile instrumentation up

# Access SigNoz dashboard
open http://localhost:8080

# Import the pgstream dashboard
# Dashboard configuration is available in ./build/docker/signoz/dashboards/pgstream.json
```

The dashboard includes panels for all key metrics, alerting rules, and provides a comprehensive view of your pgstream deployment health and performance.

## Metrics

All metrics follow the `pgstream.*` naming convention and include relevant attributes for filtering and aggregation.

### WAL Replication

> [accordion] pgstream.replication.lag
>
> **Type**: ObservableGauge\
> **Unit**: bytes\
> **Description**: Replication lag in bytes accrued by the pgstream WAL consumer

**Attributes:** None

**Usage:** Monitor replication health and detect lag buildup that could indicate performance issues.

**⚠️ Important:** This metric only tracks pgstream's consumer lag. It's strongly recommended to also monitor your source PostgreSQL metrics, particularly the built-in replication lag metrics (`pg_stat_replication.flush_lag`, `pg_stat_replication.replay_lag`) to get a complete picture of replication health.

### Pipeline Phase

> [accordion] pgstream.pipeline.phase
>
> **Type**: ObservableGauge\
> **Unit**: 1\
> **Description**: Reports `1` for the active pipeline phase and `0` for the others (\`phase=snapshot\\

**Attributes:**

* `phase`: Phase this data point refers to (`snapshot` or `replication`)

**Usage:** Distinguish whether a running process is performing the initial snapshot or streaming logical replication, without scraping logs. Both `phase="snapshot"` and `phase="replication"` series are reported on every collection, with `1` marking the active phase and `0` the inactive one, so queries like `sum by (phase)` stay unambiguous across transitions. Nothing is reported until the first phase transition. The gauge flips to `replication` when logical replication starts. The same value is also exposed via the health server `GET /status` endpoint when health checks are enabled.

### WAL Event Processing

> [accordion] pgstream.target.processing.lag
>
> **Type**: Histogram\
> **Unit**: ns\
> **Description**: Time between WAL event creation and processing

> [accordion] pgstream.target.processing.latency
>
> **Type**: Histogram\
> **Unit**: ns\
> **Description**: Time taken to process a WAL event

**Attributes:**

* `target`: Target type (e.g., "postgres", "kafka", "search")

**Usage:** Monitor end-to-end processing latency for each target and identify bottlenecks in the processing pipeline.

### Snapshot Operations

> [accordion] pgstream.snapshot.generator.latency
>
> **Type**: Histogram\
> **Unit**: ms\
> **Description**: Time taken to snapshot a source PostgreSQL database

**Attributes:**

* `snapshot_schema`: List of schemas being snapshotted
* `snapshot_tables`: List of tables being snapshotted

**Usage:** Monitor snapshot performance and identify slow-running snapshot operations.

### Kafka Operations

#### Writer Metrics

> [accordion] pgstream.kafka.writer.batch.size
>
> **Type**: Histogram\
> **Unit**: messages\
> **Description**: Distribution of message batch sizes

> [accordion] pgstream.kafka.writer.batch.bytes
>
> **Type**: Histogram\
> **Unit**: bytes\
> **Description**: Distribution of message batch sizes in bytes

> [accordion] pgstream.kafka.writer.latency
>
> **Type**: Histogram\
> **Unit**: ms\
> **Description**: Time taken to send messages to Kafka

#### Reader Metrics

> [accordion] pgstream.kafka.reader.msg.bytes
>
> **Type**: Histogram\
> **Unit**: bytes\
> **Description**: Distribution of message sizes read from Kafka

> [accordion] pgstream.kafka.reader.fetch.latency
>
> **Type**: Histogram\
> **Unit**: ms\
> **Description**: Time taken to fetch messages from Kafka

> [accordion] pgstream.kafka.reader.commit.latency
>
> **Type**: Histogram\
> **Unit**: ms\
> **Description**: Time taken to commit offsets to Kafka

> [accordion] pgstream.kafka.reader.commit.batch.size
>
> **Type**: Histogram\
> **Unit**: offsets\
> **Description**: Distribution of offset batch sizes committed

**Attributes:** None

**Usage:** Monitor Kafka throughput, latency, and batch efficiency to optimize performance.

### PostgreSQL Operations

> [accordion] pgstream.postgres.querier.latency
>
> **Type**: Histogram\
> **Unit**: ms\
> **Description**: Time taken to perform PostgreSQL queries

**Attributes:**

* `query_type`: Type of SQL operation (e.g., "SELECT", "INSERT", "UPDATE", "DELETE", "tx")
* `query`: The actual SQL query (for non-transaction operations)

**Usage:** Monitor database performance and identify slow queries.

#### Writer Metrics

> [accordion] pgstream.postgres.writer.dropped\_queries
>
> **Type**: ObservableCounter\
> **Unit**: queries\
> **Description**: Number of queries silently dropped due to non-internal (DATALOSS) failures in drop-and-continue mode

**Attributes:**

* `writer_type`: The postgres writer that dropped the query (one of "postgres\_batch\_writer", "postgres\_bulk\_ingest\_writer")

**Usage:** Alert on any non-zero value: each drop means a failing change was skipped and the checkpoint advanced past it, so the target replica may have diverged from the source. Enable `strict_mode` (`PGSTREAM_POSTGRES_WRITER_STRICT_MODE`) to make such failures stop the pipeline instead of being dropped.

### Search Operations

> [accordion] pgstream.search.store.doc.errors
>
> **Type**: Counter\
> **Unit**: errors\
> **Description**: Count of failed document indexing operations

**Attributes:**

* `severity`: Error severity level (one of "NONE", "DATALOSS", "IGNORED", "RETRIABLE")

**Usage:** Monitor search indexing health and error rates.

### Data Transformations

> [accordion] pgstream.transformer.latency
>
> **Type**: Histogram\
> **Unit**: ms\
> **Description**: Time taken to transform data values

**Attributes:**

* `transformer_type`: Type of transformer being used

**Usage:** Monitor transformation performance and identify slow transformers.

### Go Runtime Metrics

pgstream automatically collects Go runtime metrics using the [OpenTelemetry Go runtime instrumentation](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/runtime). These metrics are essential for monitoring application health and performance:

> [accordion] runtime.go.mem.heap\_alloc
>
> **Type**: Gauge\
> **Unit**: bytes\
> **Description**: Bytes of allocated heap objects

> [accordion] runtime.go.mem.heap\_idle
>
> **Type**: Gauge\
> **Unit**: bytes\
> **Description**: Bytes in idle (unused) heap spans

> [accordion] runtime.go.mem.heap\_inuse
>
> **Type**: Gauge\
> **Unit**: bytes\
> **Description**: Bytes in in-use heap spans

> [accordion] runtime.go.mem.heap\_objects
>
> **Type**: Gauge\
> **Unit**: objects\
> **Description**: Number of allocated heap objects

> [accordion] runtime.go.mem.heap\_released
>
> **Type**: Gauge\
> **Unit**: bytes\
> **Description**: Bytes of physical memory returned to the OS

> [accordion] runtime.go.mem.heap\_sys
>
> **Type**: Gauge\
> **Unit**: bytes\
> **Description**: Bytes of heap memory obtained from the OS

> [accordion] runtime.go.gc.count
>
> **Type**: Counter\
> **Unit**: collections\
> **Description**: Number of completed GC cycles

> [accordion] runtime.go.gc.pause\_ns
>
> **Type**: Histogram\
> **Unit**: ns\
> **Description**: Amount of nanoseconds in GC stop-the-world pauses

> [accordion] runtime.go.goroutines
>
> **Type**: Gauge\
> **Unit**: goroutines\
> **Description**: Number of goroutines that currently exist

> [accordion] runtime.go.lookups
>
> **Type**: Counter\
> **Unit**: lookups\
> **Description**: Number of pointer lookups performed by the runtime

> [accordion] runtime.go.mem.gc\_sys
>
> **Type**: Gauge\
> **Unit**: bytes\
> **Description**: Bytes of memory in garbage collection metadata

**Usage:** Monitor application resource usage, detect memory leaks, track GC performance, and identify goroutine leaks.

## Configuration

### Enabling Metrics

Configure metrics collection in your pgstream configuration file:

```yaml theme={null}
instrumentation:
  metrics:
    endpoint: "0.0.0.0:4317"
    collection_interval: 60 # collection interval for metrics in seconds. Defaults to 60s
  traces:
    endpoint: "0.0.0.0:4317"
    sample_ratio: 0.5 # ratio of traces that will be sampled. Must be between 0.0-1.0, where 0 is no traces sampled, and 1 is all traces sampled.
```

Or using environment variables:

```sh theme={null}
PGSTREAM_METRICS_ENDPOINT="http://localhost:4317"
PGSTREAM_METRICS_COLLECTION_INTERVAL=60s
PGSTREAM_TRACES_ENDPOINT="http://localhost:4317"
PGSTREAM_TRACES_SAMPLE_RATIO=0.5
```

### Prometheus Scraping (Pull-based)

If you'd rather scrape metrics directly than deploy an OTel collector, enable the Prometheus exporter. It's served from the existing [health endpoint](/docs/opensource/pgstream/docs/configuration#instrumentation) server, so no separate port or process is needed:

```yaml theme={null}
instrumentation:
  health:
    enabled: true
    address: "0.0.0.0:9910" # defaults to localhost:9910
  metrics:
    prometheus:
      enabled: true
      endpoint: "/metrics" # defaults to /metrics
```

Or using environment variables:

```sh theme={null}
PGSTREAM_HEALTH_CHECK_ENABLED=true
PGSTREAM_HEALTH_CHECK_ADDRESS="0.0.0.0:9910"
PGSTREAM_METRICS_PROMETHEUS_ENABLED=true
PGSTREAM_METRICS_PROMETHEUS_ENDPOINT="/metrics"
```

This works standalone — `metrics.endpoint` is not required, so you get pgstream's metrics (including the [Go runtime metrics](#go-runtime-metrics)) without an OTLP collector in the loop. It can also be combined with `metrics.endpoint` to export to both a Prometheus scraper and an OTLP collector at the same time.

## Monitoring Dashboards

### Pre-built pgstream Dashboard

The included SigNoz dashboard provides:

* **Overview Panel**: System health and key metrics at a glance
* **Replication Health**: Both pgstream and PostgreSQL replication metrics
* **Processing Performance**: Latency and throughput across all components
* **Error Tracking**: Error rates and failure patterns
* **Resource Utilization**: Memory, CPU, and network usage
* **Go Runtime Health**: Memory usage, GC performance, and goroutine tracking

![pgstream\_signoz\_dashboard](https://github.com/user-attachments/assets/c08e5029-494d-4dc6-94e1-6ce467d3abce)

### Key Performance Indicators (KPIs)

1. **Application Health**

   * `runtime.go.mem.heap_alloc` - Memory usage trends
   * `runtime.go.goroutines` - Goroutine count (detect leaks)
   * `runtime.go.gc.pause_ns` - GC pause times (performance impact)

2. **Replication Health**

   * `pgstream.replication.lag` - Should remain low and stable
   * `pgstream.target.processing.lag` - End-to-end processing delay
   * **PostgreSQL replication lag** - Monitor source database metrics

3. **Throughput**

   * `pgstream.kafka.writer.batch.size` - Messages per batch
   * `pgstream.kafka.writer.batch.bytes` - Bytes per batch
   * `rate(pgstream.postgres.querier.latency_count[5m])` - Query rate

4. **Latency**

   * `pgstream.target.processing.latency` (p95, p99) - Processing latency percentiles
   * `pgstream.kafka.writer.latency` (p95, p99) - Kafka write latency
   * `pgstream.postgres.querier.latency` (p95, p99) - Database query latency

5. **Error Rates**
   * `rate(pgstream.search.store.doc.errors[5m])` - Search indexing errors
   * Error logs from distributed traces

## Source PostgreSQL Monitoring

**⚠️ Critical:** pgstream metrics only track the consumer-side lag. For complete replication monitoring, you must also monitor your source PostgreSQL instance:

### Essential PostgreSQL Metrics

```sql theme={null}
-- WAL generation rate
SELECT
    pg_current_wal_lsn(),
    pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) / 1024 / 1024 AS wal_mb;

-- Replication slot lag for all consumers
SELECT
    slot_name,
    active,
    pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) / 1024 / 1024 AS lag_mb
FROM pg_replication_slots;
```

## Distributed Tracing

pgstream automatically creates traces for all major operations. Traces include:

* Complete request flows from WAL event to target system
* Database query execution details
* Kafka produce/consume operations
* Transformation pipeline execution
* Error context and stack traces

Use the included SigNoz setup or tools like Jaeger/Zipkin to visualize and analyze trace data for debugging and performance optimization.

![pgstream\_signoz\_tracing](https://github.com/user-attachments/assets/595cfff7-d1a2-419c-afd0-739e851e1fa7)

## Profiling

pgstream includes built-in Go profiling capabilities using Go's [`net/http/pprof`](https://pkg.go.dev/net/http/pprof) package for performance analysis and debugging.

### Enabling Profiling

Profiling can be enabled using the `--profile` flag on supported commands:

```bash theme={null}
# Enable profiling for snapshot operations
pgstream snapshot --profile --config config.yaml

# Enable profiling for run operations
pgstream run --profile --config config.yaml
```

### Profiling Modes

#### 1. HTTP Endpoint (All Commands)

When `--profile` is enabled, pgstream exposes a profiling HTTP server at `localhost:6060` with the following endpoints:

> [accordion] [http://localhost:6060/debug/pprof/](http://localhost:6060/debug/pprof/)
>
> **Description**: Profile index page

> [accordion] [http://localhost:6060/debug/pprof/profile](http://localhost:6060/debug/pprof/profile)
>
> **Description**: CPU profile (30-second sample)

> [accordion] [http://localhost:6060/debug/pprof/heap](http://localhost:6060/debug/pprof/heap)
>
> **Description**: Memory heap profile

> [accordion] [http://localhost:6060/debug/pprof/goroutine](http://localhost:6060/debug/pprof/goroutine)
>
> **Description**: Goroutine profile

> [accordion] [http://localhost:6060/debug/pprof/allocs](http://localhost:6060/debug/pprof/allocs)
>
> **Description**: Memory allocation profile

> [accordion] [http://localhost:6060/debug/pprof/block](http://localhost:6060/debug/pprof/block)
>
> **Description**: Block profile

> [accordion] [http://localhost:6060/debug/pprof/mutex](http://localhost:6060/debug/pprof/mutex)
>
> **Description**: Mutex profile

> [accordion] [http://localhost:6060/debug/pprof/trace](http://localhost:6060/debug/pprof/trace)
>
> **Description**: Execution trace

#### 2. File Output (Snapshot Command Only)

For the `snapshot` command, profiling also generates profile files:

* **`cpu.prof`** - CPU profile for the entire snapshot operation
* **`mem.prof`** - Memory profile taken at the end of the operation

### Using Profiling Data

#### With Go's pprof Tool

1. **CPU Hotspots**: Identify functions consuming the most CPU time

   ```bash theme={null}
   go tool pprof -http=:8080 http://localhost:6060/debug/pprof/profile
   ```

2. **Memory Usage**: Find memory allocation patterns and potential leaks

   ```bash theme={null}
   go tool pprof -http=:8080 http://localhost:6060/debug/pprof/heap
   ```

3. **Goroutine Analysis**: Debug goroutine leaks or blocking operations

   ```bash theme={null}
   go tool pprof -http=:8080 http://localhost:6060/debug/pprof/goroutine
   ```

4. **Blocked Operations**:
   ```bash theme={null}
   go tool pprof -http=:8080 http://localhost:6060/debug/pprof/block
   ```

### Considerations

* **Local Only**: The profiling server binds to `localhost:6060` and is not accessible externally
* **Disable in Production**: Only enable profiling for debugging/optimization sessions
* **Performance Impact**: Profiling has minimal overhead but should be used carefully

## Best Practices

1. **Use the Pre-built Dashboard**: Start with the included SigNoz dashboard for immediate visibility
2. **Monitor Both Sides**: Track both pgstream metrics AND source PostgreSQL replication metrics
3. **Watch Runtime Metrics**: Monitor memory usage, GC performance, and goroutine counts for application health
4. **Set Up Comprehensive Alerting**: Configure alerts for both application and infrastructure metrics
5. **Regular Health Checks**: Use the dashboard to perform regular health assessments
6. **Analyze Traces for Debugging**: Leverage distributed tracing for complex issue resolution
7. **Capacity Planning**: Use historical metrics data to plan for scaling needs
8. **Enable Profiling Only When Needed**: Don't run profiling continuously in production
9. **Collect Sufficient Profile Data**: Let profiling run for adequate time to collect meaningful samples

## Troubleshooting

* **High Memory Usage**: Check `runtime.go.mem.heap_alloc` and look for memory leaks in processing pipelines
* **Goroutine Leaks**: Monitor `runtime.go.goroutines` and investigate if count keeps growing
* **GC Pressure**: High `runtime.go.gc.count` rate may indicate memory allocation issues
* **High Lag**: Check both `pgstream.replication.lag` and PostgreSQL `pg_stat_replication` for the root cause
* **Write Failures**: Monitor `pgstream.kafka.writer.latency` and check Kafka connectivity
* **Slow Snapshots**: Use `pgstream.snapshot.generator.latency` with schema/table attributes to identify problematic tables
* **Query Performance**: Analyze `pgstream.postgres.querier.latency` by `query_type` to find slow operations
* **Missing Data**: Check PostgreSQL replication slot status and WAL retention policies

## Getting Help

If you encounter issues with observability setup:

1. Check the SigNoz dashboard for immediate insights
2. Review both pgstream and PostgreSQL metrics
3. Examine Go runtime metrics for application health issues
4. Use profiling to drill down into performance bottlenecks
5. Examine distributed traces for detailed execution flow
6. Consult the troubleshooting section above
7. Check PostgreSQL logs for replication related errors


# Architecture
Source: https://xata.io/docs/opensource/pgstream/docs/architecture

`pgstream` is constructed as a streaming pipeline, where data from one module streams into the next, eventually reaching the configured targets. It keeps track of schema changes and replicates them along with the data changes to ensure a consistent view of the source data downstream. This modular approach makes adding and integrating target implementations simple and painless.

![pgstream architecture v2](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_diagram_v2.svg)

![pgstream architecture with kafka v2](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_diagram_v2_kafka.svg)

At a high level the implementation is split into WAL listeners and WAL processors.

## WAL Listener

A listener is anything that listens for WAL data, regardless of the source. It has a single responsibility: consume and manage the WAL events, delegating the processing of those entries to modules that form the processing pipeline. Depending on the listener implementation, it might be required to have a checkpointer to flag the events as processed once the processor is done.

The current implementations of the listener include:

* **Postgres listener**: listens to WAL events directly from the replication slot. Since the WAL replication slot is sequential, the Postgres WAL listener is limited to run as a single process. The associated Postgres checkpointer will sync the LSN so that the replication lag doesn't grow indefinitely. It can be configured to perform an initial snapshot when pgstream is first connected to the source PostgreSQL database (see details in the [snapshots documentation](/docs/opensource/pgstream/docs/snapshots)).

* **Postgres Snapshoter**: produces events by performing a snapshot of the configured PostgreSQL database, as described in the [snapshots documentation](/docs/opensource/pgstream/docs/snapshots). It doesn't start continuous replication, so once all the snapshotted data has been processed, the pgstream process will stop.

* **Kafka reader**: reads WAL events from a Kafka topic. It can be configured to run concurrently by using partitions and Kafka consumer groups, applying a fan-out strategy to the WAL events. The data will be partitioned by database schema by default. The associated Kafka checkpointer will commit the message offsets per topic/partition so that the consumer group doesn't process the same message twice, and there's no lag accumulated.

## WAL Processor

A processor processes a WAL event. Depending on the implementation it might also be required to checkpoint the event once it's done processing it as described above. Wherever possible the processors are implemented to continuously consume the replication slot by using configurable memory guards, aiming to prevent the replication slot lag from growing out of control.

The current implementations of the processor include:

* **Kafka batch writer**: it writes the WAL events into a Kafka topic, using the event schema as the Kafka key for partitioning. This implementation allows to fan-out the sequential WAL events, while acting as an intermediate buffer to avoid the replication slot to grow when there are slow consumers. It has a memory guarded buffering system internally to limit the memory usage of the buffer. The buffer is sent to Kafka based on the configured linger time and maximum size. It treats both data and schema events equally, since it doesn't care about the content.

* **Search batch indexer**: it indexes the WAL events into an OpenSearch/Elasticsearch compatible search store. It implements the same kind of mechanism than the Kafka batch writer to ensure continuous processing from the listener, and it also uses a batching mechanism to minimise search store calls. The WAL event identity is used as the search store document id, and the LSN is used as the document version. Events that do not have an identity are not indexed.

* **Webhook notifier**: it sends a notification to any webhooks that have subscribed to the relevant wal event. It relies on a subscription HTTP server receiving the subscription requests and storing them in the shared subscription store which is accessed whenever a wal event is processed. It sends the notifications to the different subscribed webhook urls in parallel based on a configurable number of workers (client timeouts apply). Similar to the two previous processor implementations, it uses a memory guarded buffering system internally, which allows to separate the wal event processing from the webhook url sending, optimising the processor latency.

* **Postgres batch writer**: it writes the WAL events into a PostgreSQL compatible database. It implements the same kind of mechanism than the Kafka and the search batch writers to ensure continuous processing from the listener, and it also uses a batching mechanism to minimise PostgreSQL IO traffic.

In addition to the implementations described above, there are optional processor decorators, which work in conjunction with one of the main processor implementations described above. Their goal is to act as modifiers to enrich the wal event being processed. We will refer to them as modifiers.

### Modifiers

There current implementations of the processor that act as modifier decorators are:

* **Injector**: injects some of the pgstream logic into the WAL event. This includes:
  * Data events:

  * Setting the WAL event identity. It will use the table primary key/unique not null column.

  * Adding pgstream IDs to all columns. This allows us to have a constant identifier for a column, so that if there are renames the column id doesn't change. This is particularly helpful for the search store, where a rename would require a reindex, which can be costly depending on the data.

  * Schema events:

  * Extracting pgstream ids from the DDL event and caching them to be used during the data event injection step.

* **Filter**: allows to filter out WAL events for certain schemas/tables. It can be configured by providing either an include or exclude table list. WAL events for the tables in the include list will be processed, while those for the tables in the exclude list or not present in the include list will be skipped. The format for the lists is similar to the snapshot tables, tables are expected to be schema qualified, and if not, the `public` schema will be assumed. Wildcards are supported, but not regex. Example of table list: `["test_table", "public.test_table", "test_schema.test_table", "test_schema.*", "*.test", "*.*"]`. By default, the filter will include the `pgstream.schema_log` table in the include table list, since pgstream relies on it to replicate DDL changes. It can be disabled by adding it to the exclude list.

  A third list, `schema_only_tables`, can be combined with either the include or the exclude list (or used on its own): DDL (schema change) events for these tables are processed, while their data (DML) events are skipped. This keeps the table schema in sync on the target without replicating its data. The precedence rules are: the exclude list wins over everything; a table listed explicitly (no wildcards) in the include list gets full treatment even when it matches a schema-only wildcard; otherwise a schema-only match keeps DDL and drops DML. A DDL statement referencing multiple tables is replayed as a whole and is skipped if any of the referenced tables is filtered out for DDL (schema-only tables count as allowed).

* **Transformer**: it modifies the column values in insert/update events according to the rules defined in the configured yaml file. It can be used for anonymising data from the source Postgres database. An example of the rules definition file can be found in the repo under `transformer_rules.yaml`. The rules have per column granularity, and certain transformers from opensource sources, such as greenmask or neosync, are supported. More details can be found in the [transformers documentation](/docs/opensource/pgstream/docs/transformers).

## Tracking schema changes

One of the main differentiators of pgstream is the fact that it tracks and replicates schema changes automatically.

In version 0, it relied on SQL triggers that would populate a Postgres table (`pgstream.schema_log`) containing a history log of all DDL changes for a given schema. Whenever a schema change occurred, this trigger created a new row in the schema log table with the schema encoded as a JSON value. This table tracked all the schema changes, forming a linearised change log that is then parsed and used within the pgstream pipeline to identify modifications and push the relevant changes downstream.

From version 1 onwards, schema changes are captured using event triggers that emit logical messages directly into the WAL, and are then processed downstream by the relevant processors. This new approach reduces a lot of overhead, since state is no longer required (no `pgstream.schema_log` table needed), as well as simplifying the delta when applying schema changes, which had to be computed each time with the previous approach.

![v1 schema replication](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_v1_schema_replication.png)

The detailed SQL used for both versions can be found in the [migrations folder](https://github.com/xataio/pgstream/tree/main/migrations/postgres).

The schema and data changes are part of the same linear stream - the downstream consumers always observe the schema changes as soon as they happen, before any data arrives that relies on the new schema. This prevents data loss and manual intervention.


# AWS
Source: https://xata.io/docs/opensource/pgstream/docs/aws

This guide explains how to use **pgstream** with **AWS RDS/Aurora databases**, covering **snapshots** and **replication**.

👉 Throughout this guide:

* **`pgstreamsource`** refers to the user provided in the **pgstream source URL**.
* **`pgstreamtarget`** refers to the user provided in the **pgstream target URL**.

## Snapshots

### From AWS RDS/Aurora Database (Snapshots)

#### Quick Checklist

* [ ] Point the source URL at the **writer / single-instance endpoint**, **not** the Aurora reader (`cluster-ro`) or an RDS reader endpoint (see warning below).
* [ ] Create a source user (**`pgstreamsource`**) with access to required schemas/tables.
* [ ] For roles without passwords → no special config needed.
* [ ] Snapshot of roles with passwords is **not supported**.
* [ ] Update YAML config with correct snapshot settings.

> ⚠️ **Do not snapshot from an Aurora reader (`cluster-ro`) or RDS reader endpoint.** These
> load-balance across multiple instances, and the parallel data snapshot relies on an
> instance-local exported transaction snapshot. Worker connections that land on a different
> reader fail nondeterministically with `snapshot "…" does not exist`. Use the
> **writer/cluster endpoint** (a single instance). See [Snapshots](/docs/opensource/pgstream/docs/snapshots#️-the-data-snapshot-source-must-be-a-single-instance) for details; `pgstream check` flags this before a snapshot runs.

#### Steps

1. **User privileges**
   Ensure the **`pgstreamsource`** user (from the pgstream source URL) can access the database schema and tables you need.

2. **Roles handling**

   * **No roles** → no changes required.
   * **Roles without passwords** → no changes required.
   * **Roles with passwords** → **not supported by AWS**.

     If you try to snapshot roles with passwords, you'll see:

     ```
     pg_dumpall: error: query failed: ERROR:  permission denied for table pg_authid
     ```

3. **Config when not snapshotting roles**
   If roles are disabled or not manually managed, add the following to avoid failures:

   ```yaml theme={null}
   snapshot:
     schema:
       pgdump_pgrestore:
         roles_snapshot_mode: "disabled"
         no_owner: true
         no_privileges: true
   ```

   Example full configuration:

   ```yaml theme={null}
   source:
     postgres:
       url: "postgresql://pgstreamsource:password@:5432/db?sslmode=require"
       mode: snapshot
       snapshot:
         mode: full # schema + data
         tables: ["public.*"] # all tables in the public schema
         schema:
           mode: pgdump_pgrestore
           pgdump_pgrestore:
             roles_snapshot_mode: "disabled"
             no_owner: true
             no_privileges: true
   ```

ℹ️ AWS RDS/Aurora-managed roles (`rdsadmin`, `rds_reserved`, `rds_extension`, `rds_ad`, `rds_password`, `rds_iam`, `rds_replication`, `rds_superuser` and `postgres`) will not be snapshotted.

### To AWS RDS/Aurora Database (Snapshots)

#### Quick Checklist

* [ ] Create a target user (**`pgstreamtarget`**) for the pgstream target URL.
* [ ] Grant privileges for schema/database ownership.
* [ ] Grant optional privileges depending on features (create DB, create roles).
* [ ] **Use `disable_triggers: false`** (required for all non-`rds_superuser` users).

#### Steps

The **`pgstreamtarget`** user (from the pgstream target URL) must have the following privileges:

* **Schema ownership**

  ```sql theme={null}
  ALTER DATABASE db OWNER TO pgstreamtarget;
  ALTER SCHEMA public OWNER TO pgstreamtarget;
  ```

* **Database creation** (if `create_target_db` is enabled)

  ```sql theme={null}
  ALTER ROLE pgstreamtarget CREATEDB;
  ```

* **Role creation** (if `roles_snapshot_mode` is `enabled`/`no_passwords`)

  ```sql theme={null}
  ALTER ROLE pgstreamtarget CREATEROLE;
  ```

  ⚠️ `pgstreamtarget` must already hold any privileges it assigns (e.g., `REPLICATION`).

* **Disable triggers** (required for AWS RDS/Aurora)

  ⚠️ **AWS RDS/Aurora only allows setting `session_replication_role`** for `rds_superuser` roles. You **must** disable trigger management in pgstream if using a non superuser role:

  ```yaml theme={null}
  target:
    postgres:
      disable_triggers: false # Required for AWS RDS/Aurora target databases when using non superuser roles
  ```

  ❌ If using `disable_triggers: true` and a role without enough privileges, pgstream will fail with:

  ```
  permission denied to set parameter "session_replication_role"
  ```

## Replication

### From AWS RDS/Aurora Database (Replication)

#### Quick Checklist

* [ ] Ensure logical replication is enabled (`rds.logical_replication=1` in parameter group).
* [ ] **Use the `postgres` user or a user with `rds_superuser`** for initialization.
* [ ] Grant `rds_replication` role for replication privileges.
* [ ] Optionally use a different **`pgstreamsource`** user for streaming.

#### Steps

1. **Enable logical replication**

   Logical replication requires `wal_level` to be set to `logical`.

   **Check current setting:**

   ```sql theme={null}
   SHOW wal_level;
   ```

   **If not set to `logical`, update it:**

   AWS RDS and Aurora require modifying the database parameter group:

   1. Go to the [AWS RDS Console](https://console.aws.amazon.com/rds/)
   2. Navigate to **Parameter groups** in the left sidebar
   3. Create a new parameter group or select your existing custom parameter group
      * **Note:** You cannot modify the default parameter groups
   4. Search for and modify the `rds.logical_replication` parameter:
      * Set `rds.logical_replication` to `1` (enabled)
   5. Apply the parameter group to your RDS/Aurora instance:
      * Go to your database instance
      * Click **Modify**
      * Under **Database options**, select your custom parameter group
      * Apply the changes
   6. **Reboot your database instance** for the changes to take effect

   **Verification:**

   After the reboot, verify the setting:

   ```sql theme={null}
   SHOW wal_level;  -- Should return 'logical'
   SHOW rds.logical_replication;  -- Should return 'on' or '1'
   ```

   ⚠️ **Important notes:**

   * Setting `rds.logical_replication=1` automatically sets `wal_level=logical`
   * This change **requires a database reboot**
   * Logical replication may increase storage usage due to retained WAL files
   * Ensure you have adequate monitoring for replication slot lag

2. **Replication phases**

   * **Initialization** → requires elevated privileges to create schemas, event triggers, and replication slots.
   * **Streaming** → can use a dedicated `pgstreamsource` user.

#### Initialization

Initialization does the following:

* Creates `pgstream` schema
* Creates replication slot (if missing)
* Creates event triggers/functions for schema changes

⚠️ **Use a user with `rds_superuser` privileges** (typically the `postgres` user) for initialization, as creating event triggers requires elevated privileges:

```yaml theme={null}
source:
  postgres:
    url: "postgresql://postgres:password@:5432/db?sslmode=require"
    mode: replication
```

**Required privileges for initialization user:**

```sql theme={null}
-- The postgres user typically has these by default
-- If using a different user, grant:
GRANT rds_superuser TO pgstreamsource;
GRANT rds_replication TO pgstreamsource;
```

#### Streaming

After initialization, you can:

* Keep using `postgres`, or
* Transfer ownership to a dedicated **`pgstreamsource`** role.

```sql theme={null}
-- Allow ownership transfer
GRANT pgstreamsource TO current_user;

-- Transfer ownership
ALTER DATABASE db OWNER TO pgstreamsource;
ALTER SCHEMA pgstream OWNER TO pgstreamsource;
ALTER SCHEMA <schema> OWNER TO pgstreamsource;
ALTER TABLE pgstream.schema_log OWNER TO pgstreamsource;
ALTER TABLE pgstream.table_ids OWNER TO pgstreamsource;
ALTER SEQUENCE pgstream.xid_serial OWNER TO pgstreamsource;

-- Replication privileges (AWS RDS/Aurora specific)
GRANT rds_replication TO pgstreamsource;
```

### To AWS RDS/Aurora Database (Replication)

#### Quick Checklist

* [ ] Ensure **`pgstreamtarget`** has schema/database ownership.
* [ ] If combined with snapshot → follow [Snapshot target requirements](#to-aws-rdsaurora-database-snapshots).

#### Steps

Privileges required:

```sql theme={null}
ALTER DATABASE db OWNER TO pgstreamtarget;
ALTER SCHEMA <schema> OWNER TO pgstreamtarget;
```

## Troubleshooting

> [accordion] permission denied for table pg\_authid
>
> **Cause**: Roles with passwords require pg\_authid access.\
> **Fix**: AWS RDS doesn't allow access to `pg_authid`. Disable role passwords (`roles_snapshot_mode: disabled` or `no_passwords`).

> [accordion] permission denied to set parameter "session\_replication\_role"
>
> **Cause**: AWS RDS only allows setting this parameter when using an `rds_superuser` role.\
> **Fix**: Set `disable_triggers: false` in target config or use a role with enough privileges.

> [accordion] permission denied for schema public when transferring ownership
>
> **Cause**: Target role lacks CREATE privilege on schema.\
> **Fix**: Grant CREATE on schema to the role: `GRANT CREATE ON SCHEMA public TO role_name;`

> [accordion] Role creation fails
>
> **Cause**: `pgstreamtarget` lacks privileges it tries to assign.\
> **Fix**: Grant required privileges to `pgstreamtarget` first, or use a user with more privileges.

> [accordion] must be superuser to create event triggers
>
> **Cause**: Initialization user lacks sufficient privileges.\
> **Fix**: Use the `postgres` user or grant `rds_superuser`: `GRANT rds_superuser TO username;`

> [accordion] cannot set parameter "rds.logical\_replication"
>
> **Cause**: Parameter can only be set in parameter group.\
> **Fix**: Modify the parameter group via AWS Console, not via SQL.


# CLI
Source: https://xata.io/docs/opensource/pgstream/docs/cli

pgstream is a command-line tool for streaming PostgreSQL data changes to various targets. This document covers all available commands and their usage.

## Installation

```bash theme={null}
# Download the latest release
curl -L https://github.com/xataio/pgstream/releases/latest/download/pgstream.linux.amd64 -o pgstream
chmod +x pgstream
sudo mv pgstream /usr/local/bin/

# Or use go install
go install github.com/xataio/pgstream@latest

# Or build from source
git clone https://github.com/xataio/pgstream.git
cd pgstream
go build -o pgstream ./cmd

# Or install via homebrew on macOS or Linux
brew tap xataio/pgstream
brew install pgstream
```

## Global Flags

These flags are available for all commands:

> [accordion] --config, -c
>
> **Description**: .env or .yaml config file to use with pgstream if any\
> **Default**: -

> [accordion] --log-level
>
> **Description**: Log level for the application. One of trace, debug, info, warn, error, fatal, panic\
> **Default**: `debug`

> [accordion] --log-format
>
> **Description**: Log output format. One of `console` (human-readable) or `json` (structured)\
> **Default**: `console`

> [accordion] --no-color
>
> **Description**: Disable ANSI color codes in console log output (ignored when `--log-format=json`)\
> **Default**: `false`

> [accordion] --help, -h
>
> **Description**: Show help information\
> **Default**: -

Every global flag also has an equivalent `PGSTREAM_`-prefixed environment variable (for example, `PGSTREAM_LOG_FORMAT=json`, `PGSTREAM_LOG_NO_COLOR=true`). Use `--log-format=json` when shipping logs to cloud aggregators like GCP Cloud Logging or Datadog that parse structured JSON.

## Commands

### init

Initialises pgstream, creating the replication slot and the relevant tables/functions/triggers under the configured internal pgstream schema. It performs the same operations as the `--init` flag on the `run` command.

```bash theme={null}
pgstream init [flags]
```

**Description:**
The `init` command prepares your PostgreSQL database for streaming by:

* Creating a logical replication slot with the specified name
* Creating the internal pgstream schema for tracking changes
* Setting up necessary functions and triggers for change data capture
* Configuring the database objects required for logical replication

**Prerequisites:**

* PostgreSQL must have `wal_level = logical`
* User must have replication privileges (`REPLICATION` role)
* `max_replication_slots` must allow for additional slots
* User must have privileges to create schemas, functions, and triggers

**Flags:**

* `--postgres-url` - Source postgres URL where pgstream setup will be run
* `--replication-slot` - Name of the postgres replication slot to be created by pgstream on the source url
* `--with-injector` - Whether to initialize pgstream with the injector database migrations. Required for search targets (OpenSearch/Elasticsearch)
* `--migrations-only` - Whether to only run the database migrations without creating the replication slot
* `--slot-only` - Whether to only create the replication slot, without running the database migrations

**Examples:**

```bash theme={null}
pgstream init --postgres-url <source-postgres-url> --replication-slot <replication-slot-name>
pgstream init -c config.yaml
pgstream init -c config.env
pgstream init -c config.yaml --migrations-only
pgstream init --postgres-url <replica-postgres-url> --slot-only
```

**Note:** The `--migrations-only` flag runs only the database migrations (creating the pgstream schema, tables, functions, and triggers) without creating the replication slot. This is useful when you want to set up the schema separately or when using different database credentials for migrations versus replication.

**Note:** The `--slot-only` flag is the complement: it creates only the replication slot, skipping the schema and migrations entirely. This is what a **read replica** source needs, since a standby is read only and cannot take the migrations, while the schema and event trigger reach it through physical replication from the primary. See [Running pgstream from a read replica](/docs/opensource/pgstream/docs/replicas). The two flags are mutually exclusive.

### run

Run starts a continuous data stream from the configured source to the configured target.

```bash theme={null}
pgstream run [flags]
```

**Description:**
The `run` command is the main operation mode for pgstream. It:

* Establishes a connection to the source database
* Connects to the existing replication slot (created by `init`)
* Continuously reads WAL events from the replication stream
* Processes and transforms data according to configuration
* Streams changes to configured targets (Kafka, PostgreSQL, Elasticsearch, OpenSearch)
* Runs continuously until interrupted (Ctrl+C) or receives a termination signal
* Gracefully shuts down on SIGTERM/SIGINT
* Resumes from the last confirmed WAL position

**Prerequisites:**

* Database must be initialized with `pgstream init`
* Replication slot must exist and be available
* Target systems must be accessible and properly configured
* Source database must have logical replication enabled

**Flags:**

* `--source` - Source type. One of postgres, kafka
* `--source-url` - Source URL
* `--target` - Target type. One of postgres, opensearch, elasticsearch, kafka
* `--target-url` - Target URL
* `--replication-slot` - Name of the postgres replication slot for pgstream to connect to
* `--snapshot-tables` - List of tables to snapshot if initial snapshot is required, in the format `<schema>.<table>`. If not specified, the schema `public` will be assumed. Wildcards are supported
* `--reset` - Whether to reset the target before snapshotting (only for postgres target). ⚠️ Destructive: the target objects are dropped before the table data is copied, so a snapshot that fails afterwards leaves the target partially rebuilt with the previous contents gone. See [resetting the target](/docs/opensource/pgstream/docs/snapshots#️-resetting-the-target-destroys-it-before-the-new-data-lands)
* `--profile` - Whether to expose a /debug/pprof endpoint on localhost:6060
* `--init` - Whether to initialize pgstream before starting replication
* `--dump-file` - File where the pg\_dump output will be written if initial snapshot is enabled when using pgdump/restore
* `--data-only` - When used with `--snapshot-tables`, skip schema restore and only snapshot data. Use this when the schema is already present on the target. Defaults to `false`
* `--with-injector` - Whether to enable the injection of pgstream metadata to the WAL events. Required for search targets (OpenSearch/Elasticsearch)

**Examples:**

```bash theme={null}
pgstream run --source postgres --source-url <source-postgres-url> --target postgres --target-url <target-postgres-url> --init
pgstream run --source postgres --source-url <source-postgres-url> --target postgres --target-url <target-postgres-url> --snapshot-tables <schema.table> --reset
pgstream run --source kafka --source-url <kafka-url> --target elasticsearch --target-url <elasticsearch-url>
pgstream run --source postgres --source-url <postgres-url> --target kafka --target-url <kafka-url>
pgstream run --config config.yaml --log-level info
pgstream run --config config.env
```

**Output Files (when `--profile` is enabled):**

* `cpu.prof` - CPU profiling data for performance analysis
* `mem.prof` - Memory allocation profiling data

### snapshot

Snapshot performs a one-time data snapshot of a PostgreSQL database. For continuous replication or combined snapshot+replication, use the `run` command with `--snapshot-tables` flag.

```bash theme={null}
pgstream snapshot [flags]
```

**Description:**
The `snapshot` command creates a point-in-time copy of database tables. It:

* Connects to the source PostgreSQL database
* Reads all existing data from specified tables/schemas
* Transforms and streams the data to configured targets
* Exits after completing the snapshot operation

**Prerequisites:**

* Source PostgreSQL database must be accessible
* Target system must be accessible and properly configured
* User must have SELECT privileges on tables to be snapshotted
* For PostgreSQL targets: user must have write privileges

**Flags:**

* `--postgres-url` - Source postgres database to perform the snapshot from
* `--target` - Target type. One of postgres, opensearch, elasticsearch, kafka
* `--target-url` - Target URL
* `--tables` - List of tables to snapshot, in the format `<schema>.<table>`. If not specified, the schema `public` will be assumed. Wildcards are supported
* `--reset` - Whether to reset the target before snapshotting (only for postgres target). ⚠️ Destructive: the target objects are dropped before the table data is copied, so a snapshot that fails afterwards leaves the target partially rebuilt with the previous contents gone. See [resetting the target](/docs/opensource/pgstream/docs/snapshots#️-resetting-the-target-destroys-it-before-the-new-data-lands)
* `--profile` - Whether to produce CPU and memory profile files, as well as exposing a /debug/pprof endpoint on localhost:6060
* `--dump-file` - File where the pg\_dump output will be written

**Examples:**

```bash theme={null}
# Snapshot specific tables from a PostgreSQL database to a target PostgreSQL database
pgstream snapshot --postgres-url <postgres-url> --target postgres --target-url <target-url> --tables <schema.table> --reset
# Snapshot using a YAML configuration file (requires source.postgres.mode: 'snapshot')
pgstream snapshot --config config.yaml --log-level info
# Snapshot using an environment configuration file
pgstream snapshot --config config.env
```

**Use Cases:**

* Bulk data export for analytics
* Creating test datasets
* Backfilling data after system setup

**Output Files (when `--profile` is enabled):**

* `cpu.prof` - CPU profiling data for performance analysis
* `mem.prof` - Memory allocation profiling data

### status

Checks the status of pgstream initialisation and provided configuration.

```bash theme={null}
pgstream status [flags]
```

**Description:**
The `status` command provides information about:

* Replication slot status
* Internal pgstream schema and objects status
* Overall streaming infrastructure health
* Configuration validation results

**Prerequisites:**

* Access to the source PostgreSQL database
* Replication slot should exist

**Flags:**

* `--postgres-url` - Source postgres URL where pgstream has been initialised
* `--replication-slot` - Name of the postgres replication slot created by pgstream on the source url
* `--json` - Output the status in JSON format

**Examples:**

```bash theme={null}
pgstream status -c pg2pg.env
pgstream status --postgres-url <postgres-url> --replication-slot <replication-slot-name>
pgstream status -c pg2pg.yaml --json
```

**Sample Output:**

```
✅ SUCCESS  pgstream status check encountered no issues
Initialisation status:
 - Pgstream schema exists: true
 - Pgstream schema_log table exists: true
 - Migration current version: 8
 - Migration status: success
 - Replication slot name: pgstream_postgres_slot
 - Replication slot plugin: wal2json
 - Replication slot database: postgres
Config status:
 - Valid: true
Transformation rules status:
 - Valid: true
Source status:
 - Reachable: true
```

### validate

Validate different parts of the pgstream configuration.

```bash theme={null}
pgstream validate <subcommand> [flags]
```

**Description:**
The `validate` command allows you to validate specific aspects of your pgstream configuration before running it. It currently supports validating transformation rules.

#### validate rules

Validates transformation rules against the provided Postgres database schema.

```bash theme={null}
pgstream validate rules [flags]
```

**Description:**
The `validate rules` command checks your transformation rules for:

* Column existence and type compatibility
* Table and schema references
* Rule syntax and structure
* Compatibility with the source database schema
* Overall validity before applying them in production

**Prerequisites:**

* Access to the source PostgreSQL database
* Transformation rules defined in configuration or separate rules file

**Flags:**

* `--postgres-url` - Source postgres URL to validate the rules against
* `--rules-file`, `-f` - Path to a YAML file containing the transformation rules to validate
* `--json` - Output the validation status in JSON format

**Examples:**

```bash theme={null}
pgstream validate rules -c pg2pg.env
pgstream validate rules --postgres-url <postgres-url> --rules-file rules.yaml
pgstream validate rules -c pg2pg.yaml --json
```

**Sample Output:**

```
✅ SUCCESS  transformation rules are valid
Transformation rules status:
 - Valid: true
 - Errors: []
```

**Use Cases:**

* Pre-deployment validation of transformation rules
* Testing rule changes against production schema
* CI/CD pipeline integration for rule validation
* Debugging transformation rule issues

### destroy

It destroys any pgstream setup, removing the replication slot and all the relevant tables/functions/triggers, along with the internal pgstream schema.

```bash theme={null}
pgstream destroy [flags]
```

**Description:**
The `destroy` command cleans up all resources created by `pgstream init`:

* Drops the replication slot
* Removes the internal pgstream schema and all its objects
* Removes all pgstream-related functions and triggers
* **⚠️ Warning:** This is destructive and will lose replication position

**Prerequisites:**

* Access to the source PostgreSQL database
* User must have privileges to drop schemas, functions, and replication slots
* pgstream should be initialized (objects should exist to be destroyed)

**Flags:**

* `--postgres-url` - Source postgres URL where pgstream destroy will be run
* `--replication-slot` - Name of the postgres replication slot to be deleted by pgstream from the source url
* `--with-injector` - Whether to also destroy the injector related database objects
* `--migrations-only` - Whether to only revert the database migrations without dropping the replication slot
* `--slot-only` - Whether to only drop the replication slot, leaving the pgstream schema and migrations in place

**Examples:**

```bash theme={null}
pgstream destroy --postgres-url <source-postgres-url> --replication-slot <replication-slot-name>
pgstream destroy -c config.yaml
pgstream destroy -c config.env
pgstream destroy -c config.yaml --migrations-only
pgstream destroy --postgres-url <postgres-url> --slot-only
```

**Note:** The `--migrations-only` flag reverts only the database migrations (removing the pgstream schema, tables, functions, and triggers) without dropping the pgstream schema (with any tables that it might contain, such as the snapshot recorder), or dropping the replication slot. This is useful for minimal downtime migrations where you want to preserve the replication slot position.

**Note:** The `--slot-only` flag is the complement: it drops only the replication slot and leaves the pgstream schema and the `emit_ddl` event trigger in place. Use it to remove a slot that is no longer consumed — an unused logical slot pins WAL indefinitely — without disturbing DDL replication for anything still streaming. The two flags are mutually exclusive.

**⚠️ Important Notes:**

* Without `--migrations-only`:
  * It will drop the replication slot and you will lose the current replication position
  * It will drop the pgstream schema along with any tables within it (such as the snapshot recorder table)
* With `--migrations-only`:
  * The replication slot is preserved, allowing you to maintain replication position across migrations
  * The pgstream schema is preserved, allowing non migration tables to remain
* This will stop any running pgstream instances using these resources
* All pgstream tracking data will be permanently removed
* You can recreate resources later with `pgstream init`

### version

Displays version information for pgstream.

```bash theme={null}
pgstream version
```

**Description:**
Shows the current version of the pgstream binary.

**Example:**

```bash theme={null}
$ pgstream version
pgstream version v0.8.1
```

## Configuration

pgstream uses YAML or .env configuration files. The configuration can be specified via:

1. **Command-line flag:** `--config /path/to/config.yaml` or `-c /path/to/config.env`
2. **Environment variable:** `PGSTREAM_CONFIG=/path/to/config.yaml`

For more information about the configuration options, check out the [configuration documentation](/docs/opensource/pgstream/docs/configuration).

## Examples

### Complete Setup Workflow

```bash theme={null}
# Start continuous streaming with initial snapshot and initialization step
pgstream run --source postgres --source-url "postgres://user:pass@localhost:5432/source_db" --target postgres --target-url "postgres://user:pass@localhost:5432/target_db" --replication-slot "pgstream_slot" --snapshot-tables "public.*" --init
```

### Configuration File Workflow

```bash theme={null}
# 1. Create configuration file
cat > config.yaml <<EOF
source:
  postgres:
    url: "postgres://user:pass@localhost:5432/source_db"
    replication:
      replication_slot: "pgstream_slot"

target:
  postgres:
    url: "postgres://user:pass@localhost:5432/target_db"
EOF

# 2. Initialize with config file
pgstream init -c config.yaml

# 3. Check status
pgstream status -c config.yaml

# 4. Start streaming
pgstream run -c config.yaml
```

### Development with Profiling

```bash theme={null}
# Run snapshot with profiling enabled
pgstream snapshot -c config.yaml --profile

# Run continuous streaming with profiling
pgstream run -c config.yaml --profile

# In another terminal, analyze performance
go tool pprof http://localhost:6060/debug/pprof/profile
go tool pprof http://localhost:6060/debug/pprof/heap
```

### Multi-target Streaming

```bash theme={null}
# Stream to Kafka
pgstream run --source postgres --source-url "postgres://user:pass@localhost:5432/source_db" --target kafka --target-url "localhost:9092" --replication-slot "pgstream_slot"

# Stream from Kafka to Elasticsearch
pgstream run --source kafka --source-url "localhost:9092" --target elasticsearch --target-url "http://localhost:9200" --replication-slot "pgstream_slot"

# Stream from Kafka to OpenSearch
pgstream run --source kafka --source-url "localhost:9092" --target opensearch --target-url "http://localhost:9200" --replication-slot "pgstream_slot"
```

### Environment Variable Configuration

```bash theme={null}
# Set environment variables
export PGSTREAM_POSTGRES_LISTENER_URL="postgres://user:pass@localhost:5432/source_db"
export PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME="pgstream_slot"
export PGSTREAM_LOG_LEVEL="info"
export PGSTREAM_LOG_FORMAT="json"

# Run with environment configuration
pgstream init
pgstream status
pgstream run --source postgres --target kafka --target-url "localhost:9092"
```

## Troubleshooting

### Common Issues

**1. Initialization Failures**

```bash theme={null}
# Check PostgreSQL configuration
psql "postgres://user:password@host:port/database" -c "SHOW wal_level;"
psql "postgres://user:password@host:port/database" -c "SHOW max_replication_slots;"

# Verify permissions
psql "postgres://user:password@host:port/database" -c "SELECT rolreplication FROM pg_roles WHERE rolname = 'your_user';"
```

**2. Replication Slot Conflicts**

```bash theme={null}
# Check existing slots
pgstream status --postgres-url "postgres://user:pass@host:port/db" --replication-slot "slot_name"

# Remove conflicting slot if necessary
pgstream destroy --postgres-url "postgres://user:pass@host:port/db" --replication-slot "slot_name"
```

**3. Status Check Issues**

```bash theme={null}
# Get JSON output for detailed debugging
pgstream status -c config.yaml --json

# Check specific postgres URL and slot
pgstream status --postgres-url "postgres://user:pass@host:port/db" --replication-slot "slot_name"
```

### Command-specific Troubleshooting

**Init Command:**

```bash theme={null}
# Debug with verbose logging
pgstream init --postgres-url "postgres://user:pass@host:port/db" --replication-slot "slot_name" --log-level debug
```

**Run Command:**

```bash theme={null}
# Start with initialization if needed
pgstream run -c config.yaml --init

# Run with initial snapshot
pgstream run -c config.yaml --snapshot-tables "public.users,public.orders"
```

**Snapshot Command:**

```bash theme={null}
# Snapshot with schema reset (postgres target only)
pgstream snapshot -c config.yaml --reset

# Snapshot with profiling and custom dump file
pgstream snapshot -c config.yaml --profile --dump-file custom_dump.sql
```

### Best Practices

* Use `run` with the `--init` flag to ensure pgstream can properly replicate schema changes
* `snapshot` only requires read access on your Postgres source, so it's a good alternative for non invasive syncs
* If you need a snapshot and replication, use `run` with initial snapshot to prevent data loss
* Use the `status` command to validate your transformer configuration and your pgstream replication setup
* Use `destroy` carefully, since it will remove everything used by pgstream, including the replication slot

### Getting Help

For immediate help with any command:

```bash theme={null}
pgstream --help
pgstream init --help
pgstream run --help
pgstream snapshot --help
pgstream status --help
pgstream destroy --help
```

* **GitHub Issues**: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* **Documentation**: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)


# Configuration
Source: https://xata.io/docs/opensource/pgstream/docs/configuration

⚠️ Be aware that a single source and a single target must be provided or the configuration validation will fail.

## Yaml

The pgstream configuration can be provided as a yaml configuration file, which encapsulates the transformation configuration. The following sample shows the format for all supported fields.

```yaml theme={null}
instrumentation:
  metrics:
    endpoint: "0.0.0.0:4317"
    collection_interval: 60 # collection interval for metrics in seconds. Defaults to 60s
    prometheus:
      enabled: true # exposes a Prometheus scrape endpoint on the health server. Defaults to false
      endpoint: "/metrics" # path the Prometheus endpoint is served on. Defaults to /metrics
  traces:
    endpoint: "0.0.0.0:4317"
    sample_ratio: 0.5 # ratio of traces that will be sampled. Must be between 0.0-1.0, where 0 is no traces sampled, and 1 is all traces sampled.

source:
  postgres:
    url: "postgresql://user:password@localhost:5432/mydatabase"
    mode: snapshot_and_replication # options are replication, snapshot or snapshot_and_replication
    snapshot: # when mode is snapshot or snapshot_and_replication
      mode: full # one of full, schema or data. Defaults to full.
      tables: ["test", "test_schema.Test", "another_schema.*"] # tables to snapshot, can be a list of table names or a pattern
      excluded_tables: ["test_schema.Test"] # tables to exclude for snapshot, wildcards are not supported
      schema_only_tables: ["audit_log", "reports.*"] # tables for which only the schema is snapshotted, no data. Wildcards are supported. Requires snapshot mode full or schema. Tables listed explicitly in `tables` take precedence over a schema-only wildcard match; `excluded_tables` take precedence over the schema-only list. When a snapshot recorder is configured, schema-only snapshots are recorded separately from data snapshots, so moving a table from `schema_only_tables` to `tables` triggers its data snapshot on the next run (and moving it the other way doesn't repeat any work). Exception: when `tables` is the full `*.*` wildcard, its completed snapshot record covers all schemas and a promoted table won't be re-snapshotted automatically; use `pgstream snapshot` with the table listed explicitly instead. Sequence values are copied for schema-only tables, so out-of-band data backfills on the target won't collide with source-generated IDs.
      recorder:
        repeatable_snapshots: true # whether to repeat snapshots that have already been taken. Defaults to false
        postgres_url: "postgresql://user:password@localhost:5432/mytargetdatabase" # URL of the database where the snapshot status is recorded
      snapshot_workers: 4 # number of schemas to be snapshotted in parallel. Defaults to 1
      data: # when mode is full or data
        schema_workers: 4 # number of schema tables to be snapshotted in parallel. Defaults to 4
        table_workers: 4 # number of workers to snapshot a table in parallel. Defaults to 4
        batch_bytes: 83886080 # bytes to read per batch (defaults to 80MiB)
        max_connections: 50 # maximum number of connections that the data snapshot can open to Postgres. Should  be higher or equal than the number of schema/table workers.
      schema: # when mode is full or schema
        pgdump_pgrestore:
          clean_target_db: true # whether to clean the target database before restoring. Defaults to false. Destructive: the target objects are dropped before the data is copied, so a snapshot that fails afterwards leaves the target partially rebuilt with the previous contents gone. See docs/snapshots.md for how to keep a good copy
          create_target_db: true # whether to create the database on the target postgres. Defaults to false
          include_global_db_objects: true # whether to include database global objects, such as extensions or triggers, on the schema snapshot. Defaults to false
          no_owner: false # whether to remove ownership commands from the dump. Defaults to false
          no_privileges: false # whether to prevent dumping privilege commands (grant/revoke). Defaults to false
          role: postgres # role name to be used to create the dump
          roles_snapshot_mode: # no_passwords by default. Can be set to disabled to disable roles snapshotting, or can be set to enabled to include role passwords
          exclude_security_labels: ["anon"] # list of providers whose security labels will be excluded from the snapshot. Wildcard supported.
          refresh_materialized_views: false # whether to refresh materialized views (REFRESH MATERIALIZED VIEW ... WITH DATA) after the table data has been restored. Defaults to false
          index_constraint_session_settings: # optional PostgreSQL name=value session settings applied only while restoring indexes and constraints. Each entry must be a whitespace-free name=value pair. Unset or empty preserves existing behavior
            - maintenance_work_mem=4GB
            - max_parallel_maintenance_workers=4
            # statement_timeout=0 and lock_timeout=0 disable the server-side limits that would otherwise bound a stuck restore; the restore then relies on client-side cancellation. Omit them to keep those safety limits.
            # - statement_timeout=0
            # - lock_timeout=0
            # - synchronous_commit=off # faster restore, but a target crash right after the restore can lose the final index/constraint commits
          # Number of standalone CREATE INDEX/CREATE UNIQUE INDEX statements restored concurrently against the target. Statements that depend on an index
          # (constraints, comments, partition attachments, REPLICA IDENTITY, CLUSTER) always wait for every index to be created first.
          # Each worker holds one target connection and applies the index_constraint_session_settings above to it, so the target needs this many free
          # connection slots and up to workers x maintenance_work_mem of memory. Defaults to 1 (sequential), maximum 32.
          index_restore_workers: 1
          dump_file: pg_dump.sql # name of the file where the contents of the schema pg_dump command and output will be written for debugging purposes.
          # Granular object type filtering for schema snapshots. Only one of include_object_types or exclude_object_types can be set.
          # Available categories: tables, sequences, types, indexes, constraints, functions, views, materialized_views, triggers, event_triggers, policies, rules, comments, extensions, collations, text_search
          # include_object_types: ["tables", "sequences", "types"] # only include these object types in the schema snapshot
          # exclude_object_types: ["functions", "views", "triggers"] # exclude these object types from the schema snapshot
      disable_progress_tracking: false # whether to disable progress tracking for the snapshot. Defaults to false
    replication: # when mode is replication or snapshot_and_replication
      replication_slot: "pgstream_mydatabase_slot"
      plugin:
        include_xids: false # whether to include transaction IDs in the replication stream events. Defaults to false.
        add_tables: "public.*" # comma-separated list of tables to include in the wal2json output. Supports wildcards. Filters at the source decode level. Defaults to all tables.
        filter_tables: "pipelines.*,private.*" # comma-separated list of tables to exclude from the wal2json output. Supports wildcards. Filters at the source decode level. Defaults to empty.
    retry_policy: # retry policy for postgres connections, one of exponential or constant or disable_retries
      disable_retries: false
      exponential:
        initial_interval: 500 # initial interval in milliseconds
        max_interval: 10000 # maximum interval in milliseconds
      constant:
        max_retries: 5 # maximum number of retries
        interval: 1000 # interval in milliseconds
  kafka:
    servers: ["localhost:9092"]
    topic:
      name: "mytopic"
    consumer_group:
      id: "mygroup" # id for the kafka consumer group. Defaults to pgstream-consumer-group
      start_offset: "earliest" # options are earliest or latest. Defaults to earliest.
    tls:
      ca_cert: "/path/to/ca.crt" # path to CA certificate
      client_cert: "/path/to/client.crt" # path to client certificate
      client_key: "/path/to/client.key" # path to client key
    sasl: # remove this section to disable SASL authentication
      mechanism: "scram-sha-512" # options are plain, scram-sha-256 or scram-sha-512. Required
      user: "myuser" # user for the SASL authentication. Required
      password: "mypassword" # password for the SASL authentication. Required
    backoff:
      disable_retries: false
      exponential:
        max_retries: 5 # maximum number of retries
        initial_interval: 1000 # initial interval in milliseconds
        max_interval: 60000 # maximum interval in milliseconds
      constant:
        max_retries: 5 # maximum number of retries
        interval: 1000 # interval in milliseconds

target:
  postgres:
    url: "postgresql://user:password@localhost:5432/mytargetdatabase"
    max_connections: 50 # maximum number of connections in the writer pool to the target database. Defaults to 50; overrides pool_max_conns in the URL when set. The schema observer keeps its own pool, capped at 16 and never larger than this value, so the process opens at most this many plus 16.
    batch:
      timeout: 1000 # batch timeout in milliseconds. Defaults to 30s
      size: 100 # number of messages in a batch. Defaults to 20000
      max_bytes: 1572864 # max size of batch in bytes (1.5MiB). Defaults to 1.5MiB without bulk enabled, 80MiB with bulk ingest.
      max_queue_bytes: 104857600 # max size of memory guard queue in bytes (100MiB). Defaults to 100MiB
      ignore_send_errors: false # if true, log and ignore errors during batch sending. Warning: can result in consistency errors.
      auto_tune:
        enabled: true # whether to enable auto tuning of batch bytes. Defaults to false
        min_batch_bytes: 1048576 # minimum batch size in bytes (1MB). Defaults to 1MB
        max_batch_bytes: 52428800 # maximum batch size in bytes (50MB). Defaults to 50MB
        convergence_threshold: 0.01 # convergence threshold as a fraction of max batch size. Defaults to 0.01
    disable_triggers: false # whether to disable triggers on the target database. Defaults to false
    on_conflict_action: "nothing" # options are update, nothing or error. Defaults to error
    strict_mode: false # whether to stop on non-internal query failures instead of dropping them. Defaults to false
    bulk_ingest:
      enabled: true # whether to enable bulk ingest on the target postgres, using COPY FROM (supported for insert only workloads)
      copy_workers: 8 # number of concurrent COPY streams per table when bulk ingesting. Defaults to 8. Only applies when bulk_ingest.enabled is true.
    retry_policy: # retry policy for postgres connections, one of exponential or constant or disable_retries.
      disable_retries: false
      exponential:
        initial_interval: 500 # initial interval in milliseconds
        max_interval: 10000 # maximum interval in milliseconds
      constant:
        max_retries: 5 # maximum number of retries
        interval: 1000 # interval in milliseconds
    ignore_ddl: false # whether to disable processing of DDL events on the target Postgres database. Defaults to false. Consider enabling this if source and target roles have different trust levels (see docs/privileges.md).
    # Selective DDL object type filtering for replication. Only one of include_ddl_object_types or exclude_ddl_object_types can be set. Ignored if ignore_ddl is true.
    # Available categories: tables, sequences, types, indexes, constraints, functions, views, materialized_views, triggers, event_triggers, policies, rules, extensions, collations, text_search
    # include_ddl_object_types: ["tables", "sequences", "types"] # only replicate DDL for these object types
    # exclude_ddl_object_types: ["functions", "views", "triggers"] # skip DDL replication for these object types
  kafka:
    servers: ["localhost:9092"]
    topic:
      name: "mytopic" # name of the Kafka topic
      partitions: 1 # number of partitions for the topic. Defaults to 1
      partition_key: "schema" # message key strategy for DML events, one of schema, table or primary_key. Defaults to schema
      replication_factor: 1 # replication factor for the topic. Defaults to 1
      auto_create: true # whether to automatically create the topic if it doesn't exist. Defaults to false
    tls:
      ca_cert: "/path/to/ca.crt" # path to CA certificate
      client_cert: "/path/to/client.crt" # path to client certificate
      client_key: "/path/to/client.key" # path to client key
    sasl: # remove this section to disable SASL authentication
      mechanism: "scram-sha-512" # options are plain, scram-sha-256 or scram-sha-512. Required
      user: "myuser" # user for the SASL authentication. Required
      password: "mypassword" # password for the SASL authentication. Required
    batch:
      timeout: 1000 # batch timeout in milliseconds. Defaults to 1s
      size: 100 # number of messages in a batch. Defaults to 100
      max_bytes: 1572864 # max size of batch in bytes (1.5MiB). Defaults to 1.5MiB
      max_queue_bytes: 104857600 # max size of memory guard queue in bytes (100MiB). Defaults to 100MiB
      ignore_send_errors: false # if true, log and ignore errors during batch sending. Warning: can result in consistency errors.
  search:
    engine: "elasticsearch" # options are elasticsearch or opensearch
    url: "http://localhost:9200" # URL of the search engine
    hash_doc_ids: false # if true, hash document IDs using SHA256 to avoid exceeding the 512 byte limit
    batch:
      timeout: 1000 # batch timeout in milliseconds. Defaults to 1s
      size: 100 # number of messages in a batch. Defaults to 100
      max_bytes: 1572864 # max size of batch in bytes (1.5MiB). Defaults to 1.5MiB
      max_queue_bytes: 104857600 # max size of memory guard queue in bytes (100MiB). Defaults to 100MiB
      ignore_send_errors: false # if true, log and ignore errors during batch sending. Warning: can result in consistency errors.
    backoff:
      disable_retries: false
      exponential:
        max_retries: 5 # maximum number of retries
        initial_interval: 1000 # initial interval in milliseconds
        max_interval: 60000 # maximum interval in milliseconds
      constant:
        max_retries: 5 # maximum number of retries
        interval: 1000 # interval in milliseconds
  webhooks:
    subscriptions:
      store:
        url: "postgresql://user:password@localhost:5432/mydatabase" # URL of the database where the webhook subscriptions are stored
        cache:
          enabled: true # whether to enable caching for the subscription store. Defaults to false
          refresh_interval: 60 # interval in seconds to refresh the cache
      server:
        address: "localhost:9090" # address of the subscription server
        read_timeout: 60 # read timeout in seconds. Defaults to 5s
        write_timeout: 60 # write timeout in seconds. Defaults to 10s
    notifier:
      worker_count: 4 # number of notifications to be processed in parallel. Defaults to 10
      client_timeout: 1000 # timeout for the webhook client in milliseconds. Defaults to 10s
  stdout: {} # write WAL events as NDJSON to stdout. Useful for debugging and validating the pipeline without a real target.

modifiers:
  injector:
    enabled: true # whether to inject pgstream metadata into the WAL events. Defaults to false
    source_url: "postgres://postgres:postgres@localhost:5432?sslmode=disable" # optional for postgres sources (defaults to source URL), required for non-postgres sources
  filter: # one of include_tables or exclude_tables; schema_only_tables can be combined with either
    include_tables: # list of tables for which events should be allowed. Tables should be schema qualified. If no schema is provided, the public schema will be assumed. Wildcards "*" are supported.
      - "test"
      - "test_schema.test"
      - "another_schema.*"
    exclude_tables: # list of tables for which events should be skipped. Tables should be schema qualified. If no schema is provided, the public schema will be assumed. Wildcards "*" are supported.
      - "excluded_test"
      - "excluded_schema.test"
      - "another_excluded_schema.*"
    schema_only_tables: # list of tables for which DDL (schema change) events are processed but data (DML) events are skipped. Tables should be schema qualified. If no schema is provided, the public schema will be assumed. Wildcards "*" are supported. Tables listed explicitly in include_tables take precedence over a schema-only wildcard match; exclude_tables take precedence over the schema-only list.
      - "audit_log"
      - "reports.*"
  sanitize:
    strip_null_char_bytes: true # strip null bytes (0x00) from string column values. Defaults to false
  transformations:
    validation_mode: relaxed
    table_transformers:
      - schema: public
        table: test
        column_transformers:
          name:
            name: greenmask_firstname
            dynamic_parameters:
              gender:
                column: sex
```

## Environment Variables

Here's a list of all the environment variables that can be used to configure the individual modules, along with their descriptions and default values.

> Byte-size variables (those ending in `_BYTES`) accept either a plain integer or a human-readable size such as `64MiB` or `1GiB`. Units are case-insensitive and interpreted as binary multiples (`1MiB` = 1048576 bytes). The same applies to the equivalent fields in the YAML configuration.

### Sources

#### Postgres Listener

> [accordion] PGSTREAM\_POSTGRES\_LISTENER\_URL
>
> **Default**: N/A\
> **Required**: ✓ Required\
> **Description**: URL of the Postgres database to connect to for replication purposes.

> [accordion] PGSTREAM\_POSTGRES\_REPLICATION\_SLOT\_NAME
>
> **Default**: "pgstream\_dbname\_slot"\
> **Required**: Optional\
> **Description**: Name of the Postgres replication slot name.

> [accordion] PGSTREAM\_POSTGRES\_REPLICATION\_PLUGIN\_INCLUDE\_XIDS
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Whether to include XIDs in the replication events.

> [accordion] PGSTREAM\_POSTGRES\_REPLICATION\_PLUGIN\_ADD\_TABLES
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Comma-separated list of tables to include in the wal2json output. Supports wildcards (e.g. `public.*`). Filters at the source decode level, preventing wal2json from generating JSON for excluded tables.

> [accordion] PGSTREAM\_POSTGRES\_REPLICATION\_PLUGIN\_FILTER\_TABLES
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Comma-separated list of tables to exclude from the wal2json output. Supports wildcards (e.g. `pipelines.*,private.*`). Filters at the source decode level, preventing wal2json from generating JSON for excluded tables.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_MODE
>
> **Default**: "full"\
> **Required**: Optional\
> **Description**: Mode in which the snapshot will be run. It can be one of `schema`, `data` or `full` (both schema and data).

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_TABLES
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Tables for which there will be an initial snapshot generated. The syntax supports wildcards. Tables without a schema defined will be applied the public schema. Example: for `public.test_table` and all tables in the `test_schema` schema, the value would be the following: `"test_table test_schema.\*"`

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_EXCLUDED\_TABLES
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Tables that will be excluded in the snapshot process. The syntax does not support wildcards. Tables without a schema defined will be applied the public schema.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_SCHEMA\_ONLY\_TABLES
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Tables for which only the schema will be snapshotted, skipping their data. The syntax supports wildcards. Tables without a schema defined will be applied the public schema. Requires snapshot mode `full` or `schema`. Tables explicitly listed in the snapshot tables take precedence over a schema-only wildcard match; excluded tables take precedence over the schema-only list.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_SCHEMA\_WORKERS
>
> **Default**: 4\
> **Required**: Optional\
> **Description**: Number of tables per schema that will be processed in parallel by the snapshotting process.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_TABLE\_WORKERS
>
> **Default**: 4\
> **Required**: Optional\
> **Description**: Number of concurrent workers that will be used per table by the snapshotting process.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_BATCH\_BYTES
>
> **Default**: 83886080 (80MiB)\
> **Required**: Optional\
> **Description**: Max batch size in bytes to be read and processed by each table worker at a time. The number of pages in the select queries will be based on this value.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_WORKERS
>
> **Default**: 1\
> **Required**: Optional\
> **Description**: Number of schemas that will be processed in parallel by the snapshotting process.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_MAX\_CONNECTIONS
>
> **Default**: 50\
> **Required**: Optional\
> **Description**: Maximum number of Postgres connections that will be opened by the snapshotting process. This value shouldn't be lower than the number of schema/table workers selected.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_CLEAN\_TARGET\_DB
>
> **Default**: False\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, option to issue commands to DROP all the objects that will be restored. Destructive: the DROPs are applied before the table data is copied, so a snapshot that fails afterwards leaves the target partially rebuilt with the previous contents already gone. See [resetting the target](/docs/opensource/pgstream/docs/snapshots#️-resetting-the-target-destroys-it-before-the-new-data-lands).

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_INCLUDE\_GLOBAL\_DB\_OBJECTS
>
> **Default**: False\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, option to snapshot all global database objects outside of the selected schema (such as extensions, triggers, etc).

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_CREATE\_TARGET\_DB
>
> **Default**: False\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, option to create the database being restored.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_NO\_OWNER
>
> **Default**: False\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, do not output commands to set ownership of objects to match the original database.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_NO\_PRIVILEGES
>
> **Default**: False\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, do not output privilege related commands (grant/revoke).

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_EXCLUDED\_SECURITY\_LABELS
>
> **Default**: \[]\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, list of providers whose security labels will be excluded.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_REFRESH\_MATERIALIZED\_VIEWS
>
> **Default**: False\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, whether to refresh materialized views (REFRESH MATERIALIZED VIEW ... WITH DATA) after the table data has been restored.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_INDEX\_CONSTRAINT\_SESSION\_SETTINGS
>
> **Default**: \[]\
> **Required**: Optional\
> **Description**: Space-separated PostgreSQL `name=value` session settings applied only while restoring indexes and constraints, for example `maintenance_work_mem=4GB max_parallel_maintenance_workers=4`. Each setting must be a whitespace-free `name=value` pair; invalid entries fail at startup. Unset or empty preserves existing behavior.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_INDEX\_RESTORE\_WORKERS
>
> **Default**: 1\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, number of standalone `CREATE INDEX`/`CREATE UNIQUE INDEX` statements restored concurrently against the target. Statements that depend on an index (constraints added `USING INDEX`, comments, partition attachments, `REPLICA IDENTITY`, `CLUSTER`) always wait for every index to be created first. Each worker holds one target connection and applies `PGSTREAM_POSTGRES_SNAPSHOT_INDEX_CONSTRAINT_SESSION_SETTINGS` to it, so the target needs this many free connection slots and up to workers × `maintenance_work_mem` of memory. Defaults to 1 (sequential); values above 32 are rejected at startup.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_INCLUDE\_OBJECT\_TYPES
>
> **Default**: \[]\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, list of object type categories to include in the schema snapshot. Everything else is excluded. Mutually exclusive with `PGSTREAM_POSTGRES_SNAPSHOT_EXCLUDE_OBJECT_TYPES`. See [object type filtering](#object-type-filtering).

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_EXCLUDE\_OBJECT\_TYPES
>
> **Default**: \[]\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, list of object type categories to exclude from the schema snapshot. Mutually exclusive with `PGSTREAM_POSTGRES_SNAPSHOT_INCLUDE_OBJECT_TYPES`. See [object type filtering](#object-type-filtering).

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_ROLE
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, role name to be used to create the dump.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_ROLES\_SNAPSHOT\_MODE
>
> **Default**: "no\_passwords"\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, controls how roles are snapshotted. Possible values: "enabled" (snapshot all roles including passwords), "disabled" (do not snapshot roles), "no\_passwords" (snapshot roles but exclude passwords).

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_SCHEMA\_DUMP\_FILE
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: When using `pg_dump`/`pg_restore` to snapshot schema for Postgres targets, file where the contents of the schema pg\_dump command and output will be written for debugging purposes.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_STORE\_URL
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Postgres URL for the database where the snapshot requests and their status will be tracked. A table `snapshot_requests` will be created under a `pgstream` schema.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_STORE\_REPEATABLE
>
> **Default**: False (run), True (snapshot)\
> **Required**: Optional\
> **Description**: Allow to repeat snapshots requests that have been already completed successfully. If using the run command, initial snapshots won't be repeatable by default. If the snapshot command is used instead, the snapshot will be repeatable by default.

> [accordion] PGSTREAM\_POSTGRES\_SNAPSHOT\_DISABLE\_PROGRESS\_TRACKING
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Whether to disable progress tracking for the snapshot.

> [accordion] PGSTREAM\_POSTGRES\_LISTENER\_EXP\_BACKOFF\_INITIAL\_INTERVAL
>
> **Default**: 500ms\
> **Required**: Optional\
> **Description**: Initial interval for the exponential backoff policy to be applied to the Postgres connection retries.

> [accordion] PGSTREAM\_POSTGRES\_LISTENER\_EXP\_BACKOFF\_MAX\_INTERVAL
>
> **Default**: 10s\
> **Required**: Optional\
> **Description**: Max interval for the exponential backoff policy to be applied to the Postgres connection retries.

> [accordion] PGSTREAM\_POSTGRES\_LISTENER\_EXP\_BACKOFF\_MAX\_RETRIES
>
> **Default**: 20\
> **Required**: Optional\
> **Description**: Max retries for the exponential backoff policy to be applied to the Postgres connection retries.

> [accordion] PGSTREAM\_POSTGRES\_LISTENER\_BACKOFF\_INTERVAL
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Constant interval for the backoff policy to be applied to the Postgres connection retries.

> [accordion] PGSTREAM\_POSTGRES\_LISTENER\_BACKOFF\_MAX\_RETRIES
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Max retries for the backoff policy to be applied to the Postgres connection retries.

> [accordion] PGSTREAM\_POSTGRES\_LISTENER\_DISABLE\_RETRIES
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Disable any retry policy.

One of exponential/constant/disable retries retry policies can be provided for the Postgres connection retry strategy. If none is provided, the exponential defaults apply.

#### Kafka Listener

> [accordion] PGSTREAM\_KAFKA\_READER\_SERVERS
>
> **Default**: N/A\
> **Required**: ✓ Required\
> **Description**: URLs for the Kafka servers to connect to.

> [accordion] PGSTREAM\_KAFKA\_TOPIC\_NAME
>
> **Default**: N/A\
> **Required**: ✓ Required\
> **Description**: Name of the Kafka topic to read from.

> [accordion] PGSTREAM\_KAFKA\_READER\_CONSUMER\_GROUP\_ID
>
> **Default**: N/A\
> **Required**: ✓ Required\
> **Description**: Name of the Kafka consumer group for the WAL Kafka reader.

> [accordion] PGSTREAM\_KAFKA\_READER\_CONSUMER\_GROUP\_START\_OFFSET
>
> **Default**: Earliest\
> **Required**: Optional\
> **Description**: Kafka offset from which the consumer will start if there's no offset available for the consumer group.

> [accordion] PGSTREAM\_KAFKA\_TLS\_ENABLED
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Enable TLS connection to the Kafka servers.

> [accordion] PGSTREAM\_KAFKA\_TLS\_CA\_CERT\_FILE
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Path to the CA PEM certificate to use for Kafka TLS authentication.

> [accordion] PGSTREAM\_KAFKA\_TLS\_CLIENT\_CERT\_FILE
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Path to the client PEM certificate to use for Kafka TLS client authentication.

> [accordion] PGSTREAM\_KAFKA\_TLS\_CLIENT\_KEY\_FILE
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Path to the client PEM private key to use for Kafka TLS client authentication.

> [accordion] PGSTREAM\_KAFKA\_SASL\_ENABLED
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Enable SASL authentication to the Kafka servers.

> [accordion] PGSTREAM\_KAFKA\_SASL\_MECHANISM
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: SASL mechanism to use, one of `plain`, `scram-sha-256` or `scram-sha-512`.

> [accordion] PGSTREAM\_KAFKA\_SASL\_USER
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: User for the Kafka SASL authentication.

> [accordion] PGSTREAM\_KAFKA\_SASL\_PASSWORD
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Password for the Kafka SASL authentication.

> [accordion] PGSTREAM\_KAFKA\_COMMIT\_EXP\_BACKOFF\_INITIAL\_INTERVAL
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Initial interval for the exponential backoff policy to be applied to the Kafka commit retries.

> [accordion] PGSTREAM\_KAFKA\_COMMIT\_EXP\_BACKOFF\_MAX\_INTERVAL
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Max interval for the exponential backoff policy to be applied to the Kafka commit retries.

> [accordion] PGSTREAM\_KAFKA\_COMMIT\_EXP\_BACKOFF\_MAX\_RETRIES
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Max retries for the exponential backoff policy to be applied to the Kafka commit retries.

> [accordion] PGSTREAM\_KAFKA\_COMMIT\_BACKOFF\_INTERVAL
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Constant interval for the backoff policy to be applied to the Kafka commit retries.

> [accordion] PGSTREAM\_KAFKA\_COMMIT\_BACKOFF\_MAX\_RETRIES
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Max retries for the backoff policy to be applied to the Kafka commit retries.

> [accordion] PGSTREAM\_KAFKA\_COMMIT\_DISABLE\_RETRIES
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Disable any retry policy.

One of exponential/constant backoff policies can be provided for the Kafka committing retry strategy. If none is provided, no retries apply.

### Targets

#### Kafka Batch Writer

> [accordion] PGSTREAM\_KAFKA\_WRITER\_SERVERS
>
> **Default**: N/A\
> **Required**: ✓ Required\
> **Description**: URLs for the Kafka servers to connect to.

> [accordion] PGSTREAM\_KAFKA\_TOPIC\_NAME
>
> **Default**: N/A\
> **Required**: ✓ Required\
> **Description**: Name of the Kafka topic to write to.

> [accordion] PGSTREAM\_KAFKA\_TOPIC\_PARTITIONS
>
> **Default**: 1\
> **Required**: Optional\
> **Description**: Number of partitions created for the Kafka topic if auto create is enabled.

> [accordion] PGSTREAM\_KAFKA\_TOPIC\_PARTITION\_KEY
>
> **Default**: schema\
> **Required**: Optional\
> **Description**: Message key strategy for DML events, one of `schema`, `table` or `primary_key`. See the ordering trade-offs below.

> [accordion] PGSTREAM\_KAFKA\_TOPIC\_REPLICATION\_FACTOR
>
> **Default**: 1\
> **Required**: Optional\
> **Description**: Replication factor used when creating the Kafka topic if auto create is enabled.

> [accordion] PGSTREAM\_KAFKA\_TOPIC\_AUTO\_CREATE
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Auto creation of configured Kafka topic if it doesn't exist.

> [accordion] PGSTREAM\_KAFKA\_TLS\_ENABLED
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Enable TLS connection to the Kafka servers.

> [accordion] PGSTREAM\_KAFKA\_TLS\_CA\_CERT\_FILE
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Path to the CA PEM certificate to use for Kafka TLS authentication.

> [accordion] PGSTREAM\_KAFKA\_TLS\_CLIENT\_CERT\_FILE
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Path to the client PEM certificate to use for Kafka TLS client authentication.

> [accordion] PGSTREAM\_KAFKA\_TLS\_CLIENT\_KEY\_FILE
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Path to the client PEM private key to use for Kafka TLS client authentication.

> [accordion] PGSTREAM\_KAFKA\_SASL\_ENABLED
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Enable SASL authentication to the Kafka servers.

> [accordion] PGSTREAM\_KAFKA\_SASL\_MECHANISM
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: SASL mechanism to use, one of `plain`, `scram-sha-256` or `scram-sha-512`.

> [accordion] PGSTREAM\_KAFKA\_SASL\_USER
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: User for the Kafka SASL authentication.

> [accordion] PGSTREAM\_KAFKA\_SASL\_PASSWORD
>
> **Default**: ""\
> **Required**: Optional\
> **Description**: Password for the Kafka SASL authentication.

> [accordion] PGSTREAM\_KAFKA\_WRITER\_BATCH\_TIMEOUT
>
> **Default**: 1s\
> **Required**: Optional\
> **Description**: Max time interval at which the batch sending to Kafka is triggered.

> [accordion] PGSTREAM\_KAFKA\_WRITER\_BATCH\_BYTES
>
> **Default**: 1572864\
> **Required**: Optional\
> **Description**: Max size in bytes for a given batch. When this size is reached, the batch is sent to Kafka.

> [accordion] PGSTREAM\_KAFKA\_WRITER\_BATCH\_SIZE
>
> **Default**: 100\
> **Required**: Optional\
> **Description**: Max number of messages to be sent per batch. When this size is reached, the batch is sent to Kafka.

> [accordion] PGSTREAM\_KAFKA\_WRITER\_BATCH\_IGNORE\_SEND\_ERRORS
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Whether to ignore errors encountered while sending batches to the target.

> [accordion] PGSTREAM\_KAFKA\_WRITER\_MAX\_QUEUE\_BYTES
>
> **Default**: 104857600 (100MiB)\
> **Required**: Optional\
> **Description**: Max memory used by the Kafka batch writer for inflight batches.

The partition key determines which partition an event is routed to, and therefore which events are consumed in order relative to each other:

* `schema` (default): all events for a schema go to the same partition, guaranteeing ordering per schema, including between DDL and DML events. Parallelism is capped at the number of distinct schemas, so extra partitions don't help a single-schema database.
* `table`: events are keyed by schema qualified table name, guaranteeing ordering per table. DDL events remain keyed by schema, so schema changes can be consumed out of order relative to the DML events of the tables they affect.
* `primary_key`: events are keyed by schema qualified table name plus the row primary key values, guaranteeing ordering per row and allowing full use of the topic partitions. Requires the injector (`PGSTREAM_INJECTOR_STORE_POSTGRES_URL`) to identify primary key columns; events without an identifiable primary key fall back to `table` keying. As with `table`, DDL events remain keyed by schema and can be consumed out of order relative to DML.

#### Search Batch Indexer

> [accordion] PGSTREAM\_OPENSEARCH\_STORE\_URL
>
> **Default**: N/A\
> **Required**: ✓ Required\
> **Description**: URL for the opensearch store to connect to (at least one of the URLs must be provided).

> [accordion] PGSTREAM\_ELASTICSEARCH\_STORE\_URL
>
> **Default**: N/A\
> **Required**: ✓ Required\
> **Description**: URL for the elasticsearch store to connect to (at least one of the URLs must be provided).

> [accordion] PGSTREAM\_SEARCH\_INDEXER\_HASH\_DOC\_IDS
>
> **Default**: False\
> **Required**: Optional\
> **Description**: If true, hash document IDs using SHA256 to avoid exceeding the Elasticsearch 512 byte limit.

> [accordion] PGSTREAM\_SEARCH\_INDEXER\_BATCH\_TIMEOUT
>
> **Default**: 1s\
> **Required**: Optional\
> **Description**: Max time interval at which the batch sending to the search store is triggered.

> [accordion] PGSTREAM\_SEARCH\_INDEXER\_BATCH\_SIZE
>
> **Default**: 100\
> **Required**: Optional\
> **Description**: Max number of messages to be sent per batch. When this size is reached, the batch is sent to the search store.

> [accordion] PGSTREAM\_SEARCH\_INDEXER\_BATCH\_IGNORE\_SEND\_ERRORS
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Whether to ignore errors encountered while sending batches to the target.

> [accordion] PGSTREAM\_SEARCH\_INDEXER\_MAX\_QUEUE\_BYTES
>
> **Default**: 104857600 (100MiB)\
> **Required**: Optional\
> **Description**: Max memory used by the search batch indexer for inflight batches.

> [accordion] PGSTREAM\_SEARCH\_STORE\_EXP\_BACKOFF\_INITIAL\_INTERVAL
>
> **Default**: 1s\
> **Required**: Optional\
> **Description**: Initial interval for the exponential backoff policy to be applied to the search store operation retries.

> [accordion] PGSTREAM\_SEARCH\_STORE\_EXP\_BACKOFF\_MAX\_INTERVAL
>
> **Default**: 1min\
> **Required**: Optional\
> **Description**: Max interval for the exponential backoff policy to be applied to the search store operation retries.

> [accordion] PGSTREAM\_SEARCH\_STORE\_EXP\_BACKOFF\_MAX\_RETRIES
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Max retries for the exponential backoff policy to be applied to the search store operation retries.

> [accordion] PGSTREAM\_SEARCH\_STORE\_BACKOFF\_INTERVAL
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Constant interval for the backoff policy to be applied to the search store operation retries.

> [accordion] PGSTREAM\_SEARCH\_STORE\_BACKOFF\_MAX\_RETRIES
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Max retries for the backoff policy to be applied to the search store operation retries.

> [accordion] PGSTREAM\_SEARCH\_STORE\_DISABLE\_RETRIES
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Disable any retry policy.

One of exponential/constant backoff policies can be provided for the search indexer cleanup retry strategy. If none is provided, no retries apply.

One of exponential/constant/disable retries backoff policies can be provided for the search store retry strategy. If none is provided, a default exponential backoff policy applies.

#### Stdout Writer

> [accordion] PGSTREAM\_STDOUT\_WRITER\_ENABLED
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Set to true to enable the stdout writer target. WAL events are written as NDJSON (one JSON object per line) to stdout.

#### Webhook Notifier

> [accordion] PGSTREAM\_WEBHOOK\_SUBSCRIPTION\_STORE\_URL
>
> **Default**: N/A\
> **Required**: ✓ Required\
> **Description**: URL for the webhook subscription store to connect to.

> [accordion] PGSTREAM\_WEBHOOK\_SUBSCRIPTION\_STORE\_CACHE\_ENABLED
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Caching applied to the subscription store retrieval queries.

> [accordion] PGSTREAM\_WEBHOOK\_SUBSCRIPTION\_STORE\_CACHE\_REFRESH\_INTERVAL
>
> **Default**: 60s\
> **Required**: Optional\
> **Description**: Interval at which the subscription store cache will be refreshed. Indicates max cache staleness.

> [accordion] PGSTREAM\_WEBHOOK\_NOTIFIER\_MAX\_QUEUE\_BYTES
>
> **Default**: 104857600 (100MiB)\
> **Required**: Optional\
> **Description**: Max memory used by the webhook notifier for inflight notifications.

> [accordion] PGSTREAM\_WEBHOOK\_NOTIFIER\_WORKER\_COUNT
>
> **Default**: 10\
> **Required**: Optional\
> **Description**: Max number of concurrent workers that will send webhook notifications for a given WAL event.

> [accordion] PGSTREAM\_WEBHOOK\_NOTIFIER\_CLIENT\_TIMEOUT
>
> **Default**: 10s\
> **Required**: Optional\
> **Description**: Max time the notifier will wait for a response from a webhook URL before timing out.

> [accordion] PGSTREAM\_WEBHOOK\_NOTIFIER\_EXP\_BACKOFF\_INITIAL\_INTERVAL
>
> **Default**: 1s\
> **Required**: Optional\
> **Description**: Initial interval for the exponential backoff policy to be applied to failed webhook deliveries.

> [accordion] PGSTREAM\_WEBHOOK\_NOTIFIER\_EXP\_BACKOFF\_MAX\_INTERVAL
>
> **Default**: 30s\
> **Required**: Optional\
> **Description**: Max interval for the exponential backoff policy to be applied to failed webhook deliveries.

> [accordion] PGSTREAM\_WEBHOOK\_NOTIFIER\_EXP\_BACKOFF\_MAX\_RETRIES
>
> **Default**: 3\
> **Required**: Optional\
> **Description**: Max retries for the exponential backoff policy to be applied to failed webhook deliveries.

> [accordion] PGSTREAM\_WEBHOOK\_NOTIFIER\_BACKOFF\_INTERVAL
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Constant interval for the backoff policy to be applied to failed webhook deliveries.

> [accordion] PGSTREAM\_WEBHOOK\_NOTIFIER\_BACKOFF\_MAX\_RETRIES
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Max retries for the backoff policy to be applied to failed webhook deliveries.

> [accordion] PGSTREAM\_WEBHOOK\_NOTIFIER\_DISABLE\_RETRIES
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Disable any retry policy for failed webhook deliveries.

> [accordion] PGSTREAM\_WEBHOOK\_NOTIFIER\_STRICT\_MODE
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Whether to stop the pipeline on a permanently failing webhook delivery instead of dropping it and continuing. It defaults to false.

> [accordion] PGSTREAM\_WEBHOOK\_SUBSCRIPTION\_SERVER\_ADDRESS
>
> **Default**: ":9900"\
> **Required**: Optional\
> **Description**: Address for the subscription server to listen on.

> [accordion] PGSTREAM\_WEBHOOK\_SUBSCRIPTION\_SERVER\_READ\_TIMEOUT
>
> **Default**: 5s\
> **Required**: Optional\
> **Description**: Max duration for reading an entire server request, including the body before timing out.

> [accordion] PGSTREAM\_WEBHOOK\_SUBSCRIPTION\_SERVER\_WRITE\_TIMEOUT
>
> **Default**: 10s\
> **Required**: Optional\
> **Description**: Max duration before timing out writes of the response. It is reset whenever a new request's header is read.

One of exponential/constant/disable retries backoff policies can be provided for the webhook notifier retry strategy. If none is provided, a default exponential backoff policy applies (1s initial interval, 30s max interval, 3 max retries). A 2xx response is treated as success; a 429 or 5xx response is retried; any other response (e.g. 4xx) is treated as a permanent failure and is not retried. A delivery that keeps failing with a retryable error after exhausting retries is not checkpointed, so it will be retried again after a restart. A delivery that permanently fails is logged and dropped instead by default, so a single misconfigured subscriber does not block delivery to other subscribers; set `strict_mode` to stop the pipeline on permanent failures instead, matching the at-least-once guarantee at the cost of availability.

#### Postgres Batch Writer

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_TARGET\_URL
>
> **Default**: N/A\
> **Required**: ✓ Required\
> **Description**: URL for the PostgreSQL store to connect to

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_MAX\_CONNECTIONS
>
> **Default**: 50\
> **Required**: Optional\
> **Description**: Maximum number of connections in the writer pool to the target PostgreSQL database. Overrides `pool_max_conns` in the target URL when set. The schema observer keeps its own pool, capped at 16 and never larger than this value, so the process opens at most this many plus 16.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_BATCH\_TIMEOUT
>
> **Default**: 30s\
> **Required**: Optional\
> **Description**: Max time interval at which the batch sending to PostgreSQL is triggered.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_BATCH\_SIZE
>
> **Default**: 20000\
> **Required**: Optional\
> **Description**: Max number of messages to be sent per batch. When this size is reached, the batch is sent to PostgreSQL.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_MAX\_QUEUE\_BYTES
>
> **Default**: 104857600 (100MiB)\
> **Required**: Optional\
> **Description**: Max memory used by the postgres batch writer for inflight batches.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_BATCH\_BYTES
>
> **Default**: 1572864 (1.5MiB), 83886080 (80MiB) with bulk enabled\
> **Required**: Optional\
> **Description**: Max size in bytes for a given batch. When this size is reached, the batch is sent to PostgreSQL.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_BATCH\_IGNORE\_SEND\_ERRORS
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Whether to ignore errors encountered while sending events to the target.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_DISABLE\_TRIGGERS
>
> **Default**: False(run), True(snapshot)\
> **Required**: Optional\
> **Description**: Option to disable triggers on the target PostgreSQL database while performing the snaphot/replication streaming. It defaults to false when using the run command, and to true when using the snapshot command.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_ON\_CONFLICT\_ACTION
>
> **Default**: error\
> **Required**: Optional\
> **Description**: Action to apply to inserts on conflict. Options are `nothing`, `update` or `error`.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_STRICT\_MODE
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Whether to stop on non-internal query failures instead of dropping them and continuing. It defaults to false.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_BULK\_INGEST\_ENABLED
>
> **Default**: False(run), True(snapshot)\
> **Required**: Optional\
> **Description**: Whether to use COPY FROM on insert only workloads. It defaults to false when using the run command, and to true when using the snapshot command.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_BULK\_INGEST\_COPY\_WORKERS
>
> **Default**: 8\
> **Required**: Optional\
> **Description**: Number of concurrent COPY streams per table when bulk ingesting. Only applies when bulk ingest is enabled.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_EXP\_BACKOFF\_INITIAL\_INTERVAL
>
> **Default**: 500ms\
> **Required**: Optional\
> **Description**: Initial interval for the exponential backoff policy to be applied to the Postgres connection retries.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_EXP\_BACKOFF\_MAX\_INTERVAL
>
> **Default**: 10s\
> **Required**: Optional\
> **Description**: Max interval for the exponential backoff policy to be applied to the Postgres connection retries.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_EXP\_BACKOFF\_MAX\_RETRIES
>
> **Default**: 20\
> **Required**: Optional\
> **Description**: Max retries for the exponential backoff policy to be applied to the Postgres connection retries.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_BACKOFF\_INTERVAL
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Constant interval for the backoff policy to be applied to the Postgres connection retries.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_BACKOFF\_MAX\_RETRIES
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Max retries for the backoff policy to be applied to the Postgres connection retries.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_DISABLE\_RETRIES
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Disable any retry policy.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_IGNORE\_DDL
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Disable processing of DDL events on the target Postgres database. Consider enabling if source and target roles have different trust levels (see [privileges](/docs/opensource/pgstream/docs/privileges)).

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_INCLUDE\_DDL\_OBJECT\_TYPES
>
> **Default**: \[]\
> **Required**: Optional\
> **Description**: List of object type categories for which DDL is replicated. DDL for everything else is skipped. Mutually exclusive with `PGSTREAM_POSTGRES_WRITER_EXCLUDE_DDL_OBJECT_TYPES`, and ignored when `PGSTREAM_POSTGRES_WRITER_IGNORE_DDL` is true. See [object type filtering](#object-type-filtering).

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_EXCLUDE\_DDL\_OBJECT\_TYPES
>
> **Default**: \[]\
> **Required**: Optional\
> **Description**: List of object type categories for which DDL replication is skipped. Mutually exclusive with `PGSTREAM_POSTGRES_WRITER_INCLUDE_DDL_OBJECT_TYPES`, and ignored when `PGSTREAM_POSTGRES_WRITER_IGNORE_DDL` is true. See [object type filtering](#object-type-filtering).

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_BATCH\_AUTO\_TUNE\_ENABLE
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Whether to enable auto tuning of batch bytes.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_BATCH\_AUTO\_TUNE\_MIN\_BYTES
>
> **Default**: 1048576 (1MB)\
> **Required**: Optional\
> **Description**: Minimum batch size in bytes used by the auto tune process.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_BATCH\_AUTO\_TUNE\_MAX\_BYTES
>
> **Default**: 52428800 (50MB)\
> **Required**: Optional\
> **Description**: Maximum batch size in bytes used by the auto tune process.

> [accordion] PGSTREAM\_POSTGRES\_WRITER\_BATCH\_AUTO\_TUNE\_CONVERGENCE\_THRESHOLD
>
> **Default**: 0.01 (1%)\
> **Required**: Optional\
> **Description**: Minimum search space width (as a fraction of the max batch bytes) at which the auto tune process stops searching for a better batch size.

One of exponential/constant/disable retries retry policies can be provided for the Postgres connection retry strategy. If none is provided, the exponential defaults apply.

#### Modifiers

#### Injector

> [accordion] PGSTREAM\_INJECTOR\_STORE\_POSTGRES\_URL
>
> **Default**: N/A\
> **Required**: ✓ Required\
> **Description**: PostgreSQL URL for DDL replication.

#### Transformer

> [accordion] PGSTREAM\_TRANSFORMER\_RULES\_FILE
>
> **Default**: N/A\
> **Required**: Optional\
> **Description**: Filepath pointing to the yaml file containing the transformer rules.

#### Filter

> [accordion] PGSTREAM\_FILTER\_INCLUDE\_TABLES
>
> **Default**: N/A\
> **Required**: Optional\
> **Description**: List of schema qualified tables for which the WAL events should be processed. If no schema is provided, `public` schema will be assumed. Wildcards are supported.

> [accordion] PGSTREAM\_FILTER\_EXCLUDE\_TABLES
>
> **Default**: N/A\
> **Required**: Optional\
> **Description**: List of schema qualified tables for which the WAL events should be skipped. If no schema is provided, `public` schema will be assumed. Wildcards are supported.

> [accordion] PGSTREAM\_FILTER\_SCHEMA\_ONLY\_TABLES
>
> **Default**: N/A\
> **Required**: Optional\
> **Description**: List of schema qualified tables for which DDL (schema change) events are processed but data (DML) events are skipped. If no schema is provided, `public` schema will be assumed. Wildcards are supported. Can be combined with either the include or the exclude list.

#### Sanitizer

> [accordion] PGSTREAM\_PROCESSOR\_SANITIZE\_STRIP\_NULL\_CHAR\_BYTES
>
> **Default**: false\
> **Required**: Optional\
> **Description**: Strip null bytes (0x00) from string column values. Useful when the source database contains null bytes that are not allowed by the target.

### Instrumentation

#### Metrics

> [accordion] PGSTREAM\_METRICS\_ENDPOINT
>
> **Default**: N/A\
> **Required**: Optional\
> **Description**: Endpoint where the pgstream metrics will be pushed to via OTLP. Not required to use Prometheus.

> [accordion] PGSTREAM\_METRICS\_COLLECTION\_INTERVAL
>
> **Default**: 60s\
> **Required**: Optional\
> **Description**: Interval at which the pgstream metrics will be collected and exported via OTLP.

> [accordion] PGSTREAM\_METRICS\_PROMETHEUS\_ENABLED
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Exposes a Prometheus scrape endpoint on the health server. Requires the health endpoint enabled.

> [accordion] PGSTREAM\_METRICS\_PROMETHEUS\_ENDPOINT
>
> **Default**: /metrics\
> **Required**: Optional\
> **Description**: Path the Prometheus endpoint is served on.

#### Traces

> [accordion] PGSTREAM\_TRACES\_ENDPOINT
>
> **Default**: N/A\
> **Required**: Optional\
> **Description**: Endpoint where the pgstream traces will be exported to.

> [accordion] PGSTREAM\_TRACES\_SAMPLE\_RATIO
>
> **Default**: 0\
> **Required**: Optional\
> **Description**: Ratio for the trace sampling. Value must be between 0.0 and 1.0, where 0.0 is no traces sampled, and 1.0 is all traces sampled.

#### Health endpoint

Exposes `/health` (liveness, always 200), `/ready` (readiness, pings the source postgres database when configured), and `/status` (current pipeline phase: `snapshot` or `replication`). Only the `run` and `snapshot` commands start the server. Responses are JSON.

When `instrumentation.metrics.prometheus.enabled` (`PGSTREAM_METRICS_PROMETHEUS_ENABLED`) is true, the server also exposes a Prometheus scrape endpoint (default path `/metrics`, configurable via `instrumentation.metrics.prometheus.endpoint` / `PGSTREAM_METRICS_PROMETHEUS_ENDPOINT`). The endpoint returns `404` when Prometheus is disabled. This lets you scrape pgstream's metrics directly, without deploying an OTel collector.

> [accordion] PGSTREAM\_HEALTH\_CHECK\_ENABLED
>
> **Default**: False\
> **Required**: Optional\
> **Description**: Enable the health endpoint server.

> [accordion] PGSTREAM\_HEALTH\_CHECK\_ADDRESS
>
> **Default**: localhost:9910\
> **Required**: Optional\
> **Description**: Address the health server listens on. Use `:9910` or `0.0.0.0:9910` to expose externally (e.g. in k8s pods).

## Object type filtering

Both the schema snapshot and DDL replication can be restricted to a subset of database object types. Each side is configured independently, with an allowlist (`include_*`) or a denylist (`exclude_*`); setting both on the same side is rejected at startup, and so is an unknown category name.

The available categories are:

> [accordion] tables
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] sequences
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] types
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] indexes
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] constraints
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] functions
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] views
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] materialized\_views
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] triggers
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] event\_triggers
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] policies
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] rules
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] extensions
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] collations
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] text\_search
>
> **Schema snapshot**: ✅\
> **DDL replication**: ✅

> [accordion] comments
>
> **Schema snapshot**: ✅\
> **DDL replication**: ❌

Schemas themselves are never filtered out, since the remaining objects need their namespaces to exist.

For DDL replication, a single statement can create objects of several types at once (`CREATE TABLE` with a primary key produces both a table and an index). Such an event is only skipped when *all* of its objects belong to excluded categories.

> ⚠️ Filtering does not resolve dependencies between object types. Excluding a category that surviving objects depend on (for example excluding `types` while keeping tables with columns of those types) will make the snapshot or the DDL replay fail. Use this with a good understanding of your schema.


# GCP CloudSQL
Source: https://xata.io/docs/opensource/pgstream/docs/gcp_cloudsql

This guide explains how to use **pgstream** with **Google CloudSQL for Postgres**, covering **snapshots** and **replication**.

👉 Throughout this guide:

* **`pgstreamsource`** refers to the user provided in the **pgstream source URL**.
* **`pgstreamtarget`** refers to the user provided in the **pgstream target URL**.

## Snapshots

### From CloudSQL Postgres (Snapshots)

#### Quick Checklist

* [ ] Create a source user (**`pgstreamsource`**) with access to required schemas/tables.

* [ ] Decide how to handle roles:

  * Disabled → no special config.
  * Without passwords → no special config.
  * With passwords → enable `cloudsql.pg_authid_select_role` flag.

* [ ] Update YAML config with correct snapshot settings.

#### Steps

1. **User privileges**
   Ensure the **`pgstreamsource`** user (from the pgstream source URL) can access the database schema and tables you need.

2. **Roles handling**

   * **No roles** → no changes required.
   * **Roles without passwords** → no changes required.
   * **Roles with passwords** → grant `pgstreamsource` access to `pg_authid` by enabling the `cloudsql.pg_authid_select_role` database flag (via CloudSQL UI or API).

     * Reference: [GCP documentation](https://cloud.google.com/sql/docs/postgres/users)

   Without this, you’ll see:

   ```
   pg_dumpall: error: query failed: ERROR:  permission denied for table pg_authid
   ```

3. **Config when not snapshotting roles**
   If roles are disabled or not manually managed, add the following to avoid failures:

   ```yaml theme={null}
   snapshot:
     schema:
       pgdump_pgrestore:
         roles_snapshot_mode: "disabled"
         no_owner: true
         no_privileges: true
   ```

   Example full configuration:

   ```yaml theme={null}
   source:
     postgres:
       url: "postgresql://pgstreamsource:password@<cloudsql_address>:5432/db"
       mode: snapshot
       snapshot:
         mode: full # schema + data
         tables: ["public.*"] # all tables in the public schema
         schema:
           mode: pgdump_pgrestore
           pgdump_pgrestore:
             roles_snapshot_mode: "disabled"
             no_owner: true
             no_privileges: true
   ```

ℹ️ CloudSQL-managed roles (`cloudsqlsuperuser`, `postgres`) will not be snapshotted.

### To CloudSQL Postgres (Snapshots)

#### Quick Checklist

* [ ] Create a target user (**`pgstreamtarget`**) for the pgstream target URL.
* [ ] Grant privileges for schema/database ownership.
* [ ] Grant optional privileges depending on features (create DB, create roles, set replication role).
* [ ] If using CloudSQL, disable `disable_triggers`.

#### Steps

The **`pgstreamtarget`** user (from the pgstream target URL) must have the following privileges:

* **Schema ownership**

  ```sql theme={null}
  ALTER DATABASE db OWNER TO pgstreamtarget;
  ALTER SCHEMA <schema> OWNER TO pgstreamtarget;
  ```

* **Database creation** (if `create_target_db` is enabled)

  ```sql theme={null}
  ALTER ROLE pgstreamtarget CREATEDB;
  ```

* **Role creation** (if `role_snapshot_mode` is `enabled`/`no_passwords`)

  ```sql theme={null}
  ALTER ROLE pgstreamtarget CREATEROLE;
  ```

  ⚠️ `pgstreamtarget` must already hold any privileges it assigns (e.g., `REPLICATION`).
  If many privileges are needed, use the `postgres` user, which is a [pseudo-superuser](https://cloud.google.com/sql/docs/postgres/users).

* **Disable triggers** (if `disable_triggers` is enabled)

  ```sql theme={null}
  GRANT SET ON PARAMETER session_replication_role TO pgstreamtarget;
  ```

  Without this, pgstream fails with:

  ```
  permission denied to set parameter "session_replication_role"
  ```

  ⚠️ On CloudSQL, the `postgres` role cannot set `session_replication_role`. Always disable `disable_triggers` in this case.

## Replication

### From CloudSQL Postgres (Replication)

#### Quick Checklist

* [ ] Enable `cloudsql.logical_decoding` flag.
* [ ] Use `postgres` user for initialization.
* [ ] Grant `cloudsqlreplica` + replication privileges.
* [ ] Optionally transfer ownership to **`pgstreamsource`** for streaming.

#### Steps

1. **Enable logical replication**
   Enable the [`cloudsql.logical_decoding`](https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication) flag.

2. **Replication phases**

   * **Initialization** → requires `postgres` (superuser-like privileges).
   * **Streaming** → can switch to `pgstreamsource` (from the pgstream source URL).

#### Initialization

Initialization does the following:

* Creates `pgstream` schema
* Creates replication slot (if missing)
* Creates event triggers/functions for schema changes

Use the `postgres` role. Also grant replication privileges explicitly:

```sql theme={null}
GRANT cloudsqlreplica TO postgres;
ALTER ROLE postgres REPLICATION;
```

#### Streaming

After initialization, you can:

* Keep using `postgres`, or
* Transfer ownership to a dedicated **`pgstreamsource`** role.

Example setup:

```sql theme={null}
-- allow ownership transfer
GRANT pgstreamsource TO current_user;

-- transfer ownership
ALTER DATABASE db OWNER TO pgstreamsource;
ALTER SCHEMA pgstream OWNER TO pgstreamsource;
ALTER SCHEMA <schema> OWNER TO pgstreamsource;
ALTER TABLE pgstream.schema_log OWNER TO pgstreamsource;
ALTER TABLE pgstream.table_ids OWNER TO pgstreamsource;
ALTER SEQUENCE pgstream.xid_serial OWNER TO pgstreamsource;

-- replication privileges
GRANT cloudsqlreplica TO pgstreamsource;
ALTER ROLE pgstreamsource REPLICATION;
```

### To CloudSQL Postgres (Replication)

#### Quick Checklist

* [ ] Ensure **`pgstreamtarget`** has schema/database ownership.
* [ ] If combined with snapshot → follow [Snapshot target requirements](#to-cloudsql-postgres-snapshots).

#### Steps

Privileges required:

```sql theme={null}
ALTER DATABASE db OWNER TO pgstreamtarget;
ALTER SCHEMA <schema> OWNER TO pgstreamtarget;
```

## Troubleshooting

> [accordion] permission denied for table pg\_authid
>
> **Cause**: Roles with passwords require pg\_authid access.\
> **Fix**: Enable `cloudsql.pg_authid_select_role` flag.

> [accordion] permission denied to set parameter "session\_replication\_role"
>
> **Cause**: CloudSQL roles can’t set replication role.\
> **Fix**: Disable `disable_triggers`.

> [accordion] Role creation fails
>
> **Cause**: `pgstreamtarget` lacks privileges it tries to assign.\
> **Fix**: Use `postgres` or grant required privileges first.


# Glossary
Source: https://xata.io/docs/opensource/pgstream/docs/glossary

* [CDC](https://en.wikipedia.org/wiki/Change_data_capture): Change Data Capture
* [WAL](https://www.postgresql.org/docs/current/wal-intro.html): Write Ahead Logging
* [LSN](https://pgpedia.info/l/LSN-log-sequence-number.html): Log Sequence Number
* [DDL](https://en.wikipedia.org/wiki/Data_definition_language): Data Definition Language


# Neon
Source: https://xata.io/docs/opensource/pgstream/docs/neon

This guide explains how to use **pgstream** with **Neon databases**, covering **snapshots** and **replication**.

👉 Throughout this guide:

* **`pgstreamsource`** refers to the user provided in the **pgstream source URL**.
* **`pgstreamtarget`** refers to the user provided in the **pgstream target URL**.

## Snapshots

### From Neon Database (Snapshots)

#### Quick Checklist

* [ ] Create a source user (**`pgstreamsource`**) with access to required schemas/tables.
* [ ] For roles without passwords → no special config needed.
* [ ] For roles with passwords → **must use the `neondb_owner` superuser**.
* [ ] Update YAML config with correct snapshot settings.

#### Steps

1. **User privileges**
   Ensure the **`pgstreamsource`** user (from the pgstream source URL) owns the database schema and tables you want to snapshot.

2. **Roles handling**

   * **No roles** → no changes required.
   * **Roles without passwords** → no changes required.
   * **Roles with passwords** → **must use the `neondb_owner` user**.

   ⚠️ Only the `neondb_owner` user has access to the `pg_authid` table required for snapshotting roles with passwords.

   Without this, you'll see:

   ```
   pg_dumpall: error: query failed: ERROR:  permission denied for table pg_authid
   ```

3. **Config when not snapshotting roles**
   If roles are disabled or not manually managed, add the following to avoid failures:

   ```yaml theme={null}
   snapshot:
     schema:
       pgdump_pgrestore:
         roles_snapshot_mode: "disabled"
         no_owner: true
         no_privileges: true
   ```

   Example full configuration:

   ```yaml theme={null}
   source:
     postgres:
       url: "postgresql://pgstreamsource:password@<neon-host>:5432/db?sslmode=require"
       mode: snapshot
       snapshot:
         mode: full # schema + data
         tables: ["public.*"] # all tables in the public schema
         schema:
           mode: pgdump_pgrestore
           pgdump_pgrestore:
             roles_snapshot_mode: "disabled"
             no_owner: true
             no_privileges: true
   ```

ℹ️ Neon-managed roles (`neondb_owner`, `neon_service`, `neon_superuser` and `cloud_admin`) will not be snapshotted.

### To Neon Database (Snapshots)

#### Quick Checklist

* [ ] Create a target user (**`pgstreamtarget`**) for the pgstream target URL.
* [ ] Grant privileges for schema/database ownership.
* [ ] Grant optional privileges depending on features (create DB, create roles).
* [ ] **Use `disable_triggers: false`** (required for all non-`neon` users).

#### Steps

The **`pgstreamtarget`** user (from the pgstream target URL) must have the following privileges:

* **Schema ownership**

  ```sql theme={null}
  ALTER DATABASE db OWNER TO pgstreamtarget;
  ALTER SCHEMA public OWNER TO pgstreamtarget;
  ```

* **Database creation** (if `create_target_db` is enabled)

  ```sql theme={null}
  ALTER ROLE pgstreamtarget CREATEDB;
  ```

* **Role creation** (if `roles_snapshot_mode` is `enabled`/`no_passwords`)

  ```sql theme={null}
  ALTER ROLE pgstreamtarget CREATEROLE;
  ```

  ⚠️ `pgstreamtarget` must already hold any privileges it assigns (e.g., `REPLICATION`).

* **Disable triggers** (required for Neon)

  ⚠️ **Not even the pseudo-superuser `neondb_owner` user can set `session_replication_role`** on Neon databases.

  ```yaml theme={null}
  target:
    postgres:
      disable_triggers: false # Required for neon target databases
  ```

  ❌ If using `disable_triggers: true`, pgstream will fail with:

  ```
  permission denied to set parameter "session_replication_role"
  ```

## Replication

### From Neon Database (Replication)

#### Quick Checklist

* [ ] Ensure logical replication is enabled on your Neon database.
* [ ] **Use the `neondb_owner` user for initialization** (only user with event trigger privileges).
* [ ] Optionally use a different **`pgstreamsource`** user for streaming.

#### Steps

1. **Enable logical replication**

   Logical replication requires `wal_level` to be set to `logical`.

   **Check current setting:**

   ```sql theme={null}
   SHOW wal_level;
   ```

   **If not set to `logical`, update it:**

   Neon enables logical replication by default on most databases. If you see `wal_level = replica`, you can enable logical replication through the Neon Console:

   1. Go to your project in the [Neon Console](https://console.neon.tech/)
   2. Navigate to **Settings** → **Replication**
   3. Enable **Logical replication**
   4. No restart is required - Neon handles this automatically

   Alternatively, you can enable it via the [Neon API](https://api-docs.neon.tech/reference/updateproject) or contact [Neon support](https://neon.tech/docs/introduction/support).

2. **Replication phases**

   * **Initialization** → requires elevated privileges to create schemas, event triggers, and replication slots.
   * **Streaming** → can use a dedicated `pgstreamsource` user.

#### Initialization

Initialization does the following:

* Creates `pgstream` schema
* Creates replication slot (if missing)
* Creates event triggers/functions for schema changes

⚠️ **Only the `neondb_owner` user has sufficient privileges to create event triggers.** Use the `neondb_owner` user for initialization:

```yaml theme={null}
source:
  postgres:
    url: "postgresql://neondb_owner:password@<neon-host>:5432/db?sslmode=require"
    mode: replication
```

#### Streaming

After initialization, you can:

* Keep using `neondb_owner`, or
* Transfer ownership to a dedicated **`pgstreamsource`** role.

```sql theme={null}
-- Allow ownership transfer
GRANT pgstreamsource TO current_user;

-- Transfer ownership
ALTER DATABASE db OWNER TO pgstreamsource;
ALTER SCHEMA pgstream OWNER TO pgstreamsource;
ALTER SCHEMA <schema> OWNER TO pgstreamsource;
ALTER TABLE pgstream.schema_log OWNER TO pgstreamsource;
ALTER TABLE pgstream.table_ids OWNER TO pgstreamsource;
ALTER SEQUENCE pgstream.xid_serial OWNER TO pgstreamsource;

-- Replication privileges
ALTER ROLE pgstreamsource REPLICATION;
```

### To Neon Database (Replication)

#### Quick Checklist

* [ ] Ensure **`pgstreamtarget`** has schema/database ownership.
* [ ] If combined with snapshot → follow [Snapshot target requirements](#to-neon-database-snapshots).

#### Steps

Privileges required:

```sql theme={null}
ALTER DATABASE db OWNER TO pgstreamtarget;
ALTER SCHEMA <schema> OWNER TO pgstreamtarget;
```

## Troubleshooting

> [accordion] permission denied for table pg\_authid
>
> **Cause**: Roles with passwords require pg\_authid access.\
> **Fix**: Use `neondb_owner` user as source, or disable role passwords (`roles_snapshot_mode: no_passwords`).

> [accordion] permission denied to set parameter "session\_replication\_role"
>
> **Cause**: No available Neon users can set this parameter.\
> **Fix**: Set `disable_triggers: false` in target config.

> [accordion] permission denied for schema public when transferring ownership
>
> **Cause**: Target role lacks CREATE privilege on schema.\
> **Fix**: Grant CREATE on schema to the role: `GRANT CREATE ON SCHEMA public TO role_name;`

> [accordion] Role creation fails
>
> **Cause**: `pgstreamtarget` lacks privileges it tries to assign.\
> **Fix**: Grant required privileges to `pgstreamtarget` first, or use a user with more privileges.


# Privileges
Source: https://xata.io/docs/opensource/pgstream/docs/privileges

This document summarizes the privileges required for different pgstream operations in PostgreSQL.
It is **service-agnostic**, focusing only on what pgstream itself requires.

For cloud-provider-specific instructions and constraints, please see:

* [Xata Onboarding Guide](/docs/opensource/pgstream/docs/xata)
* [AWS Onboarding Guide](/docs/opensource/pgstream/docs/aws)
* [Google Cloud (CloudSQL) Onboarding Guide](/docs/opensource/pgstream/docs/gcp_cloudsql)
* [Neon Onboarding Guide](/docs/opensource/pgstream/docs/neon)

***

## Privileges Matrix

| Mode | Role | Required Privileges | Notes |
| - | - | - | - |
| **Snapshot (Source)** | `pgstreamsource` | - **Read-only access** to schemas/tables being snapshotted - *(Optional)* Read access to `pg_authid` (if snapshotting roles with passwords) | Example grants: `GRANT pg_read_all_settings TO pgstreamsource; GRANT SELECT ON pg_authid TO pgstreamsource;` |
| **Snapshot (Target)** | `pgstreamtarget` | - Ownership of database & schemas - *(Optional)* `CREATEDB` (if creating target databases) - *(Optional)* `CREATEROLE` (if restoring roles) - *(Optional)* Ability to `SET session_replication_role` (if disabling triggers during load) | Role must already hold any privileges it assigns when creating other roles. |
| **Replication (Source, Initialization)** | Initialization user (e.g., `postgres`) | - Ability to create schemas, event triggers, and functions - Replication privileges (`REPLICATION`) | Requires elevated privileges because event triggers must be created. |
| **Replication (Source, Streaming)** | `pgstreamsource` | - Ownership of replicated database, schemas, and pgstream objects - Replication privileges (`REPLICATION`) | Can be a more restricted role than the initialization user. |
| **Replication (Target)** | `pgstreamtarget` | - Ownership of database & schemas - Ability to apply DML and DDL changes | If combined with snapshot, also meet snapshot target requirements. |

ℹ️ **Terminology**:

* `pgstreamsource` → user defined in the **pgstream source URL**
* `pgstreamtarget` → user defined in the **pgstream target URL**

***

## Security note: DDL replication and the target role

When DDL replication is enabled, DDL statements captured on the source are replayed on the target using the `pgstreamtarget` role. pgstream assumes the source and target are within the same trust domain.

If roles that can run DDL on the source are **less trusted** than `pgstreamtarget`, they can cause SQL to run on the target with the target role's privileges. To limit the impact:

* Grant `pgstreamtarget` only the privileges it needs to apply schema and data changes. Avoid `SUPERUSER`, `CREATEROLE`, and `CREATEDB` unless a feature explicitly requires them.
* If your source and target roles have different trust levels, disable DDL replication (see [`ignore_ddl`](/docs/opensource/pgstream/docs/configuration)).


# v1.0.0
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.0.0

**Release Date:** February 4th, 2026

## Major Release: Stateless DDL Replication

This is a **major breaking release** that fundamentally transforms how pgstream handles DDL (Data Definition Language) replication. Version 1.0.0 introduces a stateless architecture that eliminates the need for schema log tables and provides a more robust, maintainable solution for tracking database schema changes.

## Audience

These release notes are primarily intended for:

* Users upgrading from **v0.x** (action required)
* Operators running pgstream in production
* Developers integrating pgstream replication output

**Important:** Upgrading from any v0.x version **requires re-initialization**.

## Summary of Required Actions

* Existing installations must be **re-initialized**
* All `schema_log`–related configuration must be removed
* Injector configuration must be updated (`schemalog_url` → `source_url`)
* Snapshot schema mode must be updated

## Breaking Changes

### Removal of Schema Log (High Impact)

The schema log mechanism has been fully removed.

* The `pgstream.schema_log` table no longer exists
* All schema log–based tracking has been removed
* Schema changes are no longer persisted as database state

Existing installations **must migrate** to the new DDL-based model.

### Migration Framework Restructure (High Impact)

PostgreSQL migrations are now split into two independent groups.

#### Core migrations

* Location: `migrations/postgres/core/`
* Tracking table: `pgstream.schema_migrations_core`
* Provide:
  * DDL replication
  * Event triggers
  * Core replication infrastructure
* **Required for all installations**

#### Injector migrations

* Location: `migrations/postgres/injector/`
* Tracking table: `pgstream.schema_migrations_injector`
* Provide:
  * ID injection
  * Internal table ID tracking (`pgstream.table_ids`)
* **Only required when using search indexing or ID-dependent features**

### Configuration Defaults (Medium Impact)

* All `schema_log` configuration options have been removed
* LSN is now the default and only event version identifier

### Internal API Changes (Medium Impact)

* The `pkg/schemalog` package has been removed
* Processors now consume DDL events directly
* Snapshot schema generation is based on DDL events instead of schema log entries

These changes affect internal integrations and custom processors.

## Migration Guide

### New Installations

Simply install pgstream v1.0.0 - no migration needed.

```bash theme={null}
# Option 1: Initialize separately
pgstream init --config config.yaml

# Option 2: Initialize and run in one command
pgstream run --config config.yaml --init

# Option 3: Run only database migrations (without creating replication slot)
pgstream init --config config.yaml --migrations-only
```

The system will automatically install the appropriate migrations based on your configuration.

**Note:** The `--migrations-only` flag runs only the database migrations (creating the pgstream schema, tables, functions, and triggers) without creating the replication slot. This is useful when you want to set up the schema separately or when using different database credentials for migrations versus replication.

### Upgrading Existing Installations (v0.x → v1.0.0)

Upgrading requires re-initialization.

Recommended steps:

1. Stop the running pgstream instance

2. Remove existing state

   ```bash theme={null}
   pgstream destroy --config config.yaml --migrations-only
   ```

   Using `--migrations-only` preserves the replication slot and the pgstream schema, along with any non migration tables (e.g., `snapshot_requests`) and can reduce downtime.

3. Download pgstream v1.0.0

4. Initialize v1.0.0

   ```bash theme={null}
   pgstream init --config config.yaml

   # Or initialize and run in one command:
   pgstream run --config config.yaml --init
   ```

5. Start replication

   ```bash theme={null}
   pgstream run --config config.yaml
   ```

## ️ Required Configuration Changes

### Injector Configuration Rename

Old configuration:

```yaml theme={null}
modifiers:
  injector:
    enabled: true
    schemalog_url: "..."
```

New configuration:

```yaml theme={null}
modifiers:
  injector:
    enabled: true
    source_url: "..."
```

For PostgreSQL sources, `source_url` is not required.

### Schema Log Store Removal

PostgreSQL targets no longer require `schema_log_store_url`.

### Snapshot Schema Mode Change

The `schemalog` snapshot schema mode has been removed. It will always use `pgdump_pgrestore`.

Use:

```yaml theme={null}
snapshot:
  schema:
    mode: pgdump_pgrestore
```

## ️ Architecture Changes (Background)

The following sections describe the new replication model in more detail.

### Stateless DDL Replication

DDL is now captured directly from PostgreSQL using event triggers and logical WAL messages.

High-level flow:

1. DDL is captured using PostgreSQL event triggers
2. DDL events are enriched with metadata
3. DDL events are emitted as logical WAL messages
4. Schema changes are streamed inline with data changes

This removes persistent schema state while preserving event ordering.

### WAL-Based DDL Events

Schema changes are now represented as first-class WAL events.

This release introduces new Go structs describing DDL events and their associated metadata.

### Search Processor Changes

Search indexing no longer depends on the schema log.

* Field aliasing replaces schema-log-based mappings
* Column renames preserve aliases to maintain query compatibility
* Search results use actual table and column names instead of internal IDs

### Snapshot Changes

Snapshot processing has been unified for all targets. Instead of having a dedicated schema snapshot for non Postgres targets, now:

* DDL is extracted from `pg_dump` output
* DDL is converted into WAL-style DDL events
* Snapshot and live events are processed through the same pipeline

### Modular Migration Strategy

Migrations can now be applied selectively:

* Core migrations for standard replication
* Injector migrations only when search or ID tracking is required

This reduces database objects, permissions, and operational overhead.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.0.1
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.0.1

pgstream v1.0.1 release notes

**Release Date:** February 16th, 2026

pgstream v1.0.1 is a minor release on top of the v1.0.0 stateless DDL replication base. It adds DDL-aware enforcement for strict transformation rules, optional XIDs on replication events, and a set of snapshot and replication fixes.

## DDL Enforcement in Strict Transformation Mode

Closes [#262](https://github.com/xataio/pgstream/issues/262) via [PR #723](https://github.com/xataio/pgstream/pull/723).

Before this release, DDL events bypassed the transformer entirely. New tables or columns could appear without transformation rules applying, which broke the guarantee of `strict` validation mode and risked leaking untransformed data.

The transformer now inspects DDL events and blocks the ones not covered by the configured rules when validation is `strict` (globally or per-table):

* Tables added
* Columns added
* Tables renamed
* Columns renamed

The process stops with a descriptive error so the operator can update the transformation rules before re-applying the schema change. The transformer map was also extended to track `noop` alongside active transformers, so explicit no-op rules count as valid coverage during DDL validation.

## XIDs in Replication Events

Closes [#687](https://github.com/xataio/pgstream/issues/687) via [PR #721](https://github.com/xataio/pgstream/pull/721).

Replication events can now optionally carry the originating transaction ID. This is wired through to the `wal2json` plugin parameters and is **disabled by default**. Enable it via YAML or environment variables when downstream consumers need transaction-grouped events.

## Bug Fixes

* Duplicate `ALTER ... DROP` events are no longer emitted ([PR #724](https://github.com/xataio/pgstream/pull/724)).
* Schema names with capital letters are handled correctly during schema snapshot ([PR #715](https://github.com/xataio/pgstream/pull/715)).
* Search index-exists error handling fixed ([PR #720](https://github.com/xataio/pgstream/pull/720)).
* Use `MAX(ctid)` instead of probing for missing pages during snapshot ([PR #713](https://github.com/xataio/pgstream/pull/713)).
* Test formatting variable fix ([PR #716](https://github.com/xataio/pgstream/pull/716)).

## Other Changes

* `go.opentelemetry.io/contrib/instrumentation/runtime` 0.64.0 → 0.65.0 ([PR #717](https://github.com/xataio/pgstream/pull/717)).
* `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc` 1.39.0 → 1.40.0 ([PR #718](https://github.com/xataio/pgstream/pull/718)).
* `github.com/elastic/go-elasticsearch/v8` 8.19.1 → 8.19.2 ([PR #719](https://github.com/xataio/pgstream/pull/719)).

## Upgrade

No migration is required when upgrading from v1.0.0. Replace the binary and restart pgstream.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.0.2
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.0.2

pgstream v1.0.2 release notes

**Release Date:** May 11th, 2026

pgstream v1.0.2 is a patch release. It is built using Golang 1.26.3, which includes several security fixes. It also batches WAL operations in stream mode for the Postgres-to-Postgres sink and exposes the wal2json plugin's source-side table filters. Users on the v0.9.x maintenance line can take the same Go bump from [v0.9.12](https://github.com/xataio/pgstream/releases/tag/v0.9.12).

## Batched WAL Operations in Stream Mode

Closes [#769](https://github.com/xataio/pgstream/issues/769) via [PR #774](https://github.com/xataio/pgstream/pull/774).

Before this release, the WAL streaming phase executed each DML statement individually against the Postgres target. On high-churn tables, the post-snapshot catch-up phase could not keep up with the WAL backlog and would appear stuck. Stream mode now coalesces same-table operations:

* `INSERT` events become multi-row `INSERT` statements, chunked to stay under 60,000 query parameters.
* `DELETE` events become a single `DELETE ... WHERE pk = ANY($1::type[])` for single-column primary keys, or `DELETE ... WHERE (a,b) IN ((..),(..))` for composite primary keys.
* Rows with `NULL` identity values fall back to individual `DELETE` statements.

**Note:** This applies only to the Postgres target; Elasticsearch, Kafka, and webhook targets are unchanged. Only `INSERT` and `DELETE` are coalesced. `UPDATE` and `TRUNCATE` still process individually, so catch-up workloads dominated by `UPDATE` traffic can still see backpressure.

## Decode-Time Table Filtering via wal2json

[PR #791](https://github.com/xataio/pgstream/pull/791), contributed by [@blakewatters](https://github.com/blakewatters).

Two new configuration keys pass through directly to `wal2json` at the source Postgres. Excluded tables are not decoded into JSON, which reduces CPU and network overhead before events enter the pgstream pipeline.

YAML:

```yaml theme={null}
source:
  postgres:
    replication:
      plugin:
        add_tables: "public.*"
        filter_tables: "pipelines.*,private.*"
```

Environment variables:

```bash theme={null}
PGSTREAM_POSTGRES_REPLICATION_PLUGIN_ADD_TABLES="public.*"
PGSTREAM_POSTGRES_REPLICATION_PLUGIN_FILTER_TABLES="pipelines.*,private.*"
```

Configuration reference updated in [PR #803](https://github.com/xataio/pgstream/pull/803).

## Go 1.26.3

[PR #804](https://github.com/xataio/pgstream/pull/804) bumps the Go toolchain to 1.26.3 and ships the upstream security fixes that drove this release. [v0.9.12](https://github.com/xataio/pgstream/releases/tag/v0.9.12) ([PR #808](https://github.com/xataio/pgstream/pull/808)) is the same bump backported with no functional changes.

## Bug Fixes

* `clean_target_db` no longer drops the `pgstream` schema on the target ([PR #740](https://github.com/xataio/pgstream/pull/740), fixes [#736](https://github.com/xataio/pgstream/issues/736)).
* `pg_restore` surfaces the underlying exec error when its output contains no parseable `ERROR` lines, instead of emitting `error restoring dump: %!w(<nil>)` ([PR #771](https://github.com/xataio/pgstream/pull/771)).
* `UPDATE ... WHERE` clauses against the target now emit `IS NULL` instead of `= NULL` for null comparisons.

## Other Changes

* Concurrency limit added on the snapshot recorder.
* Schema workers renamed to snapshot workers in the recorder.

## Upgrade

No migration is required. Replace the binary and restart pgstream.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.0.3
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.0.3

pgstream v1.0.3 release notes

**Release Date:** May 20th, 2026

pgstream v1.0.3 is a patch release. It adds `tsvector` support to the Postgres-to-Postgres DML adapter, fixes schema snapshots of partitioned tables, introduces a `--data-only` snapshot flag, and resolves a regression that broke updates on always-identity columns.

## tsvector Replication Support

Closes [#576](https://github.com/xataio/pgstream/issues/576) via [PR #796](https://github.com/xataio/pgstream/pull/796), contributed by [@utafrali](https://github.com/utafrali).

Tables with `tsvector` columns previously failed to replicate because the DML adapter had no handler for that type. The adapter now converts `tsvector` values arriving as `[]byte` from the WAL into strings before they are used in the copy operation, and passes existing string values through unchanged.

## Partitioned Table Snapshots and `--data-only` Flag

[PR #775](https://github.com/xataio/pgstream/pull/775), contributed by [@blakewatters](https://github.com/blakewatters).

This release fixes a schema-restore failure on databases that use declarative partitioning (PostgreSQL 17/18) and adds a flag for data-only snapshots.

* **Partitioned table restore fix.** `pg_dump -Fp` output for partitioned tables emits both `CREATE TABLE ... PARTITION OF` and a redundant `ALTER TABLE ... ATTACH PARTITION`, which fails with `"already a partition"`. That message was not matched by the existing ignorable-error patterns, so the restore was treated as critical and aborted. These errors are now classified as harmless idempotent errors, alongside `"already exists"`.
* **`--data-only` flag.** A new `--data-only` boolean flag for `pgstream run` performs a data-only initial snapshot when combined with `--snapshot-tables`. It sets the snapshot mode to `data` instead of `full`, skipping the schema-restore phase. This is useful when the schema is restored separately (for example via `pg_dump --schema-only | psql`) or is managed independently on the target.

```bash theme={null}
# Schema already exists on target — only snapshot data, then start CDC
pgstream run \
  --source postgres --source-url "postgres://..." \
  --target postgres --target-url "postgres://..." \
  --snapshot-tables "public.*,private.*" \
  --data-only
```

## Bug Fixes

* Updates on `GENERATED ALWAYS AS IDENTITY` columns no longer fail. A regression introduced in [PR #754](https://github.com/xataio/pgstream/pull/754) caused `UPDATE` statements that included an always-identity column in the `SET` clause to be rejected by Postgres. `buildUpdateQuery` now drops always-identity columns from the `SET` clause, leaving `INSERT`, bulk `INSERT`, and `COPY` paths unchanged ([PR #820](https://github.com/xataio/pgstream/pull/820), fixes [#821](https://github.com/xataio/pgstream/issues/821)).

## CLI and Documentation

* The `snapshot` and `run` command descriptions, examples, and error messages now clarify that `snapshot` performs one-time snapshots only and does not support `snapshot_and_replication` or `replication` mode; use `run` for continuous or combined replication ([PR #807](https://github.com/xataio/pgstream/pull/807), relates [#332](https://github.com/xataio/pgstream/issues/332)).
* `cnpg_pooler_pgbouncer`, the CloudNativePG PgBouncer service account, is now documented as a Xata-managed role excluded from snapshots ([PR #819](https://github.com/xataio/pgstream/pull/819)).

## Other Changes

* `go.opentelemetry.io/otel` 1.41.0 → 1.43.0 ([PR #761](https://github.com/xataio/pgstream/pull/761)).
* `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc` ([PR #815](https://github.com/xataio/pgstream/pull/815)).
* `github.com/bytedance/sonic` 1.15.0 → 1.15.1 ([PR #814](https://github.com/xataio/pgstream/pull/814)).
* `github.com/tidwall/gjson` 1.18.0 → 1.19.0 ([PR #812](https://github.com/xataio/pgstream/pull/812)).
* `docker/login-action` 3 → 4 ([PR #768](https://github.com/xataio/pgstream/pull/768)).
* `goreleaser/goreleaser-action` 6 → 7 ([PR #738](https://github.com/xataio/pgstream/pull/738)).
* `clickhouse/clickhouse-server` Docker image bump ([PR #816](https://github.com/xataio/pgstream/pull/816)).
* `opensearchproject/opensearch-dashboards` Docker image bump ([PR #793](https://github.com/xataio/pgstream/pull/793)).

## Upgrade

No migration is required when upgrading from v1.0.x. Replace the binary and restart pgstream.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.1.0
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.1.0

pgstream v1.1.0 release notes

**Release Date:** June 16th, 2026

pgstream v1.1.0 is a minor release focused on configuration ergonomics and operability. YAML configuration accepts environment variables, byte-size settings accept human-readable units, logs can be emitted as JSON, and the process exposes `/health` and `/ready` endpoints. It also adds a `stdout` target for debugging pipelines, an optional null-byte sanitizer, `ltree` and `cube` type support, custom TLS for search stores, and an `--upgrade` flag that migrates a v0.9.x installation to v1.x. No migration is required when upgrading from v1.0.x.

## Environment Variables in YAML Configuration

[PR #790](https://github.com/xataio/pgstream/pull/790), contributed by [@mvanhorn](https://github.com/mvanhorn).

Any `${VAR}` reference in a YAML configuration file — including the transformation rules file — is replaced with the value of that environment variable before the configuration is parsed. Credentials no longer have to be committed alongside the rest of the configuration:

```yaml theme={null} theme={null}
source:
  postgres:
    url: "postgresql://${PG_USER}:${PG_PASSWORD}@${PG_HOST}:5432/mydatabase"
```

Only the braced form is expanded, and a variable that is not set expands to an empty string.

## Human-Readable Byte Sizes

[PR #823](https://github.com/xataio/pgstream/pull/823), by [@kvch](https://github.com/kvch).

Byte-size settings — batch sizes, auto-tuning bounds — accept a unit suffix instead of a raw byte count. Units are case-insensitive and binary (`1MiB` is 1048576 bytes). Plain integers keep working:

```yaml theme={null} theme={null}
target:
  postgres:
    batch:
      max_bytes: 80MiB
      auto_tune:
        min_batch_bytes: 1MiB
```

## Configurable Log Format

[PR #880](https://github.com/xataio/pgstream/pull/880), by [@kvch](https://github.com/kvch).

`--log-format` selects between the human-readable `console` output (the default) and structured `json` for log collectors. It can also be set with `PGSTREAM_LOG_FORMAT` or `logging.format.type`. A companion `--no-color` flag disables ANSI codes in console output.

## Health and Readiness Endpoints

[PR #882](https://github.com/xataio/pgstream/pull/882), by [@kvch](https://github.com/kvch).

The `run` and `snapshot` commands can start a small HTTP server exposing `/health` (liveness, always `200`) and `/ready` (readiness, which pings the source database when one is configured). Responses are JSON. This makes pgstream deployable behind Kubernetes probes without a sidecar:

```yaml theme={null} theme={null}
health:
  enabled: true
  address: ":9910"
```

The server is disabled by default and listens on `localhost:9910` unless an address is given.

## New `stdout` Target

[PR #883](https://github.com/xataio/pgstream/pull/883), by [@kvch](https://github.com/kvch).

A `stdout` target writes WAL events as NDJSON — one JSON object per line — so a pipeline, its filters and its transformation rules can be validated without provisioning a real target:

```yaml theme={null} theme={null}
target:
  stdout: {}
```

## Null-Byte Sanitizer

[PR #871](https://github.com/xataio/pgstream/pull/871), contributed by [@AasheeshLikePanner](https://github.com/AasheeshLikePanner).

Postgres `text` values may contain null bytes (`0x00`) that other targets reject. An optional processor strips them from string column values:

```yaml theme={null} theme={null}
modifiers:
  sanitize:
    strip_null_char_bytes: true
```

The processor is off by default, so pipelines that carry null bytes deliberately are unaffected.

## New Type Support

* **`ltree`** ([PR #834](https://github.com/xataio/pgstream/pull/834)) and **`cube`** ([PR #831](https://github.com/xataio/pgstream/pull/831)), by [@kvch](https://github.com/kvch). Both types are now replicated instead of failing on an unknown type.
* **`pgvector`** ([PR #809](https://github.com/xataio/pgstream/pull/809)), contributed by [@rieg-ec](https://github.com/rieg-ec). The pgvector types are registered on the connection, so vector columns replicate without a custom codec.

## Custom TLS for Search Stores

[PR #832](https://github.com/xataio/pgstream/pull/832), by [@kvch](https://github.com/kvch).

Elasticsearch and OpenSearch targets accept explicit TLS settings — CA certificate, client certificate and key — instead of relying on the system trust store, which is what self-managed clusters with a private CA need.

## `--upgrade` Flag for v0.9.x Installations

[PR #788](https://github.com/xataio/pgstream/pull/788), by [@tsg](https://github.com/tsg).

Moving from the 0.9 line to 1.x required a manual `destroy`/`init` cycle. `pgstream init --upgrade` now cleans up the v0.9 objects left behind by the schema-log architecture and re-initializes in one step. See the [v1.0.0 release notes](/docs/opensource/pgstream/docs/releases/v1.0.0) for what changes between the two lines.

## Bug Fixes

* **Precision loss on large integers and `bigint`** ([PR #833](https://github.com/xataio/pgstream/pull/833)). Values beyond the exact range of a float64 were rounded in transit; they are now carried without loss.
* **Composite primary keys in WAL metadata** ([PR #805](https://github.com/xataio/pgstream/pull/805)), contributed by [@blurskye](https://github.com/blurskye). All columns of a composite key are recorded, instead of only the first.
* **Constraints restored before snapshot conflict updates** ([PR #806](https://github.com/xataio/pgstream/pull/806)), contributed by [@blurskye](https://github.com/blurskye). An `ON CONFLICT` update needs the constraint it targets to exist, so constraint restore now happens first.
* **Transformer panics with multiple workers** ([PR #884](https://github.com/xataio/pgstream/pull/884)). Greenmask's generators keep state that is not safe for concurrent use; transformers now take an instance per execution, so snapshots with more than one worker no longer panic.
* **All error messages are collected before shutdown** ([PR #841](https://github.com/xataio/pgstream/pull/841)). A failing pipeline reported the first error only; the rest are now surfaced too.
* **`includeGlobalDBObjects` with `schema: all`** ([PR #760](https://github.com/xataio/pgstream/pull/760)), by [@tsg](https://github.com/tsg).
* **CVEs in `jackc/pgx`** ([PR #810](https://github.com/xataio/pgstream/pull/810)), contributed by [@fiws](https://github.com/fiws). pgx is updated to v5.9.2.

## New Contributors

* [@mvanhorn](https://github.com/mvanhorn) in [PR #790](https://github.com/xataio/pgstream/pull/790)
* [@SAY-5](https://github.com/SAY-5) in [PR #802](https://github.com/xataio/pgstream/pull/802)
* [@blurskye](https://github.com/blurskye) in [PR #805](https://github.com/xataio/pgstream/pull/805)
* [@rieg-ec](https://github.com/rieg-ec) in [PR #809](https://github.com/xataio/pgstream/pull/809)
* [@fiws](https://github.com/fiws) in [PR #810](https://github.com/xataio/pgstream/pull/810)
* [@AasheeshLikePanner](https://github.com/AasheeshLikePanner) in [PR #871](https://github.com/xataio/pgstream/pull/871)

## Upgrade

No configuration changes are required when upgrading from v1.0.x. Replace the binary and restart pgstream. Existing configurations keep working: environment-variable expansion, human-readable sizes, the sanitizer and the health server are all opt-in.

Users still on the 0.9 line can migrate with `pgstream init --upgrade`, which removes the v0.9 objects and re-initializes for v1.x.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.1.1
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.1.1

**Release Date:** July 6th, 2026

pgstream v1.1.1 is a patch release. It hardens the logical-replication path against SQL injection, fixes five snapshot-restore correctness bugs that surface on production-shaped schemas (materialized views, inherited and partitioned tables, `CLUSTER ON` ordering, range-type typing, and legacy handler functions), and backports the security fix to the 0.9.x line as [v0.9.15](https://github.com/xataio/pgstream/releases/tag/v0.9.15). No migration is required.

## SQL Parameter Binding and Escaping

[PR #965](https://github.com/xataio/pgstream/pull/965), by [@kvch](https://github.com/kvch). Backported as [v0.9.15](https://github.com/xataio/pgstream/releases/tag/v0.9.15).

Three raw-text interpolation spots in the logical-replication path are now bound as parameters or validated before use:

* WAL identifier quoting now validates a single complete quoted token instead of trusting leading and trailing `"`, so a crafted source object name cannot break out onto the target.
* Replication slot `CREATE` and `DROP` bind the slot name as a parameter.
* `START_REPLICATION` escapes table-filter plugin options and validates the slot name before use.

## Snapshot Restore Correctness

Five fixes, all contributed by [@danddanddand](https://github.com/danddanddand) in their first contribution to pgstream. Each surfaces on real production schemas rather than a test database.

* **Materialized views are refreshed after restore** ([PR #908](https://github.com/xataio/pgstream/pull/908)). Matviews previously restored empty; pgstream now runs `REFRESH MATERIALIZED VIEW ... WITH DATA` after the table data is restored. Refresh is opt-in and off by default, because refreshing large views is expensive. Enable it in the snapshot config:

  ```yaml theme={null} theme={null}
  source:
    postgres:
      snapshot:
        schema:
          pgdump_pgrestore:
            refresh_materialized_views: true
  ```

* **Inherited and partition child rows are no longer copied twice** ([PR #909](https://github.com/xataio/pgstream/pull/909)). Snapshot CTID queries now use `ONLY`, so child rows of inheritance and partition hierarchies are excluded from the parent scan.

* **`CLUSTER ON` is restored after its referenced index** ([PR #907](https://github.com/xataio/pgstream/pull/907)). Restoring `CLUSTER ON` before the index it references failed on an ordering error; the statement is now emitted after the index exists.

* **`int4range`, `int8range`, and `tstzrange` values are typed before `INSERT`/`COPY`** ([PR #910](https://github.com/xataio/pgstream/pull/910)). Range columns previously failed on restore because the value was passed untyped. (The PR title mentions only `int4range`/`int8range`; the shipped code also types `tstzrange`.)

* **Legacy `public` PL/pgSQL handler functions are skipped on restore** ([PR #906](https://github.com/xataio/pgstream/pull/906)). Databases carrying these handlers from old Postgres upgrades no longer hit a restore conflict.

## CLI

* The `xata` CLI is rebuilt against pgstream v1.1.1.

## Upgrade

No migration is required when upgrading from v1.0.x or v1.1.0. Replace the binary and restart pgstream. Users on the 0.9.x line who only need the security fix can take [v0.9.15](https://github.com/xataio/pgstream/releases/tag/v0.9.15) without upgrading to 1.x.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.2.0
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.2.0

pgstream v1.2.0 release notes

**Release Date:** July 15th, 2026

pgstream v1.2.0 is a minor release built around catching problems before a migration starts rather than during it. It introduces the experimental `check` command, which runs preflight validation of connectivity, replication, access, schema and resources against the source and target. Alongside it: a deterministic `encrypted_aes_siv` transformer, schema-only tables, parallelized COPY workers for faster snapshots, a configurable transformer error policy, a strict mode that fails snapshots on dropped Postgres writes, and automated nightly and container image releases. No migration is required.

## New `check` Command

[PR #901](https://github.com/xataio/pgstream/pull/901) and follow-ups, by [@kvch](https://github.com/kvch), with contributions from [@kv1sidisi](https://github.com/kv1sidisi) and [@lghuy05](https://github.com/lghuy05).

`pgstream check` runs preflight validation against the configuration you are about to run, and reports what would block a snapshot or a replication run:

```bash theme={null} theme={null}
pgstream check -c pg2pg.yaml
pgstream check -c pg2pg.env --json
```

Checks are grouped into five categories — `connectivity`, `replication`, `access`, `schema` and `resources` — and the command exits non-zero when any check fails, so it can gate a deployment.

What ships in this release:

* **Connectivity.** Source and target Postgres reachability ([PR #902](https://github.com/xataio/pgstream/pull/902), [PR #905](https://github.com/xataio/pgstream/pull/905)).
* **Replication.** `wal_level`, slot availability and related replication prerequisites ([PR #904](https://github.com/xataio/pgstream/pull/904)), plus replica identity ([PR #911](https://github.com/xataio/pgstream/pull/911)).
* **Access.** `SELECT` privileges on the source tables ([PR #912](https://github.com/xataio/pgstream/pull/912)) and on sequences ([PR #947](https://github.com/xataio/pgstream/pull/947)), both contributed by [@lghuy05](https://github.com/lghuy05). Schema, table and role identifiers are quoted in the messages ([PR #944](https://github.com/xataio/pgstream/pull/944)).
* **Schema.** Schema compatibility validation, contributed by [@kv1sidisi](https://github.com/kv1sidisi) ([PR #896](https://github.com/xataio/pgstream/pull/896)), moved into preflight ([PR #951](https://github.com/xataio/pgstream/pull/951)), plus source/target extension compatibility ([PR #952](https://github.com/xataio/pgstream/pull/952)).
* **Resources.** A new category covering resource limits on the source and target ([PR #971](https://github.com/xataio/pgstream/pull/971)).

The command is experimental: the checks and their output format may change in later releases.

## New `encrypted_aes_siv` Transformer

[PR #998](https://github.com/xataio/pgstream/pull/998), contributed by [@geo-ocean](https://github.com/geo-ocean).

A deterministic authenticated-encryption transformer based on AES-SIV (RFC 5297). The same input, key and associated data always produce the same token, so equality relationships survive across rows, tables and runs — which makes it the transformer to reach for on a column covered by a unique index or used as a join key:

```yaml theme={null} theme={null}
transformations:
  table_transformers:
    - schema: public
      table: orders
      column_transformers:
        file_path:
          name: encrypted_aes_siv
          parameters:
            key_hex: "${PGSTREAM_AES_KEY}" # 64-byte key, hex-encoded
            associated_data: "public.orders.file_path"
```

Because it is reversible by anyone holding `key_hex`, it is pseudonymization rather than anonymization: keep the key in a secret store, use a different key per environment, and set `associated_data` so tokens cannot be correlated between columns. Tokens are longer than their input, so length-constrained columns must be wide enough to hold them.

## Schema-Only Tables

[PR #999](https://github.com/xataio/pgstream/pull/999), by [@tsg](https://github.com/tsg), with the underlying table selection in [PR #949](https://github.com/xataio/pgstream/pull/949).

Tables can be replicated as structure without data. On the snapshot side, `schema_only_tables` restores the table definition and copies its sequence values but skips the rows:

```yaml theme={null} theme={null}
source:
  postgres:
    snapshot:
      tables: ["*.*"]
      schema_only_tables: ["audit_log", "reports.*"]
```

The same list exists as a filter for the streaming side, where DDL events are processed and DML events are skipped. This is what large append-only audit tables need: the target keeps a usable schema without paying to copy history.

## Parallelized COPY Workers

[PR #995](https://github.com/xataio/pgstream/pull/995), by [@kvch](https://github.com/kvch).

Bulk ingest opens several concurrent COPY streams per table instead of one, governed by `copy_workers` (default 8):

```yaml theme={null} theme={null}
target:
  postgres:
    bulk_ingest:
      enabled: true
      copy_workers: 8
```

## Configurable Transformer Error Policy

[PR #958](https://github.com/xataio/pgstream/pull/958), by [@kvch](https://github.com/kvch).

What happens when a transformer fails on a value is now a choice rather than a fixed behaviour. `on_error` accepts `fail` (stop the pipeline), `pass-through` (emit the original value) or `null`:

```yaml theme={null} theme={null}
modifiers:
  transformations:
    on_error: fail
```

Note that `pass-through` emits unanonymized data on failure, which is rarely what an anonymization pipeline wants.

## Strict Mode for Dropped Writes

[PR #959](https://github.com/xataio/pgstream/pull/959), by [@kvch](https://github.com/kvch), and [PR #976](https://github.com/xataio/pgstream/pull/976), contributed by [@danddanddand](https://github.com/danddanddand).

The Postgres target can drop a failing query and continue, which keeps a pipeline alive at the cost of silent divergence. `strict_mode` stops the pipeline instead, and snapshots now fail when a write is dropped:

```yaml theme={null} theme={null}
target:
  postgres:
    strict_mode: true
```

A counter for dropped messages ships with it, so the non-strict path is measurable rather than only visible in logs.

## Nightly and Container Image Releases

[PR #955](https://github.com/xataio/pgstream/pull/955) and [PR #980](https://github.com/xataio/pgstream/pull/980), by [@kvch](https://github.com/kvch).

Container images are built and published automatically with build provenance and SBOM attestations, and a nightly release is published from `main` for testing unreleased fixes.

## Other Improvements

* **SQL parameter binding and escaping** in the replication path ([PR #965](https://github.com/xataio/pgstream/pull/965)). Also released on the 0.9 line as [v0.9.15](https://github.com/xataio/pgstream/releases/tag/v0.9.15).
* **Postgres temp schemas are excluded** from snapshot discovery ([PR #954](https://github.com/xataio/pgstream/pull/954)), contributed by [@danddanddand](https://github.com/danddanddand).
* **Legacy `public` PL/pgSQL handler functions are skipped** on snapshot restore ([PR #906](https://github.com/xataio/pgstream/pull/906)).
* **pgstream metadata DML is skipped** for Postgres targets ([PR #977](https://github.com/xataio/pgstream/pull/977)), contributed by [@danddanddand](https://github.com/danddanddand).
* **`validate` exits 1 when validation fails** ([PR #948](https://github.com/xataio/pgstream/pull/948)), contributed by [@fiws](https://github.com/fiws), so it can be used in CI.
* **Log level reloads on `SIGHUP`** ([PR #945](https://github.com/xataio/pgstream/pull/945)), contributed by [@pollychen-lab](https://github.com/pollychen-lab), for turning on debug logging without a restart.
* **Hot-path performance**: column types and timestamps are resolved once per result set ([PR #950](https://github.com/xataio/pgstream/pull/950)), the log level is checked before logging ([PR #969](https://github.com/xataio/pgstream/pull/969)), and DDL events are converted once per event in the processor chain ([PR #970](https://github.com/xataio/pgstream/pull/970)).

## Bug Fixes

* **Top-level JSONB `null` values** are treated correctly instead of being confused with a SQL NULL ([PR #991](https://github.com/xataio/pgstream/pull/991)), by [@tsg](https://github.com/tsg).
* **`exclude_tables` across multiple schemas** no longer excludes the wrong tables ([PR #992](https://github.com/xataio/pgstream/pull/992)), by [@tsg](https://github.com/tsg).
* **`snapshot` CLI auto-enables bulk ingest** when the corresponding flags are used ([PR #993](https://github.com/xataio/pgstream/pull/993)).
* **Batch sender startup race** between `NewSender` and `Close` ([PR #994](https://github.com/xataio/pgstream/pull/994)).
* **Data race in the batch-bytes auto-tuner** ([PR #957](https://github.com/xataio/pgstream/pull/957)).
* **Backoff `MaxInterval` was wired to `MaxElapsedTime`** ([PR #956](https://github.com/xataio/pgstream/pull/956)), so retry intervals grew far past their configured ceiling.
* **`GetTables` shadowed the outer result slice** during snapshots ([PR #960](https://github.com/xataio/pgstream/pull/960)).

## New Contributors

* [@kv1sidisi](https://github.com/kv1sidisi) in [PR #896](https://github.com/xataio/pgstream/pull/896)
* [@lghuy05](https://github.com/lghuy05) in [PR #912](https://github.com/xataio/pgstream/pull/912)
* [@pollychen-lab](https://github.com/pollychen-lab) in [PR #945](https://github.com/xataio/pgstream/pull/945)
* [@geo-ocean](https://github.com/geo-ocean) in [PR #998](https://github.com/xataio/pgstream/pull/998)

## Upgrade

No configuration changes are required, and no re-initialization is needed. Replace the binary and restart pgstream. The `check` command, `encrypted_aes_siv`, schema-only tables and strict mode are all opt-in.

Scripts that call `pgstream validate` should be aware that it now exits with code 1 when validation fails, where it previously exited 0.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.2.1
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.2.1

pgstream v1.2.1 release notes

**Release Date:** July 20th, 2026

pgstream v1.2.1 is a patch release. It adds pipeline phase visibility through the health server and OpenTelemetry, fixes a btree index overflow that blocked snapshot requests covering many tables, and corrects three preflight and restore behaviours that reported problems where there were none. No migration is required.

## Pipeline Phase Visibility

[PR #1000](https://github.com/xataio/pgstream/pull/1000), contributed by [@Shaurya2k06](https://github.com/Shaurya2k06) in their first contribution.

A pipeline that is still snapshotting behaves differently from one that has moved on to replication, and there was no way to tell them apart from the outside. The health server now exposes `/status`, reporting the current phase — `snapshot` or `replication` — and the same value is published as an OpenTelemetry gauge, so the transition can be alerted on:

```yaml theme={null} theme={null}
health:
  enabled: true
  address: ":9910"
```

`/status` is served alongside the existing `/health` and `/ready` endpoints by the `run` and `snapshot` commands.

## Bug Fixes

* **Snapshot requests with many tables no longer overflow the btree index** ([PR #1013](https://github.com/xataio/pgstream/pull/1013)). The unique index on snapshot requests included `table_names` directly; a request listing enough tables exceeded the btree entry size limit and could not be recorded. The column is hashed in the index instead, which bounds the entry regardless of how many tables a request names.
* **Custom enums are no longer reported as unsupported types** ([PR #1005](https://github.com/xataio/pgstream/pull/1005)). Preflight schema validation flagged user-defined enum columns as incompatible on sources that replicate them without trouble.
* **Parsing of validated schema objects is deferred** ([PR #1003](https://github.com/xataio/pgstream/pull/1003)), by [@tsg](https://github.com/tsg).
* **A failed comment restore is a warning, not an error** ([PR #1009](https://github.com/xataio/pgstream/pull/1009)). `COMMENT ON` statements that fail during restore no longer abort it; a missing comment does not affect data or structure.

## New Contributors

* [@Shaurya2k06](https://github.com/Shaurya2k06) in [PR #1000](https://github.com/xataio/pgstream/pull/1000)

## Upgrade

No configuration changes are required. Replace the binary and restart pgstream.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.2.2
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.2.2

pgstream v1.2.2 release notes

**Release Date:** July 22nd, 2026

pgstream v1.2.2 is a patch release. It makes the index and constraint restore phase of a snapshot tunable with per-session Postgres settings, warns when a snapshot source looks load-balanced, and fixes a stack-depth overflow on bulk deletes of tables with composite primary keys along with a `pg_restore` that ignored context cancellation. No migration is required.

## Session Settings for Index and Constraint Restore

[PR #990](https://github.com/xataio/pgstream/pull/990), contributed by [@jonasHanhan](https://github.com/jonasHanhan) in their first contribution.

Building indexes and validating constraints is the most resource-hungry phase of a snapshot restore, and the settings that make it fast are the ones you would not want applied to the data copy. Session settings can now be applied only to that phase:

```yaml theme={null} theme={null}
source:
  postgres:
    snapshot:
      schema:
        pgdump_pgrestore:
          index_constraint_session_settings:
            - maintenance_work_mem=4GB
            - max_parallel_maintenance_workers=4
```

Each entry must be a whitespace-free `name=value` pair. Leaving the setting unset preserves the previous behaviour.

The settings are validated at startup rather than at restore time ([PR #1021](https://github.com/xataio/pgstream/pull/1021)), so a typo fails immediately instead of hours into a snapshot, and a value cannot smuggle extra options into the session.

## Preflight Detects Load-Balanced Snapshot Sources

[PR #1015](https://github.com/xataio/pgstream/pull/1015), by [@kvch](https://github.com/kvch).

A data snapshot depends on `SET TRANSACTION SNAPSHOT` pinning one instance's view. Pointed at an Aurora or RDS reader endpoint, or a pooler that spans instances, successive connections can land on different instances and the snapshot silently mixes views. `pgstream check` now warns when the snapshot source looks load-balanced.

## Bug Fixes

* **Stack-depth overflow on bulk composite primary key deletes** ([PR #1024](https://github.com/xataio/pgstream/pull/1024)). Deletes on a table with a composite primary key were coalesced into an expression that grew with the batch, and a large enough batch exceeded Postgres' `max_stack_depth`, failing the whole statement.
* **`pg_restore` honors context cancellation** ([PR #1023](https://github.com/xataio/pgstream/pull/1023)). A restore in progress ignored cancellation, so shutdowns and timeouts hung until it finished on its own.

## New Contributors

* [@jonasHanhan](https://github.com/jonasHanhan) in [PR #990](https://github.com/xataio/pgstream/pull/990)

## Upgrade

No configuration changes are required. Replace the binary and restart pgstream. `index_constraint_session_settings` is opt-in, and invalid entries now fail at startup rather than at restore time.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.2.3
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.2.3

pgstream v1.2.3 release notes

**Release Date:** July 27th, 2026

pgstream v1.2.3 is a patch release. It fixes bulk ingest on tables with user-defined enum columns, stops Kafka `primary_key` partitioning from collapsing distinct rows onto one message key, corrects two preflight checks that reported false failures, and lets preflight run source-only checks. Kafka pipelines using key-based partitioning need to read the upgrade notes before deploying.

## Bug Fixes

* **Bulk ingest handles user-defined enum columns** ([PR #1035](https://github.com/xataio/pgstream/pull/1035)). Snapshotting a table with an enum column failed on the COPY path because pgx has no codec for a database-specific enum OID. Enum values are now sent as text and cast on the target.
* **Kafka message keys are escaped** ([PR #1036](https://github.com/xataio/pgstream/pull/1036)). With `partition_key: primary_key` or `table`, the key was built by joining identifiers and values with `.`, `:` and `,`. A value containing one of those characters produced the same key as a different row, so distinct rows shared a key and lost their per-row ordering guarantee. The delimiters are now escaped, so they only ever appear as structure.
* **Two preflight checks no longer report false failures** ([PR #1034](https://github.com/xataio/pgstream/pull/1034)). The wal2json and replica identity checks failed on correctly configured sources.
* **The Postgres version check for source size support is corrected** ([PR #1027](https://github.com/xataio/pgstream/pull/1027)).

## Preflight Improvements

* **Source-only checks** ([PR #1025](https://github.com/xataio/pgstream/pull/1025)). `pgstream check --source` runs only the source-side checks, which is what you want when the target does not exist yet.
* **Major version compatibility** ([PR #1026](https://github.com/xataio/pgstream/pull/1026)). Preflight verifies that the source and target major PostgreSQL versions are compatible before a migration starts.

Both by [@kvch](https://github.com/kvch).

## Upgrade

⚠️ **Kafka `partition_key: primary_key` and `table` users only.** Message keys are now escaped so that the `.`, `:` and `,` delimiters can only appear as structure and never as data. Any key whose identifiers or values contain one of the escaped characters is encoded differently than before and therefore hashes to a **different partition** after upgrading.

Integer and UUID primary keys are unaffected. Keys that do change include `bytea` values (replicated in Postgres hex format, `\x...`, so every one contains a backslash), array and composite typed values (which render with commas), and schemas or tables whose quoted identifiers contain `.`, `:` or `\`.

Where this applies, drain consumers to the end of the topic before upgrading: events for one row can otherwise interleave across the restart boundary, and on a **compacted** topic a record written under the old key is never superseded by its replacement under the new one.

Everything else requires no configuration changes.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.2.4
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.2.4

pgstream v1.2.4 release notes

**Release Date:** July 27th, 2026

pgstream v1.2.4 is a patch release. It fixes replicated `DELETE`s on tables whose replica identity includes a user-defined enum column, extends the fix to enum arrays and domains over enums, and stops a single failing query from taking its whole batch down with it. Pipelines replicating tables with enum primary keys should read the upgrade notes: the target may already have diverged.

## Replicated Deletes on Enum Primary Keys

[PR #1038](https://github.com/xataio/pgstream/pull/1038), by [@kvch](https://github.com/kvch).

Every `DELETE` replicated to a table whose replica identity contained a user-defined enum column failed with `unable to encode []interface {}{...} into text format for unknown type`. Consecutive same-table deletes are coalesced into a single statement that binds one array parameter per identity column, and pgx has no codec for a database-specific enum *array* OID — so the statement never left the client. Scalar enum values are plain strings and encode fine, which is why `INSERT` and `UPDATE` were unaffected and only deletes broke. Identity values for these columns are now sent as `text[]` and cast back on the target.

This is the replication counterpart of the snapshot fix in [v1.2.3](/docs/opensource/pgstream/docs/releases/v1.2.3) ([PR #1035](https://github.com/xataio/pgstream/pull/1035)), which covered only the bulk-ingest COPY path.

## Enum Arrays and Domains Over Enums

The cast target is resolved from the target catalog — the enum itself, reached through the array element type and the domain base type — rather than from the column type carried by the replication stream. Casting to the column's declared type is not equivalent: the enum comparison operators are polymorphic over `anyenum`, which does not accept a domain. This also fixes deletes on a domain-over-enum identity column, which previously failed on *every* path, including the per-row form that predates delete coalescing.

## Failure Containment

A client-side encoding failure carries no SQLSTATE, so it was classified as an internal error: the whole send failed rather than the one offending query being dropped, and with `ignore_send_errors: true` the entire batch was discarded — including co-batched writes to unrelated tables. Such failures are now recognized as deterministic and query-attributable, so only the failing query is dropped and the rest of the batch proceeds. They are also no longer retried; with the default retry policy (no attempt or elapsed-time bound) a deterministic encoding failure would otherwise retry indefinitely.

Two related fixes ship alongside: a transaction retried after a transient failure no longer reports already-committed queries as needing a retry, and the DATALOSS warning for a dropped query now records the cause and the affected schema and table.

## Upgrade

No configuration changes are required, and the SQL emitted for tables without enum identity columns is unchanged.

⚠️ **If you replicate tables with enum primary keys**, the target may already have diverged: before this release those deletes failed permanently, and with `ignore_send_errors: true` the surrounding batch was dropped with it. Upgrading fixes future deletes but does not reconcile rows that were missed — check for rows on the target that no longer exist on the source, and re-snapshot the affected tables if needed.

Operators who would rather halt than let the target diverge can set `strict_mode`, which stops the pipeline on any query that would otherwise be dropped. Note that it has no effect while `ignore_send_errors` is `true`, as that swallows the error first.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.2.5
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.2.5

pgstream v1.2.5 release notes

**Release Date:** July 28th, 2026

pgstream v1.2.5 is a patch release about data loss that used to be hard to see. The `template`, `json` and `hstore` transformers no longer corrupt their shared random generator under multi-worker snapshots; batches discarded by `ignore_send_errors` are counted and exported as metrics; snapshot failures are logged as errors instead of rendering as `{}`; snapshot request status updates are retried; and a failed restore now carries the output that explains it, with credentials redacted. No migration is required.

## Shared Template State

[PR #1045](https://github.com/xataio/pgstream/pull/1045), by [@kvch](https://github.com/kvch).

The `template`, `json` and `hstore` transformers build their templates from greenmask's toolkit function map, which closes over a single `*rand.Rand` shared by every `random*` and `noise*` function. pgstream builds one transformer per column and calls it from all snapshot workers, so a template as simple as `{{ randomInt 1 1000 }}` had several goroutines drawing from one generator: the lagged-Fibonacci source's indices were corrupted and the call failed with `index out of range [-1]`. With the batch writer's error handling this surfaced as a per-row `DATALOSS` — the transformed value dropped or nulled — rather than a crash, so it was easy to miss.

Templates are now pooled, so each execution gets an instance with function map state of its own. This is the same mechanism applied to the greenmask generators in [PR #884](https://github.com/xataio/pgstream/pull/884), which covered the string transformer and the byte generators but not the template function map. Pooling also covers the other shared state in that map: the `pgtype.Map` behind `tsModify` and `noiseDatePgInterval`.

## Accounting for Dropped Batches

[PR #1042](https://github.com/xataio/pgstream/pull/1042), by [@kvch](https://github.com/kvch).

`ignore_send_errors: true` turns a failed send into a dropped batch and keeps the pipeline running. Until now nothing counted that: each drop produced a log line, with no total anywhere and no metric at all, so the only way to size the loss was to count log lines after the fact.

Every writer that can drop — the Postgres batch and bulk-ingest writers, the Kafka writer and the search indexer — now shares one counter with the batch senders it builds, exported as two observable counters:

* `pgstream.batch.sender.dropped_batches` — batches discarded because `ignore_send_errors` is enabled
* `pgstream.batch.sender.dropped_messages` — messages lost with those batches

Both carry a `writer_type` attribute — `postgres_batch_writer`, `postgres_bulk_ingest_writer`, `kafka_batch_writer` or `search_batch_indexer` — so a run with several writers can be told apart. Each drop is logged with `severity: DATALOSS`, the batch's message count and size, and the running totals; the totals are reported once more at shutdown.

Suppression is also stated at startup rather than being visible only in hindsight: a writer configured with `ignore_send_errors` warns that failing batches will be dropped, and the Postgres batch writer says when `strict_mode` is off, since that governs the per-query drop-and-continue path.

Relatedly, `pgstream.postgres.writer.dropped_queries` is now registered only for the batch writer, the only writer that can increment it. It was previously exported for the bulk-ingest writer as well, where it published a permanent zero under a `writer_type` label naming a component that does drop data, by whole batches.

## Snapshot Reporting

[PR #1039](https://github.com/xataio/pgstream/pull/1039) and [PR #1041](https://github.com/xataio/pgstream/pull/1041), by [@kvch](https://github.com/kvch).

A snapshot that failed logged `snapshot generation completed` at info level with the error rendered as `{}`, because the error was passed as a log field rather than as an error. Failures are now logged as errors with the message `snapshot generation failed`.

Recording the outcome of a snapshot request is also no longer a single attempt: the status update is retried (3 attempts, 500ms apart). If it still cannot be persisted, the failure is logged with `severity: DATALOSS` and the status that remains stored, because a request left in a non-terminal state is never picked up again.

## Restore Errors Carry Their Output

[PR #1040](https://github.com/xataio/pgstream/pull/1040), by [@kvch](https://github.com/kvch).

When `pg_restore` or `psql` exited non-zero without a parseable error, pgstream reported only the exit status, discarding the output that said what actually went wrong. The error now carries the tail of that output (up to 4KiB), with credential material removed first: `PASSWORD '...'` literals in role statements and the row payload psql appends to a failed `COPY` are both redacted.

## Upgrade

No configuration changes are required.

⚠️ **If you use the `template`, `json` or `hstore` transformers with greenmask's `random*`/`noise*` functions**, snapshots run with more than one worker before this release may have written incomplete data: affected values were dropped or nulled per row rather than failing the run. Check the affected columns and re-snapshot if needed. The workaround for earlier versions — sprig's `randInt` in place of `randomInt`, or `table_workers`/`schema_workers` set to 1 — is no longer needed.

**Alerting on drop logs.** When `ignore_send_errors` is enabled, the per-drop line changed from `failed to send batch` to `failed to send batch: dropping it and continuing`, and now carries `severity: DATALOSS`. Alerts keyed on the old message text need updating; keying on the severity field covers this and the other silent-loss sites. The message is unchanged when `ignore_send_errors` is off, where the error still stops the pipeline.

**Metrics.** The two new counters are exported only when instrumentation is enabled. `pgstream.postgres.writer.dropped_queries` no longer appears with `writer_type: postgres_bulk_ingest_writer`; that series was always zero.

**Contributors.** `make lint` now installs golangci-lint v2.10.1 to match CI, which reports findings the previously pinned v2.5.0 did not.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.3.0
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.3.0

pgstream v1.3.0 release notes

**Release Date:** July 30th, 2026

pgstream v1.3.0 is a small minor release. It validates and documents running both the snapshot and the replication phases against a physical read replica instead of the primary, and fixes an encoding bug that turned empty strings into null bytes. No migration is required.

## Snapshot and Stream From a Read Replica

[PR #1049](https://github.com/xataio/pgstream/pull/1049), by [@tsg](https://github.com/tsg).

Running pgstream against a physical streaming replica keeps the load — and the long-running transaction a snapshot holds — off the primary. This release adds integration test coverage for that setup and a guide covering what it requires: PostgreSQL 16 or later, `wal_level = logical` on both the primary and the replica (the replica does not inherit it), `hot_standby_feedback = on`, the wal2json plugin installed on the replica's host, and a single-instance endpoint rather than a load-balanced reader.

See [Running pgstream from a read replica](https://github.com/xataio/pgstream/blob/main/docs/replicas.md) for the full setup.

## Bug Fixes

* **Empty strings are no longer encoded as null bytes** ([PR #1051](https://github.com/xataio/pgstream/pull/1051)), by [@tsg](https://github.com/tsg).

## Upgrade

No configuration changes are required. Replace the binary and restart pgstream.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.3.1
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.3.1

pgstream v1.3.1 release notes

**Release Date:** August 4th, 2026

pgstream v1.3.1 is a patch release about snapshots taken while the source schema is moving. The data snapshot now reads the columns the schema snapshot captured, so a column added mid-snapshot no longer breaks the load, and drift that the pin cannot absorb is reported instead of silently applied. It also adds a `CREATEDB` preflight check for snapshots that create the target database, and documents how DDL replication can carry privileges from a less trusted source to the target. No migration is required.

## Snapshot Column Pinning

[PR #1058](https://github.com/xataio/pgstream/pull/1058), with catalog helpers in [PR #1057](https://github.com/xataio/pgstream/pull/1057), by [@kvch](https://github.com/kvch).

The data snapshot read rows with `SELECT * FROM ONLY <table> WHERE ctid BETWEEN '(x,0)' AND '(y,0)'`. `SET TRANSACTION SNAPSHOT` pins row visibility, but not the catalog, and every page range plans its own query — so the column list was re-resolved from the live source catalog throughout the snapshot, while the target's schema stayed as the schema dump had left it. A column added to the source once the dump was done appeared in the reads of every page range planned after it, and those rows carried a column the target did not have.

Each table's read is now pinned to the columns the schema snapshot captured. `pg_dump` is preceded by a catalog read that records the column list of every table in the dump's scope, and the page range queries spell those columns out instead of `*`, so a column added afterwards is simply not read. The `ADD COLUMN` and any backfill arrive over the replication stream, which already starts from an LSN taken before the snapshot. A snapshot with no schema dump — schema-only runs, or a data snapshot configured on its own — captures nothing and keeps reading `*` as before.

## Schema Drift Is Reported Rather Than Absorbed

The pin narrows the window but does not remove it: the capture and the dump are two separate reads of the source. After the dump, the columns are read again and any table that changed in between is named in a warning, since it may be snapshotted with stale columns and needs replication to converge — or a re-snapshot if none follows.

A column dropped after the capture is dropped from the read too, so the rest of the table still loads. If *none* of a table's captured columns still exist, the table snapshot fails with `source schema changed during the snapshot` rather than falling back to `SELECT *`, which would reintroduce the bug for that table. A table dropped mid-snapshot now reports the same error instead of a bare `relation does not exist`. A table that the capture did not see — created after it, or a name that did not resolve to a catalog entry — is logged and read with the columns it has now.

## `CREATEDB` Preflight Check

[PR #996](https://github.com/xataio/pgstream/pull/996), contributed by [@lghuy05](https://github.com/lghuy05).

`pgstream check` verifies that the target role can create databases when the snapshot is configured to create the target database:

```yaml theme={null} theme={null}
snapshot:
  schema:
    pgdump_pgrestore:
      create_target_db: true
```

The check reads `rolcreatedb` from `pg_roles` for the current target session role and reports an actionable finding when the privilege is missing, instead of the run failing at restore time. It is registered only when `create_target_db` is enabled for a Postgres target.

## DDL Replication and Trust Domains

[PR #1052](https://github.com/xataio/pgstream/pull/1052), by [@tsg](https://github.com/tsg).

DDL captured on the source is replayed on the target as the `pgstreamtarget` role, which means pgstream assumes source and target are in one trust domain. Where they are not — roles that can run DDL on the source being less trusted than `pgstreamtarget` — those roles can get SQL executed on the target with the target role's privileges. [`docs/privileges.md`](https://github.com/xataio/pgstream/blob/main/docs/privileges.md) now says so, and recommends granting `pgstreamtarget` only what it needs (no `SUPERUSER`, `CREATEROLE` or `CREATEDB` unless a feature requires it), or disabling DDL replication with `ignore_ddl` where the trust levels differ.

## Upgrade

No configuration changes are required, and no re-initialization or migration is needed.

⚠️ **If you snapshot a source that receives DDL.** Snapshots run before this release may have loaded rows shaped by a column list that changed underneath them; a table that failed to write mid-snapshot for a column the target did not have needs a re-snapshot. Going forward, watch for the two new signals: the drift warning naming tables that changed during the dump, and `source schema changed during the snapshot`, which aborts that table's snapshot.

**Snapshots that previously ran to completion can now fail.** The `source schema changed during the snapshot` error is new: a table whose captured columns have all been dropped, or that was dropped outright, now stops rather than reading whatever the source has at that moment. This is the intended outcome — the alternative silently produces a table the target cannot accept — but a pipeline that snapshots concurrently with heavy DDL will surface it as a table-level failure that did not exist before.

**Preflight.** `pgstream check` with `create_target_db: true` now runs one additional query against the target and reports a finding if the role lacks `CREATEDB`. Targets already holding the privilege see no change beyond an extra passing check in the output.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# v1.4.0
Source: https://xata.io/docs/opensource/pgstream/docs/releases/v1.4.0

pgstream v1.4.0 release notes

**Release Date:** August 12th, 2026

pgstream v1.4.0 is a minor release. It adds a native Prometheus scrape endpoint, granular object type filtering for Postgres-to-Postgres snapshots and DDL replication, and validation of transformation rules against unique indexes before a run starts instead of failing mid-load. Target connection pool limits become configurable, webhook delivery gains retries and idempotency headers, and a DDL parsing bug that silently dropped `RENAME COLUMN` and `DROP COLUMN` changes is fixed. Pipelines with a lossy transformer on a unique-indexed column need attention before upgrading — see the upgrade notes.

## Prometheus Scrape Endpoint

[PR #1008](https://github.com/xataio/pgstream/pull/1008), contributed by [@pauliyobo](https://github.com/pauliyobo) in their first contribution.

`instrumentation.metrics.prometheus.enabled` exposes a `/metrics` endpoint on the existing health server, wired into the OpenTelemetry meter provider independently of the OTLP exporter. Metrics can be scraped directly, with no collector in between:

```yaml theme={null} theme={null}
instrumentation:
  metrics:
    prometheus:
      enabled: true # Defaults to false
      endpoint: "/metrics" # Defaults to /metrics
```

The endpoint returns `404` when disabled.

## Granular Object Type Filtering

[PR #735](https://github.com/xataio/pgstream/pull/735), by [@tsg](https://github.com/tsg).

Both the schema snapshot and DDL replication for Postgres-to-Postgres pipelines can be restricted to a subset of object categories — `tables`, `sequences`, `types`, `indexes`, `functions`, `views`, `triggers` and others, 16 in total — via `include_object_types` (allowlist) or `exclude_object_types` (denylist), independently for snapshot and replication. A DDL statement that touches several object types (for example `CREATE TABLE` with a primary key) is only skipped when every object it touches is excluded.

This does not resolve inter-object dependencies: excluding a category that surviving objects depend on will make the snapshot or DDL replay fail, so use it with a good understanding of your schema.

## Transformation Rules Validated Against Unique Indexes

[PR #1082](https://github.com/xataio/pgstream/pull/1082), [PR #1084](https://github.com/xataio/pgstream/pull/1084), [PR #1085](https://github.com/xataio/pgstream/pull/1085) and [PR #1086](https://github.com/xataio/pgstream/pull/1086), by [@kvch](https://github.com/kvch).

Anonymization is lossy by design: `masking` with `type: id` keeps a 6-character prefix, so every value sharing that prefix collapses to the same masked value. Applied to a column covered by a unique index, this previously surfaced as `duplicate key value violates unique constraint` hours into a data load.

Every transformer is now classified as `preserved` (distinct inputs stay distinct), `not_guaranteed` (random or hashed output, collisions possible) or `lossy` (collides by construction — masking, the greenmask and neosync name and choice generators, `literal_string`). `validate rules` and pipeline startup read `pg_index` for the tables in scope and check the configured rules against unique indexes, primary keys and unique constraints.

With a Postgres target, a `lossy` transformer on a covered column fails at parse time with `transformation rules break a unique index`; with other targets (Kafka, Elasticsearch/OpenSearch, webhooks) the same finding is a warning, since there is no unique index to violate. A `not_guaranteed` transformer always warns. Set `allow_uniqueness_loss: true` on a column rule to keep a lossy transformer anyway.

Related: a `custom` mask configured to mask zero characters (`mask_begin` equal to `mask_end`, or `unmask_begin: 0`) is now rejected at construction rather than silently passing values through unmasked ([PR #1083](https://github.com/xataio/pgstream/pull/1083)) — use `noop` to pass a column through on purpose.

## Configurable Connection Pool Limits

[PR #1048](https://github.com/xataio/pgstream/pull/1048), contributed by [@subotac](https://github.com/subotac) in their first contribution, with [PR #1092](https://github.com/xataio/pgstream/pull/1092) by [@kvch](https://github.com/kvch).

`target.postgres.max_connections` (`PGSTREAM_POSTGRES_WRITER_MAX_CONNECTIONS`) controls the writer pool size, defaulting to 50 and honoring `pool_max_conns` from the target URL when set explicitly.

Previously this setting also sized the schema observer's pool, so a target capped at 90 connections to stay under a 100-connection server limit could still open up to 180. The schema observer is now sized separately, at `min(writer pool, 16)` — its catalog lookups are a startup and DDL burst rather than sustained COPY concurrency — so the process opens at most the configured writer limit plus 16.

## Webhook Delivery

[PR #1080](https://github.com/xataio/pgstream/pull/1080), by [@kvch](https://github.com/kvch).

A failed webhook POST is now retried with configurable backoff (network errors, `429`, `5xx`); other non-2xx responses are treated as permanent and not retried. A delivery that is still failing after retries is no longer checkpointed away — it is logged and dropped so one broken subscriber cannot block delivery to everyone else, and at-least-once semantics apply on restart. New `X-Pgstream-LSN` and `Idempotency-Key` headers let subscribers dedupe redelivered events (both omitted for snapshot events, which share a zero LSN). `DISABLE_RETRIES` now actually disables retries.

## `CREATEROLE` Preflight Check

[PR #1055](https://github.com/xataio/pgstream/pull/1055), contributed by [@lghuy05](https://github.com/lghuy05).

`pgstream check` verifies that the target connection role has `CREATEROLE` (or is a superuser) when `roles_snapshot_mode` is `enabled` or `no_passwords`, catching a missing privilege before the restore fails.

## Bug Fixes

* **`RENAME COLUMN` and `DROP COLUMN` are no longer silently dropped** ([PR #1078](https://github.com/xataio/pgstream/pull/1078)). The regex used to extract column names matched the trailing statement terminator as part of a bare identifier, so `RENAME COLUMN pii_email TO email;` was parsed as renaming to `email;` — the lookup then missed and the DDL event produced an empty schema diff, dropping the change from replication.

## Documentation

* Transformer uniqueness and the unique index check are documented ([PR #1088](https://github.com/xataio/pgstream/pull/1088)).
* The destructive `--reset` snapshot behaviour is documented ([PR #1059](https://github.com/xataio/pgstream/pull/1059)).

## New Contributors

* [@pauliyobo](https://github.com/pauliyobo) in [PR #1008](https://github.com/xataio/pgstream/pull/1008)
* [@subotac](https://github.com/subotac) in [PR #1048](https://github.com/xataio/pgstream/pull/1048)

## Upgrade

⚠️ **Breaking: Postgres-to-Postgres pipelines with a lossy transformer on a unique-indexed column now fail at startup.** This is the intended outcome of the new uniqueness check — it replaces a mid-load `duplicate key value violates unique constraint` failure with an immediate, actionable one — but existing configurations that rely on this combination need either a uniqueness-preserving transformer or `allow_uniqueness_loss: true` on the affected column before upgrading.

⚠️ **Breaking: a zero-width `custom` mask (`masking`) is now rejected at startup.** If you have a `mask_begin`/`mask_end` pair or `unmask_begin: 0` that masks no characters, switch to `noop` or a real mask range.

**Connection pools.** No action needed for most setups. If you were relying on `max_connections` to size the schema observer's pool as well as the writer's, the observer is now capped at `min(writer pool, 16)` regardless — reduce the writer setting if you need a lower total connection count.

## Support

* Issues: [https://github.com/xataio/pgstream/issues](https://github.com/xataio/pgstream/issues)
* Documentation: [https://github.com/xataio/pgstream/tree/main/docs](https://github.com/xataio/pgstream/tree/main/docs)
* Discussions: [https://github.com/xataio/pgstream/discussions](https://github.com/xataio/pgstream/discussions)


# Snapshots
Source: https://xata.io/docs/opensource/pgstream/docs/snapshots

![snapshots diagram](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_snapshot_diagram.svg)

`pgstream` supports the generation of PostgreSQL schema and data snapshots. It can be done as an initial step before starting the replication, or as a standalone mode, where a snapshot of the database is performed without any replication.

The snapshot behaviour is the same in both cases, with the only difference that if we're listening on the replication slot, we will store the current LSN before performing the snapshot, so that we can replay any operations that happened while the snapshot was ongoing.

The snapshot implementation is different for schema and data.

* Schema: it relies on `pg_dump` to produce the dump of the schema to be snapshotted. For Postgres targets it relies on `pg_restore` for restoring the schema, while for other targets it emits DDL events into the WAL pipeline to be processed.

* Data: it relies on transaction snapshot ids to obtain a stable view of the database tables, and parallelises the read of all the rows by dividing them into ranges using the `ctid`.

![snapshots sequence](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_snapshot_sequence.svg)

## ⚠️ The data snapshot source must be a single instance

The data snapshot exports a transaction snapshot on one connection and imports it
(`SET TRANSACTION SNAPSHOT`) on the parallel worker connections to give every worker the
same stable view. An exported snapshot is **instance-local**, so this only works when every
connection reaches the *same* Postgres instance.

Do **not** point the snapshot source at a load-balanced endpoint that spans multiple
instances, such as:

* an Amazon **Aurora reader (`cluster-ro`) endpoint** or an **RDS reader endpoint**,
* a connection **pooler that spreads connections across multiple database instances**.

Against such an endpoint the worker connections can land on a different instance than the
one that exported the snapshot, and the snapshot fails — nondeterministically, depending on
how connections happen to be routed — with:

```
setting transaction snapshot: relation does not exist: snapshot "…" does not exist
```

Use a **single-instance / writer endpoint** for the snapshot source. A connection pooler in
front of a *single* instance (e.g. RDS Proxy to one instance, or pgbouncer to one server) is
fine.

> `pgstream check` includes a preflight check that probes the source and reports a clear
> error when it detects a load-balanced source, so you can catch this before a snapshot runs.

To keep snapshot load off a primary that cannot afford it, the snapshot can be taken from a physical read replica instead. See [Running pgstream from a read replica](/docs/opensource/pgstream/docs/replicas).

## ⚠️ Resetting the target destroys it before the new data lands

`clean_target_db` (the `--reset` flag, `PGSTREAM_POSTGRES_SNAPSHOT_CLEAN_TARGET_DB`) drops the
in-scope target objects after the source schema is dumped but **before** the table data is
copied, and the restore is neither staged nor transactional.

So anything that fails after that point — a load-balanced source (see above), a transient
error, an interrupted run — leaves the target with the new schema and incomplete or no data,
and the previous contents are not recoverable. A scheduled snapshot re-drops the target on
every retry. Treat a non-zero exit as "the target is now unusable", not "nothing happened".

To keep a good copy, snapshot into a **new, empty database** (`create_target_db: true`, target
URL naming it) and repoint consumers only once the run has succeeded. An empty target has
nothing to drop, so `clean_target_db` isn't needed there.

For more details into the snapshot implementation and performance benchmarking, check out this [blogpost](https://xata.io/blog/behind-the-scenes-speeding-up-pgstream-snapshots-for-postgresql). For details on how to use and configure the snapshot mode, check the [snapshot tutorial](/docs/opensource/pgstream/docs/tutorials/postgres_snapshot).


# Transformers
Source: https://xata.io/docs/opensource/pgstream/docs/transformers

![transformer diagram](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_transformer_diagram.svg)

pgstream supports column value transformations to anonymize or mask sensitive data during replication and snapshots. This is particularly useful for compliance with data privacy regulations.

pgstream integrates with existing transformer open source libraries, such as [greenmask](https://github.com/GreenmaskIO/greenmask), [neosync](https://github.com/nucleuscloud/neosync) and [go-masker](https://github.com/ggwhite/go-masker), to leverage a large amount of transformation capabilities, as well as having support for custom transformations.

⚠️ Most transformers do **not** preserve uniqueness. Applying one to a column covered by a unique index or primary key produces duplicate key violations. See [Uniqueness and unique indexes](#uniqueness-and-unique-indexes).

## Uniqueness and unique indexes

Anonymization is lossy by design, and that conflicts directly with unique constraints. Masking `609898123456` with `masking` type `id` yields `609898****`: the transformer keeps a 6 character prefix, so **every** source value sharing that prefix becomes the same masked value. On a column covered by a unique index the load then fails with `duplicate key value violates unique constraint`, part way through the data, long after the run started.

Each transformer declares how it behaves with respect to uniqueness:

| Uniqueness | Meaning | Validation |
| - | - | - |
| `preserved` | Distinct inputs always produce distinct outputs. Safe on unique columns. | passes |
| `not_guaranteed` | Output is random, hashed, or driven by user supplied logic. Duplicates are possible, at the birthday bound of the output space (which usually depends on parameters). | warns |
| `lossy` | Distinct inputs are mapped to the same output by construction: partial masks, fixed literals, small value sets, name dictionaries. | errors |

| Transformer | Uniqueness |
| - | - |
| `encrypted_aes_siv` | `preserved` |
| `fpe_ff1` | `preserved` |
| `email` | `not_guaranteed` |
| `greenmask_date` | `not_guaranteed` |
| `greenmask_float` | `not_guaranteed` |
| `greenmask_integer` | `not_guaranteed` |
| `greenmask_string` | `not_guaranteed` |
| `greenmask_unix_timestamp` | `not_guaranteed` |
| `greenmask_utc_timestamp` | `not_guaranteed` |
| `greenmask_uuid` | `not_guaranteed` |
| `hstore` | `not_guaranteed` |
| `json` | `not_guaranteed` |
| `neosync_email` | `not_guaranteed` |
| `neosync_string` | `not_guaranteed` |
| `pg_anonymizer` | `not_guaranteed` |
| `phone_number` | `not_guaranteed` |
| `string` | `not_guaranteed` |
| `template` | `not_guaranteed` |
| `greenmask_boolean` | `lossy` |
| `greenmask_choice` | `lossy` |
| `greenmask_firstname` | `lossy` |
| `lookup_choice` | `lossy` |
| `literal_string` | `lossy` |
| `masking` | `lossy` |
| `neosync_firstname` | `lossy` |
| `neosync_fullname` | `lossy` |
| `neosync_lastname` | `lossy` |

On an array column the classification describes the whole array transform, which depends on the configured generator. See [Array columns](#array-columns).

When a source Postgres URL is configured, pgstream reads the unique indexes, unique constraints and primary keys of every table in the transformation rules and checks them against the configured transformers. Columns with no rule, or with a `noop` rule, keep their original value and are never flagged. Run the check on its own with:

```sh theme={null}
pgstream validate rules -c pg2pg.yaml
```

With a **Postgres target**, a `lossy` transformer on a covered column fails the rules with `transformation rules break a unique index`, because the target recreates the source's indexes and the load would hit a duplicate key. With any **other target** (Kafka, Elasticsearch/OpenSearch, webhooks) there is no unique index to violate, so the same finding is reported as a warning and does not block the pipeline. A `not_guaranteed` transformer is always a warning.

To keep a `lossy` transformer on a covered column anyway — because the target does not enforce that index, or because the values are known not to collide — set `allow_uniqueness_loss` on that column rule:

```yaml theme={null}
column_transformers:
  pms_patient_id:
    name: masking
    parameters:
      type: id
    allow_uniqueness_loss: true
```

### Keeping a column unique

If you need an anonymized column to stay unique, use `encrypted_aes_siv`: it is deterministic, so equality relationships survive across rows, tables and runs, and no two distinct inputs share a token. Its output is a base64 token rather than a value in the original format.

Use `fpe_ff1` when the column must also keep its format. For example, a phone number must stay a phone number, and a code must still fit a `varchar(12)` column. `fpe_ff1` encrypts with a format-preserving algorithm. The output has the same length and the same characters as the input. Two different inputs always give two different outputs.

⚠️ `encrypted_aes_siv` and `fpe_ff1` are **pseudonymization, not anonymization**. The output is reversible by anyone holding `key_hex`, so it remains personal data, and because tokens are stable they can be correlated across tables and across successive dumps. Treat `key_hex` as a secret: inject it from a secret store rather than committing it in the rules file, use a different key per environment, and set `associated_data` (for example `schema.table.column`) so tokens cannot be correlated between columns.

### What the check does not cover

* **Expression indexes.** A unique index over an expression, such as `lower(email)`, cannot be resolved to a column, so pgstream reports a warning naming the index and leaves it to you to verify.
* **Target-only indexes.** The check reads the *source* catalog. A unique index that exists only on the target is not seen.
* **Indexes created after startup.** Rules are validated once when the pipeline starts. A unique index added to the source later is not re-checked.
* **Exclusion constraints** with equality semantics (`EXCLUDE (email WITH =)`) are not treated as unique indexes.

If a load fails with `duplicate key value violates unique constraint` on a transformed column, run `pgstream validate rules` against the source to see which rules the check flags.

## Array columns

A transformation rule on a one-dimensional array column applies the transformer to each element. The rule reads like a rule on a scalar column:

```yaml theme={null}
column_transformers:
  emails: # emails is varchar(500)[]
    name: email
    parameters:
      replacement_domain: "@example.com.invalid"
```

pgstream decodes the array before it applies the transformer, so each element reaches the transformer as the value of the element type. An `int4[]` element reaches the transformer as an integer, not as text. The same rules apply to `text` and to `text[]`: a transformer accepts an array column if it accepts the element type. This covers extension types, so a transformer that accepts `citext` also accepts `citext[]`.

Array columns need a source PostgreSQL connection, because pgstream reads the column type from the source catalog.

### Generators

The optional `array_options` block selects how the elements of the new array are produced:

```yaml theme={null}
column_transformers:
  emails:
    name: email
    array_options:
      generator: random
      min_count: 0
      max_count: 12
```

| Generator | Behavior |
| - | - |
| `map` | Applies the transformer to each source element, in order. The array keeps its length. This is the default. |
| `random` | Emits between `min_count` and `max_count` elements. Each element is the transform of a source element selected at random, with repeats. |

`min_count` and `max_count` are required with the `random` generator. They must be non-negative, `min_count` must not be greater than `max_count`, and `max_count` must not be greater than 10000. The limit protects against a mistyped value, which pgstream would otherwise apply to every row. These parameters are not valid with the `map` generator. pgstream rejects the rules at startup if these conditions are not met, and names the schema, table and column.

The `random` generator never invents a value. Each new element is the transform of a value that is in the row. An empty source array stays empty.

### NULL values and errors

* A NULL array stays NULL. pgstream does not call the transformer.
* A NULL element stays NULL. pgstream does not call the transformer for that element.
* A transformer that returns no value produces a NULL element in the same position. The array keeps its length.
* The first element that fails makes the whole column fail. The `on_error` policy then applies to the whole column: `fail` stops the run, `null` sets the whole column to NULL, and `pass-through` restores the whole original array.

### Limitations

* **Multi-dimensional arrays are not supported.** pgstream rejects a rule at startup when the column is declared with more than one dimension. PostgreSQL does not enforce the declared number of dimensions, so a column declared as one-dimensional can still hold a multi-dimensional value. On the replication path this value is a per-row error, which the `on_error` policy handles. On the snapshot path pgstream cannot detect it, because the source driver returns the elements already flattened, and it writes a one-dimensional array to the target.
* **`literal_string` and `pg_anonymizer` write into each element.** On an array column these transformers write into every element instead of into the column as a whole. This is different from the behaviour before pgstream supported array columns. `pgstream validate rules` reports a warning for each of these rules.
* **`pg_anonymizer` sends one query for each element.** A wide array multiplies the queries that pgstream sends to the source database for the row. `pgstream validate rules` reports a warning for each rule that uses this transformer on an array column.
* **Dynamic parameters are not paired element by element.** A `dynamic_parameters` sibling that is itself an array falls back to the parameter default.

### Uniqueness

The uniqueness check reads the classification of the whole array transform, not of the transformer the rule names:

* With `map`, the array keeps the classification of the transformer. `fpe_ff1` on a `text[]` column stays `preserved`, because mapping a transformer that preserves uniqueness over an array of the same length also preserves uniqueness.
* With `random`, the array is always `lossy`, whatever the transformer guarantees. Two different source arrays that share an element can produce the same output, and `min_count: 0` lets any row produce an empty array. On a PostgreSQL target this is an error. Set `allow_uniqueness_loss` on the column to override it.

## Supported transformers

### PostgreSQL Anonymizer

#### pg\_anonymizer

**Description:** Integrates with the [PostgreSQL Anonymizer](https://postgresql-anonymizer.readthedocs.io/en/stable/) extension to provide advanced data anonymization using built-in anonymizer functions.

⚠️ This transformer requires a PostgreSQL database connection to execute transformations, which may impact performance compared to other transformers in this document that generate values locally without database queries.

| Supported PostgreSQL types |
| - |
| Dependent on [anonymizer function](https://postgresql-anonymizer.readthedocs.io/en/stable/masking_functions/) |

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| anon\_function | string | N/A | Yes | Any valid anon.\* function |
| postgres\_url | string | N/A | Yes | PostgreSQL connection URL |
| salt | string | "" | No | Salt for deterministic functions |
| hash\_algorithm | string | sha256 | No | Algorithm for anon.digest. One of md5, sha224, sha256, sha384, sha512 |
| interval | string | N/A | No | Time interval for anon.dnoise function |
| ratio | float | N/A | No | Noise ratio for anon.noise function |
| sigma | float | N/A | No | Blur sigma for anon.image\_blur function |
| mask | string | N/A | No | Mask character for anon.partial function |
| mask\_prefix\_count | int | 0 | No | Prefix count for anon.partial function |
| mask\_suffix\_count | int | 0 | No | Suffix count for anon.partial function |
| min | string | N/A | No | Minimum value for anon.random\_\*\_between functions |
| max | string | N/A | No | Maximum value for anon.random\_\*\_between functions |
| range | string | "" | No | Range for anon.random\_in\_\* functions |
| locale | string | en\_US | No | Locale for dummy supported functions. One of ar\_SA, en\_US, fr\_FR, ja\_JP, pt\_BR, zh\_CN, zh\_TW |
| count | int | 0 | No | Count parameter for functions like anon.random\_string and anon.lorem\_ipsum |
| unit | string | paragraphs | No | Unit for anon.lorem\_ipsum function. One of characters, words, paragraphs |
| prefix | string | "" | No | Prefix for anon.random\_phone function |

Notes:

* The transformer executes functions directly in PostgreSQL, ensuring compatibility with all anonymizer features
* Deterministic functions (pseudo\_\*, hash, digest) produce consistent output for the same input
* Functions that don't require parameters (like anon.fake\_\*()) can be used without additional configuration

**Prerequisites:**

* PostgreSQL Anonymizer extension must be installed and enabled on the source (or the configured url)
* Extension must be loaded in `shared_preload_libraries`
* Run `SELECT anon.init();`in order to use the faking functions

**Supported Functions:**

* [Adding noise](https://postgresql-anonymizer.readthedocs.io/en/stable/masking_functions/#adding-noise)
* [Randomization](https://postgresql-anonymizer.readthedocs.io/en/stable/masking_functions/#randomization)
* [Faking](https://postgresql-anonymizer.readthedocs.io/en/stable/masking_functions/#faking)
* [Advanced faking](https://postgresql-anonymizer.readthedocs.io/en/stable/masking_functions/#advanced-faking)
* [Pseudoanonymization](https://postgresql-anonymizer.readthedocs.io/en/stable/masking_functions/#pseudonymization)
* [Generic hashing](https://postgresql-anonymizer.readthedocs.io/en/stable/masking_functions/#generic-hashing)
* [Partial scrambling](https://postgresql-anonymizer.readthedocs.io/en/stable/masking_functions/#partial-scrambling)
* [Image blurring](https://postgresql-anonymizer.readthedocs.io/en/stable/masking_functions/#image-bluring)

**Unsupported Functions:**

* [Conditional masking](https://postgresql-anonymizer.readthedocs.io/en/stable/masking_functions/#conditional-masking)
* [Generalization](https://postgresql-anonymizer.readthedocs.io/en/stable/masking_functions/#generalization)

**Example Configurations:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        first_name:
          name: pg_anonymizer
          parameters:
            anon_function: anon.fake_first_name()
        id:
          name: pg_anonymizer
          parameters:
            anon_function: anon.digest
            salt: salt
            hash_algorithm: md5
        phone:
          name: pg_anonymizer
          parameters:
            anon_function: anon.random_phone
            prefix: "+1-555-"
        api_key:
          name: pg_anonymizer
          parameters:
            anon_function: anon.random_string
            count: 32
        content:
          name: pg_anonymizer
          parameters:
            anon_function: anon.lorem_ipsum
            unit: "words"
            count: 50
        status:
          name: pg_anonymizer
          parameters:
            anon_function: anon.random_in
            range: "ARRAY['active', 'inactive', 'pending']"
```

**Input-Output Examples:**

| Input Value | Function Configuration | Output Value |
| - | - | - |
| `John` | `anon_function: anon.fake_first_name()` | `Michael` (random) |
| `john@test.com` | `anon_function: anon.pseudo_email, salt: "key123"` | `alice@test.com` (deterministic) |
| `1234567890` | `anon_function: anon.partial, mask: "*", mask_prefix_count: 3, mask_suffix_count: 3` | `123****890` |
| `sensitive_data` | `anon_function: anon.digest, salt: "key", hash_algorithm: "sha256"` | `a1b2c3d4e5f6...` (hash) |
| `100.50` | `anon_function: anon.noise, ratio: 0.1` | `95.23` (with 10% noise) |
| `2023-01-15` | `anon_function: anon.dnoise, interval: "1 day"` | `2023-01-16` (±1 day noise) |
| `password123` | `anon_function: anon.hash` | `ef92b778bafe771e89245b89ecbc08a4...` |
| `Alice Smith` | `anon_function: anon.pseudo_first_name, salt: "s1"` | `Bob Smith` (deterministic) |
| `user@company.com` | `anon_function: anon.partial_email` | `u***@company.com` |
| `42` | `anon_function: anon.random_int_between(1, 100)` | `73` (random between 1-100) |
| `/path/image.jpg` | `anon_function: anon.image_blur, sigma: 2.5` | Blurred image data |
| Any value | `anon_function: anon.fake_company()` | `Acme Corporation` (random) |
| `25` | `anon_function: anon.random_int_between, min: "18", max: "65"` | `42` (random between 18-65) |
| Any value | `anon_function: anon.random_in, range: "ARRAY['A', 'B', 'C']"` | `B` (random from array) |
| Any value | `anon_function: anon.lorem_ipsum, unit: "words", count: 5` | `Lorem ipsum dolor sit amet` |
| Any value | `anon_function: anon.random_string, count: 8` | `aB3xY9z1` (random string) |
| Any value | `anon_function: anon.random_phone, prefix: "+1-555-"` | `+1-555-123-4567` |
| `John` | `anon_function: anon.fake_first_name_locale, locale: "fr_FR"` | `Pierre` (French name) |

### Greenmask

#### greenmask\_boolean

**Description:** Generates random or deterministic boolean values (`true` or `false`).

**Uniqueness:** `lossy`. The output space has two values. Cannot be used on a column covered by a unique index. See [Uniqueness and unique indexes](#uniqueness-and-unique-indexes).

| Supported PostgreSQL types |
| - |
| `boolean` |

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| generator | string | random | No | random,deterministic |

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        is_active:
          name: greenmask_boolean
          parameters:
            generator: deterministic
```

**Input-Output Examples:**

| Input Value | Configuration Parameters | Output Value |
| - | - | - |
| `true` | `generator: deterministic` | `false` |
| `false` | `generator: deterministic` | `true` |
| `true` | `generator: random` | `true` or `false` (random) |

#### greenmask\_choice

**Description:** Randomly selects a value from a predefined list of choices.

**Uniqueness:** `lossy`. Any table with more rows than choices produces duplicates. Cannot be used on a column covered by a unique index. See [Uniqueness and unique indexes](#uniqueness-and-unique-indexes).

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar`, user-defined enum, and arrays of these |

`choices` is optional for an enum column. If you do not set it, pgstream uses the labels of the enum. If you set it, pgstream compares each value with those labels. A wrong value stops the run at startup, not at each insert.

A domain over an enum resolves to the enum. An array of an enum resolves to the enum too, so an array column receives the same choices and transforms each element.

`greenmask_choice` is the only type-specific transformer for an enum column. It is the only one that you can limit to the values of the enum. `literal_string` and `pg_anonymizer` accept all types. They also apply to an enum column. For `literal_string`, use a literal that is a valid label.

The default choices have three limits:

* **A source Postgres URL is necessary.** Without it, pgstream does not validate the rules against a catalog. A rule without `choices` then stops the run at startup with the message `greenmask_choice: choices must not be empty`. Set `choices` for a pipeline that has no Postgres source, for example a pipeline with a Kafka source.
* **pgstream reads the labels one time, at startup.** If you add or rename a label on the source, pgstream uses the new label only after a restart. Before the restart, a replicated `ALTER TYPE ... RENAME VALUE` makes pgstream write a label that the target refuses. At startup, pgstream writes the labels in use to the log for each column with default choices.
* **The default is the full set of labels.** pgstream does not read a `CHECK` constraint on a domain or on a table. It does not apply such a constraint. Set `choices` for a column with a `CHECK` constraint.

⚠️ The transformer can select the label that the source row already contains. It selects from all the choices and does not remove the source value from the set. With `generator: random`, approximately 1 row in N keeps its source value, where N is the number of choices. With `generator: deterministic`, each label maps to one fixed label, so a label that maps to itself keeps its value in every row. The transformers that select from a fixed dictionary, for example `greenmask_firstname`, operate in the same way.

⚠️ Use `generator: random` for an enum column. The `deterministic` generator maps each label to one fixed label and uses no secret key. The target contains all labels of the enum. A person who reads the target can find the source label of each value. pgstream writes a warning to the log when a rule uses `deterministic` with default enum choices.

ℹ️ The transformer writes the value as a string. This is a change for the Kafka, webhook, Elasticsearch and OpenSearch targets. Before this change, `greenmask_choice` wrote a byte array, and these targets encoded that byte array as base64 in JSON. Update the consumers that decode base64. A search index also contains base64 in the documents from before this change.

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| generator | string | random | No | random,deterministic |
| choices | string\[] | the enum's labels, if an enum | Yes, unless an enum column | N/A |

`transformers-definition.json` shows `choices` as always required. This file describes the transformer, not the column that you configure it on.

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: orders
      column_transformers:
        status:
          name: greenmask_choice
          parameters:
            generator: random
            choices: ["pending", "shipped", "delivered", "cancelled"]
        # an enum column does not need choices; pgstream uses the enum labels
        mood:
          name: greenmask_choice
```

**Input-Output Examples:**

| Input Value | Configuration Parameters | Output Value |
| - | - | - |
| `pending` | `generator: random` | `shipped` (random) |
| `shipped` | `generator: deterministic` | `pending` |
| `delivered` | `generator: random` | `cancelled` (random) |

#### greenmask\_date

**Description:** Generates random or deterministic dates within a specified range.

| Supported PostgreSQL types |
| - |
| `date`, `timestamp`, `timestamptz` |

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| generator | string | random | No | random,deterministic |
| min\_value | string (`yyyy-MM-dd`) | N/A | Yes | N/A |
| max\_value | string (`yyyy-MM-dd`) | N/A | Yes | N/A |

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: events
      column_transformers:
        event_date:
          name: greenmask_date
          parameters:
            generator: random
            min_value: "2020-01-01"
            max_value: "2025-12-31"
```

**Input-Output Examples:**

| Input Value | Configuration Parameters | Output Value |
| - | - | - |
| `2023-01-01` | `generator: random, min_value: 2020-01-01, max_value: 2025-12-31` | `2021-05-15` (random) |
| `2022-06-15` | `generator: deterministic` | `2020-01-01` |

#### greenmask\_firstname

**Description:** Generates random or deterministic first names, optionally filtered by gender.

**Uniqueness:** `lossy`. Names come from a fixed dictionary and repeat well before a table of any size is exhausted. Cannot be used on a column covered by a unique index. See [Uniqueness and unique indexes](#uniqueness-and-unique-indexes).

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar` |

| Parameter | Type | Default | Required | Values | Dynamic |
| - | - | - | - | - | - |
| generator | string | random | No | random,deterministic | No |
| gender | string | Any | No | Any,Female,Male | Yes |

`gender` can also be a dynamic parameter, referring to some other column. Please see the below example config.

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: employees
      column_transformers:
        first_name:
          name: greenmask_firstname
          parameters:
            generator: deterministic
          dynamic_parameters:
            gender:
              column: sex
```

**Input-Output Examples:**

| Input Name | Configuration Parameters | Output Name |
| - | - | - |
| `John` | `preserve_gender: true` | `Michael` |
| `Jane` | `preserve_gender: true` | `Emily` |
| `Alex` | `preserve_gender: false` | `Jordan` |
| `Chris` | `generator: random` | `Taylor` |

#### greenmask\_float

**Description:** Generates random or deterministic floating-point numbers within a specified range.

| Supported PostgreSQL types |
| - |
| `real`, `double precision`, `numeric` |

⚠️ Set `min_value` and `max_value` for a `numeric` column. These parameters are required there. The default range covers all `float32` values. With the default range, the transformer writes the same constant value in each row.

The transformer converts the `numeric` value to a `float64` and uses that `float64` as the seed for the generator. The output is also a `float64`. If the source value has more digits than a `float64` holds, the transformer rounds it. If the source value is outside the `float64` range, the transformer uses the largest `float64` value. These changes apply to the seed only, because the transformer discards the source value.

pgstream compares the range with the column when it validates the rules. If the range does not fit a `numeric(p,s)` column, the run stops. A `numeric` column without a precision holds any value. For such a column, pgstream checks only that you set both bounds.

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| generator | string | random | No | random,deterministic |
| min\_value | float | -3.40282346638528859811704183484516925440e+38 | No | N/A |
| max\_value | float | 3.40282346638528859811704183484516925440e+38 | No | N/A |

#### greenmask\_integer

**Description:** Generates random or deterministic integers within a specified range.

| Supported PostgreSQL types |
| - |
| `smallint`, `integer`, `bigint`, `real`, `double precision`, `numeric` |

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| generator | string | random | No | random,deterministic |
| size | int | 4 | No | 2,4 |
| min\_value | int | -2147483648 | No | N/A |
| max\_value | int | 2147483647 | No | N/A |

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: products
      column_transformers:
        stock_quantity:
          name: greenmask_integer
          parameters:
            generator: random
            min_value: 1
            max_value: 1000
```

#### greenmask\_string

**Description:** Generates random or deterministic strings with customizable length and character set.

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar` |

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| generator | string | random | No | random,deterministic |
| symbols | string | abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890 | No | N/A |
| min\_length | int | 1 | No | N/A |
| max\_length | int | 100 | No | N/A |

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        username:
          name: greenmask_string
          parameters:
            generator: random
            min_length: 5
            max_length: 15
            symbols: "abcdefghijklmnopqrstuvwxyz1234567890"
```

#### greenmask\_unix\_timestamp

**Description:** Generates random or deterministic unix timestamps.

| Supported PostgreSQL types |
| - |
| `smallint`, `integer`, `bigint`, `real`, `double precision`, `numeric` |

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| generator | string | random | No | random,deterministic |
| min\_value | string | N/A | Yes | N/A |
| max\_value | string | N/A | Yes | N/A |

#### greenmask\_utc\_timestamp

**Description:** Generates random or deterministic UTC timestamps.

| Supported PostgreSQL types |
| - |
| `timestamp` |

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| generator | string | random | No | random,deterministic |
| truncate\_part | string | "" | No | nanosecond,microsecond,millisecond,second,minute,hour,day,month,year |
| min\_timestamp | string (`RFC3339`) | N/A | Yes | N/A |
| max\_timestamp | string (`RFC3339`) | N/A | Yes | N/A |

#### greenmask\_uuid

**Description:** Generates random or deterministic UUIDs.

| Supported PostgreSQL types |
| - |
| `uuid`,`text`, `varchar`, `char`, `bpchar` |

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| generator | string | random | No | random,deterministic |

### Neosync

#### neosync\_email

**Description:** Anonymizes email addresses while optionally preserving length and domain.

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar`, `citext` |

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| preserve\_length | bool | false | No | |
| preserve\_domain | bool | false | No | |
| excluded\_domains | string\[] | N/A | No | |
| max\_length | int | 100 | No | |
| email\_type | string | uuidv4 | No | uuidv4,fullname,any |
| invalid\_email\_action | string | 100 | No | reject,passthrough,null,generate |
| seed | int | Rand | No | |

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: customers
      column_transformers:
        email:
          name: neosync_email
          parameters:
            preserve_length: true
            preserve_domain: true
```

**Input-Output Examples:**

| Input Email | Configuration Parameters | Output Email |
| - | - | - |
| `john.doe@example.com` | `preserve_length: true, preserve_domain: true` | `abcd.efg@example.com` |
| `jane.doe@company.org` | `preserve_length: false, preserve_domain: true` | `random@company.org` |
| `user123@gmail.com` | `preserve_length: true, preserve_domain: false` | `abcde123@random.com` |
| `invalid-email` | `invalid_email_action: passthrough` | `invalid-email` |
| `invalid-email` | `invalid_email_action: null` | `NULL` |
| `invalid-email` | `invalid_email_action: generate` | `generated@random.com` |

#### neosync\_firstname

**Description:** Generates anonymized first names while optionally preserving length.

**Uniqueness:** `lossy`. Names come from a fixed dictionary and repeat well before a table of any size is exhausted. Cannot be used on a column covered by a unique index. See [Uniqueness and unique indexes](#uniqueness-and-unique-indexes).

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar` |

| Parameter | Type | Default | Required |
| - | - | - | - |
| preserve\_length | bool | false | No |
| max\_length | int | 100 | No |
| seed | int | Rand | No |

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        first_name:
          name: neosync_firstname
          parameters:
            preserve_length: true
```

#### neosync\_lastname

**Description:** Generates anonymized last names while optionally preserving length.

**Uniqueness:** `lossy`. Names come from a fixed dictionary and repeat well before a table of any size is exhausted. Cannot be used on a column covered by a unique index. See [Uniqueness and unique indexes](#uniqueness-and-unique-indexes).

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar` |

| Parameter | Type | Default | Required |
| - | - | - | - |
| preserve\_length | bool | false | No |
| max\_length | int | 100 | No |
| seed | int | Rand | No |

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        last_name:
          name: neosync_lastname
          parameters:
            preserve_length: true
```

#### neosync\_fullname

**Description:** Generates anonymized full names while optionally preserving length.

**Uniqueness:** `lossy`. Names come from fixed dictionaries, so even first/last combinations repeat well before a large table is exhausted. Cannot be used on a column covered by a unique index. See [Uniqueness and unique indexes](#uniqueness-and-unique-indexes).

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar` |

| Parameter | Type | Default | Required |
| - | - | - | - |
| preserve\_length | bool | false | No |
| max\_length | int | 100 | No |
| seed | int | Rand | No |

max\_length must be greater than 2. If preserve\_length is set to true, generated value can might be longer than max\_length, depending on the input length.

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        full_name:
          name: neosync_fullname
          parameters:
            preserve_length: true
```

#### neosync\_string

**Description:** Generates anonymized strings with customizable length.

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar` |

| Parameter | Type | Default | Required |
| - | - | - | - |
| preserve\_length | bool | false | No |
| min\_length | int | 1 | No |
| max\_length | int | 100 | No |
| seed | int | Rand | No |

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: logs
      column_transformers:
        log_message:
          name: neosync_string
          parameters:
            min_length: 10
            max_length: 50
```

### Xata

#### template

**Description:** Transforms the data using go templates

| Supported PostgreSQL types |
| - |
| All types with a string representation |

| Parameter | Type | Default | Required |
| - | - | - | - |
| template | string | N/A | Yes |

This transformer can be used for any Postgres type as long as the given template produces a value with correct syntax for that column type. e.g It can be "5-10-2021" for a date column, or "3.14159265" for a double precision one.

`.GetValue` renders the same text from a snapshot and from replication. During a snapshot the value is encoded as Postgres text for the column type, so a numeric renders as decimal text, a uuid as `a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11`, a date as `2024-02-29`, a bytea as `\xdeadbeef`, an array as `{a,b}`, and so on. Values from other columns, read with `.GetDynamicValue`, keep the Go `time.Time` type during a snapshot so the sprig date functions can format them.

Template transformer supports a bunch of useful functions. Use `.GetValue` to refer to the value to be transformed. Use `.GetDynamicValue "<column_name>"` to refer to some other column. Other than the standard go template functions, there are many useful helper functions supported to be used with template transformer, thanks to `greenmask`'s huge set of [core functions](https://docs.greenmask.io/latest/built_in_transformers/advanced_transformers/custom_functions/core_functions/) including `masking` function by `go-masker` and various [random data generator functions](https://docs.greenmask.io/latest/built_in_transformers/advanced_transformers/custom_functions/faker_function/) powered by the open source library `faker`. Also, template transformer has support for the open source library [sprig](https://masterminds.github.io/sprig/) which has many useful helper functions.

With the below example config `pgstream` masks values in the column `email` of the table `users`, using `go-masker`'s email masking function. But first, this template checks if there's a non-empty value to be used in the column `email`. If not, it simply looks for another column named `secondary_email` and uses that instead. Then we have another check to see if it's a `@xata` email or not. Finally masking the value, only if it's not a `@xata` email, passing it without a mask otherwise.

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        email:
          name: template
          parameters:
            template: >
              {{ $email := "" }}
              {{- if and (ne .GetValue nil) (isString .GetValue) (gt (len .GetValue) 0) -}}
                {{ $email = .GetValue }}
              {{- else -}}
                {{ $email = .GetDynamicValue "secondary_email" }}
              {{- end -}}
              {{- if (not (contains "@xata" $email)) -}}
                {{ masking "email" $email }}
              {{- else -}}
                {{ $email }}
              {{- end -}}
```

#### masking

**Description:** Masks string values using the provided masking function.

**Uniqueness:** `lossy`. Every type replaces part of the value with `*`, so all values sharing the untouched part collapse to the same output — `type: id` keeps only a 6 character prefix. Cannot be used on a column covered by a unique index. See [Uniqueness and unique indexes](#uniqueness-and-unique-indexes).

A `custom` mask that would cover zero characters (for example `mask_begin: 3` with `mask_end: 3`, or `unmask_begin: 0`) is rejected at startup: it leaves values completely unmasked, which silently defeats anonymization. Use the `noop` transformer if passing a column through untouched is what you want.

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar` |

**Parameter Details:**

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| type | string | default | No | custom, password, name, address, email, mobile, tel, id, credit\_card, url, default |

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        email:
          name: masking
          parameters:
            type: email
```

**Input-Output Examples:**

| Input Value | Configuration Parameters | Output Value |
| - | - | - |
| `aVeryStrongPassword123` | `type: password` | `************` |
| `john.doe@example.com` | `type: email` | `joh****e@example.com` |
| `Sensitive Data` | `type: default` | `**************` |

With `custom` type, the masking function is defined by the user, by providing beginning and end indexes for masking. If the input is shorter than the end index, the rest of the string will all be masked. See the third example below.

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        email:
          name: masking
          parameters:
            type: custom
            mask_begin: "4"
            mask_end: "12"
```

**Input-Output Examples:**

| Input Value | Output Value |
| - | - |
| `1234567812345678` | `1234********5678` |
| `sensitive@example.com` | `sens********ample.com` |
| `sensitive` | `sens*****` |

If the begin index is not provided, it defaults to 0. If the end is not provided, it defaults to input length.

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        email:
          name: masking
          parameters:
            type: custom
            mask_end: "5"
```

| Input Value | Output Value |
| - | - |
| `1234567812345678` | `*****67812345678` |
| `sensitive@example.com` | `*****tive@example.com` |
| `sensitive` | `*****tive` |

Alternatively, since input length may vary, user can provide relative beginning and end indexes, as percentages of the input length.

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        email:
          name: masking
          parameters:
            type: custom
            mask_begin: "15%"
            mask_end: "85%"
```

| Input Value | Output Value |
| - | - |
| `1234567812345678` | `12***********678` |
| `sensitive@example.com` | `sen***************com` |
| `sensitive` | `s******ve` |

Alternatively, user can provide unmask begin and end indexes. In that case, the specified part of the input will remain unmasked, while all the rest is masked.
Mask and unmask parameters cannot be provided at the same time.

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        email:
          name: masking
          parameters:
            type: custom
            unmask_end: "3"
```

| Input Value | Output Value |
| - | - |
| `1234567812345678` | `123*************` |
| `sensitive@example.com` | `sen******************` |
| `sensitive` | `sen******` |

#### json

**Description:** Transforms json data with set and delete operations

| Supported PostgreSQL types |
| - |
| json, jsonb |

| Parameter | Type | Default | Required |
| - | - | - | - |
| operations | array | N/A | Yes |

Parameter for each operation:

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| operation | string | N/A | Yes | set, delete |
| path | string | N/A | Yes | sjson syntax<sup>\*</sup> |
| skip\_not\_exist | boolean | true | No | true, false |
| error\_not\_exist | boolean | false | No | true, false |
| value | string | N/A | Yes<sup>\*\*</sup> | Any valid JSON representation |
| value\_template | string | N/A | Yes<sup>\*\*</sup> | Any template with valid syntax |

<sup>\*</sup>*Paths should follow [sjson syntax](https://github.com/tidwall/sjson#path-syntax)*

<sup>\*\*</sup>*Either `value` or `value_template` must be provided if the operation is `set`. If both are provided, `value_template` takes precedence.*

JSON transformer can be used for Postgres types json and jsonb. This transformer executes a list of given operations on the json data to be transformed.

All operations must be either `set` or `delete`.
`set` operations support literal values as well as templates, making use of `sprig` and `greenmask`'s function sets. See template transformer section for more details. Also, like the template transformer, `.GetValue` and `.GetDynamicValue` functions are supported. Unlike template transformer, here `.GetValue` refers to the value at given path, rather than the entire JSON object; whereas `.GetDynamicValue` is again used for referring to other columns.
`delete` operations simply delete the object at the given path.
Execution of an operation will be skipped if the given path does not exists and the parameter `skip_not_exist` is set to true, which is also the default behavior.
Execution of an operation errors out if the given path does not exists and the parameter `error_not_exist` - which is false by default - is set to true; unless the operation is skipped already.

JSON transformer uses [sjson](https://github.com/tidwall/sjson) library for executing the operations. Operation paths should follow the [synxtax rules of sjson](https://github.com/tidwall/sjson#path-syntax)

With the below config `pgstream` transforms the json values in the column `user_info_json` of the table `users` by:

* First, traversing all the items in the array named `purchases`, and for each element, setting value to "-" for key "item".
* Then, deleting the object named "country" under the top-level object "address".
* Completely masking the "city" value under object "address", using `go-masker`'s default masking function supported by `pgstream`'s templating.
* Finally, setting the user's lastname after fetching it from some other column named `lastname`, using dynamic values support. Assuming there's such column, having the lastname info for users.

Example input-output is given below the config.

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        user_info_json:
          name: json
          parameters:
            operations:
              - operation: set
                path: "purchases.#.item"
                value: "-"
                error_not_exist: true
              - operation: delete
                path: "address.country"
              - operation: set
                path: "address.city"
                value_template: '"{{ masking "default" .GetValue }}"'
              - operation: set
                path: "user.lastname"
                value_template: '"{{ .GetDynamicValue "lastname" }}"'
```

For input JSON value,

```json theme={null}
{
  "user": {
    "firstname": "john",
    "lastname": "unknown"
  },
  "residency": {
    "city": "some city",
    "country": "some country"
  },
  "purchases": [
    {
      "item": "book",
      "price": 10
    },
    {
      "item": "pen",
      "price": 2
    }
  ]
}
```

the JSON transformer with above config produces output:

```json theme={null}
{
  "user": {
    "firstname": "john",
    "lastname": "doe"
  },
  "residency": {
    "city": "*********"
  },
  "purchases": [
    {
      "item": "-",
      "price": 10
    },
    {
      "item": "-",
      "price": 2
    }
  ]
}
```

#### hstore

**Description:** Transforms hstore data with set and delete operations

| Supported PostgreSQL types |
| - |
| hstore |

| Parameter | Type | Default | Required |
| - | - | - | - |
| operations | array | N/A | Yes |

Parameter for each operation:

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| operation | string | N/A | Yes | set, delete |
| key | string | N/A | Yes | |
| skip\_not\_exist | boolean | true | No | true, false |
| error\_not\_exist | boolean | false | No | true, false |
| value | string, null | N/A | Yes<sup>\*</sup> | Any string or null |
| value\_template | string | N/A | Yes<sup>\*</sup> | Any template with valid syntax |

<sup>\*</sup>*Either `value` or `value_template` must be provided if the operation is `set`. If both are provided, `value_template` takes precedence.*

Hstore transformer can be used for Postgres type hstore. This transformer executes a list of given operations on the hstore data to be transformed.

All operations must be either `set` or `delete`.
`set` operations support literal values as well as templates, making use of `sprig` and `greenmask`'s function sets. See template transformer section for more details. Also, like the template transformer, `.GetValue` and `.GetDynamicValue` functions are supported. Unlike template transformer, here `.GetValue` refers to the value for the given key, rather than the entire Hstore object; whereas `.GetDynamicValue` is again used for referring to other columns.
`delete` operations simply delete the pair with the given key.
Execution of an operation will be skipped if the given key does not exists and the parameter `skip_not_exist` is set to true, which is also the default behavior.
Execution of an operation errors out if the given key does not exists and the parameter `error_not_exist` - which is false by default - is set to true; unless the operation is skipped already.

A limitation to be aware of: When using hstore transformer templates, you cannot set a value to the string literal "\<no value>". This is because Go templates produce "\<no value>" as output when the result is `nil`, creating an ambiguity. In such cases, `pgstream` will interpret it as `nil` and set the hstore value to `NULL` rather than storing the actual string "\<no value>".

With the below config `pgstream` transforms the hstore values in the column `attributes` of the table `users` by:

* First, updating the value for key "email" to the masked version of it, using email masking function. If the key "email" is not found, it simply ignores it, since `error_not_exist` is not set to true explicitly and it is false by default.
* Then, deleting the pair where the key is "public\_key". If there's no such key, errors out, because the parameter "error\_not\_exist" is set to true.
* Completely masking the value for key "private\_key", using `go-masker`'s default masking function supported by `pgstream`'s templating.
* Finally, updating the value for key "newKey" to "newValue". Since "error\_not\_exist" is false by default, and there is no such key in the example below, this operation will be done by adding a new key-value pair.

Example input-output is given below the config.

**Example Configuration:**

```yaml theme={null}
transformations:
  validation_mode: relaxed
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        attributes:
          name: hstore
          parameters:
            operations:
              - operation: set
                key: "email"
                value_template: '{{masking "email" .GetValue}}'
              - operation: delete
                key: "public_key"
                error_not_exist: true
              - operation: set
                key: "private_key"
                value_template: '{{masking "default" .GetValue}}'
              - operation: set
                key: "newKey"
                value: "newValue"
```

For input hstore value,

```
  public_key => 12345,
  private_key => 12345abcdefg,
  email => user@email.com
```

the hstore transformer with above config produces output:

```
  private_key => ************,
  email => use***@email.com
  newKey => newValue
```

#### literal\_string

**Description:** Transforms all values into the given constant value.

**Uniqueness:** `lossy`. Every value becomes the same literal. Cannot be used on a column covered by a unique index. See [Uniqueness and unique indexes](#uniqueness-and-unique-indexes).

| Supported PostgreSQL types |
| - |
| All types with a string representation |

| Parameter | Type | Default | Required |
| - | - | - | - |
| literal | string | N/A | Yes |

Below example makes all values in the JSON column `log_message` to become `{'error': null}`.
This transformer can be used for any Postgres type as long as the given string literal has the correct syntax for that type. e.g It can be "5-10-2021" for a date column, or "3.14159265" for a double precision one.

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: logs
      column_transformers:
        log_message:
          name: literal_string
          parameters:
            literal: "{'error': null}"
```

#### phone\_number

**Description:** Generates anonymized phone numbers with customizable length.

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar` |

| Parameter | Type | Default | Required | Values | Dynamic |
| - | - | - | - | - | - |
| prefix | string | "" | No | N/A | Yes |
| min\_length | int | 6 | No | N/A | No |
| max\_length | int | 10 | No | N/A | No |
| generator | string | random | No | random, deterministic | No |

If the prefix is set, this transformer will always generate phone numbers starting with the prefix.
`prefix` can also be a dynamic parameter, referring to some other column. Please see the below example config.

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        phone:
          name: phone_number
          parameters:
            min_length: 9
            max_length: 12
            generator: deterministic
          dynamic_parameters:
            gender:
              column: country_code
```

#### email

**Description:** Anonymizes email addresses while optionally excluding domain to anonymize.

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar`, `citext` |

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| replacement\_domain | string | "@example.com" | No | |
| exclude\_domain | string | "" | No | |
| salt | string | "defaultsalt" | No | |

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: customers
      column_transformers:
        email:
          name: email
          parameters:
            exclude_domain: "example.com"
            salt: "helloworld"
```

**Input-Output Examples:**

| Input Email | Configuration Parameters | Output Email |
| - | - | - |
| `john.doe@company.org` | `exclude_domain: "company.org", salt: "helloworld"` | `john.doe@company.org` |
| `jane.doe@company.org` | `exclude_domain: "exclude.com", salt: "helloworld"` | `T79P9zlFWzmT0yCUDMEE7S@example.com` |
| `jane.doe@company.org` | `exclude_domain: "exclude.com", salt: "helloworld", replacement_domain: "@random.com"` | `6EIWw5lEa8nsY9JDOm5@random.com` |
| `invalid-email` | `exclude_domain: "exclude.com", salt: "helloworld"` | `1fk5VLgTeoRQCCvqXFoToC1@example.com` |
| `invalid-email` | `exclude_domain: "exclude.com", salt: "helloworld", replacement_domain: "@random.com"` | `6EIWw5lEa8nsY9JDOm5@random.com` |

#### encrypted\_aes\_siv

**Description:** Encrypts values with AES-SIV (RFC 5297), a deterministic authenticated encryption scheme. The same input, key and associated data always produce the same token, so equality relationships between column values are preserved across rows, tables and runs — while remaining reversible by holders of the key, unlike hashing. Tokens are authenticated: tampered or forged values fail decryption. The output is the ciphertext encoded as unpadded base64url, safe for URLs and file names.

**Uniqueness:** `preserved`. Encryption is reversible with the key, so two distinct plaintexts cannot share a token. This is the only transformer that never collides on a column covered by a unique index. Note that being reversible makes it pseudonymization rather than anonymization — see [Keeping a column unique](#keeping-a-column-unique).

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar`, `bytea` |

**Note on length-constrained columns:** the token is always longer than the input — `ceil(4 × (input_length + 16) / 3)` characters (36 for an 11-character input, 22 minimum). Length-constrained columns (`varchar(n)`, `char(n)`) must be wide enough to hold the expanded token or writes to the target will fail; prefer `text` columns.

For `bytea` columns the raw bytes are encrypted (the transformer normalizes the hex-text form delivered during replication and the raw bytes delivered during snapshots to the same plaintext), and the token is stored as the ASCII bytes of the base64url text.

| Parameter | Type | Default | Required |
| - | - | - | - |
| key\_hex | string | N/A | Yes |
| associated\_data | string | "" | No |

`key_hex` is the 64-byte AES-SIV key, hex-encoded (128 characters), e.g. generated with `openssl rand -hex 64`. AES-SIV requires the full 64-byte key (RFC 5297); shorter keys are rejected.

`associated_data` is authenticated but not encrypted: a token minted with one associated data value fails decryption under another. Use it to bind tokens to a context (such as a table/column name) so they cannot be replayed elsewhere.

**Security note:** the encryption is deterministic by design — equal inputs produce equal tokens, which reveals equality (and only equality) of the underlying values. Anyone holding the key can decrypt the tokens, so the anonymization guarantee is key custody: load the key from a secret manager and never store it alongside the transformed data.

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: orders
      column_transformers:
        file_path:
          name: encrypted_aes_siv
          parameters:
            # example key only — generate your own and load it from a secret store
            key_hex: "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f"
            associated_data: "public.orders.file_path"
```

**Input-Output Examples:**

| Input | Configuration Parameters | Output |
| - | - | - |
| `hello world` | `key_hex: "000102…3e3f"`, no `associated_data` | `Hc5d96xIxu2ute1RbFuenEftGxw-P__m1Vv_` |
| `hello world` | `key_hex: "000102…3e3f"`, `associated_data: "public.orders.file_path"` (example above) | `AsC2hoq-Y9V0iqK6JmNIxq0Fsr3SFPo27QNq` |

Every run with the same key and parameters produces the same output. Tokens can be decrypted with any RFC 5297 AES-SIV implementation, for example Tink's `daead/subtle` package in Go.

#### lookup\_choice

**Description:** Replaces a value with one taken from a column of another table, for example a foreign key column pointing at a lookup table. The values are read from the source database once, when the pipeline starts, so the configuration does not have to be regenerated when the lookup table's contents change. It is the live-table counterpart of [`greenmask_choice`](#supported-transformers), which chooses from a list written into the configuration.

**Uniqueness:** `lossy`. Any table with more rows than the lookup column has values produces duplicates, in both generator modes. Cannot be used on a column covered by a unique index. See [Uniqueness and unique indexes](#uniqueness-and-unique-indexes).

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar`, `citext`, `bytea`, `boolean`, `int2`, `int4`, `int8`, `float4`, `float8`, `uuid`, `date`, `timestamp`, `timestamptz` |

The type comes from the **lookup column**: the transformer asks PostgreSQL what it is and reports the column types its values can be written to, so a rule pointing a column at a lookup column of an incompatible type is rejected on startup. A narrower integer or float is accepted for a wider column (an `int4` lookup key can fill an `int8` foreign key). A lookup column of any other type is rejected rather than silently skipping the check.

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| lookup\_table | string | N/A | Yes | N/A |
| lookup\_column | string | N/A | Yes | N/A |
| generator | string | random | No | random, deterministic |
| max\_values | integer | 100000 | No | N/A |
| ignore\_values | any\[] | \[] | No | N/A |
| postgres\_url | string | N/A | Yes | N/A |

`lookup_table` is schema qualified, e.g. `public.countries`; an unqualified name is read from the `public` schema. Both names are quoted as written, so `Public.Countries` looks for a case-sensitive `"Countries"`.

`postgres_url` is required, but the PostgreSQL parser fills it in with the URL of the source database being read, so it only has to be written out when the source is not PostgreSQL.

`max_values` caps how many values are read. The load fails if the lookup column holds more, rather than truncating the list, because a truncated list would silently change which value every row is mapped to. Raise it if the table really is that large and the memory cost is acceptable.

`ignore_values` removes values from the list after it is read, for placeholder rows such as an "unknown" id. An entry that matches nothing is an error, so a typo or a value written in a form the column never produces is reported rather than quietly leaving the row in the choice set. If it excludes every value, or the lookup column is empty, the pipeline fails to start rather than writing the same value into every row.

`generator: deterministic` picks the value from a hash of the incoming value, so every row that pointed at the same original value still points at one single new value. `generator: random` picks independently for each row and destroys that grouping. Deterministic mode is rejected for `timestamp` and `timestamptz` lookup columns, because a snapshot and a replication event deliver a timestamp in forms that cannot be reduced to the same hash input, so the same row would be mapped differently either side of the cutover.

**Security note:** the deterministic mapping is an unsalted hash over a value set that is usually small and often public. Anyone holding the transformed data and a guess at the lookup table can compute the same hashes and recover much of the original mapping. Deterministic mode preserves structure; it does not hide the values it maps from. The same is true of `greenmask_choice`.

⚠️ **A reference can be left dangling, and the rows are dropped rather than reported.** The values are read from the lookup table in the **source** database. If that table's own key column is itself transformed, or the lookup row is deleted after the pipeline started, the value chosen here will not exist in the target and the foreign key constraint rejects it. Nothing checks this for you. During a snapshot the constraints are restored after the data, so the failure arrives at the end of the load as a constraint violation that names the constraint rather than this rule. Under replication a constraint violation is not retried: in the default mode the row is **dropped** with a `DATALOSS` log line and the checkpoint advances, so the pipeline keeps running without it, and with `deterministic` every row sharing that original value is dropped too. Under `strict_mode` the pipeline stops instead. Leave the lookup table's key column untransformed, with `noop` if the validation mode requires a rule for it.

⚠️ **The mapping only holds while the lookup column does.** The value is chosen by position in the loaded list, so inserting or deleting a single row in the lookup table — or editing `ignore_values` — remaps almost every input the next time the pipeline starts. Deterministic mode is reproducible across restarts for a **fixed** lookup set; it is not stable across changes to it. New rows in the lookup table are not picked up until a restart.

⚠️ **The whole column is loaded into memory, once per rule.** There is no paging, and each column rule runs its own query: three columns reading the same lookup table load it three times. `max_values` caps each of those loads at 100000 values by default, so a rule pointed at a large table fails on startup instead of growing until the kernel intervenes. The read is given 30 seconds, so a locked or unreachable lookup table fails startup instead of hanging it.

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: addresses
      column_transformers:
        country_id:
          name: lookup_choice
          parameters:
            lookup_table: public.countries
            lookup_column: id
            generator: deterministic
            ignore_values: [0, -1]
```

**Input-Output Examples:**

Given a `public.countries` table whose `id` column holds `1, 2, 3`:

| Input Value | Configuration Parameters | Output Value |
| - | - | - |
| `7` | `generator: random` | `3` (random) |
| `7` | `generator: deterministic` | `2` |
| `7` | `generator: deterministic` | `2` (again, next run) |
| `8` | `generator: deterministic` | `1` |

#### fpe\_ff1

**Description:** This transformer encrypts values with FF1 ([NIST SP 800-38G](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-38G.pdf)). FF1 is a format-preserving encryption algorithm. The output contains the same characters as the input, and it has the same length. An encrypted phone number is still a phone number. An encrypted product code still fits a `varchar(n)` column. The transformer copies the characters that are not in the alphabet to the same positions in the output.

**Uniqueness:** `preserved`. FF1 maps each input to one different output. Two different inputs cannot give the same output. Use this transformer when a unique column must keep its format. Use `encrypted_aes_siv` when the format is not important.

| Supported PostgreSQL types |
| - |
| `text`, `varchar`, `char`, `bpchar` |

| Parameter | Type | Default | Required | Values |
| - | - | - | - | - |
| key\_hex | string | N/A | Yes | 32, 48 or 64 hexadecimal characters |
| associated\_data | string | "" | No | Any string |
| alphabet | string | digits | No | `digits`, `letters`, `alphanumeric`, or a set of characters |
| passthrough | string | keep | No | `keep`, `error` |
| keep\_prefix | int | 0 | No | Number of characters in the alphabet to keep at the start |
| keep\_suffix | int | 0 | No | Number of characters in the alphabet to keep at the end |
| min\_length | int | 0 | No | 0 sets the minimum that FF1 permits for the alphabet |
| preserve\_from | string | "" | No | A delimiter. The transformer keeps the text from its last occurrence. |

`key_hex` is the AES key in hexadecimal. Use 32, 48 or 64 characters for a key of 128, 192 or 256 bits. To make a key, run `openssl rand -hex 32`. The `encrypted_aes_siv` transformer is different, because it needs a key of 64 bytes.

`associated_data` is the FF1 tweak. The tweak is not secret. It makes the output different for each column when you use the same key. Set it to `schema.table.column`.

`alphabet` is the set of characters that the transformer encrypts. There are three named sets: `digits` (`0-9`), `letters` (`a-zA-Z`) and `alphanumeric` (`0-9a-zA-Z`). Any other value is a set of characters that you write out. Such a set must contain two characters or more, and it must not contain a character two times. The transformer does not encrypt the characters that are not in the alphabet.

`passthrough` controls the characters that are not in the alphabet. The value `keep` copies these characters to the output. Characters such as `-`, `+` and the space stay in a phone number. The value `error` rejects the value.

`keep_prefix` and `keep_suffix` keep a number of characters without a change. Use them to keep a country code or a check digit. These parameters count only the characters in the alphabet. Separators do not increase the count, because the transformer does not encrypt them. For example, set `keep_prefix: 4` for a `digits` column. The transformer then keeps the country code and the operator code of `+36301234567`, `+36 30 123 4567` and `+36-30-123-4567`. The three formats give the same digits in the output. The transformer rejects a value that has too few characters. Refer to the first caution below.

`preserve_from` keeps the end of the value without a change. The part that the transformer keeps starts at the **last** occurrence of the delimiter. Use this parameter when a delimiter marks the part to keep. `keep_suffix` keeps a fixed number of characters, but `preserve_from` keeps a variable number. In an email address, `preserve_from: "@"` keeps the full domain. `preserve_from: "."` keeps only the top-level domain, and it encrypts the domain name. If the value does not contain the delimiter, the transformer encrypts the full value. The delimiter must not contain a character from the alphabet. If it does, pgstream rejects the configuration when it starts. The transformer copies only the characters that are not in the alphabet to the same position in the output. This condition is necessary to keep the outputs unique.

**⚠️ The transformer rejects short values.** FF1 does not encrypt a set of possible values that is smaller than 10<sup>6</sup>, because an attacker can try all of them. For `digits`, a value must contain 6 characters or more from the alphabet. For `letters` and `alphanumeric`, it must contain 4 characters or more. Count the characters after you remove the prefix, the suffix, and the characters that are not in the alphabet. The transformer returns an error for a shorter value. It does not send the original value to the target, because this makes the data visible. The error goes to the [`on_error`](#transformation-rules) policy, where you select `fail`, `pass-through` or `null`. Be careful with `pass-through`, because it writes the original value to the target. `min_length` increases the minimum, for example to 9 digits. It cannot decrease the minimum.

**⚠️ The transformer keeps the format, but the output is not realistic.** A phone number stays a possible phone number, and a code stays a valid code. But with `alphabet: letters`, a company name becomes a random text such as `Dmkh VjksJwk`. The output has the shape of a name and it is unique, but it is not a real company name.

**Email addresses.** With `alphabet: alphanumeric` and `passthrough: keep`, the transformer encrypts the local part, the domain and the top-level domain. The address `john.doe@example.com` becomes `FdSQ.v6s@udengIL.0Td`. The top-level domain is not valid, and a mail server cannot deliver to this address. Use `preserve_from` to correct this:

| `preserve_from` | Result | What the transformer encrypts |
| - | - | - |
| unset | `FdSQ.v6s@udengIL.0Td` | The full address. The top-level domain is not valid. |
| `"."` | `lHtC.Mr7@1uH9RP1.com` | The local part and the domain name. The transformer keeps the top-level domain, thus the address has a correct format. |
| `"@"` | `69pe.Nzy@example.com` | The local part only. The address is valid, but the target shows the domain. |

Select the option that agrees with your data. Do not use `preserve_from: "@"` when the domain is sensitive, because the target then contains each source domain. All three options keep the outputs unique. Use `neosync_email` with `preserve_domain: true` when a mail server must deliver to the address. That transformer makes a new local part, but its uniqueness is `not_guaranteed`. Two different addresses can give the same output.

**Security note:** The encryption is deterministic. Equal inputs give equal outputs. This shows which values are equal, but it shows no other data. A person who has the key can decrypt the values. Keep the key in a secret manager, and do not store it with the transformed data.

**Example Configuration:**

```yaml theme={null}
transformations:
  table_transformers:
    - schema: public
      table: persons
      column_transformers:
        phone_number:
          name: fpe_ff1
          parameters:
            # example key only — generate your own and load it from a secret store
            key_hex: "000102030405060708090a0b0c0d0e0f"
            associated_data: "public.persons.phone_number"
            alphabet: digits
            passthrough: keep
            # keep 4 digits: the country code and the operator code.
            # separators do not increase this count.
            keep_prefix: 4
```

**Input-Output Examples:**

| Input | Configuration Parameters | Output |
| - | - | - |
| `1234567890` | `key_hex: "000102…0e0f"`, no `associated_data` | `5102547240` |
| `1234567890` | `key_hex: "000102…0e0f"`, `associated_data: "public.persons.phone_number"` | `0453276999` |
| `+36301234567` | as the example above (`keep_prefix: 4`) | `+36303497985` |
| `+36 30 123 4567` | as the example above (`keep_prefix: 4`) | `+36 30 349 7985` |
| `(301) 555-0123` | as the example above (`keep_prefix: 4`) | `(301) 545-8564` |
| `Acme Trading` | `key_hex: "000102…0e0f"`, `alphabet: letters` | `Dmkh VjksJwk` |
| `john.doe@example.com` | `key_hex: "000102…0e0f"`, `alphabet: alphanumeric` | `FdSQ.v6s@udengIL.0Td` |
| `john.doe@example.com` | `key_hex: "000102…0e0f"`, `alphabet: alphanumeric`, `preserve_from: "."` | `lHtC.Mr7@1uH9RP1.com` |
| `john.doe@example.com` | `key_hex: "000102…0e0f"`, `alphabet: alphanumeric`, `preserve_from: "@"` | `69pe.Nzy@example.com` |

Each run gives the same output for the same key and the same parameters. Any NIST FF1 implementation can decrypt the values. It needs the key, the tweak and the alphabet.

### Transformation rules

The rules for the transformers are defined in a dedicated yaml file with the following format:

```yaml theme={null}
transformations:
  infer_from_security_labels: false # whether to infer anonymization rules from Postgres `anon` security labels. Requires a live connection to the source database. Defaults to false
  dump_inferred_rules: false # if set, dumps the inferred anonymization rules to a file in YAML format for debugging purposes. The file will be named `inferred_anon_transformation_rules.yaml` and will be created in the directory where pgstream is run. Defaults to false
  validation_mode: <validation_mode> # Validation mode for the transformation rules. Can be one of strict, relaxed or table_level. Defaults to relaxed if not provided.
  table_transformers: # List of table transformations
    - schema: <schema_name> # Name of the table schema
      table: <table_name> # Name of the table
      validation_mode: <validation_mode> # To be used when the global validation_mode is set to `table_level`. Can be one of strict or relaxed
      column_transformers: # List of column transformations
        <column_name>: # Name of the column to which the transformation will be applied
          name: <transformer_name> # Name of the transformer to be applied to the column. If no transformer needs to be applied on strict validation mode, it can be left empty or use `noop`
          allow_uniqueness_loss: false # Whether to allow a transformer that can produce duplicates on a column covered by a unique index. Defaults to false. See "Uniqueness and unique indexes"
          parameters: # Transformer parameters as defined in the supported transformers documentation
            <transformer_parameter>: <transformer_parameter_value>
          array_options: # Only valid on an array column. If omitted, the transformer is applied to each element in order. See "Array columns"
            generator: <map|random> # How the elements of the new array are produced. Defaults to map
            min_count: <min_count> # Smallest number of elements to emit. Required with the random generator
            max_count: <max_count> # Largest number of elements to emit. Required with the random generator
```

When the `infer_from_security_labels` option is enabled, the table transformers will be parsed from the source Postgres [`SECURITY LABELS`](https://www.postgresql.org/docs/current/sql-security-label.html) for the [`anon` extension](https://postgresql-anonymizer.readthedocs.io/en/stable/declare_masking_rules/). If the option is not enabled, the table transformers need to be explicitly provided.

Below is a complete example of a transformation rules YAML file:

```yaml theme={null}
transformations:
  infer_from_security_labels: false # whether to infer anonymization rules from Postgres `anon` security labels. Requires a live connection to the source database. Defaults to false
  dump_inferred_rules: false # if set, dumps the inferred anonymization rules to a file in YAML format for debugging purposes. The file will be named `inferred_anon_transformation_rules.yaml` and will be created in the directory where pgstream is run. Defaults to false
  validation_mode: table_level
  table_transformers:
    - schema: public
      table: users
      validation_mode: strict
      column_transformers:
        email:
          name: neosync_email
          parameters:
            preserve_length: true
            preserve_domain: true
        first_name:
          name: greenmask_firstname
          parameters:
            gender: Male
        username:
          name: greenmask_string
          parameters:
            generator: random
            min_length: 5
            max_length: 15
            symbols: "abcdefghijklmnopqrstuvwxyz1234567890"
    - schema: public
      table: orders
      validation_mode: relaxed
      column_transformers:
        status:
          name: greenmask_choice
          parameters:
            generator: random
            choices: ["pending", "shipped", "delivered", "cancelled"]
        order_date:
          name: greenmask_date
          parameters:
            generator: random
            min_value: "2020-01-01"
            max_value: "2025-12-31"
```

Validation mode can be set to `strict` or `relaxed` for all tables at once. Or it can be determined for each table individually, by setting the higher level `validation_mode` parameter to `table_level`. When it is set to strict, pgstream will throw an error if any of the columns in the table do not have a transformer defined. When set to relaxed, pgstream will skip any columns that do not have a transformer defined. Also in strict mode, all snapshot tables must be provided in the transformation config.
For details on how to use and configure the transformer, check the [transformer tutorial](/docs/opensource/pgstream/docs/tutorials/postgres_transformer).


# Replication using Kafka
Source: https://xata.io/docs/opensource/pgstream/docs/tutorials/postgres_kafka

## Introduction

This tutorial will showcase the use of pgstream to replicate data from a PostgreSQL database to multiple targets (PostgreSQL and OpenSearch), leveraging Kafka. Kafka can also be used to replicate to a single target, to take advantage of the fan out model that allows for parallel processing of the WAL events thanks to topic partitioning.

![kafka tutorial](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_tutorial_kafka.svg)

### Requirements

* A source PostgreSQL database
* A target PostgreSQL database
* A target OpenSearch cluster
* A Kafka cluster
* pgstream (see [installation](/docs/opensource/pgstream/docs/cli#installation) instructions for more details)

## Environment setup

The first step is to start the PostgreSQL databases that will be used as source and the target PostgreSQL database and OpenSearch cluster for replication, along with the Kafka cluster. The `pgstream` repository provides a docker installation that will be used for the purposes of this tutorial, but can be replaced by any available PostgreSQL servers, as long as they have [`wal2json`](https://github.com/eulerto/wal2json) installed, and any Kafka and OpenSearch clusters.

To start the docker provided PostgreSQL servers, OpenSearch cluster and Kafka cluster run the following command:

```sh theme={null}
docker-compose -f build/docker/docker-compose.yml --profile pg2pg --profile pg2os --profile kafka up
```

This will start two PostgreSQL databases on ports `5432` and `7654`, an OpenSearch cluster on port `9200` and a Kafka cluster on port `9092`.

## Database initialisation

Once all the resources are up and running, the next step is to initialise pgstream on the source database. This will create the `pgstream` schema in the configured Postgres database, along with the tables/functions/triggers required to keep track of the schema changes. See [Tracking schema changes](/docs/opensource/pgstream/docs/architecture#tracking-schema-changes) section for more details. This step will also create a replication slot on the source database which will be used by the pgstream service.

The initialisation step allows to provide both the URL of the PostgreSQL database and the name of the replication slot to be created. The PostgreSQL URL is required, but the replication slot name is optional. If not provided, it will default to `pgstream_<dbname>_slot`, where `<dbname>` is the name of the PostgreSQL database. The configuration can be provided either by using the CLI supported parameters, or using the environment variables.

For this tutorial, we'll create a replication slot with the name `pgstream_tutorial_slot`.

* Using CLI parameters:

```sh theme={null}
pgstream init --postgres-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --replication-slot pgstream_tutorial_slot
```

* Using environment variables:

```sh theme={null}
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot PGSTREAM_POSTGRES_LISTENER_URL=postgres://postgres:postgres@localhost:5432?sslmode=disable pgstream init
```

Successful initialisation should prompt the following message:

```
SUCCESS  pgstream initialisation complete
```

If at any point the initialisation performed by pgstream needs to be reverted, all state will be removed by running the `destroy` CLI command.

```sh theme={null}
pgstream destroy --postgres-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --replication-slot pgstream_tutorial_slot
```

## Prepare `pgstream` configuration

For this tutorial, we need to prepare 3 different configuration files, since we'll be running 3 separate instances of pgstream, each handling one of the flows described below.

### PostgreSQL -> Kafka

#### Listener

In order to run pgstream, we need to provide the configuration required to run the PostgreSQL replication. First, we configure the listener module that will be listening to the WAL on the source PostgreSQL database. This requires the PostgreSQL database URL, which will be the one from the docker PostgreSQL server we started and setup in the previous steps.

```sh theme={null}
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
```

Since we've set a custom replication slot name, the configuration variable needs to be set accordingly so that it doesn't use the default value.

```sh theme={null}
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot
```

The PostgreSQL WAL listener can be configured to perform an initial snapshot of the existing PostgreSQL database tables before starting to listen on the replication slot. In this case, we have no existing tables, so we don't need to configure the initial snapshot.

However, if there were tables with pre-existing data that we wanted to replicate to the target PostgreSQL database, we could configure it by setting the following environment variables:

```sh theme={null}
# URL of the PostgreSQL database we want to snapshot
PGSTREAM_POSTGRES_SNAPSHOT_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"

# List of tables we want to snapshot. If the tables are not schema qualified, the public schema will be assumed.
# Wildcards are supported.
#
# The following example will snapshot all tables in the `test_schema` and the table `test` from the public schema.
PGSTREAM_POSTGRES_SNAPSHOT_TABLES="test_schema.* test"
```

Further configuration can be provided to optimize the performance of the snapshot process. For more information, check the [snapshot tutorial](/docs/opensource/pgstream/docs/tutorials/postgres_snapshot).

#### Processor

We will be using a Kafka processor which will be writing the WAL events into a Kafka topic. The topic is partitioned by schema. The only required configuration are the servers URLs and the topic settings. In our case, we'll use the Kafka cluster initialised in the previous steps, and use a topic named `pgstream`.

```sh theme={null}
# List of Kafka servers
PGSTREAM_KAFKA_WRITER_SERVERS="localhost:9092"
PGSTREAM_KAFKA_TOPIC_NAME=pgstream
PGSTREAM_KAFKA_TOPIC_PARTITIONS=1
# Replication factor for the kafka topic. Defaults to 1.
PGSTREAM_KAFKA_TOPIC_REPLICATION_FACTOR=1
# Create the topic automatically if it doesn't exist. Defaults to false.
PGSTREAM_KAFKA_TOPIC_AUTO_CREATE=true
```

The Kafka processor can also be configured to use a TLS connection to the Kafka servers.

```sh theme={null}
PGSTREAM_KAFKA_TLS_ENABLED=true
# Filepaths to the certificate PEM files
PGSTREAM_KAFKA_TLS_CA_CERT_FILE="broker.cer.pem"
PGSTREAM_KAFKA_TLS_CLIENT_CERT_FILE="client.cer.pem"
PGSTREAM_KAFKA_TLS_CLIENT_KEY_FILE="client.key.pem"
```

The Kafka processor can also use SASL authentication. Give the mechanism, the user and the password. The mechanism must be `plain`, `scram-sha-256` or `scram-sha-512`. `pgstream` does not support GSSAPI or OAUTHBEARER.

```sh theme={null}
PGSTREAM_KAFKA_SASL_ENABLED=true
PGSTREAM_KAFKA_SASL_MECHANISM="scram-sha-512"
PGSTREAM_KAFKA_SASL_USER="myuser"
PGSTREAM_KAFKA_SASL_PASSWORD="mypassword"
```

The `plain` mechanism sends the password in clear text. Enable TLS with it.

These four variables apply to the Kafka reader and to the Kafka writer. Add them to the `Kafka -> PostgreSQL` and `Kafka -> OpenSearch` configuration too, with the same values.

The Kafka processor uses batching under the hood to reduce the number of IO calls to Kafka and improve performance. The batch size and send timeout can both be configured to be able to better fit the different traffic patterns. The indexer will send a batch when the timeout or the batch size is reached, whichever happens first.

```sh theme={null}
# Number of messages that will be batched and sent together in a given request. It defaults to 100.
PGSTREAM_KAFKA_WRITER_BATCH_SIZE=25
# Max delay between batch sending. The batches will be sent every 5s by default.
PGSTREAM_KAFKA_WRITER_BATCH_TIMEOUT=5s
# Max size for a batch in bytes. Defaults to 1572864.
PGSTREAM_KAFKA_WRITER_BATCH_BYTE=1572864
```

Since we'll be replicating data to OpenSearch, we need to add the injector processor in order to inject pgstream metadata into the WAL events. This helps identify the id and version of the event. By enabling the injector for the kafka processor, we make the metadata available to all processors downstream of Kafka. If we only wanted to apply it to the OpenSearch processor we could add it to its configuration instead. More details about the injector can be found in the [architecture](/docs/opensource/pgstream/docs/architecture) section.

The full configuration for the pg2kafka step can be put into a `pg2kafka_tutorial.env` file to be used later on. An equivalent `pg2kafka_tutorial.yaml` configuration can be found below the environment one, and can be used interchangeably.

* Without initial snapshot

```sh theme={null}
# Listener config
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot

# Processor config
PGSTREAM_INJECTOR_STORE_POSTGRES_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_KAFKA_WRITER_SERVERS="localhost:9092"
PGSTREAM_KAFKA_TOPIC_NAME=pgstream
PGSTREAM_KAFKA_TOPIC_PARTITIONS=1
PGSTREAM_KAFKA_TOPIC_REPLICATION_FACTOR=1
PGSTREAM_KAFKA_TOPIC_AUTO_CREATE=true
```

```yaml theme={null}
source:
  postgres:
    url: "postgres://postgres:postgres@localhost:5432?sslmode=disable"
    mode: replication
    replication:
      replication_slot: "pgstream_tutorial_slot"
target:
  kafka:
    servers: ["localhost:9092"]
    topic:
      name: "pgstream" # name of the Kafka topic
      partitions: 1 # number of partitions for the topic
      replication_factor: 1 # replication factor for the topic
      auto_create: true # whether to automatically create the topic if it doesn't exist
modifiers:
  injector:
    enabled: true # whether to inject pgstream metadata into the WAL events
```

* With initial snapshot

```sh theme={null}
# Listener config
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot
PGSTREAM_POSTGRES_SNAPSHOT_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
# Initial snapshot of all tables in the public schema
PGSTREAM_POSTGRES_SNAPSHOT_TABLES="*"

# Processor config
PGSTREAM_INJECTOR_STORE_POSTGRES_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_KAFKA_WRITER_SERVERS="localhost:9092"
PGSTREAM_KAFKA_TOPIC_NAME=pgstream
PGSTREAM_KAFKA_TOPIC_PARTITIONS=1
PGSTREAM_KAFKA_TOPIC_REPLICATION_FACTOR=1
PGSTREAM_KAFKA_TOPIC_AUTO_CREATE=true
```

```yaml theme={null}
source:
  postgres:
    url: "postgres://postgres:postgres@localhost:5432?sslmode=disable"
    mode: replication
    replication:
      replication_slot: "pgstream_tutorial_slot"
    snapshot: # when mode is snapshot or snapshot_and_replication
      mode: full # options are data_and, schema or data
      tables: ["*"] # tables to snapshot, can be a list of table names or a pattern
      recorder:
        repeatable_snapshots: true # whether to repeat snapshots that have already been taken
        postgres_url: "postgres://postgres:postgres@localhost:5432?sslmode=disable" # URL of the database where the snapshot status is recorded
      schema: # when mode is full or schema
        pgdump_pgrestore:
          clean_target_db: false # whether to clean the target database before restoring
target:
  kafka:
    servers: ["localhost:9092"]
    topic:
      name: "pgstream" # name of the Kafka topic
      partitions: 1 # number of partitions for the topic
      replication_factor: 1 # replication factor for the topic
      auto_create: true # whether to automatically create the topic if it doesn't exist
modifiers:
  injector:
    enabled: true # whether to inject pgstream metadata into the WAL events
```

### Kafka -> PostgreSQL

#### Listener

In this case the listener will be a Kafka reader that will listen for the WAL events from the Kafka topic. The only configuration required are the kafka servers, the topic name and the consumer group id. We will use a consumer group for each of the targets, in this case `pgstream-postgres-consumer-group`.

```sh theme={null}
PGSTREAM_KAFKA_READER_SERVERS="localhost:9092"
PGSTREAM_KAFKA_TOPIC_NAME=pgstream
PGSTREAM_KAFKA_READER_CONSUMER_GROUP_ID=pgstream-postgres-consumer-group
```

The configuration can be tweaked further by providing the start offset for the consumer group. Valid values are `earliest` or `latest`.

```sh theme={null}
PGSTREAM_KAFKA_READER_CONSUMER_GROUP_START_OFFSET=earliest
```

The Kafka reader is in charge of committing the messages to Kafka once they've been processed, so that they're not processed multiple times. It applies a default exponential backoff policy to this commit process, but it can be configured to select different parameters for the exponential backoff or to provide a non exponential backoff policy. Only one of the two will be applied, giving priority to the exponential backoff configuration if both are provided.

```sh theme={null}
# The amount of time the initial exponential backoff interval will apply. Defaults to 1s.
PGSTREAM_KAFKA_COMMIT_BACKOFF_INITIAL_INTERVAL=1s
# The max amount of time the exponential backoff will retry for. Defaults to 1m.
PGSTREAM_KAFKA_COMMIT_BACKOFF_MAX_INTERVAL=1m
# Maximum amount of retries the exponential backoff will retry for. Defaults to 60.
PGSTREAM_KAFKA_COMMIT_BACKOFF_MAX_RETRIES=60

# Constant interval that the backoff policy will apply between retries. Defaults to 0.
PGSTREAM_KAFKA_COMMIT_BACKOFF_INTERVAL=0
# Max number of retries the backoff policy will apply. Defaults to 0.
PGSTREAM_KAFKA_COMMIT_BACKOFF_MAX_RETRIES=0
```

#### Processor

This configuration is the same as the one from the tutorial for PostgreSQL to PostgreSQL replication. More details [here](/docs/opensource/pgstream/docs/tutorials/postgres_to_postgres#processor).

The full configuration for the kafka2pg step can be put into a `kafka2pg_tutorial.env` file to be used later on. An equivalent `kafka2pg_tutorial.yaml` configuration can be found below the environment one, and can be used interchangeably.

```sh theme={null}
# Listener config
PGSTREAM_KAFKA_READER_SERVERS="localhost:9092"
PGSTREAM_KAFKA_TOPIC_NAME=pgstream
PGSTREAM_KAFKA_READER_CONSUMER_GROUP_ID=pgstream-postgres-consumer-group

# Processor config
PGSTREAM_POSTGRES_WRITER_TARGET_URL="postgres://postgres:postgres@localhost:7654?sslmode=disable"
PGSTREAM_POSTGRES_WRITER_BATCH_SIZE=25
PGSTREAM_POSTGRES_WRITER_BATCH_TIMEOUT=5s
```

```yaml theme={null}
source:
  kafka:
    servers: ["localhost:9092"]
    topic:
      name: "pgstream"
    consumer_group:
      id: "pgstream-postgres-consumer-group"
      start_offset: "earliest" # options are earliest or latest
target:
  postgres:
    url: "postgres://postgres:postgres@localhost:7654?sslmode=disable"
    batch:
      timeout: 5000 # batch timeout in milliseconds
      size: 25 # number of messages in a batch
    disable_triggers: false # whether to disable triggers on the target database
    on_conflict_action: "nothing" # options are update, nothing or error
```

### Kafka -> OpenSearch

#### Listener

The configuration for the Kafka listener is the same as for the one in the previous step, the only difference will be the name of the consumer group id, to ensure it consumes the topic independently from the postgres one.

```sh theme={null}
PGSTREAM_KAFKA_READER_SERVERS="localhost:9092"
PGSTREAM_KAFKA_TOPIC_NAME=pgstream
PGSTREAM_KAFKA_READER_CONSUMER_GROUP_ID=pgstream-opensearch-consumer-group
```

#### Processor

This configuration is the same as the one from the tutorial for PostgreSQL to OpenSearch replication. More details [here](/docs/opensource/pgstream/docs/tutorials/postgres_to_opensearch#processor).

The full configuration for the kafka2opensearch step can be put into a `kafka2os_tutorial.env` file to be used later on. An equivalent `kafka2os_tutorial.yaml` configuration can be found below the environment one, and can be used interchangeably.

```sh theme={null}
# Listener config
PGSTREAM_KAFKA_READER_SERVERS="localhost:9092"
PGSTREAM_KAFKA_TOPIC_NAME=pgstream
PGSTREAM_KAFKA_READER_CONSUMER_GROUP_ID=pgstream-opensearch-consumer-group

# Processor config
PGSTREAM_OPENSEARCH_STORE_URL="http://admin:admin@localhost:9200"
PGSTREAM_SEARCH_INDEXER_BATCH_SIZE=25
PGSTREAM_SEARCH_INDEXER_BATCH_TIMEOUT=5s
```

```yaml theme={null}
source:
  kafka:
    servers: ["localhost:9092"]
    topic:
      name: "pgstream"
    consumer_group:
      id: "pgstream-opensearch-consumer-group"
      start_offset: "earliest" # options are earliest or latest
target:
  search:
    engine: "opensearch" # options are elasticsearch or opensearch
    url: "http://admin:admin@localhost:9200" # URL of the search engine
    batch:
      timeout: 5000 # batch timeout in milliseconds
      size: 25 # number of messages in a batch
```

## Validate `pgstream` status

We can validate that the initialisation and the configuration are valid by running the `status` command before starting `pgstream`. This can be run for all configuration files.

```sh theme={null}
# using yaml configuration file
./pgstream status -c pg2kafka_tutorial.yaml
# using env configuration file
./pgstream status -c pg2kafka_tutorial.env
```

```sh theme={null}
SUCCESS  pgstream status check encountered no issues
Initialisation status:
 - Pgstream schema exists: true
 - Migration current version: 7
 - Migration status: success
 - Replication slot name: pgstream_tutorial_slot
 - Replication slot plugin: wal2json
 - Replication slot database: postgres
Config status:
 - Valid: true
Transformation rules status:
 - Valid: true
Source status:
 - Reachable: true
```

## Run `pgstream`

With all configuration files ready, we can now run pgstream. In this case we set the log level as trace to provide more context for debugging and have more visibility into what pgstream is doing under the hood.

As mentioned above, we'll need to run 3 different pgstream services on separate terminals, one with each configuration file.

```sh theme={null}
# with the environment configuration
pgstream run -c pg2kafka_tutorial.env --log-level trace

# with the yaml configuration
pgstream run -c pg2kafka_tutorial.yaml --log-level trace

# with the CLI flags and relying on defaults
pgstream run --source postgres --source-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --target kafka --target-url "localhost:9092" --log-level trace
```

```sh theme={null}
# with the environment configuration
pgstream run -c kafka2pg_tutorial.env --log-level trace

# with the yaml configuration
pgstream run -c kafka2pg_tutorial.yaml --log-level trace

# with the CLI flags and relying on defaults
pgstream run --source kafka --source-url "localhost:9092" --target postgres --target-url "postgres://postgres:postgres@localhost:7654?sslmode=disable" --log-level trace
```

```sh theme={null}
# with the environment configuration
pgstream run -c kafka2os_tutorial.env --log-level trace

# with the yaml configuration
pgstream run -c kafka2os_tutorial.yaml --log-level trace

# with the CLI flags and relying on defaults
pgstream run --source kafka --source-url "localhost:9092" --target opensearch --target-url "http://localhost:9200" --log-level trace
```

First we connect and create a table in the source PostgreSQL database.

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:5432/postgres
```

```sql theme={null}
CREATE TABLE test(id SERIAL PRIMARY KEY, name TEXT);
```

## Verify Replication

We should now be able to see the table created in the target PostgreSQL database:

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:7654/postgres
```

```sql theme={null}
postgres@localhost:postgres> \d+ test
+--------+---------+-----------+----------+--------------+-------------+
| Column | Type    | Modifiers | Storage  | Stats target | Description |
|--------+---------+-----------+----------+--------------+-------------|
| id     | integer |  not null | plain    | <null>       | <null>      |
| name   | text    |           | extended | <null>       | <null>      |
+--------+---------+-----------+----------+--------------+-------------+
Indexes:
    "test_pkey" PRIMARY KEY, btree (id)
Has OIDs: no
```

And see the table in the schema log document of the newly created `pgstream` index on the OpenSearch cluster:

```sh theme={null}
curl -X GET -u admin:admin http://localhost:9200/pgstream/_search | jq .
```

```json theme={null}
{
  "took": 31,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "pgstream",
        "_id": "cv9esaav80ig0h81pkj0",
        "_score": 1.0,
        "_source": {
          "id": "cv9esaav80ig0h81pkj0",
          "version": 1,
          "schema_name": "public",
          "created_at": "2025-03-13 14:39:37.295222",
          "schema": "{\"tables\":[{\"oid\":\"16464\",\"name\":\"test\",\"columns\":[{\"name\":\"id\",\"type\":\"integer\",\"default\":\"nextval('public.test_id_seq'::regclass)\",\"nullable\":false,\"unique\":true,\"metadata\":null,\"pgstream_id\":\"cv9esaav80ig0h81pkjg-1\"},{\"name\":\"name\",\"type\":\"text\",\"nullable\":true,\"unique\":false,\"metadata\":null,\"pgstream_id\":\"cv9esaav80ig0h81pkjg-2\"}],\"primary_key_columns\":[\"id\"],\"pgstream_id\":\"cv9esaav80ig0h81pkjg\"}]}",
          "acked": false
        }
      }
    ]
  }
}
```

As well as seeing a `public` index, with no documents for now (since there's no data in the table):

```sh theme={null}
curl -X GET -u admin:admin http://localhost:9200/public/_search | jq .
```

```json theme={null}
{
  "took": 16,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}
```

We can insert data into the source database `test` table, which should then appear in the target PostgreSQL database as well as the OpenSearch `public` index.

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:5432/postgres
```

```sql theme={null}
INSERT INTO test(name) VALUES('alice'),('bob'),('charlie');
```

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:7654/postgres
```

```sql theme={null}
postgres@localhost:postgres> select * from test;
+----+---------+
| id | name    |
|----+---------|
| 1  | alice   |
| 2  | bob     |
| 3  | charlie |
+----+---------+
```

```sh theme={null}
curl -X GET -u admin:admin http://localhost:9200/public/_search | jq .
```

```json theme={null}
{
  "took": 399,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "public-1",
        "_id": "cv9esaav80ig0h81pkjg_1",
        "_score": 1.0,
        "_source": {
          "_table": "cv9esaav80ig0h81pkjg",
          "cv9esaav80ig0h81pkjg-2": "alice"
        }
      },
      {
        "_index": "public-1",
        "_id": "cv9esaav80ig0h81pkjg_2",
        "_score": 1.0,
        "_source": {
          "_table": "cv9esaav80ig0h81pkjg",
          "cv9esaav80ig0h81pkjg-2": "bob"
        }
      },
      {
        "_index": "public-1",
        "_id": "cv9esaav80ig0h81pkjg_3",
        "_score": 1.0,
        "_source": {
          "_table": "cv9esaav80ig0h81pkjg",
          "cv9esaav80ig0h81pkjg-2": "charlie"
        }
      }
    ]
  }
}
```

All other operations will be cascaded to both targets accordingly. For more details on each individual target, check out the [PostgreSQL to PostgreSQL replication](/docs/opensource/pgstream/docs/tutorials/postgres_to_postgres) and the [PostgreSQL to OpenSearch replication](/docs/opensource/pgstream/docs/tutorials/postgres_to_opensearch) tutorials.

## Troubleshooting

Here are some common issues you might encounter while following this tutorial and how to resolve them:

### 1. **Error: `Connection refused`**

* **Cause:** The PostgreSQL database, Kafka cluster, or OpenSearch cluster is not running.
* **Solution:**
  * Ensure the Docker containers for all services are running.
  * Verify the database, Kafka, and OpenSearch URLs in the configuration files.
  * Test the connections using the following commands:
    ```sh theme={null}
    psql postgresql://postgres:postgres@localhost:5432/postgres
    curl -X GET http://localhost:9200
    kafka-topics.sh --list --bootstrap-server localhost:9092
    ```

### 2. **Error: `Replication slot not found`**

* **Cause:** The replication slot was not created during initialization.
* **Solution:**
  * Reinitialize `pgstream` or manually create the replication slot.
  * Run the `pgstream status` command to validate the initialisation was successful.
  * Verify the replication slot exists by running:
    ```sql theme={null}
    SELECT slot_name FROM pg_replication_slots;
    ```

### 3. **Error: `Kafka topic not found`**

* **Cause:** The Kafka topic was not created automatically or the topic name is incorrect.
* **Solution:**
  * Ensure the `PGSTREAM_KAFKA_TOPIC_AUTO_CREATE` variable is set to `true` in the configuration.
  * Manually create the topic using the Kafka CLI:
    ```sh theme={null}
    kafka-topics.sh --create --topic pgstream --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
    ```

### 4. **Error: `Data not replicated to targets`**

* **Cause:** The Kafka consumer groups for PostgreSQL or OpenSearch are not configured correctly.
* **Solution:**
  * Verify the consumer group IDs in the configuration files.
  * Check the `pgstream` logs for errors:
    ```sh theme={null}
    pgstream run -c kafka2pg_tutorial.env --log-level trace
    pgstream run -c kafka2os_tutorial.env --log-level trace
    ```

### 5. **Error: `Permission denied`**

* **Cause:** The database user does not have sufficient privileges.
* **Solution:**
  * Grant the required privileges to the database user:
    ```sql theme={null}
    GRANT ALL PRIVILEGES ON DATABASE postgres TO postgres;
    ```

### 6. **Error: `OpenSearch index not created`**

* **Cause:** The OpenSearch processor is not configured correctly.
* **Solution:**
  * Verify the OpenSearch URL in the configuration file.
  * Check the `pgstream` logs for errors:
    ```sh theme={null}
    pgstream run -c kafka2os_tutorial.env --log-level trace
    ```

### 7. **Error: `Kafka TLS connection failed`**

* **Cause:** The Kafka TLS certificates are not configured correctly.
* **Solution:**
  * Verify the paths to the certificate files in the configuration:
    ```sh theme={null}
    PGSTREAM_KAFKA_TLS_CA_CERT_FILE="broker.cer.pem"
    PGSTREAM_KAFKA_TLS_CLIENT_CERT_FILE="client.cer.pem"
    PGSTREAM_KAFKA_TLS_CLIENT_KEY_FILE="client.key.pem"
    ```
  * Ensure the certificates are valid and match the Kafka server configuration.

### 8. **Error: `SASL Authentication failed`**

* **Cause:** The Kafka SASL credentials are wrong, or the broker does not offer the mechanism.
* **Solution:**
  * Verify the user and the password in the configuration:
    ```sh theme={null}
    PGSTREAM_KAFKA_SASL_MECHANISM="scram-sha-512"
    PGSTREAM_KAFKA_SASL_USER="myuser"
    PGSTREAM_KAFKA_SASL_PASSWORD="mypassword"
    ```
  * Ensure the broker offers the mechanism. The broker lists the mechanisms it accepts in its `sasl.enabled.mechanisms` setting.
  * Set the same four variables for every `pgstream` instance. The Kafka reader and the Kafka writer authenticate separately.

If you encounter issues not listed here, consult the [pgstream documentation](https://github.com/xataio/pgstream) or open an issue on the project's GitHub repository.

## Summary

In this tutorial, we successfully configured `pgstream` to replicate data from a PostgreSQL database to multiple targets (PostgreSQL and OpenSearch) using Kafka as an intermediary. We:

1. Set up the source PostgreSQL database, target PostgreSQL database, OpenSearch cluster, and Kafka cluster.
2. Initialized `pgstream` on the source database, creating the necessary schema and replication slot.
3. Configured three separate `pgstream` instances:
   * PostgreSQL to Kafka
   * Kafka to PostgreSQL
   * Kafka to OpenSearch
4. Verified that schema changes and data changes were replicated correctly across all targets.

This tutorial demonstrates how `pgstream` can leverage Kafka for scalable, real-time replication to multiple targets. For more advanced use cases, refer to the [pgstream tutorials](/docs/opensource/pgstream/overview#tutorials).


# PostgreSQL snapshots
Source: https://xata.io/docs/opensource/pgstream/docs/tutorials/postgres_snapshot

## Introduction

This tutorial will showcase the use of pgstream to snapshot data from a PostgreSQL database. For this tutorial, we'll use a PostgreSQL target.

![snapshot2pg tutorial](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_tutorial_snapshot2pg.svg)

### Requirements

* A source PostgreSQL database
* A target PostgreSQL database
* pgstream (see [installation](/docs/opensource/pgstream/docs/cli#installation) instructions for more details)

## Environment setup

The first step is to start the two PostgreSQL databases that will be used as source and target for the snapshot. The `pgstream` repository provides a docker installation that will be used for the purposes of this tutorial, but can be replaced by any available PostgreSQL servers.

To start the docker provided PostgreSQL servers, run the following command:

```sh theme={null}
# Start two PostgreSQL databases using Docker.
# The source database will run on port 5432, and the target database will run on port 7654.
docker-compose -f build/docker/docker-compose.yml --profile pg2pg up
```

This will start two PostgreSQL databases on ports `5432` and `7654`.

## Database initialisation

Normally we need to initialise pgstream on the source database. The initialisation step creates the `pgstream` schema in the configured Postgres database, along with the tables/functions/triggers required for DDL tracking. It also creates the replication slot. However, this is only required if we're going to be using the replication slot. If we're using a PostgreSQL target for the snapshot, pgstream supports using `pg_dump`/`pg_restore` for the schema snapshot, which removes the need to keep any pgstream state in the source PostgreSQL database.

## Prepare `pgstream` configuration

### Listener

In order to run pgstream, we need to provide the configuration required to run the PostgreSQL to PostgreSQL snapshot. First, we configure the listener module that will be producing the snapshot of the source PostgreSQL database. This requires the PostgreSQL database URL, which will be the one from the docker PostgreSQL server we started and setup in the previous steps.

```sh theme={null}
# URL of the source PostgreSQL database. This is where the snapshot will be taken from.
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
```

The snapshot listener needs to be configured to indicate the tables/schemas that need to be snapshotted. If the tables are not schema qualified, the `public` schema will be assumed. Wildcards are supported. For example, `test_schema.*` will snapshot all tables in the `test_schema` schema, and `test` will snapshot the `public.test` table.

```sh theme={null}
# Tables to snapshot. Use "*" to snapshot all tables in the public schema.
PGSTREAM_POSTGRES_SNAPSHOT_TABLES="*"
```

In order to exclude some tables, e.g to snapshot all the tables in the public schema except for `test_table` and `test_table_2`, `PGSTREAM_POSTGRES_SNAPSHOT_EXCLUDED_TABLES` can be used.

```sh theme={null}
PGSTREAM_POSTGRES_SNAPSHOT_TABLES="public.*"
PGSTREAM_POSTGRES_SNAPSHOT_EXCLUDED_TABLES="public.test_table public.test_table_2"
```

Further configuration can be provided to optimize the performance of the snapshot process, mostly focusing on the concurrency.

```sh theme={null}
# Number of tables being snapshotted in parallel for a given schema. Defaults to 4.
PGSTREAM_POSTGRES_SNAPSHOT_SCHEMA_WORKERS=4
# Number of concurrent workers that will be used per table by the snapshotting process. Defaults to 4.
PGSTREAM_POSTGRES_SNAPSHOT_TABLE_WORKERS=4
# Size of the batches the table workers from PGSTREAM_POSTGRES_SNAPSHOT_TABLE_WORKERS will read concurrently. Defaults to 80MiB.
PGSTREAM_POSTGRES_SNAPSHOT_BATCH_BYTES=83886080
# Number of schemas that will be processed in parallel by the snapshotting process. Defaults to 1.
PGSTREAM_POSTGRES_SNAPSHOT_WORKERS=1
```

The snapshot listener can also be configured to record and update the status of snapshot requests in a dedicated table `snapshot_requests` under the `pgstream` schema. This allows to only perform a given snapshot once by keeping track of what's already been completed. All it's needed is the URL of the database where the table should be created. For this tutorial, we'll use the source database.

```sh theme={null}
PGSTREAM_POSTGRES_SNAPSHOT_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
```

### Processor

With the listener side ready, the next step is to configure the processor. Since we want the snapshot reach a PostgreSQL database, we will set the PostgreSQL writer configuration variables. The only required value is the URL of the target database, where the snapshotted schema/data from the source database will be streamed. We use the URL of the docker PostgreSQL database we started earlier (note the port is the only difference between the source and the target PostgreSQL databases).

```sh theme={null}
PGSTREAM_POSTGRES_WRITER_TARGET_URL="postgres://postgres:postgres@localhost:7654?sslmode=disable"
```

If we need to disable triggers on the target database during the snapshot process(ie., to avoid foreign key constraint violations), we can do so by setting the following variable:

```sh theme={null}
PGSTREAM_POSTGRES_WRITER_DISABLE_TRIGGERS=true
```

For further granularity, we can also configure the action that should be taken when an insert has a conflict.

```sh theme={null}
# Insert on conflict action. Options are update, nothing or error (error is the default behaviour)
PGSTREAM_POSTGRES_WRITER_ON_CONFLICT_ACTION=nothing
```

The PostgreSQL writer uses batching under the hood to reduce the number of IO calls to the target database and improve performance. The batch size and send timeout can both be configured to be able to better fit the different traffic patterns. The writer will send a batch when the timeout or the batch size is reached, whichever happens first.

```sh theme={null}
# Number of DML queries that will be batched and sent together in a given transaction. It defaults to 100.
PGSTREAM_POSTGRES_WRITER_BATCH_SIZE=25

# Max delay between batch sending. The batches will be sent every 5s by default.
PGSTREAM_POSTGRES_WRITER_BATCH_TIMEOUT=5s
```

The full configuration for this tutorial can be put into a `snapshot2pg_tutorial.env` file to be used in the next step. An equivalent `snapshot2pg_tutorial.yaml` configuration can be found below the environment one, and can be used interchangeably.

```sh theme={null}
# Listener config
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_SNAPSHOT_TABLES="*"
PGSTREAM_POSTGRES_SNAPSHOT_SCHEMA_WORKERS=4
PGSTREAM_POSTGRES_SNAPSHOT_TABLE_WORKERS=4
PGSTREAM_POSTGRES_SNAPSHOT_BATCH_BYTES=83886080
PGSTREAM_POSTGRES_SNAPSHOT_WORKERS=1
PGSTREAM_POSTGRES_SNAPSHOT_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"

# Processor config
PGSTREAM_POSTGRES_WRITER_TARGET_URL="postgres://postgres:postgres@localhost:7654?sslmode=disable"
PGSTREAM_POSTGRES_WRITER_BATCH_SIZE=25
PGSTREAM_POSTGRES_WRITER_BATCH_TIMEOUT=5s
PGSTREAM_POSTGRES_WRITER_DISABLE_TRIGGERS=true
PGSTREAM_POSTGRES_WRITER_ON_CONFLICT_ACTION=nothing
```

```yaml theme={null}
source:
  postgres:
    url: "postgres://postgres:postgres@localhost:5432?sslmode=disable"
    mode: snapshot # options are replication, snapshot or snapshot_and_replication
    snapshot: # when mode is snapshot or snapshot_and_replication
      mode: full # options are data_and, schema or data
      tables: ["*"] # tables to snapshot, can be a list of table names or a pattern
      recorder:
        repeatable_snapshots: true # whether to repeat snapshots that have already been taken
        postgres_url: "postgres://postgres:postgres@localhost:5432?sslmode=disable" # URL of the database where the snapshot status is recorded
      snapshot_workers: 4 # number of schemas to be snapshotted in parallel
      data: # when mode is full or data
        schema_workers: 4 # number of schema tables to be snapshotted in parallel
        table_workers: 4 # number of workers to snapshot a table in parallel
        batch_bytes: 83886080 # bytes to read per batch (defaults to 80MiB)
      schema: # when mode is full or schema
        pgdump_pgrestore:
          clean_target_db: false # whether to clean the target database before restoring

target:
  postgres:
    url: "postgres://postgres:postgres@localhost:7654?sslmode=disable"
    batch:
      timeout: 5000 # batch timeout in milliseconds
      size: 25 # number of messages in a batch
    disable_triggers: true # whether to disable triggers on the target database
    on_conflict_action: "nothing" # options are update, nothing or error
```

## Preparing snapshot data

Now we can connect to the source database and create a table and populate it with some data that we'll want to snapshot.

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:5432/postgres
```

```sql theme={null}
CREATE TABLE test(id SERIAL PRIMARY KEY, name TEXT);
INSERT INTO test(name) VALUES('alice'),('bob'),('charlie');
```

## Perform snapshot

With the configuration ready, we can now run pgstream. In this case we set the log level as trace to provide more context for debugging and have more visibility into what pgstream is doing under the hood. Once the snapshot finishes, the process will stop.

```sh theme={null}
# with the environment configuration
pgstream snapshot -c snapshot2pg_tutorial.env --log-level trace

# with the yaml configuration
pgstream snapshot -c snapshot2pg_tutorial.yaml --log-level trace

# with the CLI flags and relying on defaults
pgstream snapshot --postgres-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --target postgres --target-url "postgres://postgres:postgres@localhost:7654?sslmode=disable" --tables "*" --log-level trace
```

## Verify snapshot

If we connect to the target database, we should now see the test table created and populated with the data from the snapshot.

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:7654/postgres
```

```sql theme={null}
 \d+ test
+--------+---------+-----------+----------+--------------+-------------+
| Column | Type    | Modifiers | Storage  | Stats target | Description |
|--------+---------+-----------+----------+--------------+-------------|
| id     | integer |  not null | plain    | <null>       | <null>      |
| name   | text    |           | extended | <null>       | <null>      |
+--------+---------+-----------+----------+--------------+-------------+
Indexes:
    "test_pkey" PRIMARY KEY, btree (id)
Has OIDs: no
```

```sql theme={null}
postgres@localhost:postgres> SELECT * FROM test;
+----+---------+
| id | name    |
|----+---------|
| 1  | alice   |
| 2  | bob     |
| 3  | charlie |
+----+---------+
```

We can also check the status of the snapshot by querying the `pgstream.snapshot_requests` table on the source database. It should contain the details of the snapshot that has been requested.

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:5432/postgres
```

```sql theme={null}
postgres@localhost:postgres> SELECT * FROM pgstream.snapshot_requests;
+--------+-------------+-------------+-------------------------------+-------------------------------+-----------+--------+
| req_id | schema_name | table_names | created_at                    | updated_at                    | status    | errors |
|--------+-------------+-------------+-------------------------------+-------------------------------+-----------+--------|
| 1      | public      | ['*']       | 2025-03-18 11:27:21.683361+00 | 2025-03-18 11:27:21.843207+00 | completed | <null> |
+--------+-------------+-------------+-------------------------------+-------------------------------+-----------+--------+
```

If the snapshot has completed with errors, it will be retried on the next run of pgstream.

## Troubleshooting

Here are some common issues you might encounter while following this tutorial and how to resolve them:

### 1. **Error: `Connection refused`**

* **Cause:** The PostgreSQL database is not running or the connection URL is incorrect.
* **Solution:**
  * Ensure the Docker containers for the source and target databases are running.
  * Verify the database URLs in the configuration file (`snapshot2pg_tutorial.env`).
  * Test the connection using `psql`:
    ```sh theme={null}
    psql postgresql://postgres:postgres@localhost:5432/postgres
    ```

### 2. **Error: `Snapshot failed`**

* **Cause:** There may be issues with the configuration or permissions.
* **Solution:**
  * Check the pgstream logs for detailed error messages:
    ```sh theme={null}
    pgstream run -c snapshot2pg_tutorial.env --log-level trace
    ```
  * Ensure the source database user has the necessary permissions to read the schema and data.
  * Check the `pgstream.snapshot_requests` table for error details:
    ```sql theme={null}
    SELECT * FROM pgstream.snapshot_requests;
    ```

### 3. **Error: `Target database does not contain the snapshot data`**

* **Cause:** The snapshot process did not complete successfully or the target database URL is incorrect.
* **Solution:**
  * Verify the target database URL in the configuration file.

  * Check the pgstream logs to confirm the snapshot process completed without errors.

  * Check the `pgstream.snapshot_requests` table for error details:

    ```sql theme={null}
    SELECT * FROM pgstream.snapshot_requests;
    ```

  * Query the target database to ensure the data was replicated:
    ```sql theme={null}
    SELECT * FROM test;
    ```

### 4. **Error: `pgstream: invalid configuration`**

* **Cause:** The configuration file contains invalid or missing values.
* **Solution:**
  * Double-check the `snapshot2pg_tutorial.env` file for typos or missing variables.
  * Refer to the [pgstream configuration documentation](/docs/opensource/pgstream/docs/configuration) for details on required variables.
  * Run the `pgstream status` command to validate the configuration is correct.

### 5. **Error: `Permission denied`**

* **Cause:** The database user does not have sufficient privileges.
* **Solution:**
  * Ensure the user has the necessary permissions to create tables, replication slots, and perform snapshots.
  * Grant the required privileges:
    ```sql theme={null}
    GRANT ALL PRIVILEGES ON DATABASE postgres TO postgres;
    ```

If you encounter issues not listed here, consult the [pgstream documentation](/docs/opensource/pgstream/overview) or open an issue on the project's GitHub repository.

## Summary

In this tutorial, we successfully configured `pgstream` to snapshot data from a source PostgreSQL database to a target PostgreSQL database. We covered the following steps:

1. Set up the source and target PostgreSQL databases using Docker.
2. Configured the `pgstream` listener and processor for snapshotting.
3. Created a sample table in the source database and populated it with data.
4. Ran `pgstream` to perform the snapshot.
5. Verified that the data was successfully replicated to the target database.

This process demonstrates how `pgstream` can be used to efficiently snapshot data between PostgreSQL databases. For more advanced use cases, such as continuous replication or applying transformations, refer to the other [pgstream tutorials](/docs/opensource/pgstream/overview#tutorials).


# Replication to OpenSearch
Source: https://xata.io/docs/opensource/pgstream/docs/tutorials/postgres_to_opensearch

## Introduction

This tutorial will showcase the use of pgstream to replicate data from a PostgreSQL database to an OpenSearch cluster.

![pg2os tutorial](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_tutorial_pg2os.svg)

### Requirements

* A source PostgreSQL database
* A target OpenSearch cluster
* pgstream (see [installation](/docs/opensource/pgstream/docs/cli#installation) instructions for more details)

### Demo

[YouTube video player](https://www.youtube.com/embed/HC0LeLjD220)

## Environment setup

The first step is to start the PostgreSQL database that will be used as source and the OpenSearch cluster that will be the target for replication. The `pgstream` repository provides a docker installation that will be used for the purposes of this tutorial, but can be replaced by any available PostgreSQL server, as long as they have [`wal2json`](https://github.com/eulerto/wal2json) installed, and any OpenSearch cluster.

To start the docker provided PostgreSQL servers, run the following command:

```sh theme={null}
docker-compose -f build/docker/docker-compose.yml --profile pg2os up
```

This will start a PostgreSQL database on ports `5432` and an OpenSearch cluster on port `9200`. It also starts OpenSearch dashboards UI on port `5601` to simplify visualisation of the OpenSearch documents, but we'll be relying on curl for this tutorial.

## Database initialisation

Once both the PostgreSQL server and the OpenSearch cluster are up and running, the next step is to initialise pgstream on the source database. This will create the `pgstream` schema in the configured Postgres database, along with the tables/functions/triggers required to keep track of the schema changes. See [Tracking schema changes](/docs/opensource/pgstream/docs/architecture#tracking-schema-changes) section for more details. This step will also create a replication slot on the source database which will be used by the pgstream service.

The initialisation step allows to provide both the URL of the PostgreSQL database and the name of the replication slot to be created. The PostgreSQL URL is required, but the replication slot name is optional. If not provided, it will default to `pgstream_<dbname>_slot`, where `<dbname>` is the name of the PostgreSQL database. The configuration can be provided either by using the CLI supported parameters, or using the environment variables.

For this tutorial, we'll create a replication slot with the name `pgstream_tutorial_slot`.

* Using the `--init` flag in the `run` command

* Using CLI parameters:

```sh theme={null}
pgstream init --postgres-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --replication-slot pgstream_tutorial_slot
```

* Using environment variables:

```sh theme={null}
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot PGSTREAM_POSTGRES_LISTENER_URL=postgres://postgres:postgres@localhost:5432?sslmode=disable pgstream init
```

Successful initialisation should prompt the following message:

```
SUCCESS  pgstream initialisation complete
```

If at any point the initialisation performed by pgstream needs to be reverted, all state will be removed by running the `destroy` CLI command.

```sh theme={null}
pgstream destroy --postgres-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --replication-slot pgstream_tutorial_slot
```

## Prepare `pgstream` configuration

### Listener

In order to run pgstream, we need to provide the configuration required to run the PostgreSQL to OpenSearch replication. First, we configure the listener module that will be listening to the WAL on the source PostgreSQL database. This requires the PostgreSQL database URL, which will be the one from the docker PostgreSQL server we started and setup in the previous steps.

```sh theme={null}
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
```

Since we've set a custom replication slot name, the configuration variable needs to be set accordingly so that it doesn't use the default value.

```sh theme={null}
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot
```

The PostgreSQL WAL listener can be configured to perform an initial snapshot of the existing PostgreSQL database tables before starting to listen on the replication slot. In this case, we have no existing tables, so we don't need to configure the initial snapshot.

However, if there were tables with pre-existing data that we wanted to replicate to the target PostgreSQL database, we could configure it by setting the following environment variables:

```sh theme={null}
# URL of the PostgreSQL database we want to snapshot
PGSTREAM_POSTGRES_SNAPSHOT_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"

# List of tables we want to snapshot. If the tables are not schema qualified, the public schema will be assumed.
# Wildcards are supported.
#
# The following example will snapshot all tables in the `test_schema` and the table `test` from the public schema.
PGSTREAM_POSTGRES_SNAPSHOT_TABLES="test_schema.* test"
```

Further configuration can be provided to optimize the performance of the snapshot process. For more information, check the [snapshot tutorial](/docs/opensource/pgstream/docs/tutorials/postgres_snapshot).

### Processor

With the listener side ready, the next step is to configure the processor. Since we want to replicate to an OpenSearch cluster, we will need to set the search indexer configuration variables. The only required value is the URL of the cluster, where the replicated data from the source database will be written. We use the URL of the docker OpenSearch cluster we started earlier.

```sh theme={null}
PGSTREAM_OPENSEARCH_STORE_URL="http://admin:admin@localhost:9200"
```

The search indexer uses batching under the hood to reduce the number of IO calls to OpenSearch and improve performance. The batch size and send timeout can both be configured to be able to better fit the different traffic patterns. The indexer will send a batch when the timeout or the batch size is reached, whichever happens first.

```sh theme={null}
# Number of documents that will be batched and sent together in a given request. It defaults to 100.
PGSTREAM_SEARCH_INDEXER_BATCH_SIZE=25
# Max delay between batch sending. The batches will be sent every 5s by default.
PGSTREAM_SEARCH_INDEXER_BATCH_TIMEOUT=5s
```

The search indexer will apply exponential backoff by default for all the OpenSearch cluster requests. However, the backoff policy can be tweaked further to fit the behaviour to the specific needs of the application. The configuration for the exponential backoff can be altered, but it is also possible to configure a non exponential backoff instead as well. Only one of the two will be applied, giving priority to the exponential backoff configuration if both are provided.

```sh theme={null}
# The amount of time the initial exponential backoff interval will apply. Defaults to 1s.
PGSTREAM_SEARCH_STORE_EXP_BACKOFF_INITIAL_INTERVAL=1s
# The max amount of time the exponential backoff will retry for. Defaults to 1m.
PGSTREAM_SEARCH_STORE_EXP_BACKOFF_MAX_INTERVAL=1m
# Maximum amount of retries the exponential backoff will retry for. Defaults to 0.
PGSTREAM_SEARCH_STORE_EXP_BACKOFF_MAX_RETRIES=0

# Constant interval that the backoff policy will apply between retries. Defaults to 0.
PGSTREAM_SEARCH_STORE_BACKOFF_INTERVAL=0
# Max number of retries the backoff policy will apply. Defaults to 0.
PGSTREAM_SEARCH_STORE_BACKOFF_MAX_RETRIES=0
```

The search indexer requires some metadata to be added to the WAL events in order to identify the id of the event, which will be used to index the documents. In order to add this data to the events, we need to enable the injector processor, which takes care of adding pgstream metadata into the events. This includes pgstream IDs for columns, which will be used by the indexer instead of the column names to maintain constant identifiers when renames happen, helping with performance by preventing reindexes. Check out more details about the injector in the [architecture](/docs/opensource/pgstream/docs/architecture) section.

```sh theme={null}
PGSTREAM_INJECTOR_STORE_POSTGRES_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
```

The full configuration for this tutorial can be put into a `pg2os_tutorial.env` file to be used in the next step. An equivalent `pg2os_tutorial.yaml` configuration can be found below the environment one, and can be used interchangeably.

* Without initial snapshot

```sh theme={null}
# Listener config
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot

# Processor config
PGSTREAM_INJECTOR_STORE_POSTGRES_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_OPENSEARCH_STORE_URL="http://admin:admin@localhost:9200"
PGSTREAM_SEARCH_INDEXER_BATCH_SIZE=25
PGSTREAM_SEARCH_INDEXER_BATCH_TIMEOUT=5s
```

```yaml theme={null}
source:
  postgres:
    url: "postgres://postgres:postgres@localhost:5432?sslmode=disable"
    mode: replication
    replication:
      replication_slot: "pgstream_tutorial_slot"
target:
  search:
    engine: "opensearch" # options are elasticsearch or opensearch
    url: "http://localhost:9200" # URL of the search engine
    batch:
      timeout: 5000 # batch timeout in milliseconds
      size: 25 # number of messages in a batch
modifiers:
  injector:
    enabled: true # whether to inject pgstream metadata into the WAL events
```

* With initial snapshot

```sh theme={null}
# Listener config
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot
PGSTREAM_POSTGRES_SNAPSHOT_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
# Initial snapshot of all tables in the public schema
PGSTREAM_POSTGRES_SNAPSHOT_TABLES="*"

# Processor config
PGSTREAM_INJECTOR_STORE_POSTGRES_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_OPENSEARCH_STORE_URL="http://admin:admin@localhost:9200"
PGSTREAM_SEARCH_INDEXER_BATCH_SIZE=25
PGSTREAM_SEARCH_INDEXER_BATCH_TIMEOUT=5s
```

```yaml theme={null}
source:
  postgres:
    url: "postgresql://user:password@localhost:5432/mydatabase"
    mode: snapshot_and_replication # options are replication, snapshot or snapshot_and_replication
    snapshot: # when mode is snapshot or snapshot_and_replication
      mode: full # options are data_and, schema or data
      tables: ["*"] # tables to snapshot, can be a list of table names or a pattern
      recorder:
        repeatable_snapshots: true # whether to repeat snapshots that have already been taken
        postgres_url: "postgres://postgres:postgres@localhost:5432?sslmode=disable" # URL of the database where the snapshot status is recorded

    replication:
      replication_slot: "pgstream_tutorial_slot"
target:
  search:
    engine: "opensearch" # options are elasticsearch or opensearch
    url: "http://localhost:9200" # URL of the search engine
    batch:
      timeout: 5000 # batch timeout in milliseconds
      size: 25 # number of messages in a batch
modifiers:
  injector:
    enabled: true # whether to inject pgstream metadata into the WAL events
```

## Validate `pgstream` status

We can validate that the initialisation and the configuration are valid by running the `status` command before starting `pgstream`.

```sh theme={null}
# using yaml configuration file
./pgstream status -c pg2os_tutorial.yaml
# using env configuration file
./pgstream status -c pg2os_tutorial.env
```

```sh theme={null}
SUCCESS  pgstream status check encountered no issues
Initialisation status:
 - Pgstream schema exists: true
 - Migration current version: 7
 - Migration status: success
 - Replication slot name: pgstream_tutorial_slot
 - Replication slot plugin: wal2json
 - Replication slot database: postgres
Config status:
 - Valid: true
Transformation rules status:
 - Valid: true
Source status:
 - Reachable: true
```

## Run `pgstream`

With the configuration ready, we can now run pgstream. In this case we set the log level as trace to provide more context for debugging and have more visibility into what pgstream is doing under the hood.

```sh theme={null}
# with initialization
pgstream run -c pg2os_tutorial.env --init --log-level trace

# with the environment configuration
pgstream run -c pg2os_tutorial.env --log-level trace

# with the yaml configuration
pgstream run -c pg2os_tutorial.yaml --log-level trace

# with the CLI flags and relying on defaults
pgstream run --source postgres --source-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --target opensearch --target-url "http://localhost:9200" --log-level trace
```

## Verify Replication

Now we can connect to the source database and create a table:

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:5432/postgres
```

```sql theme={null}
CREATE TABLE test(id SERIAL PRIMARY KEY, name TEXT);
```

We should be able to see a `public-1` index created in the OpenSearch cluster. The `public-1` index is where the data for our tables in the public schema will be indexed.

```sh theme={null}
➜  ~ curl -X GET -u admin:admin http://localhost:9200/_cat/indices
green  open .opensearch-observability    -AUmVpfvQ0SyMi6-9ayTzA 1 0  0 0    208b    208b
yellow open security-auditlog-2025.03.13 ealaBDugReq6oxh-rg4bbw 1 1  7 0 105.4kb 105.4kb
yellow open public-1                     I0HbCjHsSCGxQ3O5aI2N4g 1 1  0 0    208b    208b
yellow open pgstream                     PWefOB2qQMa_Nv_47FMjMg 1 1  1 0   5.9kb   5.9kb
green  open .opendistro_security         mso1dkRtRlWFNEVGg72r3A 1 0 10 0    73kb    73kb
green  open .kibana_1                    ODNoFjfmT8enKvEi9tvRcw 1 0  0 0    208b    208b
```

If we look up the documents on the `pgstream` index we can see there's the version 1 of the schema.

```sh theme={null}
➜  ~ curl -X GET -u admin:admin http://localhost:9200/pgstream/_search | jq .
```

```json theme={null}
{
  "took": 73,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "pgstream",
        "_id": "cv9bopq2e0ig0vt9s3mg",
        "_score": 1.0,
        "_source": {
          "id": "cv9bopq2e0ig0vt9s3mg",
          "version": 1,
          "schema_name": "public",
          "created_at": "2025-03-13 11:07:19.050485",
          "schema": "{\"tables\":[{\"oid\":\"16464\",\"name\":\"test\",\"columns\":[{\"name\":\"id\",\"type\":\"integer\",\"default\":\"nextval('public.test_id_seq'::regclass)\",\"nullable\":false,\"unique\":true,\"metadata\":null,\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0-1\"},{\"name\":\"name\",\"type\":\"text\",\"nullable\":true,\"unique\":false,\"metadata\":null,\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0-2\"}],\"primary_key_columns\":[\"id\"],\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0\"}]}",
          "acked": false
        }
      }
    ]
  }
}
```

And the `public-1` index (which we can refer to by its alias `public`) will show no documents yet.

```sh theme={null}
➜  ~ curl -X GET -u admin:admin http://localhost:9200/public/_search | jq .
```

```json theme={null}
{
  "took": 10,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}
```

If we now start inserting data into the source database, we will be able to see the documents being indexed on the target OpenSearch cluster.

```sql theme={null}
INSERT INTO test(name) VALUES('alice'),('bob'),('charlie');
```

```sh theme={null}
➜  ~ curl -X GET -u admin:admin http://localhost:9200/public/_search | jq .
```

```json theme={null}
{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "public-1",
        "_id": "cv9bopq2e0ig0vt9s3n0_1",
        "_score": 1.0,
        "_source": {
          "_table": "cv9bopq2e0ig0vt9s3n0",
          "cv9bopq2e0ig0vt9s3n0-2": "alice"
        }
      },
      {
        "_index": "public-1",
        "_id": "cv9bopq2e0ig0vt9s3n0_2",
        "_score": 1.0,
        "_source": {
          "_table": "cv9bopq2e0ig0vt9s3n0",
          "cv9bopq2e0ig0vt9s3n0-2": "bob"
        }
      },
      {
        "_index": "public-1",
        "_id": "cv9bopq2e0ig0vt9s3n0_3",
        "_score": 1.0,
        "_source": {
          "_table": "cv9bopq2e0ig0vt9s3n0",
          "cv9bopq2e0ig0vt9s3n0-2": "charlie"
        }
      }
    ]
  }
}
```

As we can see in the documents, instead of column names we're relying on the pgstream ids associated to each column, so that renames don't trigger a reindex. The pgstream id of the table is used to build the pgstream ids of the columns, by appending the number to them. In this example, the pgstream id for the table `test` is `cv9bopq2e0ig0vt9s3n0`, and the `name` column pgstream id is `cv9bopq2e0ig0vt9s3n0-2`. The `id` column is not explicitly mapped, but instead used as the document `_id`, prefixed by the table pgstream id to prevent collisions with other tables. Since the `id` is the primary key for this table, it was used as the document id. If the table had a composite primary key, the individual columns would be part of the `_source`, and the document `_id` would be a combination of their values.

If we decide to update the schema of the table we will see the `pgstream` index will get a new document with a new version of the schema.

```sql theme={null}
ALTER TABLE test ADD COLUMN age INT DEFAULT 0;
```

```sh theme={null}
➜  ~ curl -X GET -u admin:admin http://localhost:9200/pgstream/_search | jq .
```

```json theme={null}
{
  "took": 8,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "pgstream",
        "_id": "cv9bopq2e0ig0vt9s3mg",
        "_score": 1.0,
        "_source": {
          "id": "cv9bopq2e0ig0vt9s3mg",
          "version": 1,
          "schema_name": "public",
          "created_at": "2025-03-13 11:07:19.050485",
          "schema": "{\"tables\":[{\"oid\":\"16464\",\"name\":\"test\",\"columns\":[{\"name\":\"id\",\"type\":\"integer\",\"default\":\"nextval('public.test_id_seq'::regclass)\",\"nullable\":false,\"unique\":true,\"metadata\":null,\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0-1\"},{\"name\":\"name\",\"type\":\"text\",\"nullable\":true,\"unique\":false,\"metadata\":null,\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0-2\"}],\"primary_key_columns\":[\"id\"],\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0\"}]}",
          "acked": false
        }
      },
      {
        "_index": "pgstream",
        "_id": "cv9btii2e0ig0vt9s3ng",
        "_score": 1.0,
        "_source": {
          "id": "cv9btii2e0ig0vt9s3ng",
          "version": 2,
          "schema_name": "public",
          "created_at": "2025-03-13 11:17:30.291352",
          "schema": "{\"tables\":[{\"oid\":\"16464\",\"name\":\"test\",\"columns\":[{\"name\":\"id\",\"type\":\"integer\",\"default\":\"nextval('public.test_id_seq'::regclass)\",\"nullable\":false,\"unique\":true,\"metadata\":null,\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0-1\"},{\"name\":\"name\",\"type\":\"text\",\"nullable\":true,\"unique\":false,\"metadata\":null,\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0-2\"},{\"name\":\"age\",\"type\":\"integer\",\"default\":\"0\",\"nullable\":true,\"unique\":false,\"metadata\":null,\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0-3\"}],\"primary_key_columns\":[\"id\"],\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0\"}]}",
          "acked": false
        }
      }
    ]
  }
}
```

Also, if we update the existing rows, we will see them updated on the existing document as well, along with the newly added column age (`cv9bopq2e0ig0vt9s3n0-3`). Since adding a new column with the fault doesn't trigger a table update, the existing documents will not be updated with the new column until they're updated.

```sql theme={null}
UPDATE test SET name='a' WHERE name='alice';
```

```sh theme={null}
➜  ~ curl -X GET -u admin:admin http://localhost:9200/public/_search | jq .
```

```json theme={null}
{
  "took": 907,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "public-1",
        "_id": "cv9bopq2e0ig0vt9s3n0_2",
        "_score": 1.0,
        "_source": {
          "_table": "cv9bopq2e0ig0vt9s3n0",
          "cv9bopq2e0ig0vt9s3n0-2": "bob"
        }
      },
      {
        "_index": "public-1",
        "_id": "cv9bopq2e0ig0vt9s3n0_3",
        "_score": 1.0,
        "_source": {
          "_table": "cv9bopq2e0ig0vt9s3n0",
          "cv9bopq2e0ig0vt9s3n0-2": "charlie"
        }
      },
      {
        "_index": "public-1",
        "_id": "cv9bopq2e0ig0vt9s3n0_1",
        "_score": 1.0,
        "_source": {
          "_table": "cv9bopq2e0ig0vt9s3n0",
          "cv9bopq2e0ig0vt9s3n0-2": "a",
          "cv9bopq2e0ig0vt9s3n0-3": 0
        }
      }
    ]
  }
}
```

Truncating a table will delete all documents associated with a table.

```sql theme={null}
TRUNCATE TABLE test;
```

```sh theme={null}
➜  ~ curl -X GET -u admin:admin http://localhost:9200/public/_search | jq .
```

```json theme={null}
{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}
```

And deleting a table will delete it from the pgstream schema log.

```sh theme={null}
➜  ~ curl -X GET -u admin:admin http://localhost:9200/pgstream/_search | jq .
```

```json theme={null}
{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "pgstream",
        "_id": "cv9bopq2e0ig0vt9s3mg",
        "_score": 1.0,
        "_source": {
          "id": "cv9bopq2e0ig0vt9s3mg",
          "version": 1,
          "schema_name": "public",
          "created_at": "2025-03-13 11:07:19.050485",
          "schema": "{\"tables\":[{\"oid\":\"16464\",\"name\":\"test\",\"columns\":[{\"name\":\"id\",\"type\":\"integer\",\"default\":\"nextval('public.test_id_seq'::regclass)\",\"nullable\":false,\"unique\":true,\"metadata\":null,\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0-1\"},{\"name\":\"name\",\"type\":\"text\",\"nullable\":true,\"unique\":false,\"metadata\":null,\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0-2\"}],\"primary_key_columns\":[\"id\"],\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0\"}]}",
          "acked": false
        }
      },
      {
        "_index": "pgstream",
        "_id": "cv9btii2e0ig0vt9s3ng",
        "_score": 1.0,
        "_source": {
          "id": "cv9btii2e0ig0vt9s3ng",
          "version": 2,
          "schema_name": "public",
          "created_at": "2025-03-13 11:17:30.291352",
          "schema": "{\"tables\":[{\"oid\":\"16464\",\"name\":\"test\",\"columns\":[{\"name\":\"id\",\"type\":\"integer\",\"default\":\"nextval('public.test_id_seq'::regclass)\",\"nullable\":false,\"unique\":true,\"metadata\":null,\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0-1\"},{\"name\":\"name\",\"type\":\"text\",\"nullable\":true,\"unique\":false,\"metadata\":null,\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0-2\"},{\"name\":\"age\",\"type\":\"integer\",\"default\":\"0\",\"nullable\":true,\"unique\":false,\"metadata\":null,\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0-3\"}],\"primary_key_columns\":[\"id\"],\"pgstream_id\":\"cv9bopq2e0ig0vt9s3n0\"}]}",
          "acked": false
        }
      },
      {
        "_index": "pgstream",
        "_id": "cv9c05a2e0ig0vt9s3o0",
        "_score": 1.0,
        "_source": {
          "id": "cv9c05a2e0ig0vt9s3o0",
          "version": 3,
          "schema_name": "public",
          "created_at": "2025-03-13 11:23:01.250382",
          "schema": "{\"tables\":null}",
          "acked": false
        }
      }
    ]
  }
}
```

Dropping the schema would delete the `public` OpenSearch index.

## Troubleshooting

### 1. **Error: `Connection refused`**

* **Cause:** The PostgreSQL database or OpenSearch cluster is not running.
* **Solution:**
  * Ensure the Docker containers are running.
  * Verify the database and OpenSearch URLs in the configuration.

### 2. **Error: `Replication slot not found`**

* **Cause:** The replication slot was not created during initialization.
* **Solution:**
  * Reinitialize `pgstream` or manually create the replication slot.
  * Run the `pgstream status` command to validate the initialisation was successful.
  * Verify the replication slot exists by running:
    ```sql theme={null}
    SELECT slot_name FROM pg_replication_slots;
    ```

### 3. **Error: `Data not replicated to OpenSearch`**

* **Cause:** The OpenSearch cluster URL is incorrect or the processor configuration is invalid.
* **Solution:**
  * Verify the OpenSearch URL in the configuration file.
  * Check the `pgstream` logs for errors:
    ```sh theme={null}
    pgstream run -c pg2os_tutorial.env --log-level trace
    ```

### 4. **Error: `Permission denied`**

* **Cause:** The database user does not have sufficient privileges.
* **Solution:**
  * Grant the required privileges to the database user:
    ```sql theme={null}
    GRANT ALL PRIVILEGES ON DATABASE postgres TO postgres;
    ```

If you encounter issues not listed here, consult the [pgstream documentation](https://github.com/xataio/pgstream) or open an issue on the project's GitHub repository.

## Summary

In this tutorial, we successfully configured `pgstream` to replicate data from a PostgreSQL database to an OpenSearch cluster. We:

1. Set up the source PostgreSQL database and target OpenSearch cluster.
2. Initialized `pgstream` and created a replication slot.
3. Configured the listener and processor for OpenSearch replication.
4. Verified that both schema changes and data changes were replicated correctly.

This tutorial demonstrates how `pgstream` can be used to integrate PostgreSQL with OpenSearch for real-time indexing. For more advanced use cases, refer to the [pgstream tutorials](/docs/opensource/pgstream/overview#tutorials).


# Replication to PostgreSQL
Source: https://xata.io/docs/opensource/pgstream/docs/tutorials/postgres_to_postgres

## Introduction

This tutorial demonstrates how to use pgstream to replicate data from one PostgreSQL database to another. It covers setting up the environment, configuring pgstream, and verifying the replication of both data and schema changes.

![pg2pg tutorial](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_tutorial_pg2pg.svg)

### Requirements

* A source PostgreSQL database
* A target PostgreSQL database
* pgstream (see [installation](/docs/opensource/pgstream/docs/cli#installation) instructions for more details)

### Demo

[YouTube video player](https://www.youtube.com/embed/WCv1ZZXkUnU)

## Environment setup

The first step is to start the two PostgreSQL databases that will be used as the source and target for replication. The pgstream repository provides a Docker setup for this purpose, but any PostgreSQL servers with [`wal2json`](https://github.com/eulerto/wal2json) installed can be used.

To start the docker provided PostgreSQL servers, run the following command:

```sh theme={null}
docker-compose -f build/docker/docker-compose.yml --profile pg2pg up
```

This will start two PostgreSQL databases:

* Source database on port `5432`
* Target database on port `7654`

## Database initialisation

Once both PostgreSQL servers are up and running, initialise pgstream on the source database. This step will create the `pgstream` schema in the configured Postgres database, along with the tables/functions/triggers required to keep track of the schema changes. See [Tracking schema changes](/docs/opensource/pgstream/docs/architecture#tracking-schema-changes) section for more details. It will also create a replication slot on the source database which will be used by the pgstream service.

The initialisation step allows to provide both the URL of the PostgreSQL database and the name of the replication slot to be created. The PostgreSQL URL is required, but the replication slot name is optional. If not provided, it will default to `pgstream_<dbname>_slot`, where `<dbname>` is the name of the PostgreSQL database. The configuration can be provided either via CLI parameters or environment variables.

For this tutorial, we'll create a replication slot with the name `pgstream_tutorial_slot`.

* Using the `--init` flag in the `run` command

* Using CLI parameters:

```sh theme={null}
pgstream init --postgres-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --replication-slot pgstream_tutorial_slot
```

* Using environment variables:

```sh theme={null}
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot \
PGSTREAM_POSTGRES_LISTENER_URL=postgres://postgres:postgres@localhost:5432?sslmode=disable \
pgstream init
```

After initialization, you should see the following message:

```
SUCCESS  pgstream initialisation complete
```

If at any point the initialisation performed by pgstream needs to be reverted, all state will be removed by running the `destroy` CLI command.

```sh theme={null}
pgstream destroy --postgres-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --replication-slot pgstream_tutorial_slot
```

## Prepare `pgstream` configuration

### Listener

The listener reads changes from the source database's WAL. Configure it as follows:

```sh theme={null}
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot
```

If you want to perform an initial snapshot of existing tables, add:

```sh theme={null}
# URL of the PostgreSQL database we want to snapshot
PGSTREAM_POSTGRES_SNAPSHOT_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
# List of tables we want to snapshot. If the tables are not schema qualified, the public schema will be assumed.
# Wildcards are supported.
#
# The following example will snapshot all tables in the `test_schema` and the table `test` from the public schema.
PGSTREAM_POSTGRES_SNAPSHOT_TABLES="test_schema.* test"
```

Further configuration can be provided to optimize the performance of the snapshot process. For more information, check the [snapshot tutorial](/docs/opensource/pgstream/docs/tutorials/postgres_snapshot).

### Processor

The processor writes changes to the target database. The only required value is the URL of the target database. Configure it as follows:

```sh theme={null}
PGSTREAM_POSTGRES_WRITER_TARGET_URL="postgres://postgres:postgres@localhost:7654?sslmode=disable"
```

If we need to disable triggers on the target database during the replication process(ie., to avoid foreign key constraint violations), we can do so by setting the following variable:

```sh theme={null}
PGSTREAM_POSTGRES_WRITER_DISABLE_TRIGGERS=true
```

For further granularity, we can also configure the action that should be taken when there's a conflict on an insert.

```sh theme={null}
# Insert on conflict action. Options are update, nothing or error (error is the default behaviour)
PGSTREAM_POSTGRES_WRITER_ON_CONFLICT_ACTION=nothing
```

The PostgreSQL writer uses batching under the hood to reduce the number of IO calls to the target database and improve performance. The batch size and send timeout can both be configured to be able to better fit the different traffic patterns. The writer will send a batch when the timeout or the batch size is reached, whichever happens first.

```sh theme={null}
# Number of DML queries that will be batched and sent together in a given transaction. It defaults to 100.
PGSTREAM_POSTGRES_WRITER_BATCH_SIZE=25
# Max delay between batch sending. The batches will be sent every 5s by default.
PGSTREAM_POSTGRES_WRITER_BATCH_TIMEOUT=5s
```

Save the configuration in a file named `pg2pg_tutorial.env`. An equivalent `pg2pg_tutorial.yaml` configuration can be found below the environment one, and can be used interchangeably.

```sh theme={null}
# Listener config
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot

# Processor config
PGSTREAM_POSTGRES_WRITER_TARGET_URL="postgres://postgres:postgres@localhost:7654?sslmode=disable"
PGSTREAM_POSTGRES_WRITER_BATCH_SIZE=25
PGSTREAM_POSTGRES_WRITER_BATCH_TIMEOUT=5s
PGSTREAM_POSTGRES_WRITER_DISABLE_TRIGGERS=true
PGSTREAM_POSTGRES_WRITER_ON_CONFLICT_ACTION=nothing
```

```yaml theme={null}
source:
  postgres:
    url: "postgres://postgres:postgres@localhost:5432?sslmode=disable"
    mode: replication # options are replication, snapshot or snapshot_and_replication
    replication:
      replication_slot: pgstream_tutorial_slot

target:
  postgres:
    url: "postgres://postgres:postgres@localhost:7654?sslmode=disable"
    batch:
      timeout: 5000 # batch timeout in milliseconds
      size: 25 # number of messages in a batch
    disable_triggers: false # whether to disable triggers on the target database
    on_conflict_action: "nothing" # options are update, nothing or error
```

## Validate `pgstream` status

We can validate that the initialisation and the configuration are valid by running the `status` command before starting `pgstream`.

```sh theme={null}
# using yaml configuration file
./pgstream status -c pg2pg_tutorial.yaml
# using env configuration file
./pgstream status -c pg2pg_tutorial.env
```

```sh theme={null}
SUCCESS  pgstream status check encountered no issues
Initialisation status:
 - Pgstream schema exists: true
 - Migration current version: 7
 - Migration status: success
 - Replication slot name: pgstream_tutorial_slot
 - Replication slot plugin: wal2json
 - Replication slot database: postgres
Config status:
 - Valid: true
Transformation rules status:
 - Valid: true
Source status:
 - Reachable: true
```

## Run `pgstream`

Run pgstream with the prepared configuration. In this case we set the log level as trace to provide more context for debugging and have more visibility into what pgstream is doing under the hood.

```sh theme={null}
# with initialization
pgstream run -c pg2pg_tutorial.env --init --log-level trace

# with the environment configuration
pgstream run -c pg2pg_tutorial.env --log-level trace

# with the yaml configuration
pgstream run -c pg2pg_tutorial.yaml --log-level trace

# with the CLI flags and relying on defaults
pgstream run --source postgres --source-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --target postgres --target-url "postgres://postgres:postgres@localhost:7654?sslmode=disable" --log-level trace
```

## Verify Replication

### Schema Changes

1. Connect to the source database:

   ```sh theme={null}
   psql postgresql://postgres:postgres@localhost:5432/postgres
   ```

2. Create a table:

   ```sql theme={null}
   CREATE TABLE test(id SERIAL PRIMARY KEY, name TEXT);
   ```

3. Connect to the target database:

   ```sh theme={null}
   psql postgresql://postgres:postgres@localhost:7654/postgres
   ```

4. Verify the table was replicated:

   ```sql theme={null}
    \d+ test
   +--------+---------+-----------+----------+--------------+-------------+
   | Column | Type    | Modifiers | Storage  | Stats target | Description |
   |--------+---------+-----------+----------+--------------+-------------|
   | id     | integer |  not null | plain    | <null>       | <null>      |
   | name   | text    |           | extended | <null>       | <null>      |
   +--------+---------+-----------+----------+--------------+-------------+
   ```

Similarly when performing other DDL operations, they should be properly replicated on the target database.

```sql theme={null}
ALTER TABLE test RENAME TO tutorial_test;
ALTER TABLE tutorial_test ADD COLUMN age INT DEFAULT 0;
ALTER TABLE tutorial_test ALTER COLUMN age TYPE bigint;
ALTER TABLE tutorial_test RENAME COLUMN age TO new_age;
ALTER TABLE tutorial_test DROP COLUMN new_age;
DROP TABLE tutorial_test;
```

### Data Changes

1. Insert data into the source database:

   ```sql theme={null}
   INSERT INTO test(name) VALUES('alice'),('bob'),('charlie');
   ```

2. Verify the data was replicated to the target database:
   ```sql theme={null}
   SELECT * FROM test;
   ```

## Troubleshooting

Here are some common issues you might encounter while following this tutorial and how to resolve them:

### 1. **Error: `Connection refused`**

* **Cause:** The PostgreSQL database is not running or the connection URL is incorrect.
* **Solution:**
  * Ensure the Docker containers for the source and target databases are running.
  * Verify the database URLs in the configuration file (`pg2pg_tutorial.env`).
  * Test the connection using `psql`:
    ```sh theme={null}
    psql postgresql://postgres:postgres@localhost:5432/postgres
    ```

### 2. **Error: `Replication slot not found`**

* **Cause:** The replication slot was not created during initialization.
* **Solution:**
  * Reinitialize `pgstream` or manually create the replication slot.
  * Run the `pgstream status` command to validate the initialisation was successful.
  * Verify the replication slot exists by running:
    ```sql theme={null}
    SELECT slot_name FROM pg_replication_slots;
    ```

### 3. **Error: `Target database does not contain replicated data`**

* **Cause:** The replication process did not complete successfully or the target database URL is incorrect.
* **Solution:**
  * Verify the target database URL in the configuration file.
  * Check the `pgstream` logs to confirm the replication process completed without errors.
  * Query the target database to ensure the data was replicated:
    ```sql theme={null}
    SELECT * FROM test;
    ```

### 4. **Error: `Permission denied`**

* **Cause:** The database user does not have sufficient privileges.
* **Solution:**
  * Grant the required privileges to the database user:
    ```sql theme={null}
    GRANT ALL PRIVILEGES ON DATABASE postgres TO postgres;
    ```

### 5. **Error: `Invalid configuration`**

* **Cause:** The configuration file contains invalid or missing values.
* **Solution:**
  * Double-check the `pg2pg_tutorial.env` file for typos or missing variables.
  * Refer to the [pgstream configuration documentation](https://github.com/xataio/pgstream) for details on required variables.
  * Run the `pgstream status` command to validate the configuration is correct.

### 6. **Error: `Snapshot failed`**

* **Cause:** The initial snapshot process encountered an issue.
* **Solution:**
  * Ensure the `PGSTREAM_POSTGRES_SNAPSHOT_TABLES` variable is set if a snapshot is required.
  * Check the `pgstream` logs for detailed error messages:
    ```sh theme={null}
    pgstream run -c pg2pg_tutorial.env --log-level trace
    ```

If you encounter issues not listed here, consult the [pgstream documentation](https://github.com/xataio/pgstream) or open an issue on the project's GitHub repository.

## Summary

In this tutorial, we successfully configured `pgstream` to replicate data from a source PostgreSQL database to a target PostgreSQL database. We:

1. Set up the source and target PostgreSQL databases using Docker.
2. Initialized `pgstream` on the source database, creating the necessary schema and replication slot.
3. Configured the listener to capture changes from the source database's WAL.
4. Configured the processor to replicate changes to the target database.
5. Verified that both DML (data manipulation) and DDL (schema changes) were replicated correctly.

This tutorial demonstrates how `pgstream` can be used for real-time replication between PostgreSQL databases. For more advanced use cases, such as transformations or webhook integration, refer to the [pgstream tutorials](/docs/opensource/pgstream/overview#tutorials).


# Replication to webhooks
Source: https://xata.io/docs/opensource/pgstream/docs/tutorials/postgres_to_webhooks

## Introduction

This tutorial will showcase the use of pgstream to replicate data from a PostgreSQL database to a webhook server. You can also check out this [blogpost](https://xata.io/blog/postgres-webhooks-with-pgstream) explaining how to use pgstream with webhooks.

![pg2webhooks tutorial](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_tutorial_pg2webhooks.svg)

### Requirements

* A source PostgreSQL database
* A target webhook server
* pgstream (see [installation](/docs/opensource/pgstream/docs/cli#installation) instructions for more details)

### Demo

[YouTube video player](https://www.youtube.com/embed/fewe85P5ktk)

## Environment setup

The first step is to start the PostgreSQL database that will be used as source for replication. The `pgstream` repository provides a docker installation that will be used for the purposes of this tutorial, but can be replaced by any available PostgreSQL server, as long as it has [`wal2json`](https://github.com/eulerto/wal2json) installed.

To start the docker provided PostgreSQL server, run the following command:

```sh theme={null}
docker-compose -f build/docker/docker-compose.yml --profile pg2webhook up
```

This will start a PostgreSQL databases on port `5432`.

## Database initialisation

Once the PostgreSQL server is up and running, the next step is to initialise pgstream. This will create the `pgstream` schema in the configured Postgres database, along with the tables/functions/triggers required to keep track of the schema changes. See [Tracking schema changes](/docs/opensource/pgstream/docs/architecture#tracking-schema-changes) section for more details. This step will also create a replication slot on the source database which will be used by the pgstream service.

The initialisation step allows to provide both the URL of the PostgreSQL database and the name of the replication slot to be created. The PostgreSQL URL is required, but the replication slot name is optional. If not provided, it will default to `pgstream_<dbname>_slot`, where `<dbname>` is the name of the PostgreSQL database. The configuration can be provided either by using the CLI supported parameters, or using the environment variables.

For this tutorial, we'll create a replication slot with the name `pgstream_tutorial_slot`.

* Using the `--init` flag in the `run` command

* Using CLI parameters:

```sh theme={null}
pgstream init --postgres-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --replication-slot pgstream_tutorial_slot
```

* Using environment variables:

```sh theme={null}
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot PGSTREAM_POSTGRES_LISTENER_URL=postgres://postgres:postgres@localhost:5432?sslmode=disable pgstream init
```

Successful initialisation should prompt the following message:

```
SUCCESS  pgstream initialisation complete
```

If at any point the initialisation performed by pgstream needs to be reverted, all state will be removed by running the `destroy` CLI command.

```sh theme={null}
pgstream destroy --postgres-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --replication-slot pgstream_tutorial_slot
```

## Prepare `pgstream` configuration

### Listener

In order to run pgstream, we need to provide the configuration required to run the PostgreSQL to webhook server replication. First, we configure the listener module that will be listening to the WAL on the source PostgreSQL database. This requires the PostgreSQL database URL, which will be the one from the docker PostgreSQL server we started and setup in the previous steps.

```sh theme={null}
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
```

Since we've set a custom replication slot name, the configuration variable needs to be set accordingly so that it doesn't use the default value.

```sh theme={null}
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot
```

The PostgreSQL WAL listener can be configured to perform an initial snapshot of the existing PostgreSQL database tables before starting to listen on the replication slot. In this case, we have no existing tables, so we don't need to configure the initial snapshot.

However, if there were tables with pre-existing data that we wanted to replicate to the webhooks server, we could configure it by setting the following environment variables:

```sh theme={null}
# URL of the PostgreSQL database we want to snapshot
PGSTREAM_POSTGRES_SNAPSHOT_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"

# List of tables we want to snapshot. If the tables are not schema qualified, the public schema will be assumed.
# Wildcards are supported.
#
# The following example will snapshot all tables in the `test_schema` and the table `test` from the public schema.
PGSTREAM_POSTGRES_SNAPSHOT_TABLES="test_schema.* test"
```

Further configuration can be provided to optimize the performance of the snapshot process. For more information, check the [snapshot tutorial](/docs/opensource/pgstream/docs/tutorials/postgres_snapshot#listener).

### Processor

With the listener side ready, the next step is to configure the processor. Since we want to replicate to a webhook server, we need to configure the webhooks processor. The webhooks processor runs a server that accepts webhook subscriptions, to keep track of which webhooks need to be called when an event is received. We need to configure where we want that subscriptions table to be stored, usually the source database will be a good option.

```sh theme={null}
PGSTREAM_WEBHOOK_SUBSCRIPTION_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
```

The subscription store can be configured to use caching, to minimise the number of calls we make to the database, since this would need to be called on a per event basis, which can be a lot in high traffic workloads. It is also helpful if the subscriptions don't change often, since the cache would be up to date most of the time.

```sh theme={null}
PGSTREAM_WEBHOOK_SUBSCRIPTION_STORE_CACHE_ENABLED=true
# How often the subscription store cache will retrieved all subscriptions from the sql store, and update its state. In the worst case scenario, this represent the staleness of the cache. Defaults to 60s.
PGSTREAM_WEBHOOK_SUBSCRIPTION_STORE_CACHE_REFRESH_INTERVAL="60s"
```

Save the configuration in a file named `pg2webhook_tutorial.env`. An equivalent `pg2webhook_tutorial.yaml` configuration can be found below the environment one, and can be used interchangeably.

* Without initial snapshot

```sh theme={null}
# Listener config
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot

# Processor config
PGSTREAM_WEBHOOK_SUBSCRIPTION_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_WEBHOOK_SUBSCRIPTION_STORE_CACHE_ENABLED=true
PGSTREAM_WEBHOOK_SUBSCRIPTION_STORE_CACHE_REFRESH_INTERVAL="60s"
```

```yaml theme={null}
source:
  postgres:
    url: "postgres://postgres:postgres@localhost:5432?sslmode=disable"
    mode: replication # options are replication, snapshot or snapshot_and_replication
    replication:
      replication_slot: pgstream_tutorial_slot

target:
  webhooks:
    subscriptions:
      store:
        url: "postgres://postgres:postgres@localhost:5432?sslmode=disable" # URL of the database where the webhook subscriptions are stored
        cache:
          enabled: true # whether to enable caching for the subscription store
          refresh_interval: 60 # interval in seconds to refresh the cache

modifiers:
  injector:
    enabled: true # whether to inject pgstream metadata into the WAL events
```

* With initial snapshot

```sh theme={null}
# Listener config
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot
PGSTREAM_POSTGRES_SNAPSHOT_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
# Initial snapshot of all tables in the public schema
PGSTREAM_POSTGRES_SNAPSHOT_TABLES="*"

# Processor config
PGSTREAM_WEBHOOK_SUBSCRIPTION_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_WEBHOOK_SUBSCRIPTION_STORE_CACHE_ENABLED=true
PGSTREAM_WEBHOOK_SUBSCRIPTION_STORE_CACHE_REFRESH_INTERVAL="60s"
```

```yaml theme={null}
source:
  postgres:
    url: "postgres://postgres:postgres@localhost:5432?sslmode=disable"
    mode: replication # options are replication, snapshot or snapshot_and_replication
    replication:
      replication_slot: pgstream_tutorial_slot
    snapshot: # when mode is snapshot or snapshot_and_replication
      mode: full # options are data_and, schema or data
      tables: ["*"] # tables to snapshot, can be a list of table names or a pattern
      recorder:
        repeatable_snapshots: true # whether to repeat snapshots that have already been taken
        postgres_url: "postgres://postgres:postgres@localhost:5432?sslmode=disable" # URL of the database where the snapshot status is recorded
      schema: # when mode is full or schema
        pgdump_pgrestore:
          clean_target_db: false # whether to clean the target database before restoring
target:
  webhooks:
    subscriptions:
      store:
        url: "postgres://postgres:postgres@localhost:5432?sslmode=disable" # URL of the database where the webhook subscriptions are stored
        cache:
          enabled: true # whether to enable caching for the subscription store
          refresh_interval: 60 # interval in seconds to refresh the cache

modifiers:
  injector:
    enabled: true # whether to inject pgstream metadata into the WAL events
```

## Validate `pgstream` status

We can validate that the initialisation and the configuration are valid by running the `status` command before starting `pgstream`.

```sh theme={null}
# using yaml configuration file
./pgstream status -c pg2webhook_tutorial.yaml
# using env configuration file
./pgstream status -c pg2webhook_tutorial.env
```

```sh theme={null}
SUCCESS  pgstream status check encountered no issues
Initialisation status:
 - Pgstream schema exists: true
 - Migration current version: 7
 - Migration status: success
 - Replication slot name: pgstream_tutorial_slot
 - Replication slot plugin: wal2json
 - Replication slot database: postgres
Config status:
 - Valid: true
Transformation rules status:
 - Valid: true
Source status:
 - Reachable: true
```

## Run `pgstream`

With the configuration ready, we can now run pgstream. In this case we set the log level as trace to provide more context for debugging and have more visibility into what pgstream is doing under the hood.

```sh theme={null}
# with initialization
pgstream run -c pg2webhook_tutorial.env --init --log-level trace

# with the environment configuration
pgstream run -c pg2webhook_tutorial.env --log-level trace

# with the yaml configuration
pgstream run -c pg2webhook_tutorial.yaml --log-level trace
```

Once pgstream is running, we need to make sure the webhook server we want to use is started and ready to accept requests. For the purposes of this tutorial, we will use a dummy webhook server provided in this repository, under the `/tools/webhook` directory, which will listen on `localhost:9910`. This dummy webhook server just prints the events received to output in JSON format for validation.

```sh theme={null}
cd tools/webhook
go build
./webhook
```

```sh theme={null}
2025-03-13T10:31:44.18911+01:00 INF logger.go:37 > listening on :9910...
```

Once we have pgstream and the webhook server running, all we need to do is register a subscription to the events we're interested in for our dummy webhook server. The subscriptions server that the pgstream webhook processor runs exposes the endpoint `http://localhost:9900/webhooks/subscribe`, so that's where we should send the requests. In this example we'll subscribe to insert events for all tables.

```sh theme={null}
curl -d '{"url": "http://localhost:9910/webhook", "event_types":["I"]}' -H "Content-Type: application/json" -X POST http://localhost:9900/webhooks/subscribe
```

The supported events are `I`(insert), `U`(update), `D`(delete) and `T`(truncate). If no events are provided, all will be included. We can also subscribe to a specific schema or table by providing them in the request body. If they are not provided, all will be included.

For example, if we wanted to subscribe to all events for the `test` table of the `public` schema, the request would look like this:

```sh theme={null}
curl -d '{"url": "http://localhost:9910/webhook", "schema": "public", "table": "test"}' -H "Content-Type: application/json" -X POST http://localhost:9900/webhooks/subscribe
```

Once we have sent the subscription, we can validate that it has been created correctly by connecting to the PostgreSQL database where we configured the `webhook_subscriptions` table to be created. In our case, this is the source PostgreSQL database.

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:5432/postgres
```

```sql theme={null}
SELECT * FROM pgstream.webhook_subscriptions;
+-------------------------------+-------------+------------+-------------+
| url                           | schema_name | table_name | event_types |
|-------------------------------+-------------+------------+-------------|
| http://localhost:9910/webhook |             |            | ['I']       |
+-------------------------------+-------------+------------+-------------+
```

## Verify webhook events

Now we can start populating the source PostgreSQL database and receiving events in our webhook server.

```sql theme={null}
CREATE TABLE test(id SERIAL PRIMARY KEY, name TEXT);
```

Creating a table will generate an insert event on the `pgstream.schema_log` table that we'll be notified about, since it matches the subscription parameters.

```json theme={null}
{
  "Data": {
    "action": "I",
    "timestamp": "2025-03-13 09:51:31.19715+00",
    "lsn": "0/15A1038",
    "schema": "pgstream",
    "table": "schema_log",
    "columns": [
      {
        "id": "",
        "name": "id",
        "type": "pgstream.xid",
        "value": "cv9al8qhi0j00i9chq80"
      },
      {
        "id": "",
        "name": "version",
        "type": "bigint",
        "value": 1
      },
      {
        "id": "",
        "name": "schema_name",
        "type": "text",
        "value": "public"
      },
      {
        "id": "",
        "name": "schema",
        "type": "jsonb",
        "value": "{\"tables\": [{\"oid\": \"16471\", \"name\": \"test\", \"columns\": [{\"name\": \"id\", \"type\": \"integer\", \"unique\": true, \"default\": \"nextval('public.test_id_seq'::regclass)\", \"metadata\": null, \"nullable\": false, \"pgstream_id\": \"cv9al8qhi0j00i9chq8g-1\"}, {\"name\": \"name\", \"type\": \"text\", \"unique\": false, \"default\": null, \"metadata\": null, \"nullable\": true, \"pgstream_id\": \"cv9al8qhi0j00i9chq8g-2\"}], \"pgstream_id\": \"cv9al8qhi0j00i9chq8g\", \"primary_key_columns\": [\"id\"]}]}"
      },
      {
        "id": "",
        "name": "created_at",
        "type": "timestamp without time zone",
        "value": "2025-03-13 09:51:31.15459"
      },
      {
        "id": "",
        "name": "acked",
        "type": "boolean",
        "value": false
      }
    ],
    "identity": null,
    "metadata": {
      "schema_id": null,
      "table_pgstream_id": "",
      "id_col_pgstream_id": null,
      "version_col_pgstream_id": ""
    }
  }
}
```

Inserting data into this newly created table will also send an event to our webhook server.

```sql theme={null}
INSERT INTO test(name) VALUES('alice'),('bob'),('charlie');
```

This will generate 3 independent events for each row insert:

```json theme={null}
{
  "Data": {
    "action": "I",
    "timestamp": "2025-03-13 09:52:47.579305+00",
    "lsn": "0/15A5C70",
    "schema": "public",
    "table": "test",
    "columns": [
      {
        "id": "",
        "name": "id",
        "type": "integer",
        "value": 1
      },
      {
        "id": "",
        "name": "name",
        "type": "text",
        "value": "alice"
      }
    ],
    "identity": null,
    "metadata": {
      "schema_id": null,
      "table_pgstream_id": "",
      "id_col_pgstream_id": null,
      "version_col_pgstream_id": ""
    }
  }
}
```

```json theme={null}
{
  "Data": {
    "action": "I",
    "timestamp": "2025-03-13 09:52:47.579305+00",
    "lsn": "0/15A5D58",
    "schema": "public",
    "table": "test",
    "columns": [
      {
        "id": "",
        "name": "id",
        "type": "integer",
        "value": 2
      },
      {
        "id": "",
        "name": "name",
        "type": "text",
        "value": "bob"
      }
    ],
    "identity": null,
    "metadata": {
      "schema_id": null,
      "table_pgstream_id": "",
      "id_col_pgstream_id": null,
      "version_col_pgstream_id": ""
    }
  }
}
```

```json theme={null}
{
  "Data": {
    "action": "I",
    "timestamp": "2025-03-13 09:52:47.579305+00",
    "lsn": "0/15A5DD8",
    "schema": "public",
    "table": "test",
    "columns": [
      {
        "id": "",
        "name": "id",
        "type": "integer",
        "value": 3
      },
      {
        "id": "",
        "name": "name",
        "type": "text",
        "value": "charlie"
      }
    ],
    "identity": null,
    "metadata": {
      "schema_id": null,
      "table_pgstream_id": "",
      "id_col_pgstream_id": null,
      "version_col_pgstream_id": ""
    }
  }
}
```

Any other events will not notify our webhook server.

If we wanted to update our webhook subscription to include those events we can just send another subscription request.

```sh theme={null}
curl -d '{"url": "http://localhost:9910/webhook", "event_types":["I","U","D"]}' -H "Content-Type: application/json" -X POST http://localhost:9900/webhooks/subscribe
```

We should now see the subscriptions table updated accordingly:

```sql theme={null}
SELECT * FROM pgstream.webhook_subscriptions;
+-------------------------------+-------------+------------+-----------------+
| url                           | schema_name | table_name | event_types     |
|-------------------------------+-------------+------------+-----------------|
| http://localhost:9910/webhook |             |            | ['I', 'U', 'D'] |
+-------------------------------+-------------+------------+-----------------+
```

Those events will now notify our webhook server.

```sql theme={null}
UPDATE test SET name='alice' WHERE name='a';
```

```json theme={null}
{
  "Data": {
    "action": "U",
    "timestamp": "2025-03-13 09:59:27.02505+00",
    "lsn": "0/15A62C8",
    "schema": "public",
    "table": "test",
    "columns": [
      {
        "id": "",
        "name": "id",
        "type": "integer",
        "value": 1
      },
      {
        "id": "",
        "name": "name",
        "type": "text",
        "value": "alice"
      }
    ],
    "identity": [
      {
        "id": "",
        "name": "id",
        "type": "integer",
        "value": 1
      }
    ],
    "metadata": {
      "schema_id": null,
      "table_pgstream_id": "",
      "id_col_pgstream_id": null,
      "version_col_pgstream_id": ""
    }
  }
}
```

If we want to have the identity values populated (this is the previous values for the columns of the row being updated/deleted), we can set the `REPLICA IDENTITY` to `FULL`.

```sql theme={null}
ALTER TABLE test REPLICA IDENTITY FULL;
```

With a full replica identity the update event will now contain the old values for all columns. The same would apply for delete events, where a previous value is relevant.

```json theme={null}
{
  "Data": {
    "action": "U",
    "timestamp": "2025-03-13 10:00:47.406957+00",
    "lsn": "0/15A8F50",
    "schema": "public",
    "table": "test",
    "columns": [
      {
        "id": "",
        "name": "id",
        "type": "integer",
        "value": 1
      },
      {
        "id": "",
        "name": "name",
        "type": "text",
        "value": "alice"
      }
    ],
    "identity": [
      {
        "id": "",
        "name": "id",
        "type": "integer",
        "value": 1
      },
      {
        "id": "",
        "name": "name",
        "type": "text",
        "value": "a"
      }
    ],
    "metadata": {
      "schema_id": null,
      "table_pgstream_id": "",
      "id_col_pgstream_id": null,
      "version_col_pgstream_id": ""
    }
  }
}
```

## Populate event metadata

In this tutorial we haven't enabled the injector to populate the metadata event information, which is why it appears empty in the events. If the webhook notifier requires that metadata information, the modifiers configuration can be updated to enable the injector. The injector adds pgstream IDs and metadata to the events, which can be useful for tracking and versioning (more details can be found in the [architecture section](/docs/opensource/pgstream/docs/architecture)).

To enable the injector for webhooks, add to the configuration:

```yaml theme={null}
modifiers:
  injector:
    enabled: true
```

With the injector enabled, the metadata fields in the webhook payloads will be populated with the appropriate pgstream information for each event.

If we now run pgstream again with the updated configuration file, the events will have the metadata populated.

```json theme={null}
{
  "Data": {
    "action": "U",
    "timestamp": "2025-03-13 10:18:29.264688+00",
    "lsn": "0/15C1CC0",
    "schema": "public",
    "table": "tutorial_test",
    "columns": [
      {
        "id": "cv9al8qhi0j00i9chq8g-1",
        "name": "id",
        "type": "integer",
        "value": 1
      },
      {
        "id": "cv9al8qhi0j00i9chq8g-2",
        "name": "name",
        "type": "text",
        "value": "a"
      }
    ],
    "identity": [
      {
        "id": "cv9al8qhi0j00i9chq8g-1",
        "name": "id",
        "type": "integer",
        "value": 1
      },
      {
        "id": "cv9al8qhi0j00i9chq8g-2",
        "name": "name",
        "type": "text",
        "value": "alice"
      }
    ],
    "metadata": {
      "schema_id": "cv9b12qhi0j00i9chqag",
      "table_pgstream_id": "cv9al8qhi0j00i9chq8g",
      "id_col_pgstream_id": ["cv9al8qhi0j00i9chq8g-1"],
      "version_col_pgstream_id": ""
    }
  }
}
```

## Troubleshooting

Here are some common issues you might encounter while following this tutorial and how to resolve them:

### 1. **Error: `Connection refused`**

* **Cause:** The PostgreSQL database or webhook server is not running.
* **Solution:**
  * Ensure the Docker containers for the PostgreSQL database and webhook server are running.
  * Verify the database and webhook server URLs in the configuration.

### 2. **Error: `Replication slot not found`**

* **Cause:** The replication slot was not created during initialization.
* **Solution:**
  * Reinitialize `pgstream` or manually create the replication slot.
  * Run the `pgstream status` command to validate the initialisation was successful.
  * Verify the replication slot exists by running:
    ```sql theme={null}
    SELECT slot_name FROM pg_replication_slots;
    ```

### 3. **Error: `Webhook events not received`**

* **Cause:** The webhook subscription was not registered correctly.
* **Solution:**
  * Verify the subscription by querying the `pgstream.webhook_subscriptions` table:
    ```sql theme={null}
    SELECT * FROM pgstream.webhook_subscriptions;
    ```
  * Ensure the webhook server is running and reachable.

### 4. **Error: `Permission denied`**

* **Cause:** The database user does not have sufficient privileges.
* **Solution:**
  * Grant the required privileges to the database user:
    ```sql theme={null}
    GRANT ALL PRIVILEGES ON DATABASE postgres TO postgres;
    ```

### 5. **Error: `Invalid configuration`**

* **Cause:** The configuration file contains invalid or missing values.
* **Solution:**
  * Double-check the `pg2webhook_tutorial.env` file for typos or missing variables.
  * Refer to the [pgstream configuration documentation](https://github.com/xataio/pgstream) for details on required variables.
  * Run the `pgstream status` command to validate the configuration is correct.

### 6. **Error: `Event metadata not populated`**

* **Cause:** The injector is not enabled in the configuration.
* **Solution:**
  * Enable the injector in your configuration:
    ```yaml theme={null}
    modifiers:
      injector:
        enabled: true
    ```
  * Restart pgstream for the changes to take effect.

### 7. **Error: `Stale webhook subscription cache`**

* **Cause:** The subscription cache is not refreshing frequently enough.
* **Solution:**
  * Adjust the cache refresh interval in the configuration:
    ```sh theme={null}
    PGSTREAM_WEBHOOK_SUBSCRIPTION_STORE_CACHE_REFRESH_INTERVAL="30s"
    ```

If you encounter issues not listed here, consult the [pgstream documentation](https://github.com/xataio/pgstream) or open an issue on the project's GitHub repository.

## Summary

In this tutorial, we successfully configured `pgstream` to replicate data from a PostgreSQL database to a webhook server. We:

1. Set up the source PostgreSQL database and initialized `pgstream`.
2. Configured the listener to capture changes from the PostgreSQL WAL.
3. Configured the processor to send events to a webhook server.
4. Verified that database changes triggered webhook events.
5. Explored how to update webhook subscriptions and use the injector for metadata enrichment.

This tutorial demonstrates how `pgstream` can be used to integrate PostgreSQL with webhook-based systems, enabling real-time event-driven architectures. For more advanced use cases, refer to the other [pgstream tutorials](/docs/opensource/pgstream/overview#tutorials).


# PostgreSQL anonymization
Source: https://xata.io/docs/opensource/pgstream/docs/tutorials/postgres_transformer

## Introduction

This tutorial will showcase the use of pgstream to replicate data from PostgreSQL using transformers. We'll use a target PostgreSQL database as target, but the same would apply to any of the supported targets.

![transformer tutorial](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_tutorial_transformer.svg)

### Requirements

* A source PostgreSQL database
* A target PostgreSQL database
* pgstream (see [installation](/docs/opensource/pgstream/docs/cli#installation) instructions for more details)

### Demo

[YouTube video player](https://www.youtube.com/embed/6aHzmv1h_48)

## Environment setup

The first step is to start the two PostgreSQL databases that will be used as source and target for replication. The `pgstream` repository provides a docker installation that will be used for the purposes of this tutorial, but can be replaced by any available PostgreSQL servers, as long as they have [`wal2json`](https://github.com/eulerto/wal2json) installed.

To start the docker provided PostgreSQL servers, run the following command:

```sh theme={null}
# Start two PostgreSQL databases using Docker.
# The source database will run on port 5432, and the target database will run on port 7654.
docker-compose -f build/docker/docker-compose.yml --profile pg2pg up
```

This will start two PostgreSQL databases on ports `5432` and `7654`.

## Database initialisation

Once both PostgreSQL servers are up and running, the next step is to initialise pgstream on the source database. This will create the `pgstream` schema in the configured Postgres database, along with the tables/functions/triggers required to keep track of the schema changes. See [Tracking schema changes](/docs/opensource/pgstream/docs/architecture#tracking-schema-changes) section for more details. This step will also create a replication slot on the source database which will be used by the pgstream service.

The initialisation step allows to provide both the URL of the PostgreSQL database and the name of the replication slot to be created. The PostgreSQL URL is required, but the replication slot name is optional. If not provided, it will default to `pgstream_<dbname>_slot`, where `<dbname>` is the name of the PostgreSQL database. The configuration can be provided either by using the CLI supported parameters, or using the environment variables.

For this tutorial, we'll create a replication slot with the name `pgstream_tutorial_slot`.

* Using the `--init` flag in the `run` command

* Using CLI flags:

```sh theme={null}
pgstream init --postgres-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --replication-slot pgstream_tutorial_slot
```

* Using environment variables:

```sh theme={null}
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot PGSTREAM_POSTGRES_LISTENER_URL=postgres://postgres:postgres@localhost:5432?sslmode=disable pgstream init
```

Successful initialisation should prompt the following message:

```
SUCCESS  pgstream initialisation complete
```

If at any point the initialisation performed by pgstream needs to be reverted, all state will be removed by running the `destroy` CLI command.

```sh theme={null}
pgstream destroy --postgres-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --replication-slot pgstream_tutorial_slot
```

## Prepare `pgstream` configuration

### Listener

The listener configuration will be the same as the one in the PostgreSQL to PostgreSQL replication tutorial. Check the details [here](/docs/opensource/pgstream/docs/tutorials/postgres_to_postgres#listener).

* Without initial snapshot

```sh theme={null}
# Listener config
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot
```

* With initial snapshot

```sh theme={null}
# Listener config
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot
PGSTREAM_POSTGRES_SNAPSHOT_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
# Initial snapshot of all tables in the public schema
PGSTREAM_POSTGRES_SNAPSHOT_TABLES="*"
```

### Processor

The processor configuration will be the same as the one in the PostgreSQL to PostgreSQL replication tutorial. Check the details [here](/docs/opensource/pgstream/docs/tutorials/postgres_to_postgres#processor).

```sh theme={null}
PGSTREAM_POSTGRES_WRITER_TARGET_URL="postgres://postgres:postgres@localhost:7654?sslmode=disable"
PGSTREAM_POSTGRES_WRITER_BATCH_SIZE=25
PGSTREAM_POSTGRES_WRITER_BATCH_TIMEOUT=5s
```

In addition to the processor configuration above, we will add the transformer processor wrapper which will apply the column transformations. The only configuration required is the path to the transformer rules.

```sh theme={null}
PGSTREAM_TRANSFORMER_RULES_FILE="tutorial_transformer_rules.yaml"
```

The transformer rules follow the pattern defined in the [transfomer section](/docs/opensource/pgstream/docs/transformers#transformation-rules) of the documentation. To see all the transformers currently supported, check out the [transformers section](/docs/opensource/pgstream/docs/transformers#supported-transformers). For this tutorial we'll be using the following transformations:

```yaml theme={null}
transformations:
  validation_mode: relaxed # Specifies the validation mode for the transformer.
  table_transformers:
    - schema: public
      table: test
      column_transformers:
        email:
          name: neosync_email
          parameters:
            preserve_length: true # Ensures the transformed email has the same length as the original.
            preserve_domain: true # Keeps the domain of the original email intact.
            email_type: fullname # Specifies the type of email transformation.
        name:
          name: greenmask_firstname
          parameters:
            generator: deterministic # Ensures the same input always produces the same output.
            gender: Female # Generates female names for the transformation.
```

The full configuration for this tutorial can be put into a `pg2pg_transformer_tutorial.env` file to be used in the next step. An equivalent `pg2pg_transformer_tutorial.yaml` configuration can be found below the environment one, and can be used interchangeably.

* Without initial snapshot

```sh theme={null}
# Listener config
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot

# Processor config
PGSTREAM_TRANSFORMER_RULES_FILE="tutorial_transformer_rules.yaml"
PGSTREAM_POSTGRES_WRITER_TARGET_URL="postgres://postgres:postgres@localhost:7654?sslmode=disable"
PGSTREAM_POSTGRES_WRITER_BATCH_SIZE=25
PGSTREAM_POSTGRES_WRITER_BATCH_TIMEOUT=5s
```

```yaml theme={null}
source:
  postgres:
    url: "postgres://postgres:postgres@localhost:5432?sslmode=disable"
    mode: replication # options are replication, snapshot or snapshot_and_replication
    replication:
      replication_slot: pgstream_tutorial_slot

target:
  postgres:
    url: "postgres://postgres:postgres@localhost:7654?sslmode=disable"
    batch:
      timeout: 5000 # batch timeout in milliseconds
      size: 25 # number of messages in a batch
    disable_triggers: false # whether to disable triggers on the target database
    on_conflict_action: "nothing" # options are update, nothing or error

modifiers:
  transformations:
    validation_mode: relaxed
    table_transformers:
      - schema: public
        table: test
        column_transformers:
          email:
            name: neosync_email
            parameters:
              preserve_length: true
              preserve_domain: true
              email_type: fullname
          name:
            name: greenmask_firstname
            parameters:
              generator: deterministic
              gender: Female
```

* With initial snapshot

```sh theme={null}
# Listener config
PGSTREAM_POSTGRES_LISTENER_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
PGSTREAM_POSTGRES_REPLICATION_SLOT_NAME=pgstream_tutorial_slot
PGSTREAM_POSTGRES_SNAPSHOT_STORE_URL="postgres://postgres:postgres@localhost:5432?sslmode=disable"
# Initial snapshot of all tables in the public schema
PGSTREAM_POSTGRES_SNAPSHOT_TABLES="*"

# Processor config
PGSTREAM_TRANSFORMER_RULES_FILE="tutorial_transformer_rules.yaml"
PGSTREAM_POSTGRES_WRITER_TARGET_URL="postgres://postgres:postgres@localhost:7654?sslmode=disable"
PGSTREAM_POSTGRES_WRITER_BATCH_SIZE=25
PGSTREAM_POSTGRES_WRITER_BATCH_TIMEOUT=5s
```

```yaml theme={null}
source:
  postgres:
    url: "postgres://postgres:postgres@localhost:5432?sslmode=disable"
    mode: snapshot_and_replication # options are replication, snapshot or snapshot_and_replication
    replication:
      replication_slot: pgstream_tutorial_slot
    snapshot: # when mode is snapshot or snapshot_and_replication
      mode: full # options are data_and, schema or data
      tables: ["*"] # tables to snapshot, can be a list of table names or a pattern
      recorder:
        repeatable_snapshots: true # whether to repeat snapshots that have already been taken
        postgres_url: "postgres://postgres:postgres@localhost:5432?sslmode=disable" # URL of the database where the snapshot status is recorded
      schema: # when mode is full or schema
        pgdump_pgrestore:
          clean_target_db: false # whether to clean the target database before restoring

target:
  postgres:
    url: "postgres://postgres:postgres@localhost:7654?sslmode=disable"
    batch:
      timeout: 5000 # batch timeout in milliseconds
      size: 25 # number of messages in a batch
    disable_triggers: false # whether to disable triggers on the target database
    on_conflict_action: "nothing" # options are update, nothing or error

modifiers:
  transformations:
    validation_mode: relaxed
    table_transformers:
      - schema: public
        table: test
        column_transformers:
          email:
            name: neosync_email
            parameters:
              preserve_length: true
              preserve_domain: true
              email_type: fullname
          name:
            name: greenmask_firstname
            parameters:
              generator: deterministic
              gender: Female
```

## Validate `pgstream` status

We can validate that the initialisation and the configuration are valid by running the `status` command before starting `pgstream`.

```sh theme={null}
# using yaml configuration file
./pgstream status -c pg2pg_transformer_tutorial.yaml
# using env configuration file
./pgstream status -c pg2pg_transformer_tutorial.env
```

```sh theme={null}
SUCCESS  pgstream status check encountered no issues
Initialisation status:
 - Pgstream schema exists: true
 - Migration current version: 7
 - Migration status: success
 - Replication slot name: pgstream_tutorial_slot
 - Replication slot plugin: wal2json
 - Replication slot database: postgres
Config status:
 - Valid: true
Transformation rules status:
 - Valid: true
Source status:
 - Reachable: true
```

## Run `pgstream`

With the configuration ready, we can now run pgstream. In this case we set the log level as trace to provide more context for debugging and have more visibility into what pgstream is doing under the hood.

**Important:** Ensure that the source and target databases are running before proceeding.

```sh theme={null}
# with initialization
pgstream run -c pg2pg_transformer_tutorial.env --init --log-level trace

# with the environment configuration
pgstream run -c pg2pg_transformer_tutorial.env --log-level trace

# with the yaml configuration
pgstream run -c pg2pg_transformer_tutorial.yaml --log-level trace

# with the CLI flags and relying on defaults
PGSTREAM_TRANSFORMER_RULES_FILE="tutorial_transformer_rules.yaml" pgstream run --source postgres --source-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --target postgres --target-url "postgres://postgres:postgres@localhost:7654?sslmode=disable" --log-level trace
```

Now we can connect to the source database, create a table and start inserting data.

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:5432/postgres
```

```sql theme={null}
CREATE TABLE test(id SERIAL PRIMARY KEY, name TEXT, email TEXT);
INSERT INTO test(name,email) VALUES('alice','alice@test.com'),('bob','bob@test.com'),('charlie','charlie@test.com');
SELECT * FROM test;
+----+---------+------------------+
| id | name    | email            |
|----+---------+------------------|
| 1  | alice   | alice@test.com   |
| 2  | bob     | bob@test.com     |
| 3  | charlie | charlie@test.com |
+----+---------+------------------+
```

## Verify data transformation

When we check the target PostgreSQL database, we should see the three rows have been inserted, but both the name and the emails have been anonymised.

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:7654/postgres
```

```sql theme={null}
SELECT * FROM test;
+----+---------+------------------+
| id | name    | email            |
|----+---------+------------------|
| 1  | Herman  | ecdox@test.com   |
| 2  | Domingo | nab@test.com     |
| 3  | Elwin   | aafsack@test.com |
+----+---------+------------------+
```

Since we used the deterministic transformation for the name column, if we insert the same names again the value generated should be the same.

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:5432/postgres
```

```sql theme={null}
INSERT INTO test(name,email) VALUES('alice','alice@test.com'),('bob','bob@test.com'),('charlie','charlie@test.com');
SELECT * FROM test;
+----+---------+------------------+
| id | name    | email            |
|----+---------+------------------|
| 1  | alice   | alice@test.com   |
| 2  | bob     | bob@test.com     |
| 3  | charlie | charlie@test.com |
| 4  | alice   | alice@test.com   |
| 5  | bob     | bob@test.com     |
| 6  | charlie | charlie@test.com |
+----+---------+------------------+
```

```sh theme={null}
➜ psql postgresql://postgres:postgres@localhost:7654/postgres
```

```sql theme={null}
SELECT * FROM test;
+----+---------+------------------+
| id | name    | email            |
|----+---------+------------------|
| 1  | Herman  | ecdox@test.com   |
| 2  | Domingo | nab@test.com     |
| 3  | Elwin   | aafsack@test.com |
| 4  | Herman  | rwamy@test.com   |
| 5  | Domingo | wie@test.com     |
| 6  | Elwin   | cailbni@test.com |
+----+---------+------------------+
```

To see all the transformers currently supported, check out the [transformers section](/docs/opensource/pgstream/docs/transformers#supported-transformers).

## Troubleshooting

* **Error:** `Connection refused`

  * Ensure the Docker containers for the source and target databases are running.
  * Verify the database URLs in the configuration.

* **Error:** `Replication slot not found`

  * Check that the replication slot was created successfully during initialization.
  * Run the `pgstream status` command to validate the initialisation was successful.
  * Run the following query on the source database to verify:
    ```sql theme={null}
    SELECT slot_name FROM pg_replication_slots;
    ```

* **Error:** `Transformation rules not applied`
  * Ensure the `PGSTREAM_TRANSFORMER_RULES_FILE` points to the correct YAML file.
  * Validate the syntax of the transformer rules using the [pgstream documentation](/docs/opensource/pgstream/docs/transformers#transformation-rules).
  * Run the `pgstream status` command to validate the transformation rules are correct.

## Summary

In this tutorial, we successfully configured `pgstream` to replicate data from a source PostgreSQL database to a target PostgreSQL database, applying transformations to anonymize sensitive data during replication. For more use cases, refer to the [pgstream documentation](https://github.com/xataio/pgstream).


# Xata
Source: https://xata.io/docs/opensource/pgstream/docs/xata

This guide explains how to use **pgstream** with **Xata databases**, covering **snapshots** and **replication**.

👉 Throughout this guide:

* **`pgstreamsource`** refers to the user provided in the **pgstream source URL**.
* **`pgstreamtarget`** refers to the user provided in the **pgstream target URL**.

## Snapshots

### From Xata Database (Snapshots)

#### Quick Checklist

* [ ] Create a source user (**`pgstreamsource`**) with access to required schemas/tables.
* [ ] For roles without passwords → no special config needed.
* [ ] For roles with passwords → **must use the `xata` superuser**.
* [ ] Update YAML config with correct snapshot settings.

#### Steps

1. **User privileges**
   Ensure the **`pgstreamsource`** user (from the pgstream source URL) can access the database schema and tables you need.

2. **Roles handling**

   * **No roles** → no changes required.
   * **Roles without passwords** → no changes required.
   * **Roles with passwords** → **must use the `xata` user** as `pgstreamsource`.

     ⚠️ Only the `xata` user has access to the `pg_authid` table required for snapshotting roles with passwords.

     Without this, you'll see:

     ```
     pg_dumpall: error: query failed: ERROR:  permission denied for table pg_authid
     ```

3. **Config when not snapshotting roles**
   If roles are disabled or not manually managed, add the following to avoid failures:

   ```yaml theme={null}
   snapshot:
     schema:
       pgdump_pgrestore:
         roles_snapshot_mode: "disabled"
         no_owner: true
         no_privileges: true
   ```

   Example full configuration:

   ```yaml theme={null}
   source:
     postgres:
       url: "postgresql://pgstreamsource:password@<xata-host>:5432/db?sslmode=require"
       mode: snapshot
       snapshot:
         mode: full # schema + data
         tables: ["public.*"] # all tables in the public schema
         schema:
           mode: pgdump_pgrestore
           pgdump_pgrestore:
             roles_snapshot_mode: "disabled"
             no_owner: true
             no_privileges: true
   ```

ℹ️ Xata-managed roles (`xata`, `xata_superuser`, `cnpg_pooler_pgbouncer`) will not be snapshotted.

### To Xata Database (Snapshots)

#### Quick Checklist

* [ ] Create a target user (**`pgstreamtarget`**) for the pgstream target URL.
* [ ] Grant privileges for schema/database ownership.
* [ ] Grant optional privileges depending on features (create DB, create roles).
* [ ] **Use `disable_triggers: false`** (required for all non-`xata` users).

#### Steps

The **`pgstreamtarget`** user (from the pgstream target URL) must have the following privileges:

* **Schema ownership**

  ```sql theme={null}
  ALTER DATABASE db OWNER TO pgstreamtarget;
  ALTER SCHEMA public OWNER TO pgstreamtarget;
  ```

* **Database creation** (if `create_target_db` is enabled)

  ```sql theme={null}
  ALTER ROLE pgstreamtarget CREATEDB;
  ```

* **Role creation** (if `roles_snapshot_mode` is `enabled`/`no_passwords`)

  ```sql theme={null}
  ALTER ROLE pgstreamtarget CREATEROLE;
  ```

  ⚠️ `pgstreamtarget` must already hold any privileges it assigns (e.g., `REPLICATION`).

* **Disable triggers** (required for Xata)

  ⚠️ **Only the `xata` user can set `session_replication_role`** on Xata databases. For all other users, you **must** disable trigger management in pgstream:

  ```yaml theme={null}
  target:
    postgres:
      disable_triggers: false # Required for non-xata users
  ```

  ❌ If using a non-`xata` user with `disable_triggers: true`, pgstream will fail with:

  ```
  permission denied to set parameter "session_replication_role"
  ```

## Replication

### From Xata Database (Replication)

#### Quick Checklist

* [ ] Ensure logical replication is enabled on your Xata database.
* [ ] **Use the `xata` user for initialization** (only user with event trigger privileges).
* [ ] Optionally use a different **`pgstreamsource`** user for streaming.

#### Steps

1. **Enable logical replication**
   Make sure logical replication is enabled in your database. The `wal_level` must be set to `logical`. You can check the current setting by running the following:

   ```sql theme={null}
   SHOW wal_level;

   ```

   ⚠️ Changing wal\_level requires a full database restart.

2. **Replication phases**

   * **Initialization** → requires elevated privileges to create schemas, event triggers, and replication slots.
   * **Streaming** → can use a dedicated `pgstreamsource` user.

#### Initialization

Initialization does the following:

* Creates `pgstream` schema
* Creates replication slot (if missing)
* Creates event triggers/functions for schema changes

⚠️ **Only the `xata` user has sufficient privileges to create event triggers.** Use the `xata` user for initialization:

```yaml theme={null}
source:
  postgres:
    url: "postgresql://xata:password@<xata-host>:5432/db?sslmode=require"
    mode: replication
```

#### Streaming

After initialization, you can:

* Keep using `xata`, or
* Transfer ownership to a dedicated **`pgstreamsource`** role.

```sql theme={null}
-- Allow ownership transfer
GRANT pgstreamsource TO current_user;

-- Transfer ownership
ALTER DATABASE db OWNER TO pgstreamsource;
ALTER SCHEMA pgstream OWNER TO pgstreamsource;
ALTER SCHEMA <schema> OWNER TO pgstreamsource;
ALTER TABLE pgstream.schema_log OWNER TO pgstreamsource;
ALTER TABLE pgstream.table_ids OWNER TO pgstreamsource;
ALTER SEQUENCE pgstream.xid_serial OWNER TO pgstreamsource;

-- Replication privileges
ALTER ROLE pgstreamsource REPLICATION;
```

### To Xata Database (Replication)

#### Quick Checklist

* [ ] Ensure **`pgstreamtarget`** has schema/database ownership.
* [ ] If combined with snapshot → follow [Snapshot target requirements](#to-xata-database-snapshots).

#### Steps

Privileges required:

```sql theme={null}
ALTER DATABASE db OWNER TO pgstreamtarget;
ALTER SCHEMA <schema> OWNER TO pgstreamtarget;
```

## Troubleshooting

> [accordion] permission denied for table pg\_authid
>
> **Cause**: Roles with passwords require pg\_authid access.\
> **Fix**: Use `xata` user as source, or disable role passwords (`roles_snapshot_mode: no_passwords`).

> [accordion] permission denied to set parameter "session\_replication\_role"
>
> **Cause**: Only the `xata` user can set this parameter.\
> **Fix**: Set `disable_triggers: false` in target config, or use the `xata` user as target.

> [accordion] permission denied for schema public when transferring ownership
>
> **Cause**: Target role lacks CREATE privilege on schema.\
> **Fix**: Grant CREATE on schema to the role: `GRANT CREATE ON SCHEMA public TO role_name;`

> [accordion] Role creation fails
>
> **Cause**: `pgstreamtarget` lacks privileges it tries to assign.\
> **Fix**: Grant required privileges to `pgstreamtarget` first, or use a user with more privileges.


# Overview
Source: https://xata.io/docs/opensource/pgstream/overview

Postgres replication with DDL changes

![pgstream logo](https://raw.githubusercontent.com/xataio/pgstream/main/brand-kit/banner/pgstream-banner@2x.png)

`pgstream` is an open source CDC command-line tool and library that offers Postgres replication support with DDL changes to any provided target.

![pg2pg demo with transformers](https://github.com/user-attachments/assets/6f11b326-d8ed-44eb-b743-756910b9fedd)

## Features

* Schema change tracking and replication of DDL changes
* Support for multiple out of the box targets
  * Elasticsearch/OpenSearch
  * Webhooks
  * PostgreSQL
* Initial and on demand PostgreSQL snapshots (for when you don't need continuous replication)
* Column value transformations (anonymise your data on the go!)
* Modular deployment configuration, only requires Postgres
* Kafka support with schema based partitioning
* Extendable support for custom targets

## Usage

`pgstream` can be used via the readily available CLI or as a library. For detailed information about the CLI usage, check out the dedicated [CLI documentation section](/docs/opensource/pgstream/docs/cli).

### CLI Installation

```bash theme={null}
# Download the latest release
curl -L https://github.com/xataio/pgstream/releases/latest/download/pgstream.linux.amd64 -o pgstream
chmod +x pgstream
sudo mv pgstream /usr/local/bin/

# Or use go install
go install github.com/xataio/pgstream@latest

# Or build from source
git clone https://github.com/xataio/pgstream.git
cd pgstream
go build -o pgstream ./cmd

# Or install via homebrew on macOS or Linux
brew tap xataio/pgstream
brew install pgstream
```

### Environment setup

If you have an environment available, with at least Postgres and whichever resources you're planning on running, then you can skip this step. Otherwise, a docker setup is available in this repository with profiles that selectively start Postgres, Kafka and OpenSearch.

To run all profiles:

```
docker-compose -f build/docker/docker-compose.yml up
```

If you only want to run PostgreSQL to PostgreSQL pgstream replication you can use the `pg2pg` profile:

```
docker-compose -f build/docker/docker-compose.yml --profile pg2pg up
```

You can also run multiple profiles. For example to start two PostgreSQL instances and Kafka:

```
docker-compose -f build/docker/docker-compose.yml --profile pg2pg --profile kafka up
```

List of supported docker profiles:

* pg2pg
* pg2os
* pg2webhook
* kafka

### Configuration

Pgstream source and target need to be configured appropriately before the commands can be run. This can be done:

* Using the relevant CLI flags for each command
* Using a yaml configuration file
* Using environment variables (.env file supported)

Check the [configuration documentation](/docs/opensource/pgstream/docs/configuration) for more information about the configuration options, or check the [CLI documentation](/docs/opensource/pgstream/docs/cli) for details on the available flags. Additionally, you can find sample files for both .env and .yaml [here](https://github.com/xataio/pgstream/tree/main/docs/examples).

If you want to configure column transformations, leveraging [greenmask](https://github.com/GreenmaskIO/greenmask), [neosync](https://github.com/nucleuscloud/neosync) and [go-masker](https://github.com/ggwhite/go-masker) open source integrations, as well as custom transformers, check the [transformation rules](/docs/opensource/pgstream/docs/transformers#transformation-rules) configuration for more details, along with the list of [available transformers](/docs/opensource/pgstream/docs/transformers#supported-transformers).

### Run `pgstream`

#### Replication mode

Run will start streaming data from the configured source into the configured target. By passing the `--init` flag to the run command, pgstream will initialise the pgstream state in the source Postgres database before starting replication. It will:

* Create a `pgstream` schema
* Create tables/functions/triggers to keep track of schema changes for DDL replication (see [Tracking schema changes](/docs/opensource/pgstream/docs/architecture#tracking-schema-changes) for more details)
* Create a replication slot

Initialisation is required for pgstream replication. It can alternatively be performed by running the `pgstream init` command separately before `pgstream run`. Check out the [CLI documentation](/docs/opensource/pgstream/docs/cli#init) for more details.

Example running pgstream replication from Postgres -> OpenSearch:

```sh theme={null}
# using the environment configuration file
pgstream run -c docs/examples/pg2os.env --init --log-level trace
# using the yaml configuration file
pgstream run -c docs/examples/pg2os.yaml --init --log-level info
# using the CLI flags
pgstream run --source postgres --source-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --target opensearch --target-url "http://admin:admin@localhost:9200" --init
```

Example running pgstream with Postgres -> Kafka, and in a separate terminal, Kafka->OpenSearch:

```sh theme={null}
# using the environment configuration file
pgstream run -c docs/examples/pg2kafka.env --init --log-level trace
# using the yaml configuration file
pgstream run -c docs/examples/pg2kafka.yaml --init --log-level info
# using the CLI flags
pgstream run --source postgres --source-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --target kafka --target-url "localhost:9092" --init
```

```sh theme={null}
# using the environment configuration file
pgstream run -c docs/examples/kafka2os.env --init --log-level trace
# using the yaml configuration file
pgstream run -c docs/examples/kafka2os.yaml --init --log-level info
# using the CLI flags
pgstream run --source kafka --source-url "localhost:9092" --target opensearch --target-url "http://admin:admin@localhost:9200" --init
```

An initial snapshot can be performed before starting replication by providing `--snapshot-tables` flag or by setting the relevant configuration fields (check the [configuration documentation](/docs/opensource/pgstream/docs/configuration) for more details on advanced configuration options).

Example running pgstream with PostgreSQL -> PostgreSQL with initial snapshot enabled:

```sh theme={null}
# using the CLI flags
pgstream run --source postgres --source-url "postgres://postgres:postgres@localhost:5432?sslmode=disable" --target postgres --target-url "postgres://postgres:postgres@localhost:7654?sslmode=disable" --snapshot-tables test --init
```

#### Snapshot mode

pgstream can also be used to perform a point in time snapshot of the source database. This is helpful if you don't require continuous replication, but want to keep the source and target in sync by running nightly snapshots for example.

The `snapshot` command doesn't require any initialisation or pgstream specific state, since it only performs read operations on the source Postgres database.

Example running pgstream to perform a snapshot from PostgreSQL -> PostgreSQL:

```sh theme={null}
# using the environment configuration file
pgstream snapshot -c docs/examples/snapshot2pg.env --log-level trace
# using the yaml configuration file
pgstream snapshot -c docs/examples/snapshot2pg.yaml --log-level info
# using the CLI flags
pgstream snapshot --postgres-url="postgres://postgres:postgres@localhost:5432?sslmode=disable" --target=postgres --target-url="postgres://postgres:postgres@localhost:7654?sslmode=disable" --tables="test" --reset
```

## Tutorials

* [PostgreSQL replication to PostgreSQL](/docs/opensource/pgstream/docs/tutorials/postgres_to_postgres)
* [PostgreSQL replication to OpenSearch](/docs/opensource/pgstream/docs/tutorials/postgres_to_opensearch)
* [PostgreSQL replication to webhooks](/docs/opensource/pgstream/docs/tutorials/postgres_to_webhooks)
* [PostgreSQL replication using Kafka](/docs/opensource/pgstream/docs/tutorials/postgres_kafka)
* [PostgreSQL snapshots](/docs/opensource/pgstream/docs/tutorials/postgres_snapshot)
* [PostgreSQL column transformations](/docs/opensource/pgstream/docs/tutorials/postgres_transformer)

## Documentation

For more advanced usage, implementation details, and detailed configuration settings, please refer to the full documentation below.

1. [Architecture](/docs/opensource/pgstream/docs/architecture)
2. [Configuration](/docs/opensource/pgstream/docs/configuration)
   * [Yaml](/docs/opensource/pgstream/docs/configuration#yaml)
   * [Environment Variables](/docs/opensource/pgstream/docs/configuration#environment-variables)
   * [Examples](https://github.com/xataio/pgstream/tree/main/docs/examples)
3. [Snapshots](/docs/opensource/pgstream/docs/snapshots)
   * [Running from a read replica](/docs/opensource/pgstream/docs/replicas)
4. [Transformers](/docs/opensource/pgstream/docs/transformers)
   * [Supported Transformers](/docs/opensource/pgstream/docs/transformers#supported-transformers)
   * [Transformation Rules](/docs/opensource/pgstream/docs/transformers#transformation-rules)
5. [Observability](/docs/opensource/pgstream/docs/Observability)
6. [CLI](/docs/opensource/pgstream/docs/cli)
7. [Privileges](/docs/opensource/pgstream/docs/privileges)
   * [Xata](/docs/opensource/pgstream/docs/xata)
   * [AWS](/docs/opensource/pgstream/docs/aws)
   * [GCP CloudSQL](/docs/opensource/pgstream/docs/gcp_cloudsql)
   * [Neon](/docs/opensource/pgstream/docs/neon)
8. [Container image](/docs/opensource/pgstream/docs/container_image)
9. [Glossary](/docs/opensource/pgstream/docs/glossary)

## Benchmarks

### Snapshots

![pgstream snapshot benchmarks](https://raw.githubusercontent.com/xataio/pgstream/main/docs/img/pgstream_snapshot_benchmarks.png)

Datasets used: [IMDB database](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/2QYZBT), [MusicBrainz database](https://musicbrainz.org/doc/MusicBrainz_Database), [Firenibble database](https://github.com/PeerDB-io/ab-scale-testing/tree/main).

All benchmarks were run using the same setup, with pgstream `v0.7.2`, pg\_dump/pg\_restore (PostgreSQL) 17.4 and PostgreSQL 17.4, using identical resources to ensure a fair comparison.

For more details into performance benchmarking for snapshots to PostgreSQL with `pgstream`, check out this [blogpost](https://xata.io/blog/behind-the-scenes-speeding-up-pgstream-snapshots-for-postgresql).

## Limitations

Some of the limitations of the initial release include:

* Single Kafka topic support
* Postgres plugin support limited to `wal2json`
* No row level filtering support
* Primary key/unique not null column required for replication
* Kafka serialisation support limited to JSON

## Releases

You can find below the release notes for major pgstream versions, along with migration guides whenever relevant:

* [v1.0.0](/docs/opensource/pgstream/docs/releases/v1.0.0)

## Contributing

We welcome contributions from the community! If you'd like to contribute to pgstream, please follow [these guidelines](https://github.com/xataio/pgstream/tree/main?tab=contributing-ov-file) and adhere to our [code of conduct](https://github.com/xataio/pgstream?tab=coc-ov-file).

## License

This project is licensed under the Apache License 2.0 - see the [LICENSE](https://github.com/xataio/pgstream/blob/main/LICENSE) file for details.

## Support

If you have any questions, encounter issues, or need assistance, open an issue in this repository, and our community will be happy to help.


# Xata Documentation
Source: https://xata.io/docs/overview

Xata documentation

# Branch-native Postgres

Build with real production data safely. Xata gives teams instant Postgres branches, anonymized clones, and controlled schema changes for faster development workflows.

[Get started](quickstart)

[Explore branching](core-concepts/branching)

![Workflow diagram showing Production Postgres in AWS, GCP, or Azure connected to a Xata production clone and development database branches.](https://mintcdn.com/xata/W3vk-RcC5XaWRdBV/images/overview/branching-workflow.png?fit=max\&auto=format\&n=W3vk-RcC5XaWRdBV\&q=85\&s=2ede29e09a667af025047ad9b730cf6e)

Core concept

## [Branching](core-concepts/branching)

Create copy-on-write database branches in seconds without duplicating storage.

Tutorial

## [Production Clone](tutorials/create-production-clone)

Build realistic staging and development data sets with PII removed.

Guide

## [AI Agents](ai-agents/overview)

Give coding agents isolated Postgres branches for safe migrations, tests, and data work.

Core concept

## [Schema Changes](core-concepts/schema-changes)

Apply and roll back PostgreSQL schema changes with controlled migration workflows.

Reference

## [API Reference](api-reference)

Manage organizations, projects, branches, and platform resources programmatically.

Reference

## [CLI Reference](cli)

Work with auth, projects, branches, clone, roll, stream, and status commands.

Platform
Projects, branches, logs, metrics, extensions, and compute.

Read

[Open](platform/project)

Migrations
Move PostgreSQL data from AWS, GCP, Azure, Neon, Supabase, or self-hosted sources.

Read

[Open](migrations/overview)

Automations
Add Xata branches and clone workflows to CI and Kubernetes.

Read

[Open](automations/ga-pr)

AI Agents
Use Claude Code, Codex, Cursor, Copilot, and other coding agents with isolated branches.

Read

[Open](ai-agents/overview)

Integrations
Connect with Drizzle, Railway, and standard PostgreSQL tooling.

Read

[Open](quickstarts/drizzle)

Changelog
Track platform, console, and CLI updates.

Read

[Open](changelog)

Support
Get help with setup, migration, or production usage.

Contact

[Open](https://xata.io/contact-support)


# Account Management
Source: https://xata.io/docs/platform/account

Manage your account settings, profile, and authentication preferences

![Account settings navigation overview](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/account-settings-nav.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=3942a919dda0edbeff5d702c19391e99)

## Overview

Your Xata account is the foundation of your database management experience. From here, you can update personal information and create personal API keys.

## Profile Settings

Update your personal information and contact details in the profile section of your account settings.

![Account profile settings interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/account-settings-profile.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=9227ab254407059bdf0653489711201f)

* **User ID**: Your unique user ID for the platform
* **Email**: Primary contact email
* **First Name**: Your first name
* **Last Name**: Your last name

## Authentication Settings

Configure authentication and security preferences for access.

![Authentication settings interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/account-settings-auth.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=6977f8bb5c29593268f11c904b8c6b13)

* **Password**: Set and change your account password
* **Google**: Log in with Google SSO
* **GitHub**: Log in with GitHub SSO
* **Passkey**: Register a passkey on your device. Passkeys can be used for passwordless sign-in as a primary authentication method, or as a second factor after a password login.

## Active Sessions

View and manage your active sessions across different devices and locations.

![Active sessions management interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/account-settings-active-sessions.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=613b2b8f863b74c20748141a42dfebdd)

* **Current Sessions**: View all active login sessions
* **Metadata**: View IP, browser, session start, last access and expiration time
* **Action**: Logout of unwanted sessions

## Account Delete

Deleting an account will delete all organizations, projects and branches underneath it. Please contact [support](https://xata.io/contact-support) for assistance.


# API Keys
Source: https://xata.io/docs/platform/api-key

Generate and manage API keys for secure access to your Xata database

![API Keys navigation overview](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/api-keys-nav.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=1879cbe2786539ab73a2d10a8385e684)

## Overview

API keys provide programmatic access to your Xata database without requiring user authentication. They're essential for integrating Xata with your applications and automating database operations.

![API Keys overview interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/api-keys-overview.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=a00b47903349d93ce17199850bf6ee31)

## Scoping

Control what each key can access with granular permissions and restrictions.

![API Keys scoping interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/api-keys-scope.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=f62e1888cfc2cab30ab5fd4706aec250)

### Project and Branch Access

* **All projects and branches**: Full access across your organization
* **Specific branches**: Limited to selected development or production branches

### Functionality Access

Control access to specific platform features:

* **API keys**: Manage your API keys to authenticate requests to your organization
* **Organization**: Manage your organization settings and members
* **Organization deletion**: Submit a deletion request for an organization (`org:delete` scope)
* **Invitations**: Manage organization invitations to add new members
* **Project**: Manage your projects and their settings
* **Branch**: Manage your branches and their settings
* **Credentials**: Read and rotate database credentials for branches
* **Metrics**: View and manage your API usage metrics

Several CLI commands need more than branch access. See [Required scopes](/docs/cli#required-scopes) for which command needs which scope.

## Access Control

* **No Access**: Keys with no access cannot perform any operations
* **Read-Only**: Keys with read-only access can only read or view
* **Read and Write**: Keys with read and write access have full access

## Keys and Organization Roles

A user API key acts with the [organization role](/docs/platform/organization-roles) of the member who created it, so an Editor's key cannot perform Admin-only actions whatever its scopes. Organization API keys act with Admin access, limited by their scopes.

### From the CLI

Create and manage API keys using the Xata CLI:

```bash theme={null}
# Create a user API key
xata keys user create

# Create an organization API key
xata keys organization create

# List user API keys
xata keys user list

# List organization API keys
xata keys organization list
```

For more CLI commands, see the [CLI Reference](/docs/cli/keys).


# Branches
Source: https://xata.io/docs/platform/branch

Create isolated database environments for development, testing, and staging

![Branch list view showing all available branches in the project](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/branch-list-view.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=a46b0050ccd85553fbbb11d73a7f9e4e)

Xata's branching system uses Copy-on-Write (CoW) at the storage layer to create instant database branches. Each branch contains the exact schema and data of the parent branch at the moment of creation, but after creation, branches are independent PostgreSQL instances. Learn more about [how branching works](/docs/core-concepts/branching).

## Overview

The overview page provides a single pane of glass to understand the details of your branch.

![Branch overview showing branch details and configuration options](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/branch-overview.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=e1942e917a5ac58ac9aae23e40b32824)

* **Connect**: Copy a connection string or individual parameters. Choose an [endpoint type](#endpoint-types) to route traffic to the primary, read replicas, or a connection pooler
* **Health**: View the health and state of your primary instance and replicas
* **Instance Details**: Understand the instance health, hierarchy and metadata
* **Create Child**: Create a child branch using the current branch as the source
* **Instance Health**: Click into an instance node to quickly view the latest metrics
* **Rotate Credentials**: Rotate the database password for the `xata` user for enhanced security

![Branch node flyout showing quick actions and branch information](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/branch-node-flyout.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=a20429f025fc22252a42eebafb13c30e)

## Branch Features

> [card] [SQL Editor](/docs/platform/sql-editor)
>
> Execute and manage database queries across your branches.

> [card] [Metrics](/docs/platform/metrics)
>
> Monitor branch performance and usage across your instances within a branch.

> [card] [Logs](/docs/platform/logs)
>
> Inspect PostgreSQL logs from the primary and replicas of your branch.

> [card] [Extensions](/docs/platform/extensions)
>
> Extend your database functionality with extensions.

> [card] [Schemas](/docs/platform/schemas)
>
> Manage your database structure and schema.

> [card] [Migrations](/docs/platform/migrations)
>
> Apply schema changes to your database without downtime.

## Settings

Configure branch-specific settings including instance details, compute and PostgreSQL parameters.

### Instance Details

Configure the fundamental properties and settings for your branch instance.

![Branch settings showing instance details and configuration options](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/branch-settings-instance-details.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=837e7bdc0901d1bb5fe224ddc237efb3)

* **Branch Name**: Set and modify the human-readable branch name
* **Replicas**: Configure the number of read replicas for the branch. Replicas provide high availability, automated failover, and read scaling.
* **Instance Type**: Configure the instance type for this branch
* **Storage**: Adjust storage size for the branch in GB. Storage can only be increased, not decreased, with a maximum of 1024 Gi (1 TB). You are charged based on actual usage, not the configured limit. Note that 0.5-1% of the configured limit is reserved for filesystem overhead.
* **PostgreSQL Image**: Select from available PostgreSQL images including standard postgres, and analytics-optimized. You can upgrade to newer minor versions (e.g., from 17.5 to 17.7) but cannot change the offering type or perform major version upgrades.

> [!WARNING]
>
> **A branch with 0 replicas is not suitable for production workloads.** It has no standby promotion target and cannot automatically fail over. A primary failure, or an update that requires the primary to restart or be replaced, makes the database unavailable until the primary is available again. Configure at least 1 replica for production. See [High availability and replicas](/docs/core-concepts/deployment#high-availability-and-replicas).

### Compute

Configure scale-to-zero settings for cost optimization. [Learn more about compute settings →](/docs/platform/compute)

### Rotate Password

Generate a new password for the PostgreSQL `xata` user on this branch. Existing connections keep working, but new connections must use the new password.

You can rotate the password from the branch settings page or using the CLI:

```bash theme={null}
xata branch rotate-password <branch-name>
```

> [!NOTE]
>
> * Existing database connections remain active during rotation
> * New connections will use the new password once rotation completes
> * PgBouncer automatically picks up the new password for connection pooling
> * There may be a brief period where credentials are unavailable during rotation

### PostgreSQL Configuration

Fine-tune PostgreSQL parameters and settings for optimal database performance.

![Branch settings showing PostgreSQL configuration parameters](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/branch-settings-postgres-config.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=cc44c1f36d30cd42319e73d70c6a9df8)

* **Connections**: Configure connection and authentication settings
* **Resource Consumption**: Configure memory and resource allocation settings
* **Planner**: Configure query planner options
* **IO & Async Behaviour**: Configure input/output and asynchronous behavior
* **Write-Ahead Log**: Configure Write-Ahead logging (WAL)
* **Extensions**: Configure extension-specific parameters (available parameters depend on your branch's image type and PostgreSQL version)

> [!NOTE]
>
> Available PostgreSQL parameters and preload libraries depend on your branch's image type (postgres or analytics) and PostgreSQL major version. Extension-specific parameters only appear when the extension is available for your image type.

### Connection Pooling

Xata uses PgBouncer for connection pooling to efficiently manage database connections. The connection pooler is automatically configured based on your instance size and PostgreSQL settings.

#### Default Pool Size

The `default_pool_size` parameter controls how many server connections PgBouncer maintains per database. This value is automatically calculated as `floor(0.9 * max_connections)` based on your instance's `max_connections` setting.

| Instance | RAM | max\_connections | default\_pool\_size |
| - | - | - | - |
| xata.micro | 1 GB | 50 | 45 |
| xata.small | 2 GB | 100 | 90 |
| xata.medium | 4 GB | 200 | 180 |
| xata.large | 8 GB | 400 | 360 |
| xata.xlarge | 16 GB | 800 | 720 |
| xata.2xlarge | 32 GB | 1,600 | 1,440 |
| xata.4xlarge | 64 GB | 3,200 | 2,880 |
| xata.8xlarge | 128 GB | 5,000 | 4,500 |

If you modify `max_connections` in your PostgreSQL configuration, the `default_pool_size` will automatically adjust on the next reconciliation. You can also override this value directly in your branch configuration if needed.

#### Additional Pooler Settings

Xata configures the following PgBouncer parameters for optimal performance:

* **max\_prepared\_statements**: 1000 - Maximum number of prepared statements to cache
* **query\_wait\_timeout**: 120 seconds - Maximum time queries can wait for a connection

### Preloaded Libraries

Configure extensions and libraries to be preloaded with your branch. Extensions that require preloading must be added to this list before their configuration parameters become available in the PostgreSQL Configuration section.

When you add an extension to the preload list, its parameters will become visible and configurable. If you remove an extension from the preload list, any configured parameters for that extension will be automatically removed from your branch configuration.

## Endpoint Types

When building a connection string, you can select how traffic is routed to your branch. The endpoint type is encoded into the connection string's hostname, so a single branch can expose multiple endpoints simultaneously.

* **Primary only** (`rw`): Full read and write access to the primary instance. This is the default.
* **Primary or Read Replica** (`r`): Connects to either the primary or a replica. Use it only for read-only workloads; writes are unreliable because the connection can land on a read-only replica.
* **Read Replica only** (`ro`): Read-only access guaranteed to use read replicas, keeping load off the primary. Requires at least one replica configured on the branch.
* **Pooler** (`pooled_rw`): Pooled access to the primary through a connection pooler. Recommended for serverless and high-concurrency workloads where many short-lived connections open and close frequently.

You can switch between endpoint types from the **Connect** panel on the branch overview page, or generate a connection string directly from the CLI:

```bash theme={null}
# Pooled connection for a serverless app
xata branch url main --type pooler

# Read-only connection that targets replicas
xata branch url main --type replica
```

See the [`xata branch url`](/docs/cli/branch#url) CLI reference for all available connection types.

## Upgrading PostgreSQL Versions

You can upgrade your branch to a newer minor version of PostgreSQL while maintaining the same major version and offering type. This allows you to benefit from bug fixes and performance improvements in newer PostgreSQL releases.

### Upgrade Requirements

* **Minor versions only**: You can upgrade from one minor version to another within the same major version (e.g., 17.5 → 17.7)
* **Same offering**: The PostgreSQL offering type (postgres or analytics) must remain the same
* **No downgrades**: You cannot downgrade to an older minor version
* **No major version changes**: Major version upgrades (e.g., 16.x → 17.x) are not supported through this method
* **Separate from configuration changes**: Image upgrades cannot be combined with PostgreSQL configuration parameters, instance type, or preload library changes in the same update. Make these changes in separate requests.

> [!WARNING]
>
> Upgrading PostgreSQL versions will trigger a rolling update of your branch instances. Plan upgrades during maintenance windows to minimize impact on your applications.

### In the Console

Navigate to your branch's **Settings** page. Under **Instance Details**, the **Postgres image** field shows your current PostgreSQL version and lists only compatible minor version upgrades. Select the target version and save your changes.

### Using the CLI

Use the `xata branch set postgres-version` command to upgrade. In interactive mode, it presents a list of compatible versions:

```bash theme={null}
xata branch set postgres-version
```

You can also specify the target version directly:

```bash theme={null}
xata branch set postgres-version postgres:17.7
```

The value must be a full image name in `<offering>:<version>` format (e.g., `postgres:17.7`). If the specified version is not a valid upgrade for the current image, the CLI will return an error.

### Using the API

```bash theme={null}
curl -X PATCH "https://api.xata.tech/organizations/{organizationId}/projects/{projectId}/branches/{branchId}" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"image": "postgres:17.7"}'
```

> [!WARNING]
>
> When upgrading the PostgreSQL image, you cannot simultaneously update PostgreSQL configuration parameters, instance type, or preload libraries in the same API call. These changes must be made in separate requests.

## Rotating database credentials

You can rotate the database password for the `xata` user to enhance security. This operation is asynchronous and typically completes within a few seconds.

### Using the API

```bash theme={null}
curl -X POST "https://api.xata.tech/organizations/{organizationId}/projects/{projectId}/branches/{branchId}/credentials/rotate" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"username": "xata"}'
```

After rotation completes, retrieve the new credentials:

```bash theme={null}
curl "https://api.xata.tech/organizations/{organizationId}/projects/{projectId}/branches/{branchId}/credentials?username=xata" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

> [!NOTE]
>
> * Existing database connections remain active during rotation
> * New connections will use the new password once rotation completes
> * PgBouncer automatically picks up the new password for connection pooling
> * There may be a brief period where credentials are unavailable during rotation

## Deleting branches

Branches can be deleted from the console, the CLI, or the API. Deletion permanently removes the branch and its data, and cannot be undone.

### Using the API

```bash theme={null}
curl -X DELETE "https://api.xata.tech/organizations/{organizationId}/projects/{projectId}/branches/{branchId}" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

A successful deletion returns `204 No Content`.

### Using the CLI

```bash theme={null}
xata branch delete <branch-name>
```

See the [`xata branch delete`](/docs/cli/branch#delete) CLI reference for all options.

## From the CLI

Use the Xata CLI to create and manage branches programmatically:

```bash theme={null}
# Create a new branch
xata branch create --name "staging" --parent-branch "main"

# List all branches
xata branch list

# Get branch details
xata branch describe staging
```

For more CLI commands, see the [CLI Reference](/docs/cli/branch).


# Compute
Source: https://xata.io/docs/platform/compute

Configure scale-to-zero and hibernation options for projects and branches

## Overview

With scale to zero, branches automatically hibernate after periods of inactivity. You can also manually hibernate branches that won't be used for extended periods. Learn more about the [benefits of scale to zero](/docs/core-concepts/scale-to-zero).

## Project-Level Settings

Configure scale to zero defaults for all branches in your project. These settings apply to both base branches and child branches, providing a foundation for cost optimization across your entire project.

![Project-level compute settings showing scale to zero configuration options](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/compute-project-settings.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=4db9f195bcedb39012ae8bf574ad3d2d)

Set up automatic hibernation for inactive branches across the project.

* **Enable for base branch**: Enable scale to zero for the base branch in the project
* **Enable for child branches**: Enable scale to zero for all child branches in the project
* **Inactivity period**: Define the duration of inactivity before hibernating the branch (base or child)

## Branch-Level Settings

Override project defaults with branch-specific compute settings. Each branch can have its own scale-to-zero configuration or be manually hibernated for extended periods.

![Branch-level compute settings showing scale to zero and manual hibernation options](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/compute-branch-settings.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=78997bfa0c584a9018014abcc9c32732)

Customize compute settings for individual branches.

* **Scale to zero**: Enable or disable scale to zero for this branch
* **Inactivity period**: Define the duration of inactivity before hibernating the branch
* **Hibernate branch**: Instead of enabling scale to zero, manually hibernate the branch


# Extensions
Source: https://xata.io/docs/platform/extensions

All extensions listed below are currently available for PostgreSQL 14, 15, 16, 17, and 18. These extensions provide additional functionality to enhance your database, from spatial data processing to text search and monitoring tools.

## Available PostgreSQL extensions

Extensions that require preloading must be added to the "Preloaded Libraries" section in your branch settings before their configuration parameters become available. Once an extension is preloaded, you can configure its parameters in the PostgreSQL Configuration section. If you remove an extension from the preload list, any configured parameters for that extension will be automatically removed.

| Extension | Description | Version | Prerequisites |
| - | - | - | - |
| [address\_standardizer](https://postgis.net/docs/Extras.html#Address_Standardizer) | Standardizes address data using rules-based parsing | 3.6.3 | |
| [address\_standardizer\_data\_us](https://postgis.net/docs/Extras.html#Address_Standardizer_Tables) | US address data for address\_standardizer | 3.6.3 | |
| [amcheck](https://www.postgresql.org/docs/current/amcheck.html) | Functions for verifying relation integrity | 1.5 | |
| [auth\_delay](https://www.postgresql.org/docs/current/auth-delay.html) | Configurable delay after failed authentication attempts | 1.0 | Add `auth_delay` to "Preloaded Libraries" in branch settings to configure its parameters. |
| [auto\_explain](https://www.postgresql.org/docs/current/auto-explain.html) | Automatic query plan logging | 1.0 | Add `auto_explain` to "Preloaded Libraries" in branch settings to configure its parameters. |
| [autoinc](https://www.postgresql.org/docs/current/contrib-spi.html#id-1.11.7.47.8) | Functions for automatic increment of fields | 1.0 | |
| [bloom](https://www.postgresql.org/docs/current/bloom.html) | Bloom index access method | 1.0 | |
| [btree\_gin](https://www.postgresql.org/docs/current/btree-gin.html) | Support for indexing common datatypes in GIN | 1.3 | |
| [btree\_gist](https://www.postgresql.org/docs/current/btree-gist.html) | Support for indexing common datatypes in GiST | 1.8 | |
| [citext](https://www.postgresql.org/docs/current/citext.html) | Case-insensitive text data type | 1.8 | |
| [cube](https://www.postgresql.org/docs/current/cube.html) | Multi-dimensional cube data type | 1.5 | |
| [dblink](https://www.postgresql.org/docs/current/dblink.html) | Cross-database queries | 1.2 | |
| [dict\_int](https://www.postgresql.org/docs/current/dict-int.html) | Text search dictionary template for integers | 1.0 | |
| [dict\_xsyn](https://www.postgresql.org/docs/current/dict-xsyn.html) | Text search dictionary template for synonyms | 1.0 | |
| [earthdistance](https://www.postgresql.org/docs/current/earthdistance.html) | Earth distance calculations | 1.2 | |
| [fuzzystrmatch](https://www.postgresql.org/docs/current/fuzzystrmatch.html) | Determine similarities and distance between strings | 1.2 | |
| [hll](https://github.com/citusdata/postgresql-hll) | HyperLogLog data type for cardinality estimation | 2.19 | |
| [hstore](https://www.postgresql.org/docs/current/hstore.html) | Key-value store data type | 1.8 | |
| [hypopg](https://hypopg.readthedocs.io/) | Hypothetical indexes for query planning | 1.4.2 | |
| [intagg](https://www.postgresql.org/docs/current/intagg.html) | Integer aggregator (deprecated in favor of array\_agg) | 1.1 | |
| [intarray](https://www.postgresql.org/docs/current/intarray.html) | Functions for 1-D arrays of integers | 1.5 | |
| [insert\_username](https://www.postgresql.org/docs/current/contrib-spi.html#id-1.11.7.47.9) | Automatic username insertion | 1.0 | |
| [ip4r](https://github.com/RhodiumToad/ip4r) | IPv4 and IPv6 types and functions | 2.4 | |
| [isn](https://www.postgresql.org/docs/current/isn.html) | International Standard Number data types | 1.3 | |
| [lo](https://www.postgresql.org/docs/current/lo.html) | Large Object maintenance functions | 1.2 | |
| [ltree](https://www.postgresql.org/docs/current/ltree.html) | Tree-like structures data type | 1.3 | |
| [moddatetime](https://www.postgresql.org/docs/current/contrib-spi.html#id-1.11.7.47.10) | Automatic timestamp updates | 1.0 | |
| [pageinspect](https://www.postgresql.org/docs/current/pageinspect.html) | Low-level page inspection | 1.13 | |
| [pg\_buffercache](https://www.postgresql.org/docs/current/pgbuffercache.html) | Buffer cache inspection | 1.6 | |
| [pg\_cron](https://github.com/citusdata/pg_cron) | Job scheduler for PostgreSQL | 1.6 | Enable `pg_cron` in the "Preloaded Libraries" section of "Advanced settings" under "Branch settings". |
| [pg\_duckdb](https://github.com/duckdb/pg_duckdb) | DuckDB embedded in Postgres | 1.1.0 | **Analytics image only**. Automatically preloaded on analytics branches. Configure `duckdb.postgres_role` parameter to set the PostgreSQL role allowed to use DuckDB privileged features (defaults to `xata`). |
| [pg\_freespacemap](https://www.postgresql.org/docs/current/pgfreespacemap.html) | Free space map examination | 1.3 | |
| [pg\_hint\_plan](https://github.com/ossc-db/pg_hint_plan) | Query execution plan hinting | 1.8.0 | Available on PostgreSQL 16, 17, and 18 only. Add `pg_hint_plan` to "Preloaded Libraries" in branch settings to configure its parameters. |
| [pg\_partman](https://github.com/pgpartman/pg_partman) | Partition management extension for PostgreSQL | 5.4.1 | |
| [pg\_prewarm](https://www.postgresql.org/docs/current/pgprewarm.html) | Database prewarming | 1.2 | Add `pg_prewarm` to "Preloaded Libraries" in branch settings to configure its parameters. |
| [pg\_repack](https://github.com/reorg/pg_repack) | Remove bloat from tables and indexes without exclusive lock | 1.5.3 | |
| [pg\_surgery](https://www.postgresql.org/docs/current/pgsurgery.html) | Row surgery functions for modifying rows | 1.0 | |
| [pg\_trgm](https://www.postgresql.org/docs/current/pgtrgm.html) | Trigram matching for similarity searches | 1.6 | |
| [pg\_visibility](https://www.postgresql.org/docs/current/pgvisibility.html) | Visibility map examination | 1.2 | |
| [pg\_walinspect](https://www.postgresql.org/docs/current/pgwalinspect.html) | WAL inspection functions | 1.1 | Available on PostgreSQL 16, 17, and 18 only. |
| [pgrowlocks](https://www.postgresql.org/docs/current/pgrowlocks.html) | Row locking information | 1.2 | |
| [pg\_stat\_statements](https://www.postgresql.org/docs/current/pgstatstatements.html) | Query execution statistics | 1.12 | Add `pg_stat_statements` to "Preloaded Libraries" in branch settings to configure its parameters. |
| [pgstattuple](https://www.postgresql.org/docs/current/pgstattuple.html) | Functions to show tuple-level statistics | 1.5 | |
| [pgaudit](https://github.com/pgaudit/pgaudit) | Audit logging extension for PostgreSQL | 18.0 | Add `pgaudit` to "Preloaded Libraries" in branch settings to configure its parameters. |
| [pgcrypto](https://www.postgresql.org/docs/current/pgcrypto.html) | Cryptographic functions | 1.4 | |
| [pgrouting](https://docs.pgrouting.org/) | Routing functionality for PostGIS/PostgreSQL | 4.0.1 | |
| [pgtap](https://pgtap.org/) | Unit testing framework for PostgreSQL | 1.3.4 | |
| [plpgsql](https://www.postgresql.org/docs/current/plpgsql.html) | PL/pgSQL procedural language | 1.0 | |
| [postgres\_fdw](https://www.postgresql.org/docs/current/postgres-fdw.html) | Foreign data wrapper for PostgreSQL | 1.2 | |
| [postgis](https://postgis.net/) | PostGIS geometry and geography spatial types and functions | 3.6.3 | |
| [postgis\_raster](https://postgis.net/docs/RT_reference.html) | PostGIS raster extension | 3.6.3 | |
| [postgis\_sfcgal](https://www.postgis.net/docs/manual-dev/en/reference_sfcgal.html) | PostGIS SFCGAL support | 3.6.3 | |
| [postgis\_tiger\_geocoder](https://postgis.net/docs/Extras.html#Tiger_Geocoder) | PostGIS tiger geocoder and reverse geocoder | 3.6.3 | |
| [postgis\_topology](https://postgis.net/docs/Topology.html) | PostGIS topology spatial types and functions | 3.6.3 | |
| [prefix](https://github.com/dimitri/prefix) | Prefix matching operator | 1.2.0 | |
| [refint](https://www.postgresql.org/docs/current/contrib-spi.html#id-1.11.7.47.11) | Referential integrity triggers (example) | 1.0 | |
| [seg](https://www.postgresql.org/docs/current/seg.html) | Floating-point intervals | 1.4 | |
| [sslinfo](https://www.postgresql.org/docs/current/sslinfo.html) | Information about SSL connections | 1.2 | |
| [tablefunc](https://www.postgresql.org/docs/current/tablefunc.html) | Functions that manipulate whole tables, including crosstab | 1.0 | |
| [tcn](https://www.postgresql.org/docs/current/tcn.html) | Triggered change notifications | 1.0 | |
| [tsm\_system\_rows](https://www.postgresql.org/docs/current/tsm-system-rows.html) | TABLESAMPLE method which accepts number of rows | 1.0 | |
| [tsm\_system\_time](https://www.postgresql.org/docs/current/tsm-system-time.html) | TABLESAMPLE method which accepts time in milliseconds | 1.0 | |
| [unaccent](https://www.postgresql.org/docs/current/unaccent.html) | Text search dictionary that removes accents | 1.1 | |
| [uuid-ossp](https://www.postgresql.org/docs/current/uuid-ossp.html) | UUID generation functions | 1.1 | |
| [vector](https://github.com/pgvector/pgvector) | Vector similarity search for PostgreSQL | 0.8.1 | |
| [wal2json](https://github.com/eulerto/wal2json) | Logical replication output plugin for JSON | 2.6 | |

## Request an extension

**Need an extension we don't have?** 📩 [Request an extension](https://xata.canny.io/xata-postgres-extensions)

We're always looking to expand our extension offerings. If you need an extension that's not currently available, please let us know through our feature request portal. We'll review your request and respond there.

## Getting help

If you need help with a specific extension, we recommend checking the official documentation linked in the table above. For general PostgreSQL extension support, you can also refer to the [PostgreSQL documentation](https://www.postgresql.org/docs/current/).

If you run into an issue or discover a bug with an extension, we recommend reporting it to the extension's upstream maintainers. If a fix is released, we're happy to update to the latest version of the extension.

For additional support, you can also [contact our support team](https://xata.io/contact-support).

## Custom extensions

We don't actively maintain third-party extension code, but we're happy to help you get the extensions you need. If you require a custom extension that's not listed above, we encourage you to [contact our support team](https://xata.io/contact-support) to discuss.

### BYOC for custom extensions

If you need to install custom extensions that aren't available in our standard offering, consider our [Bring Your Own Cloud (BYOC) deployment model](/docs/core-concepts/deployment#bring-your-own-cloud-byoc). This allows you to run the entire Xata dataplane in your own Kubernetes cluster, giving you full control over which extensions are installed and configured.


# Logs
Source: https://xata.io/docs/platform/logs

Inspect PostgreSQL logs from your branch instances

Branch logs let you inspect PostgreSQL server output across the primary and replicas of a branch. Logs are useful for debugging slow queries, connection issues, replication problems, and other runtime behavior.

Branch logs are generally available on every branch — no opt-in or feature flag is required.

> [!NOTE]
>
> Logs are retained for 30 days. Entries older than that are no longer available from the dashboard, the CLI, or the API.

## Viewing logs in the dashboard

Open a branch in the Xata dashboard and select **Logs** in the branch sidebar to browse entries in a searchable viewer.

From the Logs page you can:

* Pick a preset or custom time range.
* Filter by **level** (`error`, `warning`, `info`, `debug`), **instance** (primary or a specific replica), and **process**.
* Search log message bodies with substring or regex matching.
* Click any entry to open a details panel with the full message and metadata.
* Use **Refresh** to fetch the latest entries for the current time range.

To read logs from a terminal or a script, use [`xata branch logs`](/docs/cli/branch#logs), which offers the same filters plus follow mode and `raw`, `json`, `ndjson`, and `csv` output. For anything else programmatic, such as shipping logs to an external observability tool, use the API described below.

## Retrieving logs

Use the `branchLogs` endpoint to fetch logs for a time range. The endpoint requires the `logs:read` scope on your API key.

```bash theme={null}
curl -X POST "https://api.xata.tech/organizations/{organizationId}/projects/{projectId}/branches/{branchId}/logs" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "start": "2026-06-09T00:00:00Z",
    "end":   "2026-06-09T01:00:00Z",
    "limit": 100
  }'
```

The response contains a `logs` array of entries and a `nextCursor` that you can pass back in a follow-up request to page through additional results.

```json theme={null}
{
  "start": "2026-06-09T00:00:00Z",
  "end": "2026-06-09T01:00:00Z",
  "logs": [
    {
      "timestamp": "2026-06-09T00:00:01.234Z",
      "instanceID": "branch-xxx-1",
      "level": "info",
      "process": "postgres",
      "message": "database system is ready to accept connections"
    }
  ],
  "nextCursor": null
}
```

## Request parameters

| Field | Type | Description |
| - | - | - |
| `start` | `string` (RFC 3339) | Start of the time range. Required. Logs are retained for 30 days, so ranges older than that return no entries. |
| `end` | `string` (RFC 3339) | End of the time range. Required. |
| `limit` | `integer` | Maximum number of entries to return per request. Default `100`, maximum `200`. To retrieve more, page with `cursor` as described below, or use [`xata branch logs --limit`](/docs/cli/branch#logs), which pages for you up to 1000 entries. |
| `cursor` | `string` | Pagination cursor returned by a previous response. |
| `filters` | `array` | Optional filters. Multiple filters are combined with `AND`. |

## Log entry fields

| Field | Description |
| - | - |
| `timestamp` | Time the log was emitted. |
| `instanceID` | Instance that produced the entry (primary or replica). |
| `level` | One of `debug`, `info`, `warning`, `error`. |
| `process` | Name of the PostgreSQL process that emitted the log. |
| `message` | The log message. |

## Filtering

Pass one or more `filters` to narrow results. Each filter applies to a specific `field` with an `op` and either a `value` or `values`:

| Field | Description |
| - | - |
| `instance` | Filter by instance ID. |
| `level` | Filter by log level. |
| `process` | Filter by PostgreSQL process name. |
| `body` | Filter by the log message body. |

Supported operators:

* `in` — match any value in `values` (use with `instance`, `level`, `process`).
* `contains` / `icontains` — substring match on the log body (case-sensitive / insensitive).
* `regex` / `iregex` — regular-expression match on the log body.

For example, fetch only error-level entries from the primary instance that mention `deadlock`:

```bash theme={null}
curl -X POST "https://api.xata.tech/organizations/{organizationId}/projects/{projectId}/branches/{branchId}/logs" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "start": "2026-06-09T00:00:00Z",
    "end":   "2026-06-09T01:00:00Z",
    "filters": [
      { "field": "level",    "op": "in",        "values": ["error"] },
      { "field": "instance", "op": "in",        "values": ["branch-xxx-1"] },
      { "field": "body",     "op": "icontains", "value":  "deadlock" }
    ]
  }'
```

## Credential redaction

Branch logs are scrubbed before they reach the API response. If a `CREATE ROLE`, `ALTER ROLE`, `CREATE USER`, `ALTER USER`, `CREATE GROUP`, `ALTER GROUP`, or `CREATE/ALTER USER MAPPING` statement appears in a log line, the statement is truncated at the `PASSWORD` keyword and the secret is replaced with `<REDACTED>`. This applies to both single-line and multi-line log records.

For example, a log line like:

```text theme={null}
LOG: statement: ALTER ROLE app_user WITH PASSWORD 'hunter2'
```

is returned as:

```text theme={null}
LOG: statement: ALTER ROLE app_user WITH PASSWORD <REDACTED>
```

Redaction is applied on the read path, so it covers historical entries as well as newly emitted logs. Other `password` mentions outside role and user-mapping statements are left untouched.

## Pagination

When a response includes a non-null `nextCursor`, pass it back as `cursor` in the next request to fetch the following page. The `start`, `end`, `limit`, and `filters` should remain the same across paged requests.

```bash theme={null}
curl -X POST "https://api.xata.tech/organizations/{organizationId}/projects/{projectId}/branches/{branchId}/logs" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "start":  "2026-06-09T00:00:00Z",
    "end":    "2026-06-09T01:00:00Z",
    "cursor": "<nextCursor from previous response>"
  }'
```


# MCP Server
Source: https://xata.io/docs/platform/mcp

Connect Cursor, Claude, VS Code, and other MCP clients to Xata

The Xata MCP server lets AI assistants and agents interact with your Xata organizations, projects, and branches using the [Model Context Protocol](https://modelcontextprotocol.io) (MCP).

## What is the Xata MCP server?

* A hosted MCP server that runs alongside the Xata API — there is nothing to install or run locally.
* Authenticated via OAuth in your browser, or with a Xata API key for headless environments.
* Accessible from any MCP client that supports remote servers over Streamable HTTP.

**Server URL:**

```
https://api.xata.tech/mcp
```

The server uses the **Streamable HTTP** transport. There is no SSE endpoint and no local (npm) version of the server.

## Authentication

The MCP server supports two authentication methods:

| Method | Use when | Client requirement |
| - | - | - |
| OAuth | Interactive use in an editor/chat | Support for MCP OAuth (dynamic client registration) |
| API key | Automation, CI, headless agents | Support for custom HTTP headers |

### OAuth

With OAuth-capable clients, you only need the server URL. When your client connects for the first time, it registers itself with Xata, opens a browser window, and asks you to sign in to your Xata account and approve access. Tokens are short-lived and scoped to the MCP server.

### API key

Clients that support custom headers can authenticate with a [Xata API key](/docs/platform/api-key) instead:

```
Authorization: Bearer YOUR_XATA_API_KEY
```

> [!WARNING]
>
> Create a dedicated API key for MCP access rather than reusing an existing key. Store it in an environment variable or your client's secret storage — never commit it to source control.

## Set up your MCP client

### Cursor

> [!TIP]
>
> Cursor offers a deep link for quick OAuth setup:
>
> [Add to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=xata\&config=eyJ1cmwiOiJodHRwczovL2FwaS54YXRhLnRlY2gvbWNwIn0%3D)

Alternatively, you can add it manually:

1. Open the command palette and search for "Cursor Settings".
2. Under **Tools & MCP**, click **New MCP Server**.
3. Add the Xata server to the configuration file that opens:

```json .cursor/mcp.json theme={null}
{
  "mcpServers": {
    "xata": {
      "url": "https://api.xata.tech/mcp"
    }
  }
}
```

4. Save the file. Cursor prompts you to authenticate — follow the browser flow and approve access to your Xata account.

### Claude Code

Add the server from your terminal:

```bash theme={null}
claude mcp add --transport http xata https://api.xata.tech/mcp
```

Then start Claude Code and run the `/mcp` slash command. Select the `xata` server and follow the browser instructions to authenticate.

To use an API key instead of OAuth (for example, in CI):

```bash theme={null}
claude mcp add --transport http xata https://api.xata.tech/mcp \
  --header "Authorization: Bearer YOUR_XATA_API_KEY"
```

### VS Code

MCP servers in VS Code require the [GitHub Copilot](https://marketplace.visualstudio.com/items?itemName=github.copilot) and [GitHub Copilot Chat](https://marketplace.visualstudio.com/items?itemName=github.copilot-chat) extensions.

1. Open the Command Palette (`Cmd+Shift+P` / `Ctrl+Shift+P`).
2. Run **MCP: Add Server** and choose **HTTP**.
3. Enter `https://api.xata.tech/mcp` as the URL and `xata` as the name.

Alternatively, add it to your configuration manually:

```json .vscode/mcp.json theme={null}
{
  "servers": {
    "xata": {
      "type": "http",
      "url": "https://api.xata.tech/mcp"
    }
  }
}
```

Start the server from **MCP: List Servers** and allow it to authenticate when prompted.

### Claude (web and desktop)

> [!TIP]
>
> Open Claude's custom connector dialog with Xata's details prefilled:
>
> [Connect Xata to Claude](https://claude.ai/customize/connectors?modal=add-custom-connector\&connectorName=Xata\&connectorUrl=https%3A%2F%2Fapi.xata.tech%2Fmcp)
>
> Review and confirm the connector in Claude, then authenticate with Xata.

Alternatively, add Xata as a custom connector manually:

1. Go to **Settings** → **Connectors**.
2. Click **Add custom connector**.
3. Enter `https://api.xata.tech/mcp` as the server URL and click **Add**.
4. Follow the prompts to sign in with your Xata account.

> [!NOTE]
>
> Custom connectors using remote MCP are not available on all Claude plans, and may require an organization owner to add them on team plans. See the [Claude documentation](https://support.claude.com/en/articles/11175166-getting-started-with-custom-connectors-using-remote-mcp) for details.

### ChatGPT

Connect ChatGPT to Xata using a custom connector:

1. In ChatGPT, go to **Settings** → **Connectors** → **Advanced settings** and enable **Developer mode**.
2. On the Connectors tab, create a new connector with the server URL:

```
https://api.xata.tech/mcp
```

3. Choose **OAuth** for authentication and complete the authorization flow when prompted.
4. In each chat where you want to use Xata, click the **+** button and enable the Xata connector under **Add sources**.

### Codex CLI

Add the Xata server:

```bash theme={null}
codex mcp add xata --url https://api.xata.tech/mcp
```

> [!NOTE]
>
> The `add` command may open a browser and report an OAuth error. If that happens, continue with the login command below; the `xata` server entry has already been saved.

Authenticate with Xata using explicit OAuth scopes:

```bash theme={null}
codex mcp login xata --scopes mcp-client,offline_access
```

Complete authorization in the browser. The `offline_access` scope allows Codex to refresh its Xata session without requiring another browser authorization.

Then start `codex`, run `/mcp`, and verify that `xata` is connected and authenticated.

### Antigravity CLI

Add Xata to your global MCP configuration:

```json ~/.gemini/config/mcp_config.json theme={null}
{
  "mcpServers": {
    "xata": {
      "serverUrl": "https://api.xata.tech/mcp"
    }
  }
}
```

To enable Xata only for one project, use `.agents/mcp_config.json` in that project's root instead.

Start `agy` and enter `/mcp`. In the MCP Manager, use **Authenticate** for `xata` and follow the prompts to complete OAuth.

### OpenCode

Add the Xata server to your OpenCode configuration file:

```json ~/.config/opencode/opencode.json theme={null}
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "xata": {
      "type": "remote",
      "url": "https://api.xata.tech/mcp"
    }
  }
}
```

Then authenticate from your terminal:

```bash theme={null}
opencode mcp auth xata
```

### Amp

Add the server from your terminal:

```bash theme={null}
amp mcp add xata https://api.xata.tech/mcp
```

Then start `amp` — you should be prompted to authenticate in the browser. Run `/mcp list tools` to confirm the server is connected.

### Windsurf

1. In Windsurf, open the Cascade panel and click the MCP (hammer) icon, then **Configure** to open the raw configuration file (`~/.codeium/windsurf/mcp_config.json`).
2. Add the Xata server entry:

```json ~/.codeium/windsurf/mcp_config.json theme={null}
{
  "mcpServers": {
    "xata": {
      "serverUrl": "https://api.xata.tech/mcp"
    }
  }
}
```

3. Save the file and click **Refresh** in the Cascade sidebar. Complete the OAuth flow when the browser window opens.

### Zed

1. Open **Settings** → **AI** → **MCP Servers** and click **Add Server** → **Add Remote Server**, or edit your settings file directly:

```json settings.json theme={null}
{
  "context_servers": {
    "xata": {
      "url": "https://api.xata.tech/mcp"
    }
  }
}
```

2. Zed prompts you to authenticate against the server using the standard MCP OAuth flow.

### Cline

1. Open Cline in VS Code and click the **MCP Servers** icon.
2. In the **Remote Servers** tab, enter `xata` as the name, `https://api.xata.tech/mcp` as the URL, and choose **Streamable HTTP** as the transport. Or edit the configuration JSON directly:

```json theme={null}
{
  "mcpServers": {
    "xata": {
      "type": "streamableHttp",
      "url": "https://api.xata.tech/mcp"
    }
  }
}
```

> [!NOTE]
>
> The transport type must be `streamableHttp` (camelCase). Omitting it causes Cline to fall back to the legacy SSE transport, which the Xata MCP server does not support.

### Other MCP clients

Any MCP client can connect if it supports:

* Remote MCP servers over **Streamable HTTP** (not SSE)
* **OAuth with dynamic client registration**, or **custom HTTP headers** for API key authentication

Consult your client's documentation for where to configure remote MCP servers, and use `https://api.xata.tech/mcp` as the URL.

## Verify the connection

After connecting, ask your assistant:

> Use the Xata MCP server to find the REST API operation for listing branches.

The assistant should call `search_operations` with `{"query":"list branches"}` and return the `listBranches` operation, which can be invoked through `call_read_operation`. If it does, the connection is working.

## Available tools

The Xata MCP server exposes the following tools:

| Tool | Description |
| - | - |
| `search_operations` | Find a Xata REST API operation by intent (for example, "list branches" or "invite member"). |
| `describe_operation` | Return the parameters and request/response schemas for a specific operation. |
| `call_read_operation` | Invoke a read-only Xata REST API operation. |
| `call_write_operation` | Invoke a Xata REST API operation that creates or updates data. |
| `call_destructive_operation` | Invoke a Xata REST API operation that destroys data or revokes access. Requires `confirm=true`. |
| `run_sql` | Run SQL against a branch. Read-only by default; statements that mutate data require both `write=true` and `confirm=true`. |
| `describe_schema` | List the tables and columns of a branch. |
| `list_skills` | List the available Xata skills — guided workflows for common multi-step tasks. |
| `get_skill` | Read the instructions for a specific skill. |
| `search_xata` | Search the Xata documentation. |
| `query_docs_filesystem_xata` | Read Xata documentation pages by path. |

## Security

* Prefer OAuth for interactive clients; tokens are short-lived and can be revoked by disconnecting the server in your client.
* For automation, use a dedicated [API key](/docs/platform/api-key) and rotate it regularly.
* Some tools can modify your data: `call_write_operation` and `call_destructive_operation` can change or delete resources (the latter requires `confirm=true`), and `run_sql` can mutate data when called with both `write=true` and `confirm=true`. Review the actions your assistant proposes before approving them, and keep a human in the loop for any write or delete.

## Troubleshooting

**Authentication keeps failing or loops.** Remove the Xata server from your client, restart the client, and add the server again to trigger a fresh OAuth flow.

**Authentication shows "Client not found" in the browser.** Your MCP client may have a saved OAuth registration that is no longer available on Xata's authentication server. Depending on the client, removing and re-adding the MCP server may leave this registration intact. Clear the saved OAuth registration for Xata in your client, then restart the Xata MCP server and authenticate again to allow a fresh registration.

For example, in **VS Code**:

1. Open the Command Palette (`Cmd+Shift+P` / `Ctrl+Shift+P`).
2. Run **Authentication: Remove Dynamic Authentication Providers**.
3. Select the provider associated with Xata, leaving unrelated providers unchanged.
4. Restart the Xata MCP server and complete authentication again.

**The server connects but no tools show up.** Make sure you completed the authentication step — most tools require a valid session before they appear. Re-run your client's authentication flow, then refresh its tool list. See [Available tools](#available-tools) for the full set.

**Your client can't connect at all.** Confirm the URL is exactly `https://api.xata.tech/mcp` and that your client supports Streamable HTTP. SSE-only clients are not supported.

**The server doesn't appear in your client.** Check the client's MCP configuration file syntax — the JSON shape differs between clients (`mcpServers` vs `servers` vs `context_servers`, `url` vs `serverUrl`) — and check the client's logs. Most clients require a full restart after configuration changes.


# Metrics
Source: https://xata.io/docs/platform/metrics

Monitor your database performance across instances with real-time metrics

![Metrics dashboard showing database performance monitoring with CPU, memory, and disk usage charts](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/metrics.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=78f87f8de9dfa303b23c72d1544e23aa)

Monitor your database performance across instances with real-time metrics including CPU, memory, disk usage, and more. You can view trends as far back as 30 days with minute level granularity.

The same metrics are available from a terminal with [`xata branch metrics`](/docs/cli/branch#metrics), including a live view via `-w` that renders a dashboard in an interactive terminal and streams NDJSON when piped into another process.

## Aggregations

Xata provides various aggregation methods to help you analyze your metrics data effectively.

* **Average**: Mean values over time periods
* **Min/Max**: Peak and valley values
* **Percentiles**: P50, P95, P99 for performance analysis `Coming Soon`

## Metric Definitions

By default, all metrics have a 1 minute granularity and a retention period of 30 days.

| Metric | Description | Unit |
| - | - | - |
| CPU Usage | Shows the cumulative usage by combining all available cores (e.g. 200% = two fully used cores) | Percentage |
| Memory Usage | Shows the amount of memory being used by your application | Bytes |
| Disk Usage | Shows the amount of disk being used by your application | Bytes |
| Connection Count | Shows the number of active and idle client connections to your database | Count |
| Read Throughput | Shows the amount of data read in bytes | Bytes |
| Write Throughput | Shows the amount of data written in bytes | Bytes |
| Network Ingress | Shows the amount of incoming network traffic to your database instances | Bytes |
| Network Egress | Shows the amount of outgoing network traffic from your database instances | Bytes |
| Read Latency | Shows the time it takes to complete read operations in milliseconds | Milliseconds |
| Write Latency | Shows the time it takes to complete write operations in milliseconds | Milliseconds |
| Read IOPS | Shows the number of read operations performed by your database instances | Count |
| Write IOPS | Shows the number of write operations performed by your database instances | Count |
| Wal Sync Time | Shows the time it takes to synchronize Write-Ahead Log (WAL) data to disk in milliseconds | Milliseconds |
| Replication Lag | Shows the time it takes for the replica to catch up with the primary | Seconds |


# Migrations
Source: https://xata.io/docs/platform/migrations

Zero-downtime schema migrations with automatic rollback support

![Migrations interface showing version history, recent activity, and database migration management](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/migrations.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=17b84c63929ab3f378828506c211af09)

Apply schema changes safely with zero-downtime migrations using `xata roll`. Xata provides comprehensive migration management with automatic rollback support, multi-version schema handling, and lock-safe operations. Track recent migration activity in the console and manage multiple schemas simultaneously. Learn more about [how schema changes work](/docs/core-concepts/schema-changes).

## Making Schema Changes

Xata uses [pgroll](https://github.com/xataio/pgroll) behind the scenes to enable zero-downtime, reversible schema changes for PostgreSQL. This approach allows you to apply schema changes without locking tables for extended periods, ensuring your application remains available during migrations.

### Key Features

* **Zero-downtime migrations**: Apply schema changes without service interruption
* **Automatic rollback**: Instantly revert changes if issues arise
* **Multi-version support**: Serve both old and new schema versions simultaneously
* **Lock-safe operations**: Minimal table locking during migration execution
* **Declarative approach**: Express migrations declaratively rather than using raw SQL

### Migration Process

1. **Start migration**: Begin the migration process with `xata roll migrate`
2. **Monitor progress**: Check migration status and activity in the console
3. **Test changes**: Verify application compatibility with new schema
4. **Complete migration**: Finalize changes when ready with `xata roll complete`
5. **Rollback if needed**: Use `xata roll rollback` to revert changes instantly

For detailed step-by-step instructions, see our [schema change tutorial](/docs/tutorials/schema-change).

## From the CLI

Use the Xata CLI to manage migrations programmatically:

```bash theme={null}
# Initialize migration tracking
xata roll init

# Apply migrations
xata roll migrate

# Check migration status
xata roll status

# Rollback if needed
xata roll rollback

# Complete migration
xata roll complete
```

For more CLI commands, see the [CLI Reference](/docs/cli/roll).


# Organizations
Source: https://xata.io/docs/platform/organization

Collaborate with your team in a shared organization

## What is an Organization?

An organization is a hierarchical structure that supports multiple projects, each with their own database and branches. You can invite members to an organization and projects, and this is the entity you are billed on.

Each member is either an Admin or an Editor. Admins manage the organization, including its members, billing and API keys; Editors work with its projects and branches. See [Organization roles](/docs/platform/organization-roles) for what each role can do.

![Organization overview interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/organization.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=8c6dc3416156fde7d5f3a46466b4114c)

## Create an Organization

Set up your organization to start collaborating with your team and managing multiple projects.

![Create organization interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/organization-create.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=705b068133e492fb23ee12d09cda6ec1)

### From the CLI

Use the Xata CLI to create organizations programmatically:

```bash theme={null}
# Create a new organization
xata organization create --name "My Company"
```

For more CLI commands, see the [CLI Reference](/docs/cli/organization).

## Invite Members

Admins add team members to your organization by sending invitation emails. Each invitation sets the role the new member holds once they accept, **Editor** by default. See [Organization roles](/docs/platform/organization-roles) for how to choose or change a role.

![Invite members interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/organization-invite-members.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=07493f80faece810fe5632dd1163c90b)

### Managing Invitations

You can manage organization invitations through the Console, CLI and API. Every member can list and view invitations; sending, resending and deleting them requires Admin:

* **Send invitations**: Invite users by email to join your organization as an Admin or an Editor
* **List invitations**: View all pending and expired invitations with filtering options
* **View invitation details**: Get information about a specific invitation
* **Resend invitations**: Send a fresh invitation email with an extended expiration time
* **Delete invitations**: Cancel an existing invitation

## Organization API Keys

Manage API keys at the organization level for team-wide access and permissions.

![Organization API keys interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/organization-api-key.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=7a72f1c0648755b2d01f00e99a1ed23f)

Organization API keys provide centralized access control across all projects and branches within your organization. These keys can be scoped to specific projects, branches, or functionality levels.

Only Admins can create, list and delete organization API keys. An organization API key acts with Admin access, limited by its scopes.

For more detailed information about API key management, see [Account API Keys](/docs/platform/api-key).

## Billing

Manage billing and usage across your entire organization with centralized cost tracking and allocation.

![Organization billing interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/organization-billing.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=c50d6e7bdc6b9aa115aa516f09aa9536)

Organization billing allows you to view both upcoming and historic invoices. Only Admins can view and change billing.

## Request Organization Deletion

Admins can submit a deletion request for an organization through the API. This flags the organization for deletion. Once accepted, the organization is immediately inaccessible.

```bash theme={null}
curl -X POST https://api.xata.tech/organizations/{organizationID}/deletion-request \
  -H "Authorization: Bearer $XATA_API_KEY"
```

A successful request returns `202 Accepted`.

> [!NOTE]
>
> The request is rejected with `409 Conflict` if the organization still has active projects or outstanding invoices. Delete or transfer all projects and settle outstanding invoices before requesting deletion.

This endpoint requires an API key with the `org:delete` scope. See [API Keys](/docs/platform/api-key) for details on scoping.

For the immediate destructive variant, use the `DELETE /organizations/{organizationID}` endpoint, or the CLI:

```bash theme={null}
xata organization delete --organization <id>
```


# Organization roles
Source: https://xata.io/docs/platform/organization-roles

Choose who can manage your organization's members, billing and API keys

Every member of an [organization](/docs/platform/organization) is either an **Admin** or an **Editor**. Admins manage the organization itself. Editors work with its projects and branches.

The person who creates an organization is its first Admin.

> [!NOTE]
>
> Members who joined before roles were introduced are Admins. Change anyone who does not need to manage the organization to Editor.

## Permissions

| Action | Admin | Editor |
| - | :-: | :-: |
| Create and manage projects and branches | Yes | Yes |
| Connect to and query databases | Yes | Yes |
| View the organization, its members and invitations | Yes | Yes |
| Invite and remove members | Yes | No |
| Change a member's role | Yes | No |
| Manage organization API keys | Yes | No |
| View and change billing | Yes | No |
| Configure [single sign-on](/docs/platform/sso) | Yes | No |
| Rename or delete the organization | Yes | No |

An Editor who attempts an Admin-only action through the API or CLI gets a `403 Forbidden` response.

## Invite a member

Select **Organization Settings** in the sidebar, then **Members**. Enter the email address, choose a role and select **Send Invitation**. New invitations default to **Editor**.

### From the CLI

```bash theme={null}
# Invite an Admin; omit --role to invite an Editor
xata organization members invite --email jane@example.com --role admin
```

## Change a member's role

On the **Members** page, pick a new role from the dropdown next to the member. The change applies immediately, without signing out.

An organization always keeps at least one Admin, so Xata refuses to demote or remove its last Admin. If you demote yourself, only another Admin can make you an Admin again.

### From the CLI

```bash theme={null}
# Find the member's user ID with: xata organization members list
xata organization members set-role --user-id <user-id> --role editor
```

### From the API

```bash theme={null}
curl -X PUT https://api.xata.tech/organizations/{organizationID}/members/{userID}/role \
  -H "Authorization: Bearer $XATA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"role": "editor"}'
```

## API keys

A user API key acts with the current role of the member who created it, so an Editor's key cannot perform Admin-only actions whatever its scopes. Organization API keys act with Admin access, limited by their scopes. See [API Keys](/docs/platform/api-key).


# AWS PrivateLink
Source: https://xata.io/docs/platform/privatelink

Connect to your Xata Postgres databases privately from your AWS VPC, without traffic leaving the AWS network

AWS PrivateLink lets you reach your Xata databases from your own AWS VPC over
the AWS private network. Traffic never traverses the public internet, and your
connection strings stay exactly the same.

PrivateLink is available on request. Please contact us via our regular support channels and
provide us with:

* The **AWS region** of your Xata databases (e.g. `us-east-1`)
* The **AWS account ID** (or IAM role/user ARN) you will create the endpoint
  from.

In return we will provide you with the **service name** of the Xata
VPC endpoint service to connect to. It will be in the form
`com.amazonaws.vpce.us-east-1.vpce-svc-...`.

## How it works

Xata exposes its Postgres gateway as a VPC endpoint service. You create an
**interface VPC endpoint** against it in your VPC, which places network
interfaces (ENIs) with private IPs in your subnets. With private DNS enabled,
your existing database hostnames (`<branch>.us-east-1.xata.tech`) resolve to
those private IPs inside your VPC. This means the same connection string uses the
private path from your VPC and the public path from everywhere else.

## Prerequisites

* A VPC in the same region and Availability Zone (AZ) as your Xata databases.
* The VPC must have **DNS support** and **DNS hostnames** enabled
  (`enableDnsSupport` and `enableDnsHostnames`, both on by default for most
  VPCs).

## Step 1 — Create a security group for the endpoint

The endpoint's network interfaces need a security group that allows inbound
traffic from your clients:

* TCP `5432` (Postgres wire protocol) from your application's security group
  or CIDR ranges
* TCP `443` (HTTPS) from the same sources — only needed if you use the
  serverless HTTP driver

## Step 2 — Create the interface VPC endpoint

In the AWS console, go to **VPC → PrivateLink and Lattice → Endpoints** and
click **Create endpoint**:

1. Optionally give the endpoint a **Name tag** (e.g. `xata`).
2. Under **Type**, select **Endpoint services that use NLBs and GWLBs**
   ("Find services shared with you by service name").
3. Leave **Enable Cross Region endpoint** unchecked. Your endpoint must be
   in the same region as your Xata databases.
4. In the **Service name** field, paste the service name provided by Xata
   (`com.amazonaws.vpce.us-east-1.vpce-svc-…`) and click **Verify service**.
   If this fails with a permissions error, please contact us.
5. The form expands: select your VPC and one subnet per Availability Zone
   you want to use.
6. Attach the security group from step 1.
7. Enable **Private DNS names**. This is what makes your existing Xata
   hostnames resolve to the endpoint inside your VPC. No DNS changes should be
   needed on your side. You can also enable it later from the endpoint's
   **Actions → Modify private DNS name**.
8. Click **Create endpoint**.

The Terraform equivalent:

```hcl theme={null}
resource "aws_vpc_endpoint" "xata" {
  vpc_id              = var.vpc_id
  service_name        = "<service name provided by Xata>"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.xata_endpoint.id]
  private_dns_enabled = true
}
```

## Step 3 — Connect

Connect to the database endpoints as shown in the Xata console. For example:

```bash theme={null}
psql "postgresql://<user>:<password>@<branch>.us-east-1.xata.tech:5432/<database>?sslmode=verify-full"
```

Inside your VPC, the hostname now resolves to your endpoint's private IPs and
traffic flows over PrivateLink. From anywhere else, the same hostname resolves
to the public endpoint as usual. You can verify the private path from an
instance in your VPC:

```bash theme={null}
dig +short <branch>.us-east-1.xata.tech
# should return private IPs from your VPC's subnets (the endpoint ENIs)
```


# Projects
Source: https://xata.io/docs/platform/project

Create and manage projects that contain your database branches

## Overview

A project is part of an organization that has a base branch and child branches. Projects serve as containers for your database environments, allowing you to organize and manage multiple development and production environments under a single project umbrella.

![Project overview interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/project.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=5966fe09b0f11da19e97e7e3530cc008)

## Create Project

Create a new project to start managing your database environments and branches.

![Create project interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/project-create.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=542f2c3a6f723bcf86e4d0120c87c40f)

### From the CLI

Create a project using the Xata CLI:

```bash theme={null}
# Create a new project
xata project create --name "my-app" --organization "org-id"
```

For more CLI commands, see the [CLI Reference](/docs/cli/project).

## Configure Compute

Configure compute resources for your project including scale-to-zero capabilities and manual hibernation options.

For detailed information about compute configuration, scaling, and resource management, see [Compute Resources](/docs/platform/compute).

## Default Database

Set a default database for your project to streamline access and operations. This feature allows you to select which database should be automatically selected within each branch. If the database does not exist in the branch, it will default to `xata`.

![Default database selection interface](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/default-database.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=1370fc7f8afca2f46fe8800cbee65a36)

## GitHub Repository

Link a GitHub repository to a project to create Xata preview branches from pull requests. The Xata GitHub App creates child branches from the root branch you choose and cleans them up when pull requests close.

For setup instructions and behavior details, see the [GitHub App guide](/docs/automations/github-app).

## Delete Project

Delete a project and all its associated resources. **Important**: All branches must be deleted before a project can be deleted.

### From the CLI

Delete a project using the Xata CLI:

```bash theme={null}
# Delete a project (requires confirmation)
xata project delete --project "project-id"
```

For more CLI commands, see the [CLI Reference](/docs/cli/project).


# Query insights
Source: https://xata.io/docs/platform/query-insights

Identify slow and expensive queries on your branch with statistics from pg_stat_statements

Query insights surfaces aggregated execution statistics for the queries running on a branch so you can spot slow statements, hot paths, and resource-heavy workloads at a glance. The page reads from PostgreSQL's [`pg_stat_statements`](https://www.postgresql.org/docs/current/pgstatstatements.html) extension and groups statements by their normalized form, regardless of literal values.

The same statistics are available from a terminal or a script with [`xata branch query-insights`](/docs/cli/branch#query-insights). See [From the command line](#from-the-command-line) below.

## Find slow queries

To find the slowest queries on a branch, open **Query insights** and sort by **Avg** to rank statements by mean execution time, or leave the default sort on total execution time to find the statements consuming the most database time overall. Any statement whose slowest recorded execution exceeded one second is highlighted in the **Max** column.

From there, click a row to see the full statement, how many times it ran, and its cache hit ratio. A low cache hit ratio means the query is reading from disk more than expected, which usually points at a missing index or a working set larger than available memory.

To do the same thing from a terminal, run `xata branch query-insights list --performance slow --sort mean-time`.

## Enable the extension

Query insights requires the `pg_stat_statements` extension, which must be added to the preloaded libraries for your branch before it can be used.

1. Open the **Query insights** page for a branch.
2. If the extension is not yet enabled, an **Enable pg\_stat\_statements** prompt is shown — click it to add `pg_stat_statements` to the preloaded libraries and create the extension. A branch restart is required to load the library.
3. Once the branch finishes restarting, return to the Query insights page to start exploring statistics.

You can also enable the extension manually from the [Extensions](/docs/platform/extensions) settings of your branch.

## Queries table

The main view lists every normalized statement that `pg_stat_statements` has captured since it was last reset, sorted by total execution time. Click any row to open an inline detail panel with the full statement and metadata. Use the arrows in the panel header to move to the previous or next query without closing it.

### Filters

Narrow the table using the controls above the headline:

* **Search** — match against the query text, database, or role.
* **Type** — filter by statement type (`SELECT`, `INSERT`, `UPDATE`, `DELETE`, `CREATE`, `ALTER`, `OTHER`).
* **Database** — restrict to one or more databases on the branch.
* **User** — restrict to one or more roles that ran the queries.

The row counter below the headline shows how many queries match the active filters out of the total captured.

### Columns

A column selector in the toolbar lets you toggle which statistics are visible. The **Query** column is always shown.

| Column | Description |
| - | - |
| Query | Normalized SQL text |
| Database | Database the query ran against |
| User | Role that executed the query |
| Runtime | Share of total execution time across all captured queries |
| Calls | Number of times the query was executed |
| Total | Cumulative execution time |
| Avg | Mean execution time per call |
| Min | Fastest recorded execution |
| Max | Slowest recorded execution (highlighted when over 1s) |
| Stddev | Standard deviation of execution time |
| Rows | Total rows retrieved or affected |
| Shared Hit / Read / Dirtied / Written | Shared buffer activity |
| Local Hit / Read / Dirtied / Written | Local buffer activity (temp tables) |
| Temp Read / Written | Temporary file I/O |

By default, the table shows **Query**, **Runtime**, **Calls**, **Total**, **Avg**, **Max**, and **Rows**.

## Query detail panel

Selecting a row opens a resizable detail panel, shown as a bottom sheet on narrow screens, with the full statement and a summary of its execution profile:

* **Query ID** — the `pg_stat_statements` `queryid` (copy with one click).
* **Full SQL** — the normalized statement, formatted with syntax highlighting.
* **Database** and **Role**.
* **Count** — total calls.
* **Max time** and **Total exec time**.
* **Cache hit ratio** — computed from `shared_blks_hit` and `shared_blks_read`. A low ratio suggests the query is going to disk more than expected.

## Refresh and reset statistics

The toolbar exposes two actions:

* **Refresh** — re-runs the underlying query against `pg_stat_statements` to pull the latest counters.
* **Reset** — calls `pg_stat_statements_reset()` to clear all captured statistics. Statistics begin rebuilding as new queries are executed. Resetting requires superuser privileges; if the action is denied, ask an administrator to run it for you.

> [!WARNING]
>
> Resetting statistics is branch-wide and cannot be undone. All historical query counters are discarded.

## From the command line

[`xata branch query-insights`](/docs/cli/branch#query-insights) (alias `qi`) exposes the same `pg_stat_statements` data outside the dashboard, with `--output json` and `--output ndjson` for scripts and other tools. `list` covers the queries table, `show` covers the detail view, and `enable` and `reset` match the toolbar actions.

The CLI also has one capability the dashboard does not: `xata branch query-insights active` lists the queries executing at this moment, with their age, state, and wait event, read from `pg_stat_activity`. There is no page for this in the dashboard.

Note that `query-insights` opens a direct PostgreSQL connection to the branch, so it requires the `credentials:read` scope on your API key and a healthy branch, meaning one that is not hibernated or still provisioning.

## Availability

Query insights is available on any branch that has `pg_stat_statements` enabled. The page is not available while a branch is hibernated, still provisioning, or has public access disabled. In those cases, the standard branch status view is shown instead. The `active` subcommand in the CLI is the one exception to the extension requirement: it reads `pg_stat_activity`, which is always present.


# Schemas
Source: https://xata.io/docs/platform/schemas

Design, manage, and optimize your database schema structure and relationships

![Database schemas interface showing table structure, relationships, and schema management tools](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/platform/schemas.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=91a214cdc089150e8754af98f049e3de)

Use the schema view for a 10,000 foot view of your database schema, tables, relationships and column details.
Design, manage, and optimize your database schema structure, tables, relationships, and indexes for optimal performance and data integrity.

## Supported Features

* **Table View**: View your schema in a table view for each table with visibility into data types, contstraints, relationships and indices
* **UML View**: Use the UML view for a graphical representation of your schema
* **Schema Diff**: Choose to the diff option to view a diff of your branch schema and its parent branch


# SQL Editor
Source: https://xata.io/docs/platform/sql-editor

Execute and manage database queries across your branches

![SQL Editor with saved queries in the sidebar, open queries in tabs, and query results below the editor](https://mintcdn.com/xata/fDJrx8iLz9Shdmol/images/platform/sql-editor.png?fit=max\&auto=format\&n=fDJrx8iLz9Shdmol\&q=85\&s=3a8fae40ff1a764bb838e98433a1366c)

Xata's built-in SQL Editor provides a comprehensive environment to explore your data. Run SQL statements, generate SQL with natural language, view and export your results, and save your queries for later use.

The SQL Editor page is organized into a resizable sidebar of saved queries and a workspace with a tab for each open query, split into the editor and the results panel. Both splits can be dragged to resize, and the sidebar collapses automatically on smaller screens.

## Run statements

Execute SQL queries directly in the browser with syntax highlighting and auto-completion.

* Autocomplete makes it easy to create queries without intimate schema knowledge.
* Run multiple statements separately or all at once.
* Select from multiple databases in the branch to run queries against.
* Perform exploratory queries as well as DDL statements.
* Highlight a portion of the editor and use **Run selected** to execute only the highlighted text.
* The run button reflects execution state: **ready**, **running**, **completed**, or **error**.

### Keyboard shortcuts

| Action | Shortcut |
| - | - |
| Run query | `⌘` + `Enter` |
| Save query | `⌘` + `S` |
| Open Ask AI | `⌘` + `K` |
| Rename query | `⌘` + `Shift` + `R` |
| New query | `⌘` + `J` |

## Work in tabs

Queries open in tabs, so you can keep several queries open and switch between them without losing your work. Selecting a query in the sidebar opens it in a new tab, or switches to its tab if it is already open.

* Each tab keeps its own edits, undo history, cursor position, and results.
* A running query keeps running when you switch to another tab. A spinner on the tab shows it is still running.
* Drag tabs to reorder them, or move the focused tab with `Alt` + `Shift` + `←` / `→`.
* Close a tab with its close button, a middle-click, or `Delete` when the tab is focused. Right-click a tab to **Close**, **Close others**, **Close to the right**, or **Close all**.
* When there are more tabs than fit in the tab bar, the rest are listed in a dropdown at the end of the bar.
* Open tabs are restored when you reload the page.

Closing a tab with unsaved changes asks for confirmation first, and you are warned before leaving the page while any open query has unsaved changes.

## Generate SQL with Ask AI

Use AI-powered SQL generation to create queries from natural language descriptions. Ask AI opens as an anchored popover in the editor so you can iterate on prompts without leaving your current statement.

![SQL generation interface showing natural language to SQL conversion](https://mintcdn.com/xata/pwZgANRwCfo4oBhA/images/platform/query-generate-sql.png?fit=max\&auto=format\&n=pwZgANRwCfo4oBhA\&q=85\&s=51539c37cf457694285cb30c7cb8e02a)

Open Ask AI from the editor toolbar or with `⌘K`, describe what you want in plain English, and insert the generated SQL into the editor.

## Save and manage queries

Every query can be saved for later use. The saved queries sidebar supports search, rename and delete actions from a context menu, and shows a marker on queries with unsaved changes.

### Draft and save-as

![Save query button](https://mintcdn.com/xata/pwZgANRwCfo4oBhA/images/platform/query-save.png?fit=max\&auto=format\&n=pwZgANRwCfo4oBhA\&q=85\&s=e60ef30815b3cbafd426eff6aaf65114)

When you click **New query** or the **+** button in the tab bar, a new tab opens in a draft state labeled with a **Draft** badge. Open drafts are listed under **Drafts** in the sidebar. Drafts are not persisted until you save them.

* With **Auto-save** off, use **Save as...** to name and create the query. The save button highlights when there are unsaved changes.
* With **Auto-save** on, the draft is saved as a query as soon as you start typing, then persisted automatically as you type. A **Saving** / **Saved** indicator reflects the current state.

Toggle **Auto-save** from the editor toolbar. The preference is remembered across sessions.

### Toolbar actions

The query toolbar includes quick actions for save, rename, delete, and Ask AI, along with a save status indicator. Renaming is also available by clicking the query name.

## View results

![Query results panel](https://mintcdn.com/xata/pwZgANRwCfo4oBhA/images/platform/query-results.png?fit=max\&auto=format\&n=pwZgANRwCfo4oBhA\&q=85\&s=443fdfe1afd34cfa8a1566a18648f15f)

The results panel sits below the editor and supports multiple result types in tabbed views.

* **Multi-statement tabs** — when multiple statements are run, each result is shown in its own tab.
* **Explain plans** — `EXPLAIN` output is auto-detected and rendered in a dedicated monospace view instead of a table.
* **Statement success** — non-SELECT statements (INSERT, UPDATE, CREATE TABLE, etc.) show a success message with the number of affected rows.
* **Status and row counts** — each tab shows the statement status and the number of rows returned.
* **Full-screen mode** — expand the results panel to occupy the full workspace.

### Display options

Customize how rows are rendered from the results toolbar. These preferences are stored in local storage.

* **Striped rows** — alternate row backgrounds for readability (enabled by default).
* **Compact density** — reduce row padding to show more data at once.
* **Row limit** — cap the number of rows rendered client-side. Choose between 100, 500, 1000, or no limit. When a result exceeds the limit, a notice indicates the cap.

### Export results

* **Copy as JSON** — copy the active tab's rows to your clipboard as JSON.
* **Download as CSV** — export the active tab's rows to a CSV file.

## Performance warnings

When a statement may be slow or expensive, a non-blocking, dismissible warning floats at the bottom of the editor. Dismissed warnings are remembered in local storage.

## Editor preferences

The following preferences persist in local storage and apply across sessions on the same browser:

* Auto-save
* Performance warnings
* Striped rows
* Compact density
* Row limit
* Open tabs


# Single sign-on
Source: https://xata.io/docs/platform/sso

Require your organization's members to sign in through your own identity provider

Single sign-on (SSO) lets members of your organization sign in to Xata through your company's identity provider instead of with a
password or social logins. You prove you own an email domain, connect the identity provider for that domain, and can then
require everyone with an address on it to sign in through that provider.

> [!NOTE]
>
> Single sign-on is a paid add-on. Please contact support to enable it for your organization.

## Supported identity providers

Xata connects to identity providers over OpenID Connect (OIDC):

* **Google Workspace**: members sign in with their Workspace account. Google restricts sign-in to your domain.
* **Microsoft Entra ID**: members sign in with their Entra account, restricted to your own tenant.
* **OpenID Connect**: any provider that publishes an OpenID Connect discovery document, such as Okta or Auth0.

SAML is not supported out of the box. Contact us if your identity provider only supports SAML.

## How it works

SSO is configured per email domain. Each domain goes through three steps:

1. **Verify the domain.** Publish a DNS TXT record to prove your organization controls it.
2. **Connect an identity provider.** Register Xata with your provider and enter the credentials it issues.
3. **Require SSO.** Turn on enforcement so every account on the domain signs in through your provider.

Nothing changes for anyone until you complete the third step. You can add more than one domain, and each domain has its own identity provider.

Only organization [Admins](/docs/platform/organization-roles) can view and change single sign-on settings. To open them, select **Organization Settings** in the sidebar, then **Single sign-on**.

## Step 1: Verify your domain

1. Under **Add an email domain**, enter the domain your members' email addresses end in, for example `acme.com`, and select **Add domain**.
2. Xata shows a TXT record for the domain. Add it at your DNS provider exactly as shown:
   * **Record name**: `<label>._xata-sso.<your domain>`, for example `3f9a1c0d7e2b._xata-sso.acme.com`
   * **Record type**: `TXT`
   * **Record value**: `xata-domain-verification=<value>`
3. Select **Check DNS record**. DNS changes can take up to a few hours to spread, so if the record is not found yet, check again later.

Once the record is found, the domain is marked **Verified**.

![Domain awaiting DNS verification](https://mintcdn.com/xata/Pj5PQhsXA8ZN6iw-/images/platform/organization-sso-domain.png?fit=max\&auto=format\&n=Pj5PQhsXA8ZN6iw-\&q=85\&s=4a0bf2f50eed714b9e3b50c90e419711)

> [!WARNING]
>
> Keep the TXT record published for as long as you use single sign-on. Xata checks it daily, and emails your Admins when it first finds the record missing. If the record is still missing at the first daily check after 72 hours, the domain is no longer verified, its identity provider is disabled and SSO is no longer required. Admins get a second email when this happens.

While the domain is unverified, the identity provider stays connected but members cannot sign in through it. They can sign in with the Google and GitHub buttons, or with a password. Members who have only ever signed in through SSO have no password, so they need to select **Forgot Password?** on the sign-in page to set one.

To restore SSO, publish the TXT record shown on the domain card again and select **Check DNS record**. The domain is verified again and its identity provider is enabled, but SSO is not required until you turn **Require SSO for \<domain>** back on.

The record name and value are unique to your organization. A domain can belong to only one Xata organization. If another organization has claimed it, adding it fails, even if that organization's verification has lapsed. Contact support in that case.

Subdomains are not covered by their parent domain, and wildcard domains such as `*.acme.com` are not accepted. If your members use addresses on `eu.acme.com` as well as `acme.com`, add both domains.

## Step 2: Connect your identity provider

The domain card shows a **Redirect URI** as soon as you add the domain, so you can register it with your identity provider while DNS spreads. It has the form:

```
https://auth.xata.io/realms/xata/broker/<provider-id>/endpoint
```

Each domain has its own redirect URI, so copy it from the domain you are setting up.

Once the domain is verified, choose the provider type and enter its details.

> [tab] Google Workspace
>
> 1. In the [Google Cloud console](https://console.cloud.google.com/auth/clients), create an OAuth 2.0 Client ID of type **Web application**.
> 2. Add the redirect URI from Xata under **Authorized redirect URIs**.
> 3. In Xata, select **Google Workspace** and paste the **Client ID** and **Client secret**.

> [tab] Microsoft Entra ID
>
> 1. In the Microsoft Entra admin center, register an application and add the redirect URI from Xata as a **Web** redirect URI.
> 2. Add a client secret under **Certificates & secrets**.
> 3. Under **API permissions**, make sure the application has the delegated Microsoft Graph permissions `openid`, `profile`, `email` and `User.Read`. Xata reads each member's profile from Microsoft Graph.
> 4. In Xata, select **Microsoft Entra ID** and enter:
>    * **Issuer URL**: `https://login.microsoftonline.com/<tenant-id>/v2.0`, using your tenant ID
>    * **Client ID**: the application (client) ID
>    * **Client secret**: the secret value
>
> The shared `common`, `organizations` and `consumers` issuers are refused, because they are not limited to your tenant.
>
> Xata takes each member's email address from the `mail` attribute of their Entra user, or from their user principal name if `mail` is empty. That address must be on the verified domain, or the sign-in is refused. A Microsoft Entra ID Free tenant is enough.

> [tab] OpenID Connect
>
> 1. In your provider, for example Okta or Auth0, register a web application and add the redirect URI from Xata as its sign-in redirect URI.
> 2. In Xata, select **OpenID Connect** and enter:
>    * **Issuer URL**: your provider's issuer, for example `https://acme.okta.com`. It must use `https://`.
>    * **Client ID** and **Client secret** from the application.
>
> Xata reads the endpoints from the issuer's `/.well-known/openid-configuration` document, so you do not enter them by hand.

Select **Connect** followed by the provider name, for example **Connect Google Workspace**, to save. Xata checks the client ID and secret against your provider before saving, and rejects them if the provider does.

Xata requests the `openid`, `profile` and `email` scopes, plus `User.Read` for Microsoft Entra ID. The client secret is never shown again once saved. To change any provider setting later, select **Edit credentials** and enter the secret again.

Before you require SSO, check that a sign-in through the provider works. Once the provider is connected, the domain card shows a **Test sign-in** link. Open it in a private browser window and sign in with an account on the domain. You land on your account page, which lists the provider under **Connected Accounts**. If the sign-in fails, check your provider settings and select **Edit credentials** to fix them.

![Connect an identity provider](https://mintcdn.com/xata/Pj5PQhsXA8ZN6iw-/images/platform/organization-sso-provider.png?fit=max\&auto=format\&n=Pj5PQhsXA8ZN6iw-\&q=85\&s=e803ce754d34d7021076548b41aee6af)

## Step 3: Require SSO

Turn on **Require SSO for \<domain>** on the domain card. The change takes effect immediately, and the domain shows an **SSO required** badge.

If members on the domain cannot sign in afterwards, turn **Require SSO for \<domain>** off again from a window where you are still signed in. Their previous sign-in methods work again straight away.

![Domain with SSO required](https://mintcdn.com/xata/Pj5PQhsXA8ZN6iw-/images/platform/organization-sso-required.png?fit=max\&auto=format\&n=Pj5PQhsXA8ZN6iw-\&q=85\&s=e16821e6f3d9d44351ab6419ffe80b54)

While SSO is required, for every Xata account with an email address on the domain:

* Signing in with a password redirects to your identity provider.
* The **Google** and **GitHub** sign-in buttons redirect to your identity provider.
* Signing up with a password and resetting a password are refused.

This applies to Admins too, and to every account on the domain, whether or not it is already a member of your organization.

> [!WARNING]
>
> If your identity provider stops accepting sign-ins while SSO is required, for example because the client secret was rotated or expired, nobody on the domain can sign in to Xata, including Admins. Keep the credentials in Xata up to date, and contact support if you are locked out.

You can turn enforcement off again at any time. Members can then sign in with the Google and GitHub buttons, or with a password. Members who have only ever signed in through SSO have no password and need to select **Forgot Password?** to set one.

## Signing in with SSO

Once SSO is required, members enter their work email on the Xata sign-in page and are sent to your identity provider. There is no separate SSO button.

The first time someone signs in through your identity provider, they are added to your organization automatically, without an invitation. If they already have a Xata account with the same email address, they are asked to confirm linking it to your identity provider.

Your identity provider can only sign in email addresses on its own domain. Multi-factor authentication and other sign-in policies are enforced by your identity provider.

Xata does not support SCIM provisioning. Removing someone in your identity provider stops them from signing in again, but does not remove them from your organization or end sessions they already have. Remove departed members from your organization in Xata as well.

## Remove SSO

* **Disconnect** removes the identity provider from a domain. SSO is no longer required, and members on the domain sign in with the Google and GitHub buttons or a password, using **Forgot Password?** if they never set one. The domain stays verified, so you can connect a different provider.
* **Remove domain** releases the domain from your organization. It is only available once no identity provider is connected to the domain.

Both actions ask you to type the domain name to confirm.

## Using the CLI

Every step on this page is also available from the CLI under `xata organization sso`:

```bash theme={null}
xata organization sso domains add acme.com
xata organization sso domains verify acme.com
xata organization sso providers add --domain acme.com --type google --client-id <client-id>
# Find the provider alias with: xata organization sso show
xata organization sso providers test <alias>    # prints the Test sign-in link
xata organization sso providers enforce <alias>
```

`providers add` prompts for the client secret, or reads it from `XATA_SSO_CLIENT_SECRET`, so it stays out of your shell history. See the [CLI reference](/docs/cli/organization#sso) for every command and flag.


# Quickstart
Source: https://xata.io/docs/quickstart

Quickstart guide for Xata's PostgreSQL platform

Xata is a PostgreSQL platform that provides instant Copy-on-Write branching, data masking, and separation of storage from compute. It's designed for modern teams running PostgreSQL at scale, with features like zero-downtime schema changes, realistic production clones, and cloud-agnostic deployment options.

## Let your agent do it

The fastest way to get started is to hand the setup to your coding agent. Paste this prompt into Claude Code, Cursor, Codex, or whichever agent you use:

```text theme={null}
Help me get started with Xata by following: https://xata.io/docs/ai-agents/onboarding.md
```

Cursor
Codex

Your agent installs the CLI, signs you in, creates a project and branch, connects your app, and walks you through Xata's branching workflow, explaining each step along the way.

## Or do it yourself

Prefer to click and type yourself? The steps below take about five minutes.

## 1. Sign up

Create your free Xata account at [console.xata.io](https://console.xata.io). You can sign up with GitHub, Google, or your email address.

![Xata sign-up page](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/getting-started/signup.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=78a6aedcfd60b488bb3bf3f59fe29542)

## 2. Create a project

After signing in, create a new project. A project is the top-level container for all your branches and Postgres instances.

![Create project page](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/getting-started/create-project.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=acaf41caa836a535bb5198463c3a6cdb)

## 3. Create a branch

Within your project, create your first branch (e.g., `main`). A branch is a Postgres cluster—a collection of one or more Postgres instances, which can include a primary and optional replicas. When creating your main branch, you can choose the Postgres version, region, and instance size to fit your needs. This branch can serve as your production branch or a staging environment. You can branch off `main` at any time to create isolated development or test environments.

![Create branch page](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/getting-started/create-branch.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=21bebd986e32b3139204ac9d9faa5930)

## 4. Install and configure the Xata CLI

Install the Xata CLI:

```bash theme={null}
curl -fsSL https://xata.io/install.sh | bash
```

Authenticate with your Xata account:

```bash theme={null}
xata auth login
```

Initialize your project by running this command in your project directory:

```bash theme={null}
xata init
```

This will create a `.xata` directory with your project configuration.

## 5. Insert sample data

Let's insert some sample data using `psql`. If you don't have `psql` installed:

```bash macOS theme={null}
## If you are using bash, replace `~/.zshrc` with `~/.bash_profile`
brew install libpq
echo 'export PATH="/opt/homebrew/opt/libpq/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc
```

```bash Ubuntu theme={null}
sudo apt install postgresql-client-17
```

```bash Windows theme={null}
Download from [PostgreSQL official site](https://www.postgresql.org/download/windows/).
```

Connect to `psql` with your dev branch connection string

```sh theme={null}
psql `xata branch url`
```

First, create the required tables:

```sql theme={null}
CREATE TABLE products (
  id SERIAL PRIMARY KEY,
  name TEXT NOT NULL,
  price NUMERIC(7,2) NOT NULL
);

CREATE TABLE orders (
  id SERIAL PRIMARY KEY,
  created_at TIMESTAMP DEFAULT NOW()
);

CREATE TABLE order_items (
  id SERIAL PRIMARY KEY,
  order_id INT REFERENCES orders(id),
  product_id INT REFERENCES products(id),
  qty INT NOT NULL
);
```

Then run the following SQL to insert sample data:

```sql theme={null}
INSERT INTO products(name,price) SELECT LEFT(md5(i::text),8),(random()*90+10)::numeric(7,2) FROM generate_series(1,10)i;
WITH o AS (INSERT INTO orders DEFAULT VALUES RETURNING id) INSERT INTO order_items(order_id,product_id,qty) SELECT o.id,pid,(1+floor(random()*3))::int FROM o,(SELECT id pid FROM products ORDER BY random() LIMIT 5)p;
```

## 6. Create a development branch with the CLI

Create a new branch from `main`:

```bash theme={null}
xata branch create --name dev --parent-branch `xata branch get id`
```

This command will automatically checkout the new branch for you, with the data from `main`.

## 7. Next steps: choose your workflow

Now that you have a base branch and development branch to try out, explore what the rest of the platform has to offer.

* **[Set up a production clone](/docs/tutorials/create-production-clone):** Learn how to use Xata for production clones, feature branches, and collaborative development.
* **[Schema changes](/docs/tutorials/schema-change):** Learn how to safely apply and roll back schema changes with zero downtime.
* **[Migrate to Xata](/docs/migrations/overview):** Get set up for production by migrating your existing PostgreSQL database to Xata.

***

**Need help?** Reach out to the Xata team at [info@xata.io](mailto:info@xata.io).


# Use Xata with Drizzle
Source: https://xata.io/docs/quickstarts/drizzle

Learn how to connect your Drizzle ORM project to a Xata PostgreSQL database.

[Drizzle ORM](https://orm.drizzle.team/) is a TypeScript-first ORM that provides type-safe database access with a SQL-like query builder. You can use Drizzle with Xata using either the standard PostgreSQL driver or the serverless driver for edge environments.

## Prerequisites

* Xata account with a project and branch
* Node.js project with TypeScript

## Installation

Install Drizzle ORM and the PostgreSQL driver:

> [tab] Standard (TCP)
>
> ```bash theme={null}
> npm install drizzle-orm postgres
> npm install -D drizzle-kit
> ```

> [tab] Serverless (HTTP/WebSocket)
>
> ```bash theme={null}
> npm install drizzle-orm @neondatabase/serverless
> npm install -D drizzle-kit
> ```

## Get your connection string

```bash theme={null}
xata branch url
```

Set it as an environment variable:

```bash theme={null}
DATABASE_URL=postgresql://user:password@host:port/database
```

## Connect to Xata

> [tab] Standard (TCP)
>
> Use this for traditional servers and long-running applications:
>
> ```typescript theme={null}
> import { drizzle } from 'drizzle-orm/postgres-js';
> import postgres from 'postgres';
>
> const client = postgres(process.env.DATABASE_URL!);
> export const db = drizzle(client);
> ```

> [tab] Serverless (HTTP)
>
> Use this for serverless functions with single queries:
>
> ```typescript theme={null}
> import { drizzle } from 'drizzle-orm/neon-http';
> import { neon } from '@neondatabase/serverless';
>
> const client = neon(process.env.DATABASE_URL!);
> export const db = drizzle(client);
> ```

> [tab] Serverless (WebSocket)
>
> Use this for serverless functions that need transactions:
>
> ```typescript theme={null}
> import { drizzle } from 'drizzle-orm/neon-serverless';
> import { Pool } from '@neondatabase/serverless';
>
> const pool = new Pool({ connectionString: process.env.DATABASE_URL! });
> export const db = drizzle(pool);
> ```

## Define your schema

Create a `schema.ts` file:

```typescript theme={null}
import { pgTable, serial, text, timestamp } from 'drizzle-orm/pg-core';

export const users = pgTable('users', {
  id: serial('id').primaryKey(),
  name: text('name').notNull(),
  email: text('email').notNull().unique(),
  createdAt: timestamp('created_at').defaultNow(),
});
```

## Query your database

```typescript theme={null}
import { db } from './db';
import { users } from './schema';
import { eq } from 'drizzle-orm';

// Insert a user
await db.insert(users).values({
  name: 'Alice',
  email: 'alice@example.com',
});

// Select all users
const allUsers = await db.select().from(users);

// Select with filter
const user = await db
  .select()
  .from(users)
  .where(eq(users.email, 'alice@example.com'));
```

## Configure Drizzle Kit

Create a `drizzle.config.ts` file for migrations:

```typescript theme={null}
import { defineConfig } from 'drizzle-kit';

export default defineConfig({
  schema: './schema.ts',
  out: './drizzle',
  dialect: 'postgresql',
  dbCredentials: {
    url: process.env.DATABASE_URL!,
  },
});
```

## Run migrations

Generate and apply migrations:

```bash theme={null}
# Generate migration files
npx drizzle-kit generate

# Apply migrations
npx drizzle-kit migrate
```

> [!TIP]
>
> For serverless environments, consider using the [Serverless Proxy](/docs/core-concepts/serverless-proxy) to connect over HTTP or WebSocket instead of TCP.


# Use Xata with n8n
Source: https://xata.io/docs/quickstarts/n8n

Learn how to use Xata as the memory provider for AI agents in n8n.

## What is n8n

[n8n](https://n8n.io) is a workflow automation platform that lets you connect apps, APIs, and AI models to build automated workflows. It's commonly used to build AI agents, where a large language model is given tools and memory to carry out multi-step tasks.

This page shows you how to use Xata with n8n. Since Xata is fully PostgreSQL-compatible, you can use it anywhere n8n supports PostgreSQL.

## Use Xata as memory for AI agents

The most common way to use Xata with n8n is as the memory store for an AI agent, so that the agent can persist and recall chat history across runs.

> [!NOTE]
>
> When adding a memory provider to your agent, n8n shows a dedicated **Xata** option. This option is out of date, so don't use it. Instead, use the **Postgres Chat Memory** option and point it at your Xata branch, as described below.

### Prerequisites

* A [Xata account](https://console.xata.io) with a project and branch
* An n8n instance (cloud or self-hosted)

### Steps

1. In your n8n AI agent, add a memory provider and choose **Postgres Chat Memory** (not the Xata option).

![n8n memory provider list showing Postgres Chat Memory and the out-of-date Xata option](https://mintcdn.com/xata/Nj_y4Z5mGeTrrakS/images/n8n-memory-providers.png?fit=max\&auto=format\&n=Nj_y4Z5mGeTrrakS\&q=85\&s=4773ad32d392d9836675423b554f2e6e)

2. In the [Xata console](https://console.xata.io), open the **Overview** tab of your branch and copy the connection string shown at the top of the page.

![Xata project overview showing the PostgreSQL connection string](https://mintcdn.com/xata/Nj_y4Z5mGeTrrakS/images/xata-conn-string.png?fit=max\&auto=format\&n=Nj_y4Z5mGeTrrakS\&q=85\&s=7ae07fc0c07feb0c5185bb3636bab090)

A Xata connection string looks like this:

```
postgresql://xata:<password>@<host>.us-east-1.xata.tech:5432/xata?sslmode=require
```

> [!NOTE]
>
> In the Xata UI, you can copy each field (host, poassword, port, etc.) by simply clicking it.

3. Create a new PostgreSQL credential in n8n and copy the values from the connection string field by field:

   | n8n field | Value from connection string |
   | - | - |
   | Host | The host, e.g. `<host>.us-east-1.xata.tech` |
   | Database | `xata` |
   | User | `xata` |
   | Password | The password from the connection string |
   | Port | `5432` |

![n8n PostgreSQL credential form filled in with Xata connection details](https://mintcdn.com/xata/Nj_y4Z5mGeTrrakS/images/n8n-postgresql-settings.png?fit=max\&auto=format\&n=Nj_y4Z5mGeTrrakS\&q=85\&s=ffe839b7611ba5741ef1d074873f88af)

4. Leave **Ignore SSL Issues (Insecure)** off and set **SSL** to **Allow**.

![n8n PostgreSQL SSL settings with SSL set to Allow and port 5432](https://mintcdn.com/xata/Nj_y4Z5mGeTrrakS/images/n8n-postgresql-settings-ssl.png?fit=max\&auto=format\&n=Nj_y4Z5mGeTrrakS\&q=85\&s=e9b9bc597b1560ef4a752b3c49717bc2)

5. Save the credential and test the connection. It should connect successfully, and your AI agent will now store its chat memory in Xata.

## Next steps

* **Enable scale-to-zero.** In your branch settings, enable scale-to-zero (a suggested inactivity time is 1 hour) so the branch pauses when it's not in use. This way you only incur compute costs while the branch is active. See [Scale to zero](/docs/core-concepts/scale-to-zero) for details.
* **Move beyond the free trial.** To keep using Xata after the free trial, add a credit card in the billing system. Xata charges only for the resources you use. There is no minimum fee.


# Use Xata with Netlify
Source: https://xata.io/docs/quickstarts/netlify

Connect a Netlify site to Xata PostgreSQL, and give every deploy preview its own database branch.

## What is Netlify

[Netlify](https://www.netlify.com) is a platform for building and deploying web applications, with Git-driven builds, serverless functions, and a Deploy Preview for every pull request.

Xata gives a Netlify site a managed PostgreSQL database, and gives each Deploy Preview its own branch of that database with production-like data. Branches are copy-on-write, so a preview branch takes about 3 seconds to create and stores only its diff from the parent. Idle branches scale to zero, so previews that nobody is looking at cost nothing to run.

## Prerequisites

* A Xata account and a project with at least one branch
* A Netlify site
* The [Xata CLI](/docs/cli) installed, if you want to use the CLI commands below or per-preview branches

The CLI commands in this guide read the organization, project, and branch from the folder
they run in. Run [`xata init`](/docs/cli/init) once in your project folder to link it:

```bash theme={null}
xata init
```

Without this, commands such as `xata branch url` fail with
`Missing required path parameter: organizationID`.

## Connect a Netlify site to Xata

1. In the [Xata console](https://console.xata.io), open the branch you want the site to use.
2. On the **Overview** tab, copy the pooled connection string. You can also get it from the CLI:

```bash theme={null}
xata branch url main --type pooler
```

> [!TIP]
>
> Use the `pooler` connection type for Netlify Functions. Serverless invocations create many short-lived connections, and the pooler keeps the number of PostgreSQL connections low under concurrency. See [Serverless Proxy](/docs/core-concepts/serverless-proxy) for the full list of connection types.

3. In Netlify, go to **Project configuration > Environment variables** and add a variable named `DATABASE_URL` with the connection string as its value. Leave the scopes at their default so the variable is available to builds and functions on every Netlify plan.
4. Install the driver in your project:

```bash theme={null}
npm install @neondatabase/serverless
```

> [!NOTE]
>
> Xata implements the [Neon serverless driver protocol](https://github.com/neondatabase/serverless), so the `@neondatabase/serverless` package connects to Xata over HTTP and WebSocket without changes. If you are moving a Netlify site off Neon, your data access code stays as it is.

5. On the branch's **SQL Editor** page in Xata, create the table used by this example:

```sql theme={null}
CREATE TABLE users (
  id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  name TEXT NOT NULL
);

INSERT INTO users (name) VALUES ('Ada Lovelace');
```

6. Query the database from a Netlify Function:

```typescript theme={null}
import { neon } from '@neondatabase/serverless';
import type { Context } from '@netlify/functions';

export default async (request: Request, context: Context) => {
  const sql = neon(process.env.DATABASE_URL!);
  const users = await sql`SELECT * FROM users LIMIT 10`;
  return Response.json(users);
};

export const config = { path: '/api/users' };
```

7. Deploy the site. The function now reads from your Xata branch.

Standard TCP connections work too, so ORMs and drivers such as Drizzle, Prisma, and `pg` connect with the same connection string. For long-running processes rather than functions, use `xata branch url main` without the `--type pooler` flag.

## Give every Deploy Preview its own database

Netlify builds a Deploy Preview for each pull request. To pair each preview with its own database, create a Xata branch when the pull request opens and set a branch-scoped `DATABASE_URL` on the Netlify site.

This uses a GitHub Actions workflow rather than a Netlify build plugin, because the connection string has to exist before the build starts and has to be readable by the deployed functions at runtime.

### Create the branch when a pull request opens

```yaml theme={null}
name: Create Xata preview branch

on:
  pull_request:
    types: [opened, reopened]

env:
  XATA_API_KEY: ${{ secrets.XATA_API_KEY }}
  XATA_ORGANIZATION_ID: ${{ secrets.XATA_ORGANIZATION_ID }}
  XATA_PROJECT_ID: ${{ secrets.XATA_PROJECT_ID }}
  NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
  NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}

jobs:
  create-branch:
    runs-on: ubuntu-latest
    steps:
      - name: Install the Xata CLI
        run: |
          curl -fsSL https://xata.io/install.sh | bash
          echo "$HOME/.config/xata/bin" >> $GITHUB_PATH

      - name: Create a preview branch
        run: |
          xata branch create \
            --organization "$XATA_ORGANIZATION_ID" \
            --project "$XATA_PROJECT_ID" \
            --name "pr-${{ github.event.number }}" \
            --parent-branch "${{ secrets.XATA_PREVIEW_PARENT_BRANCH_ID }}" \
            --scale-to-zero true

      - name: Wait for the branch to be ready
        run: |
          xata branch wait-ready "pr-${{ github.event.number }}" \
            --organization "$XATA_ORGANIZATION_ID" \
            --project "$XATA_PROJECT_ID"

      - name: Point the Deploy Preview at the branch
        run: |
          DATABASE_URL=$(xata branch url "pr-${{ github.event.number }}" --type pooler \
            --organization "$XATA_ORGANIZATION_ID" \
            --project "$XATA_PROJECT_ID")
          if [ -z "$DATABASE_URL" ]; then
            echo "No connection string returned for pr-${{ github.event.number }}" >&2
            exit 1
          fi
          npx netlify-cli env:set DATABASE_URL "$DATABASE_URL" \
            --context branch:${{ github.head_ref }}
```

Netlify applies a branch-scoped value to that branch's Deploy Previews, branch deploys, and deploy permalinks, so the preview build and its functions both pick up the branch connection string. Production keeps whatever `DATABASE_URL` you set on the site.

> [!WARNING]
>
> The CLI reads `XATA_API_KEY` from the environment, but not the organization or project.
> Outside a folder linked with `xata init`, pass `--organization` and `--project` on every
> command, as the workflow above does. Without them each command fails with
> `Missing required path parameter: projectID`.

### Clean up when the pull request closes

```yaml theme={null}
name: Delete Xata preview branch

on:
  pull_request:
    types: [closed]

env:
  XATA_API_KEY: ${{ secrets.XATA_API_KEY }}
  XATA_ORGANIZATION_ID: ${{ secrets.XATA_ORGANIZATION_ID }}
  XATA_PROJECT_ID: ${{ secrets.XATA_PROJECT_ID }}
  NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
  NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}

jobs:
  delete-branch:
    runs-on: ubuntu-latest
    steps:
      - name: Install the Xata CLI
        run: |
          curl -fsSL https://xata.io/install.sh | bash
          echo "$HOME/.config/xata/bin" >> $GITHUB_PATH

      - name: Delete the branch
        run: |
          xata branch delete "pr-${{ github.event.number }}" --yes \
            --organization "$XATA_ORGANIZATION_ID" \
            --project "$XATA_PROJECT_ID"

      - name: Remove the Netlify variable
        run: |
          npx netlify-cli env:unset DATABASE_URL \
            --context branch:${{ github.head_ref }}
```

> [!NOTE]
>
> `XATA_API_KEY` should be an organization API key. Create one with `xata keys organization create`. The other IDs come from `xata organization get id`, `xata project get id`, and `xata branch get id`. Set `XATA_PREVIEW_PARENT_BRANCH_ID` to the ID of the branch that previews should copy. Run these commands from a folder you have linked with `xata init`, on CLI 1.7.0 or later.

For the full set of Xata GitHub Actions, including schema migration checks on pull requests, see [Automations](/docs/automations/ga-pr).

## Anonymize preview data

Deploy Previews on a public repository can be triggered by anyone who opens a pull request, so a preview branch forked from production may expose real customer data.

Branch creation copies its parent's data without anonymizing it. Use [`xata clone`](/docs/core-concepts/anonymization) to create an anonymized staging branch, then set `XATA_PREVIEW_PARENT_BRANCH_ID` to that branch's ID. Each preview branch will inherit the sanitized data. See Netlify's [sensitive variable policy](https://docs.netlify.com/build/environment-variables/get-started/#sensitive-variable-policy) for controlling which deploys can read your variables.

## Next steps

* [Branching](/docs/core-concepts/branching): how copy-on-write branches work
* [Scale to Zero](/docs/core-concepts/scale-to-zero): how idle branches hibernate and wake
* [Serverless Proxy](/docs/core-concepts/serverless-proxy): HTTP and WebSocket connections for functions and edge runtimes


# Use Xata with Railway
Source: https://xata.io/docs/quickstarts/railway

Learn how to connect your Railway project with a Xata PostgreSQL database.

## What is Railway

[Railway](https://railway.com?utm_medium=integration\&utm_source=docs\&utm_campaign=xata) is Platform as a Service (PaaS) that supports almost any application stack or open-source framework. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

While Railway offers built-in PostgreSQL templates, this page shows you how to use Xata as a full-managed PostgreSQL service for Railway.

## Prerequisites

* Xata account
* Railway account

## Getting started

1. Create a project and branch in the [Xata console](https://console.xata.io).
2. In the **Overview** tab of the branch, copy the connection string.

![Branch overview showing connection string and database details](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/guides/railway-branch-overview.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=c871b185aca450dd92f0f81713563924)

3. In the Railway Marketplace you can find the following Xata specific templates:

* [Xata PostgreSQL for Node.js and Drizzle](https://railway.com/deploy/xata-postgresql-nodejs-drizzle?utm_medium=integration\&utm_source=docs\&utm_campaign=xata)
* [Xata PostgreSQL for Django](https://railway.com/deploy/xata-postgresql-django?utm_medium=integration\&utm_source=docs\&utm_campaign=xata)

4. Click **Deploy Now** in the template page.
5. In the creation modal, use the connection string from step 2 as the `DATABASE_URL` setting.

![Railway template installation modal showing database URL configuration](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/guides/railway-template-install.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=60e926d3297218142dd497e5f8b67bb4)

6. Click **Deploy**. Once the deployment is done, visit the resulting page at the assign domain. On success, it should show the PostgreSQL version.

![Railway template success page showing PostgreSQL version and deployment confirmation](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/guides/railway-template-success.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=4178f3de20371646120d873104ae1788)

> [!TIP]
>
> The above screenshot is from the Node.js template. The other templates have a different success screen.

## Add Xata PostgreSQL to your existing Railway project

You can also add Xata to your existing project. To do this, you would typically replace the PostgreSQL service from the Railway canvas. To do this:

1. Create a `DATABASE_URL` variable and set it to the PostgreSQL connection string, which you can copy from the Xata branch **Overview** page.
2. Use `DATABASE_URL` as your PostgreSQL connection string in your application. See the various framework and programming language guides on how to do this.
3. If you had PostgreSQL on your Railway canvas, you can remove it now.


# Create a production clone
Source: https://xata.io/docs/tutorials/create-production-clone

Use Xata's clone and branching features to create a safe, anonymized production clone.

This guide shows you how to create a realistic production clone in Xata by cloning your production database, optionally anonymizing sensitive data.

![Creating a production clone of your database](https://mintcdn.com/xata/zunLvnbMMVhTIsZq/images/deployment-models/xata-deployment-staging.png?fit=max\&auto=format\&n=zunLvnbMMVhTIsZq\&q=85\&s=19ef06a16fb3d83a6d6bff838395b5b1)

## 1. Prerequisites

* A Xata account ([sign up here](https://console.xata.io))
* The [Xata CLI](/docs/cli) installed:
  ```bash theme={null}
  curl -fsSL https://xata.io/install.sh | bash
  ```
* Access to your production PostgreSQL database

## 2. Create a Xata project and branch

In the Console, create a new project and then click the **Create main branch** button to create the PostgreSQL instance that will hold the production clone.

![Create new branch](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/create-new-branch.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=cbb47662022af582812118fea4057b47)

Since this is a non-production setup, you can typically select 0 replicas. It is a good idea to start with a similar
instance size as you use for production, but you can also use a smaller instance size if the production clone traffic will be much smaller. You can adjust the size later.

> [!TIP]
>
> If you're using Xata for production, we recommend creating a separate organization for your production clone. This allows you to control access to production data more effectively by only granting production access to team members who need it. Within an organization, [roles](/docs/platform/organization-roles) control who can manage members, billing and API keys, but every member can access all of its projects and branches.

## 3. Configure the Xata CLI

Authenticate the CLI by running:

```sh theme={null}
xata auth login
```

Initialize the project by running:

```sh theme={null}
xata init
```

You would typically run `xata init` command in your application source directory, or in any directory/repository where you'd like to keep the configuration. It will create a `.xata` folder with several configuration files.

Follow the instructions from the prompt to select the project and branch that you created at the previous step. Leave the database name set to `xata`.

## 4. Get the connection string of your current provider

This step depends on your Postgres provider, but keep the following recommendations in mind:

* If your provider offers a read-only endpoint, it is recommended to use that one in order to avoid extra load to your primary instance.
* If your provider offers a "pooler" or "pgbouncer" endpoint, avoid that one, because `xata clone` uses a long running transaction.
* The Postgres connection string generally has the form `postgres://user:pass@host:5432/dbname`.

## 5. Configure cloning

First, generate a configuration for the Xata clone process:

```bash theme={null}
xata clone config --source-url $CONN_STRING --mode=prompt
```

Where the `CONN_STRING` variable is set to the production connection string from the previous step.

> [!TIP]
>
> if your Postgres hosting provider uses self-signed certificates, you can disable the strict certificate verification like this:

```bash theme={null}
NODE_TLS_REJECT_UNAUTHORIZED=0 xata clone config --source-url $CONN_STRING --mode=prompt
```

Follow the prompts to specify whether you want anonymization enabled, which tables and columns to anonymize, and which transformers to use. For more details about transformers, see the [anonymization](/docs/core-concepts/anonymization) page.

> [!TIP]
>
> To simplify the process, the interactive prompt suggests which columns require anonymization. This is based on simple heuristics (name, email, etc.)

The result is a YAML configuration file written in `.xata/clone.yaml`. You can edit this file to further configure your anonymization process.

### 6. Start the clone

Now, run the clone command to snapshot your production database into Xata:

```bash theme={null}
xata clone start --source-url $CONN_STRING
```

* The clone process will copy data and apply your anonymization rules to the current production clone branch.

***

## 7. Create a branch from the production clone

Now, list your branches to get the production clone branch ID:

```bash theme={null}
xata branch list
```

Grab the production clone branch ID from the list and create a new `dev` branch from it:

```bash theme={null}
xata branch create --name dev --parent-branch <production-clone-branch-id>
```

Wait for the branch to become ready:

```bash theme={null}
xata branch wait-ready
```

Connect to the newly created dev branch:

```sh theme={null}
psql `xata branch url`
```

***

## Summary

* You now have a realistic, anonymized production clone in Xata.
* You can safely test schema migrations, new indexes, or any other database changes using instant Copy-on-Write branches.

For more details on how to set up an advanced schema changes workflow, see the [schema changes](/docs/core-concepts/schema-changes) section.

To automate this process and keep your production clone up-to-date, see the [scheduled clone](/docs/automations/kubernetes-clone) documentation.


# Find and fix slow queries on a Xata branch
Source: https://xata.io/docs/tutorials/find-slow-queries

Use query insights, branch logs, and instance metrics to find slow PostgreSQL queries, understand why they are slow, and confirm the fix worked.

When a database gets slow, you usually need three things: which statements are expensive, what the instance is doing while they run, and what the server logged. On Xata all three are available for any branch, in the console and from the CLI, so you can work through them in one place.

This guide walks the loop end to end: find the expensive statements, read one query's profile, see what is running right now, correlate against logs and instance metrics, then confirm the fix.

## 1. Prerequisites

* A Xata branch with a workload on it. A [production clone](/docs/tutorials/create-production-clone) is a good place to do this without touching production.
* The [Xata CLI](/docs/cli) installed, if you want to follow the terminal steps:
  ```bash theme={null}
  curl -fsSL https://xata.io/install.sh | bash
  ```
* An API key with the `credentials:read` and `logs:read` scopes. `credentials:read` is what lets the CLI open a database connection for query insights. See [Required scopes](/docs/cli#required-scopes).

## 2. Enable pg\_stat\_statements

Query statistics come from PostgreSQL's [`pg_stat_statements`](https://www.postgresql.org/docs/current/pgstatstatements.html) extension, which has to be in the branch's preloaded libraries before it can collect anything.

From the console, open **Query insights** on the branch and click **Enable pg\_stat\_statements**. From the CLI:

```bash theme={null}
xata branch query-insights enable my-branch
```

Either route restarts the branch, because loading a shared library requires it. Wait for the branch to come back:

```bash theme={null}
xata branch wait-ready my-branch --wake
```

> [!NOTE]
>
> The extension only records statements executed after it was loaded. If you enable it and immediately look, the table will be close to empty. Let real traffic run for a while before drawing conclusions.

## 3. Find the expensive statements

Two different questions get confused here, and they have different answers.

**Which statement consumes the most database time overall** is the default sort, on total execution time. This is the one to start with, because a query taking 20ms that runs a million times costs more than a 3-second report that runs nightly.

**Which statement is slowest per call** is a sort on mean execution time. This is what you want when a specific request feels slow.

From the console, open **Query insights** and sort by **Total** or **Avg**. Anything whose slowest recorded run exceeded one second is highlighted in the **Max** column.

From the CLI:

```bash theme={null}
# Biggest consumers of total database time
xata branch query-insights list my-branch

# Slowest per call, narrowed to the statements Xata classifies as slow
xata branch query-insights list my-branch --performance slow --sort mean-time

# Narrow to one database and only writes
xata branch qi list my-branch --db app --type UPDATE,DELETE
```

`qi` is an alias for `query-insights`, and `list` is the default subcommand, so `xata branch qi my-branch` does the same thing as the first command above.

In table output the CLI appends a **Potential issues** block calling out statements with a high mean execution time or a low cache hit rate, each with the exact command to inspect it next.

## 4. Read one query's profile

Take a query ID from the previous step and look at it in detail. In the console, click the row. From the CLI:

```bash theme={null}
xata branch qi show -6744440887696913970 --db app --role app my-branch
```

The `--db` and `--role` flags matter more than they look. A `pg_stat_statements` query ID is only unique within a database and role pair, so the same normalized statement running as two different roles is two separate rows. If the ID you pass matches more than one, the command warns you and exits non-zero rather than guessing.

What to look at, in rough order of usefulness:

* **Mean versus max execution time.** A large gap means the query is usually fine and occasionally terrible, which points at lock contention, a cold cache, or a plan that flips on certain parameter values. A mean close to the max means it is uniformly slow, which is usually the plan itself.
* **Cache hit ratio**, computed from shared buffer hits against reads. A low ratio means the query is going to disk more than it should. That is either a missing index or a working set bigger than available memory, and step 6 tells you which.
* **Calls and rows.** A query returning far more rows than the application needs is often a missing `LIMIT` or an over-broad join, and it is cheaper to fix in the application than in the database.

## 5. See what is running right now

Aggregated statistics tell you what has been expensive. They do not tell you what is happening at this moment, which is what you want when the database is slow *now*.

```bash theme={null}
xata branch qi active my-branch
```

This reads `pg_stat_activity` and lists each running query with its process ID, age, state, and wait event. The wait event is the useful column: a query waiting on a lock is a different problem from one burning CPU, and they have different fixes.

To watch it, refreshing every five seconds:

```bash theme={null}
xata branch qi active my-branch --watch 5
```

> [!NOTE]
>
> `active` is the one subcommand that does not need `pg_stat_statements`, since `pg_stat_activity` is always present. It is also only available from the CLI; there is no page for it in the console.

## 6. Correlate with logs and instance metrics

A slow query is sometimes the cause and sometimes the symptom. Before optimizing a statement, check whether the instance was simply saturated while it ran.

```bash theme={null}
# CPU, memory, disk, connections, IOPS, replication lag
xata branch metrics my-branch

# Live view, refreshed every 5 seconds
xata branch metrics my-branch -w --refresh 5s
```

If CPU is pinned or the connection count is at its ceiling, the slow query may be a victim rather than the culprit. If disk read IOPS are high while your query shows a low cache hit ratio, those two facts are the same fact, and an index is the likely fix.

Then read what the server said:

```bash theme={null}
# Errors in the last 15 minutes
xata branch logs my-branch --level error --start 15m

# Lock and timeout trouble specifically
xata branch logs my-branch --output raw | rg 'timeout|deadlock|lock'
```

`--search` does case-insensitive substring matching only. For regex, pipe `--output raw` into `rg` or `grep`, as above.

## 7. Confirm the fix

After you ship an index or a query change, you need a clean baseline, because the accumulated averages include every slow execution from before the fix.

```bash theme={null}
xata branch query-insights reset my-branch
```

This calls `pg_stat_statements_reset()`, discards all counters for the branch, and cannot be undone. Let traffic run again, then re-check the same sort from step 3 and compare mean execution time and cache hit ratio against what you wrote down earlier.

> [!WARNING]
>
> Reset is branch-wide. Everyone looking at query insights on that branch loses their history too, so on a shared branch it is worth telling people first.

## Running this from a script or an agent

Every command above produces machine-readable output, so the same loop works unattended. `qi list`, `logs`, and `metrics` take `-o`/`--output` with a format: `json`, `ndjson`, and in some cases `csv` or `tui`. The rest of the subcommands used here, `qi show`, `qi active`, `qi enable`, `qi reset`, and `wait-ready`, take a plain `--json` flag instead, and passing `--output` to them is an error.

```bash theme={null}
# The slow statements, as query IDs
xata branch qi list my-branch --performance slow --output json | jq -r '.queries[].queryid'

# Stream errors as they arrive, one JSON object per entry
xata branch logs my-branch -f --output ndjson | jq -r 'select(.level=="error") | .message'

# A metrics sample every 5 seconds, one report per line
xata branch metrics my-branch -w --refresh 5s --output ndjson

# What is running right now, with wait events. Note --json, not --output json
xata branch qi active my-branch --json | jq -r '.queries[] | "\(.pid) \(.wait_event_type // "-")/\(.wait_event // "-") \(.query)"'
```

`ndjson` means something slightly different per command, which is worth knowing before you write a parser. For `logs` it is one object per log entry, so it streams. For `metrics -w` it is one report per refresh, so it also streams. For `qi list` it is the whole result set on a single line, identical in content to `--output json`, with the statements under a `queries` key.

Two behaviors worth knowing if a coding agent is the one running these. The CLI detects when it is running in CI or inside an agent and skips interactive prompts instead of blocking on input. And `xata branch metrics -w` adapts to its caller: an interactive terminal gets a live dashboard, while a pipe gets an NDJSON stream, with no flag change.

## Next steps

* [Query insights](/docs/platform/query-insights) for the console reference
* [Logs](/docs/platform/logs) and [Metrics](/docs/platform/metrics) for what each surface collects
* [`xata branch` CLI reference](/docs/cli/branch#query-insights) for the full flag surface
* [Create a production clone](/docs/tutorials/create-production-clone) to do this work against realistic data without touching production


# Make Schema Changes
Source: https://xata.io/docs/tutorials/schema-change

Learn how to safely apply and roll back schema changes in Xata using the CLI.

You'll use the `xata roll` commands to manage migrations, and an example with changing column types.

To better understand the use case and our novel approach to the challenge of schema changes, see the [schema changes core concept](/docs/core-concepts/schema-changes) page for more details.

`xata roll` is embedding [pgroll](https://pgroll.com) behind the scenes and additional details around schema changes can be found in the [pgroll documentation](https://pgroll.com/docs/latest).

## 1. Prerequisites

* A Xata account ([sign up here](https://console.xata.io))
* The [Xata CLI](/docs/cli) installed:
  ```bash theme={null}
  curl -fsSL https://xata.io/install.sh | bash
  ```
* Access to your Xata PostgreSQL branch

## 2. Create a Xata project and branch

In the Console, create a new project and then click the **Create main branch** button to create the PostgreSQL instance.

![Create new branch](https://mintcdn.com/xata/PDDxPY9xptrEGBCP/images/create-new-branch.png?fit=max\&auto=format\&n=PDDxPY9xptrEGBCP\&q=85\&s=cbb47662022af582812118fea4057b47)

Since this is a production setup, we recommend using at least 1 replica. It's a good idea to start with a similar instance size as you are running in your current PostgreSQL provider. You will be able to adjust it later.

## 3. Configure the Xata CLI

Authenticate the CLI by running:

```sh theme={null}
xata auth login
```

Initialize the project by running:

```sh theme={null}
xata init
```

You would typically run `xata init` command in your application source directory, or in any directory/repository where you'd like to keep the configuration. It will create a `.xata` folder with several configuration files.

Follow the instructions from the prompt to select the project and branch that you created at the previous step.

## 4. Prepare a migration directory

Initialize migration tracking on your current Xata branch:

```sh theme={null}
xata roll init
```

This command sets up migration history for your branch.

## 5. Create initial tables

First, let's create a migration file for our initial schema:

```sh theme={null}
cat > .xata/migrations/001_initial_schema.yaml << 'EOL'
operations:
  - create_table:
      name: products
      columns:
        - name: id
          type: serial
        - name: name
          type: text
        - name: price
          type: numeric(7,2)
      constraints:
        - name: products_pk
          type: primary_key
          columns:
            - id
  - create_table:
      name: orders
      columns:
        - name: id
          type: serial
        - name: created_at
          type: timestamptz
          default: now()
      constraints:
        - name: orders_pk
          type: primary_key
          columns:
            - id
  - create_table:
      name: order_items
      columns:
        - name: order_id
          type: int
        - name: product_id
          type: int
        - name: qty
          type: int
      constraints:
        - name: order_items_pk
          type: primary_key
          columns:
            - order_id
            - product_id
        - name: order_items_order_id_fk
          type: foreign_key
          columns:
            - order_id
          references:
            table: orders
            columns:
              - id
        - name: order_items_product_id_fk
          type: foreign_key
          columns:
            - product_id
          references:
            table: products
            columns:
              - id
EOL
```

Apply the migration:

```sh theme={null}
xata roll migrate --complete true
```

## 6. Insert sample data

Now let's insert some sample data using psql:

```sh theme={null}
psql `xata branch url`
```

Then run the following SQL to insert sample data:

```sql theme={null}
INSERT INTO products(name,price) SELECT LEFT(md5(i::text),8),(random()*90+10)::numeric(7,2) FROM generate_series(1,10)i;
WITH o AS (INSERT INTO orders DEFAULT VALUES RETURNING id) INSERT INTO order_items(order_id,product_id,qty) SELECT o.id,pid,(1+floor(random()*3))::int FROM o,(SELECT id pid FROM products ORDER BY random() LIMIT 5)p;
```

## 7. View and pull changes

You can view the changes in your branch details page in the [Xata Console](https://console.xata.io). The changes will be tracked automatically.

To pull the latest migrations locally, run:

```sh theme={null}
xata roll pull
```

This will write the complete schema history as .yaml files to the `.xata/migrations` directory.

## 8. Create a new schema change

Let's create a migration that:

1. Changes the `qty` column in the `order_items` table from integer to float
2. In the `up` function (current schema → new schema):
   * Converts any quantity less than 1 to 1.0
   * Converts all other values from integer to float
3. In the `down` function (new schema → current schema):
   * Uses the column value directly for rollback

Create the migration file with this command:

```sh theme={null}
cat > .xata/migrations/002_alter_order_items.yaml << 'EOL'
operations:
  - alter_column:
      table: order_items
      column: qty
      type: float
      up: |
        CASE 
          WHEN qty < 1 THEN 1.0
          ELSE qty::float
        END
      down: qty
EOL
```

## 9. Start the migration

Start the migration using:

```sh theme={null}
xata roll migrate
```

This command starts the migration but leaves it uncompleted, allowing you to roll back if needed.

## 10. Roll back the migration

When a migration has started, it is serving up both version of the schema simultaneously. You can instantly revert any schema changes during an active migraiton by rolling back the changes. To test rolling back, run:

```sh theme={null}
xata roll rollback
```

## 11. Check migration status

Let's start the migration again and check its status:

```sh theme={null}
xata roll migrate
xata roll status
```

The migration should be `In progress` and you can view all statuses available [here](https://pgroll.com/docs/latest/cli/status).

## 12. Complete the migration

To finalize the migration, run:

```sh theme={null}
xata roll complete
```

In a production scenario, you would only complete the migration when the application code is up to date with the database changes.

## 13. (Optional) Add to CI/CD with GitHub Actions

To automate schema changes in your CI/CD pipeline, you can use our GitHub Actions workflow. This workflow creates a dedicated Xata branch for each pull request, allowing you to test database changes in isolation.

For details about setting up the workflow and available configuration options, see our [GitHub Actions PR workflow documentation](/docs/automations/ga-pr).

## 14. Summary

* Use `xata roll migrate` to apply migrations
* Use `xata roll status` to check migration state
* Use `xata roll rollback` to undo migrations if needed
* Use `xata roll complete` to finalize migrations when your application is ready
* For more details on up/down migrations, see the [pgroll documentation](https://pgroll.com/docs/latest).
* See the [CLI reference](/docs/cli/roll) for all available `xata roll` commands.


# Create Streaming Replica
Source: https://xata.io/docs/tutorials/streaming-replication

Use Xata's streaming replication to keep your database continuously synchronized with real-time changes.

This guide shows you how to set up continuous logical streaming replication from your production PostgreSQL database to Xata, enabling real-time data synchronization with optional anonymization.

## 1. Prerequisites

* A Xata account ([sign up here](https://console.xata.io))
* The [Xata CLI](/docs/cli) installed:
  ```bash theme={null}
  curl -fsSL https://xata.io/install.sh | bash
  ```
* A PostgreSQL database with:
  * Logical replication enabled
  * Role with permissions to create a replication slow (`xata clone stream` command does that automatically)
  * Network connectivity from Xata to your database

## 2. Enable logical replication on source database

First, ensure your source PostgreSQL database has logical replication enabled. You'll need to set these parameters:

```sql theme={null}
-- Check current settings
SHOW wal_level;
SHOW max_replication_slots;
SHOW max_wal_senders;
```

If not already configured, update your PostgreSQL configuration:

```sql theme={null}
ALTER SYSTEM SET wal_level = logical;
ALTER SYSTEM SET max_replication_slots = 10;
ALTER SYSTEM SET max_wal_senders = 10;
```

Restart your PostgreSQL instance for the changes to take effect.

## 3. Create a Xata project and branch

In the Console, create a new project and then click the **Create main branch** button to create the PostgreSQL instance.

For streaming replication, consider using at least 1 replica to ensure high availability during continuous synchronization. Select an instance size that can handle your expected write throughput.

> **Note:** Streaming replication maintains a persistent connection to your source database. Ensure your network allows stable, long-lived connections between Xata and your PostgreSQL instance.

## 4. Configure the Xata CLI

Authenticate the CLI by running:

```sh theme={null}
xata auth login
```

Initialize the project by running:

```sh theme={null}
xata init
```

## 5. Configure streaming replication

Generate a configuration for the streaming process:

```bash theme={null}
xata clone config --source-url $CONN_STRING
```

Where `CONN_STRING` is your PostgreSQL connection string with replication permissions.

The configuration prompt will ask you to:

* Select tables to replicate
* Set up transformation pipelines i.e. anonymization rules

This creates a configuration file at `.xata/clone.yaml` that you can further customize.

## 6. Initialize and start streaming

```bash theme={null}
xata clone stream --source-url $CONN_STRING
```

This command will:

* Create an initial snapshot of your specified tables
* Set up the streaming pipeline
* Begin continuous replication

## 7. Advanced configuration

### Filtering specific tables

To stream only specific tables, use the `--filter-tables` flag:

```bash theme={null}
xata clone stream --source-url $CONN_STRING \
  --filter-tables "users.*,orders.*,products.*"
```

If this option is not specified it defaults to `*.*`

### Custom transformations

Edit your `.xata/clone.yaml` file to add custom transformations:

```yaml theme={null}
transforms:
  - table: users
    columns:
      - name: email
        transformer: mask_email
      - name: phone
        transformer: redact
  - table: orders
    columns:
      - name: credit_card
        transformer: mask_credit_card
```

### Streaming without DDL tracking

DDL tracking via event triggers requires elevated privileges (e.g., `rds_superuser` on AWS RDS/Aurora, or the `postgres` role on Google Cloud SQL). Most managed PostgreSQL services support this — see the [pgstream setup guides](/docs/opensource/pgstream/docs/aws) for provider-specific instructions on configuring the right roles.

However, if your database user does not have the required privileges to create event triggers, you can use the `--skip-ddl-tracking` flag:

```bash theme={null}
# First, create a replication slot on your source database
psql $CONN_STRING -c "SELECT pg_create_logical_replication_slot('xata_replication_slot', 'pgoutput');"

# Then start streaming with DDL tracking disabled
xata clone stream --source-url $CONN_STRING \
  --skip-ddl-tracking \
  --replication-slot xata_replication_slot
```

When using `--skip-ddl-tracking`:

* You must manually create a replication slot on the source database before streaming
* DDL changes (schema modifications) will not be automatically tracked
* The replication slot name must be provided via the `--replication-slot` flag

## 10. Handling failures and recovery

If the streaming connection is interrupted, the replication slot ensures no data is lost. Simply restart the streaming command:

```bash theme={null}
xata clone stream --source-url $CONN_STRING
```

The process will resume from where it left off, catching up with any changes that occurred during the downtime.
However, if the too much lag accumulates then the Postgres server might slow down as it has to do both catching up on the lag and its normal operations.

If you terminate the `xata clone stream` process and do not wish to run streaming replication again, clean up the replication slot and
other `pgstream` objects using `xata stream destroy` command.

Not cleaning up the replication slot will cause the WAL to be aggregated continuously and that would lead to full disk space. Use options like `max_slot_wal_keep_size`
to keep the max WAL size in check.

## Summary

* You now have real-time streaming replication (Postgres's logical replication) from your PostgreSQL database to Xata
* Changes in your source database are automatically synchronized
* Your data can be anonymized in transit using configurable transformers
* The replication slot ensures no data loss during network interruptions

For more details on advanced streaming configurations and monitoring, see the [clone command documentation](/docs/cli/clone).