Data masking and anonymization
for PostgreSQL

pgstream is an open source project for PostgreSQL data replication and transformation. It is using PostgreSQL logical replication (including DDL statements) as well as parallel snapshotting in order to copy data at maximum throughput.

You can schedule pgstream via your CI/CD pipeline to execute nightly and create an anonymized snapshot of your production database, which is then written to your staging database.

The snapshots can be created from a read-replica, in order to minimize the impact on the production database.

Anonymization is accomplished by applying a set of column value transformations. Deterministic transformers ensure that the same data will be masked the same way every time. This means that your relational data stays consistent.

pgstream integrates with existing transformer open source libraries, such as Greenmask, NeoSync and go-masker, to leverage a large amount of transformation capabilities, as well as having support for custom transformations written in Golang. This way even the most complex anonymization requirements can be met.

In case your production data is measured in terabytes, it might be more practical to create a smaller staging dataset, yet still large enough to catch performance issues.

Creating this subset is challenging, because it requires keeping track of the relations between tables and subsetting each table in such a way that the data is consistent and interconnected.

In the example diagram, we request that the 'orders' table is subsetted to 5% of the total size. The subsetting logic automatically follows the foreign key relations to the 'users' and 'products' tables and filters the data accordingly.