Skip to main content

Design Notes

pgrwl is designed to always stream WAL data to the local filesystem first. This design ensures durability and correctness, especially in synchronous replication setups where PostgreSQL waits for the replica to confirm the commit.

  • Incoming WAL data is written directly to *.partial files in a local directory.
  • These *.partial files are synced (fsync) after each write to ensure that WAL segments are fully durable on disk.
  • Once a WAL segment is fully received, the *.partial suffix is removed, and the file is considered complete.

Compression and encryption are applied only after a WAL segment is completed:

  • Completed files are passed to the uploader worker, which may compress and/or encrypt them before uploading to a remote backend (e.g., S3, SFTP).
  • The uploader worker ignores partial files and operates only on finalized, closed segments.

This model avoids the complexity and risk of streaming incomplete WAL data directly to remote storage, which can lead to inconsistencies or partial restores. By ensuring that all WAL files are locally durable and only completed files are uploaded, pgrwl guarantees restore safety and clean segment handoff for disaster recovery.

In short: PostgreSQL requires acknowledgments for commits in synchronous setups, and relying on external systems for critical paths (like WAL streaming) could introduce unacceptable delays or failures. This architecture mitigates that risk.