CDC pipeline — Neon logical replication to S3 Iceberg¶
| ID | MOD-042 |
| System | SD07 |
| Repo | bank-platform |
| Build status | Deployed |
| Deployed | Yes |
| Last commit | 39bbf8c1b8255fa6429add330de438d2899dc558 |
Scheduled Lambda (60-second interval via EventBridge Scheduler) connects directly to each Neon
domain database, reads committed WAL changes via pg_logical_slot_get_changes(), and publishes
records to Kinesis Firehose. Firehose writes Apache Iceberg files to S3, catalogued in AWS Glue
Data Catalog. Snowflake reads via External Iceberg Tables — zero-copy, no Snowpipe ingestion cost.
One replication slot per domain database. Last-acknowledged LSN persisted in S3 alongside the data files. Monitoring alerts if the Lambda fails for more than 30 continuous hours (Neon drops inactive slots at ~40 hours).
See ADR-003.
Module dependencies¶
Depends on¶
| Module | Title | Required? | Contract | Reason |
|---|---|---|---|---|
| MOD-104 | AWS shared infrastructure bootstrap | Required | — | AWS shared infrastructure provisioned by MOD-104 (EventBridge buses, S3, KMS, Kinesis, Cognito) is required before this module can be deployed. |
| MOD-102 | Snowflake account configuration & governance | Required | — | Snowflake account and governance provisioned by MOD-102 must exist before this module can read or write Snowflake. |
Required by¶
| Module | Title | As | Contract |
|---|---|---|---|
| MOD-016 | Rule-based typology engine | Hard dependency | — |
| MOD-017 | ML behavioural scoring model | Hard dependency | — |
| MOD-023 | Transaction fraud scorer | Hard dependency | — |
| MOD-027 | Affordability calculator | Optional enhancement | — |
| MOD-028 | Credit score & risk rating | Optional enhancement | — |
| MOD-030 | Stage allocation model | Hard dependency | — |
| MOD-032 | LCR / NSFR calculator | Hard dependency | — |
| MOD-033 | RWA & capital ratio engine | Hard dependency | — |
| MOD-034 | Stress testing scenario engine | Hard dependency | — |
| MOD-035 | IRRBB / EVE / NII model | Hard dependency | — |
| MOD-037 | AUSTRAC / RBNZ AML reporting pipeline | Hard dependency | — |
| MOD-038 | Data quality & reconciliation monitor | Hard dependency | — |
| MOD-039 | Customer risk score model | Hard dependency | — |
| MOD-040 | Churn & health score engine | Hard dependency | — |
| MOD-041 | Categorisation & merchant enrichment model | Hard dependency | — |
| MOD-057 | Statistical returns & survey engine | Hard dependency | — |
| MOD-060 | FATCA/CRS/AEOI reporting engine | Hard dependency | — |
| MOD-079 | Snowflake decision publication service | Hard dependency | — |
| MOD-080 | Statutory financial reporting & ERP integration | Hard dependency | — |
| MOD-086 | Funds transfer pricing engine | Hard dependency | — |
| MOD-100 | External asset connector | Optional enhancement | — |
| MOD-101 | Wealth intelligence engine | Hard dependency | — |
| MOD-150 | Risk management platform | Hard dependency | — |
| MOD-152 | Climate risk assessment | Hard dependency | — |
Policies satisfied¶
| Policy | Title | Mode | How |
|---|---|---|---|
| DT-004 | Data Governance Policy | AUTO |
All operational data changes flow through a single governed CDC pipeline — no shadow extracts or parallel data taps permitted |
| REP-005 | Data Quality & Assurance Policy | AUTO |
Regulatory data sourced from the same Iceberg snapshots as all consumers — no divergent copies or selective replication |
| AML-005 | Transaction Monitoring Policy | AUTO |
Transaction events available to the AML monitoring engine within 5 minutes of posting via S3 Iceberg External Table |
Capabilities satisfied¶
(No capabilities mapped)
Part of SD07 — Data Platform & Governance Infrastructure
Compiled 2026-05-22 from source/entities/modules/MOD-042.yaml