ConnectHub is a Kafka Connect control plane we engineered from the ground up. It collapses weeks of CDC pipeline setup into a guided 6-step workflow with zero JSON configuration, live schema discovery, column-level PII masking, and one-click Kubernetes manifest generation.
A platform that lets any data engineer deploy a production-grade Debezium CDC pipeline in minutes, with PII compliance built in, Kubernetes manifests auto-generated, and silent failures surfaced before they become incidents.
The Problem We Solved
A production-grade Debezium CDC pipeline from PostgreSQL to OpenSearch requires mastering 80+ connector properties, hand-writing error-prone JSON, managing plugin installation on every worker node, configuring obscure SMT chains, building Kubernetes manifests, and monitoring task failures across multiple REST endpoints. All of this with no visibility into data lag or silent failures dropping messages into a dead-letter queue nobody is watching.
80+ configuration properties per connector, no validation until runtime, written entirely by hand. A single typo in table.include.list silently excludes an entire table from the CDC stream with no error logged.
The correct Debezium JAR must be manually installed on every Kafka Connect worker. No guided path, no version validation, no detection of missing plugins before the connector fails at runtime with a cryptic ClassNotFoundException.
No native mechanism to mask, hash, or exclude sensitive columns before data reaches Kafka brokers. GDPR and HIPAA compliance requires engineering each masking rule by hand, a process that scales to every table, every column, every connector.
Connectors show RUNNING in the UI while silently dropping messages into a dead-letter queue with no alerts. Consumer lag accumulates for hours before anyone notices. MTTR stretches because the failure timeline is invisible.
Writing Strimzi KafkaConnector CRDs and Secret objects manually for every pipeline. Credentials embedded in CRDs. No GitOps-safe output. Every new connector is a fragile copy-paste exercise compounding over time.
Zero visibility into which source databases feed which Kafka topics, which sinks consume them, or what transformations are applied. Lineage documentation is always out of date the moment it is written, making it a compliance audit risk waiting to happen.
The real problem wasn't the complexity of any single step. It was that every step depended on knowing all the others, and a mistake in step one wouldn't surface until step seven, in production, at 2am.
Real Scenario: Ecommerce at Peak
It's 11:58pm on Black Friday. A developer adds a discount_code column to the orders table.
Debezium picks up the schema change fine. But the OpenSearch sink connector hits a mapping exception on the new field - the index has strict dynamic mapping set. The sink goes FAILED and starts writing records to the DLQ. The source connector keeps showing RUNNING. Your homepage serves stale stock data for four hours while the DLQ fills up. Customers check out items that are gone. Returns and chargebacks follow.
This is the exact failure mode that hits ecommerce teams managing these connectors by hand. The source looks healthy. Only the sink is failing. And nobody is watching the DLQ.
The Solution
ConnectHub's deployment wizard walks engineers from source to sink without writing a single line of configuration. Live JDBC schema discovery, per-column PII controls, SMT composition, and automated artifact generation are built into every step of the workflow.
How It Works
The wizard enforces a logical sequence that mirrors how a senior data engineer would think about a CDC pipeline: source first, schema second, compliance third, destination fourth, transforms fifth, deploy last. Each step validates the previous one before proceeding.
Choose PostgreSQL, SQL Server, MySQL, Oracle CDC via Debezium, or Kafka-to-Kafka replication via MirrorMaker 2. Plugin availability is checked against your cluster immediately.
Live JDBC connection enumerates every table and column in your source database. Point-and-click selection with no typing table names and no typos in table.include.list.
Per-column controls: include, exclude, mask with asterisks, SHA-256 hash (GDPR-compliant), or truncate to N characters. Rules translate directly into SMT configuration with no manual JSON required.
OpenSearch for search and analytics, Amazon S3 for data lake archival, or JDBC for any database target. Kafka topics are auto-derived from your earlier table selection.
9 guided transforms: ExtractNewRecordState, MaskField, TimestampConverter, ByLogicalTableRouter, ReplaceField, InsertField, Filter, Cast, ValueToKey, each with guided configuration and field-level validation.
One-click direct deploy via Kafka Connect REST API, or export connector JSON, Docker Compose, Helm commands, and Strimzi Kubernetes manifests for GitOps workflows.
Artifacts generated from a single wizard flow
Deploys source and sink connectors to your Kafka Connect cluster via REST API instantly. No copy-paste, no manual steps.
Downloadable source.json + sink.json ready for CI/CD pipelines and version control.
Ready-to-run compose stack with KRaft Kafka, Kafka Connect workers, and all environment variables pre-populated.
KafkaConnector CRDs with credentials in a separate Secret object, base64-encoded and reference-linked for clean GitOps. Namespace-configurable.
helm repo add, helm install, and kubectl heredoc commands ready to paste into a terminal or pipeline.
Step-by-step instructions: enable CDC in PostgreSQL, set WAL level, create replication user, grant publication permissions.
Engineering Capabilities
Thirteen capabilities we built to cover the full lifecycle of a Kafka Connect deployment, from connector discovery and initial pipeline setup through day-two operations, compliance, and multi-cluster management.
The wizard fills safe defaults for 80+ connector properties, auto-derives Kafka topic names, and deploys directly. A new data engineer can deploy a production Debezium CDC pipeline on their first day with no documentation required.
Core FeatureConnects to your source database at configuration time and enumerates every table, every column, every data type in a point-and-click interface. Eliminates the typos in table.include.list that cause silent data quality failures at snapshot time.
Column-level controls: mask with asterisks, salted SHA-256 hash (pseudonymization under GDPR Article 4(5)), exclude entirely, or truncate to N characters. Rules are enforced at the Kafka Connect layer so no PII ever reaches the Kafka broker. Masking proof is persisted in config history for auditors.
ConnectHub calls only the Kafka Connect REST API. It is not a Kafka consumer, not a JDBC proxy in production. Data flows source to broker to sink exactly as with raw connector JSON. This eliminates data sovereignty objections in financial services and healthcare.
SecurityPipeline Health Check surfaces what the standard Kafka Connect UI hides: DLQ accumulation, consumer lag per partition, tables with zero messages (snapshot gap), error tolerance misconfigurations, schema mismatches, and auth failures, all categorised in plain language.
OperationsAlert rules scoped to connector name patterns using regex. Configurable duration threshold prevents noise from transient restarts. Webhooks to PagerDuty, Slack, OpsGenie, or custom endpoints. Auto-resolves when the condition clears. Full event history for post-mortems.
OperationsA visual SVG directed acyclic graph automatically derived from live connector configurations, with no manual metadata entry required. Source database to connector to Kafka topic to sink to destination. Auto-refreshes every 60 seconds and is always current.
VisibilityPer-cluster: connector uptime percentage, mean time to recovery computed from FAILED-to-RUNNING transitions, hourly error timeline, and consumer lag trends per sink connector. Time windows from 6h to 7d. Auto-refreshes every 60 seconds.
OperationsGenerates Strimzi KafkaConnector CRDs plus a separate Secret object, where credentials are never embedded in the CRD. Namespace-configurable. Base64-encoded and reference-linked for clean GitOps workflows. Helm commands included in the same view.
InfrastructureEvery configuration change persisted with a diff view: which properties changed, old and new values, timestamp, who made the change. Passwords masked in history. Answers "who changed this connector config and when?" immediately after a 2am incident.
CompliancePer-connector Schemas tab auto-derives topics, fetches key and value subjects, shows full version history. Evolution warnings via the SR compatibility endpoint surface schema incompatibilities before production failures occur. Avro, Protobuf, and JSON Schema supported.
Core FeatureRegister dev, staging, prod, and multi-region Kafka Connect clusters in a single pane of glass. Full isolation prevents accidental prod deploys. Deploy to dev first, inspect configuration, then promote. Each cluster independently scoped per organisation.
OperationsBrowse the full Confluent Hub connector catalog directly inside ConnectHub. Every available source and sink connector is listed with its required and optional configuration properties, accepted values, and defaults. Engineers can evaluate and compare connectors before starting a pipeline, without leaving the platform or reading external documentation.
DiscoveryDelivered Outcomes
The metrics that matter to a data engineering team shipping real pipelines to production at scale. Not feature counts, but operational outcomes.
What previously required 2-3 days of a senior data engineer's time, including researching connector properties, hand-writing JSON, configuring SMT chains, and writing K8s manifests, now completes in under 30 minutes through the guided wizard. Day-one engineers deploy production CDC pipelines correctly, every time.
Column-level masking and salted SHA-256 hashing applied before data reaches any Kafka broker. No PII in transit, no PII at rest downstream. Compliance teams get an auditable config history with every masking rule, every change, and every timestamp. GDPR and HIPAA requirements addressed structurally, not by policy.
Pipelines that previously showed RUNNING while silently accumulating DLQ messages are now caught within two health-check cycles (60 seconds). DLQ accumulation, snapshot gaps, consumer lag anomalies, and schema mismatches surface in plain language before they become incidents.
Strimzi KafkaConnector CRDs and credential Secrets generated with correct base64 encoding and namespace configuration. Engineers previously spending hours writing and debugging K8s manifests now export them directly from the same view they deployed from.
A visual DAG derived from live connector configurations, not from documentation that goes stale. Auto-refreshes every 60 seconds. Compliance audits that previously required days of manual lineage reconstruction now complete in seconds.
Single Message Transforms including ExtractNewRecordState, MaskField, TimestampConverter, and ByLogicalTableRouter configured through a UI, not by reading Kafka Connect documentation and hand-constructing transform chains that break silently when mis-ordered.
Dev, staging, production, and regional clusters managed from a single interface. Accidental cross-environment deploys eliminated by org-level isolation. Teams promote pipeline configurations from dev to prod with confidence, as the config is identical and only the target cluster changes.
Ecommerce Outcomes
Product stock levels in OpenSearch reflect the live orders table within seconds, not hours. Actual latency depends on sink connector batch settings - teams running lower batch sizes get sub-10s end-to-end. Either way, your homepage is not serving data from six hours ago.
Consumer lag alerts fire before backlog accumulates. During flash sales and peak traffic spikes, the monitoring catches throughput increases before they stall the pipeline and cause inventory reads to go stale.
Card data is provably never in Kafka in plain text. Masking rules are documented in config history with timestamps. PCI-DSS auditors get a clear, reproducible paper trail without anyone pulling connector logs at 9pm.
Multi-region ecommerce deployments, dev and prod clusters, and multiple source databases all managed from the same interface. Promote a pipeline config from staging to prod without rewriting a single property.
Technical Architecture
ConnectHub is architecturally isolated from the customer data plane. All business data flows directly from source to Kafka broker to sink. ConnectHub calls only the Kafka Connect REST API to manage connector lifecycle. This is a structural guarantee, not just a policy.
"ConnectHub never receives, stores, or proxies any of your business data. Source DB credentials are sent directly to the Kafka Connect REST API during the wizard and are not retained. This is the architecture, not a marketing claim."
Data Lineage — Zero Configuration
The lineage DAG is derived entirely from live connector configurations via the Kafka Connect REST API. No manual metadata entry, no separate lineage tool, no YAML to maintain. The moment you deploy a new connector, it appears in the lineage graph.
Supported Connectors & Transforms
Four Debezium CDC source connectors spanning every major enterprise database, plus Kafka-to-Kafka replication via MirrorMaker 2. Three sink connectors covering the most common downstream targets, and nine guided Single Message Transforms for production-grade data shaping.
Source Connectors — CDC via Debezium 2.7
| Connector | CDC Method | License |
|---|---|---|
| PostgreSQL | WAL pgoutput / wal2json | Apache 2.0 |
| SQL Server | CDC + transaction log | Apache 2.0 |
| MySQL | Binlog | Apache 2.0 |
| Oracle | LogMiner | Apache 2.0 |
Kafka-to-Kafka replication via MirrorMaker 2 is also supported as a Kafka Connect source connector (MirrorSourceConnector), for cross-cluster and cross-region topic mirroring.
Need a connector not listed above? ConnectHub's architecture is built to onboard any new Debezium source or Kafka Connect sink in 2 to 3 days of engineering effort. MariaDB, MongoDB, Cassandra, or a custom sink — the wizard, schema discovery, PII controls, and Strimzi manifest generation all extend to new connectors with minimal plumbing. The platform grows with your stack.
Sink Connectors
| Connector | Use Case |
|---|---|
| OpenSearch (Aiven) | Search and analytics indexing with full document upsert support |
| Amazon S3 (Confluent) | Data lake and cold archival using Parquet, Avro, or JSON format |
| JDBC Sink (Confluent) | Any JDBC-compatible database target with upsert, insert, or delete mode |
The guided wizard covers the connectors above. The Connector Catalog goes further — it surfaces every connector available on Confluent Hub, complete with all configuration properties and accepted values, so your team can evaluate any connector before requesting it for a new pipeline. No external documentation needed.
Single Message Transforms — 9 Guided Transforms
ExtractNewRecordStateUnwraps the Debezium CDC envelope to a flat row representation. Essential for all CDC pipelines: without it, sinks receive the full Debezium change event structure, not the row data.
MaskFieldReplaces sensitive fields with null or a fixed string. Applied per-column from the PII configuration step with no manual SMT JSON required.
TimestampConverterConverts timestamp formats between representations: Unix milliseconds to ISO-8601, epoch to string, and more. Eliminates sink-side type mismatch errors.
ByLogicalTableRouterRoutes multiple source tables into a single Kafka topic with a discriminator field. Used for multi-tenant schemas and table sharding patterns.
ReplaceFieldInclude, exclude, or rename specific fields in the message payload. Strips internal Debezium metadata fields before data reaches downstream consumers.
InsertField · Filter · Cast · ValueToKeyAdd metadata fields (e.g. __source_table) to every message; drop events by field condition; cast field data types (String → Int32, Long → Double) to fix schema mismatches before data reaches sinks; promote payload fields to the Kafka message key for correct partition assignment and log compaction.
Security Model
Multi-tenancy enforced at the database query level, not just application logic. Every JWT carries an orgId claim and every query is scoped to the org. Cross-tenant data access is structurally impossible.
Stateless JWT tokens with configurable expiry. BCrypt password hashing with appropriate cost factor. All API endpoints require a valid JWT with org-scoped claims. No session state is stored server-side, making it horizontally scalable without sticky sessions.
Every JWT carries an orgId claim. Every database query filters by orgId at the query level so clusters, connectors, alert rules, health history, and lag snapshots are all fully org-isolated. A compromised token from Org A cannot reach Org B's data by any code path.
ConnectHub never receives, stores, or proxies your business data. Source database credentials entered during the wizard are transmitted directly to the Kafka Connect REST API and are not retained in ConnectHub's database. This is the architecture, not a configuration option.
Passwords masked in UI displays and config history diff views. Kubernetes manifests generate separate Secret objects, where credentials are never embedded in KafkaConnector CRDs. Base64-encoded and reference-linked for clean GitOps workflows that pass security scanning.
The hardest security property to achieve is the one that eliminates an entire class of risk by design. Zero data custody isn't a control; it's an architectural constraint that means there's nothing to breach in the first place.
Observability
Health snapshots every 30 seconds. Alert evaluation every 60 seconds. Consumer lag polled via Kafka AdminClient every 120 seconds. Built-in monitoring dashboards expose connector health, host metrics (CPU, memory, disk), and Kafka JMX metrics (JVM heap, GC pause, thread count). Prometheus and Grafana are bundled with ConnectHub's deployment — no separate observability stack to provision.
Running snapshots divided by total snapshots in the selected time window. Per connector, per cluster, with trend line.
Average time from first FAILED snapshot to next RUNNING. An objective measure of pipeline brittleness that tracks improvement over time.
Hourly bar chart of snapshots where tasksFailed > 0 or state equals FAILED. Correlate with deployment events.
Dead-letter queue accumulation surfaced in the Pipeline Health Check. Catches messages being silently dropped before they become data loss.
Area chart per sink connector from the lag snapshots table. Total lag plus trend line for early warning before sinks fall behind irreversibly.
Schema Registry compatibility checks per connector, flagging schema compatibility violations before they cause downstream consumer failures.
LSN or binlog position per partition tracked over time. Derive data velocity from offset delta to detect CDC stalls before replication lag becomes visible.
Warns when tables are added to table.include.list after the initial snapshot, a common cause of missing historical rows that silently corrupts downstream aggregations.
Platform Walkthrough
A complete visual tour of ConnectHub, from registering a cluster through the full 6-step deploy wizard to the monitoring dashboards. Every screenshot is a real screen from the running application.
Competitive Positioning
Confluent Control Center and Lenses.io address cluster management and observability. Neither was designed to solve the setup complexity, PII compliance, or Kubernetes automation problems that kill adoption of Kafka Connect CDC in regulated industries.
| Capability | Confluent Control Center | Lenses.io | Manual / Scripts | ConnectHub |
|---|---|---|---|---|
| 6-step guided deployment wizard | Partial | No | No | Yes, fully guided |
| Live schema discovery via JDBC | No | No | No | Yes, live JDBC |
| Per-column PII masking in wizard | No | No | No | Yes, mask / hash / exclude |
| Missing plugin detection + install guide | No | No | No | Yes, auto-detected |
| Strimzi K8s manifest generation | No | No | Manual | Yes, one click |
| Silent failure detection (DLQ, snapshot gap) | Partial | No | No | Yes, full pipeline check |
| Data lineage auto-derived from configs | Partial | Yes | No | Yes, zero config |
| SMT configuration UI (9 transforms) | No | No | No | Yes, guided UI |
| Monitoring with MTTR computation | Partial | Yes | No | Yes, 6h to 7d windows |
| Schema Registry with evolution warnings | Partial | Yes | No | Yes, per connector |
| Self-hostable | SaaS only | Yes | N/A | Yes, self-host or SaaS |
| Data custody | Confluent sees data | Lenses sees data | N/A | Zero (pure control plane) |
Ecommerce Use Cases
These are the source tables ecommerce data teams deal with daily. Here is how ConnectHub fits into each pipeline.
| Table | Source DB | Sink | ConnectHub Feature | Why It Matters |
|---|---|---|---|---|
| orders | PostgreSQL | OpenSearch + S3 | Schema discovery, auto topic naming | Live order search and data lake archival without maintaining two connector configs manually |
| customers | PostgreSQL | S3 data lake / Snowflake | PII masking: email SHA-256, phone masked, address excluded from CDC stream | GDPR compliance enforced at the connector layer - masked fields never reach any downstream store via this pipeline |
| inventory | MySQL | JDBC (read replica / analytics DB) | Consumer lag monitoring, webhook alerts | Alert fires when lag exceeds threshold so the team knows before a product page shows wrong stock |
| products | PostgreSQL | OpenSearch | Schema Registry, evolution warnings | New product attributes are caught before they break the search index mapping |
| payments | SQL Server | Snowflake (native Kafka connector) | PCI-DSS masking: BIN kept, PAN truncated to last 4, CVV excluded | Card data never travels through Kafka in plain text - provable to auditors via config history |
Work With Our Data Engineering Team
Our data engineering team built and operates ConnectHub end-to-end - from Debezium CDC internals to Spring Boot backends to Strimzi Kubernetes deployments. We bring this exact expertise to client engagements in streaming data architecture, pipeline operations, and compliance-ready data engineering. Explore our broader engineering services or see other case studies.
Ecommerce CTO
Your platform processes 50K orders a day. A schema change at peak breaks your CDC pipeline. You find out via a customer complaint, not an alert. ConnectHub surfaces that failure within two health-check cycles.
Data Engineering Lead
You manage 12 Debezium connectors by hand across 3 environments. One typo in table.include.list silently drops a table from CDC. You will not know until the warehouse is stale and a stakeholder asks why the numbers look wrong.
Compliance Lead
Your PCI-DSS scope covers every system that touches card data. ConnectHub's masking proves card fields never leave the Connect layer - and the config history gives auditors a timestamped record without anyone pulling logs.
Running PostgreSQL + Kafka on an ecommerce platform? We built ConnectHub. Let's map your order-to-sink pipeline.
