ItdaStreamKafka-Compatible Interface · Tiered Storage · Real-time Iceberg Streaming
ItdaStream Key Features
Overcoming Kafka limitations by separating compute and storage for unlimited retention and lower cost — and landing topic messages into an Iceberg lakehouse in real time, exactly-once.
Full Kafka Protocol Compatibility
Seamless migration using existing Kafka clients and ecosystem without any code changes.
No-Code Kafka → Iceberg Streaming
Point a topic at a target Iceberg table in the Admin UI and messages land in Iceberg exactly-once — no code. When records carry no event-time column, an ingestion-time column is synthesised and the table is hidden-partitioned by hour, so time-pruned scans work out of the box. A built-in Flink-style engine (controller broker as master, the rest as workers) needs no separate streaming cluster. Set a WAP branch in the sink config to stage streaming writes on a non-main branch, audit them in isolation, then publish to main with a fast-forward via REST (/admin/iceberg/publish) — Write-Audit-Publish.
SDK Multi-Sink Pipelines
Submit filter/select/map/flatMap transforms and multi-sink pipelines — Iceberg, Elasticsearch, JDBC, Kafka — with the Java SDK or submit.sh. Custom map/flatMap transform JARs are uploaded with the job and loaded by a per-job URLClassLoader, so no broker restart is needed. Credentials are registered once in the Connection Registry and referenced by connectionId.
Separation of Compute & Storage
Cloud-native architecture with high-performance Stateless brokers and independent S3 storage scaling.
S3-based Tiered Storage
Incoming messages are taken by an in-memory buffer and a local RocksDB staging store and acknowledged immediately; a background worker batches them into S3 (Cloud tier). Local disk is a buffer, not the archive — so disk capacity never caps your retention.
High Throughput at Low Cost
Writes are acknowledged from memory and local staging while S3 uploads are batched asynchronously, so write latency is never tied to an object-store round trip. Reads are fronted by an in-memory segment cache that removes most round trips on repeat access.
Unlimited Data Retention
Secure long-term data preservation using S3 Object Storage, independent of local disk constraints.
Enterprise Security & KMS
Enhanced security with built-in KMS applying AES-GCM 256-bit Envelope Encryption.
Zero-Rebalancing Scaling
A broker owns no data, so adding one is just starting a process — nothing to move, no replication to wait out.
Drain a Broker Without Dropping a Request
Mark a broker draining before you stop it: every other broker leaves it out of the metadata handed to clients while it keeps answering the requests it already has. Clients move away on their next metadata refresh, so taking a node out costs no failed requests. Shutdown does this automatically, and it is verified end-to-end on a three-broker cluster — drain, stop, restart, zero messages lost.
Integrated Admin UI
Real-time TPS monitoring, topic management, IAM policies, and integrated log browsing.
Drastic Infrastructure Cost Savings
Reduces storage costs by up to 80% compared to traditional Kafka by utilizing S3 over expensive SSDs.
Key Use Cases
Large-scale Log Aggregation
Cost-effectively store and query terabytes of logs for the long term.
Event-driven Microservices
Handle MSA events based on standard Kafka APIs with minimal operational overhead.
Real-time Lakehouse Ingestion
Land topic JSON/Avro messages into Iceberg tables exactly-once with no-code setup — auto-partitioned by time and ready for immediate analytics.
Data Lake Integration
Directly accumulate and analyze real-time events into S3 data lakes.
Cloud Native Migration
Seamlessly transition legacy Kafka infrastructure to elastic cloud environments.
Considering ItdaStream for your infrastructure?
Kafka API Compatible. S3 Tiered Storage. Unlimited Retention.
Innovate with the next-generation distributed streaming platform that enables cost savings and unlimited scalability.