Ontul Key Features
Discover the core features of the distributed data engine that unifies batch processing, stream processing, and interactive SQL in a single engine.
Unified Data Engine
Run batch processing, stream processing, and interactive SQL queries in a single cluster. Consolidate all data workloads without separate systems.
Arrow-Native Execution Engine
Process all data in Apache Arrow columnar format. Iceberg Parquet data files are decoded column-at-a-time directly into Arrow vectors, with Iceberg file pruning and Parquet row-group skipping to minimize bytes read; columnar aggregation and bounded-heap Top-N with zero-copy execution deliver best-in-class analytical performance.
Interactive SQL
JDBC connections (DBeaver, DataGrip) via Arrow Flight SQL with multi-catalog federation queries. Standard SQL — JOINs (INNER/LEFT/RIGHT/FULL), GROUP BY/HAVING, CTEs, DISTINCT. Anything the engine cannot evaluate exactly is rejected with an explicit error before it runs, so a silently wrong number never reaches a dashboard or an agent answer — plus a compiled-plan cache and a snapshot-keyed result cache that skips execution entirely for repeated reads over unchanged data, lifting interactive QPS for BI and AI-agent workloads. The MCP server services JSON-RPC batch requests concurrently, so an agent's multiple tool calls take one round-trip.
Ontul Flow — Visual Streaming Pipelines
Continuous processing — events are handled as they arrive, not in Spark-style micro-batches (TUMBLING/SLIDING/SESSION windows, multi-worker hash shuffle). Build source→transform→sink pipelines in the Admin UI Ontul Flow editor by drag-and-drop or YAML/JSON, then save them as named flows and start/stop them while watching live metrics, streamed logs, data preview and run history on one screen. Sources: Kafka, Debezium CDC (Postgres/MySQL/SQL Server/Oracle/Db2), Iceberg (incremental & changelog), S3 files, NeorunBase. Sinks: Iceberg, JDBC, Kafka, REST, Elasticsearch, NeorunBase — with Iceberg/JDBC CDC-apply (SCD Type 1/2, hard/soft/ignore deletes) to keep an OLTP replica in real time. Flows are governed by owner-scoped IAM and auto-recover from the last checkpoint on worker failure. A record the pipeline cannot decode does not quietly disappear: it never reaches the sink, it is counted by reason and shown in the Flow list's Rejected column, and naming a quarantine table with errorSink keeps the raw payload, the reason and the source offset so you can fix it and replay. When the source schema moves — an upstream ALTER TABLE, a producer adding a field — the target Iceberg table gains the column automatically (with int→long / float→double promotion), or you can choose to ignore the drift or fail on it. Lag is reported on two axes, records behind and how old the newest record is, so falling behind is visible rather than inferred.
Exchange Manager
Unified fault-tolerance infrastructure for Query, Batch, and Streaming. Handles data spill on memory pressure and streaming checkpoint state — all through a single system with KMS envelope encryption. Choose local-disk or S3-primary (Trino-style) mode; in S3 mode spill and checkpoints are read/written directly to S3 so a failed task recovers on a different worker, and a failed streaming task auto-restarts from its last checkpoint.
Exactly-Once Semantics
Master-coordinated barrier checkpoint guarantees exactly-once delivery for transactional sinks (Iceberg, JDBC, NeorunBase, Kafka Transactions). Sink commit before offset commit ensures data consistency.
Connector Architecture
Access diverse data sources through plugin-based connectors. Dynamically register and unregister Iceberg, NeorunBase, JDBC, Lance, Elasticsearch, and more at runtime, with catalog and connection registration in the Admin UI. Elasticsearch is a source + sink: query indices as tables (numeric/boolean predicate pushdown, parallel sliced-scroll) and write via _bulk (upsert with idField). Kafka is a streaming source/sink.
Federation Queries
Execute cross-catalog joins across multiple data sources in a single SQL query. Combine Iceberg, NeorunBase, and JDBC tables seamlessly.
Semantic Layer
Define metrics, dimensions, multilingual synonyms, governance, conformed-dimension joins, derived metrics, and multi-tenant mandatory filters once. Ontul rewrites SELECT revenue FROM sales into the full aggregation, JOIN, GROUP BY, and row filter server-side — clients never duplicate the formula.
Agentic AI Ready
Built-in MCP server gives LLM agents metric discovery, natural-language search (Korean 매출 ↔ revenue), and certification metadata. The semantic layer handles aggregation, JOINs, and RBAC server-side, so agents only need column names — multi-tenant policies follow the authenticated user automatically.
Native Apache Iceberg v2 & v3
Native support for both Iceberg v2 and v3 — distributed INSERT/CTAS plus merge-on-read DELETE/UPDATE/MERGE, hidden partitioning, schema evolution, time travel, branches and tags. On v3, deletes are written and read as deletion vectors (Puffin) instead of position-delete files. All operational capabilities in a single engine. Write-Audit-Publish (WAP) is supported too: stage INSERT/UPDATE/DELETE/MERGE on a non-main branch via SET, audit in isolation, then publish to main with ALTER TABLE EXECUTE fast_forward / cherrypick. It also ships Spark-style table-maintenance procedures (optimize, expire_snapshots, rewrite_manifests, remove_orphan_files, rollback) via ALTER TABLE EXECUTE, with fine-grained parameters such as retain_last, min_input_files, dry_run, and window_hours (window_hours does incremental compaction of just the last N hours of small files — ideal for streaming churn). From the Admin UI you can run per-table auto-maintenance with per-operation toggles, those parameters, and a CRON schedule. Iceberg metadata tables ($snapshots, $history, $files, $metadata_files) are also directly queryable in SQL — for commit statistics (added/total records, files, bytes), small-file diagnosis, and even the UUID-generation boundary of a table recreated at the same path. That metadata is also raised into continuous observability: the Iceberg Health page in the Admin UI keeps collecting per-table condition and turns it into a 0-100 health score with named findings — small files, snapshot accumulation, delete-file debt, metadata.json growth, manifest fragmentation, and maintenance that is off, overdue or failing — each one offering the maintenance operation that fixes it as a button. Collection is split into two tiers so that watching the tables never becomes the load itself: the cheap tier reads only the snapshot summary and runs often (60s by default), while the expensive tier that reads every manifest to build the file-size distribution runs rarely (30m by default). Most importantly, every maintenance run is measured before and after it commits, so the platform records what the run actually changed (data files 9 to 1, delete files 3 to 0, score 65 to 100) — the loop closes at verification, not at detection. The same signals are exported as table-labelled Prometheus gauges for existing monitoring. REST catalog sessions also survive their own expiry: the Iceberg refresh loop stops scheduling after a single failed attempt, and from then on the catalog answers 401 to everything the moment its token runs out. Ontul rebuilds the session and retries once when a call comes back unauthorized, and renews ahead of expiry on a timer — so a catalog that has been serving all day does not start refusing valid credentials an hour in.
Lance Connector — Vector & Multimodal Lakehouse
Integrates LanceDB's Lance format as a first-class catalog connector (vendor-neutral org.lance lineage, format-compatible with the Spark and Trino lance connectors). Vector ANN and full-text (BM25) search are exposed as the SQL functions vector_search() and match(); Korean/CJK full-text uses lindera/ko-dic morphological tokenization via a pylance sidecar (the bare noun "데이터베이스" matches the inflected "데이터베이스를"). CREATE INDEX (btree, bitmap, ivf, inverted) builds secondary, vector and FTS indexes. Writes are distributed (workers write Lance fragments, the master single-commits) for INSERT/CTAS, with native deletion-vector DELETE, native mergeInsert MERGE (upsert), and engine-rewrite UPDATE. A Kafka→Lance streaming sink and a streaming-friendly two-step maintenance (OPTIMIZE compaction + VACUUM reclaim) are included — OPTIMIZE takes window_hours, cooldown, min_input_files and dry_run to incrementally compact only recent commits without scanning the whole table (Lance has no hidden partitions, so the fragment window is derived from version timestamps). Unstructured data — video, audio, images, documents — lives directly in the table as a Blob V2 column, co-located with its embeddings and metadata in the same transactional, versioned row, so a search hit hands you the original asset from the very same row. The bytes are read back through the engine over a governed, IAM-gated streaming path (and the Python SDK) in bounded ranges, so large assets never bypass access control; multi-GB originals can be kept as external references while only their derivatives (embeddings, thumbnails) sit inline in a hybrid layout. With Apache Polaris Generic Tables as the shared catalog, the same Lance tables interoperate across engines — read and written by Spark and Trino alongside Ontul — and stay consistent after maintenance (compaction and cleanup). It also works as a semantic-retriever backend for in-engine RAG vector/FTS search, with IAM column-masking and row-level policies applied at the scan output.
Security (IAM & KMS)
AES-256-GCM envelope encryption, built-in KMS, Exchange Manager data encryption, catalog/table/column/row-level IAM policies, and STS temporary credentials.
BI Integration (Tableau · Power BI · Looker)
Tableau, Power BI, Looker, and DBeaver connect live via Arrow Flight SQL JDBC. Semantic views expose measures and dimensions with the right classification, and /api/v1/bi/connection-info returns driver coordinates plus per-tool setup hints in one call.
Ontology & Actions (Objects · Links · Actions)
An ontology layer that models your data in business terms. Define entities and relationships with object types (property↔column) and link types (relational JOIN or NeorunBase graph traversal); agents and apps query instances and traverse relationships by logical property name. Writes go through governed Actions — DML against the system of record, or a REST operation to an operational system (ERP, CRM, payment, …) — with RBAC, approval, idempotency, and audit enforced. Actions are callable from SQL as well as REST: `CALL catalog.schema.action(param => value)` over the same JDBC connection a BI tool or agent already uses. Both entry points run the identical path, so authorization, idempotency, approval, audit and lineage hold whichever one triggered the write — and the right to write into an external system is granted separately from the right to read. Multiple actions compose into an Action Workflow: a governed DAG authored in YAML and run as a Saga that compensates completed steps in reverse on failure, with execution delegated to workers like batch/streaming jobs — so an agent reads and changes data safely through MCP tools. Definitions carry certification. Certifying is a distinct act whose signer the server records — nothing can declare itself certified in a payload — and it stamps a fingerprint of the definition at that moment. Repoint the read source or rebind a property afterwards and it drops to STALE automatically; editing a description or a tag does not revoke the sign-off. Trust never exceeds what it is built on, so an object type over a draft view, or a link with an uncertified endpoint, reports as draft. The verdict travels on every read as a single effectiveStatus field, so an agent can state "this came from certified definitions" without working it out itself.
Maintenance Mode — Close Writes Without Killing Queries
Closes the cluster-wide write path for work where a concurrent write would be a problem: restoring a backup, rotating KMS keys, rewriting tables with an external tool. SQL DML and DDL, CALL actions, ontology action invokes and job submission are refused with 503 and Retry-After, while SELECT keeps being served and catalogs, connections, IAM and KMS stay editable — because editing them is usually the reason the window was opened. The authorization check that the Trino, Spark and Flink plugins call stays open too: refusing it would fail those engines closed and stop every query in the lakehouse, not just Ontul's. So does job kill, since stopping the jobs still writing is the first thing you do once the window is open. The setting lives in the replicated RocksDB metadata store, so every master agrees immediately and a master restarted mid-window comes back still refusing writes.
Data Lineage & Audit (Multi-Engine)
Lineage for CTAS/INSERT/MERGE/VIEW is auto-extracted from the Calcite query plan at table and column granularity, and an audit log records who, which engine, which tables, and what query touched the data. Audit is written on the leader and replicated to standby masters so it survives failover; the write path bypasses IAM (an internal component is the sole writer) while the read path enforces IAM. From the Admin UI you can tier older audit to S3 Iceberg (v2) or Parquet — bulk-written via Ontul’s own Iceberg handler — and enabling Iceberg tiering auto-registers the audit table for hourly Iceberg maintenance (compaction, snapshot expiry, orphan cleanup). Read auditing is bounded by sampling and retention days, and Trino/Spark/Flink authz plugins push their lineage and audit into Ontul over REST. In the Admin UI, click a user→engine→table relationship node to drill down into its read/write audit, or use the dedicated search page.
Semantic Layer — Single Source of Truth for Agentic AI
One definition of truth per metric — enforced server-side.
Ontul's semantic layer gives agents two things. ① Numbers (metrics) — define an analytics measure like revenue or margin once, and LLM agents, Tableau, and analysts all see the same number. ② Relevant context (retrievers) — multi-modal search that finds related documents by text or image. If a metric answers "how much revenue?", a retriever answers "find the related documents." Agents get both through one interface.
Core Capabilities
Server-Side Query Rewriting
SELECT revenue, customer.region FROM sales becomes SUM(amount * (1-discount)) with LEFT JOIN customer ON ... and GROUP BY customer.region — automatically. Clients never have to memorize the formula.
MCP-Native Metric Discovery
LLM agents use ontul_search_metrics and ontul_describe_semantic_view to find metrics across multilingual synonyms (매출 · revenue · net_revenue · sales_amount) and read their definitions. One definition is shared by every agent.
Derived Metrics
profit = revenue - cost, profit_margin = (revenue - cost) / revenue — define metrics in terms of other metrics. Ontul resolves them recursively at plan time, with cycle detection.
Conformed-Dimension Joins
Declare a JOIN once; Ontul injects it only when its columns are referenced. SELECT customer.region, revenue auto-adds LEFT JOIN customer ON ..., while unused joins stay out of the plan — declared joins cost nothing until used.
Multi-Tenant Mandatory Filters
Declare row-scoping predicates like tenant_id = ${user.attr.tenant_id} at the view or per-metric level. Substituted from the authenticated user context, so the same RLS policy applies whether the caller is a BI dashboard or an LLM agent.
Governance & RBAC
Per-metric allowedRoles for access control, DRAFT → CERTIFIED → DEPRECATED lifecycle, certifier audit, free-form tags. Enforced at rewrite time — unauthorized users never see the formula in error messages.
Retrievers — Multi-Modal Search in One Call
The search object an agent uses to find "what's related" by text or image. It runs vector (meaning), keyword (BM25), and graph (relationships) together on NeorunBase, protected by the same IAM and permissions as metrics. The agent writes no SQL — it just fills in the values, and that's RAG. The order the candidates come back in can be refined by the re-ranking stage below. (HYBRID_SEARCH / GRAPH_NEIGHBORS defined as governed retriever objects, pushed down through Ontul.) The caller identity and IAM attributes are bound into the template as ${user.id} / ${user.roles} / ${user.attr.X}, so the same retriever called with the same question returns different rows to different people — scoped inside the search rather than filtered by the application afterwards.
Re-ranking — Reorder by Actual Relevance
Recall embeds the question and the documents separately, so it cannot tell "a document about refunds" from "the document that answers how many days". Re-ranking puts the two in front of a cross-encoder together and keeps only the best few. The model sits on an endpoint outside Ontul while the retriever holds just a connection id, so a 0.6B→4B swap or a host move changes neither the retriever definition nor the Ontul deployment. If the endpoint goes down, results fall back to the recall ordering and the response says why — quality never degrades silently.
One line is enough
SELECT customer.region, profit_margin
FROM saas.core.sales
WHERE ship_date >= DATE '2024-01-01';SELECT customer.region,
(SUM(amount) - SUM(unit_cost * quantity)) / SUM(amount)
AS profit_margin
FROM saas.core.sales
LEFT JOIN saas.core.customer customer
ON sales.customer_id = customer.id
WHERE ship_date >= DATE '2024-01-01'
AND tenant_id = 'acme-co' -- auto RLS
AND status = 'COMPLETED' -- per-metric filter
GROUP BY customer.region;What this means for Agentic AI
No Hallucinated Metrics
Formulas live once, server-side. Even if an LLM guesses AVG instead of SUM — as long as the metric name is right, the correct aggregation runs every time.
IAM Auto-Propagation
The metrics and rows an agent can see are exactly what the user's IAM policy allows. No prompt-level permission logic, no bypass.
Multilingual by Default
"매출 어떻게 돼?" finds the revenue metric via synonym matching. Business terminology varies by team — the semantic layer bridges that gap.
BI · AI Consistency
The revenue Tableau shows and the revenue an LLM agent answers are computed by the same SQL. The two channels never disagree on the number.
Use Cases
Unified Data Processing
Handle all data workloads — batch, streaming, and SQL — with a single Ontul cluster instead of separate systems.
AI Agent Analytics
LLM agents discover metrics through MCP tools and translate natural-language questions into Ontul SQL. The semantic layer handles aggregation, joins, and IAM — so agents answer with certified business definitions, not hallucinated formulas.
Real-Time Data Pipelines
Ingest data from Kafka, process in Ontul, and load into Iceberg tables for real-time ETL pipelines.
Data Lake Analytics
Run federation queries across Iceberg, JDBC, and other sources for unified analytics.
Analytics + RAG in One Backend
Run metrics (analytics) and retrievers (multi-modal search) on one engine under one governance — no separate semantic-analytics tool and vector/graph search stack to operate. An agent pulls "the numbers" and "the supporting context" together in a single MCP session.
Considering Ontul for your data platform?
Unified. Arrow-Native. Agentic AI-Ready.
A distributed data engine that unifies batch, streaming, SQL, and a production semantic layer — so BI dashboards and AI agents answer with the same truth.