kiok Key Features
Discover the core capabilities of the workflow orchestration platform.
DAGs as Code
Define workflows in YAML, Python, or Java — all compile once to a single DagSpec. Beyond shell, python, and http, built-in Livy (Spark), Trino, and Ontul operators orchestrate external engine jobs directly.
Flexible Scheduling
Cron schedules and on-demand triggers, with catchup to recover the intervals that passed while the cluster was down. It resumes from the DAG’s last run and creates one run per missed interval, each stamped with the interval it stands for rather than the moment it was created, so a recovered run carries the time its data belongs to. A long gap is capped — the most recent intervals are recovered and the dropped range is logged, because releasing a week of runs at once is another outage, not a recovery. #{ } date macros slide the date on every run, so a DAG compiled once executes with the right date each time.
Distributed Execution & Failover
A master/worker cluster runs tasks in parallel across per-worker task slots, and driver failover keeps runs alive through worker loss — without re-running completed tasks.
Git Deployment & Connections
Auto-sync DAGs from multiple git repositories (including private ones) or upload bundles for air-gapped clusters. Credentials live in an encrypted connection store, used only via ${conn}/${secret} references — never exposed in DAGs or git.
Built-in Security
Native KMS envelope encryption secures all cluster state at rest — metadata, IAM, connections, logs — and AWS IAM-style policies control access down to individual DAGs.
Observability & Operations
Watch live run graphs, streaming job logs, metrics, and cluster topology in the built-in admin UI, with incremental S3 backup and restore for production-grade operations.
Single Sign-On (SSO)
Hand authentication to your existing provider over OIDC, SAML 2.0 or LDAP / Active Directory. The console redirects a browser (Authorization Code with PKCE, or SAML Web Browser SSO); a CI pipeline, an SDK or submit.sh posts directory credentials to the login route and receives the same bearer token an access key yields, so existing clients are unmodified. Permissions come from the mapped kiok group’s IAM policies rather than from the provider, so per-DAG access control applies to a federated user exactly as it does to a local one. The federated session rides the IAM snapshot to the follower masters and the workers, and the login state is sealed with a key every master derives from the master key and carried in the state parameter — so a login succeeds whichever master the load balancer hands the callback to. A SAML assertion cannot be replayed: the record is an atomic ZooKeeper write, so replaying it against another master is refused too. Local passwords are stored as PBKDF2-HMAC-SHA256 hashes, and an older digest is re-hashed on its owner’s next login.
Maintenance Mode — Hold the Scheduler Still
For the moments when work starting underneath you is the problem — restoring a backup, rotating keys — the cluster stops creating work for itself. The cron scheduler creates no runs, pending runs are not dispatched to workers, git sync stops registering DAG definitions, and manual triggers are refused with 503 and Retry-After. Runs already executing are not killed: cancel the ones you need stopped, and that path stays open precisely so you can. Backups and driver-failure detection keep running. Cron firings inside the window are skipped rather than replayed on exit — releasing a night of runs at once would be an outage of its own — and every skipped firing is logged. The setting lives in the replicated RocksDB store, so a leader restarted mid-window, or a follower that wins the election mid-window, comes up already paused.
Use Cases
Data Pipeline Orchestration
Schedule and run ETL/ELT and analytics jobs with Livy (Spark), Trino, and Ontul operators, driven by task dependencies.
Job Automation
Automate recurring batch jobs reliably across a scalable worker fleet.
Workflow as Code
Version-control workflows in git and deploy them to the cluster with a single push.
Reliable Operations
Backup and restore, leader failover, and encrypted state for production-grade reliability.
Considering kiok for your workflows?
Author. Schedule. Orchestrate.
Run your workflows on a distributed, secure orchestration platform.