As a seasoned professional, I have gained extensive experience as a maintainer of Apache Druid k8s operator and have co-authored and committed to various k8s operators/libraries. Through my experience, I have recognized the need to establish application-specific control planes to facilitate the smooth running of big data on Kubernetes.

In my past roles, I have built Kubernetes control planes for Druid at RillData, Pinot at Startree, and Clickhouse at Chistadata. Having worked on all three OLAP stores, I am now fully dedicated to my latest venture, DataInfra, a company focused on building highly efficient control planes that enable the seamless running of data on Kubernetes.

Why Control Plane for Pinot?

We believe that Kubernetes can serve as a control plane for any application, including those with data and stateful sets. While Helm charts are useful for configuration management, they don't maintain the state of the application. That's why we've built a control plane based on the Kubernetes operator pattern that acts as a bridge between your application's requirements and Kubernetes infrastructure.

What's in store for DevOps in 2023

Our 2023 "DevOps: CI/CD, Application Delivery, and Release Orchestration" report provides developers with the latest information on AIOps and MLOps, IaC, GitOps, automation techniques, and more.

Pinot control plane for k8s is specifically designed to improve the user experience of running Apache Pinot clusters. As a distributed database, Pinot can be challenging to run on Kubernetes without the right tools. Our project is based on the DSOI Spec and built using the Operator-Runtime library to provide a more user-friendly and Kubernetes-friendly experience.

This project is based on the Kubernetes operator pattern, but it is not exclusively limited to this pattern. Given the complexity of Pinot, relying solely on Kubernetes operators may or may not be sufficient to effectively manage its operations. Our vision for the project is to create a comprehensive set of tools and utilities that enable seamless deployment and operation of Pinot on Kubernetes.

Installing Pinot Control Plane

Pinot control plane supports Helm Chart-based deployment. To install the chart run the following commands. We will use the Makefile commands to bootstrap the control plane.

• Clone the Repo

git clone https://github.com/datainfrahq/pinot-control-plane-k8s.git

• Deploy Pinot Control Plane — The following commands deploys the control plane using helm in the pinot-control-plane namespace.

make helm-install-pinot-control-plane

• Deploy Zookeeper Operator and Zookeeper — The following command will take a few minutes to complete. It will deploy the zookeeper operator in the zookeeper-operator namespace and zk statefulset in the pinot namespace

make helm-install-zk-operator 

Installing Pinot Cluster

Export Kubernetes storage class.

export STORAGE_CLASS_NAME=standard

• Install Pinot Cluster

envsubst < examples/00-pinot-basic/pinot-basic.yaml  | kubectl apply -f - -n pinot

Installing Kafka Cluster and Create Topics

• Add Helm repo and install Kafka cluster

# Add Kafka
helm repo add kafka https://charts.bitnami.com/bitnami
# Deploy kafka
helm install -n pinot kafka kafka/kafka --set replicas=1,zookeeper.image.tag=latest

•  Once Kafka Pods are up and running, create topics

# Add Kafka
helm repo add kafka https://charts.bitnami.com/bitnami
# Deploy kafka
helm install -n pinot kafka kafka/kafka --set replicas=1,zookeeper.image.tag=latest

Create Pinot Schema

• Pinot control plane supports the creation of schema.

kubectl apply -f examples/00-pinot-basic/pinotschema-basic.yaml -n pinot

Create Pinot Table

• Pinot control plane supports the creation of tables.

kubectl apply -f examples/00-pinot-basic/pinottable-basic.yaml -n pinot

Ingest Data Into Kafka

• Ingest Data into kafka topics

kubectl apply -f examples/00-pinot-basic/pinot-realtime-kafka.yaml

Access Pinot Console

• Access the pinot console by port-forward controller pod or svc.

kubectl port-forward pinot-controller-controller-0 -n pinot 9000