Running MySQL in Kubernetes/ Openshift

Vadim Tkachenko, CTO, Percona

What is Kubernetes?

Kubernetes is a portable, extensible open-source platform for managing containerized workloads and

services, that facilitates both declarative configuration and automation.

What is Kubernetes? (my version)

Kubernetes is a way to “automagically” manage data center resources requested to run

applications and tasks

Kubernetes is an Operating System for multiple computing, networking, and storage resources

Why Kubernetes?

● Cloud Agnostic○ Unified API

● A standard supported by multiple cloud and software vendors● Based on modern container-based deployment

Massive support from the industry leaders● OpenShift from RedHat● Amazon Elastic Container Service for Kubernetes● Google Kubernetes Engine● Azure Kubernetes Service● Pivotal Kubernetes Service● Digital Ocean Kubernetes (Early Access)● 35+ public distributions of Kubernetes on the market

Let’s look under the hood

Kubernetes Node

Kubernetes Nodes

Properties● Container can be destroyed and restarted automatically● POD can be destroyed and restarted on a different Node● POD can be multiplied - scaling out● Great for “stateless” application, e.g. web server● No so great for databases○ We want to keep our data

Databases as first class citizens● Made possible with○ Stateful Sets○ Persistent Storage

MySQL In Kubernetes● A single instance is easy○ https://kubernetes.io/docs/tasks/run-application/run-single-

instance-stateful-application/

A complex setup is … more complex● High Availability● Persistent Storage● Backup and recovery● Traffic routing● Monitoring

Disclaimer● The further information is based on our Proof Of Concept● Use it as a guidance for your own setup, not as a production

ready product● https://github.com/Percona-Lab/percona-openshift

Replication

Replication Options● Traditional MySQL replication● Group Replication● Percona XtraDB Cluster

Traditional Replication challenges● No automatic failover○ New master detection can be quite complicated

● No automatic slave setup

Group Replication● Not mature enough● Still no automatic slave setup

What is Percona XtraDB Cluster (PXC)?

Why PXC? Auto provisioning...

PXC: Data transfer

Storage

Storage options● Local storage○ Can be fast (SSD, NVMe options)○ Still limited management capabilities in Kubernetes

● Network storage○ Management and Flexibility if supported by your setup

■ Supported by major cloud providers

○ On-premise setup may require Ceph or SAN

○ The dynamic provision of persistent volumes is required

Traffic routing

Traffic routing● ProxySQL○ High-performance MySQL proxy○ Query Routing○ Supports Failover

What is ProxySQL?

ProxySQL read/write split

Query routing is one of the core features of ProxySQL.

Read/write split is perhaps one of the most common query routing use

ProxySQL read-write split

PXC + ProxySQL on Kubernetes / Openshift

Monitoring● Percona Monitoring

And Management

OpenShift Percona Labshttps://github.com/Percona-Lab/percona-openshift$ git clone git@github.com:Percona-Lab/percona-openshift.git

Deployment options

Deployment options● Manual● Helm Charts● Operator

Manual● Manual juggling multiple YAML files

● Cumbersome and error-prone● Practically not feasible for a complicated setup● Deployment logic and configuration parameters are mixed

together

backup-volume.yamlmysql-configmap.yamlproxysql-pxc.yamlpxc.yamlsecret.yamlxtrabackup-job.yaml

Helm ChartsHelm - The package manager for Kubernetes

● Separate configuration from logic

To continue “Kubernetes is an Operating System” analogy -Helm chart is like RPM/DEB packages

Operator● Method of packaging, deploying and managing a

Kubernetes application● The most advanced way of the deployment● Requires software engineering efforts

● A way to think about “Operator” is it’s goal is to replace a human in typical operations

● Assumes a high degree of automation

* image: RedHat OpenShift documentation

The current state of our work● The finished prototype: Helm charts● Planned work: Operator

Demo

I assume● You have Kubernetes running● The dynamic provision of persistent volumes is provided● Root inside containers is allowed (required for PMM)

Start with monitoring● PMM Server○ Attn!!! Requires running from root inside a container

■ OpenShift does not like it■ Not a recommended practice■ We are working to fix this

● helm install --name monitoring . -f values.yaml

$ kubectl get serviceNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEmonitoring-service LoadBalancer 10.11.248.149 35.193.173.250 80:30701/TCP,443:32249/TCP 50s

You can try https://35.193.173.250 (will be down soon after webinar ends)

Let’s start our cluster$ helm install --name cluster1 . -f values.yaml

Will start: ● 3 nodes of Percona XtraDB Cluster, each with PMM Client● ProxySQL instance

Configure ProxySQL to route traffic

$ kubectl get pods NAME READY STATUS RESTARTS AGE cluster1-node-0 2/2 Running 0 5m cluster1-node-1 2/2 Running 0 4m cluster1-node-2 2/2 Running 0 2m cluster1-proxysql-0 2/2 Running 0 5m monitoring-0 1/1 Running 0 16m

Let’s take a minute hereWith a single command we

● Started multiple MySQL nodes (and Kubernetes will handle a node failure)

● Started and configured ProxySQL● Added all nodes under PMM Server

What’s so big deal about it?

● One can do it with reasonably simple Ansible playbooks?

Kubernetes is Cloud-agnostic; provides an uniform API; “write once - run anywhere” approach

Demo / Howto - pods

$ kubectl get pods

NAME READY STATUS RESTARTS cluster1-proxysql-0 2/2 Running 0 cluster1-node-0 2/2 Running 0 cluster1-node-1 2/2 Running 0

cluster1-node-2 2/2 Running 0

Demo / Howto

$ kubectl get services

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)

cluster1-proxysql ClusterIP 10.11.251.66 <none> 3306/TCP,6032/TCP

MySQL Port(for app) Proxy Admin Port

kubectl exec cluster1-proxysql-0 -it -- mysql -uadmin -padmin -h127.0.0.1 -P6032

mysql> select * from mysql_servers;+--------------+-----------+------+--------+---------+-------------+-----------------+---------------------+---------+----------------+---------+| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |+--------------+-----------+------+--------+---------+-------------+-----------------+---------------------+---------+----------------+---------+| 10 | 10.8.2.10 | 3306 | ONLINE | 1000000 | 0 | 1000 | 0 | 0 | 0 | WRITE || 11 | 10.8.1.6 | 3306 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | READ || 11 | 10.8.0.10 | 3306 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | READ |+--------------+-----------+------+--------+---------+-------------+-----------------+---------------------+---------+----------------+---------+

Let’s kill one pod…

NAME READY STATUS RESTARTS AGEcluster1-node-0 0/2 Terminating 0 41mcluster1-node-1 2/2 Running 0 41mcluster1-node-2 2/2 Running 0 40mcluster1-proxysql-0 2/2 Running 0 41mmonitoring-0 1/1 Running 0 35m

Kubernetes will restart itmysql> select * from mysql_servers;+--------------+-----------+------+--------------+---------+-------------+-----------------+---------------------+---------+----------------+---------+| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |+--------------+-----------+------+--------------+---------+-------------+-----------------+---------------------+---------+----------------+---------+| 11 | 10.8.2.10 | 3306 | OFFLINE_HARD | 1000 | 0 | 1000 | 0 | 0 | 0 | READ || 11 | 10.8.1.6 | 3306 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | READ || 10 | 10.8.0.10 | 3306 | ONLINE | 1000000 | 0 | 1000 | 0 | 0 | 0 | WRITE || 11 | 10.8.2.12 | 3306 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | READ |+--------------+-----------+------+--------------+---------+-------------+-----------------+---------------------+---------+----------------+---------+

Let’s scale it upkubectl scale --replicas=4 statefulset/cluster1-nodestatefulset "cluster1-node" scaled

mysql> select * from mysql_servers;+--------------+-----------+------+--------------+---------+-------------+-----------------+---------------------+---------+----------------+---------+| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |+--------------+-----------+------+--------------+---------+-------------+-----------------+---------------------+---------+----------------+---------+| 11 | 10.8.2.10 | 3306 | OFFLINE_HARD | 1000 | 0 | 1000 | 0 | 0 | 0 | READ || 11 | 10.8.1.6 | 3306 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | READ || 10 | 10.8.0.10 | 3306 | ONLINE | 1000000 | 0 | 1000 | 0 | 0 | 0 | WRITE || 11 | 10.8.2.12 | 3306 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | READ || 11 | 10.8.1.7 | 3306 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | READ |+--------------+-----------+------+--------------+---------+-------------+-----------------+---------------------+---------+----------------+---------+

The Road Ahead● Operators● Encryptions○ At Rest○ End-to-end

● Key Management

Additional resources● Oracle’s MySQL Operator (Alpha quality)

https://github.com/oracle/mysql-operator● Presslab’s Operator

https://github.com/presslabs/mysql-operator

Thank you!