Skip to content

Operator

In this tutorial, we will first understand how an operator works, then we will try to use the python client of kubernetes to list and watch our customize resource Nginx. And finally, we will build a local operator to create deployments for each of our Nginx resource.

1. Concept

sequenceDiagram;
    participant User;
    participant API Server;
    Operator-->>API Server: Watch custom resources' changes;
    Note right of Operator: Perform actions <br>if resources are changed;
    User->>API Server: Create a custom resource;
    API Server->>Operator: New custom resource created;
    Operator-->>API Server: Create corresponding resources,<br> e.g. pods, if applicable;
    User->>API Server: Update a custom resource;
    API Server->>Operator: Custom resource updated;
    Operator-->>API Server: Update corresponding resources,<br> e.g. pods, if applicable;
    User->>API Server: Delete a custom resource;
    API Server->>Operator: Custom resource deleted;
    Operator-->>API Server: Delete corresponding resources,<br> e.g. pods, if applicable;

An operator:

  • watches the changes of the custom resources
    • create, update, delete
  • preforms some actions if the custom resources are changed

2. Prepare

  • k8s, kubectl
  • CRD created in the tutorial CRD
  • python3

There is already a good enough operator sdk out there, but we won't use it.

Let's build an operator in python from scratch, and in this way, you can fully understand the underneath mechanism.

3. Use kubernetes client to list pods locally

Let's warmup a bit and get familiar with list function provided by the python client of kubernetes.

Script

Save the following codes into list_pods.py.

from kubernetes import client, config


if __name__ == '__main__':
    # load config
    config.load_kube_config()
    # create a V1API instance
    v1 = client.CoreV1Api()
    # list pods in all namespaces
    pods = v1.list_pod_for_all_namespaces(watch=False)
    # print out the pods' info
    print("{:20}{:40}{:20}".format('Namespace', 'Pod', 'Start Time'))
    for pod in pods.items:
        print("{:20}{:40}{:20}".format(
            pod.metadata.namespace,
            pod.metadata.name,
            pod.status.start_time.strftime('%Y-%m-%d:%H:%M:%S')))

python list_pods.py

Namespace           Pod                                     Start Time
container-registry  registry-d7d7c8bc9-ltgx9                2019-12-23:08:28:43
default             busybox-sleep                           2020-03-02:07:22:03
kube-system         coredns-9b8997588-hvhhq                 2019-12-23:08:19:48
kube-system         hostpath-provisioner-7b9cb5cdb4-nk8tf   2019-12-23:08:18:47

Tips

We will not discuss how the client's codes work here. You can find the explanations in another article Client - Python

4. List Nginx Locally

Now, let's use the CustomObjectsApi to list our custom resource Nginx created in the previous tutorial CRD.

Script

Save the following codes into list_nginx.py.

from kubernetes import client, config


if __name__ == '__main__':
    # load config
    config.load_kube_config()
    # create a custom resource api instance
    crapi = client.CustomObjectsApi()
    # list all the nginx
    nginx = crapi.list_cluster_custom_object('stable.underneathall.com', 'v1', 'nginx')
    # print out the nginx's info
    print("{:20}{:40}{:20}".format('Namespace', 'Pod', 'Start Time'))
    for ngx in nginx.get('items', []):
        print("{:20}{:40}{:20}".format(
            ngx['metadata']['namespace'],
            ngx['metadata']['name'],
            ngx['metadata']['creationTimestamp']))

python list_nginx.py

Namespace           Pod                                     Start Time
default             underneathall-nginx                     2021-01-15T09:01:29Z

5. Watch Nginx Locally

5.1 Start Watch

Script

Save the following codes into watch_nginx.py.

import datetime
from kubernetes import client, config, watch


if __name__ == '__main__':
    # load config
    config.load_kube_config()
    # create a custom resource api instance
    crapi = client.CustomObjectsApi()
    # create a kubernetes watch instance
    watch = watch.Watch()
    # initialize the count no.
    i = 0
    # list custom resource function
    list_cr = client.CustomObjectsApi().list_cluster_custom_object
    # watch the stream and print out the events' info
    print("{:5}{:10}{:20}{:30}".format('No.', 'Type', 'Receive Time', 'Name'))
    for e in watch.stream(list_cr, 'stable.underneathall.com', 'v1', 'nginx'):
        i += 1
        print("{:<5}{:<10}{:<20}{:<30}".format(
            i,
            e['type'],
            datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
            e['object']['metadata']['name']))

python watch_nginx.py

The process will not exit, and will continue to watch the resources. 

No.  Type      Receive Time        Name
1    ADDED     2021-01-16 20:03:21 underneathall-nginx

5.2 Create a New Nginx

Now, let's not kill the watch process. Launch another shell and create a new nginx custom resource.

New Nginx-1

Run the following command to save the yaml into nginx-1.yaml

cat <<NGINX> nginx-1.yaml
apiVersion: "stable.underneathall.com/v1"
kind: Nginx
metadata:
  name: underneathall-nginx-1
  namespace: default
spec:
  replicas: 3
NGINX

Run: 

kubectl apply -f nginx-1.yaml

Output: 

nginx.stable.underneathall.com/underneathall-nginx-1 created

Added Event

Then, let's switch back to the watch process, and you will see the output similar to below, the new nginx-1 is received.

No.  Type      Receive Time        Name
1    ADDED     2021-01-16 20:03:21 underneathall-nginx
2    ADDED     2021-01-16 20:07:44 underneathall-nginx-1

5.3 Update Nginx

Update Nginx-1

Let's change the replicas in nginx-1.yaml to 1.

cat <<NGINX> nginx-1.yaml
apiVersion: "stable.underneathall.com/v1"
kind: Nginx
metadata:
  name: underneathall-nginx-1
  namespace: default
spec:
  replicas: 1
NGINX

Run: 

kubectl apply -f nginx-1.yaml

Output: 

nginx.stable.underneathall.com/underneathall-nginx-1 configured

Modified Event

Then, let's switch back to the watch process, and you will see the output similar to below, the update event of nginx-1 is received.

No.  Type      Receive Time        Name
1    ADDED     2021-01-16 20:03:21 underneathall-nginx
2    ADDED     2021-01-16 20:07:44 underneathall-nginx-1
3    MODIFIED  2021-01-16 20:10:39 underneathall-nginx-1

5.3 Delete Nginx

Delete Nginx-1

Run: 

kubectl delete -f nginx-1.yaml

Output: 

nginx.stable.underneathall.com "underneathall-nginx-1" deleted

Deleted Event

Then, let's switch back to the watch process, and you will see the output similar to below, the deletion event of nginx-1 is received.

No.  Type      Receive Time        Name
1    ADDED     2021-01-16 20:03:21 underneathall-nginx
2    ADDED     2021-01-16 20:07:44 underneathall-nginx-1
3    MODIFIED  2021-01-16 20:10:39 underneathall-nginx-1
4    DELETED   2021-01-16 20:12:19 underneathall-nginx-1

6. Operator

6.1 Codes

Now let's add a new file nginx.py to define a class Nginx to help us manage the events.

Nginx

from kubernetes import client


class Nginx:
    def __init__(self, name, namespace, **kwargs):
        self.name = name
        self.deploy_name = f'{name}-deploy'
        self.namespace = namespace
        self.kwargs = kwargs

    @property
    def api_client(self):
        if not getattr(self, '_api_client', None):
            self._api_client = client.AppsV1Api()
        return self._api_client

    @property
    def label_selector(self):
        return f'underneathall-app = nginx-{self.name}'

    @property
    def labels(self):
        return {
            'underneathall-app': f'nginx-{self.name}',
            'app': 'underneathall'
        }

    @property
    def body(self):
        return {
            'kind': 'Deployment',
            'apiVersion': 'apps/v1',
            'metadata': {
                'labels': self.labels,
                'namespace': self.namespace,
                'name': self.deploy_name
            },
            'spec': {
                'selector': {
                    'matchLabels': self.labels
                },
                'replicas': self.kwargs.get('replicas', 1),
                'template': {
                    'metadata': {
                        'labels': self.labels
                    },
                    'spec': {
                        'containers': [
                            {
                                'image': 'nginx:1.7.9',
                                'name': 'nginx',
                                'ports': [
                                    {'containerPort': 80}
                                ]
                            }
                        ]
                    }
                }
            }
        }

    @property
    def exists(self):
        if not getattr(self, '_deploy', None):
            deploys = self.api_client.list_namespaced_deployment(
                namespace=self.namespace,
                label_selector=self.label_selector)
            if len(deploys.items) == 0:
                return False
            self._deploy = deploys.items[0]
        return True

    def sync(self, event_type="ADDED"):
        event_type = event_type.upper()
        if event_type not in ["ADDED", "MODIFIED", "DELETED"]:
            print(f'Invalid event type received: {event_type}')
            return False
        if event_type in ["ADDED", "MODIFIED"] and not self.exists:
            ret = self.api_client.create_namespaced_deployment(
                namespace=self.namespace,
                body=self.body
            )
        elif event_type == "MODIFIED":
            ret = self.api_client.replace_namespaced_deployment(
                name=self.deploy_name,
                namespace=self.namespace,
                body=self.body
            )
        elif event_type == "DELETED" and self.exists:
            ret = self.api_client.delete_namespaced_deployment(
                name=self.deploy_name,
                namespace=self.namespace
            )
        return True

class Nginx

We will look at the codes step by step in a later section Operator Explained.

Now, change a little bit of our watch-nginx.py, add a handle_event function and call it during watch.

Save it to a new file operator.py

Operator Entrypoint

import datetime
from kubernetes import client, config, watch
from nginx import Nginx


# load config
config.load_kube_config()


def handle_event(event):
    nginx = Nginx(
        name=event['object']['metadata']['name'],
        namespace=event['object']['metadata']['namespace'],
        replicas=event['object']['spec']['replicas']
    )
    nginx.sync(event_type=event['type'])


if __name__ == '__main__':
    # create a custom resource api instance
    crapi = client.CustomObjectsApi()
    # create a kubernetes watch instance
    watch = watch.Watch()
    # initialize the count no.
    i = 0
    # list custom resource function
    list_cr = client.CustomObjectsApi().list_cluster_custom_object
    # watch the stream and print out the events' info
    print("{:5}{:10}{:20}{:30}".format('No.', 'Type', 'Receive Time', 'Name'))
    for e in watch.stream(list_cr, 'stable.underneathall.com', 'v1', 'nginx'):
        i += 1
        print("{:<5}{:<10}{:<20}{:<30}".format(
            i,
            e['type'],
            datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
            e['object']['metadata']['name']))
        handle_event(e)

6.2 Run the Operator

First, let's check current environment.

Current Status

There should be one nginx we've created.

Run

kubectl get nginx

Output 

NAME                  AGE
underneathall-nginx   7s

There should be no deployments.

Run

kubectl get deploy

Output 

No resources found in default namespace.

There should be no pods.

Run

kubectl get pod

Output 

No resources found in default namespace.

Now let's start our operator program.

Start Operator

python operator.py

No.  Type      Receive Time        Name                          
1    ADDED     2021-01-17 17:03:38 underneathall-nginx

Now, let's check again our resource.

Current Status

There should be one nginx we've created.

Run

kubectl get nginx

Output 

NAME                  AGE
underneathall-nginx    1m

There should be one 2-replica deployment.

Run

kubectl get deploy

Output 

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
underneathall-nginx-deploy   2/2     2            2           81s

There should be two pods running.

Run

kubectl get pod

Output 

NAME                                         READY   STATUS    RESTARTS   AGE
underneathall-nginx-deploy-8c56bb567-gtjm6   1/1     Running   0          5m25s
underneathall-nginx-deploy-8c56bb567-xlhcg   1/1     Running   0          5m25s

6.3 Create a New Nginx

Now, let's not kill the process. Launch another shell and create a new nginx custom resource.

New Nginx-1

Run the following command to save the yaml into nginx-1.yaml

cat <<NGINX> nginx-1.yaml
apiVersion: "stable.underneathall.com/v1"
kind: Nginx
metadata:
  name: underneathall-nginx-1
  namespace: default
spec:
  replicas: 3
NGINX

Run:

kubectl apply -f nginx-1.yaml

Output:

nginx.stable.underneathall.com/underneathall-nginx-1 created

Added Event

Then, let's switch back to the watch process, and you will see the output similar to below, the new nginx-1 is received.

No.  Type      Receive Time        Name                          
1    ADDED     2021-01-17 17:03:38 underneathall-nginx
2    ADDED     2021-01-17 17:10:26 underneathall-nginx-1

There should be two nginx we've created.

Run

kubectl get nginx

Output

NAME                  AGE
underneathall-nginx    8m
underneathall-nginx-1  5s

There should be another 3-replica deployment.

Run

kubectl get deploy

Output

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
underneathall-nginx-deploy     2/2     2            2           7m31s
underneathall-nginx-1-deploy   3/3     3            3           43s

There should be totally 5 pods running.

Run

kubectl get pod

Output 

NAME                                            READY   STATUS    RESTARTS   AGE
underneathall-nginx-deploy-8c56bb567-gtjm6      1/1     Running   0          8m19s
underneathall-nginx-deploy-8c56bb567-xlhcg      1/1     Running   0          8m19s
underneathall-nginx-1-deploy-75ddffcbbf-zr7vm   1/1     Running   0          91s
underneathall-nginx-1-deploy-75ddffcbbf-4tvdb   1/1     Running   0          91s
underneathall-nginx-1-deploy-75ddffcbbf-mfmmf   1/1     Running   0          91s

6.4 Update Nginx

Update Nginx-1

Let's change the replicas in nginx-1.yaml to 1.

cat <<NGINX> nginx-1.yaml
apiVersion: "stable.underneathall.com/v1"
kind: Nginx
metadata:
  name: underneathall-nginx-1
  namespace: default
spec:
  replicas: 1
NGINX

Run: 

kubectl apply -f nginx-1.yaml

Output: 

nginx.stable.underneathall.com/underneathall-nginx-1 configured

Modified Event

Then switch back to the watch process, and you will see the output similar to below, the update of nginx-1 is received.

No.  Type      Receive Time        Name                          
1    ADDED     2021-01-17 17:03:38 underneathall-nginx
2    ADDED     2021-01-17 17:10:26 underneathall-nginx-1
3    MODIFIED  2021-01-17 17:13:25 underneathall-nginx-1

There should be two nginx we've created.

Run

kubectl get nginx

Output 

NAME                  AGE
underneathall-nginx   11m
underneathall-nginx-1  3m

underneathall-nginx-1-deploy should be a 1-replica deployment now.

Run

kubectl get deploy

Output 

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
underneathall-nginx-deploy     2/2     2            2           10m
underneathall-nginx-1-deploy   1/1     1            1           3m42s

There should be totally 3 pods running now.

Run

kubectl get pod

Output 

NAME                                            READY   STATUS    RESTARTS   AGE
underneathall-nginx-deploy-8c56bb567-gtjm6      1/1     Running   0          11m
underneathall-nginx-deploy-8c56bb567-xlhcg      1/1     Running   0          11m
underneathall-nginx-1-deploy-75ddffcbbf-zr7vm   1/1     Running   0          4m28s

6.4 Delete Nginx

Delete Nginx-1

Run: 

kubectl delete -f nginx-1.yaml

Output: 

nginx.stable.underneathall.com "underneathall-nginx-1" deleted

Deleted Event

Then, let's switch back to the watch process, and you will see the output similar to below, the deletion event of nginx-1 is received.

No.  Type      Receive Time        Name                          
1    ADDED     2021-01-17 17:03:38 underneathall-nginx
2    ADDED     2021-01-17 17:10:26 underneathall-nginx-1
3    MODIFIED  2021-01-17 17:13:25 underneathall-nginx-1
4    DELETED   2021-01-17 17:16:45 underneathall-nginx-1

There should be only one nginx.

Run

kubectl get nginx

Output 

NAME                  AGE
underneathall-nginx   15m

There should only be one deployment now.

Run

kubectl get deploy

Output 

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
underneathall-nginx-deploy   2/2     2            2           13m

There should be totally 3 pods running now.

Run

kubectl get pod

Output 

NAME                                         READY   STATUS    RESTARTS   AGE
underneathall-nginx-deploy-8c56bb567-gtjm6   1/1     Running   0          15m
underneathall-nginx-deploy-8c56bb567-xlhcg   1/1     Running   0          15m

7. Summary

Great! You have suceessfully create an operator running in local.

Next you can