Application Deployment using k8s Operator on a Tanzu Kubernetes Cluster

What is Operator ?

Operators makes it easy to manage complex stateful applications on top of Kubernetes. Operator Framework behind the scene does this for us. > To know more about Operator framework, Refer https://github.com/operator-framework

Operator SDK is component of Operator Framework that manages Kubernetes native application, called operator. > Operators can be created using

  1. Go
  2. Ansible
  3. Helm

If you are planning to create an operator, Refer below picture to understand the operator capability based on what is used to create an operator.

Operator capability level

In this blog post, I will talk about setting up operator sdk, then creating a sample ansible based operator and deploying it on a Tanzu Kuberentes Cluster.


What You will learn:-

  1. Installing an Operator SDK
  2. Creating a memcached operator based on ansible
  3. Deploying memcached operator on a Tanzu Kubernetes Cluster

1. Installing an operator SDK

I will be demonstrating the steps for MAC, You can follow the steps based on your operating system.

# Install operator-sdk

$ brew install operator-sdk

Once installed, verify the operator-sdk version.

$ operator-sdk version

operator-sdk version: "v1.13.0", commit: "6e84414b468029c5c3e07352cabe64cf3a682111", kubernetes version: "v1.21", go version: "go1.17.1", GOOS: "darwin", GOARCH: "amd64"

If you are on another operating system, Refer this link for installing operator-sdk

2. Creating a memcached operator based on ansible

  • Create a new project
$ mkdir memcached-operator

$ ls memcached-operator

  • Go to the newly created directory
$ cd memcached-operator
  • Initialize a memcached-operator project
$ operator-sdk init --plugins=ansible --domain dineshtripathi30
Writing kustomize manifests for you to edit...
Next: define a resource with:
$ operator-sdk create api
  • See, what is created. you will see different files. Explore them.
$ ls 
Dockerfile       PROJECT          molecule         requirements.yml watches.yaml
Makefile         config           playbooks        roles
  • Next, create a memcached api
$ operator-sdk create api --group cache --version v1alpha1 --kind Memcached --generate-role
  • Above step will create memcached api and sample files inside ansible role that we can modify to install memcached.
  • Validate the content created inside memcached-operator project. Explore them. > Note: We have only created the basic content and not yet added a logic to install memcached.
$ tree .                                                
.
├── Dockerfile
├── Makefile
├── PROJECT
├── config
│   ├── crd
│   │   ├── bases
│   │   │   └── cache.dineshtripathi30_memcacheds.yaml
│   │   └── kustomization.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── manifests
│   │   └── kustomization.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── memcached_editor_role.yaml
│   │   ├── memcached_viewer_role.yaml
│   │   ├── role.yaml
│   │   ├── role_binding.yaml
│   │   └── service_account.yaml
│   ├── samples
│   │   ├── cache_v1alpha1_memcached.yaml
│   │   └── kustomization.yaml
│   ├── scorecard
│   │   ├── bases
│   │   │   └── config.yaml
│   │   ├── kustomization.yaml
│   │   └── patches
│   │       ├── basic.config.yaml
│   │       └── olm.config.yaml
│   └── testing
│       ├── debug_logs_patch.yaml
│       ├── kustomization.yaml
│       ├── manager_image.yaml
│       └── pull_policy
│           ├── Always.yaml
│           ├── IfNotPresent.yaml
│           └── Never.yaml
├── molecule
│   ├── default
│   │   ├── converge.yml
│   │   ├── create.yml
│   │   ├── destroy.yml
│   │   ├── kustomize.yml
│   │   ├── molecule.yml
│   │   ├── prepare.yml
│   │   ├── tasks
│   │   │   └── memcached_test.yml
│   │   └── verify.yml
│   └── kind
│       ├── converge.yml
│       ├── create.yml
│       ├── destroy.yml
│       └── molecule.yml
├── playbooks
├── requirements.yml
├── roles
│   └── memcached
│       ├── README.md
│       ├── defaults
│       │   └── main.yml
│       ├── files
│       ├── handlers
│       │   └── main.yml
│       ├── meta
│       │   └── main.yml
│       ├── tasks
│       │   └── main.yml
│       ├── templates
│       └── vars
│           └── main.yml
└── watches.yaml

28 directories, 58 files
  • We will deploy this operator now (first without adding a logic to install memcached). This step is just needed to understand how the CRD’s will be created on a Tanzu Kubernetes Cluster.
  • So to create the container image of operator we just created, Lets build the image and push it to container registry. In this case, I am using my own dockerhub.
$ make docker-build docker-push IMG="dineshtripathi30/memcached-operator:v1.0.0"
docker build -t dineshtripathi30/memcached-operator:v1.0.0 .
[+] Building 25.0s (11/11) FINISHED                                                                                                         
 => [internal] load build definition from Dockerfile                                                                                   0.1s
 => => transferring dockerfile: 356B                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                      0.0s
 => => transferring context: 2B                                                                                                        0.0s
 => [internal] load metadata for quay.io/operator-framework/ansible-operator:v1.13.0                                                   5.5s
 => [1/6] FROM quay.io/operator-framework/ansible-operator:v1.13.0@sha256:e72c55158f66bb0db216028ea38e53d54ce5c0280e13e0cdab355931fae  9.2s
 => => resolve quay.io/operator-framework/ansible-operator:v1.13.0@sha256:e72c55158f66bb0db216028ea38e53d54ce5c0280e13e0cdab355931fae  0.0s
 => => sha256:e72c55158f66bb0db216028ea38e53d54ce5c0280e13e0cdab355931faeb376e 1.36kB / 1.36kB                                         0.0s
 => => sha256:44f5fb7008c71336fb13f94a4f51367ed9c2632d003bd0223cb4666ae70b7022 1.99kB / 1.99kB                                         0.0s
 => => sha256:efe5815f59af1035b0ea46905d221ed409d1bf224783d138579ca87a1feeb61b 6.44kB / 6.44kB                                         0.0s
 => => sha256:9a8a5acd1e64898f0d2cb3f6a127cc9ddc4f4788d2cf2fc73400b8fcbb48dc07 21.41MB / 21.41MB                                       6.7s
 => => sha256:8506e556f852044dafb679876de54d37876d56add6306959caac72acf24b2d5d 581B / 581B                                             1.9s
 => => extracting sha256:8506e556f852044dafb679876de54d37876d56add6306959caac72acf24b2d5d                                              0.0s
 => => extracting sha256:9a8a5acd1e64898f0d2cb3f6a127cc9ddc4f4788d2cf2fc73400b8fcbb48dc07                                              1.9s
 => [internal] load build context                                                                                                      0.2s
 => => transferring context: 4.82kB                                                                                                    0.1s
 => [2/6] COPY requirements.yml /opt/ansible/requirements.yml                                                                          0.2s
 => [3/6] RUN ansible-galaxy collection install -r /opt/ansible/requirements.yml  && chmod -R ug+rwx /opt/ansible/.ansible             9.3s
 => [4/6] COPY watches.yaml /opt/ansible/watches.yaml                                                                                  0.1s
 => [5/6] COPY roles/ /opt/ansible/roles/                                                                                              0.0s 
 => [6/6] COPY playbooks/ /opt/ansible/playbooks/                                                                                      0.0s
 => exporting to image                                                                                                                 0.2s
 => => exporting layers                                                                                                                0.1s
 => => writing image sha256:1949f3126c1e8572862f06b33f32303d794f5d34249ab978de1b4e03be0f2b3c                                           0.0s
 => => naming to docker.io/dineshtripathi30/memcached-operator:v1.0.0                                                                  0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
docker push dineshtripathi30/memcached-operator:v1.0.0
The push refers to repository [docker.io/dineshtripathi30/memcached-operator]
c96a2a8b059e: Pushed 
62d1cf1c30dd: Pushed 
5e335fe5e034: Pushed 
e2e2d56f391c: Pushed 
ab2f333ea343: Pushed 
af8378298237: Pushed 
b04f8ce435c4: Pushed 
047f99f97b37: Layer already exists 
a1fc8c6dd0aa: Layer already exists 
e28bbcda30e0: Layer already exists 
8a4a82f5e7f8: Layer already exists 
47f871cfc125: Layer already exists 
173357979467: Layer already exists 
v1.0.0: digest: sha256:9e833c305cf7a435acd717c92cc6341570e58d219e2bf03c0dfc2f05785dd1bb size: 3031

Deploy the operator on a Tanzu Kubernetes Cluster.

> Note: Ensure you have cluster-admin permission > Also, Ensure that kubectl command is working from where you are going to run below command.

$ make deploy IMG="dineshtripathi30/memcached-operator:v1.0.0"
cd config/manager && /Users/dinetrip/operator/memcached-operator/bin/kustomize edit set image controller=dineshtripathi30/memcached-operator:v1.0.0
/Users/dinetrip/operator/memcached-operator/bin/kustomize build config/default | kubectl apply -f -
namespace/memcached-operator-system created
customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.dineshtripathi30 unchanged
serviceaccount/memcached-operator-controller-manager created
role.rbac.authorization.k8s.io/memcached-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/memcached-operator-manager-role unchanged
clusterrole.rbac.authorization.k8s.io/memcached-operator-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/memcached-operator-proxy-role unchanged
rolebinding.rbac.authorization.k8s.io/memcached-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-manager-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-proxy-rolebinding unchanged
configmap/memcached-operator-manager-config created
service/memcached-operator-controller-manager-metrics-service created
deployment.apps/memcached-operator-controller-manager created
  • Lets validate what is done on TKG Cluster. You will find that the new namespace called memcached-operator-system is created.
$ k get all -n memcached-operator-system           
NAME                                                         READY   STATUS    RESTARTS   AGE
pod/memcached-operator-controller-manager-55cd4b99df-btd95   2/2     Running   0          84s

NAME                                                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/memcached-operator-controller-manager-metrics-service   ClusterIP   10.107.165.109           8443/TCP   84s

NAME                                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/memcached-operator-controller-manager   1/1     1            1           84s

NAME                                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/memcached-operator-controller-manager-55cd4b99df   1         1         1       84s

NAME                                                                  SHORT NAME    DUCKS   READY   REASON
clusterducktype.discovery.knative.dev/addressables.duck.knative.dev   Addressable                   
clusterducktype.discovery.knative.dev/bindings.duck.knative.dev       Binding                       
clusterducktype.discovery.knative.dev/channelables.duck.knative.dev   Channelable                   
clusterducktype.discovery.knative.dev/podspecables.duck.knative.dev   PodSpecable                   
clusterducktype.discovery.knative.dev/sources.duck.knative.dev        Source          
  • See the CRD created
$ k get crd | grep -i memcached
memcacheds.cache.dineshtripathi30                        2021-10-08T08:54:47Z
  • Now, If you try to validate if there any instance of memcached. You will find nothing as we have not yet created.
$ k get memcached                    
No resources found in default namespace.

In this section, We will define how to deploy memcached.

Let’s start.

  • Modify the file roles/memcached/tasks/main.yml and add the manifest to deploy memcached.
---
- name: start memcached
  community.kubernetes.k8s:
    definition:
      kind: Deployment
      apiVersion: apps/v1
      metadata:
        name: '{{ ansible_operator_meta.name }}-memcached'
        namespace: '{{ ansible_operator_meta.namespace }}'
      spec:
        replicas: "{{size}}"
        selector:
          matchLabels:
            app: memcached
        template:
          metadata:
            labels:
              app: memcached
          spec:
            containers:
            - name: memcached
              command:
              - memcached
              - -m=64
              - -o
              - modern
              - -v
              image: "docker.io/memcached:1.4.36-alpine"
              ports:
                - containerPort: 11211

> Note: > – This memcached role will: > > Ensure a memcached Deployment exists > Set the Deployment size

  • Now modify config/samples/cache_v1alpha1_memcached.yaml and specify the the number of instances for memcached.
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  size: 2
  • If you notice in above manifest file, We are mentioning kind: Memcached. This is possible because now operator already created a CRD that understands what is Memcached. > You can Validate by running kubectl command. > – $ kubectl get memcached

  • Lets build the image again, so that the changes made can be packaged and then deploy.

$ make docker-build docker-push IMG="dineshtripathi30/memcached-operator:v1.0.1"
docker build -t dineshtripathi30/memcached-operator:v1.0.1 .
[+] Building 1.5s (11/11) FINISHED                                                                                                                               
 => [internal] load build definition from Dockerfile                                                                                                        0.0s
 => => transferring dockerfile: 37B                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                           0.0s
 => => transferring context: 2B                                                                                                                             0.0s
 => [internal] load metadata for quay.io/operator-framework/ansible-operator:v1.13.0                                                                        1.2s
 => [1/6] FROM quay.io/operator-framework/ansible-operator:v1.13.0@sha256:e72c55158f66bb0db216028ea38e53d54ce5c0280e13e0cdab355931faeb376e                  0.0s
 => [internal] load build context                                                                                                                           0.0s
 => => transferring context: 928B                                                                                                                           0.0s
 => CACHED [2/6] COPY requirements.yml /opt/ansible/requirements.yml                                                                                        0.0s
 => CACHED [3/6] RUN ansible-galaxy collection install -r /opt/ansible/requirements.yml  && chmod -R ug+rwx /opt/ansible/.ansible                           0.0s
 => CACHED [4/6] COPY watches.yaml /opt/ansible/watches.yaml                                                                                                0.0s
 => CACHED [5/6] COPY roles/ /opt/ansible/roles/                                                                                                            0.0s
 => CACHED [6/6] COPY playbooks/ /opt/ansible/playbooks/                                                                                                    0.0s
 => exporting to image                                                                                                                                      0.0s
 => => exporting layers                                                                                                                                     0.0s
 => => writing image sha256:1949f3126c1e8572862f06b33f32303d794f5d34249ab978de1b4e03be0f2b3c                                                                0.0s
 => => naming to docker.io/dineshtripathi30/memcached-operator:v1.0.1                                                                                       0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
docker push dineshtripathi30/memcached-operator:v1.0.1
The push refers to repository [docker.io/dineshtripathi30/memcached-operator]
c96a2a8b059e: Layer already exists 
62d1cf1c30dd: Layer already exists 
5e335fe5e034: Layer already exists 
e2e2d56f391c: Layer already exists 
ab2f333ea343: Layer already exists 
af8378298237: Layer already exists 
b04f8ce435c4: Layer already exists 
047f99f97b37: Layer already exists 
a1fc8c6dd0aa: Layer already exists 
e28bbcda30e0: Layer already exists 
8a4a82f5e7f8: Layer already exists 
47f871cfc125: Layer already exists 
173357979467: Layer already exists 
v1.0.1: digest: sha256:9e833c305cf7a435acd717c92cc6341570e58d219e2bf03c0dfc2f05785dd1bb size: 3031
  • Now, Deploy the operator.
$ make deploy IMG="dineshtripathi30/memcached-operator:v1.0.1"                  
cd config/manager && /Users/dinetrip/operator/memcached-operator/bin/kustomize edit set image controller=dineshtripathi30/memcached-operator:v1.0.1
/Users/dinetrip/operator/memcached-operator/bin/kustomize build config/default | kubectl apply -f -
namespace/memcached-operator-system unchanged
customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.dineshtripathi30 unchanged
serviceaccount/memcached-operator-controller-manager unchanged
role.rbac.authorization.k8s.io/memcached-operator-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/memcached-operator-manager-role unchanged
clusterrole.rbac.authorization.k8s.io/memcached-operator-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/memcached-operator-proxy-role unchanged
rolebinding.rbac.authorization.k8s.io/memcached-operator-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-manager-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-proxy-rolebinding unchanged
configmap/memcached-operator-manager-config unchanged
service/memcached-operator-controller-manager-metrics-service unchanged
deployment.apps/memcached-operator-controller-manager configured
  • If you look at the output carefully, You will see that the deployment is configured now. Rest is unchanged.
  • Validate the deployment and memcached instance.
$ k get deploy
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
memcached-sample-memcached   2/2     2            2           10s
  • Validate pods
$ k get po
NAME                                         READY   STATUS    RESTARTS   AGE
memcached-sample-memcached-54946946b-7hnzz   1/1     Running   0          10s
memcached-sample-memcached-54946946b-k25lk   1/1     Running   0          10s

  • Now, Let validate the memcached crd
$ k get memcached
NAME               AGE
memcached-sample   10s

  • Describe memcached-sample to see more detail.
$ k describe memcached memcached-sample
Name:         memcached-sample
Namespace:    default
Labels:       
Annotations:  
API Version:  cache.dineshtripathi30/v1alpha1
Kind:         Memcached
Metadata:
  Creation Timestamp:  2021-10-08T07:54:10Z
  Generation:          2
  Managed Fields:
    API Version:  cache.dineshtripathi30/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:size:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2021-10-08T07:54:10Z
    API Version:  cache.dineshtripathi30/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
    Manager:         ansible-operator
    Operation:       Update
    Time:            2021-10-08T07:54:12Z
  Resource Version:  25438606
  UID:               1f25635d-635e-4134-a540-f23cea4488a2
Spec:
  Size:  2
Status:
  Conditions:
    Ansible Result:
      Changed:             0
      Completion:          2021-10-09T05:32:21.193601
      Failures:            0
      Ok:                  1
      Skipped:             0
    Last Transition Time:  2021-10-08T07:54:08Z
    Message:               Awaiting next reconciliation
    Reason:                Successful
    Status:                True
    Type:                  Running
Events:                    

> Basically, now memcached CRD is just another resource for kubernetes and you can try other commands like edit etc.

Editing memcached instance and create three pods.

$ k edit  memcached memcached-sample
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: cache.dineshtripathi30/v1alpha1
kind: Memcached
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cache.dineshtripathi30/v1alpha1","kind":"Memcached","metadata":{"annotations":{},"name":"memcached-sample","namespace":"default"},"spec":{"size":2}}
  creationTimestamp: "2021-10-08T07:54:10Z"
  generation: 2
  name: memcached-sample
  namespace: default
  resourceVersion: "25438606"
  uid: 1f25635d-635e-4134-a540-f23cea4488a2
spec:
  **size: 3**
status:
  conditions:
  - ansibleResult:
      changed: 0
      completion: 2021-10-09T05:32:21.193601
      failures: 0
      ok: 1
      skipped: 0
    lastTransitionTime: "2021-10-08T07:54:08Z"
    message: Awaiting next reconciliation
    reason: Successful
    status: "True"
    type: Running
  • Save and you will see that now there are three pods for memcached.
$ k edit  memcached memcached-sample
memcached.cache.dineshtripathi30/memcached-sample edited
$ k get pods
NAME                                         READY   STATUS    RESTARTS   AGE
memcached-sample-memcached-54946946b-2w275   1/1     Running   0          4s
memcached-sample-memcached-54946946b-7hnzz   1/1     Running   0          60s
memcached-sample-memcached-54946946b-k25lk   1/1     Running   0          60s

That’s all for this post. In next post, I will explain advance concept like OLM and its integration with operator to manage the lifecycle of deployed application.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s