What is Operator ?
Operators makes it easy to manage complex stateful applications on top of Kubernetes. Operator Framework behind the scene does this for us. > To know more about Operator framework, Refer https://github.com/operator-framework
Operator SDK is component of Operator Framework that manages Kubernetes native application, called operator. > Operators can be created using
- Go
- Ansible
- Helm
If you are planning to create an operator, Refer below picture to understand the operator capability based on what is used to create an operator.
Operator capability level
In this blog post, I will talk about setting up operator sdk, then creating a sample ansible based operator and deploying it on a Tanzu Kuberentes Cluster.
What You will learn:-
- Installing an Operator SDK
- Creating a memcached operator based on ansible
- Deploying memcached operator on a Tanzu Kubernetes Cluster
1. Installing an operator SDK
I will be demonstrating the steps for MAC, You can follow the steps based on your operating system.
# Install operator-sdk
$ brew install operator-sdk
Once installed, verify the operator-sdk version.
$ operator-sdk version
operator-sdk version: "v1.13.0", commit: "6e84414b468029c5c3e07352cabe64cf3a682111", kubernetes version: "v1.21", go version: "go1.17.1", GOOS: "darwin", GOARCH: "amd64"
If you are on another operating system, Refer this link for installing operator-sdk
2. Creating a memcached operator based on ansible
- Create a new project
$ mkdir memcached-operator
$ ls memcached-operator
- Go to the newly created directory
$ cd memcached-operator
- Initialize a memcached-operator project
$ operator-sdk init --plugins=ansible --domain dineshtripathi30
Writing kustomize manifests for you to edit...
Next: define a resource with:
$ operator-sdk create api
- See, what is created. you will see different files. Explore them.
$ ls
Dockerfile PROJECT molecule requirements.yml watches.yaml
Makefile config playbooks roles
- Next, create a memcached api
$ operator-sdk create api --group cache --version v1alpha1 --kind Memcached --generate-role
- Above step will create memcached api and sample files inside ansible role that we can modify to install memcached.
- Validate the content created inside memcached-operator project. Explore them. > Note: We have only created the basic content and not yet added a logic to install memcached.
$ tree .
.
├── Dockerfile
├── Makefile
├── PROJECT
├── config
│ ├── crd
│ │ ├── bases
│ │ │ └── cache.dineshtripathi30_memcacheds.yaml
│ │ └── kustomization.yaml
│ ├── default
│ │ ├── kustomization.yaml
│ │ ├── manager_auth_proxy_patch.yaml
│ │ └── manager_config_patch.yaml
│ ├── manager
│ │ ├── controller_manager_config.yaml
│ │ ├── kustomization.yaml
│ │ └── manager.yaml
│ ├── manifests
│ │ └── kustomization.yaml
│ ├── prometheus
│ │ ├── kustomization.yaml
│ │ └── monitor.yaml
│ ├── rbac
│ │ ├── auth_proxy_client_clusterrole.yaml
│ │ ├── auth_proxy_role.yaml
│ │ ├── auth_proxy_role_binding.yaml
│ │ ├── auth_proxy_service.yaml
│ │ ├── kustomization.yaml
│ │ ├── leader_election_role.yaml
│ │ ├── leader_election_role_binding.yaml
│ │ ├── memcached_editor_role.yaml
│ │ ├── memcached_viewer_role.yaml
│ │ ├── role.yaml
│ │ ├── role_binding.yaml
│ │ └── service_account.yaml
│ ├── samples
│ │ ├── cache_v1alpha1_memcached.yaml
│ │ └── kustomization.yaml
│ ├── scorecard
│ │ ├── bases
│ │ │ └── config.yaml
│ │ ├── kustomization.yaml
│ │ └── patches
│ │ ├── basic.config.yaml
│ │ └── olm.config.yaml
│ └── testing
│ ├── debug_logs_patch.yaml
│ ├── kustomization.yaml
│ ├── manager_image.yaml
│ └── pull_policy
│ ├── Always.yaml
│ ├── IfNotPresent.yaml
│ └── Never.yaml
├── molecule
│ ├── default
│ │ ├── converge.yml
│ │ ├── create.yml
│ │ ├── destroy.yml
│ │ ├── kustomize.yml
│ │ ├── molecule.yml
│ │ ├── prepare.yml
│ │ ├── tasks
│ │ │ └── memcached_test.yml
│ │ └── verify.yml
│ └── kind
│ ├── converge.yml
│ ├── create.yml
│ ├── destroy.yml
│ └── molecule.yml
├── playbooks
├── requirements.yml
├── roles
│ └── memcached
│ ├── README.md
│ ├── defaults
│ │ └── main.yml
│ ├── files
│ ├── handlers
│ │ └── main.yml
│ ├── meta
│ │ └── main.yml
│ ├── tasks
│ │ └── main.yml
│ ├── templates
│ └── vars
│ └── main.yml
└── watches.yaml
28 directories, 58 files
- We will deploy this operator now (first without adding a logic to install memcached). This step is just needed to understand how the CRD’s will be created on a Tanzu Kubernetes Cluster.
- So to create the container image of operator we just created, Lets build the image and push it to container registry. In this case, I am using my own dockerhub.
$ make docker-build docker-push IMG="dineshtripathi30/memcached-operator:v1.0.0"
docker build -t dineshtripathi30/memcached-operator:v1.0.0 .
[+] Building 25.0s (11/11) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 356B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for quay.io/operator-framework/ansible-operator:v1.13.0 5.5s
=> [1/6] FROM quay.io/operator-framework/ansible-operator:v1.13.0@sha256:e72c55158f66bb0db216028ea38e53d54ce5c0280e13e0cdab355931fae 9.2s
=> => resolve quay.io/operator-framework/ansible-operator:v1.13.0@sha256:e72c55158f66bb0db216028ea38e53d54ce5c0280e13e0cdab355931fae 0.0s
=> => sha256:e72c55158f66bb0db216028ea38e53d54ce5c0280e13e0cdab355931faeb376e 1.36kB / 1.36kB 0.0s
=> => sha256:44f5fb7008c71336fb13f94a4f51367ed9c2632d003bd0223cb4666ae70b7022 1.99kB / 1.99kB 0.0s
=> => sha256:efe5815f59af1035b0ea46905d221ed409d1bf224783d138579ca87a1feeb61b 6.44kB / 6.44kB 0.0s
=> => sha256:9a8a5acd1e64898f0d2cb3f6a127cc9ddc4f4788d2cf2fc73400b8fcbb48dc07 21.41MB / 21.41MB 6.7s
=> => sha256:8506e556f852044dafb679876de54d37876d56add6306959caac72acf24b2d5d 581B / 581B 1.9s
=> => extracting sha256:8506e556f852044dafb679876de54d37876d56add6306959caac72acf24b2d5d 0.0s
=> => extracting sha256:9a8a5acd1e64898f0d2cb3f6a127cc9ddc4f4788d2cf2fc73400b8fcbb48dc07 1.9s
=> [internal] load build context 0.2s
=> => transferring context: 4.82kB 0.1s
=> [2/6] COPY requirements.yml /opt/ansible/requirements.yml 0.2s
=> [3/6] RUN ansible-galaxy collection install -r /opt/ansible/requirements.yml && chmod -R ug+rwx /opt/ansible/.ansible 9.3s
=> [4/6] COPY watches.yaml /opt/ansible/watches.yaml 0.1s
=> [5/6] COPY roles/ /opt/ansible/roles/ 0.0s
=> [6/6] COPY playbooks/ /opt/ansible/playbooks/ 0.0s
=> exporting to image 0.2s
=> => exporting layers 0.1s
=> => writing image sha256:1949f3126c1e8572862f06b33f32303d794f5d34249ab978de1b4e03be0f2b3c 0.0s
=> => naming to docker.io/dineshtripathi30/memcached-operator:v1.0.0 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
docker push dineshtripathi30/memcached-operator:v1.0.0
The push refers to repository [docker.io/dineshtripathi30/memcached-operator]
c96a2a8b059e: Pushed
62d1cf1c30dd: Pushed
5e335fe5e034: Pushed
e2e2d56f391c: Pushed
ab2f333ea343: Pushed
af8378298237: Pushed
b04f8ce435c4: Pushed
047f99f97b37: Layer already exists
a1fc8c6dd0aa: Layer already exists
e28bbcda30e0: Layer already exists
8a4a82f5e7f8: Layer already exists
47f871cfc125: Layer already exists
173357979467: Layer already exists
v1.0.0: digest: sha256:9e833c305cf7a435acd717c92cc6341570e58d219e2bf03c0dfc2f05785dd1bb size: 3031
Deploy the operator on a Tanzu Kubernetes Cluster.
> Note: Ensure you have cluster-admin permission > Also, Ensure that kubectl command is working from where you are going to run below command.
$ make deploy IMG="dineshtripathi30/memcached-operator:v1.0.0"
cd config/manager && /Users/dinetrip/operator/memcached-operator/bin/kustomize edit set image controller=dineshtripathi30/memcached-operator:v1.0.0
/Users/dinetrip/operator/memcached-operator/bin/kustomize build config/default | kubectl apply -f -
namespace/memcached-operator-system created
customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.dineshtripathi30 unchanged
serviceaccount/memcached-operator-controller-manager created
role.rbac.authorization.k8s.io/memcached-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/memcached-operator-manager-role unchanged
clusterrole.rbac.authorization.k8s.io/memcached-operator-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/memcached-operator-proxy-role unchanged
rolebinding.rbac.authorization.k8s.io/memcached-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-manager-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-proxy-rolebinding unchanged
configmap/memcached-operator-manager-config created
service/memcached-operator-controller-manager-metrics-service created
deployment.apps/memcached-operator-controller-manager created
- Lets validate what is done on TKG Cluster. You will find that the new namespace called
memcached-operator-systemis created.
$ k get all -n memcached-operator-system
NAME READY STATUS RESTARTS AGE
pod/memcached-operator-controller-manager-55cd4b99df-btd95 2/2 Running 0 84s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/memcached-operator-controller-manager-metrics-service ClusterIP 10.107.165.109 8443/TCP 84s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/memcached-operator-controller-manager 1/1 1 1 84s
NAME DESIRED CURRENT READY AGE
replicaset.apps/memcached-operator-controller-manager-55cd4b99df 1 1 1 84s
NAME SHORT NAME DUCKS READY REASON
clusterducktype.discovery.knative.dev/addressables.duck.knative.dev Addressable
clusterducktype.discovery.knative.dev/bindings.duck.knative.dev Binding
clusterducktype.discovery.knative.dev/channelables.duck.knative.dev Channelable
clusterducktype.discovery.knative.dev/podspecables.duck.knative.dev PodSpecable
clusterducktype.discovery.knative.dev/sources.duck.knative.dev Source
- See the CRD created
$ k get crd | grep -i memcached
memcacheds.cache.dineshtripathi30 2021-10-08T08:54:47Z
- Now, If you try to validate if there any instance of memcached. You will find nothing as we have not yet created.
$ k get memcached
No resources found in default namespace.
In this section, We will define how to deploy memcached.
Let’s start.
- Modify the file
roles/memcached/tasks/main.ymland add the manifest to deploy memcached.
---
- name: start memcached
community.kubernetes.k8s:
definition:
kind: Deployment
apiVersion: apps/v1
metadata:
name: '{{ ansible_operator_meta.name }}-memcached'
namespace: '{{ ansible_operator_meta.namespace }}'
spec:
replicas: "{{size}}"
selector:
matchLabels:
app: memcached
template:
metadata:
labels:
app: memcached
spec:
containers:
- name: memcached
command:
- memcached
- -m=64
- -o
- modern
- -v
image: "docker.io/memcached:1.4.36-alpine"
ports:
- containerPort: 11211
> Note: > – This memcached role will: > > Ensure a memcached Deployment exists > Set the Deployment size
- Now modify
config/samples/cache_v1alpha1_memcached.yamland specify the the number of instances for memcached.
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
name: memcached-sample
spec:
size: 2
-
If you notice in above manifest file, We are mentioning
kind: Memcached. This is possible because now operator already created a CRD that understands what isMemcached. > You can Validate by running kubectl command. > – $ kubectl get memcached -
Lets build the image again, so that the changes made can be packaged and then deploy.
$ make docker-build docker-push IMG="dineshtripathi30/memcached-operator:v1.0.1"
docker build -t dineshtripathi30/memcached-operator:v1.0.1 .
[+] Building 1.5s (11/11) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for quay.io/operator-framework/ansible-operator:v1.13.0 1.2s
=> [1/6] FROM quay.io/operator-framework/ansible-operator:v1.13.0@sha256:e72c55158f66bb0db216028ea38e53d54ce5c0280e13e0cdab355931faeb376e 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 928B 0.0s
=> CACHED [2/6] COPY requirements.yml /opt/ansible/requirements.yml 0.0s
=> CACHED [3/6] RUN ansible-galaxy collection install -r /opt/ansible/requirements.yml && chmod -R ug+rwx /opt/ansible/.ansible 0.0s
=> CACHED [4/6] COPY watches.yaml /opt/ansible/watches.yaml 0.0s
=> CACHED [5/6] COPY roles/ /opt/ansible/roles/ 0.0s
=> CACHED [6/6] COPY playbooks/ /opt/ansible/playbooks/ 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:1949f3126c1e8572862f06b33f32303d794f5d34249ab978de1b4e03be0f2b3c 0.0s
=> => naming to docker.io/dineshtripathi30/memcached-operator:v1.0.1 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
docker push dineshtripathi30/memcached-operator:v1.0.1
The push refers to repository [docker.io/dineshtripathi30/memcached-operator]
c96a2a8b059e: Layer already exists
62d1cf1c30dd: Layer already exists
5e335fe5e034: Layer already exists
e2e2d56f391c: Layer already exists
ab2f333ea343: Layer already exists
af8378298237: Layer already exists
b04f8ce435c4: Layer already exists
047f99f97b37: Layer already exists
a1fc8c6dd0aa: Layer already exists
e28bbcda30e0: Layer already exists
8a4a82f5e7f8: Layer already exists
47f871cfc125: Layer already exists
173357979467: Layer already exists
v1.0.1: digest: sha256:9e833c305cf7a435acd717c92cc6341570e58d219e2bf03c0dfc2f05785dd1bb size: 3031
- Now, Deploy the operator.
$ make deploy IMG="dineshtripathi30/memcached-operator:v1.0.1"
cd config/manager && /Users/dinetrip/operator/memcached-operator/bin/kustomize edit set image controller=dineshtripathi30/memcached-operator:v1.0.1
/Users/dinetrip/operator/memcached-operator/bin/kustomize build config/default | kubectl apply -f -
namespace/memcached-operator-system unchanged
customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.dineshtripathi30 unchanged
serviceaccount/memcached-operator-controller-manager unchanged
role.rbac.authorization.k8s.io/memcached-operator-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/memcached-operator-manager-role unchanged
clusterrole.rbac.authorization.k8s.io/memcached-operator-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/memcached-operator-proxy-role unchanged
rolebinding.rbac.authorization.k8s.io/memcached-operator-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-manager-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-proxy-rolebinding unchanged
configmap/memcached-operator-manager-config unchanged
service/memcached-operator-controller-manager-metrics-service unchanged
deployment.apps/memcached-operator-controller-manager configured
- If you look at the output carefully, You will see that the deployment is configured now. Rest is unchanged.
- Validate the deployment and memcached instance.
$ k get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
memcached-sample-memcached 2/2 2 2 10s
- Validate pods
$ k get po
NAME READY STATUS RESTARTS AGE
memcached-sample-memcached-54946946b-7hnzz 1/1 Running 0 10s
memcached-sample-memcached-54946946b-k25lk 1/1 Running 0 10s
- Now, Let validate the memcached crd
$ k get memcached
NAME AGE
memcached-sample 10s
- Describe memcached-sample to see more detail.
$ k describe memcached memcached-sample
Name: memcached-sample
Namespace: default
Labels:
Annotations:
API Version: cache.dineshtripathi30/v1alpha1
Kind: Memcached
Metadata:
Creation Timestamp: 2021-10-08T07:54:10Z
Generation: 2
Managed Fields:
API Version: cache.dineshtripathi30/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:size:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-10-08T07:54:10Z
API Version: cache.dineshtripathi30/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
Manager: ansible-operator
Operation: Update
Time: 2021-10-08T07:54:12Z
Resource Version: 25438606
UID: 1f25635d-635e-4134-a540-f23cea4488a2
Spec:
Size: 2
Status:
Conditions:
Ansible Result:
Changed: 0
Completion: 2021-10-09T05:32:21.193601
Failures: 0
Ok: 1
Skipped: 0
Last Transition Time: 2021-10-08T07:54:08Z
Message: Awaiting next reconciliation
Reason: Successful
Status: True
Type: Running
Events:
> Basically, now memcached CRD is just another resource for kubernetes and you can try other commands like edit etc.
Editing memcached instance and create three pods.
$ k edit memcached memcached-sample
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: cache.dineshtripathi30/v1alpha1
kind: Memcached
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"cache.dineshtripathi30/v1alpha1","kind":"Memcached","metadata":{"annotations":{},"name":"memcached-sample","namespace":"default"},"spec":{"size":2}}
creationTimestamp: "2021-10-08T07:54:10Z"
generation: 2
name: memcached-sample
namespace: default
resourceVersion: "25438606"
uid: 1f25635d-635e-4134-a540-f23cea4488a2
spec:
**size: 3**
status:
conditions:
- ansibleResult:
changed: 0
completion: 2021-10-09T05:32:21.193601
failures: 0
ok: 1
skipped: 0
lastTransitionTime: "2021-10-08T07:54:08Z"
message: Awaiting next reconciliation
reason: Successful
status: "True"
type: Running
- Save and you will see that now there are three pods for memcached.
$ k edit memcached memcached-sample
memcached.cache.dineshtripathi30/memcached-sample edited
$ k get pods
NAME READY STATUS RESTARTS AGE
memcached-sample-memcached-54946946b-2w275 1/1 Running 0 4s
memcached-sample-memcached-54946946b-7hnzz 1/1 Running 0 60s
memcached-sample-memcached-54946946b-k25lk 1/1 Running 0 60s
That’s all for this post. In next post, I will explain advance concept like OLM and its integration with operator to manage the lifecycle of deployed application.
