Sonobuoy is an opensource tool that helps to check the k8s cluster conformance. You can run Sonobuoy utility on your laptop and perform the test or other option is through tanzu cli that is part of Tanzu Community Edition (TCE).
In this tutorial, I will walk you through the later option and will run the end to end test using Tanzu CLI against Tanzu Standalone cluster running on a laptop.
Pre-requirement
- Tanzu Standalone cluster is up and running, You can refer this post to setup the same: https://mappslearning.wordpress.com/2021/10/06/setting-up-tanzu-community-edition-tce-on-a-mac/
- Tanzu CLI is setup, Validate tanzu CLI by running below command
$ tanzu
Tanzu CLI
Usage:
tanzu [command]
Available command groups:
Admin
builder Build Tanzu components
Build
accelerator Manage accelerators in a Kubernetes cluster
apps Applications on Kubernetes
Run
cluster Kubernetes cluster operations
conformance Run Sonobuoy conformance tests against clusters
diagnostics Cluster diagnostics
imagepullsecret Manage image pull secret operations. Image pull secrets enable the package and package repository consumers to authenticate to private registries.
kubernetes-release Kubernetes release operations
management-cluster Kubernetes management cluster operations
package Tanzu package management
standalone-cluster Create clusters without a dedicated management cluster
System
completion Output shell completion code
config Configuration for the CLI
init Initialize the CLI
login Login to the platform
plugin Manage CLI plugins
update Update the CLI
version Version information
Flags:
-h, --help help for tanzu
Use "tanzu [command] --help" for more information about a command.
Logged in to tkgmgmtcluster
Running E2E Test
The Kubernetes end-to-end testing plugin (the e2e plugin) is used to run tests which are maintained by the upstream Kubernetes community.
We can run the conformance test just by running tanzu conformance run command but before we run, let’s validate what resources will be created on a cluster. Run below command:
$ tanzu conformance gen
---
apiVersion: v1
kind: Namespace
metadata:
name: sonobuoy
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
component: sonobuoy
name: sonobuoy-serviceaccount
namespace: sonobuoy
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
component: sonobuoy
namespace: sonobuoy
name: sonobuoy-serviceaccount-sonobuoy
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: sonobuoy-serviceaccount-sonobuoy
subjects:
- kind: ServiceAccount
name: sonobuoy-serviceaccount
namespace: sonobuoy
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
component: sonobuoy
namespace: sonobuoy
name: sonobuoy-serviceaccount-sonobuoy
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '/metrics'
- '/logs'
- '/logs/*'
verbs:
- 'get'
---
apiVersion: v1
data:
config.json: |
{"Description":"DEFAULT","UUID":"","Version":"v0.53.2","ResultsDir":"/tmp/sonobuoy","Resources":["apiservices","certificatesigningrequests","clusterrolebindings","clusterroles","componentstatuses","configmaps","controllerrevisions","cronjobs","customresourcedefinitions","daemonsets","deployments","endpoints","ingresses","jobs","leases","limitranges","mutatingwebhookconfigurations","namespaces","networkpolicies","nodes","persistentvolumeclaims","persistentvolumes","poddisruptionbudgets","pods","podlogs","podsecuritypolicies","podtemplates","priorityclasses","replicasets","replicationcontrollers","resourcequotas","rolebindings","roles","servergroups","serverversion","serviceaccounts","services","statefulsets","storageclasses","validatingwebhookconfigurations","volumeattachments"],"Filters":{"Namespaces":".*","LabelSelector":""},"Limits":{"PodLogs":{"Namespaces":"","SonobuoyNamespace":true,"FieldSelectors":[],"LabelSelector":"","Previous":false,"SinceSeconds":null,"SinceTime":null,"Timestamps":false,"TailLines":null,"LimitBytes":null,"LimitSize":"","LimitTime":""}},"QPS":30,"Burst":50,"Server":{"bindaddress":"0.0.0.0","bindport":8080,"advertiseaddress":"","timeoutseconds":21600},"Plugins":null,"PluginSearchPath":["./plugins.d","/etc/sonobuoy/plugins.d","~/sonobuoy/plugins.d"],"Namespace":"sonobuoy","WorkerImage":"sonobuoy/sonobuoy:v0.53.2","ImagePullPolicy":"IfNotPresent","ImagePullSecrets":"","ProgressUpdatesPort":"8099"}
kind: ConfigMap
metadata:
labels:
component: sonobuoy
name: sonobuoy-config-cm
namespace: sonobuoy
---
apiVersion: v1
data:
plugin-0.yaml: |
podSpec:
containers: []
nodeSelector:
kubernetes.io/os: linux
restartPolicy: Never
serviceAccountName: sonobuoy-serviceaccount
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
- key: CriticalAddonsOnly
operator: Exists
- key: kubernetes.io/e2e-evict-taint-key
operator: Exists
sonobuoy-config:
driver: Job
plugin-name: e2e
result-format: junit
spec:
command:
- /run_e2e.sh
env:
- name: E2E_EXTRA_ARGS
value: --progress-report-url=http://localhost:8099/progress
- name: E2E_FOCUS
value: \[Conformance\]
- name: E2E_PARALLEL
value: "false"
- name: E2E_SKIP
value: \[Disruptive\]|NoExecuteTaintManager
- name: E2E_USE_GO_RUNNER
value: "true"
- name: SONOBUOY_K8S_VERSION
value: v1.21.2
image: k8s.gcr.io/conformance:v1.21.2
imagePullPolicy: IfNotPresent
name: e2e
resources: {}
volumeMounts:
- mountPath: /tmp/results
name: results
plugin-1.yaml: |
sonobuoy-config:
driver: DaemonSet
plugin-name: systemd-logs
result-format: raw
spec:
command:
- /bin/sh
- -c
- /get_systemd_logs.sh; while true; do echo "Plugin is complete. Sleeping indefinitely
to avoid container exit and automatic restarts from Kubernetes"; sleep 3600; done
env:
- name: CHROOT_DIR
value: /node
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: RESULTS_DIR
value: /tmp/results
- name: SONOBUOY_K8S_VERSION
value: v1.21.2
image: sonobuoy/systemd-logs:v0.3
imagePullPolicy: IfNotPresent
name: systemd-logs
resources: {}
securityContext:
privileged: true
volumeMounts:
- mountPath: /tmp/results
name: results
- mountPath: /node
name: root
kind: ConfigMap
metadata:
labels:
component: sonobuoy
name: sonobuoy-plugins-cm
namespace: sonobuoy
---
apiVersion: v1
kind: Pod
metadata:
labels:
component: sonobuoy
run: sonobuoy-master
sonobuoy-component: aggregator
tier: analysis
name: sonobuoy
namespace: sonobuoy
spec:
containers:
- env:
- name: SONOBUOY_ADVERTISE_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: sonobuoy/sonobuoy:v0.53.2
imagePullPolicy: IfNotPresent
name: kube-sonobuoy
volumeMounts:
- mountPath: /etc/sonobuoy
name: sonobuoy-config-volume
- mountPath: /plugins.d
name: sonobuoy-plugins-volume
- mountPath: /tmp/sonobuoy
name: output-volume
restartPolicy: Never
serviceAccountName: sonobuoy-serviceaccount
tolerations:
- key: "kubernetes.io/e2e-evict-taint-key"
operator: "Exists"
volumes:
- configMap:
name: sonobuoy-config-cm
name: sonobuoy-config-volume
- configMap:
name: sonobuoy-plugins-cm
name: sonobuoy-plugins-volume
- emptyDir: {}
name: output-volume
---
apiVersion: v1
kind: Service
metadata:
labels:
component: sonobuoy
sonobuoy-component: aggregator
name: sonobuoy-aggregator
namespace: sonobuoy
spec:
ports:
- port: 8080
protocol: TCP
targetPort: 8080
selector:
sonobuoy-component: aggregator
type: ClusterIP
So, As you can see the above specification, that is what going to run on a cluster and start testing the conformance parameters.
Now, Lets run the test.
$ tanzu conformance run
INFO[0000] created object name=sonobuoy namespace= resource=namespaces
INFO[0000] created object name=sonobuoy-serviceaccount namespace=sonobuoy resource=serviceaccounts
INFO[0000] created object name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterrolebindings
INFO[0000] created object name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterroles
INFO[0000] created object name=sonobuoy-config-cm namespace=sonobuoy resource=configmaps
INFO[0000] created object name=sonobuoy-plugins-cm namespace=sonobuoy resource=configmaps
INFO[0000] created object name=sonobuoy namespace=sonobuoy resource=pods
INFO[0000] created object name=sonobuoy-aggregator namespace=sonobuoy resource=services
Check the status of test:
$ tanzu conformance status
The Sonobuoy aggregator is in the 'Pending' state. This is normal as the pod is created and begins to run, but if this state persists, use kubectl to debug further.
And you see above because pods are still being created, Check the status.
$ k get po -n sonobuoy
NAME READY STATUS RESTARTS AGE
sonobuoy 1/1 Running 0 20s
sonobuoy-e2e-job-0619425a176943e5 0/2 ContainerCreating 0 12s
sonobuoy-systemd-logs-daemon-set-8cdf7bf30e3a4e40-djgmk 0/2 ContainerCreating 0 12s
sonobuoy-systemd-logs-daemon-set-8cdf7bf30e3a4e40-l57fc 0/2 ContainerCreating 0 12s
Wait for few mins to run the pods and then run the command again and you will the status:
$ tanzu conformance status
PLUGIN STATUS RESULT COUNT PROGRESS
e2e running 1 2/337 (0 failures)
systemd-logs complete 2
Sonobuoy is still running. Runs can take 60 minutes or more depending on cluster and plugin configuration.
As you can see, first systemd-logs plugin will be executed and then e2e plugin. It says about 60 mins but in my case, it took around 3 hours. So wait.
Eventually, when all tests are run, see the status again.
$ tanzu conformance status
PLUGIN STATUS RESULT COUNT PROGRESS
e2e complete failed 1 337/337 (3 failures)
systemd-logs complete passed 2
In my case, there are three test failed. Before you dig, Lets explore few more useful options.
Fetch the Test Result
Run the below command to retrive the test result.
$ tanzu conformance retrieve .
202110301357_sonobuoy_827a65bd-8fd3-4046-8b2e-31dbc1a91f53.tar.gz
Validate the result
$ tanzu conformance results 202110301357_sonobuoy_827a65bd-8fd3-4046-8b2e-31dbc1a91f53.tar.gz
Plugin: e2e
Status: failed
Total: 5771
Passed: 334
Failed: 3
Skipped: 5434
Failed tests:
[sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance]
[sig-apps] Daemon set [Serial] should rollback without unnecessary restarts [Conformance]
[sig-scheduling] SchedulerPreemption [Serial] validates lower priority pod preemption by critical pod [Conformance]
Plugin: systemd-logs
Status: passed
Total: 2
Passed: 2
Failed: 0
Skipped: 0
Check the logs
Run below command to check the status
$ tanzu conformance logs
View Detailed Report with failure message
Add --mode detailed option to see the result in detail
$ tanzu conformance results 202110301357_sonobuoy_827a65bd-8fd3-4046-8b2e-31dbc1a91f53.tar.gz --mode detailed
Finally, If you don’t want to run full end to end test or change some paramters like image etc, see the available options.
$ tanzu conformance run --help
Starts a Sonobuoy run by launching the Sonobuoy aggregator and plugin pods.
Usage:
tanzu conformance run [flags]
Flags:
--aggregator-node-selector nodeSelectors Node selectors to add to the aggregator. Values can be given multiple times and are in the form key:value (default map[])
--config Sonobuoy config Path to a sonobuoy configuration JSON file.
--context string Context in the kubeconfig to use.
--dns-namespace string The namespace to check for DNS pods during preflight checks. (default "kube-system")
--dns-pod-labels strings The label selectors to use for locating DNS pods during preflight checks. Can be specified multiple times or as a comma-separated list. (default [k8s-app=kube-dns,k8s-app=coredns])
--e2e-focus envModifier Specify the E2E_FOCUS value for the e2e plugin, specifying which tests to run. Shorthand for --plugin-env=e2e.E2E_FOCUS=<string> (default \[Conformance\])
--e2e-repo-config yaml-filepath Specify a yaml file acting as KUBE_TEST_REPO_LIST, overriding registries for test images.
--e2e-skip envModifier Specify the E2E_SKIP value for the e2e plugin, specifying which tests to skip. Shorthand for --plugin-env=e2e.E2E_SKIP=<string> (default \[Disruptive\]|NoExecuteTaintManager)
-f, --file - If set, loads the file as if it were the output from sonobuoy gen. Set to - to read from stdin.
-h, --help help for run
--image-pull-policy string Set the ImagePullPolicy for the Sonobuoy image and all plugins. Valid options are Always, Never, IfNotPresent. (default "IfNotPresent")
--kube-conformance-image image Container image override for the e2e plugin. Shorthand for --plugin-image=e2e:<string> (default map[])
--kubeconfig Kubeconfig Path to explicit kubeconfig file.
--kubernetes-version string Use default E2E image, but override the version. Default is 'auto', which will be set to your cluster's version if detected, erroring otherwise. 'ignore' will try version resolution but ignore errors. 'latest' will find the latest dev image/version upstream.
-m, --mode Mode What mode to run the e2e plugin in. Valid modes are [non-disruptive-conformance certified-conformance quick]. (default non-disruptive-conformance)
-n, --namespace string The namespace to run Sonobuoy in. Only one Sonobuoy run can exist per namespace simultaneously. (default "sonobuoy")
-p, --plugin pluginList Which plugins to run. Can either point to a URL, local file/directory, or be one of the known plugins (e2e or systemd-logs). Can be specified multiple times to run multiple plugins.
--plugin-env pluginenvvar Set env vars on plugins. Values can be given multiple times and are in the form plugin.env=value (default map[])
--plugin-image plugin:image Override a plugins image from what is in its definition (e.g. myPlugin:testimage) (default map[])
--rbac RBACMode Whether to enable RBAC on Sonobuoy. Valid modes are Enable, Disable, and Detect (query the server to see whether to enable RBAC). (default Detect)
--show-default-podspec If true, include the default pod spec used for plugins in the output.
--skip-preflight If true, skip all checks before starting the sonobuoy run.
--sonobuoy-image string Container image override for the sonobuoy worker and aggregator. (default "sonobuoy/sonobuoy:v0.53.2")
--ssh-key yamlFile Path to the private key enabling SSH to cluster nodes. May be required by some tests from the e2e plugin.
--ssh-user envModifier SSH user for ssh-key. Required if running e2e plugin with certain tests that require SSH access to nodes.
--systemd-logs-image image Container image override for the systemd-logs plugin. Shorthand for --plugin-image=systemd-logs:<string> (default map[])
--timeout int How long (in seconds) Sonobuoy aggregator will wait for plugins to complete before exiting. 0 indicates no timeout. (default 21600)
--wait int[=1440] How long (in minutes) for the CLI to wait for sonobuoy run to be completed or fail, where 0 indicates do not wait. If specified, the default wait time is 1 day.
--wait-output string Specify the type of output Sonobuoy should produce when --wait is used. Valid modes are silent, spinner, or progress (default "Silent")
Global Flags:
-v, --verbose int32 Number for the log level verbosity(0-9)
Cleanup
Run below command to cleanup Sonobuoy run
$ tanzu conformance delete
INFO[0000] deleted kind=namespace namespace=sonobuoy
INFO[0000] deleted kind=clusterrolebindings
INFO[0000] deleted kind=clusterroles
Modern Apps Platform Learning 