VMware announced Tanzu Community Edition (TCE) this week. It’s an OpenSource and free to use version. However, from a capability perspective, it got all the capabilties that TKG comes with including Tanzu packages. TCE support different deployment model as shown in below pictures.
In the first picture, you see the flow fo creating a standalone cluster where bootstrap cluster and then workload cluster. In other picture, you can see the general flow where bootstrap cluster creates a management cluster and then we can have 1 or more workload clusters.
In this blog post, I would like to provide you the step by step intructions on how to setup TCE standalone cluster on a MAC.
Pre-requirements
- You have Docker Desktop running. You can find steps to setup Docker desktop on MAC here
- Docker is running with minimum of 2 CPU and 6GB RAM
- Kubectl is installed on your workstation to interact with TCE Cluster after setup. You can find the steps to setup kubectl on mac
Note: Docker desktop setting for resources.
Installing TCE on Docker Destop running on MAC
- Add a repository and install tanzu cli
# Run the below commands
$ brew tap vmware-tanzu/tanzu
$ brew install tanzu-community-edition
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 5 taps (cloudfoundry/tap, homebrew/core, homebrew/cask, homebrew/services and vmware-tanzu/carvel).
==> New Formulae
aws-sso-util clickhouse-odbc cmake-docs texlive
ca-certificates cloudfoundry/tap/cf-cli@8 git-credential-libsecret
==> Updated Formulae
Updated 348 formulae.
==> New Casks
bespoke plistedplus remarkable
fig rancher thetimemachinemechanic
==> Updated Casks
Updated 258 casks.
==> Deleted Casks
google-backup-and-sync google-photos-backup-and-sync nosqlclient
==> Tapping vmware-tanzu/tanzu
Cloning into '/usr/local/Homebrew/Library/Taps/vmware-tanzu/homebrew-tanzu'...
remote: Enumerating objects: 53, done.
remote: Counting objects: 100% (53/53), done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 53 (delta 22), reused 23 (delta 9), pack-reused 0
Receiving objects: 100% (53/53), 21.41 KiB | 5.35 MiB/s, done.
Resolving deltas: 100% (22/22), done.
Tapped 1 formula (19 files, 53.8KB).
==> Downloading https://github.com/vmware-tanzu/community-edition/releases/download/v0.9.1/tce-darwin-amd64-v
==> Downloading from https://github-releases.githubusercontent.com/303802332/f34557b0-acad-41d4-a12a-24474258
######################################################################## 100.0%
==> Installing tanzu-community-edition from vmware-tanzu/tanzu
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/tanzu
Target /usr/local/bin/tanzu
already exists. You may want to remove it:
rm '/usr/local/bin/tanzu'
To force the link and overwrite all conflicting files:
brew link --overwrite tanzu-community-edition
To list all files that would be deleted:
brew link --overwrite --dry-run tanzu-community-edition
Possible conflicting files are:
/usr/local/bin/tanzu
==> Thanks for installing Tanzu Community Edition!
==> The Tanzu CLI has been installed on your system
==>
==> ******************************************************************************
==> * To initialize all plugins required by Tanzu Community Edition, an additional
==> * step is required. To complete the installation, please run the following
==> * shell script:
==> *
==> * /usr/local/Cellar/tanzu-community-edition/v0.9.1/libexec/configure-tce.sh
==> *
==> ******************************************************************************
==>
==> * To cleanup and remove Tanzu Community Edition from your system, run the
==> * following script:
==> /usr/local/Cellar/tanzu-community-edition/v0.9.1/libexec/uninstall.sh
==>
==> Summary
/usr/local/Cellar/tanzu-community-edition/v0.9.1: 15 files, 642.6MB, built in 10 seconds
Note: If you are seeing the below warning, this may come in case you have another Tanzu cli running
-
Warning: vmware-tanzu/tanzu/tanzu-community-edition v0.9.1 is already installed, it's just not linked.Run the below command -
To link this version
-
run:
$ brew link tanzu-community-edition
Linking /usr/local/Cellar/tanzu-community-edition/v0.9.1... 1 symlinks created.
- Validate the Tanzu CLI by running the below command
$ tanzu
Tanzu CLI
Usage:
tanzu [command]
Available command groups:
Run
cluster Kubernetes cluster operations
kubernetes-release Kubernetes release operations
management-cluster Kubernetes management cluster operations
package Tanzu package management
System
completion Output shell completion code
config Configuration for the CLI
init Initialize the CLI
login Login to the platform
plugin Manage CLI plugins
update Update the CLI
version Version information
Flags:
-h, --help help for tanzu
Use "tanzu [command] --help" for more information about a command.
- Validat the Tanzu CLI version.
tanzu version
version: v0.2.1
buildDate: 2021-09-29
sha: ceaa474
- Configure the Tanzu CLI by running the following command
# Note: You will find the below path including script name after executing very first command, Check the output and copy the command from above.
$ /usr/local/Cellar/tanzu-community-edition/v0.9.1/libexec/configure-tce.sh
MY_DIR: /usr/local/Cellar/tanzu-community-edition/v0.9.1/libexec
/Users/dinetrip/Library/Application Support
Removing old plugin cache from /Users/dinetrip/.cache/tanzu/catalog.yaml
Making a backup of your Kuberentes config files int /tmp
| initializing ✔ successfully initialized CLI
Installation complete!
- Now, Validate the tanzu cli and you will notice the differences. There are more plugins and commands avilable.
$ tanzu
Tanzu CLI
Usage:
tanzu [command]
Available command groups:
Admin
builder Build Tanzu components
Run
cluster Kubernetes cluster operations
conformance Run Sonobuoy conformance tests against clusters
diagnostics Cluster diagnostics
kubernetes-release Kubernetes release operations
management-cluster Kubernetes management cluster operations
package Tanzu package management
standalone-cluster Create clusters without a dedicated management cluster
System
completion Output shell completion code
config Configuration for the CLI
init Initialize the CLI
login Login to the platform
plugin Manage CLI plugins
update Update the CLI
version Version information
Flags:
-h, --help help for tanzu
Use "tanzu [command] --help" for more information about a command.
- Let’s run a command to create a standalone cluster.
$ tanzu standalone-cluster create -i docker tkgondocker
Downloading TKG compatibility file from 'projects.registry.vmware.com/tkg/framework-zshippable/tkg-compatibility'
Downloading the TKG Bill of Materials (BOM) file from 'projects.registry.vmware.com/tkg/tkg-bom:v1.4.0'
Downloading the TKr Bill of Materials (BOM) file from 'projects.registry.vmware.com/tkg/tkr-bom:v1.21.2_vmware.1-tkg.1'
Validating the pre-requisites...
Identity Provider not configured. Some authentication features won't work.
Setting up standalone cluster...
Validating configuration...
Using infrastructure provider docker:v0.3.23
Generating cluster configuration...
Setting up bootstrapper...
Bootstrapper created. Kubeconfig: /Users/dinetrip/.kube-tkg/tmp/config_tImxi3O1
Installing providers on bootstrapper...
Start creating standalone cluster...
Saving standalone cluster kubeconfig into /Users/dinetrip/.kube/config
Waiting for bootstrap cluster to get ready for save ...
Waiting for addons installation...
Moving all Cluster API objects from bootstrap cluster to standalone cluster...
Context set for standalone cluster tkgondocker as 'tkgondocker-admin@tkgondocker'.
Cleaning up unneeded resources (for standalone clusters)...
Standalone cluster created!
You can now use Kubectl to access your cluster:
kubectl get pods -A
You can also delete the cluster by running the following:
tanzu standalone-cluster delete [name]
Some addons might be getting installed! Check their status by running the following:
kubectl get apps -A
- while above command is running, you can grab the kubeconfig file from below location and check the progress status.
# find the latest kubeconfig file from below directory
$ ls -ltra ~/.kube-tkg/tmp
total 48
-rw-r--r-- 1 dinetrip staff 5660 Nov 6 2020 config_THfjUqmF
-rw-r--r-- 1 dinetrip staff 5664 Nov 9 2020 config_7kMvqdKt
-rw-r--r-- 1 dinetrip staff 5688 Oct 4 20:03 config_tImxi3O1
drwxr-xr-x 5 dinetrip staff 160 Oct 4 20:03 .
drwxr-xr-x 4 dinetrip staff 128 Oct 4 20:08 ..
# view the list of pods running on bootstrap node.
(main) $ k get po -A --kubeconfig ~/.kube-tkg/tmp/config_tImxi3O1
NAMESPACE NAME READY STATUS RESTARTS AGE
capd-system capd-controller-manager-5885f59558-9tp8t 2/2 Running 0 3m25s
capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-6494884869-hs2tm 2/2 Running 0 3m31s
capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-857d687b9d-lpgf6 2/2 Running 0 3m28s
capi-system capi-controller-manager-778bd4dfb9-d65g2 2/2 Running 0 3m36s
capi-webhook-system capi-controller-manager-9995bdc94-jf7jp 2/2 Running 0 3m40s
capi-webhook-system capi-kubeadm-bootstrap-controller-manager-68845b65f8-7w5ms 2/2 Running 0 3m34s
capi-webhook-system capi-kubeadm-control-plane-controller-manager-9847c6747-wmngn 2/2 Running 0 3m30s
cert-manager cert-manager-77f6fb8fd5-mrtps 1/1 Running 0 5m3s
cert-manager cert-manager-cainjector-6bd4cff7bb-vtsw9 1/1 Running 0 5m3s
cert-manager cert-manager-webhook-fbfcb9d6c-csmwz 1/1 Running 0 5m2s
kube-system coredns-8dcb5c56b-cs4b7 1/1 Running 0 5m45s
kube-system coredns-8dcb5c56b-kchrv 1/1 Running 0 5m45s
kube-system etcd-tkg-kind-c5dh03sbcv49n0qus2kg-control-plane 1/1 Running 0 5m54s
kube-system kindnet-bcrth 1/1 Running 1 5m45s
kube-system kube-apiserver-tkg-kind-c5dh03sbcv49n0qus2kg-control-plane 1/1 Running 0 5m54s
kube-system kube-controller-manager-tkg-kind-c5dh03sbcv49n0qus2kg-control-plane 1/1 Running 0 5m59s
kube-system kube-proxy-w4smg 1/1 Running 0 5m45s
kube-system kube-scheduler-tkg-kind-c5dh03sbcv49n0qus2kg-control-plane 1/1 Running 0 5m54s
local-path-storage local-path-provisioner-8b46957d4-vxzzb 1/1 Running 0 5m45s
(main) $
- You can also see the
kindcluster running
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b25052dc2da9 projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1 "/usr/local/bin/entr…" 4 minutes ago Up 4 minutes 127.0.0.1:57265->6443/tcp tkg-kind-c5dh03sbcv49n0qus2kg-control-plane
- Once the
kindcluster is ready, you will notice that the ha-proxy load balancer and control plane containers will start
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
225d61b6f9ba projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1 "/usr/local/bin/entr…" 2 minutes ago Up 2 minutes tkgondocker-md-0-7b9758c6d6-wqdlk
347bf31db466 projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1 "/usr/local/bin/entr…" 4 minutes ago Up 3 minutes 40819/tcp, 127.0.0.1:40819->6443/tcp tkgondocker-control-plane-cmqpg
8167fc8ee5e4 kindest/haproxy:v20210715-a6da3463 "haproxy -sf 7 -W -d…" 4 minutes ago Up 4 minutes 41431/tcp, 0.0.0.0:41431->6443/tcp tkgondocker-lb
b25052dc2da9 projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1 "/usr/local/bin/entr…" 8 minutes ago Up 8 minutes 127.0.0.1:57265->6443/tcp tkg-kind-c5dh03sbcv49n0qus2kg-control-plane
- After few mins, when the cluster is created, you will notice that the
kindcontainer is deleted.
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
225d61b6f9ba projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1 "/usr/local/bin/entr…" 3 minutes ago Up 3 minutes tkgondocker-md-0-7b9758c6d6-wqdlk
347bf31db466 projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1 "/usr/local/bin/entr…" 5 minutes ago Up 5 minutes 40819/tcp, 127.0.0.1:40819->6443/tcp tkgondocker-control-plane-cmqpg
8167fc8ee5e4 kindest/haproxy:v20210715-a6da3463 "haproxy -sf 7 -W -d…" 5 minutes ago Up 5 minutes 41431/tcp, 0.0.0.0:41431->6443/tcp tkgondocker-lb
- Now, Set the right context
# Set the contexts
$ kubectl config use-context tkgondocker-admin@tkgondocker
Switched to context "tkgondocker-admin@tkgondocker".
# View the post of pods from TCE Cluster
$ k get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system antrea-agent-ddvx9 2/2 Running 0 5m55s
kube-system antrea-agent-ztc7p 2/2 Running 1 5m55s
kube-system antrea-controller-86f8988c5f-r2w8d 1/1 Running 0 5m56s
kube-system coredns-8dcb5c56b-btk8j 1/1 Running 0 6m34s
kube-system coredns-8dcb5c56b-g97v6 1/1 Running 0 6m34s
kube-system etcd-tkgondocker-control-plane-cmqpg 1/1 Running 0 6m32s
kube-system kube-apiserver-tkgondocker-control-plane-cmqpg 1/1 Running 0 6m32s
kube-system kube-controller-manager-tkgondocker-control-plane-cmqpg 1/1 Running 1 6m32s
kube-system kube-proxy-9wsjt 1/1 Running 0 5m56s
kube-system kube-proxy-ckjqs 1/1 Running 0 6m35s
kube-system kube-scheduler-tkgondocker-control-plane-cmqpg 1/1 Running 1 6m32s
tkg-system kapp-controller-6499b8866-jxdrd 1/1 Running 0 5m56s
tkg-system tanzu-capabilities-controller-manager-6ff97656b8-45qf2 1/1 Running 0 6m13s
tkr-system tkr-controller-manager-6bc455b5d4-2pq2c 1/1 Running 0 6m15s
- View the list of nodes
$ k get nodes
NAME STATUS ROLES AGE VERSION
tkgondocker-control-plane-cmqpg Ready control-plane,master 8m42s v1.21.2+vmware.1-360497810732255795
tkgondocker-md-0-7b9758c6d6-wqdlk Ready 7m57s v1.21.2+vmware.1-360497810732255795
- Lets validate the newly created TCE cluster by creating an nginx pod
# create a pod
$ k run nginx --image nginx
pod/nginx created
$ k get po
NAME READY STATUS RESTARTS AGE
nginx 0/1 ContainerCreating 0 2s
# Check the pod status
$ k get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 0 62s 100.96.1.6 tkgondocker-md-0-7b9758c6d6-wqdlk
Now, TCE cluster is up and running and it got 1 master node and 1 worker node. Both nodes are running as docker container.
Let’s deploy one of the package too.
Deploying Tanzu Package on a TCE Cluster
- Add a TCE package repository
$ tanzu package repository add tce-repo --url projects.registry.vmware.com/tce/main:0.9.1 --namespace tanzu-package-repo-global
/ Adding package repository 'tce-repo'...
Added package repository 'tce-repo'
$ tanzu package repository list --namespace tanzu-package-repo-global
/ Retrieving repositories...
NAME REPOSITORY STATUS DETAILS
tce-repo projects.registry.vmware.com/tce/main:0.9.1 Reconciling
$ tanzu package repository list --namespace tanzu-package-repo-global
/ Retrieving repositories...
NAME REPOSITORY STATUS DETAILS
tce-repo projects.registry.vmware.com/tce/main:0.9.1 Reconcile succeeded
$
- View the list of available packages.
$ tanzu package available list
/ Retrieving available packages...
NAME DISPLAY-NAME SHORT-DESCRIPTION
cert-manager.community.tanzu.vmware.com cert-manager Certificate management
contour.community.tanzu.vmware.com Contour An ingress controller
external-dns.community.tanzu.vmware.com external-dns This package provides DNS synchronization functionality.
fluent-bit.community.tanzu.vmware.com fluent-bit Fluent Bit is a fast Log Processor and Forwarder
gatekeeper.community.tanzu.vmware.com gatekeeper policy management
grafana.community.tanzu.vmware.com grafana Visualization and analytics software
harbor.community.tanzu.vmware.com Harbor OCI Registry
knative-serving.community.tanzu.vmware.com knative-serving Knative Serving builds on Kubernetes to support deploying and serving of applications and functions as serverless containers
local-path-storage.community.tanzu.vmware.com local-path-storage This package provides local path node storage and primarily supports RWO AccessMode.
multus-cni.community.tanzu.vmware.com multus-cni This package provides the ability for enabling attaching multiple network interfaces to pods in Kubernetes
prometheus.community.tanzu.vmware.com prometheus A time series database for your metrics
velero.community.tanzu.vmware.com velero Disaster recovery capabilities
- List one of the package detail that we are going to install.
$ tanzu package available list cert-manager.community.tanzu.vmware.com
/ Retrieving package versions for cert-manager.community.tanzu.vmware.com...
NAME VERSION RELEASED-AT
cert-manager.community.tanzu.vmware.com 1.3.3 2021-08-06T12:31:21Z
cert-manager.community.tanzu.vmware.com 1.4.4 2021-08-23T16:47:51Z
cert-manager.community.tanzu.vmware.com 1.5.3 2021-08-23T17:22:51Z
- Install the
cert-managerpackage
$ tanzu package install cert-manager \
--package-name cert-manager.community.tanzu.vmware.com \
--version 1.4.4
/ Installing package 'cert-manager.community.tanzu.vmware.com'
| Getting namespace 'default'
| Getting package metadata for 'cert-manager.community.tanzu.vmware.com'
| Creating service account 'cert-manager-default-sa'
| Creating cluster admin role 'cert-manager-default-cluster-role'
| Creating cluster role binding 'cert-manager-default-cluster-rolebinding'
- Creating package resource
- Package install status: Reconciling
Added installed package 'cert-manager' in namespace 'default'
- Validate the package installation
$ k get ns
NAME STATUS AGE
cert-manager Active 24s
default Active 16m
kube-node-lease Active 16m
kube-public Active 16m
kube-system Active 16m
tanzu-package-repo-global Active 16m
tkg-system Active 16m
tkg-system-public Active 16m
tkr-system Active 16m
(main) $ k get po -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-547fbbfbfb-b9bw6 1/1 Running 0 28s
cert-manager-cainjector-64bf4696c7-rz2fz 1/1 Running 0 28s
cert-manager-webhook-7497dbb564-bfwfp 1/1 Running 0 28s
$ tanzu package installed list
/ Retrieving installed packages...
NAME PACKAGE-NAME PACKAGE-VERSION STATUS
cert-manager cert-manager.community.tanzu.vmware.com 1.4.4 Reconcile succeeded
Deleting a TCE cluster
Run the below command to delete the TCE Cluster.
$ tanzu standalone-cluster delete tkgondocker
Loading bootstrap cluster config for standalone cluster at '/Users/dinetrip/.config/tanzu/tkg/clusterconfigs/tkgondocker.yaml'
no bootstrap cluster config found - using default config
Deleting standalone cluster 'tkgondocker'. Are you sure? [y/N]: y
loading cluster config file at
Setting up cleanup cluster...
Installing providers to cleanup cluster...
Moving all Cluster API objects from bootstrap cluster to standalone cluster...
Waiting for the Cluster API objects to be ready after restore ...
Deleting standalone cluster...
Standalone cluster 'tkgondocker' deleted.
Deleting the standalone cluster context from the kubeconfig file '/Users/dinetrip/.kube/config'
warning: this removed your active context, use "kubectl config use-context" to select a different one
Standalone cluster deleted!
Removing temporary bootstrap cluster config for standalone cluster at '/Users/dinetrip/.config/tanzu/tkg/configs/tkgondocker_ClusterConfig'
no bootstrap cluster config found - skipping
Removing temporary UI bootstrap cluster config for standalone cluster at '/Users/dinetrip/.config/tanzu/clusterconfigs/tkgondocker.yaml'
no UI bootstrap cluster config found - skipping
- You will notice that the another
kindcluster created, notice the container.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
**768442c470da projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1 "/usr/local/bin/entr…" 9 seconds ago Up 2 seconds 127.0.0.1:59192->6443/tcp tkg-kind-c5dhe4sbcv4a5gqv1dv0-control-plane**
225d61b6f9ba projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1 "/usr/local/bin/entr…" 21 minutes ago Up 21 minutes tkgondocker-md-0-7b9758c6d6-wqdlk
347bf31db466 projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1 "/usr/local/bin/entr…" 23 minutes ago Up 23 minutes 40819/tcp, 127.0.0.1:40819->6443/tcp tkgondocker-control-plane-cmqpg
8167fc8ee5e4 kindest/haproxy:v20210715-a6da3463 "haproxy -sf 7 -W -d…" 23 minutes ago Up 23 minutes 41431/tcp, 0.0.0.0:41431->6443/tcp tkgondocker-lb
- Finally, it is cleaned up
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
