Traditional containers run on the same operating system kernel, which may have some security vulnerabilities, such as privilege escalation and kernel vulnerabilities. If you are concerned about container security, run KubeBlocks on Kata Containers might be a solution. We have built a Kata Containers environment and performed some basic functional verification. Kata Containers (Kata for short) is an open-source project that provides a secure and high-performance container runtime environment. The goal of Kata Containers is to combine virtualization technology with container technology, providing a user experience similar to lightweight containers while offering higher isolation and security.
grep vmx /proc/cpuinfo
slc@kata-1:~$ grep vmx /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat vnmi md_clear arch_capabilities
vmx flags : vnmi preemption_timer invvpid ept_x_only ept_ad flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest vapic_reg vid shadow_vmcs
If the server supports virtualization, the output of the grep vmx /proc/cpuinfo
command will show information related to the 'vmx' flag, which indicates the support for hardware-assisted virtualization (Intel VT-x).
2.Creating Virtual Machines on GCP The level of support for nested virtualization varies among different cloud providers. Among them, GCP (Google Cloud Platform) is said to have the best support for nested virtualization. For more details, please refer to the GCP nested virtualization documentation. To create 3 nested virtualization servers on GCP, you can use the following gcloud command:
slc@bogon aws % gcloud compute instances create kata-1 \
--enable-nested-virtualization \
--project=apecloud-labs \
--zone=us-central1-c \
--machine-type=n1-standard-4 \
--network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default \
--maintenance-policy=MIGRATE \
--provisioning-model=STANDARD \
--service-account=822787046197-compute@developer.gserviceaccount.com \
--scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append \
--create-disk=auto-delete=yes,boot=yes,device-name=instance-20240407-035747,image=projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319,mode=rw,size=100,type=projects/apecloud-labs/zones/us-central1-c/diskTypes/pd-balanced \
--no-shielded-secure-boot \
--shielded-vtpm \
--shielded-integrity-monitoring \
--labels=goog-ec-src=vm_add-gcloud \
--reservation-affinity=any
To create the 3 VMs that support the vmx
flag, you can execute the gcloud command 3 times.
This will create the 3 VMs that support the vmx
flag. Additionally, you'll need to configure the VPC firewall rules to allow ipip
and dns53
traffic between the Kubernetes internal nodes for proper communication. This can be done through the GCP console or using the gcloud
command-line tool.
We have chosen to use K3s as the Kubernetes foundation for our deployment. K3s is a lightweight, production-ready Kubernetes distribution that is simple and quick to install. See https://k3s.io/.
Steps:
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
sudo k3s token create
curl -sfL https://get.k3s.io | K3S_URL=https://10.128.0.49:6443 K3S_TOKEN="K1027e02d430f1a7c8a4e9a67b9a8a354875ff92c366a830d66ceada5784518e8c8::2bhamg.xd7vwu3dlkqyipvh" sh -
kubectl get nodes
slc@kata-1:~$ kg nodes
NAME STATUS ROLES AGE VERSION
kata-2 Ready <none> 100m v1.28.8+k3s1
kata-1 Ready control-plane,master 104m v1.28.8+k3s1
kata-3 Ready <none> 99m v1.28.8+k3s1
Here we use cloud hypervisor as the default hypervisor. See kata-deploy.
git clone https://github.com/kata-containers/kata-containers.git
cd kata-containers/tools/packaging/kata-deploy/
kubectl apply -f kata-rbac/base/kata-rbac.yaml
kubectl apply -k kata-deploy/overlays/k3s
kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/runtimeclasses/kata-runtimeClasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-clh.yaml
kubectl get pods -A
slc@kata-1:~$ kg pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-6c86858495-z4fm5 1/1 Running 0 106m
kube-system svclb-traefik-5010304c-zldjh 2/2 Running 0 106m
kube-system svclb-traefik-5010304c-wlmtb 2/2 Running 0 103m
kube-system svclb-traefik-5010304c-49jz5 2/2 Running 0 102m
kube-system kata-deploy-6hw8t 1/1 Running 0 98m
kube-system kata-deploy-g4j9s 1/1 Running 0 98m
kube-system kata-deploy-drn8w 1/1 Running 0 98m
kube-system helm-install-traefik-zl849 0/1 Completed 1 106m
kube-system helm-install-traefik-crd-rhfl4 0/1 Completed 0 106m
default php-apache-kata-clh-67f67d6f89-h2dw6 1/1 Running 0 94m
kube-system metrics-server-54fd9b65b-hqpm7 1/1 Running 0 106m
kube-system coredns-6799fbcd5-68wxv 1/1 Running 0 106m
kube-system traefik-f4564c4f4-4crfq 1/1 Running 0 106m
We can see the corresponding runtime.
root 6902 0.0 0.2 1419432 39560 ? Sl 07:22 0:03 /opt/kata/bin/containerd-shim-kata-v2 -namespace k8s.io -address /run/k3s/containerd/containerd.sock -publish-binary -id 9e7c835d4868398b062d9735e
root 6914 0.0 0.0 2612 1920 ? S 07:22 0:00 \_ /opt/kata/libexec/virtiofsd --syslog --cache=auto --shared-dir=/run/kata-containers/shared/sandboxes/9e7c835d4868398b062d9735eb28115be58e5ab8ce
root 6919 0.0 0.1 2106124 26756 ? Sl 07:22 0:00 | \_ /opt/kata/libexec/virtiofsd --syslog --cache=auto --shared-dir=/run/kata-containers/shared/sandboxes/9e7c835d4868398b062d9735eb28115be58e5a
root 6915 0.3 0.9 2385428 151808 ? Sl 07:22 0:18 \_ /opt/kata/bin/cloud-hypervisor --api-socket /run/vc/vm/9e7c835d4868398b062d9735eb28115be58e5ab8ce7111f0e729f651b7a3c2c3/clh-api.sock
curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash
kbcli kubeblocks list-versions
kbcli kubeblocks install --version="0.8.2"
Error message:
slc@kata-1:~$ kbcli kubeblocks install --version="0.8.2"
The connection to the server localhost:8080 was refused - did you specify the right host or port?
The reason for the issue is that the kbcli tool was unable to locate the kubeconfig file when enumerating the Kubernetes contexts. As a result, it ended up trying to connect to localhost:8080 as a last resort.
To resolve this, we can create a symbolic link (soft link) to the kubeconfig
file, which should allow kbcli
to properly detect and use the correct Kubernetes context.
ln -sf /etc/rancher/k3s/k3s.yaml /home/slc/.kube/.kubeconfig
Wait until KubeBlocks is installed.
Steps:
kbcli cluster create --cluster-definition=apecloud-mysql
kbcli cluster connect aloe03
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| kubeblocks |
| mydb |
| mysql |
| performance_schema |
| sys |
+--------------------+
6 rows in set (0.00 sec)
Runtime information:
root 11801 0.1 0.0 1238384 15136 ? Sl 08:21 0:02 /var/lib/rancher/k3s/data/ef92c15b8f1c59f266ea4a9589be87b1b799d1682673f29721e8a1f7b0a4a97b/bin/containerd-shim-runc-v2 -namespace k8s.io -id 604c9d603425ef5c172e94a7598af3fd1b7e2004dfef4e7517e5e731e5123fc7 -address /run/k3s/containerd/containerd.sock
65535 11822 0.0 0.0 972 512 ? Ss 08:21 0:00 \_ /pause
ubuntu 12032 0.9 1.4 1105756 217004 ? Ssl 08:21 0:25 \_ mysqld
root 12181 0.0 0.0 22716 2320 ? Ss 08:21 0:00 | \_ crond -i -s
root 12590 0.0 0.0 1604 896 ? Ss 08:21 0:00 \_ /bin/sh /scripts/agamotto.sh
root 12602 0.1 0.6 885824 99328 ? Sl 08:21 0:03 | \_ /bin/agamotto --config=/opt/agamotto/agamotto-config.yaml
root 12752 0.0 0.0 2388 1664 ? Ss 08:22 0:00 \_ /bin/bash /scripts/vttablet.sh
root 14978 0.0 0.0 2312 1280 ? S 09:06 0:00 | \_ sleep 60
65532 12911 0.1 0.3 1287692 52068 ? Ssl 08:22 0:03 \_ lorry --port 3501 --grpcport 50001
65532 12950 0.0 0.2 1271036 35072 ? Ssl 08:22 0:00 \_ /bin/reloader --log-level info --operator-update-enable --tcp 9901 --config /opt/config-manager/config-manager.yaml
podSpec:
runtimeClassName: kata-clh
Download kubeblocks-addon version 0.8.3
wget https://github.com/apecloud/kubeblocks-addons/archive/refs/tags/v0.8.3-beta.5.tar.gz
tar zxf v0.8.3-beta.5.tar.gz
Change ClusterDefinition of apecloud-mysql addon
cd kubeblocks-addons-0.8.3-beta.5/addons
vi apecloud-mysql/templates/clusterdefinition.yaml
All podSpec support kata
podSpec:
runtimeClassName: kata-clh
containers:
- name: mysql
Save and render the new apecloud-mysql addon.
helm template apecloud-mysql --set resourceNamePrefix="kata-apecloud-mysql" > /tmp/a
kubectl apply -f /tmp/a
slc@kata-1:~$ kubectl get cd
NAME MAIN-COMPONENT-NAME STATUS AGE
redis redis Available 105m
clickhouse clickhouse Available 105m
mongodb-sharding mongos Available 105m
mongodb mongodb Available 105m
kafka kafka-server Available 105m
pulsar-zookeeper zookeeper Available 105m
postgresql postgresql Available 105m
apecloud-mysql mysql Available 105m
pulsar pulsar-broker Available 105m
kata-apecloud-mysql mysql Available 13m
Create apecloud-mysql cluster on Kata.
slc@kata-1:~$ kbcli cluster create --cluster-definition=kata-apecloud-mysql
Info: --cluster-version is not specified, ClusterVersion kata-apecloud-mysql-8.0.30 is applied by default
Cluster cornel23 created
Login to kata mysql.
kbcli cluster connect cornel23
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| kubeblocks |
| mydb |
| mysql |
| performance_schema |
| sys |
+--------------------+
6 rows in set (0.00 sec)
Test other command.
slc@kata-1:~$ kubectl exec -it cornel23-mysql-0 -- /bin/sh
Defaulted container "mysql" out of: mysql, metrics, vttablet, lorry, config-manager
sh-4.2# ls
anaconda-post.log apecloud bin data dev docker-entrypoint-initdb.d etc home lib lib64 media mnt opt proc root run sbin scripts srv sys tmp usr var
sh-4.2#
Check runtime progress.
root 19681 0.2 0.2 1419688 40612 ? Sl 09:52 0:01 /opt/kata/bin/containerd-shim-kata-v2 -namespace k8s.io -address /run/k3s/containerd/containerd.sock -publish-binary -id 115051aa1546b66657f14cd9a
root 19702 0.0 0.0 2612 1920 ? S 09:52 0:00 \_ /opt/kata/libexec/virtiofsd --syslog --cache=auto --shared-dir=/run/kata-containers/shared/sandboxes/115051aa1546b66657f14cd9a19f4223f98370377b
root 19708 1.4 2.4 3155940 380292 ? Sl 09:52 0:10 | \_ /opt/kata/libexec/virtiofsd --syslog --cache=auto --shared-dir=/run/kata-containers/shared/sandboxes/115051aa1546b66657f14cd9a19f4223f98370
root 19703 13.1 3.8 3436120 591872 ? Sl 09:52 1:31 \_ /opt/kata/bin/cloud-hypervisor --api-socket /run/vc/vm/115051aa1546b66657f14cd9a19f4223f98370377b3145354b8703bbcf2939c5/clh-api.sock
With KubeBlocks, we can easily run data on Kata Containers. However, to better support Kata Containers, KubeBlocks still needs to natively support the Kubernetes runtimeClass mechanism, in order to support more runtime environments.
Furthermore, we can also feel that due to its good compatibility with CRI, Kata Containers can provide a good user experience while ensuring security.