Deployment example (redmine)
Refer to the following deployment composed (deployment-redmine.yaml)
apiVersion: apps/v1
kind: Deployment
metadata:
name: redmine
labels:
app: redmine
spec:
replicas: 1
selector:
matchLabels:
app: redmine
strategy:
type: Recreate
template:
metadata:
labels:
app: redmine
spec:
containers:
- name: redmine
image: bitnami/redmine:4.1.1
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3000
protocol: TCP
env:
- name: REDMINE_USERNAME
value: admin
- name: REDMINE_PASSWORD
value: admin
- name: REDMINE_DB_USERNAME
value: root
- name: REDMINE_DB_PASSWORD
value: P2mobile8!886
- name: REDMINE_DB_NAME
value: bitnami_redmine
- name: REDMINE_DB_MYSQL
value: mariadb.redmine.svc.cluster.local
lifecycle:
postStart:
exec:
command: ["sh","-c","/usr/bin/curl http://storage.dev.hkg.internal.p2mt.com/storage/redmine/purple.tar --output purple.tar; tar -xvf purple.tar -C /opt/bitnami/redmine/public/themes/;\n"]
volumeMounts:
- mountPath: /bitnami
name: nfs-redmine
restartPolicy: Always
volumes:
- name: nfs-redmine
persistentVolumeClaim:
claimName: redmine-data-claim
Here are some points worth noting
Different types of "labels"
A. metadata.labels (with value app: redmine) - This label is to manage the deployment itself. This may be used to filter the deployment based on this label. E.g. when you run ""kubectl get deploy --show-labels, the following outputs
NAME READY UP-TO-DATE AVAILABLE AGE LABELS
redmine 1/1 1 1 20h app=redmine
We can see LABELS field with "app=redmine", which is what it was defined above, so when we want to filter specific deployment, we can also use this label by "kubectl get deployments -L app=redmine -o wide", the following will be outputted
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
redmine 1/1 1 1 20h redmine bitnami/redmine:4.1.1 app=redmine
B. spec.selector.matchLabels (with value app: redmine) - This label tells ReplicaSet, an application managed in Deployment, to use this label to identify and manage how many instances of this label of pods to maintain the numbers, this label must equal to the one specified in spec.template.spec.metadata.labels
C. spec.template.spec.metadata.labels (with value app: redmine) - Label of the pod, must be the same as spec.selector.matchLabels
Spec and templates
Spec and templates defines the desired state of the deployment. Here are some configs worth noting at.
1. deployment.spec: Defines status of pods within the deployment, e.g. how many copies of pods required for the service containers, how the pod is formed (by defining 1 or more containers within a pod)
2. deployment.spec.selector.matchLabels: A required field which must match with deployment.spec.template.metadata.labels, when specified, the deployment knows which pods underhood are being targeted.
3. deployment.spec.template.metadata.labels: The name of the template, which must match to deployment.spec.selector.matchLabels
4. deployment.spec.template.spec.containers: The formation in terms of containers that the deployment is defined to, more than 1 container can be specified, like docker compose, container name, image etc. can be designated. deployment.spec.template.spec.containers.[n].ports.containerPort specify the port used (internally) by container, it can then exposed with an external port / IP if service container is needed to be accessed externally.
deployment.spec.template.spec.containers.[n].lifecycle can be used to put some commands executed in different cycles, e.g. one may want to execute an entrypoint command after container in up successfully using .lifecycle.podStart.exec.command.
Volumes
In this redmine deployment example, we use nfs (Network File Storage) situated in our company's synology NAS as the permanent storage source, here is how to create a volume
1. Create PV (Persistent Volume) and PVC (Persistent Volume Claim)
In K8S, we need to defined PV and PVC first before we can actually "frame" them into the deployment yaml. Here are the yamls of PV and PVC.
Persistent Volume (PV), pv-redmine.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-redmine
spec:
storageClassName: redmine-class
capacity:
storage: 200Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
nfs:
path: /volume1/k8s-nfs/redmine
server: 10.240.0.3
PV defines the specifications of the volume, in this case, we use nfs as volume source, we must ensure the connectivity between the node and the NAS (10.240.0.3) before applying the manifest. And we need to also create and intialize the nfs server first (as shown below)
nfs.path is the physical location of the NAS, in our case, we need to refer to the NAS's nfs server mount point settings and fill in the mount point value. In Synology NAS, this information can be retrieved through "Shared folder > NFS rights", the mount path can be found at the bottom.
spec.storageClassName and spec.accessModes are worth noting, storageClassName is used to bind the PV with the PVC, the corresponding storage class name needs to be the same in PVC. ReadWriteOnce access mode means the volume can only be read / write by a single node.
Persistent Volume (PVC), pvc-redmine.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redmine-data-claim
spec:
storageClassName: redmine-class
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 200Gi
The PVC is created to claim the volume (PV), spec.storageClassName should match the one in PV or the claim cannot linked the PV.
Linking PVC to deployment
After creating PV and PVC, deployment.spec.template.spec.volumes should be created, it is part of the pod specifications, the deployment.spec.template.spec.volumes.persistentVolumeClaim.claimName should match the pvc.meta.name so as to link up the PV with the deployment.
Furthermore, the deployment.spec.tempplate.spec.containers.[n].volumeMounts should also be configured, the volumeMounts.[n].mountPath should be filled with the pod's mount path, e.g. /etc/config, /bitnami etc.
The volumePath.name should also match the pv.metadata.name to let the mountPath knows where the mount volume should it refer to.
Service
In k8s, service is needed to expose network application (usually containers running in pods), e.g. a redmine server.
Manifests
The following objects of service manifest (redmine-service.yaml) are worth noting.
apiVersion: v1
kind: Service
metadata:
annotations:
metallb.universe.tf/address-pool: metallb-address-pool
metallb.universe.tf/loadBalancerIPs: 10.230.0.12
labels:
app: redmine
name: redmine
spec:
ports:
- name: "redmine"
port: 80
targetPort: 3000
selector:
app: redmine
type: LoadBalancer
status:
loadBalancer: {}
The "specs.selector.app" instructs the service to be applied to the pod (usually defined in deployment yaml) which also contains the same label selector. In this case, pod with label app:redmine will be selected and apply this service (with name "redmine")
Identification
Labels and annotations are used to identify pods when cluster becomes sophisticated, labels are typically valued as release, stable etc. which are identifiable and can be selected by selector.
Labels can be multipled
Annotations can mark some informational data such as author, versioning, release contact etc. These are all arbitrary for easy identification
Troubleshooting
1. Reading logs
System wide
Sometimes logs are useful to track where the problem is, use "kubectl get events --sort-by=.metadata.creationTimestamp" or "kubectl describe pods" to do so.
Deployments
kubectl logs deployment/redmine dumps logs of the redmine container (assuming 1 container only in the deployment), and kubectl logs deployment/redmine -c <containerName> for multiple containers
Pods
For single pod logs, use kubectl logs -l app=redmine, to dump logs with designated label
2. Throttling error when trying to retrieve information
When receving "Throttling request took..." while running " kubectl get all -n <ns>" , run " sudo chown -R <user>:<group> .kube/" to solve the issue3. The connection to the server <master>:6443 was refused - did you specify the right host or port? when running any kubectl command
kubectl is unable to access K8S API server port, troubleshoot following the below steps
1. Check KUBECONFIG status by env | grep -i kub
2. Check status of docker service bysystemctl status docker.service
3. Check kubelet service by
systemctl status kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Sun 2023-05-28 13:19:36 HKT; 1s ago
Docs: https://kubernetes.io/docs/home/
Process: 4048306 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
Main PID: 4048306 (code=exited, status=1/FAILURE)
Kubelet unable to initialize, further check system log
4. Check system log by
journalctl -f -u kubelet
where -f means follow and -u to specify particular service
-- Logs begin at Tue 2023-04-25 19:25:06 HKT. --
May 28 13:18:45 jacky-MS-7817-master systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
May 28 13:18:45 jacky-MS-7817-master systemd[1]: Started kubelet: The Kubernetes Node Agent.
May 28 13:18:45 jacky-MS-7817-master kubelet[4048052]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
May 28 13:18:45 jacky-MS-7817-master kubelet[4048052]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
May 28 13:18:45 jacky-MS-7817-master kubelet[4048052]: I0528 13:18:45.187814 4048052 server.go:440] "Kubelet version" kubeletVersion="v1.22.1"
May 28 13:18:45 jacky-MS-7817-master kubelet[4048052]: I0528 13:18:45.188249 4048052 server.go:868] "Client rotation is on, will bootstrap in background"
May 28 13:18:45 jacky-MS-7817-master kubelet[4048052]: E0528 13:18:45.189222 4048052 bootstrap.go:265] part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2022-06-12 17:11:44 +0000 UTC
May 28 13:18:45 jacky-MS-7817-master kubelet[4048052]: E0528 13:18:45.189256 4048052 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
May 28 13:18:45 jacky-MS-7817-master systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
May 28 13:18:45 jacky-MS-7817-master systemd[1]: kubelet.service: Failed with result 'exit-code'.
May 28 13:18:55 jacky-MS-7817-master systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1647394.
Certificate is expired, needs to check and renew the certificate.
sudo kubeadm certs renew all
sudo kubeadm certs check-expiration
which outputs
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
MISSING! certificate the apiserver uses to access etcd
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
and
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf May 27, 2024 05:04 UTC 364d no
apiserver May 27, 2024 05:04 UTC 364d ca no
apiserver-etcd-client May 27, 2024 05:06 UTC 364d etcd-ca no
apiserver-kubelet-client May 27, 2024 05:04 UTC 364d ca no
controller-manager.conf May 27, 2024 05:04 UTC 364d no
etcd-healthcheck-client May 27, 2024 05:04 UTC 364d etcd-ca no
etcd-peer May 27, 2024 05:04 UTC 364d etcd-ca no
etcd-server May 27, 2024 05:04 UTC 364d etcd-ca no
front-proxy-client May 27, 2024 05:04 UTC 364d front-proxy-ca no
scheduler.conf May 27, 2024 05:04 UTC 364d no
indicating successful of certificate renewal
5. Check the system log again, but still fails to initiailize the service
May 28 13:07:32 jacky-MS-7817-master kubelet[4044890]: E0528 13:07:32.789268 4044890 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
May 28 13:07:32 jacky-MS-7817-master systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
May 28 13:07:32 jacky-MS-7817-master systemd[1]: kubelet.service: Failed with result 'exit-code'.
6. Re-initialize k8s control plane node using the renew certs
sudo cp -r /etc/kubernetes /etc/kubernetes-bak #backup k8s folders
sudo rm -rf $HOME/.kube # Remove all configurations in .kube folder
sudo rm -rf $HOME/.kube
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf /root/.kube/config # Copy the admin config back to .kube directory
sudo rm -rf /etc/kubernetes/*.conf
sudo kubeadm init phase kubeconfig all # redo the initialization phase, this will re-generate all kinds of certs
systemctl restart kubelet
The kubelet should back to its running state and kubectl should run successfully by running
systemctl status kubelet.service
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sun 2023-05-28 13:38:05 HKT; 11h ago
Docs: https://kubernetes.io/docs/home/
Main PID: 4052761 (kubelet)
Tasks: 19 (limit: 18434)
Memory: 149.8M
CGroup: /system.slice/kubelet.service
└─4052761 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.4.1
4. The connection to the server localhost:8080 was refused - did you specify the right host or port? when running any kubectl command
This is due to the absence of admin configuration in .kube directory, simply follow the official instruction to copy the admin.conf to the .kube config directory
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
5. Cannot connect to Docker daemon, Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory" when running pipeline in k8s gitlab runner
6. How do I allow private repository in minikube
Step1: Install minikube plugins for private repository and configure ecr
https://minikube.sigs.k8s.io/docs/handbook/registry/
Step2: Configure the deployment yaml
https://minikube.sigs.k8s.io/docs/tutorials/configuring_creds_for_aws_ecr/
7. My minikube load balancer service external IP's status is in "PENDING" state
Step1: Assign external IP by "minikube service <service_name>", a browser window will popup with the accessible IP with external port
8. I made some mistakes in ConfigMap / Secrets, how can I do the update in an already running deployment?
Basically, a more k8s way to update the manifests are creating a new yaml file with -v* to indicate the version, this allow easy rolback to the previous deployment.
e.g. configmap-runner-script-gitlab-runner-v3, copy all the contents of configmap-runner-script-gitlab-runner-v2 and start the modification.
Run
kubectl apply -f ./configmap-runner-script-gitlab-runner-v3.yaml
to apply the configMap once again, by default the changes apply will immediately propagated to all the pods, but it only applies when the configMap itself is mounted as volume, in our case, we need to rollout the statefulset to reflect the configMap changes by 
