Renew kubernetes (k8s) certificates

Background

In 12th Dec 2022, the certificates of all the k8s components expired, which cause the failure to perform CI tasks in the company software builds. However after updating the certificates following the wiki of redmine, I still cannot make the CI work. Thus restarting the master node (IP: 10.240.0.59).

Problem comes even worse, the kubelet service is not start up after reboot, all k8s services containers are not running. 

Troubleshooting

I try to find the logs for errors, but is somehow problematic. Before restarting the master node, I try the following command to retrieve logs of the gitlab runner pod, but seems like it is not working because there are 2 containers within this pod, k8s needs us to specify particular container to read the logs.

kubectl logs -f -n gitlab runner-yngqxnkk-project-737-concurrent-0zwbtj

Then I try to get the names of the containers within the pod with

kubectl describe pod -n gitlab runner-yngqxnkk-project-737-concurrent-0zwbtj | more

However I cannot get any information regarding to the container names. Then I use

kubectl get pods -n gitlab runner-yngqxnkk-project-737-concurrent-0zwbtj -o jsonpath='{.spec.containers[*].name}'

And then it list out "build helper", which is the name of the containers in the pod, I further run

kubectl logs -f -n gitlab runner-yngqxnkk-project-737-concurrent-0zwbtj -c helper

But nothing is output, my guess is that the containers are not even running anything (since pod is in pending state), therefore no container logs are being outputted. 

After that I decided restarting the whole master node (10.240.0.59), but sadly all kubelet service is not starting up, I again try to troubleshoot system log by

systemctl status kubelet -l

journalctl -xeu kubelet

And found out some useful log...

kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

So the bootstrap-kubelet.conf really is not existed, and that caused the inability of start the kubelet service.

Solutions

First update the certificates for ALL master nodes. (my case: 10.240.0.59 / 51 / 52). The command to check the certificate expiration is 

sudo kubeadm alpha certs check-expiration

This command list all k8s components' certificate status, and let you know whether they are expired or not. Renew the certificates with 

sudo kubeadm alpha certs renew all

Check the status with the first command again, the date should be updated +1 year. Then manually renew the certificate following the commands below

$ cd /etc/kubernetes/pki/

$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/

$ kubeadm init phase certs all --apiserver-advertise-address <IP>

$ cd /etc/kubernetes/

$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/

$ kubeadm init phase kubeconfig all

$ reboot

The apiserver-advertise-address is the node IP (in my case 10.240.0.59). The above script basically backup the current keys (in case any problems occur), and re-initialize the cert generation phase, backup all the files in "/etc/kubernetes" before re-initializing kubeconfig phase.

After reboot, copy the admin.conf generated in kubeconfig generation phase to the kube configuration home directory

cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

Check the status of the kubelet service by

sudo systemctl status kubelet.service

References

• Renew kubernetes pki after expired
  
• kubelet.service won't start
  
• change master ip address --apiserver-advertise-address
  
• Kubernetes / kubectl - "A container name must be specified" but seems like it is?

Share resources to 3rd party with link in Office 365

Problems

A requirement to share some certificates and / or documents to the 3rd parties, notorious office 365 complex settings makes the whole configuration difficult.

Solutions

1. Navigate to Microsoft 365 admin center

2. Navigate to "Teams & groups", "Activate teams & groups"

3. Navigate to "Security" tab

4. Create a share group (if you don't have one) and add all desired members to the group

5. Navigate to "Sharepoint" 

6. Navigate to "Policies > Sharing", expand "More external sharing settings", check "Allow only users in specific security groups to share externally", click "Manage security groups", add the group created in step 4.

7. Drag the share bar of the sharepoint to "most permissive"

8. Navigate to "Sites", create or edit existing sites, select the target site, select "Policies" tab, in "External sharing", select "Files and folders on this site can be shared with anyone". Then select "Permissions" tab, add the security group created in step 4 and all single members as site admin.

9. Share the folders or files in sites (by right clicking target entries) and select "Copy link", then "Anyone with the link can view"

References

• Sharing options are greyed out when sharing from SharePoint Online or OneDrive
• Manage security groups
  

Docker compose .env confusion

Background

I want to specify the location of .env file for variable substitution of docker-compose yml file, there is a need to separate .env, .production.env for different usage scenarios.

Problems

Some responses in stackoverflow quotes "env_file" keyword and --env-file tag which are not the same thing as .env file, env_file and --env-file tag only pass environment variables to containers, but not variable substitutions of docker-compose file, they are totally different things.

Solutions

cat the env file content to the docker compose file to read contents of .env with custom name. E.g.: alt.env

env $(cat alt.env) docker-compose up --build

References

- https://stackoverflow.com/a/67495905/2361494

Adding body parser in loopback JS

Problems

By default loopback JS does not include body parser although it is based on Express JS, and official documentation does not mention how to configure in order to add the body parser feature.

Solutions

Update $APP_HOME/server/middleware.json and add the body parser block

"parse": {

    "body-parser#json": {},

    "body-parser#urlencoded": {"params": { "extended": true }}

}

References

- How can I use body-parser with LoopBack?

 

Install dedicated AMD drivers for laptop S540

Problems

Laptop S540 has dedicated AMD 4800u integrated graphics, at first, I downloaded the driver from the lenovo official site but recently I want to upgrade the graphics card following AMD's official driver, which is more update, but everytime after the upgrade, Windows 11 automatically revert the old version driver making the new AMD driver being warned with not compatiable.

Solutions

The whole concept is to prevent Windows from checking and changing the driver automatically.

1. Navigate to Device Properties and find the AMD graphics hardware, click `XXX Grahpics Properties`  

At the same time, open Local Group Policy Editor and navigate to Administrative Templates > System > Device Installation > Device Installation Restrictions, find and edit the below entry

Enable the settings above with the entry values shown in Device Properties.

Apply the settings for each driver upgrade, disable the settings next time when a new driver update is needed

Retrieve tag from Apple Music using mp3tag

Problems

Finding tags for ripped music files sometimes are pain because the source is quite difficult to find. Ever since Amazon forbids the free use to retrieve tag of mp3tag, Apple Music becomes a new source for tags.

Solutions

1. Download and install mp3tag

2. Install the Japanese UI language (optional), unzip, copy all content to %PROGRAMFILES%\Mp3tag\lang

3. Navigate to %appdata%\Mp3tag\data\sources\Mp3tag\data\sources, copy all the content of files to this directory

4. Restart mp3tag

5. Drag music files / directories to the table

6. Right click and find tag using the newly added sources

References

1. Apple Music for MP3Tag

Configure chrome remote desktop in Ubuntu 20.04 (Updated June 2025)

Problems

Chrome remote desktop creates a new session for remote session, which is not suit my needs, like Windows's remote, I need to "resume" the current session of the original desktop.

Solutions

1. In the host Ubuntu machine, install chrome remote desktop, using the deb provided by Google making use of the command sudo dpkg -i chrome-remote-desktop_current_amd64.deb

2. In the host machine, install chrome remote desktop application, perform the remote host setup following the instructions until you see the remote device appeared in the remote list.

You should see status "Online" in the device list, if it does not,

• 
  
  

• Check whether the chrome remote desktop status is on by sudo systemctl status chrome-remote-desktop. It should output as follows

• There are cases where "Loaded: masked (Reason: Unit chrome-remote-desktop.service is masked.)" error occurs, if so, run sudo rm /lib/systemd/system/your_app.service  to remove the systemd service of the chrome-remote-desktop and reload the systemd daemon by running sudo systemctl daemon-reload. Re-check the status of the chrome-remote-desktop and active status should be achieved

3.  Modify the file /opt/google/chrome-remote-desktop/chrome-remote-desktop,  change variable FIRST_X_DISPLAY_NUMBER to the output value of the command echo $DISPLAY.

4. Apply the differences manually for the remaining changes mentioned here. Do not use the patch command to patch the file as it may cause failure.

5. Create Xsession file under the config profile of chrome remote desktop using the following command. echo "1" > ~/.config/chrome-remote-desktop/Xsession, where "1" is the $DISPLAY value output mentioned in step #3. (value may vary depending on devices)

6. Restart the chrome remote desktop daemon with sudo systemctl daemon-reload followed by sudo /etc/init.d/chrome-remote-desktop restart. Re-check the chrome remote desktop status by sudo systemctl status chrome-remote-desktop.

Appendix

1. This method does not support wayland, only support Xorg, make sure wayland support is disabled. Edit file /etc/gdm3/custom.conf, make sure WaylandEnable parameter is configured to false

2. When activating through chrome remote desktop UI is not working (e.g.: Unknown error), try using headless enable, navigate to the chrome remote desktop in the remote client, in the left hand side menu, choose "configure through SSH", copy the command provided

DISPLAY= /opt/google/chrome-remote-desktop/start-host --code="4/0AdQt8qgifMrurEQn65gbwYJOA3VrpusSF64i8Fwxfl9Nn6uL7L2KLUdw75HXhpFqlfGG-w" --redirect-url="https://remotedesktop.google.com/_/oauthredirect" --name=$(hostname)

You may need to re-generate the key if the time is expired, copy and paste it to the remote host, follow the setup netvigation to complete the remote host setup

Updated: 4/6/2024

As of chrome remote desktop version v121+,

1. Download the patch file

2. Remove the first 4 lines of the patch file

3. Navigate to /opt/google/chrome-remote-desktop, apply the patch by "sudo patch -Np1 -i /path/to/use_existing_session.patch", update the FIRST_X_DISPLAY_NUMBER variable to the one output using "echo $DISPLAY", and also update value of ~/.config/chrome-remote-desktop/Xsession to the DISPLAY value

4. Update the XSession to disable wayland, set WaylandEnable to false, run "sudo systemctl restart gdm3" to take effect.

5. Re-login Ubuntu (re-configure the chrome remote desktop where possible) "DISPLAY= /opt/google/chrome-remote-desktop/start-host --code="4/0AdLIrYeNopk9TMNaSyMPAXCr25kRpDPi5oUt9w7IJkC5ALTp75JlaKajanli3b3XQ_RDlw" --redirect-url="https://remotedesktop.google.com/_/oauthredirect" --name=$(hostname)"

Patch file backup

Updated: 30/6/2025

As of latest "Ubuntu chrome remote" current session solution, just follow the guide to apply the update of file "/opt/google/chrome-remote-desktop", following the instruction marked here , (backup as following)

rm ~/.config/chrome-remote-desktop/Xsession
echo $DISPLAY | tr -d ':' >> ~/.config/chrome-remote-desktop/Xsession && cd /opt/google/chrome-remote-desktop
sudo cp chrome-remote-desktop chrome-remote-desktop_bak
sudo wget -qO- "https://gist.githubusercontent.com/nightuser/2ec1b91a66ec33ef0a0a67b6c570eb40/raw/b09250dcc9a28f09de93192a948dffb4963c2aca/use_existing_session.patch" | tail -n +5 > ~/use_existing_session.patch
sudo patch -Np1 -i ~/use_existing_session.patch
rm ~/use_existing_session.patch
systemctl restart chrome-remote-desktop@$USER.service
systemctl enable chrome-remote-desktop@${USER}.service
systemctl status chrome-remote-desktop@$USER.service

Path file

References

1. Use existing session chrome remote desktop patch (Note: the .patch file cannot applied directly to the origin "/opt/google/chrome-remote-desktop/chrome-remote-desktop" , manual applying the differences is necessary.)
2. How to Install chrome-remote-desktop on Arch
   
3. systemd: How to unmask a service whose unit file is empty
   
4. Disable wayland support
   
5. Sample configuration files
• /etc/gdm3/custom.conf
• /opt/google/chrome-remote-desktop/chrome-remote-desktop