Fix gitlab runner error after VM backup restore processes

Background

The gitlab runner service is unable to start after performing VM backup and restore process (for the sake of increasing storage size)

Problems

The error messages come as follows when I try to start the gitlab runner manually

prodev@storage:~$ sudo gitlab-runner run
Runtime platform                                    arch=amd64 os=linux pid=20037 revision=ac8e767a version=12.6.0
Starting multi-runner from/etc/gitlab-runner/config.toml...  builds=0
Running in system-mode.

Configuration loaded                                builds=0
listen_address not defined, metrics & debug endpoints disabled  builds=0
[session_server].listen_address not defined, session endpoints disabled  builds=0
ERROR: Checking for jobs... forbidden               runner=99ab6a69
ERROR: Checking for jobs... forbidden               runner=eac73045
ERROR: Checking for jobs... forbidden               runner=99ab6a69
ERROR: Checking for jobs... forbidden               runner=eac73045
ERROR: Checking for jobs... forbidden               runner=99ab6a69
ERROR: Runner https://repo.gitlab.dev.hkg.internal.p2mt.com99ab6a6972442b2a52ffe7358cbf8f is not healthy and will be disabled!
ERROR: Checking for jobs... forbidden               runner=eac73045
ERROR: Runner https://gitlab.com/eac73045875b618fea0464590a0fb2 is not healthy and will be disabled!

Solutions

1. Rename config.toml in /etc/gitlab-runner to config.toml.bk

2. Restart gitlab service by "gitlab-runner start"

3. Re-name config.toml.bk back to config.toml

4. Re-start the service again by "gitlab-runner restart"

Pipeline failed after changing the desktop machine and upgrading Ubuntu

Background

Recently I have changed my desktop to a mini PC and update the Ubuntu system from v18 to v20, which cause the gitlab runner failed to run.

The annoying part is that it only outputs "Job failed! exit status 1" which provides no clue for my debugging.

Troubleshooting and Solutions

Retrieve more debugging information using "journalctl -u gitlab-runner", which did provides some clues with the following

But it still is not enough for my debugging, I then find out with the help of the reference links, stated that the ".bash_logout" script of the user "gitlab-runner" is the cause of the issue. Every time when the runner use user gitlab-runner and exit (logout) the account, under the logout script, clear_console script 

is being called which gitlab-runner user does not have any privileges. Commenting the clear console script and re-run the runner solved the issue.

References

• https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3849#note_134536963
• https://www.cnblogs.com/zhang-guansheng/p/13429527.html
  
• https://forum.gitlab.com/t/self-hosted-gitlab-ci-runner-shell-executer-fails-immediately-with-error-job-failed-exit-status-1/33252
  
• https://docs.gitlab.com/runner/faq/#run-in---debug-mode
  
• https://unix.stackexchange.com/questions/225401/how-to-see-full-log-from-systemctl-status-service
  

Setting up custom variables in gitlab pipeline

Background

Having a new pipeline needed to setup, I try to put some frequently use variables e.g. AWS login ID to the custom variable convenient for my pipeline usage, but somehow it fails to read the variable 

Reason

Settings problem, I configure the variable as a protected variable, which only allows protected branches or tags to use only, as the target branch is not configured, the custom variable is not recognized

Solutions

We need to configure our target branch as a protected branch in Settings > Repository > Protected Tags, in that section we can input regular expression for the tags identified as "protected" branch, after that, the custom variable is recognized successfully

Multiple accounts management in gitlab

Background

Having 2 different gitlab accounts one is for company and the other one is personal, I would like both work independently on the machine at home, pull source code separately with their corresponding username and repository.

Problems

Very often I encountered a situation with permission denied in my company account while listing the remote repository of do anything remote (like git pull), but I can only make my personal account work without success on my company account. Turns out git client is wrongly picking the my personal account ssh information to login to my working account causing the "permission right" error.

Solutions

1. Refer to ~/.ssh/config and update the config file, add 1 more ssh entry to my company gitlab account as follows

Host anywhere.gitlab.com
  PubkeyAcceptedKeyTypes +ssh-rsa
  HostName gitlab.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id_rsa2
  User ihmcjacky

Host maxwellhk.gitlab.com
  PubkeyAcceptedKeyTypes +ssh-rsa
  HostName gitlab.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id_rsa
  User maxwellhk

Here, we have 2 users, 1 is ihmcjacky for my company usage, another is for my private usage, HostName parameter both specified as gitlab.com, but the Host which is very important, it acts as an alias for git client to identify which configuration it should refer to when trying to login to gitlab. Name anything familiar to you. In my case anywhere.gitlab.com and maxwellhk.gitlab.com.

2. Go to the desired project (in my case, company repository), navigate to .git folder and open config, configure the username and user email manually so as to let git client not to use globally configured settings for log in. Also in remote-origin url settings, configure as follows

url = git@anywhere.gitlab.com:p2Firmware/p2_controller.git

The anywhere.gitlab.com here is important, it is the alias we specified in ~/.ssh/config.

For safety, we can follow References > Multiple SSH keys for different accounts on Github or Gitlab, go through the processes, delete the cached keys and add back the ssh entries and check the ssh connection

3. Finally, add back the new origin URL to the project by using git remote set-url origin new.git.url/here

References

• Git pushing to remote GitHub repository as wrong user
  
• Multiple SSH keys for different accounts on Github or Gitlab
  
• Configure SSH to a different directory
  
• https://medium.com/uncaught-exception/setting-up-multiple-gitlab-accounts-82b70e88c437
  

Configuring Gitlab account in Linux & Windows

Background

I need to create a git working environment on my Windows 10 installed desktop, I then setup the git account, add the ssh key of the machine through the gitlab control panel but gitlab replied me with no access right when trying to git pull my project.

I tried

• Re-generate the RSA key and load again

But not working...

Solution

Turn out the username and the user account must match the one created in gitlab, git client will make use that account for the verification process with git server. I misunderstand that the user account can be arbitrary created on the fly.

Procedure

1. Configure the username and user account (MUST match with the one in gitlab / github, skip the --global parameter if you are managing multiple users)
• git config --global user.name "yourname"
• git config --global user.email“your@email.com"
2. Remove known_hosts (may not be necessary, it depends)
3. ssh-keygen -t rsa -C "whateverName"
4. Copy all text in /Users/your_user_directory/.ssh/id_rsa.pub
5. Go to setting in gitlab and add the public key for authentication
6. Set the git remote url with "git remote set-url origin https://gitUsername:gitPassword@gitlab.com/yourRepo.git"

Points to Note

Sometimes, login still failed with

sign_and_send_pubkey: signing failed: agent refused operation
Permission denied (publickey).
fatal: Could not read from remote repository.

Due to ssh-agent cannot find the keys attached, in such case we need to add the private key identities by running command

ssh-add

repeat step 6 and the remote origin may now be read


But in case... If it sill fails to log in for some error messages like no access rights to the repo, navigate to ~/.ssh/config, add the entry as follows.

Note: You can also specify multiple IdentityFile entries if you have same hostname using different key pairs, just add 1 more IdentityFile entry would be done, remember, you should put the private key in the IdentityFile option

Host gitlab.com
  PubkeyAcceptedKeyTypes +ssh-rsa
  HostName gitlab.com
  IdentityFile ~/.ssh/id_rsa_anywhere /*Private key directory here!!*/
  User git

And use all the possible keys for login

ssh git@gitlab.com -vv

After the configuration, gitlab should be able to login and fetch updates again

References

• https://gitlab.com/gitlab-com/support-forum/-/issues/638#note_32211207
• https://blog.csdn.net/jingtingfengguo/article/details/51892864
• https://stackoverflow.com/questions/4220416/can-i-specify-multiple-users-for-myself-in-gitconfig
• https://askubuntu.com/questions/762541/ubuntu-16-04-ssh-sign-and-send-pubkey-signing-failed-agent-refused-operation
• https://dotblogs.com.tw/Echo/2017/06/28/Linux_AuthorizedKeys_Setting
  

Gitlab email notification related to pipeline

Gitlab has email notification on various events such as pipeline failure / success, commitment etc. Unfortunately, the setting itself is confusing which make it hard to configure.


My Wish
When I tag a release, pipeline runs, after that send the pipeline results to the person I want.

You can see only a small request takes me half day to figure out where the configuration is. Here is how it can be achieve and the misleading configurations.

You may think that the notification should be well configured via "project > notifications", then you are wrong, the desired recipients will never receive the message, because that setting means ONLY AUTHOR will receive the pipeline status changes. I am quite surprise, not because of the result, but because of the lack of explanation regarding on the configurations.

So how can we achieve to send the email notification BESIDES AUTHOR? You may need to navigate to "Project > Settings > Integrations", enable "Pipelines emails", enter the desired email addresses and gotcha.

What a genius configuration settings, I really hope that software designers can design software based on "human beings" but not only for engineering mindset. It is unacceptable for a task being so complicated to accomplish.

Configuring git-lab runner in Windows 10 Home

Background
One of a routine task during my development flow is deployment, this is quite messy and time consuming, especially when the system consist of many sub-systems which may require testing and runner to run in different machines. In my case, I have developed a new feature which send email using localhost through PHP, and I have tested this feature using docker in my Windows 10 laptop. Unfortunately, Windows 10 Home does not featured with hyper-V, it is an essential element to run native Docker. I therefore make use of docker toolbox and virtual box combination as the runner of my Lenovo laptop.

Environment
- Lenovo ThinkPad X260
- Windows 10 Home
- Gitlab runner installed
- Docker toolbox installed
- Virtual box installed

Problems
All the installation processes of different components performed successfully, however when it comes to some setup, problem comes by.

- When the job is running and using the runner, the following error occurs

error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.29/version: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.

This indicates the docker engine is not running at the host machine. I feel strange as docker toolbox has already installed and well configured. The problem comes from the config.toml, this is the configuration file to setup the runner

concurrent = 1

check_interval = 0

[[runners]]

  name = "lenovo_laptop_jackylam"

  url = "https://repo.gitlab.dev.internal.pm.com/"

  token = "b4032157c339f90523a77e5e8881a7"

  executor = "docker"

  [runners.docker]

    tls_verify = true

    host = "tcp://192.168.99.100:2376"

    tls_cert_path = "C:\\Users\\p2m\\.docker\\machine\\certs\\"

    image = "test/2938:latest"

    pull_policy = 'never'

    privileged = true

    disable_cache = false

    volumes = ["/cache"]

    shm_size = 0

  [runners.cache]

Here, executor is important, you need to told gitlab-runner to use docker as the runner, but the problem also come from this, when you say "executor: docker", you are telling the specified host machine to run docker command for the test, the problem is you are using virtual box as the intermediate to run the docker, and the "Docker Quickstart Terminal" 's environment is linked to the virtual machine, that's why you can directly run docker command through the host terminal. When gitlab-runner run the docker command, the environment does not contain docker executable!

To solve this issue, you can rely on config.toml, in [runners.docker] column, specify the host with the virtual machine's host, and also the certifications required to run the command, another important setup would be "pull_policy", setting "never" tells gitlab-runner not to pull the docker images from the docker hub, it is extremely useful when your test is not replying on the docker images situated in docker hub. After setting up the remote connection pointing to the virtual machine, the docker command can be run successfully.

- When you want to remove the docker-machine, make sure the virtual machine in virtual box is completely shutdown before removing docker-machine, otherwise error will occur during removal.

Remember if you don't specify anything, gitlab-runner will assume the host machine can be "docker-runnable", but actually the runner is located in the VM.