顯示具有 runner 標籤的文章。 顯示所有文章
顯示具有 runner 標籤的文章。 顯示所有文章

2021年9月23日 星期四

Pipeline failed after changing the desktop machine and upgrading Ubuntu

Background

Recently I have changed my desktop to a mini PC and update the Ubuntu system from v18 to v20, which cause the gitlab runner failed to run.








The annoying part is that it only outputs "Job failed! exit status 1" which provides no clue for my debugging.


Troubleshooting and Solutions

Retrieve more debugging information using "journalctl -u gitlab-runner", which did provides some clues with the following





But it still is not enough for my debugging, I then find out with the help of the reference links, stated that the ".bash_logout" script of the user "gitlab-runner" is the cause of the issue. Every time when the runner use user gitlab-runner and exit (logout) the account, under the logout script, clear_console script 






is being called which gitlab-runner user does not have any privileges. Commenting the clear console script and re-run the runner solved the issue.


References

2017年8月17日 星期四

Configuring git-lab runner in Windows 10 Home

Background
One of a routine task during my development flow is deployment, this is quite messy and time consuming, especially when the system consist of many sub-systems which may require testing and runner to run in different machines. In my case, I have developed a new feature which send email using localhost through PHP, and I have tested this feature using docker in my Windows 10 laptop. Unfortunately, Windows 10 Home does not featured with hyper-V, it is an essential element to run native Docker. I therefore make use of docker toolbox and virtual box combination as the runner of my Lenovo laptop.

Environment
- Lenovo ThinkPad X260
- Windows 10 Home
- Gitlab runner installed
- Docker toolbox installed
- Virtual box installed

Problems
All the installation processes of different components performed successfully, however when it comes to some setup, problem comes by.

- When the job is running and using the runner, the following error occurs
error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.29/version: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.

This indicates the docker engine is not running at the host machine. I feel strange as docker toolbox has already installed and well configured. The problem comes from the config.toml, this is the configuration file to setup the runner

concurrent = 1
check_interval = 0

[[runners]]
  name = "lenovo_laptop_jackylam"
  url = "https://repo.gitlab.dev.internal.pm.com/"
  token = "b4032157c339f90523a77e5e8881a7"
  executor = "docker"
  [runners.docker]
    tls_verify = true
    host = "tcp://192.168.99.100:2376"
    tls_cert_path = "C:\\Users\\p2m\\.docker\\machine\\certs\\"
    image = "test/2938:latest"
    pull_policy = 'never'
    privileged = true
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]

Here, executor is important, you need to told gitlab-runner to use docker as the runner, but the problem also come from this, when you say "executor: docker", you are telling the specified host machine to run docker command for the test, the problem is you are using virtual box as the intermediate to run the docker, and the "Docker Quickstart Terminal" 's environment is linked to the virtual machine, that's why you can directly run docker command through the host terminal. When gitlab-runner run the docker command, the environment does not contain docker executable!

To solve this issue, you can rely on config.toml, in [runners.docker] column, specify the host with the virtual machine's host, and also the certifications required to run the command, another important setup would be "pull_policy", setting "never" tells gitlab-runner not to pull the docker images from the docker hub, it is extremely useful when your test is not replying on the docker images situated in docker hub. After setting up the remote connection pointing to the virtual machine, the docker command can be run successfully.

- When you want to remove the docker-machine, make sure the virtual machine in virtual box is completely shutdown before removing docker-machine, otherwise error will occur during removal.

Remember if you don't specify anything, gitlab-runner will assume the host machine can be "docker-runnable", but actually the runner is located in the VM.