[Apache] Updating http server to https

Background

No one will delay the importance of https, to secure the data transaction between client and server, however in the past when https is not very common in Internet, we developers suffer from registration cost of a certificate and complicated setup of Apache server. One of our client although not specifically request, needs https at all time of their official web site.

Solutions

Steps are not difficult nowadays to complete the https setup, here is my system setup. 

- A LAMP docker image (mattrayner/lamp:latest-1804) https://hub.docker.com/r/mattrayner/lamp which has already setup and work in production.

- Amazon lightsail service for hosting

- A valid hostname registered in hosting speed with full control of the domain name through the domain name panel

Steps

1. Register the SSL certificate (FREE) in letsencrypt (https://letsencrypt.org/) through lightsail web terminal or any ways you can think of accessing the virtual server. Install software-properties-common and certbot accordingly (Details refer to https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-using-lets-encrypt-certificates-with-wordpress)

2. Specify the domain name and the wildcard in the environment variable, use certbot to request Let's Encrypt for the new certificate.

3. Use the following command to start certbot in interactive mode, follow the instruction to complete the registration.

sudo certbot -d $DOMAIN -d $WILDCARD --manual --preferred-challenges dns certonly

Note: There is a process where Let's Encrypt verifies the ownership of the domain, and you as the domain owner needs to add a TXT DNS record to complete the challenge. I am stucked at this as I wrongly follow the unclear instructions of Let's Encrypt's instruction. The TXT record required to fill in your DNS panel is _acme-challenge.example.com with a series of long string, but I wrongly put the whole address in the below table which caused failure of the challenges. It indeed needs only the first part "_acme-challenge" since the .goldenthumb.com.hk has already added for you during the DNS enquiry, so NO NEED to put the whole address to the name field of the DNS record panel.

4. Complete the challenges in the interactive shell and your certificate will be issued. Mark down the directory in which the certificates are stored

5. Update the docker-compose file volumes configuration so that volume is mapped to include the certificate files in the docker container, they will be used as https's certificate afterwards, at the same time, add 443:443 port mapping in ports configuration to allow correct functions of https

6. You can also map the path /etc/apache2/sites-available to local for easy access and configure the apache web server settings

7. Update the 000-default.conf to read the certificate files in step 5, the file sample is as follows

https://drive.google.com/file/d/1usQ9kHb38SyCxW8oqYT1fMAja1_Xj54S/view

The sample configuration includes pointing the certificate files, setting up 443 port based virtual server, redirecting all non https request permenantly to the https URL

8. Update the docker-compose file again, add build configuration and remove image configuration, because we are adding custom commands / scripts to the new yml file

9. Create a new dockerFile with source using the LAMP container (mattrayner/lamp:latest-1804), add the following custom commands in the dockerFile yml

- a2enmod ssl

- service apache2 restart

10. Navigate to lightsail management panel, open the port inbound for port 443 used in https

11. Stop the running containers, rebuild and make the containers up again, your server is now https protected

Certificate Renewal

1. Run "/usr/bin/certbot renew >> /var/log/certbot"

2. Reply the interactive terminal, provide emails, agree terms etc.

3. Add DNS record (TXT) for the challenges  

4. Create a file to accept the second challenge  

5. Restart the web server

Congrats, certificate renewed

References

- Nginx and Let’s Encrypt with Docker in Less Than 5 Minutes

- SSL/TLS Strong Encryption: How to
- Using Multiple SSL Certificates in Apache with One IP Address
- Apache Redirect to HTTPS
- Tutorial: Using Let’s Encrypt SSL certificates with your WordPress instance in Amazon Lightsail

- Cert renewal

- .well-known attached to domain name, slash missing

- An authentication script must be provided with –manual-auth-hook

Dot env file is not found in docker compose file

Background

So there is a case I need to substitue the variables defined in .env to the docker compose for further process, but when I run the docker-compose up command, the variable refer in docker compose file cannot be interpreted.  

Reason

Normally the .env file sits to the same directory as the docker compose yml file, and is expected the docker compose command being executed in the current directory of the docker compose file and the .env file, which I am not in this case.

Solutions

If the current directory is not the place where the compose file is sitting in, we need to specify --project-directory to allow docker-compose program to be able to recognize the directory where the yml file and the .env file sits in, the correct command after fixing is as follows

docker-compose -f {docker-compose-file-dir} --project-directory={directory-of-docker-compose-file} up -d

Setting up custom variables in gitlab pipeline

Background

Having a new pipeline needed to setup, I try to put some frequently use variables e.g. AWS login ID to the custom variable convenient for my pipeline usage, but somehow it fails to read the variable 

Reason

Settings problem, I configure the variable as a protected variable, which only allows protected branches or tags to use only, as the target branch is not configured, the custom variable is not recognized

Solutions

We need to configure our target branch as a protected branch in Settings > Repository > Protected Tags, in that section we can input regular expression for the tags identified as "protected" branch, after that, the custom variable is recognized successfully

Unable to deploy webpage through docker of AWS lightsail instance

Background

So I am trying to build and deploy a very simple LAMP application through docker, but encounters difficulties of repeatedly restarting mySQL server with the following errors

mysqld entered running state exit status 0 expected

It repeatedly restarts and never stops until I turn off the docker services. I am annoying on this because deploying a LAMP based web site through docker should be short and shouldn't take too much time.

Reason

So I finally figure out I can investigate through mySQL log through /var/log/mariadb/mariadb.log, you can determine the exact log path in the my.cnf file located in /etc/my.cnf

And this shows up as 1 of the log

innodb: cannot allocate memory for the buffer pool

Bingo ! So memory insufficient is the cause

Solutions

As I am using AWS Lightsail, the cheapest plan with only 512mb ram, and the remaining RAM is just several MBs, which is totally not enough for running all 2 containers (lamp docker container & ftp).

The I quickly think of swap spaces, which use fast SSD as memory, following URL's guidance, the LAMP docker service can be run successfully 

References

• How To Add Swap on Ubuntu 14.04
• Mysql cannot allocate memory for the buffer pool
• How do I allocate memory to work as swap space in an Amazon EC2 instance by using a swap file?
• Amazon Linux 2 - install docker and docker-compose
• How to increase swap space?
  
  

[Linux & cli] Useful Linux command

Details

The following are some useful commands which usually made use and always being forgotten when needed, here is a note down.

• Edit /etc/fstab to mount new drives (files system, mount point, type, options, dump, pass)
• /dev/sdb1   /home/jacky/Documents/2tb-hdd   ext4   defaults  0   0
• mount -av (run from root, show the mount point)
• Run command and pipe all output to file
• do_something 2>&1 | tee -a some_file (2>&1 is to direct all stderr to stdout)
• Clear all text in a text file
• > someTxtFile.txt
• Kill process occupied by a known port (e.g. 32769)
• sudo kill -9 $(sudo lsof -t -i:32769)

References

• Reload fstab
  
• Linux Tutorial - 11. Piping and Redirection
  
• direct-stderr-and-stdout-in-bash
  
• Understanding crontab
  

[Chrome & Debug] Create and debug with test snippets in Chrome

Background

Very often it would be convenient and time consuming to have a fast setup and runnable js snippet playground to quickly test for the logic or debug etc.

Solutions

1. Open chrome

2. Right click "inspect" or press F12 shortcut

3. Navigate to "Sources"

4. At the left hand side panel, click the double right arrow, click "New snippet"

5. A snippet panel pops and you can freely write the test code

6. Press "Ctrl + Enter" to execute 

Frequently used git command

Details

There are some cases I need to use some specific git commands for some special purposes, like 

• Undoing git merge which haven't be pushed
• git reflog (Get the commit desired to return)
• git reset --hard sha
  
  
• Cherry-picking specific file changes from 1 commit to another
• git checkout <branch or sha of commit> -- filename
  
  
• Retrieve just 1 file from specific branch
• git checkout experiment -- app.js
  
  
• Show content changes which have been staged
• git diff --cached
  
  
• Show the diff of the same file between different commits of the same branch
• git diff HEAD^^ HEAD someFile (Show differences of someFile 2 commits before HEAD)
  
  
• Show file changes on specific commit (without diff information)
• git show --pretty="" --name-only {commit-sha}
  
  
• Show commits differences between 2 branches since branch created (file list)
• git log --oneline master..myBranch (Show changes since master branch)
• git log --oneline --no-merges master..myBranch (Show changes since master branch skipping merge commits)
  
  
• Get a list of branches created ordered with create date
• git branch --sort=-committerdate  # DESC
• git branch --sort=committerdate  # ASC
  
  
• Commit with standard input
• echo -e 'Update version number to v1.6.72.0\n\nfrom v1.6.70.0 to v1.6.72.0' | git commit /home/jacky/Documents/gitproj/p2_controller/docker/docker-compose-prod.yml /home/jacky/Documents/gitproj/p2_controller/docker/docker-compose-prod-upgrade.yml /home/jacky/Documents/gitproj/p2_controller/p2_controller/static_data/VERSION -F -

References

• Undo a Git merge that hasn't been pushed yet
• show commits since branch creation
• How to list all the files in a commit?
• How to cherry-pick changes from one file to another file?
• View git log without merge commits
• how-do-i-diff-the-same-file-between-two-different-commits-on-the-same-branch
• How to get just one file from another branch
• How can I get a list of Git branches, ordered by most recent commit?
  
• How do I show the changes which have been staged?