Dot env file is not found in docker compose file

Background

So there is a case I need to substitue the variables defined in .env to the docker compose for further process, but when I run the docker-compose up command, the variable refer in docker compose file cannot be interpreted.  

Reason

Normally the .env file sits to the same directory as the docker compose yml file, and is expected the docker compose command being executed in the current directory of the docker compose file and the .env file, which I am not in this case.

Solutions

If the current directory is not the place where the compose file is sitting in, we need to specify --project-directory to allow docker-compose program to be able to recognize the directory where the yml file and the .env file sits in, the correct command after fixing is as follows

docker-compose -f {docker-compose-file-dir} --project-directory={directory-of-docker-compose-file} up -d

Setting up custom variables in gitlab pipeline

Background

Having a new pipeline needed to setup, I try to put some frequently use variables e.g. AWS login ID to the custom variable convenient for my pipeline usage, but somehow it fails to read the variable 

Reason

Settings problem, I configure the variable as a protected variable, which only allows protected branches or tags to use only, as the target branch is not configured, the custom variable is not recognized

Solutions

We need to configure our target branch as a protected branch in Settings > Repository > Protected Tags, in that section we can input regular expression for the tags identified as "protected" branch, after that, the custom variable is recognized successfully

Unable to deploy webpage through docker of AWS lightsail instance

Background

So I am trying to build and deploy a very simple LAMP application through docker, but encounters difficulties of repeatedly restarting mySQL server with the following errors

mysqld entered running state exit status 0 expected

It repeatedly restarts and never stops until I turn off the docker services. I am annoying on this because deploying a LAMP based web site through docker should be short and shouldn't take too much time.

Reason

So I finally figure out I can investigate through mySQL log through /var/log/mariadb/mariadb.log, you can determine the exact log path in the my.cnf file located in /etc/my.cnf

And this shows up as 1 of the log

innodb: cannot allocate memory for the buffer pool

Bingo ! So memory insufficient is the cause

Solutions

As I am using AWS Lightsail, the cheapest plan with only 512mb ram, and the remaining RAM is just several MBs, which is totally not enough for running all 2 containers (lamp docker container & ftp).

The I quickly think of swap spaces, which use fast SSD as memory, following URL's guidance, the LAMP docker service can be run successfully 

References

• How To Add Swap on Ubuntu 14.04
• Mysql cannot allocate memory for the buffer pool
• How do I allocate memory to work as swap space in an Amazon EC2 instance by using a swap file?
• Amazon Linux 2 - install docker and docker-compose
• How to increase swap space?
  
  

[Linux & cli] Useful Linux command

Details

The following are some useful commands which usually made use and always being forgotten when needed, here is a note down.

• Edit /etc/fstab to mount new drives (files system, mount point, type, options, dump, pass)
• /dev/sdb1   /home/jacky/Documents/2tb-hdd   ext4   defaults  0   0
• mount -av (run from root, show the mount point)
• Run command and pipe all output to file
• do_something 2>&1 | tee -a some_file (2>&1 is to direct all stderr to stdout)
• Clear all text in a text file
• > someTxtFile.txt
• Kill process occupied by a known port (e.g. 32769)
• sudo kill -9 $(sudo lsof -t -i:32769)

References

• Reload fstab
  
• Linux Tutorial - 11. Piping and Redirection
  
• direct-stderr-and-stdout-in-bash
  
• Understanding crontab
  

[Chrome & Debug] Create and debug with test snippets in Chrome

Background

Very often it would be convenient and time consuming to have a fast setup and runnable js snippet playground to quickly test for the logic or debug etc.

Solutions

1. Open chrome

2. Right click "inspect" or press F12 shortcut

3. Navigate to "Sources"

4. At the left hand side panel, click the double right arrow, click "New snippet"

5. A snippet panel pops and you can freely write the test code

6. Press "Ctrl + Enter" to execute 

Frequently used git command

Details

There are some cases I need to use some specific git commands for some special purposes, like 

• Undoing git merge which haven't be pushed
• git reflog (Get the commit desired to return)
• git reset --hard sha
  
  
• Cherry-picking specific file changes from 1 commit to another
• git checkout <branch or sha of commit> -- filename
  
  
• Retrieve just 1 file from specific branch
• git checkout experiment -- app.js
  
  
• Show content changes which have been staged
• git diff --cached
  
  
• Show the diff of the same file between different commits of the same branch
• git diff HEAD^^ HEAD someFile (Show differences of someFile 2 commits before HEAD)
  
  
• Show file changes on specific commit (without diff information)
• git show --pretty="" --name-only {commit-sha}
  
  
• Show commits differences between 2 branches since branch created (file list)
• git log --oneline master..myBranch (Show changes since master branch)
• git log --oneline --no-merges master..myBranch (Show changes since master branch skipping merge commits)
  
  
• Get a list of branches created ordered with create date
• git branch --sort=-committerdate  # DESC
• git branch --sort=committerdate  # ASC
  
  
• Commit with standard input
• echo -e 'Update version number to v1.6.72.0\n\nfrom v1.6.70.0 to v1.6.72.0' | git commit /home/jacky/Documents/gitproj/p2_controller/docker/docker-compose-prod.yml /home/jacky/Documents/gitproj/p2_controller/docker/docker-compose-prod-upgrade.yml /home/jacky/Documents/gitproj/p2_controller/p2_controller/static_data/VERSION -F -

References

• Undo a Git merge that hasn't been pushed yet
• show commits since branch creation
• How to list all the files in a commit?
• How to cherry-pick changes from one file to another file?
• View git log without merge commits
• how-do-i-diff-the-same-file-between-two-different-commits-on-the-same-branch
• How to get just one file from another branch
• How can I get a list of Git branches, ordered by most recent commit?
  
• How do I show the changes which have been staged?

[Docker] Docker run emits "name is already in use" error

Background

When trying to docker run through docker compose, name is already in use appeared.

Solutions

Using docker ps -a, we will find there are docker containers created with the same name, it may due to docker compose stop not cleanly close all the containers.

To solve, find out, we can try to close all the running containers (you can find which one's name is duplicated, but I want to make life easier), docker stop $(docker ps -a -q), re-run the docker compose script again

References

• docker run --> 'name is already in use by container'
• https://coderwall.com/p/ewk0mq/stop-remove-all-docker-containers
• Docker error bind address already in use

[CSS] Meaning of margin:auto

Background

Sometimes we need to fulfill situation which we want to line up several components in a row and the last component leans on the right most of the div

Solutions

This comes to marginLeft/Right: auto, when we try to mark the container component with margin-left: auto, the container takes all the spaces available to the left hence position the switch button at the right most of the container.

That explains why when we left / right margin the DIV, we can center it at the center.

References

• What does margin auto mean?
  
• What is the meaning of `auto` value in a CSS property.
  

[JS] Onclick listener confusion in div inside div

Background

Sometimes we face situations when we have 2 layers of components (e.g. DIVs), while original (outer) div already configured with onclick listener but on top of the parent div, we also want the child div to have the same onClick listener.

Needless to say, the onclick action listener will be at the top will be fired first, and the parent one (underneath), which is not what we want, we want only child one to be detected and fired events

Solutions

This is related to event bubbling and event capturing. For event bubbling, clicking the inside DIV will trigger the event outside DIV, we need to stop both bubbling event and capturing event by e.stopPropagation and e.cancelBubble

References

• Howto: div with onclick inside another div with onclick javascript
• Binding and event handler — passing the event object
  
• React prevent event bubbling in nested components on click

Multiple accounts management in gitlab

Background

Having 2 different gitlab accounts one is for company and the other one is personal, I would like both work independently on the machine at home, pull source code separately with their corresponding username and repository.

Problems

Very often I encountered a situation with permission denied in my company account while listing the remote repository of do anything remote (like git pull), but I can only make my personal account work without success on my company account. Turns out git client is wrongly picking the my personal account ssh information to login to my working account causing the "permission right" error.

Solutions

1. Refer to ~/.ssh/config and update the config file, add 1 more ssh entry to my company gitlab account as follows

Host anywhere.gitlab.com
  PubkeyAcceptedKeyTypes +ssh-rsa
  HostName gitlab.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id_rsa2
  User ihmcjacky

Host maxwellhk.gitlab.com
  PubkeyAcceptedKeyTypes +ssh-rsa
  HostName gitlab.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id_rsa
  User maxwellhk

Here, we have 2 users, 1 is ihmcjacky for my company usage, another is for my private usage, HostName parameter both specified as gitlab.com, but the Host which is very important, it acts as an alias for git client to identify which configuration it should refer to when trying to login to gitlab. Name anything familiar to you. In my case anywhere.gitlab.com and maxwellhk.gitlab.com.

2. Go to the desired project (in my case, company repository), navigate to .git folder and open config, configure the username and user email manually so as to let git client not to use globally configured settings for log in. Also in remote-origin url settings, configure as follows

url = git@anywhere.gitlab.com:p2Firmware/p2_controller.git

The anywhere.gitlab.com here is important, it is the alias we specified in ~/.ssh/config.

For safety, we can follow References > Multiple SSH keys for different accounts on Github or Gitlab, go through the processes, delete the cached keys and add back the ssh entries and check the ssh connection

3. Finally, add back the new origin URL to the project by using git remote set-url origin new.git.url/here

References

• Git pushing to remote GitHub repository as wrong user
  
• Multiple SSH keys for different accounts on Github or Gitlab
  
• Configure SSH to a different directory
  
• https://medium.com/uncaught-exception/setting-up-multiple-gitlab-accounts-82b70e88c437
  

[HTML&CSS] Vertical align 2 elements

Background

Very often we want to align 2 DIVs vertically and middle aligned with each other, like image and div, or 2 DIVs.

How

1. Declare the css "display: flex" or "display: inline-flex" at the parent container with "align-items: center"
2. Put the items we want to align inside the parent container

Quoting from stackoverflow's answer

References

• css: vertical-align image with text inside span
  
• https://wcc723.github.io/css/2017/07/21/css-flex
• A Complete Guide to Flexbox
  

[AWS & Docker] Add ec2 instance to specific cluster

Background

By default, when we create a new ec2 instance, ecs will automatically assign that ec2 instance to the cluster named "default", but we want ecs to put it in an already created cluster.

How

When creating the ec2 instance, remember 3 major things

1. Select "ECS-optimized" instances in AMI community
2. Create IAM "ecsInstanceRole" to allow ecs have the right to run commands in ec2 instances
• Refer to https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html for more info
3. Configure "User Data" entry to let ecs know which cluster we want to place the ec2 instance to

#!/bin/bash
echo ECS_CLUSTER=[CLUSTER_NAME] >> /etc/ecs/ecs.config

References

• [AWS ECS] 將建立的 EC2 instance 加入指定的 Cluster
• Launching an Amazon ECS Container Instance Point (g)

[AWS / Docker] Update Docker Images only in ECS Tasks

Background

So here is the problem, usually web applications needs updating, and more often the update is not about server setup, but application content, like updating the product images, prices etc. In this case, docker images needs to be updated, well of course, we need to rebuild and push the images to AWS docker registry again. But how do we update the service to reflect the new changes of the mounted images?

Solutions

We used "update-service" command, and "force-deployment" to force update the docker images although they are with the same tags.

aws ecs update-service --service my-service --force-new-deployment --cli-input-json myConfigJson

where service is the AWS service name and cli-input-json parameter specifies other useful attributes needed.

References

• Update docker images in ECS tasks

Useful bash command sytax (keep updating...)

Below are the useful syntax / tricks of bash commands that are easily forgot but handy. Keep them noted here

General Bash Command
1. Assignment of the value of variable word to parameter if and only if it is null

${parameter:=word}

2. Text processing using AWK example 1

echo $CODEBUILD_BUILD_ID | awk -F":" '{print $2}'

Output and pipe the string of variable CODEBUILD_BUILD_ID to awk as input, use ":" as separator and output the second element of the separated words

3. Cut out a range of characters for further processing

echo abcde | cut -c 1-2

cut accepts standard input and file as the reading source, cut -c 1-2 "xxx"  is not allowed

4. Find row of string existence and replace the whole line

SOME_VER=v1.0 && sed -i "/LINE TO FIND/c\VER=$SOME_VER" ./dir/to/file

Variable "SOME_VER" is defined, and then search for the line contains words "ROW TO FIND" and substitute with VER=v1.0 (variable substituted)

Note: The double quote "", single quote cannot perform variable substitution, single quote does not interpolate anything

"c\SomeText" is to replace the selected line with SomeText

Docker command
1. Close all running containers

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

2. Show docker entrypoint command without truncate

docker ps --no-trunc --format '{{.Command}}'

[Docker] Open Multiple Instances of Web App

Background
While developing my web application, I used docker compose to start 4 containers for the routine development, but my friend wants to try some new features / bug fixes which is completed, I don't want to interrupt his experiences while repeatedly start and stop the server during the development processes.

I then tried to re-run docker-compose command again but failed because same project name is used (default folder name).

Solutions
Outcome I need use docker-compose -p  to specify exactly the project name to allow multiple instances initialization of my project. It is very useful when you need to start the same instance of web application within the same machine. Remember to adjust the port number.

References

• Using p flag in docker compose (official)
• Run multiple docker compose instance

[AWS] Setting Up Multiple Load Balancers & A Very Dumb Mistake

Background
Continue from the previous AWS dynamic port mapping story, although familiarizing the concepts of how a load balancer works in AWS, and recognizing the theory of how different containers with the same container port can co-exists in the same service, but I still got a VERY VERY VERY huge issue not yet solved.

How to Configure Multiple Target Group Within a ECS Service?

The reason why this is a blocking issue for my project is that without multiple target groups, the following requirements cannot be achieved.

• HTTPS is a must for all URLs
• Extra port in URL (e.g.: www.gme.com:3002) is not okay because HTTPS cannot be made use (already occupied 443 port)
• Same container port (80) should be assigned to frontend and loopback containers for ALB
• ALB's rule-set configured to
• Redirect all incoming HTTP requests to HTTPS requests for security
• When gme.com or www.gme.com is requested, direct to gme.com and forward requests to FRONTEND container
• When api.gme.com or www.api.gme.com is requested, direct to api.gme.com and forward requests to LOOPBACK container

We can see the last requirement requires us to have 1 ALB and 2 target groups to accomplish, basically the idea is to identify which container to handle the requests by different URLs which in turn being allocated to different containers for responses. When api.gme.com is requested, the DNS (which is created by the LB) which execute the rule-set and know it needs to add a 301/302 redirection HTTPS and forward the request to gme-loopback-target-gp for responses, same for front-end requests

As we have 2 web servers (1 for serving front-end, 1 for middleware (loopback)), we cannot serve 2 servers using the same host port, we instead make use of different ephemeral ports or high number ports (32768 and 32769) for identification. 

So the most important part is to create an AWS service with 2 target groups associated with frontend and loopback container respectively. 

Problems and Solving Procedures
As of this tutorial, I get used to use ecs cli compose service up command to manage containers composing of the service and task definitions, however ecs cli compose DOES NOT SUPPORT MULTIPLE TARGET GROUPS! At least at the time when I write this article, I am frustrated for making only 1 frontend target group only.

But hey! I want 1 more... for my loopback container. At the end I got stuck 7-10 days wondering how can it be achieved. Furthermore requesting help in stackoverflow also but no luck. I did find the article saying multiple target groups did not currently support but this official article clearly mentioned the support of multiple target groups and even multiple LB, but they aren't using esc cli compose up command, turn out I follow the very basic service creation & task definition combination method to solve my problem. Here is the procedures



1. Under the cluster, create a new version of task, press "Run New Task"


2. Because I already created tasks before (through ecs cli compose syntactic sugar command), just creating a new version of task suit my needs. My option is to configure the task through AWS console, but we can configure through CLI too.

The important configuration here is the "task group", this affects how AWS handles task placement, in my case, the mw-gme is the same as my service name, this "connects" the task and the service.

The funny thing is it has 2 "Tasks" columns here, if you are not linking them while configuring task definition, they are detached, we will not see the task definition "mw-gme:42" in Tasks of cluster. So better specify the service name in "task group"



3. Click "Run Task" button and your tasks will start running the containers



4. Next we deal with creating service, here we use --cli-input-json parameter, feed in the json file for parameters, here are the context of the json file

{
  "cluster": "mw-ecs-gme",
  "serviceName": "mw-gme",
  "taskDefinition": "mw-gme:42",
  "schedulingStrategy": "REPLICA",
  "launchType": "EC2",
  "loadBalancers": [
      {
          "targetGroupArn": "arn:aws:elasticloadbalancing:us-east-2:916381200858:targetgroup/gme-frontend-gp/e2cbd06d8e11d633",
          "containerName": "frontend",
          "containerPort": 80
      },
      {
        "targetGroupArn": "arn:aws:elasticloadbalancing:us-east-2:916381200858:targetgroup/gme-loopback-gp/30e3eccadbb1f904",
        "containerName": "loopback",
        "containerPort": 80
    }
  ],
  "desiredCount": 1
}

The most important parameters here are cluster, serviceName, taskDefinition, and loadBalancers! Yes! And you see it allows me to put in an array! I can put multiple target groups here! Amazing, I can now setup gme-target-group-frontend to forward the matching rules to frontend container, while gme-target-group-loopback to forward the matching rules to loopback container, and both have host port of 80, that's why we can apply HTTPS on both of the containers to prevent the following errors

This error encounters when frontend (https://gme.com) is made HTTPS but not http://api.gme.com, we can't have mixture of HTTPS and HTTP put together, we must have all HTTPS equipped. That is why we need both 80 as container port.


And we can see the service name "mw-gme" here is the same as the name of the task group specified in step 2, this is very important they are with the same name, when doing so, tasks in cluster will show up the task definition entry (mw-gme:42).


5. Run aws ecs create-service --cli-input-json file://ecs-service.json, where ecs-service.json file is the one we specified in step 4.


And that's it! You will see service is started with 2 target groups in load balancing column and task definition is with status "RUNNING", for configuration of ELB and target group, refer to the previous thread.

You may wonder what takes me 7-10 days for such setup, 2 main points stuck me during the deployment.


1. Insisting there is such a way to allow me to deploy multiple target groups using ecs cli compose service up command, but indeed it is NOT SUPPORTED! Look at the command provided in official doc

ecs-cli compose --file ./docker-compose-aws-prod.yml --cluster mw-ecs-gme --ecs-profile mw-ecs-profile --cluster-config mw-ecs-gme-config --ecs-params ./ecs-params.yml --project-name mw-gme service up --target-group-arn arn:aws:elasticloadbalancing:us-east-2:916381200858:targetgroup/gme-gp/ed4eb5d138cf37df --container-name frontend --container-port 80  

We can see here how convenient this syntactic sugar command provided for us to easily bring up both the services and the associated task definition together, frankly speaking it is with no differences from the method I mentioned above (creating service and task definition), but it has its limitation. We can see parameter "--target-group-arn", "--container-name" and "--container-port" only allows creating 1 set of target group (with 1 set of container match). Stuck here for several days, turn out multiple target group is not supported in ecs-cli compose service command! Hope Amazon will improve this command in the future.


2. And the dumbest thing and spent me further 4-6 days is when I used service creation and task definition to finally created a multiple target groups load balancer service, the loopback server is still failing the health test of the LB, it is weird as I have confirmed these things already

- Host port 32769 incoming port has opened in security rule
- 2 target groups are created
- Containers and port mapping configured successfully through info provided by AWS console UI
- A record entry has been added in route 53
- Frontend (https://gme.com) is doing fine (passing the healthy test of LB, target port 32768)


I started a thread and seek help in stackoverflow, confirming my approach and steps are correct, but still no luck... And after 4-5 days struggling, I found a huge huge mistake, making all these not working, it is NOT related to all the settings of AWS, all are good and fine, then I started to think it may be come from the exposed port settings defined in loopback server itself...

As we can see, I can't image the problem comes from the loopback server configuration, I totally forgot to change the port settings to 80, while keeping it as the development port, after changing it back to port 80, healthy check passed and both https://gme.com and https://api.gme.com is up and running fine, what a silly mistake I have made!


Further Notes
You may not want to stop the service and renew the task definition again if you are just updating the docker registries, in this case, "aws ecs update-service" command is the way to go.

Executing "aws ecs update-service --service mw-gme --force-new-deployment --cli-input-json file://ecs-service-update.json", will update the service with the parameters provided, and most important is the "--force-new-deployment parameters, forcing ecs to retrieve the docker images again even they have the same tag.

And 1 more, 301 (permanent redirection) is very dangerous, if you tried HTTP 301 settings in server, redirecting all HTTP requests to HTTPS requests, in client devices, these permanent redirection is being cached with no expiry date, which means when you try to remove HTTPS and turn back to HTTP (I needed to do so at that time because I am stuck in setting up LB and need to provide services to users first), client device will still being cached to navigate to HTTPS version of the site which will cause 404 error, I would definitely recommend 302 (temporary redirect) if it is the halfway of the project and you intend to switch back temporarily.


References

• How to register multiple target groups via aws ecs cli service command
• AWS application load balancer not working
• ECS - how do you resolve unable to place a task
• ECS container ALB health checking
• What is application load balancer
• How to run more than 1 app on 1 instance of ec2
• Listener for application load balancer
• Difference between a task and a service (AWS)
• Linux - check whether the port is in used
• Is it possible to run multiple web instance in the same AWS EC2?
• Containers behind ELB in ECS?
• Listener rules for AWS
• Add support for creating services with multiple ELB target groups
• ecs-cli compose service up with a load balancer
• Registering multiple target groups with a service
• Gentle introduction to how AWS ECS works with example tutorial
• aws ecs create-service
• aws ecs update-service
• ECS container ALB health check failing
• Docker container own binds to ipv6
• Check if port is in use (Linux)
• 301 vs 302 redirect
• Differences between an AWS tasks and services