Useful bash command sytax (keep updating...)

Below are the useful syntax / tricks of bash commands that are easily forgot but handy. Keep them noted here

General Bash Command
1. Assignment of the value of variable word to parameter if and only if it is null

${parameter:=word}

2. Text processing using AWK example 1

echo $CODEBUILD_BUILD_ID | awk -F":" '{print $2}'

Output and pipe the string of variable CODEBUILD_BUILD_ID to awk as input, use ":" as separator and output the second element of the separated words

3. Cut out a range of characters for further processing

echo abcde | cut -c 1-2

cut accepts standard input and file as the reading source, cut -c 1-2 "xxx"  is not allowed

4. Find row of string existence and replace the whole line

SOME_VER=v1.0 && sed -i "/LINE TO FIND/c\VER=$SOME_VER" ./dir/to/file

Variable "SOME_VER" is defined, and then search for the line contains words "ROW TO FIND" and substitute with VER=v1.0 (variable substituted)

Note: The double quote "", single quote cannot perform variable substitution, single quote does not interpolate anything

"c\SomeText" is to replace the selected line with SomeText

Docker command
1. Close all running containers

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

2. Show docker entrypoint command without truncate

docker ps --no-trunc --format '{{.Command}}'

[Docker] Open Multiple Instances of Web App

Background
While developing my web application, I used docker compose to start 4 containers for the routine development, but my friend wants to try some new features / bug fixes which is completed, I don't want to interrupt his experiences while repeatedly start and stop the server during the development processes.

I then tried to re-run docker-compose command again but failed because same project name is used (default folder name).

Solutions
Outcome I need use docker-compose -p  to specify exactly the project name to allow multiple instances initialization of my project. It is very useful when you need to start the same instance of web application within the same machine. Remember to adjust the port number.

References

• Using p flag in docker compose (official)
• Run multiple docker compose instance

[AWS] Setting Up Multiple Load Balancers & A Very Dumb Mistake

Background
Continue from the previous AWS dynamic port mapping story, although familiarizing the concepts of how a load balancer works in AWS, and recognizing the theory of how different containers with the same container port can co-exists in the same service, but I still got a VERY VERY VERY huge issue not yet solved.

How to Configure Multiple Target Group Within a ECS Service?

The reason why this is a blocking issue for my project is that without multiple target groups, the following requirements cannot be achieved.

• HTTPS is a must for all URLs
• Extra port in URL (e.g.: www.gme.com:3002) is not okay because HTTPS cannot be made use (already occupied 443 port)
• Same container port (80) should be assigned to frontend and loopback containers for ALB
• ALB's rule-set configured to
• Redirect all incoming HTTP requests to HTTPS requests for security
• When gme.com or www.gme.com is requested, direct to gme.com and forward requests to FRONTEND container
• When api.gme.com or www.api.gme.com is requested, direct to api.gme.com and forward requests to LOOPBACK container

We can see the last requirement requires us to have 1 ALB and 2 target groups to accomplish, basically the idea is to identify which container to handle the requests by different URLs which in turn being allocated to different containers for responses. When api.gme.com is requested, the DNS (which is created by the LB) which execute the rule-set and know it needs to add a 301/302 redirection HTTPS and forward the request to gme-loopback-target-gp for responses, same for front-end requests

As we have 2 web servers (1 for serving front-end, 1 for middleware (loopback)), we cannot serve 2 servers using the same host port, we instead make use of different ephemeral ports or high number ports (32768 and 32769) for identification. 

So the most important part is to create an AWS service with 2 target groups associated with frontend and loopback container respectively. 

Problems and Solving Procedures
As of this tutorial, I get used to use ecs cli compose service up command to manage containers composing of the service and task definitions, however ecs cli compose DOES NOT SUPPORT MULTIPLE TARGET GROUPS! At least at the time when I write this article, I am frustrated for making only 1 frontend target group only.

But hey! I want 1 more... for my loopback container. At the end I got stuck 7-10 days wondering how can it be achieved. Furthermore requesting help in stackoverflow also but no luck. I did find the article saying multiple target groups did not currently support but this official article clearly mentioned the support of multiple target groups and even multiple LB, but they aren't using esc cli compose up command, turn out I follow the very basic service creation & task definition combination method to solve my problem. Here is the procedures



1. Under the cluster, create a new version of task, press "Run New Task"


2. Because I already created tasks before (through ecs cli compose syntactic sugar command), just creating a new version of task suit my needs. My option is to configure the task through AWS console, but we can configure through CLI too.

The important configuration here is the "task group", this affects how AWS handles task placement, in my case, the mw-gme is the same as my service name, this "connects" the task and the service.

The funny thing is it has 2 "Tasks" columns here, if you are not linking them while configuring task definition, they are detached, we will not see the task definition "mw-gme:42" in Tasks of cluster. So better specify the service name in "task group"



3. Click "Run Task" button and your tasks will start running the containers



4. Next we deal with creating service, here we use --cli-input-json parameter, feed in the json file for parameters, here are the context of the json file

{
  "cluster": "mw-ecs-gme",
  "serviceName": "mw-gme",
  "taskDefinition": "mw-gme:42",
  "schedulingStrategy": "REPLICA",
  "launchType": "EC2",
  "loadBalancers": [
      {
          "targetGroupArn": "arn:aws:elasticloadbalancing:us-east-2:916381200858:targetgroup/gme-frontend-gp/e2cbd06d8e11d633",
          "containerName": "frontend",
          "containerPort": 80
      },
      {
        "targetGroupArn": "arn:aws:elasticloadbalancing:us-east-2:916381200858:targetgroup/gme-loopback-gp/30e3eccadbb1f904",
        "containerName": "loopback",
        "containerPort": 80
    }
  ],
  "desiredCount": 1
}

The most important parameters here are cluster, serviceName, taskDefinition, and loadBalancers! Yes! And you see it allows me to put in an array! I can put multiple target groups here! Amazing, I can now setup gme-target-group-frontend to forward the matching rules to frontend container, while gme-target-group-loopback to forward the matching rules to loopback container, and both have host port of 80, that's why we can apply HTTPS on both of the containers to prevent the following errors

This error encounters when frontend (https://gme.com) is made HTTPS but not http://api.gme.com, we can't have mixture of HTTPS and HTTP put together, we must have all HTTPS equipped. That is why we need both 80 as container port.


And we can see the service name "mw-gme" here is the same as the name of the task group specified in step 2, this is very important they are with the same name, when doing so, tasks in cluster will show up the task definition entry (mw-gme:42).


5. Run aws ecs create-service --cli-input-json file://ecs-service.json, where ecs-service.json file is the one we specified in step 4.


And that's it! You will see service is started with 2 target groups in load balancing column and task definition is with status "RUNNING", for configuration of ELB and target group, refer to the previous thread.

You may wonder what takes me 7-10 days for such setup, 2 main points stuck me during the deployment.


1. Insisting there is such a way to allow me to deploy multiple target groups using ecs cli compose service up command, but indeed it is NOT SUPPORTED! Look at the command provided in official doc

ecs-cli compose --file ./docker-compose-aws-prod.yml --cluster mw-ecs-gme --ecs-profile mw-ecs-profile --cluster-config mw-ecs-gme-config --ecs-params ./ecs-params.yml --project-name mw-gme service up --target-group-arn arn:aws:elasticloadbalancing:us-east-2:916381200858:targetgroup/gme-gp/ed4eb5d138cf37df --container-name frontend --container-port 80  

We can see here how convenient this syntactic sugar command provided for us to easily bring up both the services and the associated task definition together, frankly speaking it is with no differences from the method I mentioned above (creating service and task definition), but it has its limitation. We can see parameter "--target-group-arn", "--container-name" and "--container-port" only allows creating 1 set of target group (with 1 set of container match). Stuck here for several days, turn out multiple target group is not supported in ecs-cli compose service command! Hope Amazon will improve this command in the future.


2. And the dumbest thing and spent me further 4-6 days is when I used service creation and task definition to finally created a multiple target groups load balancer service, the loopback server is still failing the health test of the LB, it is weird as I have confirmed these things already

- Host port 32769 incoming port has opened in security rule
- 2 target groups are created
- Containers and port mapping configured successfully through info provided by AWS console UI
- A record entry has been added in route 53
- Frontend (https://gme.com) is doing fine (passing the healthy test of LB, target port 32768)


I started a thread and seek help in stackoverflow, confirming my approach and steps are correct, but still no luck... And after 4-5 days struggling, I found a huge huge mistake, making all these not working, it is NOT related to all the settings of AWS, all are good and fine, then I started to think it may be come from the exposed port settings defined in loopback server itself...

As we can see, I can't image the problem comes from the loopback server configuration, I totally forgot to change the port settings to 80, while keeping it as the development port, after changing it back to port 80, healthy check passed and both https://gme.com and https://api.gme.com is up and running fine, what a silly mistake I have made!


Further Notes
You may not want to stop the service and renew the task definition again if you are just updating the docker registries, in this case, "aws ecs update-service" command is the way to go.

Executing "aws ecs update-service --service mw-gme --force-new-deployment --cli-input-json file://ecs-service-update.json", will update the service with the parameters provided, and most important is the "--force-new-deployment parameters, forcing ecs to retrieve the docker images again even they have the same tag.

And 1 more, 301 (permanent redirection) is very dangerous, if you tried HTTP 301 settings in server, redirecting all HTTP requests to HTTPS requests, in client devices, these permanent redirection is being cached with no expiry date, which means when you try to remove HTTPS and turn back to HTTP (I needed to do so at that time because I am stuck in setting up LB and need to provide services to users first), client device will still being cached to navigate to HTTPS version of the site which will cause 404 error, I would definitely recommend 302 (temporary redirect) if it is the halfway of the project and you intend to switch back temporarily.


References

• How to register multiple target groups via aws ecs cli service command
• AWS application load balancer not working
• ECS - how do you resolve unable to place a task
• ECS container ALB health checking
• What is application load balancer
• How to run more than 1 app on 1 instance of ec2
• Listener for application load balancer
• Difference between a task and a service (AWS)
• Linux - check whether the port is in used
• Is it possible to run multiple web instance in the same AWS EC2?
• Containers behind ELB in ECS?
• Listener rules for AWS
• Add support for creating services with multiple ELB target groups
• ecs-cli compose service up with a load balancer
• Registering multiple target groups with a service
• Gentle introduction to how AWS ECS works with example tutorial
• aws ecs create-service
• aws ecs update-service
• ECS container ALB health check failing
• Docker container own binds to ipv6
• Check if port is in use (Linux)
• 301 vs 302 redirect
• Differences between an AWS tasks and services

[AWS] Dynamic Port Mapping & SSL Equipment through ELB

Background
Deployment of the container in Amazon cloud has completed, and there are other challenges which include

• Adding SSL to current production site for increased security
• Dynamic port mapping for serving multiple services within 1 cluster

The reason why I put these 2 features together is because of their similarity on how they configure. Dynamic port mapping is used to deal with running multiple services within a cluster, imagine when more than 1 web server is built in a ec2 instance, how is the port 80 being "shared" by these 2 services (web server)? Dynamic port mapping is the solution.

Concepts
The solution is to make use of ELB (Elastic Load Balancers), which is divided into ALB (Application Load Balancers) and NLB (Network Load Balancers), ALB is to load balance the traffic in application level, L7, which is our case (HTTP & HTTPS). NLB is for binary protocols (non HTTPx protocols).


Here is how application load balancers (ALB) deals with dynamic port mapping

1. User fire requests either HTTP (port 80) or HTTPS (port 443) directly to the load balancer

2. Each load balancer has its corresponding listeners which checks the connections of the port / protocol. Note that we need to pass the healthy test before the load balancer operates

3. When such connection happens, listeners will use the listener rules (user defined, can be more than 1, depending on your needs) to direct the traffic to designated target groups

4. Here, the target group is "TG1", this group contains the frontend container which is composed by the docker compose file. We need to wire up the group arn to the container when we create the service using "ecs cli create service" command, note we cannot modify it after the service has started. And the limitation using this command is that we can only bind 1 target group to 1 container.

5.A "registered target" should be configured in the target group, and the load balancer will use the registered target to check the healthy status before operate. When the request is forwarded to the target group, the listener will choose the port specified in the registered target to transfer / map the request to the desired containers.

e.g.: If we configured a target group as ec2-instance (target instance) | 32768 (port) | ... The listener will know for the port 80, which container port should be used to map the host port 80 request to.

But remember, to ensure this happens, we need to configure the security groups to allow inbound port 80 or any desired host ports to allow load balancers correctly listens to outside requests

6. Sometimes there may be more than 1 container served in the same ec2 instance, we therefore need dynamic port mapping to accomplish 2 web servers situated in the same ec2 instance, by dividing the 2 containers into 2 different target groups, serving 2 websites using 1 ec2 instance become possible.

Procedures
1. Follow AWS tutorial to create load balancers, choose HTTP / HTTPS application load balancer

2. Choose "Internet facing" scheme and add both HTTP and HTTPS load balancer protocol if you want SSL over HTTP.





















3. Choose the desired VPC to apply the load balancer to

4. Choose the certificate type (we normally use ACM as it is super easy to deploy the site with SSL enabled), choose the certificate registered in ACM (I am using route 53 for the domain and use Amazon ACM here)

5. Choose security group, if the outgoing port is 80 and 443, you need to allow inbound connection of port 80 and 443 to allow load balancer to do its job.

6. Configure the target group and health check parameters. Here, remember we use HTTP protocol (port 80) as the protocol for the load balancer to forward traffic to designated target group, we left SSL handling to load balancer and prevent configuring the certificates server configuration, deployment and renewal in application level, we let ACM to handle them all for us. So, we only need load balancer to do health check on port 80. (As our docker compose file exposed port 80 for the LB to do the health check)

7. After creating the load balancer, navigate to "target groups", you will see the group we created when we create the load balancer, head to "Description", copy the arn of the target group and use it to associate with the service using the following command

ecs-cli compose --file ./docker-compose-aws-prod.yml --cluster mw-ecs --ecs-profile mw-ecs-profile --cluster-config mw-ec2 --project-name supremeav-2-dot-0 --ecs-params ./ecs-params.yml service up --target-group-arn arn:aws:elasticloadbalancing:us-east-2:916381200858:targetgroup/http-port-80-access/222d18e3d1cddf3a --container-name frontend --container-port 80

Note that "target-group-arn" is the arn you copied in description and "container-name", "container-port" must be specified to let balancer forward the request to the right container.

8. After the container is initialized, you should see a new entry appeared in the registered target (in the target groups) automatically when the container is up and run. Health check should be carried out accordingly.

9. Finally, navigate to load balancer and copy the DNS name, navigate to route 53, find the domain name registered, add or edit the existing A type entry, with alias selected, choose from the input the DNS A type entry (or you can paste the DNS you just copied) and you are all done!

Now when user access the website through Internet, they are just forming connection with the load balancer and the load balancer will forward / direct the request to the container port through evaluating the rules.

For SSL connection, there is no extra procedure needed, just ensure there is an entry in listeners of the load balancers, the rest would be handled by them, no extra nginx configuration play around!

References

1. https://stackoverflow.com/questions/58587976/nginx-docker-container-on-aws-ecs-the-plain-http-request-was-sent-to-https-po
2. SSL redirection in docker container on aws ecs
3. Understanding dynamic port mapping in amazon ecs with load balancer