Docker / docker compose cheatsheet

Background

Very often there are commands which are very useful for debugging and troubleshooting in docker and docker compose, this post is to document them down for future use.

1. Gather health check command status information of specific container
   

   docker inspect --format "{{json .State.Health }}" anm11_log_server_1 | jq

   jq bash command is needed to install beforehand. Following results will be outputted
   
   

   {
     "Status": "healthy",
     "FailingStreak": 0,
     "Log": [
       {
         "Start": "2023-05-09T15:31:55.974151402+08:00",
         "End": "2023-05-09T15:31:56.180881836+08:00",
         "ExitCode": 0,
         "Output": ""
       },
       {
         "Start": "2023-05-09T15:32:56.185509821+08:00",
         "End": "2023-05-09T15:32:56.322503697+08:00",
         "ExitCode": 0,
         "Output": ""
       },
       {
         "Start": "2023-05-09T15:33:56.327354557+08:00",
         "End": "2023-05-09T15:33:56.506142901+08:00",
         "ExitCode": 0,
         "Output": ""
       },
       {
         "Start": "2023-05-09T15:37:16.863314985+08:00",
         "End": "2023-05-09T15:37:17.0210078+08:00",
         "ExitCode": 0,
         "Output": ""
       },
       {
         "Start": "2023-05-09T15:38:17.025502598+08:00",
         "End": "2023-05-09T15:38:17.155554148+08:00",
         "ExitCode": 0,
         "Output": ""
       }
     ]
   }
   

   
   

Docker compose .env confusion

Background

I want to specify the location of .env file for variable substitution of docker-compose yml file, there is a need to separate .env, .production.env for different usage scenarios.

Problems

Some responses in stackoverflow quotes "env_file" keyword and --env-file tag which are not the same thing as .env file, env_file and --env-file tag only pass environment variables to containers, but not variable substitutions of docker-compose file, they are totally different things.

Solutions

cat the env file content to the docker compose file to read contents of .env with custom name. E.g.: alt.env

env $(cat alt.env) docker-compose up --build

References

- https://stackoverflow.com/a/67495905/2361494

Multi-container restaurant project

Background

A project includes a restaurant owner who wants to deploy our online ordering system with all his restaurants (total 8) to include online ordering features, currently, only a static page goldenthumb.com.hk is running (with https enabled), our target is to use a single AWS lightsail instance to serve for 

1. Providing portal page (run by Gatsby JS), currently goldenthumb.com.hk, act as a portal page to allow user to quickly select different restaurants for ordering, reference site: https://www.maximsmx.com.hk/takeaway_promotion/?utm_source=eatizen

2. Use subdomain to distinguish different restaurants, e.g. lck.goldenthumb.com.hk is the ordering system of restaurant located at Lai Chi Kok

3. api.lck.goldenthumb.com.hk is the API URL served by loopback JS container to provide RESTful service to access database (mongoDB)'s data

4. Run in https and able to check and renew the cert automatically

5. Provide ftp access for the static goldenthumb.com.hk file updates for site maintenance

Problems

This project setup is very much similar to another post, but more complicated, as it includes 2 more containers, certbot and static portal page container which co-exists with the nginx, loopback, mongo express, mongoDB containers (the ordering web application formation) as well as the ftp container and another nginx container.

The following are the problems I have faced during the system configuration

1. The lck.goldenthumb.com.hk requests are not forwarded to the desired nginx container, showing "502 bad gateway"

Turns out I have wrongly restart the container that is never meant to be restarted, normally the static web container "goldenthumbStatic" is used as the main server to receive client requests, the proxy_server settings should be configured inside "goldenthumbStatic" as follows

server {
  # Serving api url for internal use
  server_name lck.goldenthumb.com.hk;
  
  ssl_certificate /etc/letsencrypt/live/goldenthumb.com.hk/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/goldenthumb.com.hk/privkey.pem;
  
  listen 80 ;
  listen 443 ssl ;
  
  # access_log /var/log/nginx/access.loopback.log;
  # We redirect all outside request to appropriate loopback container
  location / {
    proxy_pass http://lck.goldenthumb.com.hk;
  }

  location /admin {
    proxy_pass http://lck.goldenthumb.com.hk;
    try_files $uri /index.html;
  }

  location /.well-known/acme-challenge/ {
    root /var/www/certbot;
  }
}

The location / block is important, it tells the server when request comes as "lck.goldenthumb.com.hk", then forward the request to server block "http://lck.goldenthumb.com.hk", which is defined below 

# Groups of server for proxy_pass usage in below server block
upstream lck.goldenthumb.com.hk {
  # We put docker container service name (internal docker IP) for pointing to right API server (8082:80)
  server frontend;
}

Here the upstream block is used to define blocks of server(s) for reference of proxy_pass directives, the value in proxy_pass is related to this upstream block and forward the request to frontend.

Although configurated correctly, I keep restarting the wrong docker container which the settings are therefore yet to apply 

2. The api.lck.goldenthumb.com.hk requests are not forwarded to the desired loopback container, showing "Connection refused"

This is turn out a silly mistake which forgot to update loopback configuration file (config.json), 80 port should be configured to accept the API connections from other containers

3. The admin management page lck.goldenthumb.com.hk/admin is not working, showing "Connection refused"

The nginx conf is not configured correctly, below is the corrected config, the "try_files" is important to guide nginx server to query original index.html when api.lck.goldenthumb.com.hk/admin is navigated, and let react route to handle the rest of the routing tasks instead of the server itself

  location /admin {
    proxy_pass http://lck.goldenthumb.com.hk;
    try_files $uri /index.html;
  }

4. The APIs used in admin page all fired as "lck.goldenthumb.com.hk/xxx" which is different from expected "api.lck.goldenthumb.com.hk/xxx"

This is due to the wrongly configured dockerfile setup, missing a step to copy the compiled js files to the production docker container, causing the updated code not reflecting the changes.

5. The https certbot challenges for multiple subdomains

HTTPS needs to apply for every sub-domain, lck.goldenthumb.com.hk, api.lck.goldenthumb.com.hk needs to be applied separately, update the certbot script (init-letsencrypt.sh) first, with domain marked to all desired sub-domains

domains=(goldenthumb.com.hk www.goldenthumb.com.hk lck.goldenthumb.com.hk api.lck.goldenthumb.com.hk)

Then in docker compose file, configure the all container volumes which need https certificates by adding /var/www/certbot to accept the challenges

./app/certbot/www:/var/www/certbot

Building web server of multiple docker instances with ssl (https) protection in AWS lightsail

Background

Continue from last tutorial of creating http server with ssl protection, this time I want a bid more advanced, here is the situation.

I have a take away web application, which consists of 5 servers, they are frontend server powered by react, middleware server powered by loopback (nodeJS server), mongoExpress for monitoring mongoDB in UI, mongoDB database and a ftp server for updating the menu and meal information. System architecture is as shown as follow

So there exists 5 docker containers, running in the same local network, I want to put them altogether as a web application to serve my client, but the problem is the SSL cert, I want security transaction and need to register and deploy the certificate for my react front-end and middleware loopback server, so how can I deploy all of them (with same internal 80 port) to the Internet?

Problems

The following are the requirements & problems needed to solve

1. Getting a signed certificate from trusted party

2. Allow "api.goodmaneat.com" to be surfed by front-end server to acquire middleware functionalities through port 443 (https) (which co-exists with the front-end container)

3. Is the cert being shared by *.goodmaneat.com and goodmaneat.com?

4. The following is what needed to achieve

- www.goodmaneat.com -> goodmaneat.com

- www.goodmaneat.com/admin -> goodmaneat.com/admin

- http://goodmaneat.com -> https://goodmaneat.com

So basically, I want the server to strip the www prefix and force redirect to https

Solutions

The below items are what I have done to tackle the problems

1. Create a lightsail AWS instance of type Amazon Linux 2, which is good if you have any service needed to use aws cli

2. Install docker and docker compose to the Amazon Linux (https://gist.github.com/npearce/6f3c7826c7499587f00957fee62f8ee9)

    - Note: Logout / restart the instance after installing or docker cannot function properly

3. Assume you deploy your docker images in AWS, configure your login credential first before accessing Amazon registry (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html)

4. Git clone or upload all files to the server and use docker compose to build up the docker images to containers, I have the following directories for making the whole application functions. Check all 5 containers are up and run.

Docker-compose file reference: (https://docs.google.com/document/d/1SPWOBeLL23E75W_D9U38jZACVNR9jZVwbZqtljIWX2I/edit?usp=sharing)

Note for some important points for the docker-compose yml file

- Loopback container's port setting should be 8082:80 (host:container), we cannot have 80:80 as frontend has already occupied the port 80

- Add 443:443 port settings to frontend container to serve for https connection

- Add nginx and letsencrypt folder to store nginx configuration (which we will deal with in later steps) and certificates (Note: the whole /etc/letsecrypt folder should be mounted for https to work properly)

- Update the dockerFile of the frontend container as the following (https://docs.google.com/document/d/1hJyFyWSazhE_qx9G2dBGc0of9BlDuBgdk0VdKGRRGwk/edit?usp=sharing), here, we install certbox for obtaining certificates

5. docker exec into frontend container, follow step 2 to step 6 to complete the retrieval of certificate process (https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-using-lets-encrypt-certificates-with-wordpress)

Note: 

- Here we assume you already have a domain name registered and have full control as you need to add TXT records to complete the letsencrypt challenges, you should have also configured the route 53 record (assume you are using AWS as your domain name service provider) to add the domain name and IP mapping of "www.goodmaneat.com", "goodmaneat.com" and other sub domain name required to the route 53 record table.

6. Still in frontend container, assuming you are using nginx as web server, head to /etc/nginx/conf.d/nginx.conf (https://docs.google.com/document/d/18LuvdXzsE1qsyBP3fFpP1A1v528MLiYA_Ime-_yptfY/edit?usp=sharing), update the configuration file as the specified URL to 

- Locate the certificate registered in step 5

- Update port settings to only accept https connections

- 301 permanent redirect when www.goodmaneat.com/* is detected, the first sever block configuration accomplish such effect by recognizing domain name "www.goodmaneat.com" and all its subdomains, and return 301 header redirect to its https and www removed URL.

- The second server block serves only https URLs

- The third server block is the most tricky part, it detects server name "api.goodmaneat.com", we still need to provide certificate file here because we accepts only https connection even for internal docker container access. 

- The "proxy_pass" in location block is important, it works with upstream block to guide nginx server when api.goodmaneat.com (http/https) is being accessed, it reverses proxy to send the request to the requested server (this time it is our "loopback container", which can be referred using docker service name), nginx then fetches the response and send it back to our client (frontend container), the upstream block provides group of servers for proxy_pass directive to refer to, e.g.: the value of "proxy_pass http://api.goodmaneat.com" will be parsed as "proxy_pass http://(internal docker IP of loopback container)"

6. After all the nginx configurations, reload the nginx server by "nginx -s reload -c /etc/nginx/conf.d/nginx.conf"

Finally, the file structure of the host server (not the docker container) should be as follows

- Letsencrypt folder volume amount is for storing the certificate and key files in frontend nginx web server container

- nginx-conf folder to map and permanently store the server configuration files in step 5 in frontend nginx web server container

Note

When cert is being updated, some cert file's ownership and user rights will change to root, which makes reading of certificate failed, kindly change to appropriate user and user right before deployment when cert is being renewed

References

- Nginx multiple server blocks listening to same port

- Update: Using Free Let’s Encrypt SSL/TLS Certificates with NGINX

- Multiple docker containers accessible by nginx reverse proxy 

- How to Host Multiple Docker Containers on a Single Droplet with Nginx Reverse Proxy?

- How to proxy_pass to a node docker container on port 80 with nginx container

- How To Redirect HTTP To HTTPS In Nginx

- nginx with Let’s Encrypt in Docker container

- Multiple SSL certificates for a single domain on different servers

- How nginx processes a request

- NGINX multiple server blocks with reverse proxy

- Module ngx_http_upstream_module

- Differences Between A and CNAME Records

- Secure your site with HTTPS

- http directive error in nginx.conf

- Example for a reverse multi-domain proxy using nginx and docker

- Automated nginx proxy for Docker containers using docker-gen

- Using Amazon ECR with the AWS CLI

Dot env file is not found in docker compose file

Background

So there is a case I need to substitue the variables defined in .env to the docker compose for further process, but when I run the docker-compose up command, the variable refer in docker compose file cannot be interpreted.  

Reason

Normally the .env file sits to the same directory as the docker compose yml file, and is expected the docker compose command being executed in the current directory of the docker compose file and the .env file, which I am not in this case.

Solutions

If the current directory is not the place where the compose file is sitting in, we need to specify --project-directory to allow docker-compose program to be able to recognize the directory where the yml file and the .env file sits in, the correct command after fixing is as follows

docker-compose -f {docker-compose-file-dir} --project-directory={directory-of-docker-compose-file} up -d

[Docker] Docker run emits "name is already in use" error

Background

When trying to docker run through docker compose, name is already in use appeared.

Solutions

Using docker ps -a, we will find there are docker containers created with the same name, it may due to docker compose stop not cleanly close all the containers.

To solve, find out, we can try to close all the running containers (you can find which one's name is duplicated, but I want to make life easier), docker stop $(docker ps -a -q), re-run the docker compose script again

References

• docker run --> 'name is already in use by container'
• https://coderwall.com/p/ewk0mq/stop-remove-all-docker-containers
• Docker error bind address already in use

[AWS & Docker] Add ec2 instance to specific cluster

Background

By default, when we create a new ec2 instance, ecs will automatically assign that ec2 instance to the cluster named "default", but we want ecs to put it in an already created cluster.

How

When creating the ec2 instance, remember 3 major things

1. Select "ECS-optimized" instances in AMI community
2. Create IAM "ecsInstanceRole" to allow ecs have the right to run commands in ec2 instances
• Refer to https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html for more info
3. Configure "User Data" entry to let ecs know which cluster we want to place the ec2 instance to

#!/bin/bash
echo ECS_CLUSTER=[CLUSTER_NAME] >> /etc/ecs/ecs.config

References

• [AWS ECS] 將建立的 EC2 instance 加入指定的 Cluster
• Launching an Amazon ECS Container Instance Point (g)

[AWS / Docker] Update Docker Images only in ECS Tasks

Background

So here is the problem, usually web applications needs updating, and more often the update is not about server setup, but application content, like updating the product images, prices etc. In this case, docker images needs to be updated, well of course, we need to rebuild and push the images to AWS docker registry again. But how do we update the service to reflect the new changes of the mounted images?

Solutions

We used "update-service" command, and "force-deployment" to force update the docker images although they are with the same tags.

aws ecs update-service --service my-service --force-new-deployment --cli-input-json myConfigJson

where service is the AWS service name and cli-input-json parameter specifies other useful attributes needed.

References

• Update docker images in ECS tasks

[Docker] Open Multiple Instances of Web App

Background
While developing my web application, I used docker compose to start 4 containers for the routine development, but my friend wants to try some new features / bug fixes which is completed, I don't want to interrupt his experiences while repeatedly start and stop the server during the development processes.

I then tried to re-run docker-compose command again but failed because same project name is used (default folder name).

Solutions
Outcome I need use docker-compose -p  to specify exactly the project name to allow multiple instances initialization of my project. It is very useful when you need to start the same instance of web application within the same machine. Remember to adjust the port number.

References

• Using p flag in docker compose (official)
• Run multiple docker compose instance

[AWS] Dynamic Port Mapping & SSL Equipment through ELB

Background
Deployment of the container in Amazon cloud has completed, and there are other challenges which include

• Adding SSL to current production site for increased security
• Dynamic port mapping for serving multiple services within 1 cluster

The reason why I put these 2 features together is because of their similarity on how they configure. Dynamic port mapping is used to deal with running multiple services within a cluster, imagine when more than 1 web server is built in a ec2 instance, how is the port 80 being "shared" by these 2 services (web server)? Dynamic port mapping is the solution.

Concepts
The solution is to make use of ELB (Elastic Load Balancers), which is divided into ALB (Application Load Balancers) and NLB (Network Load Balancers), ALB is to load balance the traffic in application level, L7, which is our case (HTTP & HTTPS). NLB is for binary protocols (non HTTPx protocols).


Here is how application load balancers (ALB) deals with dynamic port mapping

1. User fire requests either HTTP (port 80) or HTTPS (port 443) directly to the load balancer

2. Each load balancer has its corresponding listeners which checks the connections of the port / protocol. Note that we need to pass the healthy test before the load balancer operates

3. When such connection happens, listeners will use the listener rules (user defined, can be more than 1, depending on your needs) to direct the traffic to designated target groups

4. Here, the target group is "TG1", this group contains the frontend container which is composed by the docker compose file. We need to wire up the group arn to the container when we create the service using "ecs cli create service" command, note we cannot modify it after the service has started. And the limitation using this command is that we can only bind 1 target group to 1 container.

5.A "registered target" should be configured in the target group, and the load balancer will use the registered target to check the healthy status before operate. When the request is forwarded to the target group, the listener will choose the port specified in the registered target to transfer / map the request to the desired containers.

e.g.: If we configured a target group as ec2-instance (target instance) | 32768 (port) | ... The listener will know for the port 80, which container port should be used to map the host port 80 request to.

But remember, to ensure this happens, we need to configure the security groups to allow inbound port 80 or any desired host ports to allow load balancers correctly listens to outside requests

6. Sometimes there may be more than 1 container served in the same ec2 instance, we therefore need dynamic port mapping to accomplish 2 web servers situated in the same ec2 instance, by dividing the 2 containers into 2 different target groups, serving 2 websites using 1 ec2 instance become possible.

Procedures
1. Follow AWS tutorial to create load balancers, choose HTTP / HTTPS application load balancer

2. Choose "Internet facing" scheme and add both HTTP and HTTPS load balancer protocol if you want SSL over HTTP.





















3. Choose the desired VPC to apply the load balancer to

4. Choose the certificate type (we normally use ACM as it is super easy to deploy the site with SSL enabled), choose the certificate registered in ACM (I am using route 53 for the domain and use Amazon ACM here)

5. Choose security group, if the outgoing port is 80 and 443, you need to allow inbound connection of port 80 and 443 to allow load balancer to do its job.

6. Configure the target group and health check parameters. Here, remember we use HTTP protocol (port 80) as the protocol for the load balancer to forward traffic to designated target group, we left SSL handling to load balancer and prevent configuring the certificates server configuration, deployment and renewal in application level, we let ACM to handle them all for us. So, we only need load balancer to do health check on port 80. (As our docker compose file exposed port 80 for the LB to do the health check)

7. After creating the load balancer, navigate to "target groups", you will see the group we created when we create the load balancer, head to "Description", copy the arn of the target group and use it to associate with the service using the following command

ecs-cli compose --file ./docker-compose-aws-prod.yml --cluster mw-ecs --ecs-profile mw-ecs-profile --cluster-config mw-ec2 --project-name supremeav-2-dot-0 --ecs-params ./ecs-params.yml service up --target-group-arn arn:aws:elasticloadbalancing:us-east-2:916381200858:targetgroup/http-port-80-access/222d18e3d1cddf3a --container-name frontend --container-port 80

Note that "target-group-arn" is the arn you copied in description and "container-name", "container-port" must be specified to let balancer forward the request to the right container.

8. After the container is initialized, you should see a new entry appeared in the registered target (in the target groups) automatically when the container is up and run. Health check should be carried out accordingly.

9. Finally, navigate to load balancer and copy the DNS name, navigate to route 53, find the domain name registered, add or edit the existing A type entry, with alias selected, choose from the input the DNS A type entry (or you can paste the DNS you just copied) and you are all done!

Now when user access the website through Internet, they are just forming connection with the load balancer and the load balancer will forward / direct the request to the container port through evaluating the rules.

For SSL connection, there is no extra procedure needed, just ensure there is an entry in listeners of the load balancers, the rest would be handled by them, no extra nginx configuration play around!

References

1. https://stackoverflow.com/questions/58587976/nginx-docker-container-on-aws-ecs-the-plain-http-request-was-sent-to-https-po
2. SSL redirection in docker container on aws ecs
3. Understanding dynamic port mapping in amazon ecs with load balancer

[Docker] Console Log Problem

Background
Making a docker application through docker compose, the log terminal does not return any of the console logging returned from my node server


Cause & Solution
Turn out the problem comes from my mistakes on making a "command substitution" in entrypoint of the dockerFile. Here is my entrypoint command

ENTRYPOINT /bin/bash -c "npm install && if [ $NODE_ENV == '"development"' ]; then \
$(nodemon --ignore node_modules/ mongoEngine.js); else $(node mongoEngine.js); fi"

Outputting in command substitution mode will not be captured. Remove the $(...) quoting solves the problem

References

• https://unix.stackexchange.com/questions/440088/what-is-command-substitution-in-a-shell
• https://www.cnblogs.com/sparkdev/p/8461576.html

Putting the docker composed application online to AWS - a painful journey

Preface
Amazon Web Service (AWS) offers a very attractive limited period free tier package for those who wants to taste a bit of fantastic features of AWS like me and try to make use its services to put my docker compose application online. Of course the application is just a playground for me, my main purpose is to try out how good AWS would be in my situation. After a week of hard works, I finally made it work as my expectation but it comes with series of pain during the processes.


Background
Like I mentioned in preface, I made a web application, which contains 4 containers

• Frontend (A web interface for user to interact)
• Loopback framework (A RESTful handling framework)
• MongoDB (Database container)
• Mongo Express (Database management container)

And the following are the docker compose file I composed

• https://docs.google.com/document/d/19OiG0-Dqp6nIRJrOFOXHLPeliJFhtOww83Q-SziHi0w/edit?usp=sharing

My target is to make the above 4 containers up and running, and AWS would be the platform for me to run and maintain these containers.

Procedures & Experiences 

I developed using the above compose file, and after the development, I created another branch and changed the docker compose settings to fulfill the production environment. E.g.: changing the node environment variable to "production", changing port back to 80 etc. And hence a new docker-compose-prod.yml is created. My strategy is at least I made the production yml file up and running in local first, and I know in certain extent, it is easier to kick off my AWS journey.

The following is the docker-compose-prod.yml file I used

• https://docs.google.com/document/d/1ORxGFu3lGFd8nasnV6y-zP5bG5FGJcGVGml-amimMpc/edit?usp=sharing

This docker-compose-prod is nothing special but a copy of docker-compose.yml and basically

• Change the port to production ready port
• Remove frontend, loopback container's volume dependencies (we include all production files in the docker image instead)
• Change some context docker file to its docker-prod file

So far so good, I make the production containers up and run with the production yml in my machine, happy! Let's see how to accomplish the same thing in AWS!

By the way, I use this docker command to make those containers up and run

docker-compose -f ./docker-compose-prod.yml up

I think working on AWS is as easy as I did in local machine, but outcome it did not. After 2 weeks of struggling in AWS, here are some of the experiences and thinking I would like to share.

In Preface section, I mentioned about the attractive free tier trial. Indeed there are quite a number of "traps", maybe it is unfair to Amazon, but I have to say being a newbie of AWS is hard, AWS is comprehensive in a good way, but too complicated to familiar with is its drawbacks.

Why hard? Because Amazon is confusing us with tons of different unfamiliar names, EC2, ECS, ECR, EBS, t2.micro, Amazon Cloud Watch, fargate blablabla... You may ask why are these terms so important that I need to care? First of all money, remember the 750 hour free tier usage? If you are not familiar with how the AWS game works, you will probably being "tricked" by those "free" ads. And finally find the bill have some charges that you are not noticed

In traditional hosting, we only pay monthly or yearly (specific amount), like hosting speed, we pick a plan, and we will get 10 GB web file storage, certain amount of emails, a domain name, a control panel and we are good to go for our web application!

However in AWS, game is not playing in this way. Amazon tries to divide different parts of the hosting services as many many tiny different sub-services. Say in my case, I want to put my docker compose application to the public through AWS, I need the following components

• Amazon EC2 (Elastic Compute Cloud): Consider it as an utmost base service (actually a remote virtual machine) to execute your cloud application, root of all the services. E.g.: boot up the server instances in order to run your web server, restful API service, mongoDB,  blabla... A prerequisite in most of the cases.
• Amazon EBS (Elastic Block Storage): Nothing special but can see as a general storage spaces to store your application files.
• Amazon ECS (Elastic Container Service):  As the name implies, provide services to run docker containers. I first confused EC2 and ECS, I think ECS is necessary for running docker containers, indeed it isn't. ECS is a cluster (group of EC2 instances). So a very simple question, why we need ECS? ECS acts as a "proxy-like" role, we run ecs-cli command to tell ECS to launch container on EC2. But I am confused, why can't I directly launch them in EC2 instances? Why I need 1 more layer to do so?
  
  Analogy ECS as your mom, EC2 as myself. I can cook myself (launch container), but why we still need mom to do so? (manage to cook the food). Because mom knows how to cook delicious food, she knows how the ingredients work with each other, quantity, combination etc. To make use of the best effective resources to cook the delicious food. That's why we need mom (ECS)
  
  ECS knows very good on how to manage EC2 effectively, organizes optimal resources (CPU, RAM, Storage...). And ECS use docker to initialize containers in EC2 virtual machines. So nothing magic on this ECS. ECS is meaningless if no EC2s are associated with. On the other hand, you can live without ECS, but not EC2. The following better illustrate the relationships (from Amazon).
  
  
• Amazon ECR (Elastic Container Registry): Nothing special but a place to store the docker images. A docker image repository.

The above is pretty much all the components. The following are the procedures and the difficulties experienced. For the procedures, skipped some of the details (e.g.: IAM role setup) for simpler representation. Detailed tutorials will be hyperlinked referenced.

1. Sign Up An AWS Account - Nothing special, but you must put your credit card down first... :)

2. Install and Configure AWS CLI
To access different AWS services and if you plan to manage your services through the command line interface, you must install and setup AWS CLI

https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

https://docs.aws.amazon.com/cli/latest/userguide/install-cliv1.html

Login to AWS console, search "security", pick "IAM", choose Users in left panel, choose designated users you want to access the AWS service, and choose "Security Credentials" tab, then click "Create Access Key", copy the credential data out which is needed to do the authentication before accessing any of the AWS services. For details
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration

Basically follow the instruction for the installation & the credential creation will be fine (my installation version is v1.x). After installation, run

aws configure

AWS CLI will ask for the following

aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

Something needs to pay attention
- Region name is important, fill in the one exactly the same as you choose in the control panel
- Fill in the access key and secret with the one created above
- Country list can be found here: https://docs.aws.amazon.com/general/latest/gr/rande.html#endpoint-tables

After the setup, we have right to run authenticated (signed) AWS command for every actions we do through the aws cli.

3. Create IAM (Identity and Access Management) 
- Allow aws cli to have access right to access your services, like running docker commands on behalf of you in ec2 instances, therefore it needs your permission grant. Some security stuff needed to perform first.

- Create IAM group and attach policy to that group

https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html#getting-started_create-admin-group-cli

- Create user and add user to the IAM group
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_cliwpsapi

Amazon strongly recommend not to use AWS account root user for tasks not require root accesses. Instead create an administrator group and put newly created users to that group for management.

3b. However, if your first target is starting from ECS following tutorial in https://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html, (which is indeed suit my case), then it will guide you through all the IAM, group policy and credentials needed to use the ECS services, in this case, just follow the link for the whole setup and ignore step 3 as it is basically a subset of step 3, but specifically for ECS (for docker containers)

Indeed I completely followed the CLI tutorial in this step's URL to create IAM group, users, policies and credentials. And if you already attach "AdministratorAccess" policy to your IAM group, you have already gained accesses to all the AWS services, there is no need to create extras unless you want to grant accesses to other users.

4. Using the ECS CLI to Setup the ECS Cluster and the EC2 Instances
Actually you can follow the first run guide (if you have no docker compose file), the panel will guide you through the setup processes of the container https://us-east-2.console.aws.amazon.com/ecs/home?region=us-east-2#/firstRun, however, I have already defined all the docker containers through docker compose, this method is not suit for me.

I instead use ECS CLI to create the cluster, attach the ec2 instance to cluster and run my containers (through my docker-compose file)

First install and configure the ECS CLI

- Installation of ECS-CLI
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI.html
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_installation.html

- Configure the ECS-CLI
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_Configuration.html

I used the following commands to create the ecs cli connection profile. The access key ID and key is the one created in step 2

Create ECS credential profile to tell ECS how to connect to the remote machine (by providing the credential), profile will be saved as file for later connection use in "ecs-cli up" command

ecs-cli configure profile --profile-name ec2-mw-good-man-eat --access-key $AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY

And the following command to create a cluster profile (for future creation of cluster). Region must be the same as the one chosen in AWS console in step 2. This defines some simple cluster information, the name and the related region (which in turn stored many of the EC2 instances). The --config-name parameter should be referred by the "ecs-cli up" command so that ECS knows how to initialize the cluster.

ecs-cli configure --cluster ec2-mw-good-man-eat --default-launch-type EC2 --config-name mw-good-man-eat-conf --region us-east-2

And finally bring up our ECS cluster using the profiles we have created. The keypair are the one generated in https://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html#create-a-key-pair, note that keypair is not the local one, but the name of the keypair located in "ec2 > Key pairs"

ecs-cli up --keypair keypairName --capability-iam --size 1 --instance-type t2.micro --cluster-config mw-good-man-eat-conf --ecs-profile ec2-mw-good-man-eat

Note that, instance type should be specified as the remote machine type you preferred, for free tier, t2.micro is suited, the --cluster-config parameter is the configuration parameters we have made in "ecs-cli configure"

Now the ECS has created and initialized the cluster and the related EC2 instances, we can arrange the docker containers to be run in the EC2 instances.

5. Setup ECR
Before arranging the docker containers to be run in EC2 instances, 1 of the tricky thing here is that AWS ECS CLI won't have the ability to docker compose build from scratch like what we did in local, instead we need to set up an image registry beforehand to store the custom images for the ecs-cli compose up command. following the above ECR tutorial. ECS will NOT help you to build the image in the EC2 instances "locally" for you. You must build it locally and push to amazon ECR first.
https://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_GetStarted.html

So basically, what "official" images mean images that can be directly fetch from docker official repository, if you directly use official image without modifying, then it is no need to create image repository.

In my case, I have 2 customized images, i need to create 2 container registry to store the images. Note that the images should built in local before using docker-compose command and push using docker-compose push.

Here are the related steps / commands I used

5.1. Create container registry in AWS
Navigate to ecr services, choose "create repository" to create 2 different namespaces to store the 2 images.

Or you can follow this https://stackoverflow.com/questions/44052999/docker-compose-push-image-to-aws-ecr, to manually create repository through command line

5.2. Modify the docker compose file 
Modify the docker compose file as follows
https://docs.google.com/document/d/1Y8fvJiV_YYbwZsoFD9A92eP54-1wzrHdicmXAijd1co/edit?usp=sharing

Points to Note:
- Change the docker-compose version to 2, as ECS does not support minor versions
- Update the image tag to the repository URL created in ECR, to tell ECS which repository the image needs to be pushed to the AWS registry
- Change depends_on to links for inter-container identifications as ECS does not recognize depends_on for the docker-compose file
- Add logging tag to allow capture of logs in case there are errors, we can have some logs for debugging

5.3. Prepare and Push Images to AWS

aws ecr get-login --region us-east-2 --no-include-email

Use the output from the command, directly copy and paste to the command prompt and run again. We are now logged in to AWS ECR with our docker client, we are ready to push the images to the AWS cloud to serve our containers

We should re-build the images in local to make sure they are all up-to-date

docker-compose -f ./docker-compose-prod.yml build

And finally, all we need to do is to push all images

docker-compose -f ./docker-compose-prod.yml push

6. Start up Containers
Basically we have reached the very end of the step, docker compose in the through ecs cli command and up command for initializing the container in the ec2.

One of the difference between local compose and ecs compose is that we have to specify the resources needed for each containers using the ecs parameters file, here is my parameter file

https://docs.google.com/document/d/1PyogvbSTPtarrQyH4MEUBh26CzGRXcW95ezxNKdEXtU/edit?usp=sharing

It specifies how much CPU, RAM resources limited for each container, AWS will follow the specified information to distribute the resources accordingly.

Run ecs-cli compose up command as follows

ecs-cli compose --file ./docker-compose-prod.yml --cluster cluster-name --ecs-profile ecs-profile --cluster-config cluster-config-file --project-name project-name --ecs-params ./ecs-params.yml service up

ECS will parse the docker compose file using their customized docker rules, and if everything is going well, you can see the 4 containers up and running.

And that's it. Your servers are up and running now and are ready to serve. Of course we also need to subscribe Amazon Route 53 service for domain name, but I haven't tried yet, maybe later I will fire another thread to talk about.

Other AWS Features
Usually, we will be satisfied after step 6 as all desired containers are up and running. However a responsible and "talented" developers should think of afterwards domain name for deploying your projects, performance, maintenance and continuous development, thus making this session useful.

1. AWS Backup
Definitely a must do routine works for our cloud server to perform to prevent loss of data. Here is what I have done.

1a. Search and navigate to AWS Backup in service search box

1b. Create the backup plan
Usually I would choose the pre-defined backup plan for the sake of time saving, but I will take a custom backup plan as an example. Basically just follow the guide to fill in the backup details would be just fine except the tags which is really important for the identification of which service's docker volumes you are backing up.

1c. Assigning the backup plan to current resources
This is the most important part of the backup process, after 1b, the backup will not start until you assign the resource to backup plan. Think of backup plan as a clothes, it does not function (keep you warm) until you apply to anyone (wear it).

So the point is, how can we locate the resource to backup. "BY TAGS". After creating the backup plan, there is nothing happen, you need to then navigate to backup plans and select the name of the backup plan you have just created.

Click "Resource assignments > Assign resources", find "Assign resources", and you can choose identified by tags or resources ids, this time I choose "tags", we need to navigate to the EC2 dashboard,  navigate to "volume" (Located under EBS (Elastic Block Storage) > Volumes), navigate to tags, and make the tag ourselves. This tag will be our tag to be filled out in our AWS backup > Assign resources field.

1d. Confirm the backup
If the backup plan is successfully created, you should be able to see there are some backup operations operated in the AWS backup dashboard page

2. Configuring SSL over HTTP and Dynamic Port Mapping
https://jackygeek.blogspot.com/2020/03/aws-dynamic-port-mapping-ssl-equipment.html

3. Register domain name via Route 53
To be updated...

Epilogue
You can see how complicated starting from 0 to make your containers up and running. I can say the whole AWS ecosystem is not so user friendly, especially when you are not quite familiar with linux commands, dockers etc. And the most important thing is Amazon tries to package their services and sell them in quite a number of tiny pieces, with bunch of hard to understand terms and not quite managed documentation, I found their documentation a bit ... confusing, or somehow difficult to follow. And sometimes when following 1 tutorial and then the other tutorial seems performing the same thing with different methods, which spend me much time to make myself clear on its mechanism.

I have already made this tutorial as simple as I could but still complicated, I will try to improve this and create another one specific for the concept of AWS only after getting more familiar with this "game"


References

Thanks for the below references, without them, I definitely cannot make it.

• Free Tier AWS Services & How They Are Billed:  https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-hour-billing/
• Basic EC2 FAQs: https://aws.amazon.com/ec2/faqs/
• What is ECS: https://aws.amazon.com/ecs/
• EC2 and ECS, the differences: https://stackoverflow.com/questions/40575584/what-is-the-difference-between-amazon-ecs-and-amazon-ec2
• Creating Cluster: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-ec2.html
• Getting Started with Amazon ECS: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_GetStarted_EC2.html
• ECS CLI Command
• References: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli.html
• References 2: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_reference.html
• ECS CLI Compose Command
• References: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose.html
• ECS params
• References: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose-ecsparams.html
• ECR
• Introduction: https://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_GetStarted.html
• Use Case: https://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_AWSCLI.html
• Push Images to ECR: https://stackoverflow.com/questions/44052999/docker-compose-push-image-to-aws-ecr
• Route 53
• Routing to ec2 instance:
  https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-ec2-instance.html
• Configuring new domain:
  https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-configuring-new-domain.html
• DNS Configuring:
  https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-configuring.html
• AWS Backup
• Detailed introduction and how-to
  https://docs.aws.amazon.com/aws-backup/latest/devguide/getting-started.html
• Others
• AWS ECS Compose Not Support "depends_on": https://github.com/aws/amazon-ecs-cli/issues/708
• Anonymous Volume Fixed in MongoDB: 
• https://stackoverflow.com/questions/53509236/mongo-authentication-inside-docker
• https://stackoverflow.com/questions/51169488/docker-mongodb-cannot-sign-in-as-root
• Making nginx container for lightweight react server: https://medium.com/@pentacent/nginx-and-lets-encrypt-with-docker-in-less-than-5-minutes-b4b8a60d3a71
• Dockerizing a react app: https://mherman.org/blog/dockerizing-a-react-app/
• Formatting code to blogger: http://hilite.me/