Jacky's Treasure Box: [AWS] Setting Up Multiple Load Balancers & A Very Dumb Mistake

Background
Continue from the previous AWS dynamic port mapping story, although familiarizing the concepts of how a load balancer works in AWS, and recognizing the theory of how different containers with the same container port can co-exists in the same service, but I still got a VERY VERY VERY huge issue not yet solved.

How to Configure Multiple Target Group Within a ECS Service?

The reason why this is a blocking issue for my project is that without multiple target groups, the following requirements cannot be achieved.

HTTPS is a must for all URLs
Extra port in URL (e.g.: www.gme.com:3002) is not okay because HTTPS cannot be made use (already occupied 443 port)
Same container port (80) should be assigned to frontend and loopback containers for ALB
ALB's rule-set configured to

Redirect all incoming HTTP requests to HTTPS requests for security
When gme.com or www.gme.com is requested, direct to gme.com and forward requests to FRONTEND container
When api.gme.com or www.api.gme.com is requested, direct to api.gme.com and forward requests to LOOPBACK container

We can see the last requirement requires us to have 1 ALB and 2 target groups to accomplish, basically the idea is to identify which container to handle the requests by different URLs which in turn being allocated to different containers for responses. When api.gme.com is requested, the DNS (which is created by the LB) which execute the rule-set and know it needs to add a 301/302 redirection HTTPS and forward the request to gme-loopback-target-gp for responses, same for front-end requests

As we have 2 web servers (1 for serving front-end, 1 for middleware (loopback)), we cannot serve 2 servers using the same host port, we instead make use of different ephemeral ports or high number ports (32768 and 32769) for identification.

So the most important part is to create an AWS service with 2 target groups associated with frontend and loopback container respectively.

Problems and Solving Procedures
As of this tutorial, I get used to use ecs cli compose service up command to manage containers composing of the service and task definitions, however ecs cli compose DOES NOT SUPPORT MULTIPLE TARGET GROUPS! At least at the time when I write this article, I am frustrated for making only 1 frontend target group only.

But hey! I want 1 more... for my loopback container. At the end I got stuck 7-10 days wondering how can it be achieved. Furthermore requesting help in stackoverflow also but no luck. I did find the article saying multiple target groups did not currently support but this official article clearly mentioned the support of multiple target groups and even multiple LB, but they aren't using esc cli compose up command, turn out I follow the very basic service creation & task definition combination method to solve my problem. Here is the procedures

1. Under the cluster, create a new version of task, press "Run New Task"

2. Because I already created tasks before (through ecs cli compose syntactic sugar command), just creating a new version of task suit my needs. My option is to configure the task through AWS console, but we can configure through CLI too.

The important configuration here is the "task group", this affects how AWS handles task placement, in my case, the mw-gme is the same as my service name, this "connects" the task and the service.

The funny thing is it has 2 "Tasks" columns here, if you are not linking them while configuring task definition, they are detached, we will not see the task definition "mw-gme:42" in Tasks of cluster. So better specify the service name in "task group"

3. Click "Run Task" button and your tasks will start running the containers

4. Next we deal with creating service, here we use --cli-input-json parameter, feed in the json file for parameters, here are the context of the json file

{
  "cluster": "mw-ecs-gme",
  "serviceName": "mw-gme",
  "taskDefinition": "mw-gme:42",
  "schedulingStrategy": "REPLICA",
  "launchType": "EC2",
  "loadBalancers": [
      {
          "targetGroupArn": "arn:aws:elasticloadbalancing:us-east-2:916381200858:targetgroup/gme-frontend-gp/e2cbd06d8e11d633",
          "containerName": "frontend",
          "containerPort": 80
      },
      {
        "targetGroupArn": "arn:aws:elasticloadbalancing:us-east-2:916381200858:targetgroup/gme-loopback-gp/30e3eccadbb1f904",
        "containerName": "loopback",
        "containerPort": 80
    }
  ],
  "desiredCount": 1
}

The most important parameters here are cluster, serviceName, taskDefinition, and loadBalancers! Yes! And you see it allows me to put in an array! I can put multiple target groups here! Amazing, I can now setup gme-target-group-frontend to forward the matching rules to frontend container, while gme-target-group-loopback to forward the matching rules to loopback container, and both have host port of 80, that's why we can apply HTTPS on both of the containers to prevent the following errors

This error encounters when frontend (https://gme.com) is made HTTPS but not http://api.gme.com, we can't have mixture of HTTPS and HTTP put together, we must have all HTTPS equipped. That is why we need both 80 as container port.

And we can see the service name "mw-gme" here is the same as the name of the task group specified in step 2, this is very important they are with the same name, when doing so, tasks in cluster will show up the task definition entry (mw-gme:42).

5. Run aws ecs create-service --cli-input-json file://ecs-service.json, where ecs-service.json file is the one we specified in step 4.

And that's it! You will see service is started with 2 target groups in load balancing column and task definition is with status "RUNNING", for configuration of ELB and target group, refer to the previous thread.

You may wonder what takes me 7-10 days for such setup, 2 main points stuck me during the deployment.

1. Insisting there is such a way to allow me to deploy multiple target groups using ecs cli compose service up command, but indeed it is NOT SUPPORTED! Look at the command provided in official doc

ecs-cli compose --file ./docker-compose-aws-prod.yml --cluster mw-ecs-gme --ecs-profile mw-ecs-profile --cluster-config mw-ecs-gme-config --ecs-params ./ecs-params.yml --project-name mw-gme service up --target-group-arn arn:aws:elasticloadbalancing:us-east-2:916381200858:targetgroup/gme-gp/ed4eb5d138cf37df --container-name frontend --container-port 80

We can see here how convenient this syntactic sugar command provided for us to easily bring up both the services and the associated task definition together, frankly speaking it is with no differences from the method I mentioned above (creating service and task definition), but it has its limitation. We can see parameter "--target-group-arn", "--container-name" and "--container-port" only allows creating 1 set of target group (with 1 set of container match). Stuck here for several days, turn out multiple target group is not supported in ecs-cli compose service command! Hope Amazon will improve this command in the future.

2. And the dumbest thing and spent me further 4-6 days is when I used service creation and task definition to finally created a multiple target groups load balancer service, the loopback server is still failing the health test of the LB, it is weird as I have confirmed these things already

- Host port 32769 incoming port has opened in security rule
- 2 target groups are created
- Containers and port mapping configured successfully through info provided by AWS console UI
- A record entry has been added in route 53
- Frontend (https://gme.com) is doing fine (passing the healthy test of LB, target port 32768)

I started a thread and seek help in stackoverflow, confirming my approach and steps are correct, but still no luck... And after 4-5 days struggling, I found a huge huge mistake, making all these not working, it is NOT related to all the settings of AWS, all are good and fine, then I started to think it may be come from the exposed port settings defined in loopback server itself...

As we can see, I can't image the problem comes from the loopback server configuration, I totally forgot to change the port settings to 80, while keeping it as the development port, after changing it back to port 80, healthy check passed and both https://gme.com and https://api.gme.com is up and running fine, what a silly mistake I have made!

Further Notes
You may not want to stop the service and renew the task definition again if you are just updating the docker registries, in this case, "aws ecs update-service" command is the way to go.

Executing "aws ecs update-service --service mw-gme --force-new-deployment --cli-input-json file://ecs-service-update.json", will update the service with the parameters provided, and most important is the "--force-new-deployment parameters, forcing ecs to retrieve the docker images again even they have the same tag.

And 1 more, 301 (permanent redirection) is very dangerous, if you tried HTTP 301 settings in server, redirecting all HTTP requests to HTTPS requests, in client devices, these permanent redirection is being cached with no expiry date, which means when you try to remove HTTPS and turn back to HTTP (I needed to do so at that time because I am stuck in setting up LB and need to provide services to users first), client device will still being cached to navigate to HTTPS version of the site which will cause 404 error, I would definitely recommend 302 (temporary redirect) if it is the halfway of the project and you intend to switch back temporarily.

References

How to register multiple target groups via aws ecs cli service command

AWS application load balancer not working

ECS - how do you resolve unable to place a task

ECS container ALB health checking

What is application load balancer

How to run more than 1 app on 1 instance of ec2

Listener for application load balancer

Difference between a task and a service (AWS)

Linux - check whether the port is in used

Is it possible to run multiple web instance in the same AWS EC2?

Containers behind ELB in ECS?

Listener rules for AWS