Ecs, CircleCI, NLB and Grpc

I finished a migration from Heroku to ECS using CircleCI orbs for AWS.

I detail the aspect of the migration and the peculiarities of the project.

CircleCI

 

I knew of CircleCI back from 2016. I just knew that it was a more accessible tool to configure than Jenkins, and the project I was working on already had it.

Back then, I did not pay much attention to it since my main task was developing.

Cue to mid-2019, the company I'm working on switches from Jenkins to CircleCI. They did not want to spend resources, time, or anything. They want to "have it working without spending time fixing it.".

They move to CircleCI, and everything is working without the problems we had with Jenkins.

This new project I was working on was using Heroku with a bash script.

We opted to use ECS, and we found that we had support for it.

The documentation for the ECS orb is not clear at all if you don't spend time reading the terms of CircleCI orbs.

There is no clear differentiation between the job and the workflow, and I spent days confused.

ECS

 

I already talked about ECS, three years later and I'm back with it. Things are pretty much the same as before, now we've got fargate, that I have not used, but the documentation on crucial aspects like green/blue deployment is tailored towards that, which sucks, but is what it is.

Things are the same, you create a cluster, you create services, and you place tasks on it.

The new things I'm using this year are;

1. Green/Blue deployment (soon).
2. NLB for GRPC (more on it in the GRPC section).

What I do have to recommend is that if you are planning on expensive IO operations and you are planning on using "t2.small" as the best machine, then stay away from ECS. You are going to have downtime because the IO consumes the CPU credits fast.

We've got a situation with a monolithic application that we won't spend time refactoring. Still, the section of the code that works with PDF's has a lot of IO code, some parts are abusing IO due to the library they are using, and some parts abuse IO in the monolithic application itself.

Grpc

 

Again with GRPC. In the same situation, learn a bit more about how to debug the server internally, how to force the channel closure.

The "options" parameter for the server and the channel on the client receive a list of tuples. It lacks documentation in Python, and you need to dig deep into the source for figure how things work.

Options that you can use are also missing in Python, so I ended up reading Go examples that gave me an idea.

NLB

 

I'm writing this document on 11/21/19, NLB does indeed work with ECS EC2 deployment type without any problem.

The crux of the problem

 

I am using ECS (EC2) with a network load balancer to have the GRPC server working.

We are using internal NLB because we are not exposing GRPC to the outside world since this is just for our microservices.

The main problem is that the NLB balancer does not balance the machines behind it. Once the NLB opens a connection, it keeps on reusing it, no matter if you close the channel in the client.

That cascades the problem that If I create an autoscaling group because the EC2 instance serving is degrading, I don't have a way to force that without having downtime, which indeed it sucks.

In theory, GRPC also offers to balance at the client level, but since I'm doing a deployment in ECS, I don't have an excellent way to fix the IP of the server since the IP will change after the first deploy.

I don't know, and perhaps I could opt for a strategy like placing an elastic IP and find a way to articulate this in ECS?.

I will write my findings if I ever find a solution to this problem.

Sources and links used during this research.

• Resume of someone that came to the same conclusions as me a good couple of months before.
• How to debug the server
• Environmental variables
• Examples of servers and how to pass options
• How to deploy in production and how to pass options. Excellent read.
• More argument variables

Micro services

Intro

And we are back.

After 10 years, I'm still working in software development.

What I'm working on.

Well, after what I believe to be a failed contract, my company decided that I should be researching microservices, AWS (ECS, EC2, SES, SQS, SNS), Elastic (soon to be replaced by mongo).

What is each acronym that I'm using there?

AWS: Amazon Web Services. Yeah, I'm still on AWS. Been using it on my previous employer (a company in Argentina, that had a contract with someone from the States), where I had some sort of devops / developer position.

ECS: This is an Amazon service that allows you to use Docker inside amazon EC2 containers.

Before giving you my explanation, without reading this article by Yevgeniy Brikman, I could have never ever set up my docker tasks in ECS.

Give it a try to that link first, is the best explanation I found, way better than the amazon documentation.

The main problem that I see with the amazon guide is that you don't have an ordered tutorial.

The article by Yevgeniy Brikman is excellent.

You have the documentation, but it doesn't follow a proper order.

The first thing you will need to have if you are going to use this is an EC2 instance of the type that is optimized for ECS, for example (amzn-ami-2016.03.d-amazon-ecs-optimized)

Amazon covers the containers on the documentation, but is not the first thing.

They start the documentation with creating private docker registries that you will need way later on.

SES: Is used to send emails using the SES amazon service. SES stands for Simple Email Service.

SQS: Amazon Simple Queue Service. A simple queue, self-explanatory

SNS: Amazon Simple Notification Service. This is used to deliver notifications.

Together, SQS + SNS = Like rabbitmq. Amazon does all the heavy lifting for you. Though it has some peculiar things. Due to high replication, there is a chance that depending on how you configure your SQS, one or more workers may receive the same message published by the SNS. They state that in the documentation, your workers must be able to handle properly if a message was already processed. You can handle the visibility of a message once is consumed with a timeframe that you can configure.

The other part of my highlight is going to be the SES part.

SES is like mandril (or MailChimp now). The main thing is when you start the service, you are in the sandbox mode. You will need to verify an email, and you will be only able to send emails to that verified domain.

You need to fulfill a request, so they allow you to deliver emails.

The most important part here is that they ask you how you handle bounces and complaints.

You can create an SNS topic that will receive the bounces/complaints/ deliveries, and you will hook it up with an SQS consumer.

All that, by clicking with the mouse.

Seems pretty straight forward, but they didn't put that on the page.

The main idea is that if you receive a complaint or a bounce, you just take it out of circulation (i.e., don't try to deliver emails to that recipient).

With which language?

Python 3.5.1 (yeah, I decided to use something newer)

I'm using uwsgi / flask, with the following extensions (used across some of the microservices)

• flask-jwt
• alembic
• flask-sqlalchemy
• greenlet
• boto3
• flasgger
• swagger-ui

Flask-JWT: Is an extension to create JWT tokens, that will be consumed by a JS client in my case.

The extension has some gotchas, you need to read the code if you need to change things.

With some monkey-patching and inheritance, you can alter the functionality of JWT completely, but you need to spend some time reading.

flasgger: I love flasgger. I was using flask-swagger, but being honest. The heredocs under the function names make the whole thing illegible. Flasgger lets you point to a file. If you don't know what is this, this is used to write documentation for swagger-UI.

Swagger-ui: The only complex thing that I needed to do here, was to add support for the JWT token, since some endpoints are protected by JWT tokens, I didn't had a way to push the token there.

I found the way.

Authentication.

I was able to handle this by using JWT. When you are developing microservices, and you have to manage authentication, there are multiple things to take into account.

Disclaimer: this is not a definitive guide. At the moment I'm experimenting. So far is my best solution, but I believe that there are other solutions out there.

If you split a user service and a permission service, moving that token will become tricky, and it's difficult to think sometimes.

The idea is that you pass around that token until it expires.

No token, no service. Simple.

Obtaining the permissions of the entity after logging is a must, and I think that a caching strategy will be needed there.

A simple example, I log in, I obtain my permissions and put some key, like user_id: permissions in a Redis storage.

On the next request, I will ask the Redis for that user permissions.

When for some reason I decide to revoke a permission, I will delete the Redis record, and well, they will need to fetch the permissions again.

This is a work in progress, which I may extend much further later on.