Ecs, CircleCI, NLB and Grpc

I finished a migration from Heroku to ECS using CircleCI orbs for AWS.

I detail the aspect of the migration and the peculiarities of the project.

CircleCI

 

I knew of CircleCI back from 2016. I just knew that it was a more accessible tool to configure than Jenkins, and the project I was working on already had it.

Back then, I did not pay much attention to it since my main task was developing.

Cue to mid-2019, the company I'm working on switches from Jenkins to CircleCI. They did not want to spend resources, time, or anything. They want to "have it working without spending time fixing it.".

They move to CircleCI, and everything is working without the problems we had with Jenkins.

This new project I was working on was using Heroku with a bash script.

We opted to use ECS, and we found that we had support for it.

The documentation for the ECS orb is not clear at all if you don't spend time reading the terms of CircleCI orbs.

There is no clear differentiation between the job and the workflow, and I spent days confused.

ECS

 

I already talked about ECS, three years later and I'm back with it. Things are pretty much the same as before, now we've got fargate, that I have not used, but the documentation on crucial aspects like green/blue deployment is tailored towards that, which sucks, but is what it is.

Things are the same, you create a cluster, you create services, and you place tasks on it.

The new things I'm using this year are;

1. Green/Blue deployment (soon).
2. NLB for GRPC (more on it in the GRPC section).

What I do have to recommend is that if you are planning on expensive IO operations and you are planning on using "t2.small" as the best machine, then stay away from ECS. You are going to have downtime because the IO consumes the CPU credits fast.

We've got a situation with a monolithic application that we won't spend time refactoring. Still, the section of the code that works with PDF's has a lot of IO code, some parts are abusing IO due to the library they are using, and some parts abuse IO in the monolithic application itself.

Grpc

 

Again with GRPC. In the same situation, learn a bit more about how to debug the server internally, how to force the channel closure.

The "options" parameter for the server and the channel on the client receive a list of tuples. It lacks documentation in Python, and you need to dig deep into the source for figure how things work.

Options that you can use are also missing in Python, so I ended up reading Go examples that gave me an idea.

NLB

 

I'm writing this document on 11/21/19, NLB does indeed work with ECS EC2 deployment type without any problem.

The crux of the problem

 

I am using ECS (EC2) with a network load balancer to have the GRPC server working.

We are using internal NLB because we are not exposing GRPC to the outside world since this is just for our microservices.

The main problem is that the NLB balancer does not balance the machines behind it. Once the NLB opens a connection, it keeps on reusing it, no matter if you close the channel in the client.

That cascades the problem that If I create an autoscaling group because the EC2 instance serving is degrading, I don't have a way to force that without having downtime, which indeed it sucks.

In theory, GRPC also offers to balance at the client level, but since I'm doing a deployment in ECS, I don't have an excellent way to fix the IP of the server since the IP will change after the first deploy.

I don't know, and perhaps I could opt for a strategy like placing an elastic IP and find a way to articulate this in ECS?.

I will write my findings if I ever find a solution to this problem.

Sources and links used during this research.

• Resume of someone that came to the same conclusions as me a good couple of months before.
• How to debug the server
• Environmental variables
• Examples of servers and how to pass options
• How to deploy in production and how to pass options. Excellent read.
• More argument variables

2017 - 2018 - 2019 update

A long-time has passed since my last entry.

Let's recap. I was stuck in a really awful contract in fin-tech.

I couldn't bear it anymore and got a better job.

Just about 3 months until the end of 2017, I joined Shiftgig in Argentina.

Better pay, serious architecture, a promise to learn more, a lot of developers, and a possible career in leadership.

And the company was sold in December 2017.

Jobless.

I spend Christmas and the end of the year without spending too much cash.

Small savings, big mistake.

Fuck, I was lost.

I spent less than a month and got a new contract with Thirstie.

Remote work, much more pay than Shiftgig.

0 hassle.

Things went well for a year, I don't have much to comment, I was supposed to become a leader again.

January 1st, 2019, break up with my girlfriend of 2 years.

February 15th, 2019, my tutor and director of operations in Argentina that was preparing me to be a leader has to hire a CTO.

CTO fires him when he visits us.

The promise to become a leader vanishes, the team members that were with me do not acknowledge that we have a leader, despite me being the supposed person.

August 2019, Argentina again with the your dollars are not your dollars. My income is jeopardized. They force you to convert your hard-earn money to peso, the national currency that is worth a roll of toilet paper.

I took most of my dollars from the bank account and put them in a safe location.

I pay taxes, I'm a "responsable inscripto."

I pay taxes, 35% of my annual income next year.

I made advanced payments due to my income, about 40k pesos for 5 months.

They nuked my income with this law.

I can't save no more. For the first time in my life, I had savings, enough to live for years without working. They force me into that devaluated currency they use and take most of what I win.

In Argentina, there is no way to pay what I earn in the States, they devaluate the peso, and you don't have a way to cancel your debt with the State. Boom, welcome fiscal problems.

The utter stupid ideas that I hear when they tell me this is not what it is is ridiculous.

The only 100% legal options are buying dollars, which they will forbid soon. The previous government, Cristina Fernandez de Kirchner (CFK for short) authoritarian party did the same. They "offered dollars" but you had to ask permission AFIP (IRS if you are from the States) to buy and most of the time, you couldn't. Politicians could get their shitty hands in dollars without problems though.

The other option is to get into FCI investment, which this week I read news about people that invested in that got scammed and lost.

Doing things properly and following the law in Argentina is stupid, dangerous, and counterproductive.

Paying taxes is counterproductive, the system is made to bankrupt you, no matter what you do.

I'm not a huge company, so I don't have the resources to defend myself against the State. I don't have a viable future in this country. Being a remote developer works when we've got free access to dollars. *We bring dollars into the country*, we don't move the dollars out.

August, I left Thirstie, nothing to learn, nothing to gain here.

I start working for Vero, will see what happens. At the moment I've got a nice contract, but the economic situation here is awful.

Future?.

Well, Alberto Fernandez seems to be the one that is going to win, and he comes back with Cristina Fernandez de Kirchner.

So this means rampant corruption, authoritarianism. 

Now they come back with a payback.

Welcome back the mindset that peso is good, the dollar is bad.

Forbid the entrance of external goods, block all imports, so you have the idiots here that buy the cheapest, shittiest Chinese material, slap a sticker that says "made in Argentina" and voila, the national industry is back!. Hey, that's important, because it creates jobs. I'm sure that after the protectionism is lifted, we will have the most advanced solutions for everything like it always has happened in this country. It's impossible to think that they will use the subsidies to get a metric shit-ton of money and become rich. No sir. They will invest in the development of my country. Yes, no concerns at all.

Macri was a total and absolute disaster. I've never seen someone fuck up so badly, crash, burn and take us down with him, well, except that he won't have problems.

Fintech

More experiences

And this is a new entry on this forgotten blogger site I've created and barely update.

What I'm up to.... the microservices fiasco.

Long story short, the microservices was just a charade because they didn't have any offer to give me after the contract was lost.

That project grew up a bit until they didn't have more funds.

Then they put me in whatever project they could find for me, and that was a company that installed solar rooftops. A 4-page flask application with a lot of heavy trigonometry calculation with 0 documentation and full of callbacks with 0 documentation.

They had tests for everything, but unluckily for me, I was stuck with a specific case of an AutoCAD file that when it was going through the flow, it produced an error.

The code was complicated, hard to read, and I'm not useful in trigonometry or electrical things, so my ability to provide reasonable solutions was 0. That is why I decided to leave.

Now 

I received an offer to work with a fin-tech company. I was excited, finance was a sector I always wanted to get involved with, since I read test-driven development by example, back in 2009.

And let me tell you. I wish I had never fulfilled my dreams.

Reality is awful.

The only place for my TDD book in this company would be to put it under a monitor to lift up the monitor.

The company is a startup, they honestly don't give a damn at all for any kind of testing.

No unit tests.

No TDD.

No integration testing.

No pep8.

I've overheard them that they use threading. I asked, don't you have problems with the GIL?.

They answered me, "what is the GIL?".

That is not the answer I was expecting and this is not an elaborated joke.

And no, I'm not using hyperbole, they literally answered me that.

I don't like this new job.

The goal of this client is focused on UI, creating Facebook pages,

and literally doesn't care if one of his customers is purchasing stock options has an error.

What happens if the transaction never occurs or happens and you have an error.

The company hires US university students and doesn't pay them.

They lure them with one of those fucking internships where they get "recommendations" for their services.

This girl that they hired, she did excellent work, and the pieces of shit of my coworkers laughed about her.

Micro services

Intro

And we are back.

After 10 years, I'm still working in software development.

What I'm working on.

Well, after what I believe to be a failed contract, my company decided that I should be researching microservices, AWS (ECS, EC2, SES, SQS, SNS), Elastic (soon to be replaced by mongo).

What is each acronym that I'm using there?

AWS: Amazon Web Services. Yeah, I'm still on AWS. Been using it on my previous employer (a company in Argentina, that had a contract with someone from the States), where I had some sort of devops / developer position.

ECS: This is an Amazon service that allows you to use Docker inside amazon EC2 containers.

Before giving you my explanation, without reading this article by Yevgeniy Brikman, I could have never ever set up my docker tasks in ECS.

Give it a try to that link first, is the best explanation I found, way better than the amazon documentation.

The main problem that I see with the amazon guide is that you don't have an ordered tutorial.

The article by Yevgeniy Brikman is excellent.

You have the documentation, but it doesn't follow a proper order.

The first thing you will need to have if you are going to use this is an EC2 instance of the type that is optimized for ECS, for example (amzn-ami-2016.03.d-amazon-ecs-optimized)

Amazon covers the containers on the documentation, but is not the first thing.

They start the documentation with creating private docker registries that you will need way later on.

SES: Is used to send emails using the SES amazon service. SES stands for Simple Email Service.

SQS: Amazon Simple Queue Service. A simple queue, self-explanatory

SNS: Amazon Simple Notification Service. This is used to deliver notifications.

Together, SQS + SNS = Like rabbitmq. Amazon does all the heavy lifting for you. Though it has some peculiar things. Due to high replication, there is a chance that depending on how you configure your SQS, one or more workers may receive the same message published by the SNS. They state that in the documentation, your workers must be able to handle properly if a message was already processed. You can handle the visibility of a message once is consumed with a timeframe that you can configure.

The other part of my highlight is going to be the SES part.

SES is like mandril (or MailChimp now). The main thing is when you start the service, you are in the sandbox mode. You will need to verify an email, and you will be only able to send emails to that verified domain.

You need to fulfill a request, so they allow you to deliver emails.

The most important part here is that they ask you how you handle bounces and complaints.

You can create an SNS topic that will receive the bounces/complaints/ deliveries, and you will hook it up with an SQS consumer.

All that, by clicking with the mouse.

Seems pretty straight forward, but they didn't put that on the page.

The main idea is that if you receive a complaint or a bounce, you just take it out of circulation (i.e., don't try to deliver emails to that recipient).

With which language?

Python 3.5.1 (yeah, I decided to use something newer)

I'm using uwsgi / flask, with the following extensions (used across some of the microservices)

• flask-jwt
• alembic
• flask-sqlalchemy
• greenlet
• boto3
• flasgger
• swagger-ui

Flask-JWT: Is an extension to create JWT tokens, that will be consumed by a JS client in my case.

The extension has some gotchas, you need to read the code if you need to change things.

With some monkey-patching and inheritance, you can alter the functionality of JWT completely, but you need to spend some time reading.

flasgger: I love flasgger. I was using flask-swagger, but being honest. The heredocs under the function names make the whole thing illegible. Flasgger lets you point to a file. If you don't know what is this, this is used to write documentation for swagger-UI.

Swagger-ui: The only complex thing that I needed to do here, was to add support for the JWT token, since some endpoints are protected by JWT tokens, I didn't had a way to push the token there.

I found the way.

Authentication.

I was able to handle this by using JWT. When you are developing microservices, and you have to manage authentication, there are multiple things to take into account.

Disclaimer: this is not a definitive guide. At the moment I'm experimenting. So far is my best solution, but I believe that there are other solutions out there.

If you split a user service and a permission service, moving that token will become tricky, and it's difficult to think sometimes.

The idea is that you pass around that token until it expires.

No token, no service. Simple.

Obtaining the permissions of the entity after logging is a must, and I think that a caching strategy will be needed there.

A simple example, I log in, I obtain my permissions and put some key, like user_id: permissions in a Redis storage.

On the next request, I will ask the Redis for that user permissions.

When for some reason I decide to revoke a permission, I will delete the Redis record, and well, they will need to fetch the permissions again.

This is a work in progress, which I may extend much further later on.