Mixing Node.js API with AWS Lambda

Question

I have a never ending discussion with my manager about the usage of AWS Lambda. I would like to get some help from one of you guys.

I am a bit hesitated to utilize serverless architecture for production level projects yet. First of all, its a bit time-consuming to test what I am building in the local setup. Even if we can test the code through the unit testing, we cannot remove the possiblity of faliure of mocked request and response objects. The fact that I cannot invoke the lambda in my local setup makes me very bored when I am testing my Lambda-written API during the development process. Secondly, as far as I know so far, there is no promised SLA for AWS Lambda for its availability and reliability. This just make me kind of hesitate adopting Lambda to build RESTful API.

So now what I do is to use Lambda only the case when to catch events triggered from AWS. For example, to do something after a user uploaded his profile picture to S3 bucket or to do something after a user registered throuh Cognito.

However, what my manager expects is to mix my Node.js API with AWS Lambda for a single project. From my perspective it totally doesnt make sense. Once we have set up Node API on EC2 instances, I think it will be more productive to think about setting up auto-scaling or how to utilize all the resources running on the current EC2s. But my manager insists me to set up both Node API and Lambda API altogether. For example, the services A and B will be served by Node API and the services C, D, and E by AWS Lambdas.

I tried it before, but it caused a lot of confusion to me. I feel like it will be better to choose either Node API or AWS Lambda API when building an API instead of mixing them together.

I don't want to say that my manager is totally false and I am right. I want to just have a clear answer in this case. I would really appreciate any comment and answers on this situation.

Answer 1

Just adding some thoughts on the previous answers:

First of all, its a bit time-consuming to test what I am building in the local setup. Even if we can test the code through the unit testing, we cannot remove the possiblity of faliure of mocked request and response objects. The fact that I cannot invoke the lambda in my local setup makes me very bored when I am testing my Lambda-written API during the development process

For sure you can build, test and simulate a Lambda Invoke in your local environment, it's just a new paradigm and there's some tools to help you out.

Secondly, as far as I know so far, there is no promised SLA for AWS Lambda for its availability and reliability. This just make me kind of hesitate adopting Lambda to build RESTful API.

AWS Lambda operates on the AWS "compute layer" infrastructure, so I believe if they face an issue on the compute layer, for sure your EC2 instances will also face an outage.

Once we have set up Node API on EC2 instances, I think it will be more productive to think about setting up auto-scaling or how to utilize all the resources running on the current EC2s

I don't think so. The serverless stack scales with zero effort, you don't need to manage infrastructure.

I tried it before, but it caused a lot of confusion to me. I feel like it will be better to choose either Node API or AWS Lambda API when building an API instead of mixing them together.

Welcome to micro and decoupled services! Where developing the service is easy, but managing the whole infrastructure is hard.

Another thing to keep in mind when talking to managers about architecture: Cost

It's hard to argue and makes every manager's eyes shine when they see the possibility of running their businesses with a low cost. And having your service running with a serverless stack is really cheap.

Bottom line:

No, it's not a bad idea to mix resources as your manager wants.

Yes, it's easier to just get a framework to write the api, setup an EC2 instance and a auto-scaling group.

Yes, there's a heavy lift when decoupling services, but it pays its price when running in production.

Answer 2

OK let's go one by one. First thing first. Your first problem is to test Lambda in your local and it completely possible to do with SAM.

Please have a look on - http://docs.aws.amazon.com/lambda/latest/dg/test-sam-local.html

The most important design decision, If your application is monolithic and you don't want to redesign it to microservices then stick with EC2.

Next regarding your design for hybrid API (Lambda and EC2). I don't think that is an anti-pattern or bad idea to do. It completely based on your requirment. Say you have an existing set of API's in EC2 (May be monolithic) and you want to slowly migrate it to serverless and micro services. You don't need to migrate it all together to serverless. You can start one by one. Remember the communication is happening through Http and it doesn't matter if your services are ditributed between EC2 and Lambda. In micro service world it doesn't matter services are in same server or ditributed across many servers.

The communication speed has drastically changed over last few years and that's one of the main reasons of emergence of micro service.

So in a nut shell it is not a bad idea to have hybrid API's but completely based on your design architecture. If monolithic then don't go for Lambda.

Answer 3

There are instances, where you do need to run Lambda and EC2 instances together (E.g Monolith to Microservices migration projects, NodeJS with Express as a WebServer) which can make sense.

There are multiple approaches to achieve this. Two common approaches(For request-response) are to use.

1. If you plan to serve only RESTful APIs from both Lambda & NodeJS EC2 instances you can use API Gateway as a proxy for both of them.
2. If NodeJS EC2 instance is used as a WebServer to serve both dynamic pages & etc, then you can use AWS CloudFront as a proxy & still you will require API Gateway to connect with Lambda.

Note: For asynchronous workflows, you can use Lambda along with other AWS services such as AWS Step Functions, SQS, SNS based on the nature of workflow.