AWS Next.js CRUD app (infrastructure)

Wednesday, September 18, 2024

Updated: Thursday, September 26, 2024

In today's digital landscape, creating custom software to manage user data is crucial for organizations looking to gain insights and improve their operations. Many large-scale SaaS solutions collect, analyze, and leverage user information, but understanding how to build these capabilities from the ground up is essential for anyone looking to develop tailored applications. In this post, we'll walk through the steps to create a simple CRUD (Create, Read, Update, Delete) app using AWS and (in a subsquest post) Next.js. This hands-on guide will help you establish a practical foundation for building custom software that empowers organizations to collect, manage, and utilize user data effectively. Whether you're new to development or seeking to enhance your skills, this project will give you the tools to get started on building data-driven applications.

In this project, we'll be building and deploying a Next.js app running in ECS (Elastic Container Service) that will serve as the front-end, and AWS services like DynamoDB and Lambda which will serve as the back-end. We will store the data in DynamoDB (a NoSQL database) and use Lambda functions to perform our CRUD operations against the database. To get started, you will need to configure your local machine to run Terraform against an AWS account, set up your Terraform remote backend state resources, configure the Terraform provider.tf file, etc., there is a brief guide on how to do this with external links at the start of my first blog post. Additionally, code we'll be going over is available here on Github.

So let's get started, we will begin by provisioning our back-end with Lambda functions and a DynamoDB table, with the Lambda functions being created using my Lambda Terraform module which can be found here, also on Github.

However, before getting started, let's take some time to understand Lambda functions, what are they and how can they help us create our backend?

Lambda Functions Overview

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. Below is a breakdown of how Lambda functions work and the options available for securing their endpoints through Lambda Function URLs.

What is a Lambda Function?

AWS Lambda is a serverless compute service that automatically runs your code in response to various events, such as HTTP requests, file uploads to S3, changes in DynamoDB tables, and many more. Serverless computing means that you don't have to worry about provisioning or managing servers. Instead, the infrastructure is entirely managed by AWS, allowing you to focus solely on writing your code.

One of the key benefits of AWS Lambda and serverless computing, in general, is its cost-effectiveness. With traditional server setups, you would need to provision and pay for servers to run your applications, often resulting in unused capacity during off-peak times. In contrast, AWS Lambda only charges you for the compute time that your code actually consumes. You are billed based on the number of requests and the duration it takes for your function to run, measured in milliseconds. This "pay-per-execution" model ensures that you are only charged when your code is actively running, making it an efficient and cost-effective solution, especially for applications with variable workloads.

This model makes Lambda ideal for workloads that don't require constant server uptime. For example, functions that are triggered by events like user uploads or occasional API calls can run without the overhead of managing idle resources. Even for high-scale applications, AWS Lambda automatically scales based on the incoming requests, ensuring that you don't have to pay for over-provisioned infrastructure during quieter periods, making it a highly scalable and budget-friendly choice.

Lambda Function URLs

Lambda Function URLs provide a simple way to expose your Lambda functions as HTTP endpoints. This allows you to create RESTful (Representational State Transfer) APIs (Application Programming) directly from Lambda without needing an API Gateway. When you create a function URL for your Lambda function, AWS automatically provides a URL that can be used to invoke your function from any HTTP client.

The Lambda Function URL supports two authentication options, depending on your security needs: NONE and AWS_IAM.

Authentication Options: NONE vs AWS_IAM

When configuring a Lambda Function URL, you can choose between two authentication methods:

NONE: This option makes the function URL publicly accessible. Anyone with the function URL can invoke the Lambda function without any authentication or authorization, which is useful for testing or for public endpoints where security is not a concern.

AWS_IAM: This option secures the function URL using AWS IAM (Identity and Access Management). Only users or services with the necessary IAM permissions can invoke the Lambda function. This is useful for internal APIs or endpoints that require secure access. By using AWS_IAM, you can control who has access to the function URL based on their IAM roles and policies, ensuring secure communication with the Lambda function.

Event-Driven Architecture

Lambda functions are often part of an event-driven architecture. They are triggered by various AWS services, like API Gateway, S3, DynamoDB, and EventBridge. When an event occurs, the Lambda function is invoked automatically, scaling up as necessary to handle the event load. This makes it an excellent choice for scenarios like real-time data processing, responding to HTTP requests, or performing automated tasks based on cloud events.

By using event triggers, Lambda enables seamless integration with the rest of your AWS infrastructure without worrying about managing underlying resources.

AWS Lambda is a powerful serverless computing option that provides scalability, cost-efficiency, and integration with other AWS services. Lambda Function URLs further simplify the process of exposing Lambda functions as HTTP endpoints, while giving you the flexibility to secure them with authentication options suited to your needs. Whether you need public access (NONE) or secure, controlled access (AWS_IAM), Lambda can handle a variety of use cases with minimal infrastructure management.

For this demonstration, we will work with the NONE authentication option, but if you want to explore using AWS_IAM instead, contact me and I can assist. Moving on, now that we understand what Lambda is, let's dive in. We will be creating four separate Lambda functions, each for the operations we'll be executing from our front-end app. Our operations will follow the RESTful API methodology, and if you're unfamiliar with RESTful, a definition is provided below:

A RESTful API is a web service that allows different applications to communicate over HTTP by following a set of architectural principles. It treats everything as a resource, which is accessible via a unique URL, and uses standard HTTP methods like GET, POST, PUT, and DELETE to perform actions on these resources. RESTful APIs are stateless, meaning each request must contain all the necessary information, making them highly scalable and flexible. They are widely used due to their simplicity, scalability, and interoperability with different platforms and languages.

Take some time to look through the four Lambda functions' Node.js code, you'll see that I'm just calling DynamoDB functions from the AWS SDK (Software Development Kit) for Node.js, these functions imported will vary based on the RESTful methods associated with each function. The two functions running PUT and DELETE operations will expect data in the request's body, data such as the unique identifier for the item we will try to update or delete.

Before writing Terraform code for our Lambda functions, we need to prepare our configuration for the DynamoDB table the functions will execute operations against. Let's also understand what DynamoDB is and it's features.

DynamoDB Overview

Amazon DynamoDB is a fully managed NoSQL database service designed to deliver fast and predictable performance with seamless scalability. Below is a breakdown of how DynamoDB works and key features like Primary Keys, Sort Keys, and more.

What is DynamoDB?

Amazon DynamoDB is a fully managed NoSQL database service that provides fast and flexible database performance for applications that require low-latency and high throughput access to data. DynamoDB is designed for use cases such as real-time analytics, mobile applications, gaming, IoT and e-commerce.

As a fully managed service, DynamoDB automatically scales the storage and throughput capacity to meet the demands of your application. This makes it ideal for workloads that experience unpredictable traffic patterns, as it ensures high availability and low operational overhead.

Data Structure: Tables, Items, and Attributes

In DynamoDB, data is organized in tables. A table is a collection of items, where each item is a group of attributes. This is similar to rows and columns in a relational database. However, in DynamoDB, different items in a table can have different sets of attributes, offering flexibility in how data is structured.

Items in a DynamoDB table are uniquely identified by a primary key. You can optionally use a sort key to organize your data further, allowing multiple items to share the same primary key but be differentiated by the sort key.

Primary Keys and Sort Keys

DynamoDB uses Primary Keys to uniquely identify items within a table. A primary key can be either a simple primary key (Partition Key) or a composite primary key (Partition Key and Sort Key):

Partition Key: The partition key is a single attribute that uniquely identifies an item in a table. DynamoDB uses the partition key’s value to determine how data is distributed across partitions in storage.

Composite Key (Partition Key and Sort Key): In tables with a composite primary key, each item is uniquely identified by the combination of both a partition key and a sort key. The partition key is used to group related items, and the sort key is used to order those items within the partition. This is useful for queries that need to retrieve items based on the same partition key but sort them by another attribute.

Scalability and Performance

DynamoDB is built to scale horizontally by distributing data across multiple partitions based on the partition key. This allows the database to handle large volumes of read and write operations seamlessly, making it an ideal solution for applications with high throughput requirements.

The performance of DynamoDB can be adjusted based on the required read and write capacity through either on-demand or provisioned modes. The on-demand mode automatically scales to meet your traffic needs, while provisioned mode allows you to reserve specific read/write capacity for predictable workloads.

Event-Driven Architecture with DynamoDB Streams

DynamoDB Streams provide a powerful way to enable event-driven architectures. Streams capture a time-ordered sequence of item-level changes in your DynamoDB table, such as inserts, updates, and deletes. These events can then trigger other services, such as AWS Lambda, to perform additional actions or real-time processing when data changes in your DynamoDB tables.

DynamoDB Streams make it easy to replicate data, update search indexes, or even trigger business workflows based on data changes, making it a powerful addition to your serverless architecture.

DynamoDB is a highly scalable and fully managed NoSQL database that provides reliable performance for applications of any size. Its flexibility with primary and sort keys allows for efficient data retrieval and organization, while features like DynamoDB Streams enable real-time, event-driven architectures. Whether you are building a mobile app, an IoT platform, or a serverless solution, DynamoDB offers the scalability and performance needed to handle your application’s data demands.

dydb.tf


resource "aws_dynamodb_table" "messages" {
  name         = var.APP_NAME
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "PK" # Partition key (unique identifier for the message)
  # Define table attributes
  attribute {
    name = "PK"
    type = "S"
  }
  tags = {
    Name = var.APP_NAME
  }
}

To maintain consistency across our Terraform configuration, we'll also define some variables we can use in our files. APP_NAME will be used to name our Lambda functions, DynamoDB table, ECS service, etc.

variables.tf


variable "APP_NAME" {
  type    = string
  default = "aws-next-js-crud-app"
  # ^ feel free to change this to whatever you want your 
  # app's name to be, keep in mind that this value will 
  # be used to name Lambda functions, the DynamoDB table 
  # and ECS service
}

variable "REGION" {
  type    = string
  default = "us-east-1"
}

variable "LAMBDA_PATH" {
  type    = string
  default = "lambda"
}

Now we can start to build out our Lambda functions, and as mentioned before, we will be using the Lambda module I personally manage, I use this module to statisfy a myriad of different needs across all my different apps, e.g., setting up Lambda to work as API Gateway integrations with Cognito authentication. Having said that, for this project we won't be using API Gateway or any sort of authentication. These Lambda functions will be exposed publically, hence function_url_public = true in the module instantiation, therefore, it might be a good idea to destroy your resources after you've tried things out.

lambda.tf


module "lambda_get" {
  source              = "git::https://github.com/tldrlw/terraform-modules.git//apig-lambda"
  source_dir          = var.LAMBDA_PATH
  handler_file_prefix = "app-get"
  REST_method         = "GET"
  function_name       = "${var.APP_NAME}-get"
  environment_variables = {
    DYDB_TABLE_NAME = aws_dynamodb_table.messages.name,
    REGION          = var.REGION
  }
  is_s3                  = false
  is_dydb                = true
  dydb_table_arn         = aws_dynamodb_table.messages.arn
  dydb_table_permissions = ["dynamodb:Scan", "dynamodb:DescribeTable"]
  function_url_public    = true
}

module "lambda_post" {
  source              = "git::https://github.com/tldrlw/terraform-modules.git//apig-lambda"
  source_dir          = var.LAMBDA_PATH
  handler_file_prefix = "app-post"
  REST_method         = "POST"
  function_name       = "${var.APP_NAME}-post"
  environment_variables = {
    DYDB_TABLE_NAME = aws_dynamodb_table.messages.name,
    REGION          = var.REGION
  }
  is_s3                  = false
  is_dydb                = true
  dydb_table_arn         = aws_dynamodb_table.messages.arn
  dydb_table_permissions = ["dynamodb:BatchWriteItem"]
  function_url_public    = true
}

module "lambda_delete" {
  source              = "git::https://github.com/tldrlw/terraform-modules.git//apig-lambda"
  source_dir          = var.LAMBDA_PATH
  handler_file_prefix = "app-delete"
  REST_method         = "DELETE"
  function_name       = "${var.APP_NAME}-delete"
  environment_variables = {
    DYDB_TABLE_NAME = aws_dynamodb_table.messages.name,
    REGION          = var.REGION
  }
  is_s3                  = false
  is_dydb                = true
  dydb_table_arn         = aws_dynamodb_table.messages.arn
  dydb_table_permissions = ["dynamodb:DeleteItem"]
  function_url_public    = true
}

module "lambda_put" {
  source              = "git::https://github.com/tldrlw/terraform-modules.git//apig-lambda"
  source_dir          = var.LAMBDA_PATH
  handler_file_prefix = "app-put"
  REST_method         = "PUT"
  function_name       = "${var.APP_NAME}-put"
  environment_variables = {
    DYDB_TABLE_NAME = aws_dynamodb_table.messages.name,
    REGION          = var.REGION
  }
  is_s3                  = false
  is_dydb                = true
  dydb_table_arn         = aws_dynamodb_table.messages.arn
  dydb_table_permissions = [""]
  function_url_public    = true
}

In our Lambda configurations, we're adhering to AWS's principle of least privilege by assigning only the necessary permissions for each function's specific tasks. For instance, lambda_get has read-only permissions (dynamodb:Scan, dynamodb:DescribeTable), while lambda_postis limited to write permissions (dynamodb:BatchWriteItem), ensuring each Lambda can only perform its required operations. This approach avoids over-permissioning and ensures that no unnecessary actions, such as wildcard permissions, are granted. Additionally, by referencing the specific DynamoDB table ARN (aws_dynamodb_table.messages.arn), we further restrict access to the intended table, preventing unauthorized access to other resources.

To enhance security, continue to review and audit these permissions regularly, ensuring that no Lambda has broader access than needed. It's also important to ensure that each function's permissions are explicitly defined and modular, as I've done withdydb_table_permissions in the Lambda module instantiation, ensuring future flexibility. This strategy keeps our functions secure, maintains control over access, and adheres to best practices in AWS IAM management. You can read more about the principle of least privilege here.

After provisioning our Terraform configuration thus far, we can test out these Lambda functions using their published function URL endpoints, so let's set up an outputs.tf file that will log the endpoints in the Terraform apply logs after the Lambda functions are created.

outputs.tf


output "lambda_get_function_url" {
  value = module.lambda_get.function_url
}

output "lambda_get_arn" {
  value = module.lambda_get.arn
}

output "lambda_post_function_url" {
  value = module.lambda_post.function_url
}

output "lambda_post_arn" {
  value = module.lambda_post.arn
}

output "lambda_delete_function_url" {
  value = module.lambda_delete.function_url
}

output "lambda_delete_arn" {
  value = module.lambda_delete.arn
}

At this point, you can run the following Terraform commands from your local to provision all our resources to AWS. Despite still having the front-end infrastructure to set up, we can deploy the Lambda functions and test out their functionality using Postman, check out this guide on installing Postman, and this guide on how to make HTTP requests using endpoints.

  • terraform init
  • terraform plan
  • terraform apply --auto-approve

Assuming your Terraform run went through at this point, the outputs should clearly delineate the endpoints corresponding to your four Lambda functions. Using these, you can test out the functionality of your new back-end APIs, see the screenshots below to understand what request parameters to add for the POST, PUT and DELETE calls you'll be making. While it's not safe for me to reveal my endpoints, I will be destroying these Lambda functions, in order to preclude malicious actors from taking advantage of this vulnerability and inundating my DynamoDB table with items, I suggest you do the same after you've completed this exercise. Running a simple terraform destroy should take care of that.

radiotodaydhaka screenshot
gc-res screenshot
gc-res screenshot
radiotodaydhaka screenshot

When making DELETE or PUT calls, you can run the GET call and look at all the messages you've previously added to the table with the POST command, find the unique identifier for whatever you want to get rid of, or update, and include that as the messageId in the request's JSON body. For PUT calls, you'll also need to include a key-value pair with the key being newMessage. The Lambda functions' source code is configured to only accept these specific keys (messageId and newMessage), so be mindful of that. If you'd like, you can modify the source code to be able to take in other parameters in the request body, but for the purposes of this demonstration, I decided to keep it simple and work with "messages".

In part two of this series, we'll build out the Next.js front-end, where these APIs we just created can be used. It will be a simple form that allows users to see, delete, update existing messages and add new messages.