Pre-requisites: A basic idea of AWS DynamoDB and AWS Lambda functions.
In today's fast-paced digital landscape, businesses are constantly searching for ways to gain a competitive edge. At the heart of this pursuit lies the ability to deliver seamless user experiences via real-time data processing, which apparently is an increasingly critical challenge with the rise of cloud computing and scalable, distributed databases.
Let’s say you run an e-commerce platform, and you want to enable a feature such that customers receive instant notifications when products they're interested in become available or when their orders are updated. In such a scenario, you would need to capture and process data changes in real time with minimum latency. This is precisely where Amazon DynamoDB Streams, a powerful and versatile feature of Amazon Web Services (AWS), comes to the rescue. DynamoDB Streams offers a streamlined solution for addressing the challenge of real-time data processing. In this blog post, we will explore what DynamoDB Streams is, why it's essential, and how you can leverage it to enhance the capabilities of your applications.
However, this problem has been existing from a long time, before the concept of DynamoDB Streams were introduced. How were we able to tackle this hurdle?
To answer this, let’s have an overview of the traditional methods. Whereas the conventional relational databases excels at storing and managing structured data, it is not originally designed to handle the demands of real time processing. When we make changes in a row in a relational database, there is no inherent mechanism to notify the other parts of our application about those changes in real time. This leads to a crucial gap in your ability to provide timely updates and reactions to data changes, potentially causing user frustration, missed opportunities, or compliance issues. To address this gap, developers often resort to polling the database at regular intervals to check for changes. This approach is resource-intensive, inefficient, and can lead to unnecessary delays in processing critical data. In today's always-on world, where milliseconds can make a significant difference, such delays are unacceptable.
Enter DynamoDB Streams, a game-changing feature that seamlessly integrates with Amazon DynamoDB, AWS's managed NoSQL database service. With DynamoDB Streams, you can transform your static data into a dynamic force that keeps pace with the ever-evolving needs of your users and your business. Therefore, one of the major advantages of DynamoDB Streams is that it enables the movement of non-critical business logic to an asynchronous process, which offers benefits like improved responsiveness, scalability, reduced blocking, fault tolerance, efficient resource usage, enhanced user experience, task parallelism, support for long-running operations, and more. The approach it follows is capturing real-time changes in DynamoDB tables and triggering AWS Lambda functions to execute that logic, which decouples processing from the main application, scales independently, handles errors, optimises costs, and enhances responsiveness, making it ideal for modern, event-driven architectures.
Now that we have understood why we need DynamoDB streams, let’s delve deeper into the technical aspects.
Imagine you have a DynamoDB table named "Users," which stores user profiles for your application. Whenever a user's profile is updated, it triggers a cascade of actions that need to be executed in real time. Three essential Lambda functions come into play here:
Let's break down this scenario step by step:
time. Every time a record is inserted, updated, or deleted in the "Users" table, a corresponding event is captured by DynamoDB Streams. This event-driven approach ensures that changes to your user data are instantly detected and can trigger subsequent actions.
When changes occur in your DynamoDB table, such as the "Users" table, Amazon DynamoDB Streams captures these changes in real time. The event data provided by DynamoDB Streams contains essential information, including:
1. Type of Events : DynamoDB Streams captures three main types of events:
2.Type of Data: DynamoDB Streams provides different types of data depending on the event type and your configuration:
a. "onUserObjectChange" of AppUserManagementService and WorkplaceUserV2 Service:
This AWS Lambda function is designed to handle user profile changes as they occur. When DynamoDB Streams detects an update in the "Users" table, it triggers both the "onUserObjectChange" Lambda functions.
b. “"upsertWorkplaceUserToElasticSearchV2" of WorkplaceUserV2 Service
The "workplaceUserV2" service includes another Lambda function named "upsertWorkplaceUserToElasticSearchV2." This function gets triggered whenever there's any change in the “WorkplaceUsers_V2" table.
4. Event Filters for Lambda Functions
Now, that we have configured DynamoDB streams, we might find that even the minor updates which have no immediate impact on our application is created. Therefore, we would want to process the events efficiently by avoiding unnecessary lambda invocations in order to keep the costs under control. Hence, “Event Filters” come to our rescue, a logic evaluation that has been pushed onto AWS.
Event filters allow us to filter out events that are irrelevant to our business logic by defining a filter criteria in order to control which events can invoke a lambda function. Suppose if we want to trigger lambda function for modify events in the batch of 10 events, the following changes will be required in our serverless.yml file.
The specific filter conditions you can apply may vary depending on the event source, but here are common types of filters you can use:
However, factors such as your application's architecture, scalability requirements, technology stack, and the specific database system you are using, also needs to be taken under consideration while deciding what approach you are going to move forward with, in order to solve the use case of real time data processing. Now that we have discussed how DynamoDB streams and AWS lambda functions can be used, lets look into some other approaches of handling this issue:
In conclusion, the combination of DynamoDB Streams and AWS Lambda functions provides a dynamic and efficient solution for processing real-time events from a DynamoDB database. It offers real-time responsiveness, cost-effectiveness through serverless computing, simplified architecture, scalability, flexibility in implementing custom logic, easy integration with other AWS services, and robust monitoring capabilities. This combination empowers developers to build event-driven architectures that react swiftly to changes in the database, enhancing application performance and user experiences.
This blog post was originally published here.
Build Apps That Fit Your Business OperationsGet Started - It's free