Creating AWS IoT Rules And Deploy Lambda Functions via CDK

We develop a lot of IoT related things at Ovyl, one of which is web applications for connected devices to talk too. Handling the device messages and maintaining the infrastructure can be tough sometimes, until we learned about AWS’s CDK Tool.

AWS CDK - What Is It?

There have been many attempts at making infrastructure as code some even from AWS like: SAM but the problem with SAM is that you still have to write CloudFormation YAML templates which can be very time consuming and error prone. That’s all going to change with AWS’s new tool: CDK

CDK is AWS’s Cloud Development Kit - Define cloud infrastructure using familiar programming languages. This is powerful because it allows editors that have auto-complete and code lookup to help you implement your infrastructure and catch errors before you even try to deploy it.

I can’t count how many times I’ve had to deploy 20 micro changes because I’ve forgotten an option or misspelled something in my CloudFormation YAML. Now the errors show up right in my editor as I’m working. This allows me to fix the errors and have confidence that the code will deploy.

Under the hood CDK is generating CloudFormation templates from your code. This is good because if you need/want to inspect the YAML templates before deploying you can. It also helps you understand some of the patterns it uses to create your resources.

CDK has 3 concepts:

• App - is an application written in TypeScript, JavaScript, Python, Java, or C# that uses the AWS CDK to define AWS infrastructure.
• Stacks - Stacks (equivalent to AWS CloudFormation stacks) contain constructs. That was taken from their docs, but another way to think about stacks is where you group your resources together. You could put all your resources like API gateway, Lambda functions, and SQL databases in one stack or your could separate them out into different stacks and perform updates to individual stacks.
• Construct - Defines one or more concrete AWS resources, such as Amazon S3 buckets, Lambda functions, Amazon DynamoDB tables, and so on. This is the one that you’ll be using the most

(from AWS)

Constructs come in 3 different levels.

• L1 is essentially like writing a CloudFormation yaml and it’s up to you to complete the rest of the plumbing.
• L2 adds on top of L1 but adds functions and properties making it easier to work with. They still only create 1 AWS resource.
• L3 can create multiple AWS resources and do all the needed plumbing for you like: setting up IAM policies, VPC connections, and security groups.

Now that we know a little bit about what CDK is, let’s use it to create a database and IoT Rule to save the messages in the database.

AWS IoT Rules - What Are They?

AWS IoT Rules allow you to setup simple actions powered by SQL. Some examples of AWS IoT Rules:

• Take any message heard from a device and add additional content to it
• For a given subject or message filter based on content we might need
• Push a message to an AWS SQS Queue, Lambda Function, CloudWatch Logs or Apache Kafka

What’s neat about these services is that we can stream our incoming data with them. We can create pipelines for the rest of our application to hook into or even react to messages in a way that could automate a device on the other end.

Here is an example of a simple Rule:

SELECT *, timestamp() as recievedAt FROM 'device/tempature'

Normally you have to set this up manually using the AWS Console, but who wants to do that? We can use Git to manage the rule “source”, and CDK to deploy it. The benefits of this is that we can track changes to our infrastructure to help us pin point where an error my have been introduced.

Getting Started With CDK

Since CDK supports multiple languages, but we are going to go with Typescript.

Before we get started you’ll need to make sure you have configured the AWS CLI. CDK uses the CLI’s default profile to provision resources. If you have multiple profiles you can specify which ones you’d like to use. I’ll show you that later.

CDK has a project generator to generate a project npx cdk init --language typescript. That was the command I used to create this project.

You can follow along at this repo

The first thing you do on any fresh CDK project is bootstrap it. The bootstrap command will create and run a CloudFormation Stack that will create all the needed IAM policies that you will need to deploy your applications. It will also create a S3 bucket for storing assets for your deployments.

Let’s run that now: npm run cdk bootstrap

tip: if you want to use a different AWS profile for bootstrapping CDK you can prepend the --profile [aws_profile] option to all the CDK commands

After that completes you can look at your AWS console to see the CloudFormation stack and S3 bucket

Setting Up A Database & Rule

Let’s start with adding our Timestream (Time Series) Database and an AWS IoT rule. The IoT rule will forward our MQTT messages to our database.

Todo this we’ll add some code to lib/temp-monitor-stack.ts

import { Stack, StackProps, aws_iam, aws_iot, aws_timestream } from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class TempMonitorStack extends Stack {
 constructor(scope: Construct, id: string, props?: StackProps) {
   super(scope, id, props);

   // This the role used by our iot Rule that allows it to access our Timeseries database and tables
   const iotTimestreamRole = new aws_iam.Role(this, 'IotTempRuleTimestreamRole', {
     assumedBy: new aws_iam.ServicePrincipal('iot.amazonaws.com'),
   })

   // Timeseries database
   const datastore = new aws_timestream.CfnDatabase(this, 'TemperatureDatastore', {
     databaseName: 'TemperatureDatastore',
   })

   // Our temperature table inside of the our Timeseries database.
   // This will hold all the temperature readings from our devices.
   const temperatureDataTable = new aws_timestream.CfnTable(this, 'TemperatureTable', {
     databaseName: datastore.databaseName!,
     tableName: 'TemperatureTable'
   })
   // Since we're dealing with L1 constructs we need to tell cloudformation to
   // wait for the database to be created before making our table in the database.
   temperatureDataTable.addDependsOn(datastore)

   // this were we assign the policies to our roles using the
   // database and tables that were created before this.
   iotTimestreamRole.addToPrincipalPolicy(new aws_iam.PolicyStatement({
     resources:[temperatureDataTable.attrArn],
     actions: [
       'timestream:WriteRecords'
     ]
   }))
   iotTimestreamRole.addToPrincipalPolicy(new aws_iam.PolicyStatement({
     resources:['*'],
     actions: [
       'timestream:DescribeEndpoints',
     ]
   }))

   // This is our IoT Rule that will forward messsages to our
   // timeseries database.
   // It using interpolation to allow us to extract and add data before
   // sending it to our database. You can see that in the dimensions field
   const tempRule = new aws_iot.CfnTopicRule(this, 'TemperatureRule', {
     ruleName: 'forward_readings_to_timeseries',
     topicRulePayload: {
       awsIotSqlVersion: '2016-03-23',
       sql: "SELECT * FROM 'temp/+/reading'",
       actions: [
         {timestream: {
           roleArn: iotTimestreamRole.roleArn,
           databaseName: temperatureDataTable.databaseName,
           tableName: temperatureDataTable.tableName!,
           dimensions: [
             {name: 'temperature', value: '${temperature}'},
             {name: 'deviceId', value: '${topic(2)}'}
           ]
         }}
       ]
     }
   })

 }
}

Now that we’ve got our code changes in place we can see what the CloudFormation template will look like the CDK generates. You view it with the npm run cdk synth command.

Deploying

Let’s deploy the code. Todo that you can run:

npm run cdk deploy

This will show you all the actions/changes that will be applied during the deployment. After you review it, you can accept it and continue with the deployment by responding with y to the prompt.

Let's go look at the IoT Rule it created:

It has the SQL statement and timeseries database action we created. The description is empty. Let's go back and add one. Todo this we'll add the description key to our topicRulePayload.

const tempRule = new aws_iot.CfnTopicRule(this, 'TemperatureRule', {
     ruleName: 'forward_readings_to_timeseries',
     topicRulePayload: {
       description: 'This rule forwards our temperature messages to our timeseries database.',
       awsIotSqlVersion: '2016-03-23',
       sql: "SELECT * FROM 'temp/+/reading'",
       actions: [
         {timestream: {
           roleArn: iotTimestreamRole.roleArn,
           databaseName: temperatureDataTable.databaseName,
           tableName: temperatureDataTable.tableName!,
           dimensions: [
             {name: 'temperature', value: '${temperature}'},
             {name: 'deviceId', value: '${topic(2)}'}
           ]
         }}
       ]
     }
   })

After that let’s run our deploy command again: npm run cdk deploy

You might have noticed something different after running the command. It never prompted us if we want to continue with the changes, it just went ahead and updated our resources. This is because there was no destructive changes, i.e. it didn't need to create, destroy, or recreate a resource.

After the deploy we can refresh the AWS Console and see our description has been added.

We can also look at our Database we created in the console.

Let’s test to make sure our database is getting the messages using the AWS IoT Test console.

Then we’ll head over to the Database and run a query to see the messages.

With about 60 lines of code we were able to create everything we need to handle millions of devices and store millions of records! Even better we can even redeploy these exact changes to another AWS Account for different environments to test out new changes if we need to.

Once you’re done and you don’t want these resources on your account anymore CDK makes it extremely easy to remove it all. All you do is run npm run cdk destroy and everything will be removed from your account. Your bootstrap cloudformation stack will still be there and you have to go to the console to remove that.

Add closing statements

‍