Using the AWS SDK for Rust with SQS and a Go Lambda Function to Update DynamoDB Tables

I am learning Rust in my spare time, and I am curious to see how I can use the AWS SDK for Rust in a solution workflow that includes one of my Go Lambda functions. Like Bill Murray, you may be thinking...

Why Rust?

I first heard of Rust a few years ago, and its popularity is growing quickly in the industry. For example, AWS selected Rust for the firecracker project. AWS also announced an alpha release of the AWS SDK for Rust recently. Last, but not least, the Rust Foundation counts AWS, Google, Mozilla, Microsoft, and other industry leaders among its supporters.

The Solution Architecture

I wrote a simple Rust program to be an SQS Producer that uses the AWS SDK to write data to SQS. The SQS queue triggers a Lambda function written in Go that reads the data from the queue and writes it to a DynamoDB table. This is a familiar pattern for incorporating fault tolerance to your system where user input collection is decoupled from user input processing.

Without a queue, if there is a problem with the data processing server, the user input is lost. With a highly-available queue service like SQS, the user input is persisted until the processing server completes its work. If there is a problem with the processing server, a replacement server can retry the work. I chose Lambda functions to be my SQS Consumers because they provide a cost-effective option for provisioning compute resources only when there is data in the queue to be processed (i.e. event-driven compute). Here is a solution diagram:

The Rust SQS Producer Program

There are a couple things to be aware of with the SDK. First, as of the date of this article's publication, your AWS credentials need to be specified as environment variables wherever you are running your Rust application. FYI, the SDK team is working on a solution to have the Rust SDK work similar to other AWS SDK's where credentials can be retrieved from the file system. Second, the SDK is an alpha release, and implementation details are subject to change at any time. My program runs successfully with v0.0.12-alpha of the SDK.

If you need a refresher on how to begin writing a Rust program, I recommend you review how to get started and also use any of the learning resources listed here.

Now, I'll highlight some important items from the sample program.

In my program's Cargo.toml file, I add the SQS package from the AWS SDK for Rust. This allows my program to use the package:

# Cargo.toml
[dependencies]
sqs = { git = "https://github.com/awslabs/aws-sdk-rust", tag = "v0.0.12-alpha", package = "aws-sdk-sqs"}

If you are new to Rust, Cargo.toml contains metadata about your program (ex: author, version number) and lists the packages your program depends on. It is automatically generated for you when you run the cargo init command before you write your code.

I tell my program to use one of the standard libraries for including environment variables since I am also setting my AWS account ID as an environment variable instead of hardcoding it in the source:

 use std::env;

I initialize the SQS client, set the name of my SQS queue, and I retrieve my AWS account ID from my environment variable:

    let client = sqs::Client::from_env();
    let myqueue = "test_queue";
    let account = match env::var("AWS_ACCOUNT"){
       Ok(account)  => account,
       Err(e) => {
           eprintln!("You did not set the AWS_ACCOUNT environment variable. Please do so and try again. {}", e);
            exit(1);
       }
    };

The match statement is convenient for dealing with Result return types that either contain an error or the data you expect to receive. In the case of an Ok() variant, I set account to the environment variable. Otherwise, I inform the user to set the environment variable and abort the program.

This next line concatenates multiple string values to form the SQS queue's URL, which consists of the SQS endpoint for us-east-1 (Northern Virginia), my AWS account ID, and my SQS queue name.

let queue_url = format!("{}{}{}{}","https://sqs.us-east-1.amazonaws.com/",account,"/",myqueue);

There are multiple idiomatic approaches to concatenating strings in Rust, and they are all different compared to how this is accomplished in other popular languages. For example, in other languages, I am accustomed to using the + operator to concatenate strings, and I initially attempted that in my Rust program...

let queue_url = "https://sqs.us-east-1.amazonaws.com/"+account+"/"+myqueue;

There are two challenges to this approach:

1. Rust has multiple types for handling string values: String and &str

2. Rust has a concept of ownership for safe memory management.

To properly use the + operator, I would have to update the above line as follows...

let queue_url = "https://sqs.us-east-1.amazonaws.com/".to_owned()+&account+"/"+myqueue;

This syntax is less intuitive than the idiomatic syntax I shared earlier: the first value must be a String type, and the to_owned() method is converting my url prefix from &str to String accordingly. The values that follow the + operators must be &str type. So, I would need to use the & symbol to borrow my account variable's value, which is a String type due to the env::var("AWS_ACCOUNT") function call. The "/" value and myqueue values are already of the &str type and can follow the + operator without any issues.

TL;DR Use an idomatic method for String concatenation in Rust, and pay attention to whether you need to use a String type or a &str type for your expressions.

Finally, I use the SQS client to send messages to the queue. Rust allows you to chain functions, and I use this appraoch in the code. I use Rust's support for asynchronus processing in the tokio runtime so that my SQS API requests are non-blocking.

    for word in words.iter() {
        let rsp = client
            .send_message()
            .queue_url(&queue_url)
            .message_body(word.to_string())
            .send()
            .await?;
        println!("Response from sending a message: {:#?}", rsp);
    }
    Ok(())

Where Can I Get the Sample Code?

I covered how to use Go Lambda functions before in a previous article if you want a refresher. The source code for my SQS consumer Lambda function is on GitHub.

The sample Rust App's source code is shared below, and you can also find it on GitHub.

# Rust Sample App
 use std::process::exit;
 use std::env;

 /// Sends a message to and receives the message from a queue.
 #[tokio::main]
 async fn main() -> Result<(), sqs::Error> {
    tracing_subscriber::fmt::init();
    let client = sqs::Client::from_env();
    let myqueue = "test_queue";
    let account = match env::var("AWS_ACCOUNT"){
       Ok(account)  => account,
       Err(e) => {
           eprintln!("You did not set the AWS_ACCOUNT environment variable. Please do so and try again. {}", e);
            exit(1);
       }
    };

		let queue_url = format!("{}{}{}{}","https://sqs.us-east-1.amazonaws.com/",account,"/",myqueue);
 
    println!(
        "Sending messages to SQS Queue: `{}` in account `{:#?}`",
        queue_url,
        account
    );
    let words = ["cat","dog","horse","pig", "Mercury","Gemini","Apollo","Skylab","Skylab B","ISS"];
    for word in words.iter() {
        let rsp = client
            .send_message()
            .queue_url(&queue_url)
            .message_body(word.to_string())
            .send()
            .await?;
        println!("Response from sending a message: {:#?}", rsp);
    }
    Ok(())
 }

References

• AWS SDK for Rust
• Tokio asynchronous runtime for Rust
• Sample code that I adapted for my program

If you found this article useful, let me know on Twitter!