Error during dispatch: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

[ferologics]

Describe the bug

Encountered the following error when calling IotDataClient.publish:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: HttpDispatch(HttpDispatchError { message: "Error during dispatch: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer" })', src/main.rs:62:13

Expected Behavior

publishes to mqtt topic

Current Behavior

fails to publish to mqtt topic

Reproduction Steps

1. create a lambda with arm64 custom Amazon Linux 2 runtime (or x86_64, both have the same issue)
2. package attached lambda with cargo lambda build --release --arm64 --output-format zip
3. set up the following IoT rule trigger: SELECT * FROM "rust_lambda_mqtt"
4. send a test event json { "command": "test" } to rust_lambda_mqtt topic to trigger the lambda
5. observe the above reported error

Possible Solution

This looks like some low level TLS error. Custom runtime v2 with rust doesn't seem to be working with the base container. Might just be something with the path of the SSL certs that causes this. The lambda works fine when it's ran locally, the problem starts occurring once it's deployed on Amazon Linux 2 (AL2). Running the lambda on the x86_64 arch in AL2 yields the same error.

Workaround is to use the provided Amazon Linux 1 runtime and compile the lambda with cargo lambda build --release --target x86_64-unknown-linux-gnu.2.17 --output-format zip which produces a working bootstrap.zip.

Implementation

Cargo.toml

[package]
name = "lambda"
version = "0.1.0"
edition = "2021"

# Use cargo-edit(https://github.com/killercup/cargo-edit#installation)
# to manage dependencies.
# Running `cargo add DEPENDENCY_NAME` will
# add the latest version of a dependency to the list,
# and it will keep the alphabetic ordering for you.

[dependencies]
bincode = "1.3.3"
bytes = "1.1.0"
lambda_runtime = "0.5.1"
serde = "1.0.136"
tokio = { version = "1", features = ["macros"] }
tracing = { version = "0.1", features = ["log"] }
tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt"] }
rusoto_core = { version = "0.48.0", default_features = false, features = ["rustls"] }
rusoto_iot_data = { version = "0.48.0", default_features = false, features = ["rustls"] }

main.rs

use lambda_runtime::{run, service_fn, Error, LambdaEvent};
use serde::{Deserialize, Serialize};
use rusoto_core::Region;
use rusoto_iot_data::{IotData, IotDataClient, PublishRequest};
use bytes::Bytes;

/// This is a made-up example. Requests come into the runtime as unicode
/// strings in json format, which can map to any structure that implements `serde::Deserialize`
/// The runtime pays no attention to the contents of the request payload.
#[derive(Deserialize)]
struct Request {
    command: String,
}

/// This is a made-up example of what a response structure may look like.
/// There is no restriction on what it can be. The runtime requires responses
/// to be serialized into json. The runtime pays no attention
/// to the contents of the response payload.
#[derive(Serialize)]
struct Response {
    msg: String
}

/// This is the main body for the function.
/// Write your code inside it.
/// There are some code example in the following URLs:
/// - https://github.com/awslabs/aws-lambda-rust-runtime/tree/main/lambda-runtime/examples
/// - https://github.com/aws-samples/serverless-rust-demo/
async fn function_handler(event: LambdaEvent<Request>) -> Result<Response, Error> {
    // Extract some useful info from the request
    let command = event.payload.command;
    let reversed_cmd = command.chars().rev().collect::<String>();
    
    // Prepare the response
    let resp = Response { msg: reversed_cmd };
    let payload = Some(Bytes::from(bincode::serialize(&resp).unwrap()));
    
    // Publish response to topic
    let iot_client = IotDataClient::new(Region::UsWest2);
    iot_client.publish(
        PublishRequest {
            topic: String::from("rust_lambda_mqtt"),
            payload: payload,
            qos: None,
        }
    ).await.unwrap();
    
    // Return `Response` (it will be serialized to JSON automatically by the runtime)
    Ok(resp)
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::INFO)
        // disabling time is handy because CloudWatch will add the ingestion time.
        .without_time()
        .init();

    run(service_fn(function_handler)).await
}

IAM role policy used

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iot:Publish"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Similar issues found online

• Certificate errors when trying to publish messages using IotDataClient rusoto/rusoto#1862
• IoT dataplane - BadCertificate (UnknownIssuer) #468
• https://stackoverflow.com/questions/62644469/hyper-error-invalid-certificate-unknownissuer

Was told to open an issue here.

Version

the command doesn't work for me

Resources

• the source for rust-lambda.zip
• the deployed arm64 bootstrap.zip and x86_64 bootstrap.zip

Environment details (OS name and version, etc.)

• macOS 12.4 (21F79)
• MacBook Pro (16-inch, 2021)
• Apple M1 Pro
• Custom runtime: Amazon Linux 2 (arm64 & x86_64)
• lambda runtime 0.5.1

Logs

START RequestId: 5af5960f-60de-42c1-98fd-a25316268468 Version: $LATEST
[src/main.rs:75] "tomraff main" = "tomraff main"
DEBUG hyper::client::connect::http: connecting to 127.0.0.1:9001
DEBUG hyper::client::connect::http: connected to 127.0.0.1:9001
DEBUG hyper::proto::h1::io: flushed 109 bytes
DEBUG hyper::proto::h1::io: parsed 7 headers
DEBUG hyper::proto::h1::conn: incoming body is content-length (18 bytes)
DEBUG hyper::proto::h1::conn: incoming body completed
DEBUG hyper::client::pool: pooling idle connection for ("http", 127.0.0.1:9001)
ERROR lambda: tomraff first
[src/main.rs:52] "creating ito client" = "creating ito client"
DEBUG hyper::client::connect::dns: resolving host="data.iot.us-west-2.amazonaws.com"
DEBUG hyper::client::connect::http: connecting to 52.35.93.134:443
DEBUG hyper::client::connect::http: connected to 52.35.93.134:443
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: HttpDispatch(HttpDispatchError { message: "Error during dispatch: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer" })', src/main.rs:62:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
END RequestId: 5af5960f-60de-42c1-98fd-a25316268468
REPORT RequestId: 5af5960f-60de-42c1-98fd-a25316268468	Duration: 882.35 ms	Billed Duration: 947 ms	Memory Size: 128 MB	Max Memory Used: 26 MB	Init Duration: 64.35 ms	
RequestId: 5af5960f-60de-42c1-98fd-a25316268468 Error: Runtime exited with error: exit status 101
Runtime.ExitError

[ferologics]

Another error log worth mentioning is when we removed use of rustls in Cargo.toml, thereby relying on openssl:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: HttpDispatch(HttpDispatchError { message: "Error during dispatch: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1921: (unable to get local issuer certificate)" })', src/main.rs:53:13

[Velfi]

Hey @ferologics, thanks for submitting this. It sounds like the SDK could be resolving and incorrect endpoint. We'll take a look at this as soon as we're able to. In the meantime, you could try using native-tls instead of RusTLS or, if you can find anothe cert, manually insert it.

To enable native-tls, you can activate the feature in your Cargo.toml:

[dependencies]
aws-sdk-iotdataplane = { version = "0.15.0", default_features = false, features = ["rt-tokio", "native-tls"] }

That might end up failing with the same issue but it's worth a shot.

[ferologics]

@Velfi i tried enabling native-tls according to your suggestion but now I get:

Error: Couldn't find valid bootstrap(s): [/var/task/bootstrap /opt/bootstrap]
Runtime.InvalidEntrypoint

This is with Amazon Linux 2 custom runtime using arm64.
bootstrap.zip

[jdisanti]

The code in your implementation looks like its using Rusoto instead of aws-sdk-iotdataplane. Are you sure this is an aws-sdk-rust issue? Maybe I'm misunderstanding.

At a higher level, I remember there being an issue with the IoT DataPlane endpoint having a bad certificate, but I believe this was resolved by updating the endpoint. If you switch to the latest aws-sdk-rust, this may just work correctly.

[ferologics]

@jdisanti it was suggested that this is an aws sdk issue by @calavera here, so I supposed that rusoto may be using the sdk under the hood, somehow.

[jdisanti]

OK. To my knowledge, Rusoto doesn't use aws-sdk-rust at all, so this is a Rusoto issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[ferologics]

I'm not sure if it's a rusoto issue because 1) it works with a local debug setup, and 2) it works with Amazon Linux 1 runtime. Something about Amazon Linux 2 is misconfigured. Doesn't that suggest it's the aws rust runtime issue @jdisanti @Velfi?