Add a "How-To" example for performing HTTP requests by jvff · Pull Request #3509 · linera-io/linera-protocol

jvff · 2025-03-08T16:00:07Z

Motivation

Performing HTTP requests from applications is an important feature for Linera. There are a few different ways for applications to perform them, and there are tradeoffs between these ways. Therefore, this feature should be documented and a "How-To" example demonstrating how to use and test the feature would be useful.

Proposal

Add an example that performs HTTP requests in all different ways, and test them.

Test Plan

Unit and integration tests were added to test not just the application, but also to serve as testing examples and to also test the HTTP allow-list.

Release Plan

Nothing to do, because this is just a new example being added to the repository.

Links

reviewer checklist

## Motivation For security reasons, we would like to initially limit the hosts that applications can send HTTP requests to. ## Proposal Add an allow-list of hosts to the `ResourceControlPolicy` configured by a committee. Only perform HTTP requests in the execution state actor if the target host is in the list. ## Test Plan Tests that exercise the allow list were added in PR #3509. ## Release Plan - Nothing to do / These changes follow the usual release cycle, because this is a new feature that should be included in the next release and testnet. ## Links - [reviewer checklist](https://github.com/linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)

deuszx · 2025-03-10T09:04:06Z

linera-base/src/crypto/secp256k1.rs

 impl Secp256k1PublicKey {
    /// A fake public key used for testing.
-    #[cfg(with_testing)]
+    #[cfg(all(with_testing, not(target_arch = "wasm32")))]


Did you encounter problems with compilation here?

Yes, I believe when it tried to compile the tests for wasm32-unknown-unknown.

deuszx · 2025-03-10T09:06:39Z

examples/how-to/perform-http-requests/tests/http_requests.rs

+
+/// Tests if service query can't perform HTTP requests to hosts that aren't allowed.
+#[test_log::test(tokio::test)]
+#[should_panic(expected = "UnauthorizedHttpRequest")]


Does it expect a "Rust panic" in the service part? How will that behave on the validator node (client)? Will it propagate the panic and shut down the node?

Not exactly. This expects a panic from the ActiveChain used as the test helper:

linera-protocol/linera-sdk/src/test/chain.rs

Line 217 in 524c61a

.expect("Failed to execute block.")

To help with writing tests, the types in linera_sdk::test usually don't return Result and panic in case of an error, because that's usually what the test does anyway. There are exceptions with try_* methods, and I think the cases in this example would make more sense to assert the error type instead.

deuszx · 2025-03-10T09:07:32Z

examples/how-to/perform-http-requests/tests/http_requests.rs

+    chain
+        .graphql_mutation(application_id, "mutation { performHttpRequest }")
+        .await;
+
+    Ok(())


It looks like it's strictly less test than service_query_performs_http_request?

Not exactly, because this uses mutation { performHttpRequest } while the other test uses query { performHttpRequest }, so the entrypoints are completely different. Maybe the names should be different to avoid confusion. Either having the mutation something like performHttpRequestAndSendToContract (but shorter 😅) or having the query be performLocalHttpQuery.

deuszx · 2025-03-10T09:14:53Z

examples/how-to/perform-http-requests/tests/http_requests.rs

+        .await;
+
+    chain
+        .graphql_mutation(application_id, "mutation { performHttpRequest }")


It'd be nice to be able to really see that the block proposal failed for exactly the expected reason here.

I agree, I'll see if I can add a try_graphql_mutation method that returns a Result instead.

deuszx

A more general question:

Why do we need all these ways to perform an HTTP query? If only the service was allowed to make queries then it could be used both on~ and off~ chain:

A client node calls the service only and consumes the results to construct a block
A contract calls into its service and consumes the result (acting as an oracle)

examples/how-to/perform-http-requests/README.md

ma2bd

see comment

Also the example is not run in CI, is it?

…3540) ## Motivation Epoch changes were recently changed to use events (#3432), but the `TestValidator::change_resource_control_policy` was still trying to receive epoch change system messages instead of creating an operation to process the new epoch. ## Proposal Use the new `SystemOperation::ProcessNewEpoch` to update the chains to use the new epoch, and skip updating the admin chain because it fails to try to process the same epoch twice. ## Test Plan Tested with a rebased version of the tests in #3509. The tests stop working after #3432 but pass again after this PR. ## Release Plan - These changes follow the usual release cycle, because it fixes an issue with SDK that is not present in any released version. ## Links - [reviewer checklist](https://github.com/linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)

## Motivation HTTP requests can take a long time to complete, or attempt to stay open indefinitely. This should not be allowed because it may stall a chain. ## Proposal Add a `http_request_timeout` policy to `ResourceControlPolicy`, and configure all HTTP requests to use that timeout. ## Test Plan Wrote some [tests](https://github.com/jvff/linera-protocol-archive/compare/95da70cabb..test-http-timeouts-p1) out of this branch, possibly to be added to #3509. ## Release Plan - Backporting is not possible but we may want to deploy a new `devnet` and `testnet` and release a new SDK soon, because this contains backwards incompatible changes to the chain state. ## Links - Closes #3522 - [reviewer checklist](https://github.com/linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)

## Motivation Applications could cause excessive bandwidth costs to validators by performing HTTP requests to obtain large amounts of data as fast as possible. ## Proposal Ensure that HTTP responses are limited in size to a value agreed among the validators. ## Test Plan Wrote some unit tests in a separate branch, using the code from #3509: https://github.com/jvff/linera-protocol-archive/blob/test-http-response-limits-p1/examples/how-to/perform-http-requests/tests/response_size_tests.rs ## Release Plan - Backporting is not possible but we may want to deploy a new `devnet` and release a new SDK soon, because this contains changes to the consensus critical `ResourceControlPolicy`. ## Links - Closes #3521 - [reviewer checklist](https://github.com/linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)

The contract should successfully execute an operation with a valid off-chain HTTP response.

Ensure that the contract panics if the off-chain HTTP response can't be trusted.

The received response is sent to the contract to validate.

Ensure that the service schedules the operation with a received HTTP response.

Ensure that it forwards the received HTTP response to the contract to verify.

Ensure that if the service reaches an untrusted HTTP server, the contract rejects the HTTP response.

Demonstrate how an HTTP request is made in the contract.

Add a unit test showing how to mock the response.

Ensure that the response is being verified.

Allow the service to request the contract to perform an HTTP request.

Ensure that the mutation schedules the operation for the contract.

And that it is accepted if the response is valid.

It should reject a response that it does not trust.

Request the service to perform an HTTP request for the contract, returning only the relevant deterministic output.

Demonstrate how to mock the expected service query.

A reminder that the oracle's HTTP response also needs to be verified.

Provide a mutation to create an operation that uses the service as an oracle.

Ensure that it produces the operation for the contract to use the service as an oracle.

Ensure that the service performs an HTTP request when it is called as an oracle.

A reminder that the oracle's HTTP response also needs to be verified.

Describe the different ways HTTP requests can be made, and compare their usages.

Run a very simple HTTP server for executing the README example.

Ensure that the README instructions are up-to-date.

## Motivation The integration tests for the HTTP Requests How-To example (#3509) used the `#[should_panic]` attribute for testing failures. However, as [pointed out during review](#3509 (comment)), these are quite broad and may lead to mistakes. Being able to assert the exact error leads to better tests and a better example for external developers. ## Proposal Add `try_query`, `try_graphql_query` and `try_graphql_mutation` helper methods to the `ActiveChain` test interface. Use the new methods to replace usages of `#[should_panic]` with error type assertions. To reduce boilerplate code, a `WorkerError::expect_execution_error` test helper method was added, that dives into the internal `ExecutionError` stored inside a `WorkerError`. ## Test Plan CI should catch any regressions caused by this refactor to the tests. ## Release Plan - These changes follow the usual release cycle, because this only includes small backward-compatible API additions to the public API and SDK. ## Links - [reviewer checklist](https://github.com/linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)

jvff mentioned this pull request Mar 8, 2025

Only allow HTTP requests authorized by the committee #3427

Merged

jvff force-pushed the how-to-perform-http-requests-example branch from 2d7b1b5 to f6778af Compare March 9, 2025 03:52

jvff self-assigned this Mar 9, 2025

jvff added the enhancement New feature or request label Mar 9, 2025

jvff added this to Web3 SDK Mar 9, 2025

jvff added this to the Testnet #2 milestone Mar 9, 2025

jvff force-pushed the how-to-perform-http-requests-example branch 2 times, most recently from 971dcb8 to ce3adba Compare March 9, 2025 04:26

jvff requested review from MathieuDutSik, Twey, afck, deuszx and ma2bd and removed request for MathieuDutSik March 9, 2025 04:59

jvff marked this pull request as ready for review March 9, 2025 04:59

deuszx reviewed Mar 10, 2025

View reviewed changes

ma2bd reviewed Mar 10, 2025

View reviewed changes

examples/how-to/perform-http-requests/README.md Outdated Show resolved Hide resolved

ma2bd requested changes Mar 10, 2025

View reviewed changes

This was referenced Mar 11, 2025

Fix epoch update in TestValidator::change_resource_control_policy #3540

Merged

Add timeouts for HTTP requests #3541

Merged

jvff mentioned this pull request Mar 12, 2025

Limit HTTP response size #3553

Merged

jvff mentioned this pull request Mar 19, 2025

Limit the size of oracle responses #3597

Merged

jvff added 20 commits March 28, 2025 03:11

Test if contract accepts valid off-chain response

0e17a65

The contract should successfully execute an operation with a valid off-chain HTTP response.

Test if contract rejects an invalid HTTP response

9928d05

Ensure that the contract panics if the off-chain HTTP response can't be trusted.

Add service mutation that performs a HTTP request

18dd81b

The received response is sent to the contract to validate.

Test if service schedules HandleHttpResponse

cc55c16

Ensure that the service schedules the operation with a received HTTP response.

Test if mutation performs HTTP request

4bf43eb

Ensure that it forwards the received HTTP response to the contract to verify.

Test if contract rejects invalid HTTP response

0d9af9a

Ensure that if the service reaches an untrusted HTTP server, the contract rejects the HTTP response.

Add an operation to perform an HTTP request

7e4fb6c

Demonstrate how an HTTP request is made in the contract.

Test if contract performs HTTP request

04c681d

Add a unit test showing how to mock the response.

Test if contract rejects invalid HTTP response

3106470

Ensure that the response is being verified.

Add a mutation to perform HTTP request in contract

f3505cd

Allow the service to request the contract to perform an HTTP request.

Test if service schedules PerformHttpRequest op.

454a6c4

Ensure that the mutation schedules the operation for the contract.

Test if contract performs HTTP request

833f779

And that it is accepted if the response is valid.

Test if contract validates the response it gets

4ae4a86

It should reject a response that it does not trust.

Add an operation to use the service as an oracle

d023e4e

Request the service to perform an HTTP request for the contract, returning only the relevant deterministic output.

Test if contract can use service as an oracle

7f42e7d

Demonstrate how to mock the expected service query.

Test if contract rejects invalid oracle response

4334ab1

A reminder that the oracle's HTTP response also needs to be verified.

Add a mutation for the service as oracle example

4d85a60

Provide a mutation to create an operation that uses the service as an oracle.

Test performHttpRequestAsOracle mutation

4f5077f

Ensure that it produces the operation for the contract to use the service as an oracle.

Test service used as an oracle

bf94567

Ensure that the service performs an HTTP request when it is called as an oracle.

Test if contract rejects invalid oracle response

8806eda

A reminder that the oracle's HTTP response also needs to be verified.

jvff force-pushed the how-to-perform-http-requests-example branch from 04bba32 to 681f405 Compare March 28, 2025 03:11

jvff added 3 commits March 28, 2025 03:15

Add a README file for the HTTP request how-to

efa1c49

Describe the different ways HTTP requests can be made, and compare their usages.

Add a test HTTP server binary for README test

4937938

Run a very simple HTTP server for executing the README example.

Test README instructions in CI

43c5a08

Ensure that the README instructions are up-to-date.

jvff force-pushed the how-to-perform-http-requests-example branch from 681f405 to 43c5a08 Compare March 28, 2025 03:15

jvff merged commit 0a17a70 into linera-io:main Mar 28, 2025
23 checks passed

github-project-automation bot moved this to Done in Web3 SDK Mar 28, 2025

jvff deleted the how-to-perform-http-requests-example branch March 28, 2025 14:56

jvff mentioned this pull request Apr 2, 2025

Replace #[should_panic] with assertions in HTTP example tests #3705

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a "How-To" example for performing HTTP requests#3509

Add a "How-To" example for performing HTTP requests#3509
jvff merged 32 commits intolinera-io:mainfrom
jvff:how-to-perform-http-requests-example

jvff commented Mar 8, 2025 •

edited

Loading

Uh oh!

deuszx Mar 10, 2025

Uh oh!

jvff Mar 15, 2025

Uh oh!

deuszx Mar 10, 2025

Uh oh!

jvff Mar 15, 2025

Uh oh!

deuszx Mar 10, 2025

Uh oh!

jvff Mar 15, 2025

Uh oh!

deuszx Mar 10, 2025

Uh oh!

jvff Mar 15, 2025

Uh oh!

deuszx left a comment •

edited

Loading

Uh oh!

Uh oh!

ma2bd left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jvff commented Mar 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Proposal

Test Plan

Release Plan

Links

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deuszx left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ma2bd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jvff commented Mar 8, 2025 •

edited

Loading

deuszx left a comment •

edited

Loading