Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: gateway conformance improvements #85

Conversation

SgtPooki
Copy link
Member

@SgtPooki SgtPooki commented May 22, 2024

Title

fix: gateway conformance improvements

Description

A lot of changes here to fix requests from the gateway-conformance tests. This PR unblocks the ability to start improving
the gateway conformance of @helia/verified-fetch because most requests are now fully received and responded to by our basic-server.

This change depends on a bug fix in @helia/utils, ipfs/helia#542

Fixes ipfs/service-worker-gateway#266
Fixes #84

Summary of changes

  • gateway-conformance changes
    • Implement IPFS_NS_MAP resolution
      • dnslinks are resolved with local dns resolver
      • ipns records are loaded into Helia datastore (this could likely be improved, but TestGatewayIPNSPath is passing at 100% now)
    • removes reverse-proxy.js since it's no longer needed (less moving parts.. closer to verified-fetch)
    • fixes shutdown logic of basic-server and test fixtures.
    • Implements a HACK for gateway-conformance lack of support for subdomains with ports. see fixingGwcAnnoyance in basic-server.js
    • adds request url suffix to logger in basic-server.js to help debug issues with the gateway conformance tests.
    • constructs explicit Helia instance that is passed to verified-fetch so we can override things as necessary
      • remove delegated routing (we don't want remote work)
      • customize datastore so we can load/resolve ipns records
    • changes individual tests we run that are substrings of other individual test names to explicitly skip the longer name (so the tests are more isolated). e.g. TestTrustlessRaw should not run TestTrustlessRawRanges tests.
  • verified-fetch changes
    • Added getRedirectResponse util that is called prior to calling codecHandlers in order to satisfy the gateway conformance tests
      • We will likely need to adjust this logic because this dropped successRate of TestDagPbConversion and others.
    • Re-orders the REGEX order in matchURLString to resolve /ipns/ paths trigger false-positive DNSLink lookup service-worker-gateway#266 and adds a test for it
    • Returns a 404 for raw requests that have a path, since we can't walk paths on raw IPFS content.
      • Fixes test is passed a filename from a deep traversal if it is available that broke when implementing this change to actually construct a folder with nested content.

Notes & open questions

Returns a 404 for raw requests that have a path, since we can't walk paths on raw IPFS content.

Are there any nuances with the fix I added here we need to handle? maybe something with query/search handler.. but i believe that's already being handled elsewhere (e.g. parseUrl/resource)

Change checklist

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation if necessary (this includes comments as well)
  • I have added tests that prove my fix is effective or that my feature works

Copy link
Member Author

@SgtPooki SgtPooki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self review.. will be applying my own suggestions

packages/gateway-conformance/.aegir.js Outdated Show resolved Hide resolved
packages/gateway-conformance/src/conformance.spec.ts Outdated Show resolved Hide resolved
packages/verified-fetch/src/utils/handle-redirects.ts Outdated Show resolved Hide resolved
packages/verified-fetch/src/utils/handle-redirects.ts Outdated Show resolved Hide resolved
packages/verified-fetch/src/utils/handle-redirects.ts Outdated Show resolved Hide resolved
packages/verified-fetch/src/utils/handle-redirects.ts Outdated Show resolved Hide resolved
packages/verified-fetch/src/verified-fetch.ts Outdated Show resolved Hide resolved
@SgtPooki SgtPooki marked this pull request as ready for review May 23, 2024 20:43
@SgtPooki SgtPooki requested a review from a team as a code owner May 23, 2024 20:43
@@ -109,7 +109,7 @@ const tests: TestConfig[] = [
{
name: 'TestGatewayBlock',
run: ['TestGatewayBlock'],
successRate: 37.93
successRate: 20.69
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How come some of these tests have a lower sucessRate threshold?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly not 100% sure but i think its because requests werent making it to the endpoint and we had false negatives.

Since gwc is still being fleshed out & unit tests &interop are passing i think this is acceptable for now.

This should be the last time we have to adjust successRate to satisfy wishy-washy issues.

It would be nice to be able to output gwc failing/successful tests so we could compare more easily than manually parsing the json

} finally {
urlLog.trace('Cleaning up request')
clearTimeout(reqTimeout)
requestController.abort()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you abort here?

Copy link
Member Author

@SgtPooki SgtPooki May 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To kill verified-fetch work that may still be ongoing (session findProvider loop is one I can think of that may still be going)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, this seems a little counter intuitive that there may still be ongoing requests even after the promise is resolved.

Maybe it's something worth documenting with regards to sessions?

return path.endsWith('/') ? path : `${path}/`
}

export async function getRedirectResponse ({ resource, options, logger, cid }: GetRedirectResponse): Promise<null | Response> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about adding a test for this function that illustrate when it redirects?

Suggested change
export async function getRedirectResponse ({ resource, options, logger, cid }: GetRedirectResponse): Promise<null | Response> {
// See https://specs.ipfs.tech/http-gateways/path-gateway/#location-response-header
export async function getRedirectResponse ({ resource, options, logger, cid }: GetRedirectResponse): Promise<null | Response> {

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It really should have a test, but gwc tests it, so i figured it was fine for now. Since we already have gwc tests, redoing all that coverage in unit tests is not ideal.

Ill add a basic test

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@2color 2color left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some comments and suggestions. No major objections from me as this in essence should increase confidence with the tests

Copy link
Member Author

@SgtPooki SgtPooki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some comments

Comment on lines +26 to +34
// TODO: do we need to do anything with CNAME resolution...?
// if (questions.some((q) => q.type === 5)) {
// answers.push({
// name: domain,
// type: 5,
// TTL: 180,
// data: ''
// })
// }
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unsure about this one, but tests seem to work for now. I would like to leave this until i'm sure gwc tests don't depend on it. If we are debugging a test failure and end up here, it may trigger idea for a fix.

return path.endsWith('/') ? path : `${path}/`
}

export async function getRedirectResponse ({ resource, options, logger, cid }: GetRedirectResponse): Promise<null | Response> {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SgtPooki SgtPooki merged commit 7281078 into main May 24, 2024
20 checks passed
@SgtPooki SgtPooki deleted the 84-fix-infinite-querying-bug-reproducible-on-some-gateway-conformance-tests-that-were-disabled-in-httpsgithubcomipfshelia-verified-fetchpull81 branch May 24, 2024 19:09
github-actions bot pushed a commit that referenced this pull request May 24, 2024
Copy link

🎉 This PR is included in version 1.4.3 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

github-actions bot pushed a commit that referenced this pull request May 24, 2024
## @helia/verified-fetch-gateway-conformance [1.1.2](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-gateway-conformance-1.1.1...@helia/verified-fetch-gateway-conformance-1.1.2) (2024-05-24)

### Bug Fixes

* gateway conformance improvements ([#85](#85)) ([7281078](7281078))

### Dependencies

* **@helia/verified-fetch:** upgraded to 1.4.3
Copy link

🎉 This PR is included in version 1.1.2 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

github-actions bot pushed a commit that referenced this pull request May 24, 2024
## @helia/verified-fetch-interop [1.24.1](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-interop-1.24.0...@helia/verified-fetch-interop-1.24.1) (2024-05-24)

### Bug Fixes

* gateway conformance improvements ([#85](#85)) ([7281078](7281078))

### Dependencies

* bump kubo from 0.27.0 to 0.28.0 ([#54](#54)) ([3579844](3579844))

### Dependencies

* **@helia/verified-fetch:** upgraded to 1.4.3
Copy link

🎉 This PR is included in version 1.24.1 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

github-actions bot pushed a commit that referenced this pull request Oct 14, 2024
## 1.0.0 (2024-10-14)

### ⚠ BREAKING CHANGES

* upgrade to helia v5 (#107)

### Features

* abort signals are respected ([#26](#26)) ([30148fe](30148fe))
* allow use as a singleton ([#413](https://github.com/ipfs/helia-verified-fetch/issues/413)) ([7cbeed0](7cbeed0))
* create @helia/verified-fetch ([#392](https://github.com/ipfs/helia-verified-fetch/issues/392)) ([06d36fe](06d36fe))
* customize ipns dnsResolvers ([#445](https://github.com/ipfs/helia-verified-fetch/issues/445)) ([8f60822](8f60822))
* download tars from @helia/verified-fetch ([#442](https://github.com/ipfs/helia-verified-fetch/issues/442)) ([70ddd00](70ddd00))
* require content-type parser to set content-type ([#423](https://github.com/ipfs/helia-verified-fetch/issues/423)) ([b39d07c](b39d07c))
* support `Accept` header in `@helia/verified-fetch` ([#438](https://github.com/ipfs/helia-verified-fetch/issues/438)) ([54c4383](54c4383))
* support downloading car files from @helia/verified-fetch ([#441](https://github.com/ipfs/helia-verified-fetch/issues/441)) ([703980c](703980c))
* support http range header ([#10](#10)) ([9f5078a](9f5078a))
* support IPFS/IPNS paths, Gateways, etc ([#4](#4)) ([e7f1816](e7f1816))
* support redirects for UnixFS directories ([#5](#5)) ([4601d46](4601d46))
* support requesting raw IPNS records in @helia/verified-fetch ([#443](https://github.com/ipfs/helia-verified-fetch/issues/443)) ([e92086a](e92086a))
* upgrade to helia v5 ([#107](#107)) ([91a6473](91a6473))
* use blockstore sessions ([#50](#50)) ([541dd64](541dd64))
* use the waterworks trustless gateway by default ([#94](#94)) ([2db8d1e](2db8d1e))

### Bug Fixes

* @helia/verified-fetch init args are optional ([#412](https://github.com/ipfs/helia-verified-fetch/issues/412)) ([e6d41a0](e6d41a0))
* aborted signals throw libp2p AbortError ([#30](#30)) ([4575791](4575791))
* add interop tests and update project config ([fdc83b8](fdc83b8))
* allow contentTypeParser with Helia instance ([#427](https://github.com/ipfs/helia-verified-fetch/issues/427)) ([b5c95e6](b5c95e6))
* append query path to path resolved from IPNS name ([#3](#3)) ([fd86e6a](fd86e6a))
* byte range request end should never equal file size ([#24](#24)) ([aafc567](aafc567))
* content-type response header hints how to process response ([#426](https://github.com/ipfs/helia-verified-fetch/issues/426)) ([8b78e79](8b78e79))
* decodeURIComponent on path parts ([#40](#40)) ([f628cf6](f628cf6))
* do not coerce `undefined` to `null` for JSON serialization ([#2](#2)) ([d36ce29](d36ce29))
* gateway conformance improvements ([#85](#85)) ([7281078](7281078))
* gateway conformance tests ([#81](#81)) ([d0a3b6b](d0a3b6b))
* identity CIDs use contentTypeParser ([#49](#49)) ([3014498](3014498))
* implicit accept header can be overridden by format query ([#36](#36)) ([75c0b75](75c0b75))
* missing paths returns 404 instead of 502 ([#59](#59)) ([291a054](291a054)), closes [#53](#53)
* pass url and body to badRequestResponse ([#44](#44)) ([cc228e6](cc228e6))
* prevent interop timeouts with fast fixture loading ([#73](#73)) ([a43d994](a43d994))
* reduce dagPb and dagCbor handler complexity ([#45](#45)) ([3b41752](3b41752))
* remove redundant abort controller ([#41](#41)) ([04b220d](04b220d))
* remove stubs from verified-fetch tests ([#431](https://github.com/ipfs/helia-verified-fetch/issues/431)) ([cbab8b0](cbab8b0))
* set cache-control header correctly ([#19](#19)) ([114f3a4](114f3a4)), closes [#17](#17) [#23](#23) [#10](#10)
* support https?://<dnsLink>.ipns.<gateway> urls ([#16](#16)) ([0ece19a](0ece19a))
* ttl and caching for ipns urls ([#34](#34)) ([44ac5a1](44ac5a1))
* unixfs dir redirect ([#33](#33)) ([32ca87f](32ca87f))
* update @helia/ipns and dns config ([#18](#18)) ([9f88c54](9f88c54))
* use CID object for verified-fetch progress events ([#425](https://github.com/ipfs/helia-verified-fetch/issues/425)) ([b010184](b010184))
* use ipfs-unixfs-exporter directly ([#42](#42)) ([4532bf1](4532bf1))
* verified-fetch etag header ([#434](https://github.com/ipfs/helia-verified-fetch/issues/434)) ([f9b1ffe](f9b1ffe))
* verified-fetch header cache-control typo ([#433](https://github.com/ipfs/helia-verified-fetch/issues/433)) ([ea39b48](ea39b48))
* walking dag-cbor paths ([#39](#39)) ([99668ce](99668ce))
* X-Ipfs-Path is set correctly ([#46](#46)) ([5bb6685](5bb6685))

### Trivial Changes

* add tests for directory redirects for gateways ([#15](#15)) ([269609d](269609d))
* fix build ([#22](#22)) ([01261fe](01261fe))
* initial import ([4b4fb8c](4b4fb8c))
* **release:** 1.0.0 [skip ci] ([ef5b6a5](ef5b6a5)), closes [#68](#68)
* **release:** 1.0.0 [skip ci] ([a584a1a](a584a1a))
* **release:** 1.0.0 [skip ci] ([5207a64](5207a64))
* **release:** 1.0.1 [skip ci] ([5499724](5499724)), closes [#81](#81)
* **release:** 1.0.1 [skip ci] ([8d2fdda](8d2fdda)), closes [#2](#2)
* **release:** 1.0.2 [skip ci] ([104af2d](104af2d)), closes [#3](#3)
* **release:** 1.1.0 [skip ci] ([e300af2](e300af2))
* **release:** 1.1.0 [skip ci] ([f4fcb4d](f4fcb4d)), closes [#4](#4) [#5](#5) [#6](#6)
* **release:** 1.1.0 [skip ci] ([b9b959b](b9b959b))
* **release:** 1.1.1 [skip ci] ([8ff0b8b](8ff0b8b)), closes [#83](#83)
* **release:** 1.1.1 [skip ci] ([6f2c6bd](6f2c6bd)), closes [#15](#15) [#14](#14)
* **release:** 1.1.2 [skip ci] ([6161aba](6161aba)), closes [#85](#85)
* **release:** 1.1.2 [skip ci] ([4e1abc0](4e1abc0)), closes [#16](#16)
* **release:** 1.1.3 [skip ci] ([14c2302](14c2302)), closes [#18](#18)
* **release:** 1.10.0 [skip ci] ([4589c26](4589c26))
* **release:** 1.10.1 [skip ci] ([140bab3](140bab3)), closes [#33](#33)
* **release:** 1.11.0 [skip ci] ([39bb14e](39bb14e))
* **release:** 1.12.0 [skip ci] ([50682bf](50682bf))
* **release:** 1.13.0 [skip ci] ([e00d41c](e00d41c))
* **release:** 1.14.0 [skip ci] ([2521838](2521838))
* **release:** 1.15.0 [skip ci] ([a38a0c6](a38a0c6))
* **release:** 1.15.1 [skip ci] ([f59e862](f59e862)), closes [#47](#47)
* **release:** 1.16.0 [skip ci] ([0a18c44](0a18c44))
* **release:** 1.17.0 [skip ci] ([f5299cf](f5299cf))
* **release:** 1.18.0 [skip ci] ([7771af2](7771af2))
* **release:** 1.19.0 [skip ci] ([089635d](089635d))
* **release:** 1.2.0 [skip ci] ([befd270](befd270))
* **release:** 1.2.0 [skip ci] ([e836abf](e836abf)), closes [#10](#10) [#22](#22)
* **release:** 1.2.0 [skip ci] ([77d5e9e](77d5e9e))
* **release:** 1.2.1 [skip ci] ([718644b](718644b)), closes [#24](#24)
* **release:** 1.20.0 [skip ci] ([abaaeab](abaaeab))
* **release:** 1.21.0 [skip ci] ([aeb5a89](aeb5a89))
* **release:** 1.21.1 [skip ci] ([f7ac2e7](f7ac2e7)), closes [#73](#73)
* **release:** 1.22.0 [skip ci] ([6d4b26c](6d4b26c)), closes [#50](#50)
* **release:** 1.23.0 [skip ci] ([754e219](754e219))
* **release:** 1.24.0 [skip ci] ([c1a795f](c1a795f))
* **release:** 1.24.1 [skip ci] ([9f8fb86](9f8fb86)), closes [#85](#85) [#54](#54)
* **release:** 1.25.0 [skip ci] ([3d8bb78](3d8bb78))
* **release:** 1.3.0 [skip ci] ([43c41e4](43c41e4)), closes [#26](#26) [#17](#17) [#23](#23) [#10](#10)
* **release:** 1.3.0 [skip ci] ([bf8af38](bf8af38))
* **release:** 1.3.1 [skip ci] ([0bf5847](0bf5847)), closes [#30](#30)
* **release:** 1.3.10 [skip ci] ([6768fbd](6768fbd)), closes [#51](#51)
* **release:** 1.3.11 [skip ci] ([8606b15](8606b15))
* **release:** 1.3.12 [skip ci] ([969a2e0](969a2e0)), closes [#42](#42)
* **release:** 1.3.13 [skip ci] ([a180547](a180547)), closes [#53](#53) [#46](#46)
* **release:** 1.3.14 [skip ci] ([31f238a](31f238a)), closes [#63](#63)
* **release:** 1.3.2 [skip ci] ([2e141ca](2e141ca)), closes [#33](#33)
* **release:** 1.3.3 [skip ci] ([19574cf](19574cf)), closes [#34](#34)
* **release:** 1.3.4 [skip ci] ([7b06a56](7b06a56)), closes [#40](#40)
* **release:** 1.3.5 [skip ci] ([554bb3e](554bb3e)), closes [#41](#41)
* **release:** 1.3.6 [skip ci] ([489a1d6](489a1d6)), closes [#36](#36)
* **release:** 1.3.7 [skip ci] ([d0114c1](d0114c1)), closes [#39](#39)
* **release:** 1.3.8 [skip ci] ([0a37856](0a37856)), closes [#44](#44) [#47](#47)
* **release:** 1.3.9 [skip ci] ([768e3ff](768e3ff)), closes [#49](#49)
* **release:** 1.4.0 [skip ci] ([dab598a](dab598a)), closes [#50](#50)
* **release:** 1.4.0 [skip ci] ([e008b20](e008b20))
* **release:** 1.4.1 [skip ci] ([e3c0a68](e3c0a68)), closes [#68](#68) [#72](#72) [#72](#72)
* **release:** 1.4.2 [skip ci] ([2633d31](2633d31)), closes [#45](#45)
* **release:** 1.4.3 [skip ci] ([924a6df](924a6df)), closes [#85](#85)
* **release:** 1.5.0 [skip ci] ([4885fde](4885fde)), closes [#94](#94)
* **release:** 1.5.0 [skip ci] ([71d613d](71d613d))
* **release:** 1.5.1 [skip ci] ([4e6775c](4e6775c)), closes [#12](#12)
* **release:** 1.6.0 [skip ci] ([03d0409](03d0409))
* **release:** 1.7.0 [skip ci] ([8bf9c9f](8bf9c9f))
* **release:** 1.8.0 [skip ci] ([a240056](a240056))
* **release:** 1.9.0 [skip ci] ([8715740](8715740))
* start running CI again ([#64](#64)) ([9399f95](9399f95))
* stub blocks interface in tests ([#418](https://github.com/ipfs/helia-verified-fetch/issues/418)) ([a06f64b](a06f64b))
* unskip IPNS test ([#6](#6)) ([76485a4](76485a4))
* Update .github/pull_request_template.md [skip ci] ([5b825e6](5b825e6))
* Update .github/workflows/stale.yml [skip ci] ([3e40ad0](3e40ad0))
* update project config ([#419](https://github.com/ipfs/helia-verified-fetch/issues/419)) ([cb9dc90](cb9dc90))

### Documentation

* add link to blog post and ready-to-run example ([#63](#63)) ([696ed57](696ed57))
* add link to the blog post and example ([#87](#87)) ([f30f979](f30f979))
* change cidv0 to cidv1 in the readme ([#105](#105)) ([060e726](060e726))
* fix readme link ([#51](#51)) ([8a41c57](8a41c57))
* generate readme ([#70](#70)) ([a75567a](a75567a)), closes [#68](#68) [#72](#72) [#72](#72)
* move the package docs to root of repo ([#1](#1)) ([8fb4a76](8fb4a76))
* update api docs link in readme ([#14](#14)) ([d615633](d615633))
* update documented default value ([161a470](161a470)), closes [/github.com/multiformats/js-dns/blob/a56c9e0b953d644392cf10fd0792757da0d61c32/src/resolvers/default.browser.ts#L6-L7](https://github.com/ipfs//github.com/multiformats/js-dns/blob/a56c9e0b953d644392cf10fd0792757da0d61c32/src/resolvers/default.browser.ts/issues/L6-L7)

### Dependencies

* bump aegir from 42.2.11 to 44.1.4 ([#108](#108)) ([e36fbff](e36fbff))
* bump kubo from 0.26.0 to 0.27.0 ([#12](#12)) ([92cad49](92cad49))
* bump kubo from 0.27.0 to 0.28.0 ([#54](#54)) ([3579844](3579844))
* updating all deps ([#47](#47)) ([6d0ffd8](6d0ffd8))

### Tests

* add gateway conformance tests ([#67](#67)) ([30958fb](30958fb)), closes [#68](#68)
* use successRate for gateway conformance tests ([#83](#83)) ([5f71a33](5f71a33))
Copy link

🎉 This PR is included in version 1.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants