Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

access-api: invoking store/list in production errors with HandlerExecutionError + HTTPError #363

Closed
gobengo opened this issue Jan 18, 2023 · 5 comments
Assignees

Comments

@gobengo
Copy link
Contributor

gobengo commented Jan 18, 2023

Motivation:

Context:

Problem

  • Error message seen in github actions run
    # testing aud=did:web:web3.storage url=https://up.web3.storage/
    # testing aud=did:web:web3.storage url=https://access.web3.storage/
    # unexpected result from store/list invocation aud=did:web:web3.storage url=https://access.web3.storage/ {
    #   name: 'HandlerExecutionError',
    #   cause: {
    #     url: 'https://up.web3.storage/',
    #     name: 'HTTPError',
    #     stack: 'HTTPError: HTTP Request failed\\n' +
    #       '    at lN.throw (d1-beta-facade.entry.js:14585:11)\\n' +
    #       '    at dN.request (d1-beta-facade.entry.js:14579:135)\\n' +
    #       '    at async WN (d1-beta-facade.entry.js:15030:45)\\n' +
    #       '    at async Object. (d1-beta-facade.entry.js:15037:132)\\n' +
    #       '    at async IB (d1-beta-facade.entry.js:14077:12)\\n' +
    #       '    at async AB (d1-beta-facade.entry.js:14065:12)\\n' +
    #       '    at async TB (d1-beta-facade.entry.js:14059:42)\\n' +
    #       '    at async PO (d1-beta-facade.entry.js:15097:11)\\n' +
    #       '    at async ed.fetch (d1-beta-facade.entry.js:9434:17)\\n' +
    #       '    at async Object.fetch (d1-beta-facade.entry.js:20183:13)',
    #     status: 530,
    #     message: 'HTTP Request failed',
    #     statusText: ''
    #   },
    #   error: true,
    #   stack: 'HandlerExecutionError: service handler {can: "store/list"} error: HTTP Request failed\\n' +
    #     '    at IB (d1-beta-facade.entry.js:14079:13)\\n' +
    #     '    at async AB (d1-beta-facade.entry.js:14065:12)\\n' +
    #     '    at async TB (d1-beta-facade.entry.js:14059:42)\\n' +
    #     '    at async PO (d1-beta-facade.entry.js:15097:11)\\n' +
    #     '    at async ed.fetch (d1-beta-facade.entry.js:9434:17)\\n' +
    #     '    at async Object.fetch (d1-beta-facade.entry.js:20183:13)',
    #   message: 'service handler {can: "store/list"} error: HTTP Request failed',
    #   capability: {
    #     can: 'store/list',
    #     with: 'did:key:z6MknC4PQP4MSBxgnDdotXtgTCAXtrGker1BM4R5LfG5u3Pa'
    #   }
    # }
    
  • The same doesn't happen when I run access-api locally configured with env.{DID,PRIVATE_KEY} same as production. This leads me to think it's something specific about our production deployment running in cloudflare workers
    TAP version 13
    # Subtest: /Users/bengo/bengo.is/monorepo/experiments/w3protocol-test/dist/src/w3protocol.test.js
        # testing aud=did:web:staging.web3.storage url=https://staging.up.web3.storage/
        # testing aud=did:web:staging.web3.storage url=https://w3access-staging.protocol-labs.workers.dev/
        # testing aud=did:web:web3.storage url=https://up.web3.storage/
        # testing aud=did:web:web3.storage url=http://localhost:8787/
        # Subtest: can list items in a space
        ok 1 - can list items in a space
          ---
          duration_ms: 5206.01275
          ...
        # Subtest: w3protocol-test can upload file
        ok 2 - w3protocol-test can upload file # SKIP
          ---
          duration_ms: 0.061333
          ...
        1..2
    ok 1 - /Users/bengo/bengo.is/monorepo/experiments/w3protocol-test/dist/src/w3protocol.test.js
      ---
      duration_ms: 5373.008917
      ...
    1..1
    # tests 1
    # pass 1
    # fail 0
    

Possible Solutions

@gobengo
Copy link
Contributor Author

gobengo commented Jan 18, 2023

The status: 530 in the HTTPError is pretty unusual. It could be cloudflare https://community.cloudflare.com/t/community-tip-fixing-error-530-error-1016-origin-dns-error/44264

@gobengo
Copy link
Contributor Author

gobengo commented Jan 19, 2023

gobengo added a commit that referenced this issue Jan 19, 2023
…y non-ok responses (to debug) (#364)

…that logs unexpected responses with status=530

Motivation:
* gather more info for [this error happening in
production](#363 (comment))

Plan
* gather info, then revert this PR
@gobengo gobengo self-assigned this Jan 19, 2023
gobengo added a commit that referenced this issue Jan 19, 2023
…or, by default catches HTTPError -> error result w/ status=502 (#366)

, which defaults to something that catches HTTPErrors and rewrites to a
status=502 error result

Motivation:
* relates to: #363
* instead of that behavior of getting a `HandlerExecutionError` (due to
internally having an uncaught `HTTPError`), after this PR we should
* not have the uncaught `HTTPError`, so no uncaught
`HandlerExecutionError`
* the result will have `status=502` and `x-proxy-error` with information
about the proxy invocation request that errored (which will help debug
#363)

Limitations
* This inlcudes in `result['x-proxy-error']` any info from the
underlying `@ucanto/transport/http` `HTTPError`, but unless/until we put
more info on that error, we still won't have the raw response object and
e.g. won't have response headers.
* but if those properties are ever added to `HTTPError`, they should
show up in `x-proxy-error`
@gobengo
Copy link
Contributor Author

gobengo commented Jan 19, 2023

phew I recovered response text from the mysterious 530 response:

<!DOCTYPE html>
<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->
<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->
<!--[if IE 8]>    <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->
<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->
<head>
<title>Origin DNS error | up.web3.storage | Cloudflare</title>
<meta charset=\"UTF-8\" />
<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />
<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />
<meta name=\"robots\" content=\"noindex, nofollow\" />
<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />
<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/main.css\" />


<script>
(function(){if(document.addEventListener&&window.XMLHttpRequest&&JSON&&JSON.stringify){var e=function(a){var c=document.getElementById(\"error-feedback-survey\"),d=document.getElementById(\"error-feedback-success\"),b=new XMLHttpRequest;a={event:\"feedback clicked\",properties:{errorCode:1016,helpful:a,version:1}};b.open(\"POST\",\"https://sparrow.cloudflare.com/api/v1/event\");b.setRequestHeader(\"Content-Type\",\"application/json\");b.setRequestHeader(\"Sparrow-Source-Key\",\"c771f0e4b54944bebf4261d44bd79a1e\");
b.send(JSON.stringify(a));c.classList.add(\"feedback-hidden\");d.classList.remove(\"feedback-hidden\")};document.addEventListener(\"DOMContentLoaded\",function(){var a=document.getElementById(\"error-feedback\"),c=document.getElementById(\"feedback-button-yes\"),d=document.getElementById(\"feedback-button-no\");\"classList\"in a&&(a.classList.remove(\"feedback-hidden\"),c.addEventListener(\"click\",function(){e(!0)}),d.addEventListener(\"click\",function(){e(!1)}))})}})();
</script>

<script defer src=\"https://performance.radar.cloudflare.com/beacon.js\"></script>
</head>
<body>
  <div id=\"cf-wrapper\">
    <div class=\"cf-alert cf-alert-error cf-cookie-error hidden\" id=\"cookie-alert\" data-translate=\"enable_cookies\">Please enable cookies.</div>
    <div id=\"cf-error-details\" class=\"p-0\">
      <header class=\"mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-15 antialiased\">
         <h1 class=\"inline-block md:block mr-2 md:mb-2 font-light text-60 md:text-3xl text-black-dark leading-tight\">
           <span data-translate=\"error\">Error</span>
           <span>1016</span>
         </h1>
         <span class=\"inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed\">Ray ID: 78c2234cf35dfab6 &bull;</span>
         <span class=\"inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed\">2023-01-19 19:57:10 UTC</span>
        <h2 class=\"text-gray-600 leading-1.3 text-3xl lg:text-2xl font-light\">Origin DNS error</h2>
      </header>

      <section class=\"w-240 lg:w-full mx-auto mb-8 lg:px-8\">
          <div id=\"what-happened-section\" class=\"w-1/2 md:w-full\">
            <h2 class=\"text-3xl leading-tight font-normal mb-4 text-black-dark antialiased\" data-translate=\"what_happened\">What happened?</h2>
            <p>You've requested a page on a website (up.web3.storage) that is on the <a href=\"https://www.cloudflare.com/5xx-error-landing/\" target=\"_blank\">Cloudflare</a> network. Cloudflare is currently unable to resolve your requested domain (up.web3.storage).
            
          </div>

          
          <div id=\"resolution-copy-section\" class=\"w-1/2 mt-6 text-15 leading-normal\">
            <h2 class=\"text-3xl leading-tight font-normal mb-4 text-black-dark antialiased\" data-translate=\"what_can_i_do\">What can I do?</h2>
            <p><strong>If you are a visitor of this website:</strong><br />Please try again in a few minutes.</p><p><strong>If you are the owner of this website:</strong><br />Check your DNS settings. If you are using a CNAME origin record, make sure it is valid and resolvable. <a rel=\"noopener noreferrer\" href=\"https://support.cloudflare.com/hc/en-us/articles/234979888-Error-1016-Origin-DNS-error\">Additional troubleshooting information here.</a></p>
          </div>
          
      </section>

      <div class=\"feedback-hidden py-8 text-center\" id=\"error-feedback\">
    <div id=\"error-feedback-survey\" class=\"footer-line-wrapper\">
        Was this page helpful?
        <button class=\"border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded\" id=\"feedback-button-yes\" type=\"button\">Yes</button>
        <button class=\"border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded\" id=\"feedback-button-no\" type=\"button\">No</button>
    </div>
    <div class=\"feedback-success feedback-hidden\" id=\"error-feedback-success\">
        Thank you for your feedback!
    </div>
</div>


      <div class=\"cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300\">
  <p class=\"text-13\">
    <span class=\"cf-footer-item sm:block sm:mb-1\">Cloudflare Ray ID: <strong class=\"font-semibold\">78c2234cf35dfab6</strong></span>
    <span class=\"cf-footer-separator sm:hidden\">&bull;</span>
    <span id=\"cf-footer-item-ip\" class=\"cf-footer-item hidden sm:block sm:mb-1\">
      Your IP:
      <button type=\"button\" id=\"cf-footer-ip-reveal\" class=\"cf-footer-ip-reveal-btn\">Click to reveal</button>
      <span class=\"hidden\" id=\"cf-footer-ip\">2a06:98c0:3600::103</span>
      <span class=\"cf-footer-separator sm:hidden\">&bull;</span>
    </span>
    <span class=\"cf-footer-item sm:block sm:mb-1\"><span>Performance &amp; security by</span> <a rel=\"noopener noreferrer\" href=\"https://www.cloudflare.com/5xx-error-landing\" id=\"brand_link\" target=\"_blank\">Cloudflare</a></span>
    
  </p>
  <script>(function(){function d(){var b=a.getElementById(\"cf-footer-item-ip\"),c=a.getElementById(\"cf-footer-ip-reveal\");b&&\"classList\"in b&&(b.classList.remove(\"hidden\"),c.addEventListener(\"click\",function(){c.classList.add(\"hidden\");a.getElementById(\"cf-footer-ip\").classList.remove(\"hidden\")}))}var a=document;document.addEventListener&&a.addEventListener(\"DOMContentLoaded\",d)})();</script>
</div><!-- /.error-footer -->


    </div><!-- /#cf-error-details -->
  </div><!-- /#cf-wrapper -->

  <script>
  window._cf_translation = {};
  
  
</script>

</body>
</html>

gobengo added a commit that referenced this issue Jan 19, 2023
…of up.web3.storage (#376)

Motivation
* work around #363 by
not using the domain name that cloudflare workers can't resolve. Instead
bypass that dns rule and just request the aws http gateway domain
@gobengo
Copy link
Contributor Author

gobengo commented Jan 19, 2023

#status - we worked around this by using a different prod URL that goes directly to aws, but will leave this open until there is more update from my support ticket to cloudflare

gobengo added a commit that referenced this issue Jan 23, 2023
Motivation:
* cloudflare support asked for a URL to reproduce the undelrying cause
of #363
* this adds a route just for that. once they use it to reproduce and
diagnose, we remove it
@gobengo
Copy link
Contributor Author

gobengo commented Jan 31, 2023

cloudflare support got back to me basically saying this is because it's resolving up.web3.storage using their internal DNS, which doesn't resolve the same as 'external dns', and that he put in a feature request to support it.... I think we will keep using the workaround indefinitely (avoid cloudflare dns alltogether by using https://3bd9h7xn3j.execute-api.us-west-2.amazonaws.com/ instead of https://up.web3.storage (only when requesting from inside cloudflare workers access-api)

@gobengo gobengo closed this as completed Jan 31, 2023
gobengo added a commit that referenced this issue Apr 11, 2023
…y non-ok responses (to debug) (#364)

…that logs unexpected responses with status=530

Motivation:
* gather more info for [this error happening in
production](#363 (comment))

Plan
* gather info, then revert this PR
gobengo added a commit that referenced this issue Apr 11, 2023
…or, by default catches HTTPError -> error result w/ status=502 (#366)

, which defaults to something that catches HTTPErrors and rewrites to a
status=502 error result

Motivation:
* relates to: #363
* instead of that behavior of getting a `HandlerExecutionError` (due to
internally having an uncaught `HTTPError`), after this PR we should
* not have the uncaught `HTTPError`, so no uncaught
`HandlerExecutionError`
* the result will have `status=502` and `x-proxy-error` with information
about the proxy invocation request that errored (which will help debug
#363)

Limitations
* This inlcudes in `result['x-proxy-error']` any info from the
underlying `@ucanto/transport/http` `HTTPError`, but unless/until we put
more info on that error, we still won't have the raw response object and
e.g. won't have response headers.
* but if those properties are ever added to `HTTPError`, they should
show up in `x-proxy-error`
gobengo added a commit that referenced this issue Apr 11, 2023
…of up.web3.storage (#376)

Motivation
* work around #363 by
not using the domain name that cloudflare workers can't resolve. Instead
bypass that dns rule and just request the aws http gateway domain
gobengo added a commit that referenced this issue Apr 11, 2023
Motivation:
* cloudflare support asked for a URL to reproduce the undelrying cause
of #363
* this adds a route just for that. once they use it to reproduce and
diagnose, we remove it
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant