Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Offline Development Collection Test Suite #4022

Open
2 tasks done
pieterocp opened this issue Feb 13, 2025 · 0 comments
Open
2 tasks done

Offline Development Collection Test Suite #4022

pieterocp opened this issue Feb 13, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@pieterocp
Copy link

I have checked the following:

  • I use the newest version of bruno.
  • I've searched existing issues and found nothing related to my issue.

Describe the bug

When working on my fork of the whole bruno project to replicate a defect that my colleague is having, I checked out and started running the test suite as detailed in packages/bruno-tests/readme.md, but when I tried to run the Local tests, some of them work, but some of them are failing.

This is probably something where if we fixed this, and part of that would probably entail making the Local suite also run inside the github action, it might help accelerate any/all developers working on the project since they would be able to have the suite run much faster and can continue working when their internet dies or is slow.

I've done a bit of an investigation, but the answer will probably require some changes to both the collections and the testbench, as well as a bit of an extension, e.g. have a very barebones graphql implementation instead of using spacex's in Local/Prod tests. I've detailed a bit of work required to be done below

Failing Steps

Looking through each test case(s) folder by folder in the tree structure:

  • auth/: ~ 75ms locally, ~ 4000 ms using Prod.
  • echo/: This fails out the box since it's using echo-host locally, when the prod has this. With a slight modification (which can come in a pull request), 2586 ms using Prod, ~150ms if I remove some of the xml scenarios that seem to hang locally.
  • graphql/: This calls spacex - 837 ms, will require a bit inside the Local to emulate graphql, but should be a lot faster.
  • preview/: Both making network requests, ~ 500ms
  • redirections/: ~ 41ms ~ 2016 ms using Prod
  • scripting/: 12 failed, ~ 1400ms, ~11,494 ms using Prod but nearly all passed. One with a couple of changes needed. There's a setTimeout with 1000ms which could be shaved down and the axios-pre-req-script.bru uses a hardcoded testbench too.
  • string interpolation/: 3 failed, 39ms vs ~1300ms using Prod, all passing.

In total with failures, getting the local experience of ~2567 ms vs ~19101 ms using Prod without major changes. Which once we swap out the timeout and network requests, it should be able to run around a second.

Below is the local run visualised using a helpful website to show which tests dominate the time taken in any test suite.

visualisation of the local running of the test using https://marmelab.com/phpunit-d3-report/ since it makes some nice bubbles.

.bru file to reproduce the bug

No response

Screenshots/Live demo link

There is nothing to screenshot, this is a test suite speedup.

@pieterocp pieterocp added the bug Something isn't working label Feb 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant