Expectation failures should be reported on all pact URLs, not just on the first one #88
Assignees
Comments
alexeits
changed the title
|
I can't tell if this is a problem with the go streams, or the ruby streams. Will investigate when I get a chance. |
I can't see any difference between the first and second in the ruby verifier - the json output goes to stdout both times, and only the warnings are shown in stderr. Let me know if you find anything different @mefellows |
|
@bethesque you may want to try with I can see the first pact output going to stdout, the second to stderr: |
|
Hm. This has turned out to be trickier than I expected. I've recreated it, I've just got to work out the best way to fix it. |
|
Ok, I've fixed this, but I'm not entirely sure I've solved it the best way. The way the pacts are being passed into the verifier one at a time, so we get a separate JSON document for each pact. This means that the stdout cannot be JSON parsed directly, but needs to be split at the joins If I pass all the pacts in together, then there will just be one overall JSON results document. That would seem to be a better solution. However, there is an outstanding request to send the results for each pact to a separate file, so for that, I'd need to pass the pacts in separately (as they currently are), and give each a unique file path to write the JSON results to. I'm in a dilemma! Perhaps I'll need to add an option to allow either verifying them separately, or as a group. |
|
@bethesque I think this approach (JSONS) is quite acceptable. I would only recommend using a new line in between JSON objects to turn it into JSONL. It is much easier to parse by using line-reading streams, which are available in pretty much every language. Of course, in either case the client, e.g. pact-go would need to change to split the JSON objects first. |
|
Hm. That could work. I'd need to make a modification somewhere to add the new line in, but that shouldn't be too hard. |
|
Thanks @bethesque , I can take a look at fixing this in Pact Go tomorrow. Just to re-cap, you've fixed the verifier so that all results come back in JSON without the line separator. Should I wait for the JSONL implementation before updating? |
bethesque
added a commit
to pact-foundation/pact-provider-verifier
that referenced
this issue
Jul 12, 2018
This allows the stream to be parsed as jsonl Fixes: pact-foundation/pact-go#88
|
Done. Each JSON document is now separated by a new line as per jsonl. Please use 1.52.0 of the standalone. |
|
Boom 💥
I'll get onto this today, thanks Beth.
…On Fri, Jul 13, 2018 at 9:08 AM, Beth Skurrie ***@***.***> wrote:
Done. Each JSON document is now separated by a new line as per jsonl.
Please use 1.52.0 of the standalone.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#88 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AADSjPST9zl7eTFvHWee2jsGi4POuVQBks5uF9b5gaJpZM4UxYHG>
.
--
Matt Fellows
- http://www.onegeek.com.au/
|
|
@bethesque, @mefellows thanks a lot for the quick turnaround. The responsiveness of your team is amazing. |
|
No worries @alexeits - we ❤️ it! |
|
Latest release is out @alexeits - tag is Let us know how you go! |
|
I got results for both pacts after the upgrade. It works! Thank you very much! |
|
Excellent! 👍
…On Sat, Jul 14, 2018, 3:37 AM Alex Tsibulya ***@***.***> wrote:
I got results for both pacts after the upgrade. It works! Thank you very
much!
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#88 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AADSjFHVMpCtIglgXCsXD7aR4G8aGa8Dks5uGNrrgaJpZM4UxYHG>
.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Software versions
OS: macOS High Sierra 10.13.5
Consumer Pact library: Pact Go CLI v1.0.0-beta.1, using CLI tools version 1.44.1
Provider Pact library:
Golang Version:
go version go1.10.1 darwin/amd64Golang environment:
Expected behaviour
Actual behaviour
It the first pact succeeds expectation failures on all subsequent pacts are ignored
Steps to reproduce
Relevant log files
Running pacts by hand with
pact-provider-verifiershows that the result for the first pact URL go tostdout:{ "version": "3.7.1", "examples": [ { "id": "/opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.26.0/lib/pact/provider/rspec.rb[1:1:1:1:1:1]", "description": "has status code 200", "full_description": "Verifying a pact between agent-desktop and auth Given michelle.smith exists in example.com org create desktop token with POST /api/v3/desktop_tokens returns a response which has status code 200", "status": "passed", "file_path": "/opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.26.0/lib/pact/provider/rspec.rb", "line_number": 148, "run_time": 0.030802, "pending_message": null }, // ... cut ... ], "summary": { "duration": 0.038427, "example_count": 6, "failure_count": 0, "pending_count": 0, "errors_outside_of_examples_count": 0 }, "summary_line": "6 examples, 0 failures" }while failures for the second pact URL go to
stderr:{ "version": "3.7.1", "examples": [ { "id": "/opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.26.0/lib/pact/provider/rspec.rb[1:1:1:1:1:1]", "description": "has status code 200", "full_description": "Verifying a pact between api and auth Given michelle.smith exists in example.com org get existing user from API token with POST /api/v1/api_users returns a response which has status code 200", "status": "failed", "file_path": "/opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.26.0/lib/pact/provider/rspec.rb", "line_number": 148, "run_time": 0.008817, "pending_message": null, "exception": { "class": "RSpec::Expectations::ExpectationNotMetError", "message": "\nexpected: 200\n got: 404\n\n(compared using eql?)\n", "backtrace": [ "/opt/pact/lib/vendor/ruby/2.2.0/gems/rspec-support-3.7.1/lib/rspec/support.rb:97:in `block in <module:Support>'", "/opt/pact/lib/vendor/ruby/2.2.0/gems/rspec-support-3.7.1/lib/rspec/support.rb:106:in `call'", "/opt/pact/lib/vendor/ruby/2.2.0/gems/rspec-support-3.7.1/lib/rspec/support.rb:106:in `notify_failure'", // ... cut .... ], "summary": { "duration": 0.017386, "example_count": 4, "failure_count": 2, "pending_count": 0, "errors_outside_of_examples_count": 0 }, "summary_line": "4 examples, 2 failures" }It appears that
stderris ignored by the client.go, only stdout is processedThe text was updated successfully, but these errors were encountered: