Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sporadic test failure (∞): AkkaHttpServerInstrumentationTestAsync #430

Closed
trask opened this issue May 22, 2020 · 9 comments
Closed

Sporadic test failure (∞): AkkaHttpServerInstrumentationTestAsync #430

trask opened this issue May 22, 2020 · 9 comments
Labels
contribution welcome Request makes sense, maintainers probably won't have time, contribution would be welcome

Comments

@trask
Copy link
Member

trask commented May 22, 2020 •
edited
Loading 

Condition failed with Exception:

assertTheTraces(1, null, null, method, ERROR)
|                              |
|                              GET
java.util.concurrent.TimeoutException: Timeout waiting for 2 completed/filtered trace(s), found 1 completed/filtered trace(s) and 1 total trace(s): [[SpanData{traceId=TraceId{traceId=cdc3e2a59e677f1082681603d5b26510}, spanId=SpanId{spanId=d034159e506516b8}, traceFlags=TraceFlags{sampled=true}, traceState=TraceState{entries=[]}, parentSpanId=SpanId{spanId=0000000000000000}, resource=Resource{attributes={}}, instrumentationLibraryInfo=InstrumentationLibraryInfo{name=io.opentelemetry.auto.akka-http-10.0, version=null}, name=HTTP GET, kind=SERVER, startEpochNanos=1590172653480002009, attributes={http.url=AttributeValueString{stringValue=http://localhost:42881/error-status}, http.status_code=AttributeValueLong{longValue=500}, http.method=AttributeValueString{stringValue=GET}}, timedEvents=[], links=[], status=Status{canonicalCode=UNKNOWN, description=null}, endEpochNanos=1590172653481120521, hasRemoteParent=false, hasEnded=true, totalRecordedEvents=0, totalRecordedLinks=0, totalAttributeCount=3}, SpanData{traceId=TraceId{traceId=cdc3e2a59e677f1082681603d5b26510}, spanId=SpanId{spanId=f38e568d4252f579}, traceFlags=TraceFlags{sampled=true}, traceState=TraceState{entries=[]}, parentSpanId=SpanId{spanId=d034159e506516b8}, resource=Resource{attributes={}}, instrumentationLibraryInfo=InstrumentationLibraryInfo{name=io.opentelemetry.auto, version=null}, name=controller, kind=INTERNAL, startEpochNanos=1590172653480558345, attributes={}, timedEvents=[], links=[], status=Status{canonicalCode=OK, description=null}, endEpochNanos=1590172653480849804, hasRemoteParent=false, hasEnded=true, totalRecordedEvents=0, totalRecordedLinks=0, totalAttributeCount=0}]]
	at io.opentelemetry.auto.test.InMemoryExporter.waitForTraces(InMemoryExporter.java:169)
	at io.opentelemetry.auto.test.asserts.InMemoryExporterAssert.assertTraces(InMemoryExporterAssert.groovy:46)
	at io.opentelemetry.auto.test.AgentTestRunner.assertTraces(AgentTestRunner.java:207)
	at io.opentelemetry.auto.test.base.HttpServerTest.assertTheTraces(HttpServerTest.groovy:378)
	at io.opentelemetry.auto.test.base.HttpServerTest.test error(HttpServerTest.groovy:298)

	at io.opentelemetry.auto.test.base.HttpServerTest.test error(HttpServerTest.groovy:298)
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for 2 completed/filtered trace(s), found 1 completed/filtered trace(s) and 1 total trace(s): [[SpanData{traceId=TraceId{traceId=cdc3e2a59e677f1082681603d5b26510}, spanId=SpanId{spanId=d034159e506516b8}, traceFlags=TraceFlags{sampled=true}, traceState=TraceState{entries=[]}, parentSpanId=SpanId{spanId=0000000000000000}, resource=Resource{attributes={}}, instrumentationLibraryInfo=InstrumentationLibraryInfo{name=io.opentelemetry.auto.akka-http-10.0, version=null}, name=HTTP GET, kind=SERVER, startEpochNanos=1590172653480002009, attributes={http.url=AttributeValueString{stringValue=http://localhost:42881/error-status}, http.status_code=AttributeValueLong{longValue=500}, http.method=AttributeValueString{stringValue=GET}}, timedEvents=[], links=[], status=Status{canonicalCode=UNKNOWN, description=null}, endEpochNanos=1590172653481120521, hasRemoteParent=false, hasEnded=true, totalRecordedEvents=0, totalRecordedLinks=0, totalAttributeCount=3}, SpanData{traceId=TraceId{traceId=cdc3e2a59e677f1082681603d5b26510}, spanId=SpanId{spanId=f38e568d4252f579}, traceFlags=TraceFlags{sampled=true}, traceState=TraceState{entries=[]}, parentSpanId=SpanId{spanId=d034159e506516b8}, resource=Resource{attributes={}}, instrumentationLibraryInfo=InstrumentationLibraryInfo{name=io.opentelemetry.auto, version=null}, name=controller, kind=INTERNAL, startEpochNanos=1590172653480558345, attributes={}, timedEvents=[], links=[], status=Status{canonicalCode=OK, description=null}, endEpochNanos=1590172653480849804, hasRemoteParent=false, hasEnded=true, totalRecordedEvents=0, totalRecordedLinks=0, totalAttributeCount=0}]]
	at io.opentelemetry.auto.test.InMemoryExporter.waitForTraces(InMemoryExporter.java:169)
	at io.opentelemetry.auto.test.asserts.InMemoryExporterAssert.assertTraces(InMemoryExporterAssert.groovy:46)
	at io.opentelemetry.auto.test.AgentTestRunner.assertTraces(AgentTestRunner.java:207)
	at io.opentelemetry.auto.test.base.HttpServerTest.assertTheTraces(HttpServerTest.groovy:378)
	... 1 more

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-auto-instr-java/1096/workflows/78db301d-a95f-4840-ae1d-26ce1a738c02/jobs/9129

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-auto-instr-java/1100/workflows/567e0241-54fc-48b8-9837-58653e5b605a/jobs/9187

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-auto-instr-java/1109/workflows/411adb56-143f-45e4-8acd-d4c2baf6a731/jobs/9254/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-auto-instr-java/1114/workflows/16408ace-5ece-46f0-9e7b-d0d83fd58e60/jobs/9291/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-auto-instr-java/1132/workflows/8e2880a3-ae6f-420b-be5e-133e77828cc7/jobs/9431/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/1253/workflows/e9e863bd-1a72-46cd-9925-63d44d1c90df/jobs/10253/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/1250/workflows/3c0acfa5-fd04-49ed-8649-d2b7a3eaf349/jobs/10230/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/1283/workflows/9f67fa8d-09d4-4010-b6a4-ba7eb6186180/jobs/10493/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/1343/workflows/8f399956-977c-4363-8038-10b5f4b43cda/jobs/10894/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/1496/workflows/b0c9e9aa-863a-4b36-9860-f5dbb5d82335/jobs/11897/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/1613/workflows/1cc2a1f3-a4ad-47da-aad9-6482f7cd7649/jobs/12594/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/1614/workflows/a233fe76-ec29-4f9c-8dac-039da7fc30cb/jobs/12602

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/1644/workflows/c0eebf21-0cc4-46b0-ab66-7c840014d22d/jobs/12780/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/1703/workflows/24115aca-c02f-49a5-99e6-242e9392cddb/jobs/13003/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/1705/workflows/65e2ce17-687b-434e-8b6b-74a94a46989f/jobs/13028/steps

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/1862/workflows/f6a4e87d-2416-4871-bffa-e1baa3f72baa/jobs/13505/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/2128/workflows/c87c0537-7547-4f76-a7ea-1e29945d5824/jobs/14403

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/2162/workflows/8fefb78f-18e5-4d71-b34f-70ccb3148c22/jobs/14526

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/2178/workflows/f0cc9e98-3d96-40ab-b9c6-59fcb264dedf/jobs/14584/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/2181/workflows/5d7d7365-1ac1-413b-9465-ef1c327f805c/jobs/14588

Now with context leak propagation path:

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/2408/workflows/d7e4453d-2ba1-4e2b-bc5a-860d19b942f1/jobs/15466/tests

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/2417/workflows/a9a775e4-140f-43fe-a70b-53b6e53275cb/jobs/15490
@trask trask changed the title Sporadic test failure: test error[0] - AkkaHttpServerInstrumentationTestAsync Sporadic test failure: AkkaHttpServerInstrumentationTestAsync May 23, 2020
@trask trask added the contribution welcome Request makes sense, maintainers probably won't have time, contribution would be welcome label May 26, 2020
@trask trask mentioned this issue May 26, 2020
Merged
@trask trask changed the title Sporadic test failure: AkkaHttpServerInstrumentationTestAsync Sporadic test failure (4x): AkkaHttpServerInstrumentationTestAsync May 27, 2020
@trask trask changed the title Sporadic test failure (4x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (5x): AkkaHttpServerInstrumentationTestAsync May 27, 2020
@iNikem iNikem changed the title Sporadic test failure (5x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (6x): AkkaHttpServerInstrumentationTestAsync Jun 5, 2020
@iNikem iNikem changed the title Sporadic test failure (6x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (7x): AkkaHttpServerInstrumentationTestAsync Jun 6, 2020
@trask trask changed the title Sporadic test failure (7x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (8x): AkkaHttpServerInstrumentationTestAsync Jun 9, 2020
@trask trask changed the title Sporadic test failure (8x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (9x): AkkaHttpServerInstrumentationTestAsync Jun 9, 2020
@trask trask changed the title Sporadic test failure (9x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (10x): AkkaHttpServerInstrumentationTestAsync Jun 11, 2020
@trask trask changed the title Sporadic test failure (10x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (11x): AkkaHttpServerInstrumentationTestAsync Jun 15, 2020
@trask trask changed the title Sporadic test failure (11x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (12x): AkkaHttpServerInstrumentationTestAsync Jun 24, 2020
@iNikem iNikem changed the title Sporadic test failure (12x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (13x): AkkaHttpServerInstrumentationTestAsync Jun 25, 2020
@trask trask changed the title Sporadic test failure (13x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (14x): AkkaHttpServerInstrumentationTestAsync Jun 29, 2020
@trask trask changed the title Sporadic test failure (14x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (15x): AkkaHttpServerInstrumentationTestAsync Jun 29, 2020
@trask trask changed the title Sporadic test failure (15x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (16x): AkkaHttpServerInstrumentationTestAsync Jul 2, 2020
@iNikem iNikem changed the title Sporadic test failure (16x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (18x): AkkaHttpServerInstrumentationTestAsync Jul 6, 2020
@trask trask changed the title Sporadic test failure (18x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (19x): AkkaHttpServerInstrumentationTestAsync Jul 7, 2020
@anuraaga
Copy link
Contributor

anuraaga commented Jul 7, 2020

Hate this test :)

@trask trask changed the title Sporadic test failure (19x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (20x): AkkaHttpServerInstrumentationTestAsync Jul 13, 2020
@iNikem iNikem changed the title Sporadic test failure (20x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (21x): AkkaHttpServerInstrumentationTestAsync Jul 23, 2020
@trask trask changed the title Sporadic test failure (21x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (22x): AkkaHttpServerInstrumentationTestAsync Jul 25, 2020
@trask trask changed the title Sporadic test failure (22x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (23x): AkkaHttpServerInstrumentationTestAsync Jul 27, 2020
@trask
Copy link
Member Author

trask commented Jul 27, 2020

Hate this test :)

Me too :(

@trask trask changed the title Sporadic test failure (23x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (24x): AkkaHttpServerInstrumentationTestAsync Jul 27, 2020
@iNikem
Copy link
Contributor

iNikem commented Jul 27, 2020 

10:20:44.829 [Test worker] DEBUG http-client - --> GET http://localhost:44743/exception |  
  | 10:20:44.835 [my-system-akka.actor.default-dispatcher-16] DEBUG io.opentelemetry.auto.test.InMemoryExporter - >>> SPAN START: TEST_SPAN id=677a0ea01e291a44 traceid=512f3642887272204514ad48b69b6c9f parent=26a477984cf36f4c, library=InstrumentationLibraryInfo{name=io.opentelemetry.auto, version=null} |  
  | 10:20:44.835 [my-system-akka.actor.default-dispatcher-16] DEBUG io.opentelemetry.auto.test.InMemoryExporter - <<< SPAN END: TEST_SPAN id=677a0ea01e291a44 traceid=512f3642887272204514ad48b69b6c9f parent=26a477984cf36f4c, attributes= |  
  | 10:20:44.835 [my-system-akka.actor.default-dispatcher-16] DEBUG io.opentelemetry.auto.test.InMemoryExporter - >>> SPAN START: akka.request id=e7f80a84a45db352 traceid=c3f22294dbeb4020ab09e425bfd08061 parent=0000000000000000, library=InstrumentationLibraryInfo{name=io.opentelemetry.auto.akka-http-10.0, version=null} |  
  | 10:20:44.835 [my-system-akka.actor.default-dispatcher-59] DEBUG io.opentelemetry.auto.test.InMemoryExporter - >>> SPAN START: controller id=a55a6da4544f2a84 traceid=c3f22294dbeb4020ab09e425bfd08061 parent=e7f80a84a45db352, library=InstrumentationLibraryInfo{name=io.opentelemetry.auto, version=null} |  
  | 10:20:44.837 [my-system-akka.actor.default-dispatcher-59] DEBUG io.opentelemetry.auto.test.InMemoryExporter - <<< SPAN END: controller id=a55a6da4544f2a84 traceid=c3f22294dbeb4020ab09e425bfd08061 parent=e7f80a84a45db352, attributes=Attribute error.type=java.lang.ExceptionAttribute error.msg=controller exceptionAttribute error.stack=java.lang.Exception: controller exception |  

Looks like the same issue as in #787

@trask trask changed the title Sporadic test failure (24x): AkkaHttpServerInstrumentationTestAsync Sporadic test failure (∞): AkkaHttpServerInstrumentationTestAsync Jul 29, 2020
@trask
Copy link
Member Author

trask commented Jul 29, 2020

I stopped updating the count on this a while ago.

Bumping it to p1.

This was referenced Aug 2, 2020
Merged
Merged
@trask
Copy link
Member Author

trask commented Aug 12, 2020

Closing per #925

@trask trask closed this as completed Aug 12, 2020
@trask
Copy link
Member Author

trask commented Aug 20, 2020

Unfortunately, still happening:

https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-java-instrumentation/2943/workflows/a2df14e1-dcad-4039-a9c1-db5785c182f0/jobs/16895/tests

@trask trask reopened this Aug 20, 2020
@trask
Copy link
Member Author

trask commented Aug 20, 2020

(see #926)

@trask
Copy link
Member Author

trask commented Aug 20, 2020

but dropping to p2 because it doesn't seem to be happening as often

@trask
Copy link
Member Author

trask commented Apr 3, 2021

Closing, I don't think this has been giving us a problem lately.

@trask trask closed this as completed Apr 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
contribution welcome Request makes sense, maintainers probably won't have time, contribution would be welcome
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@anuraaga @trask @iNikem