Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request with "parent+children" spans to the zipkin collector service fails with error: index out of range #3404

Closed
alexsuraykin opened this issue Nov 22, 2021 · 4 comments · Fixed by #4160
Labels

Comments

@alexsuraykin
Copy link

If we send parent and children spans in request, jaeger will be failed with error: "runtime error: index out of range"

Steps to reproduce:

  1. Run jaeger.
  2. Send POST request to the "<jaeger_address>: <zipkin_collector_port>/api/v1/spans" with next json in the body:
[
    {
        "traceId": "4ed11df465275600",
        "id": "4ed11df465275600",
        "name": "get /",
        "timestamp": 1633073248674949,
        "duration": 14861,
        "localEndpoint": {
            "serviceName": "first_service",
            "ipv4": "10.0.2.15"
        },
        "annotations": [
            {
                "timestamp": 1633073248674949,
                "value": "sr"
            },
            {
                "timestamp": 163307324868981,
                "value": "ss"
            }
        ]
    },
    {
        "traceId": "4ed11df465275600",
        "parentId": "4ed11df465275600",
        "id": "c943743e25dc2cdf",
        "name": "get /api",
        "timestamp": 1633073248678309,
        "duration": 3360,
        "localEndpoint": {
            "serviceName": "first_service",
            "ipv4": "10.0.2.15"
        },
        "annotations": [
            {
                "timestamp": 1633073248678309,
                "value": "cs"
            },
            {
                "timestamp": 1633073248681669,
                "value": "sr"
            },
            {
                "timestamp": 1633073248685029,
                "value": "cr"
            },
            {
                "timestamp": 1633073248688388,
                "value": "ss"
            }
        ]
    }
]

Expected result: 202 Accepted
Actual result: 500 Internal Server Error

  • Jaeger version: 1.28.0, master
  • Deployment: Docker, Kubernetes

Additional context
During the debug, we found that the problem occurs in the SubmitZipkinBatch function.

After "Sanitize" there are "mSpans" array with 3 items and "bools" with labels of processed spans :

image

image

Where span with "parentId" field was converted in two items.

And then function creates "responses" array with length=2(from original "spans"), but iterates through "bools" with length =3. Eventually there is rises error "index out of range [2] with length 2".

@pavolloffay
Copy link
Member

@alexsuraykin thanks for reporting this. Would you like to work on a fix?

@frzifus
Copy link
Member

frzifus commented Dec 17, 2021

hey, out of interest, I went down the street and noticed this comment:

if the span is client and server we split it into two separate spans

Looking at the 2nd span, it matches this condition.

@rgroothuijsen
Copy link

The problem appears to be that the IsRPCClient() method only checks if the span has a kind field set to CLIENT, but this field was not introduced until v2. If I understand correctly, the v1 equivalent would be detecting the presence of a cs annotation.

@yurishkuro
Copy link
Member

@alexsuraykin thanks for the reproducer. I think I have a fix, but I am curious how you came by such an input, because the second span is a dual client/server span which should not have been produced by the instrumentation, because it contains client-send an server-receive events that should be logged in different spans (that can share the ID). Are you migrating data from a Zipkin installation to Jaeger? (at query time Zipkin server may merge the two spans into one)

yurishkuro added a commit that referenced this issue Jan 15, 2023
…pan (#4160)

## Which problem is this PR solving?
- Resolves #3404 

## Short description of the changes
- Handle case where one input zipkin span is transformed into two Jaeger
spans, but still return response with the same length as input
- Add a test for dial client/server span provided in #3404 
- Side effect: refactor `cmd/collector/app/zipkin` to avoid circular
dependencies

Signed-off-by: Yuri Shkuro <[email protected]>
shubbham1215 pushed a commit to shubbham1215/jaeger that referenced this issue Feb 22, 2023
…pan (jaegertracing#4160)

## Which problem is this PR solving?
- Resolves jaegertracing#3404 

## Short description of the changes
- Handle case where one input zipkin span is transformed into two Jaeger
spans, but still return response with the same length as input
- Add a test for dial client/server span provided in jaegertracing#3404 
- Side effect: refactor `cmd/collector/app/zipkin` to avoid circular
dependencies

Signed-off-by: Yuri Shkuro <[email protected]>
shubbham1215 pushed a commit to shubbham1215/jaeger that referenced this issue Mar 5, 2023
…pan (jaegertracing#4160)

## Which problem is this PR solving?
- Resolves jaegertracing#3404

## Short description of the changes
- Handle case where one input zipkin span is transformed into two Jaeger
spans, but still return response with the same length as input
- Add a test for dial client/server span provided in jaegertracing#3404
- Side effect: refactor `cmd/collector/app/zipkin` to avoid circular
dependencies

Signed-off-by: Yuri Shkuro <[email protected]>
Signed-off-by: shubbham1215 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants