feat: support JSON-RPC over HTTP request

[titaneric]

This is golang's uprobe implementation for #1976 .

I try to capture http request body by introducing new hook for net/http.(*body).Read function.
The body struct is initialized in the readTransfer in readRequest.

readTransfer prepare the body reader, and body's io.Reader implementation is here.

Note that I capture partial http request body and retrieve the Content-Type in bpf, and send them into userspace.

Test JSON-RPC code:

package main

import (
    "io"
    "log"
    "net"
    "net/http"
    "net/rpc"
    "net/rpc/jsonrpc"
)

// Args defines the arguments for the RPC methods.
type Args struct {
    A, B int
}

// Arith provides methods for arithmetic operations.
type Arith struct{}

// Multiply multiplies two numbers and returns the result.
func (t *Arith) Multiply(args *Args, reply *int) error {
    *reply = args.A * args.B
    return nil
}

// ReadWriteCloserWrapper wraps an io.Reader and io.Writer to implement io.ReadWriteCloser.
type ReadWriteCloserWrapper struct {
    io.Reader
    io.Writer
}
// Close is a no-op to satisfy the io.ReadWriteCloser interface.
func (w *ReadWriteCloserWrapper) Close() error {
    return nil
}
func main() {
    // Register the Arith service.
    arith := new(Arith)
    rpc.Register(arith)

    // Set up an HTTP handler for JSON-RPC.
    http.HandleFunc("/jsonrpc", func(w http.ResponseWriter, r *http.Request) {
        if r.Method != http.MethodPost {
            http.Error(w, "Only POST method is allowed", http.StatusMethodNotAllowed)
            return
        }
	// Wrap the request body and response writer in a ReadWriteCloser.
        conn := &ReadWriteCloserWrapper{Reader: r.Body, Writer: w}
        // Serve the request using JSON-RPC codec.
        rpc.ServeCodec(jsonrpc.NewServerCodec(conn))
    })

    // Start the HTTP server.
    go func() {
        log.Println("JSON-RPC server is listening on HTTP port 8080...")
        if err := http.ListenAndServe(":8080", nil); err != nil {
            log.Fatalf("Error starting HTTP server: %v", err)
        }
    }()

    // Optionally, start a TCP server as well.
    listener, err := net.Listen("tcp", ":1234")
    if err != nil {
        log.Fatalf("Error starting TCP server: %v", err)
    }
    log.Println("JSON-RPC server is listening on TCP port 1234...")

    for {
        conn, err := listener.Accept()
        if err != nil {
            log.Printf("Connection error: %v", err)
            continue
        }
        // Serve the connection using JSON-RPC.
        go rpc.ServeCodec(jsonrpc.NewServerCodec(conn))
    }
}

Sample request

curl -X POST http://localhost:8080/jsonrpc \
           -H "Content-Type: application/json" \
           -d '{
             "jsonrpc": "2.0",
             "method": "Arith.Multiply",
             "params": [{"A": 7, "B": 8}],
             "id": 1
           }'

Output:

time=2025-05-31T16:49:36.797Z level=DEBUG msg="tp: 00-3c8e60d3794797646aa6a9edebeb9247-6699afcf4d748176-01" component=BPFLogger pid=90581 comm=server
time=2025-05-31T16:49:36.797Z level=DEBUG msg="Found content type in ongoing request: application/json" component=BPFLogger pid=90581 comm=server
time=2025-05-31T16:49:36.797Z level=DEBUG msg="ServeHTTP method: POST, path: /jsonrpc, content length: 140" component=BPFLogger pid=90581 comm=server
...
time=2025-05-31T16:49:36.807Z level=DEBUG msg="goroutine_addr 4000106a80" component=BPFLogger pid=90581 comm=server
time=2025-05-31T16:49:36.807Z level=DEBUG msg="tp: 00-3c8e60d3794797646aa6a9edebeb9247-6699afcf4d748176-01\xd5a\x01" component=BPFLogger pid=90581 comm=server
time=2025-05-31T16:49:36.807Z level=DEBUG msg="ServeHTTP_ret tp: 00-3c8e60d3794797646aa6a9edebeb9247-6699afcf4d748176-01\xd5a\x01" component=BPFLogger pid=90581 comm=server
time=2025-05-31T16:49:36.807Z level=DEBUG msg="ServeHTTP_ret method: POST" component=BPFLogger pid=90581 comm=server
time=2025-05-31T16:49:36.807Z level=DEBUG msg="ServeHTTP_ret path: /jsonrpc" component=BPFLogger pid=90581 comm=server
time=2025-05-31T16:49:36.807Z level=DEBUG msg="ServeHTTP_ret content_type: application/json" component=BPFLogger pid=90581 comm=server
...
time=2025-05-31T16:49:36.877Z level=DEBUG msg="submitting traces on timeout" component=ringbuf.Tracer len=1
2025-05-31 16:49:36.53144936 (9.356261ms[9.332928ms]) HTTP 200 Arith.Multiply /jsonrpc [::1 as ::1:46882]->[::1 as ::1:8080] contentLen:140B responseLen:0B svc=[server (deleted) go] traceparent=[00-3c8e60d3794797646aa6a9edebeb9247-6699afcf4d748176[0000000000000000]-01]

I hope that present direction is correct for the implementation. I would like to hear more discussion and feedback, and I would continue the work for response body capturing, error code handling, etc.

[grcevski]

This is a great start @titaneric ! Thanks for putting this together. I left a few comments/questions.

[grcevski]

I wonder if we can make this more solid. One problematic example would be if the request JSON doesn't fit in 256 bytes, then I think the unmarshal call will fail with error. What do you think about looking for the method and version by string search?

[titaneric]

Sounds reasonable, matching the method and version may be relatively robust way to match partitioned JSON-RPC in request body

[rafaelroquetto]

Hey @titaneric, thanks for bringing this up. Adding to what was said above, I think this should be done at the ebpf level if feasible (just in case that is not already the idea), as there's non-negligible overhead in shipping non-trivial amounts of data to userspace, allocating strings and instantiating a json parser.

Since you are already reading the body and the content types in go_nethttp.c, there are few things I reckon you could do:

• when reading content_type, you can simply check on the spot if it matches one of the valid types and instead of storing the string, you can store an enum value (such as JSON)
• later when reading the body, you can perform some sort of string search or parsing as @grcevski is suggesting - it's not as trivial in ebpf, but we have a few examples scattered across the code that can serve as inspiration, for instance, in https://github.com/grafana/beyla/blob/main/bpf/common/tc_common.h there are a few functions we use for looking things up - in your case, the good news is that you know the buffer sizes at compile time, so that will keep the ebpf verifier happy.

You can then ultimately fill in the method and any other info you may have directly from the ebpf layer, leaving the http_request_trace_t untouched.

Feel free to reach out on the Slack channel if you need further help with this. Happy to assist :)

[titaneric]

May I introduce new event type for JSON-RPC over HTTP, or I could simply reuse http event and overwrite the method / error code?

[rafaelroquetto]

I'd say whatever is the easiest (provided that we don't end up doing too many gymnastics in the code)- what do you think?

[titaneric]

I wound try to reuse current http event type, overwrite the method and status code inside the bpf.

[titaneric]

I detect the JSON-RPC in bpf and overwrite the method for http event type successfully.
Present implementation works, and I would like to try to detect JSON-RPC and extract method in single pass function.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.30%. Comparing base (a2ba6e5) to head (2407702).
Report is 21 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2008      +/-   ##
==========================================
+ Coverage   69.48%   72.30%   +2.81%     
==========================================
  Files         182      186       +4     
  Lines       20162    20474     +312     
==========================================
+ Hits        14010    14804     +794     
+ Misses       5395     4881     -514     
- Partials      757      789      +32     

Flag	Coverage Δ
integration-test	56.08% <100.00%> (-0.59%)	⬇️
k8s-integration-test	55.15% <100.00%> (?)
oats-test	?
unittests	45.49% <0.00%> (-0.52%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rafaelroquetto]

Great stuff! I did another pass, will dig deeper into it after we iterate. I also triggered the tests, it's mostly looking good - we'll see if OATS pass, and if you need any help ping me on Slack!

[rafaelroquetto]

likewise here you could reuse the function I suggested above.

I wonder if we can further reduce this to:

// returns the start of value (i.e. after the key and whitespace) inside 'body`, NULL otherwise
const char *value = json_rpc2_value(key, key_size);
const char *value_end  = jsonrpc2_value_end(value);

value_end can use memchar or find_first_pos_of in tc_common.h.

[titaneric]

That's a good idea!

[titaneric]

I don't think we no longer need json_value_offset to skip space and colon since we have json_str_value function, but we could keep it for possible usage to match not only string type but any types as well?

[rafaelroquetto]

it's best to move body_addr under status - and content_type after path - the idea here is to reduce padding altogether

[titaneric]

I think we still need 5 additional bytes padding in the end after moving it.
May we increase the METHOD_MAX_LEN from 7 bytes to a higher value such as 12 or 16?

[titaneric]

Please note that in this PR I set JSONRPC_METHOD_BUF_SIZE to 16.

[rafaelroquetto]

For the OATS tests, it may be related to #2028 - so you may wanna rebase (or merge)

[titaneric]

Please give me some time to apply the patch into https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/main, and I would raise the PR there.

[grcevski]

Thank you so much!

[marctc]

The code that this PR is touching is moved to https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation and Beyla vendors it, so this is code is not here anymore. Please re-open the PR in that repository if this still relevant. Thanks and sorry for the inconveniences.