error while running instrumentors #367

vasiliy-grinko · 2023-10-09T14:50:14Z

Describe the bug

I'm running my app with the instrumentation in K8s, I have enabled the feature gate operator.autoinstrumentation.go.

"msg":"error while running instrumentors","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argument: regs=200 stack=0 before 423: (63) *(u32 *)(r
10 -96) = r9: regs=200 stac (truncated, 1043 line(s) omitted)","stacktrace":"main.main\n\t/app/cli/main.go:88\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"

Environment

OS: Ubuntu 22.04.3 LTS
Go Version: 1.19.1
Version: v0.3.0-alpha

Expected behavior

The instrumentation container is not crushing

Logs

{"level":"info","ts":1696862058.7040868,"caller":"cli/main.go:37","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1696862058.7041774,"caller":"opentelemetry/controller.go:119","msg":"Establishing connection to OTLP receiver ..."}
{"level":"info","ts":1696862060.7122269,"caller":"process/discover.go:55","msg":"found process","pid":1496429}
{"level":"info","ts":1696862060.7129314,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496429}
{"level":"info","ts":1696862060.7129982,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496449}
{"level":"info","ts":1696862060.7130327,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496450}
{"level":"info","ts":1696862060.7131026,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496451}
{"level":"info","ts":1696862060.7131865,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496452}
{"level":"info","ts":1696862060.713268,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496453}
{"level":"info","ts":1696862060.7133186,"caller":"ptrace/ptrace_linux.go:121","msg":"attach successfully","tid":1496454}
{"level":"info","ts":1696862060.7219062,"caller":"process/analyze.go:94","msg":"Detaching from process","pid":1496429}
{"level":"info","ts":1696862060.7220416,"caller":"process/analyze.go:139","msg":"mmaped remote memory","start_addr":"7F9AC2800000","end_addr":"7F9AC4000000"}
{"level":"info","ts":1696862060.890518,"caller":"process/analyze.go:168","msg":"found relevant function for instrumentation","function":"net/http.(*Client).do","start":2624320,"returns":[2624852,2627125,2627214,2627635,2627830,2627927,2628224,2628322
,2628428]}
{"level":"info","ts":1696862060.890655,"caller":"process/analyze.go:168","msg":"found relevant function for instrumentation","function":"net/http.HandlerFunc.ServeHTTP","start":2933216,"returns":[2933272]}
{"level":"info","ts":1696862060.8910005,"caller":"process/analyze.go:168","msg":"found relevant function for instrumentation","function":"github.com/gorilla/mux.(*Router).ServeHTTP","start":5012320,"returns":[5012798,5013120]}
{"level":"info","ts":1696862060.891502,"caller":"cli/main.go:79","msg":"target process analysis completed","pid":1496429,"go_version":"1.19.1","dependencies":{"code.cloudfoundry.org/lager":"v2.0.0+incompatible","git.swisscom.com/ctt/ctt-integration-d
op":"(devel)","github.com/KyleBanks/depth":"v1.2.1","github.com/PuerkitoBio/purell":"v1.1.1","github.com/PuerkitoBio/urlesc":"v0.0.0-20170810143723-de5bf2ad4578","github.com/alecthomas/template":"v0.0.0-20190718012654-fb15b899a751","github.com/alecth
omas/units":"v0.0.0-20210208195552-ff826a37aa15","github.com/beorn7/perks":"v1.0.1","github.com/cespare/xxhash/v2":"v2.1.1","github.com/dgrijalva/jwt-go":"v3.2.0+incompatible","github.com/go-openapi/jsonpointer":"v0.19.5","github.com/go-openapi/jsonr
eference":"v0.19.6","github.com/go-openapi/spec":"v0.20.4","github.com/go-openapi/swag":"v0.19.15","github.com/golang/protobuf":"v1.5.2","github.com/google/uuid":"v1.3.0","github.com/gorilla/mux":"v1.8.0","github.com/josharian/intern":"v1.0.0","githu
b.com/lestrrat-go/backoff/v2":"v2.0.8","github.com/lestrrat-go/blackmagic":"v1.0.0","github.com/lestrrat-go/httpcc":"v1.0.0","github.com/lestrrat-go/iter":"v1.0.1","github.com/lestrrat-go/jwx":"v1.2.9","github.com/lestrrat-go/option":"v1.0.0","github
.com/mailru/easyjson":"v0.7.7","github.com/matttproud/golang_protobuf_extensions":"v1.0.1","github.com/mitchellh/mapstructure":"v1.1.2","github.com/opentracing/opentracing-go":"v1.2.0","github.com/openzipkin/zipkin-go":"v0.2.6-0.20210321195400-815bc6
4e1a60","github.com/pborman/uuid":"v1.2.1","github.com/pivotal-cf/brokerapi/v8":"v8.2.0","github.com/pkg/errors":"v0.9.1","github.com/prometheus/client_golang":"v1.7.1","github.com/prometheus/client_model":"v0.2.0","github.com/prometheus/common":"v0.
21.0","github.com/prometheus/procfs":"v0.1.3","github.com/sanity-io/litter":"v1.5.0","github.com/sirupsen/logrus":"v1.8.1","github.com/swaggo/files":"v0.0.0-20190704085106-630677cd5c14","github.com/swaggo/http-swagger":"v1.0.0","github.com/swaggo/swa
g":"v1.7.0","golang.org/x/crypto":"v0.0.0-20201217014255-9d1352758620","golang.org/x/net":"v0.0.0-20211101193420-4a448f8816b3","golang.org/x/sys":"v0.0.0-20211103235746-7861aae1554b","golang.org/x/text":"v0.3.7","golang.org/x/tools":"v0.1.7","google.
golang.org/grpc":"v1.30.0","google.golang.org/protobuf":"v1.26.0","gopkg.in/alecthomas/kingpin.v2":"v2.2.6","gopkg.in/retry.v1":"v1.0.3","gopkg.in/yaml.v2":"v2.4.0"},"total_functions_found":3}
{"level":"info","ts":1696862060.8916557,"caller":"cli/main.go:85","msg":"invoking instrumentors"}
{"level":"info","ts":1696862060.89224,"logger":"allocator","caller":"allocator/allocator_linux.go:39","msg":"Loading allocator","start_addr":140302664859648,"end_addr":140302690025472}
{"level":"info","ts":1696862060.8937452,"caller":"instrumentors/runner.go:85","msg":"loading instrumentor","name":"net/http/client"}
{"level":"info","ts":1696862060.895833,"caller":"inject/injector.go:102","msg":"Injecting variables","vars":{"ctx_ptr_pos":232,"end_addr":140302690025472,"headers_ptr_pos":56,"is_registers_abi":true,"method_ptr_pos":0,"path_ptr_pos":56,"start_addr":1
40302664859648,"total_cpus":4,"url_ptr_pos":16}}
{"level":"error","ts":1696862060.9951704,"caller":"instrumentors/runner.go:88","msg":"error while loading instrumentors, cleaning up","name":"net/http/client","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argu
ment: regs=200 stack=0 before 423: (63) *(u32 *)(r10 -96) = r9: regs=200 stac (truncated, 1043 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentors.(*Manager).load\n\t/app/internal/pkg/instrumentors/runner.go:88\ngo.ope
ntelemetry.io/auto/internal/pkg/instrumentors.(*Manager).Run\n\t/app/internal/pkg/instrumentors/runner.go:36\nmain.main\n\t/app/cli/main.go:86\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
{"level":"info","ts":1696862060.9953918,"caller":"server/probe.go:243","msg":"closing net/http instrumentor"}
{"level":"info","ts":1696862060.9954069,"caller":"client/probe.go:226","msg":"closing net/http/client instrumentor"}
{"level":"info","ts":1696862060.9954352,"caller":"mux/probe.go:222","msg":"closing gorilla/mux instrumentor"}
{"level":"error","ts":1696862060.9954455,"caller":"cli/main.go:88","msg":"error while running instrumentors","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argument: regs=200 stack=0 before 423: (63) *(u32 *)(r
10 -96) = r9: regs=200 stac (truncated, 1043 line(s) omitted)","stacktrace":"main.main\n\t/app/cli/main.go:88\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

The text was updated successfully, but these errors were encountered:

RonFed · 2023-10-16T13:20:46Z

Hi, can you please retry with the latest release ? (v0.7.0-alpha)
In case it is not resolved, please attach the output of uname -r

zsdyx · 2023-11-28T09:26:29Z

I have this problem, how to troubleshoot
go version 1.20.9
ubuntu: 20.04

{"level":"error","ts":1701165351.0630052,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:166","msg":"error while loading probes, cleaning up","name":"net/http/client","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argument: 491: (55) if r8 != 0x0 goto pc+21: R0=map_value(id=0,off=0,ks=4,vs=8,imm=0) R6=inv(id=0) R7=inv524277 R8=inv0 R9=inv(id=0) R10=fp0 fp-48=???mmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmm???? fp-104=00000000 (truncated, 904 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:166\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:121\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:150\nmain.main\n\t/app/cli/main.go:99\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
--
Tue, Nov 28 2023 5:55:51 pm | {"level":"info","ts":1701165351.4821517,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:191","msg":"Cleaning bpffs"}
Tue, Nov 28 2023 5:55:51 pm | {"level":"error","ts":1701165351.4822602,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:100","msg":"instrumentation crashed","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argument: 491: (55) if r8 != 0x0 goto pc+21: R0=map_value(id=0,off=0,ks=4,vs=8,imm=0) R6=inv(id=0) R7=inv524277 R8=inv0 R9=inv(id=0) R10=fp0 fp-48=???mmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmm???? fp-104=00000000 (truncated, 904 line(s) omitted)","stacktrace":"main.main\n\t/app/cli/main.go:100\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}

RonFed · 2023-11-30T14:54:57Z

I have this problem, how to troubleshoot go version 1.20.9 ubuntu: 20.04

{"level":"error","ts":1701165351.0630052,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:166","msg":"error while loading probes, cleaning up","name":"net/http/client","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argument: 491: (55) if r8 != 0x0 goto pc+21: R0=map_value(id=0,off=0,ks=4,vs=8,imm=0) R6=inv(id=0) R7=inv524277 R8=inv0 R9=inv(id=0) R10=fp0 fp-48=???mmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmm???? fp-104=00000000 (truncated, 904 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:166\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:121\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:150\nmain.main\n\t/app/cli/main.go:99\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
--
Tue, Nov 28 2023 5:55:51 pm | {"level":"info","ts":1701165351.4821517,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:191","msg":"Cleaning bpffs"}
Tue, Nov 28 2023 5:55:51 pm | {"level":"error","ts":1701165351.4822602,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:100","msg":"instrumentation crashed","error":"field UprobeHttpClientDo: program uprobe_HttpClient_Do: load program: invalid argument: 491: (55) if r8 != 0x0 goto pc+21: R0=map_value(id=0,off=0,ks=4,vs=8,imm=0) R6=inv(id=0) R7=inv524277 R8=inv0 R9=inv(id=0) R10=fp0 fp-48=???mmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmm???? fp-104=00000000 (truncated, 904 line(s) omitted)","stacktrace":"main.main\n\t/app/cli/main.go:100\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}

Please set the environment variable OTEL_GO_AUTO_SHOW_VERIFIER_LOG=true and attach the full log.

zsdyx · 2023-12-02T07:31:33Z

Hi @RonFed this is log

{"level":"info","ts":1701502016.4696648,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:91","msg":"building OpenTelemetry Go instrumentation ..."}
--
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.470296,"logger":"Instrumentation.Analyzer","caller":"process/discover.go:69","msg":"found process","pid":13}
Sat, Dec 2 2023 3:26:58 pm | {"level":"error","ts":1701502018.470825,"logger":"Instrumentation.Analyzer","caller":"process/module.go:195","msg":"error parsing module info","error":"Malformed version: (devel)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/process.parseModules\n\t/app/internal/pkg/process/module.go:195\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).getModuleDetails\n\t/app/internal/pkg/process/module.go:50\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).Analyze\n\t/app/internal/pkg/process/analyze.go:83\ngo.opentelemetry.io/auto.NewInstrumentation\n\t/app/instrumentation.go:120\nmain.main\n\t/app/cli/main.go:92\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.4708626,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:113","msg":"No symbols found in binary, trying to find functions using .gosymtab"}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5066905,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"net/http.(*Client).do","Offset":2807904,"ReturnOffsets":[2808436,2810803,2810900,2811320,2811522,2811619,2811913,2812010,2812108]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5067766,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"net/http.HandlerFunc.ServeHTTP","Offset":3119968,"ReturnOffsets":[3120024]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5067866,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"github.com/gin-gonic/gin.(*Engine).ServeHTTP","Offset":7861312,"ReturnOffsets":[7861820]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5067918,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*loopyWriter).headerHandler","Offset":8282880,"ReturnOffsets":[8283196,8283240,8283269,8283310,8283427,8283569]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.506797,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*http2Client).NewStream","Offset":8336736,"ReturnOffsets":[8337204,8338339,8338424,8338553,8338656,8339129,8339246]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.506802,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders","Offset":8376416,"ReturnOffsets":[8377291,8377425,8384119,8384769,8384945,8385046,8385281,8385617,8385751,8386465,8388136,8388382]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5068076,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(*ClientConn).Invoke","Offset":8557952,"ReturnOffsets":[8558514,8558580]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5068126,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(*Server).handleStream","Offset":8669472,"ReturnOffsets":[8671159,8672482,8672554,8672640,8672725]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5068178,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"database/sql.(*DB).execDC","Offset":16124320,"ReturnOffsets":[16125251,16125570,16125650,16125838,16125888]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5068228,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"database/sql.(*DB).queryDC","Offset":16127936,"ReturnOffsets":[16129173,16129251,16129298,16129517,16129564]}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5069354,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":13}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5069778,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":14}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5070016,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":15}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5070238,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":16}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5070474,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":17}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5070684,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":18}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5070937,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":19}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5071194,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":20}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5071454,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":21}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5071669,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":22}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5071957,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":23}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5072172,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":24}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5072372,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":25}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5072594,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":26}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5072818,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":27}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5072997,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":28}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.507321,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":29}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5073404,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":30}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.50738,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":31}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5074024,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":48}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5075006,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:234","msg":"Detected linux kernel version","version":"5.4.0"}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5075793,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:246","msg":"mlock ret","ret":18446744073709551604}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5075881,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:64","msg":"Detaching from process","pid":13}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5076582,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:43","msg":"mmaped remote memory","start_addr":"FFFFFFFFFFFFFFF5","end_addr":"7FFF5"}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5076702,"logger":"Instrumentation","caller":"app/instrumentation.go:132","msg":"target process analysis completed","pid":13,"go_version":"1.20.9","dependencies":{"-buildmode=exe":null,"-compiler=gc":null,"-ldflags=\"-s":null,"-tags=jsoniter":null,"CGO_ENABLED=0":null,"GOAMD64=v1":null,"GOARCH=amd64":null,"GOOS=linux":null,"cloud.google.com/go":"0.110.7","cloud.google.com/go/bigquery":"1.53.0","cloud.google.com/go/compute/metadata":"0.2.3","cloud.google.com/go/iam":"1.1.1","cloud.google.com/go/storage":"1.31.0","command-line-arguments":null,"github.com/ClickHouse/ch-go":"0.51.2","github.com/ClickHouse/clickhouse-go/v2":"2.5.1","github.com/Conviva-Internal/mario-compiler":"1.3.5-mai1228-1","github.com/KyleBanks/depth":"1.2.1","github.com/andybalholm/brotli":"1.0.4","github.com/apache/arrow/go/v12":"12.0.0","github.com/apache/thrift":"0.16.0","github.com/beorn7/perks":"1.0.1","github.com/cenkalti/backoff/v4":"4.2.1","github.com/cespare/xxhash/v2":"2.2.0","github.com/dablelv/go-huge-util":"0.0.48","github.com/dgryski/go-rendezvous":"0.0.0-20200823014737-9f7001d12a5f","github.com/dlclark/regexp2":"1.10.0","github.com/gabriel-vasile/mimetype":"1.4.2","github.com/getsentry/sentry-go":"0.19.0","github.com/gin-contrib/cors":"1.3.1","github.com/gin-contrib/sse":"0.1.0","github.com/gin-contrib/timeout":"0.0.3","github.com/gin-gonic/gin":"1.9.1","github.com/go-faster/city":"1.0.1","github.com/go-faster/errors":"0.6.1","github.com/go-logr/logr":"1.3.0","github.com/go-logr/stdr":"1.2.2","github.com/go-logr/zapr":"1.2.4","github.com/go-openapi/jsonpointer":"0.19.6","github.com/go-openapi/jsonreference":"0.20.2","github.com/go-openapi/spec":"0.20.8","github.com/go-openapi/swag":"0.22.3","github.com/go-playground/locales":"0.14.1","github.com/go-playground/universal-translator":"0.18.1","github.com/go-playground/validator/v10":"10.14.0","github.com/go-redis/redis/v8":"8.11.5","github.com/go-redsync/redsync/v4":"4.8.1","github.com/go-sql-driver/mysql":"1.7.1","github.c
  | om/goccy/go-json":"0.10.2","github.com/golang/groupcache":"0.0.0-20210331224755-41bb18bfe9da","github.com/golang/protobuf":"1.5.3","github.com/golang/snappy":"0.0.4","github.com/google/flatbuffers":"2.0.8+incompatible","github.com/google/s2a-go":"0.1.4","github.com/google/uuid":"1.3.1","github.com/googleapis/enterprise-certificate-proxy":"0.2.5","github.com/googleapis/gax-go/v2":"2.12.0","github.com/grpc-ecosystem/grpc-gateway/v2":"2.16.0","github.com/hashicorp/errwrap":"1.1.0","github.com/hashicorp/go-multierror":"1.1.1","github.com/josharian/intern":"1.0.0","github.com/json-iterator/go":"1.1.12","github.com/klauspost/compress":"1.15.15","github.com/klauspost/cpuid/v2":"2.2.4","github.com/leodido/go-urn":"1.2.4","github.com/mailru/easyjson":"0.7.7","github.com/mattn/go-isatty":"0.0.19","github.com/matttproud/golang_protobuf_extensions":"1.0.4","github.com/modern-go/concurrent":"0.0.0-20180306012644-bacd9c7ef1dd","github.com/modern-go/reflect2":"1.0.2","github.com/oliveagle/jsonpath":"0.0.0-20180606110733-2e52cf6e6852","github.com/osteele/liquid":"1.3.0","github.com/osteele/tuesday":"1.0.3","github.com/patrickmn/go-cache":"2.1.0+incompatible","github.com/paulmach/orb":"0.8.0","github.com/pelletier/go-toml/v2":"2.0.8","github.com/pierrec/lz4/v4":"4.1.17","github.com/pkg/errors":"0.9.1","github.com/prometheus/client_golang":"1.16.0","github.com/prometheus/client_model":"0.3.0","github.com/prometheus/common":"0.42.0","github.com/prometheus/procfs":"0.10.1","github.com/segmentio/asm":"1.2.0","github.com/shopspring/decimal":"1.3.1","github.com/swaggo/files":"1.0.0","github.com/swaggo/gin-swagger":"1.5.3","github.com/swaggo/swag":"1.8.10","github.com/ugorji/go/codec":"1.2.11","github.com/wangjia184/sortedset":"0.0.0-20220209072355-af6d6d227aa7","github.com/zeebo/xxh3":"1.0.2","go.opencensus.io":"0.24.0","go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin":"0.46.1","go.opentelemetry.io/otel":"1.21.0","go.opentelemetry.io/otel/exporters/otlp/otlptrace":"1.21.0","go.opentelemetry.io/otel/exporter
  | s/otlp/otlptrace/otlptracehttp":"1.21.0","go.opentelemetry.io/otel/exporters/stdout/stdouttrace":"1.16.0","go.opentelemetry.io/otel/metric":"1.21.0","go.opentelemetry.io/otel/sdk":"1.21.0","go.opentelemetry.io/otel/trace":"1.21.0","go.opentelemetry.io/proto/otlp":"1.0.0","go.uber.org/atomic":"1.11.0","go.uber.org/multierr":"1.11.0","go.uber.org/zap":"1.24.0","golang.org/x/crypto":"0.15.0","golang.org/x/net":"0.18.0","golang.org/x/oauth2":"0.11.0","golang.org/x/sync":"0.3.0","golang.org/x/sys":"0.14.0","golang.org/x/text":"0.14.0","golang.org/x/tools":"0.9.1","golang.org/x/xerrors":"0.0.0-20220907171357-04be3eba64a2","google.golang.org/api":"0.134.0","google.golang.org/appengine":"1.6.7","google.golang.org/genproto":"0.0.0-20230822172742-b8732ec3820d","google.golang.org/genproto/googleapis/api":"0.0.0-20230822172742-b8732ec3820d","google.golang.org/genproto/googleapis/rpc":"0.0.0-20230822172742-b8732ec3820d","google.golang.org/grpc":"1.59.0","google.golang.org/protobuf":"1.31.0","gopkg.in/yaml.v2":"2.4.0","gopkg.in/yaml.v3":"3.0.1","mario":null,"std":"1.20.9"},"total_functions_found":10}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5078304,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:98","msg":"starting instrumentation..."}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5081096,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:178","msg":"Mounting bpffs","allocations_details":{"StartAddr":18446744073709551605,"EndAddr":524277}}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.5084653,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:163","msg":"loading probes","name":"google.golang.org/grpc"}
Sat, Dec 2 2023 3:26:58 pm | {"level":"info","ts":1701502018.6622827,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:163","msg":"loading probes","name":"google.golang.org/grpc/server"}
Sat, Dec 2 2023 3:26:58 pm | Verifier log: load program: no space left on device:
Sat, Dec 2 2023 3:26:58 pm | func#0 @0
Sat, Dec 2 2023 3:26:58 pm | 0: R1=ctx(id=0,off=0,imm=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | ; if (is_registers_abi)
Sat, Dec 2 2023 3:26:58 pm | 0: (18) r2 = 0xffff8c5811f750f8
Sat, Dec 2 2023 3:26:58 pm | 2: R1=ctx(id=0,off=0,imm=0) R2_w=map_value(id=0,off=40,ks=4,vs=64,imm=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 2: (71) r2 = *(u8 *)(r2 +0)
Sat, Dec 2 2023 3:26:58 pm | R1=ctx(id=0,off=0,imm=0) R2_w=map_value(id=0,off=40,ks=4,vs=64,imm=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 3: R1=ctx(id=0,off=0,imm=0) R2_w=inv1 R10=fp0
Sat, Dec 2 2023 3:26:58 pm | ; if (is_registers_abi)
Sat, Dec 2 2023 3:26:58 pm | 3: (15) if r2 == 0x0 goto pc+2
Sat, Dec 2 2023 3:26:58 pm | last_idx 3 first_idx 0
Sat, Dec 2 2023 3:26:58 pm | regs=4 stack=0 before 2: (71) r2 = *(u8 *)(r2 +0)
Sat, Dec 2 2023 3:26:58 pm | 4: R1=ctx(id=0,off=0,imm=0) R2_w=invP1 R10=fp0
Sat, Dec 2 2023 3:26:58 pm | ; return (void *)GO_PARAM2(ctx);
Sat, Dec 2 2023 3:26:58 pm | 4: (79) r7 = *(u64 *)(r1 +40)
Sat, Dec 2 2023 3:26:58 pm | 5: R1=ctx(id=0,off=0,imm=0) R2_w=invP1 R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 5: (05) goto pc+9
Sat, Dec 2 2023 3:26:58 pm | 15: R1=ctx(id=0,off=0,imm=0) R2_w=invP1 R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | ; bpf_probe_read(&header_fields, sizeof(header_fields), (void *)(frame_ptr + frame_fields_pos));
Sat, Dec 2 2023 3:26:58 pm | 15: (18) r1 = 0xffff8c5811f750e8
Sat, Dec 2 2023 3:26:58 pm | 17: R1_w=map_value(id=0,off=24,ks=4,vs=64,imm=0) R2_w=invP1 R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 17: (79) r1 = *(u64 *)(r1 +0)
Sat, Dec 2 2023 3:26:58 pm | R1_w=map_value(id=0,off=24,ks=4,vs=64,imm=0) R2_w=invP1 R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 18: R1_w=inv8 R2_w=invP1 R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | ; bpf_probe_read(&header_fields, sizeof(header_fields), (void *)(frame_ptr + frame_fields_pos));
Sat, Dec 2 2023 3:26:58 pm | 18: (bf) r3 = r7
Sat, Dec 2 2023 3:26:58 pm | 19: R1_w=inv8 R2_w=invP1 R3_w=inv(id=0) R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 19: (0f) r3 += r1
Sat, Dec 2 2023 3:26:58 pm | 20: R1_w=inv8 R2_w=invP1 R3_w=inv(id=0) R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | 20: (b7) r6 = 0
Sat, Dec 2 2023 3:26:58 pm | 21: R1_w=inv8 R2_w=invP1 R3_w=inv(id=0) R6_w=inv0 R7_w=inv(id=0) R10=fp0
Sat, Dec 2 2023 3:26:58 pm | ; struct go_slice header_fields = {};
Sat, Dec 2 2023 3:26:58 pm | 21: (7b) *(u64 *)(r10 -8) = r6
Sat, Dec 2 2023 3:26:58 pm | last_idx 21 first_idx 0
Sat, Dec 2 2023 3:26:58 pm | regs=40 stack=0 before 20: (b7) r6 = 0
Sat, Dec 2 2023 3:26:58 pm | 22: R1_w=inv8 R2_w=invP1 R3_w=inv(id=0) R6_w=invP0 R7_w=inv(id=0) R10=fp0 fp-8_w=00000000
Sat, Dec 2 2023 3:26:58 pm | 22: (7b) *(u64 *)(r10 -16) = r6
Sat, Dec 2 2023 3:26:58 pm | 23: R1_w=inv8 R2_w=invP1 R3_w=inv(id=0) R6_w=invP0 R7_w=inv(id=0) R10=fp0 fp-8_w=00000000 fp-16_w=00000000
Sat, Dec 2 2023 3:26:58 pm | 23: (7b) *(u64 *)(r10 -24) = r6
Sat, Dec 2 2023 3:26:58 pm | 24: R1_w=inv8 R2_w=invP1 R3_w=inv(id=0) R6_w=invP0 R7_w=inv(id=0) R10=fp0 fp-8_w=00000000 fp-16_w=00000000 fp-24_w=00000000
Sat, Dec 2 2023 3:26:58 pm | 24: (bf) r1 = r10
Sat, Dec 2 2023 3:26:58 pm | 25: R1_w=fp0 R2_w=invP1 R3_w=inv(id=0) R6_w=invP0 R7_w=inv(id=0) R10=fp0 fp-8_w=00000000 fp-16_w=00000000 fp-24_w=00000000
Sat, Dec 2 2023 3:26:58 pm | ;
Sat, Dec 2 2023 3:26:58 pm | 25: (07) r1 += -24
Sat, Dec 2 2023 3:26:58 pm | 26: R1_w=fp-24 R2_w=invP1 R3_w=inv(id=0) R6_w=invP0 R7_w=inv(id=0) R10=fp0 fp-8_w=00000000 fp-16_w=00000000 fp-24_w=00000000
Sat, Dec 2 2023 3:26:58 pm | ; bpf_probe_read(&header_fields, sizeof(header_fields), (void *)(frame_ptr + frame_fields_pos));

zsdyx · 2023-12-02T08:41:38Z

Is this error because my go.mod file version cannot be parsed，Can you tell me which one

func parseModules(mod string, logger logr.Logger) map[string]*version.Version {
	lines := strings.Split(mod, "\n")
	result := make(map[string]*version.Version)
	for _, line := range lines {
		parts := strings.Fields(line)
		if len(parts) > 1 {
			modType := parts[0]
			modPackage := parts[1]
			result[modPackage] = nil
			if modType == "dep" {
				if len(parts) < 3 {
					logger.Error(fmt.Errorf("missing version: %s", modPackage), "error parsing module info")
					continue
				}
				v, err := version.NewVersion(parts[2])
				if err != nil {
					logger.Error(err, "error parsing module info")
					continue
				}
				result[modPackage] = v
			}
		}
	}

	return result
}

zsdyx · 2023-12-04T09:53:26Z

Why won't this packet parse
{"level":"info","ts":1701682620.3500223,"logger":"Instrumentation.Analyzer","caller":"process/module.go:195","msg":"version detected---->","version":"v0.0.0-20230706204954-ccb25ca9f130","package":"google.golang.org/genproto","type":"dep"}

RonFed · 2023-12-09T08:55:41Z

Hi @zsdyx I don't think the error you get is a result of the parsing.
I updated the way we print the verifier log to print its tail.
Can you please re-run this with the latest main branch and with OTEL_GO_AUTO_SHOW_VERIFIER_LOG=true and attach the log ?
(I can't reproduce this issue on my side)

vbavireddi · 2024-01-07T18:51:21Z

Hi @RonFed
I also have similar issue

Describe the bug

I'm running my app with the instrumentation in EKS, I have enabled the feature gate operator.autoinstrumentation.go.

{"level":"error","ts":1704652636.7818418,"logger":"Instrumentation.Analyzer","caller":"process/module.go:195","msg":"error parsing module info","error":"Malformed version: (devel)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/process.parseModules\n\t/app/internal/pkg/process/module.go:195\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).getModuleDetails\n\t/app/internal/pkg/process/module.go:50\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).Analyze\n\t/app/internal/pkg/process/analyze.go:83\ngo.opentelemetry.io/auto.NewInstrumentation\n\t/app/instrumentation.go:120\nmain.main\n\t/app/cli/main.go:92\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}

Environment

OS: Amazon linux 2
Go Version: 1.18
Version: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.8.0-alpha

To Reproduce

Steps to reproduce the behavior:
Instrumentation config:
go:
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: >-
http://simplest-collector.opentelemetry-operator-system.svc.cluster.local:4318
- name: OTEL_TRACES_SAMPLER
value: always_on
- name: OTEL_RESOURCE_ATTRIBUTES
value: env=dev
- name: OTEL_METRICS_EXPORTER
value: prometheus
- name: OTEL_GO_AUTO_SHOW_VERIFIER_LOG
value: 'true'
image: >-
ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.8.0-alpha
resourceRequirements:
limits:
cpu: 500m
requests:
cpu: 50m
memory: 32Mi

Deployment annotations:
annotations:
instrumentation.opentelemetry.io/inject-go: 'true'
instrumentation.opentelemetry.io/otel-go-auto-target-exe: /main

opentelemetry-auto-instrumentation container restarting with exit 0 and with below error.

{"level":"error","ts":1704652636.7818418,"logger":"Instrumentation.Analyzer","caller":"process/module.go:195","msg":"error parsing module info","error":"Malformed version: (devel)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/process.parseModules\n\t/app/internal/pkg/process/module.go:195\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).getModuleDetails\n\t/app/internal/pkg/process/module.go:50\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).Analyze\n\t/app/internal/pkg/process/analyze.go:83\ngo.opentelemetry.io/auto.NewInstrumentation\n\t/app/instrumentation.go:120\nmain.main\n\t/app/cli/main.go:92\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
....
...
...
{"level":"info","ts":1704652637.253412,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:163","msg":"loading probes","name":"google.golang.org/grpc/server"}
Verifier log: load program: no space left on device:
	func#0 @0
	0: R1=ctx(id=0,off=0,imm=0) R10=fp0
	; if (is_registers_abi)
	0: (18) r2 = 0xffff888473a92af8
	2: R1=ctx(id=0,off=0,imm=0) R2_w=map_value(id=0,off=40,ks=4,vs=64,imm=0) R10=fp0
	2: (71) r2 = *(u8 *)(r2 +0)
	 R1=ctx(id=0,off=0,imm=0) R2_w=map_value(id=0,off=40,ks=4,vs=64,imm=0) R10=fp0
	3: R1=ctx(id=0,off=0,imm=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0
	; if (is_registers_abi)
	3: (15) if r2 == 0x0 goto pc+2
	 R1=ctx(id=0,off=0,imm=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0
	4: R1=ctx(id=0,off=0,imm=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0
	; return (void *)GO_PARAM2(ctx);
	4: (79) r7 = *(u64 *)(r1 +40)
	5: R1=ctx(id=0,off=0,imm=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R7_w=inv(id=0) R10=fp0
	5: (05) goto pc+9
	15: R1=ctx(id=0,off=0,imm=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R7_w=inv(id=0) R10=fp0
	; bpf_probe_read(&header_fields, sizeof(header_fields), (void *)(frame_ptr + frame_fields_pos));
	15: (18) r1 = 0xffff888473a92ae8

Container details:

/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                 100.0G     49.8G     50.2G  50% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                    30.9G         0     30.9G   0% /sys/fs/cgroup
/dev/nvme0n1p1          100.0G     49.8G     50.2G  50% /configMap
shm                      64.0M         0     64.0M   0% /dev/shm
/dev/nvme0n1p1          100.0G     49.8G     50.2G  50% /etc/hosts
/dev/nvme0n1p1          100.0G     49.8G     50.2G  50% /dev/termination-log
/dev/nvme0n1p1          100.0G     49.8G     50.2G  50% /etc/hostname
/dev/nvme0n1p1          100.0G     49.8G     50.2G  50% /etc/resolv.conf
tmpfs                    58.7G      4.0K     58.7G   0% /run/secrets/eks.amazonaws.com/serviceaccount
tmpfs                    58.7G     12.0K     58.7G   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                    30.9G         0     30.9G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/latency_stats
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                    30.9G         0     30.9G   0% /sys/firmware
/ # ps aux
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    7 root      8:18 ./main
  186 root      0:00 sh -c clear; (bash || ash || sh)
  193 root      0:00 ash
  256 root      0:00 sh -c clear; (bash || ash || sh)
  263 root      0:00 ash
  266 root      0:00 ps aux
/ # 

/ # uname -a
Linux <app-name>-588b7bfb48-l4qzn 5.4.249-163.359.amzn2.x86_64 #1 SMP Wed Jul 12 18:58:58 UTC 2023 x86_64 Linux

RonFed · 2024-01-07T19:15:41Z

@vbavireddi please try with the latest version https://github.com/open-telemetry/opentelemetry-go-instrumentation/releases/tag/v0.10.0-alpha
There were few PRs regarding the gRPC server instrumentation which seems to be the problem in the log you attached.

vbavireddi · 2024-01-07T20:23:34Z

@RonFed Thanks for the update.
I've updated the version. Still the container restarting continuously.

image: >-
ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.10.0-alpha

Error log:

{"level":"error","ts":1704658707.8408637,"logger":"Instrumentation.Analyzer","caller":"process/module.go:195","msg":"error parsing module info","error":"Malformed version: (devel)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/process.parseModules\n\t/app/internal/pkg/process/module.go:195\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).getModuleDetails\n\t/app/internal/pkg/process/module.go:50\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).Analyze\n\t/app/internal/pkg/process/analyze.go:83\ngo.opentelemetry.io/auto.NewInstrumentation\n\t/app/instrumentation.go:122\nmain.main\n\t/app/cli/main.go:101\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
...
...
{"level":"info","ts":1704658707.8676953,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:107","msg":"no functions found for probe, removing","name":"github.com/gin-gonic/gin"} {"level":"info","ts":1704658707.8677032,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:107","msg":"starting instrumentation..."} {"level":"info","ts":1704658707.8677793,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:177","msg":"Mounting bpffs","allocations_details":{"StartAddr":139637300117504,"EndAddr":139637301166080}} {"level":"info","ts":1704658707.8678958,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:162","msg":"loading probe","name":"google.golang.org/grpc/client"} {"level":"info","ts":1704658708.169515,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:162","msg":"loading probe","name":"google.golang.org/grpc/server"} Verifier log: load program: no space left on device: (14227 line(s) omitted) {"level":"error","ts":1704658708.521125,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:165","msg":"error while loading probes, cleaning up","name":"google.golang.org/grpc/server","error":"field UprobeHttp2ServerOperateHeader: program uprobe_http2Server_operateHeader: load program: no space left on device: 54: (67) r3 <<= 32: 55: R0=inv(id=0) R1_w=fp-122 R2_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R3_w=inv(id=0,smax_value=9223372032559808512,umax_value=18446744069414584320, (truncated, 14325 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:165\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:120\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:152\nmain.main\n\t/app/cli/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"} 49: (71) r3 = *(u8 *)(r1 +0) 50: R0=inv(id=0) R1_w=fp-124 R2_w=inv(id=0) R3_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R4_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R5_w=inv(id=0,smax_value=9223372036854775792,umax_value=18446744073709551600,var_off=(0x0; 0xfffffffffffffff0)) R6_w=invP1 R7=inv(id=0) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=0000000m fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=mmmmmmmm fp-384=mmmmmmmm fp-392=mmmmmmmm fp-400=mmmmmmmm fp-408=mmmmmmmm fp-416=mmmmmmmm fp-424=fp fp-432=fp 50: (67) r3 <<= 56 51: R0=inv(id=0) R1_w=fp-124 R2_w=inv(id=0) R3_w=inv(id=0,smax_value=9151314442816847872,umax_value=18374686479671623680,var_off=(0x0; 0xff00000000000000)) R4_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R5_w=inv(id=0,smax_value=9223372036854775792,umax_value=18446744073709551600,var_off=(0x0; 0xfffffffffffffff0)) R6_w=invP1 R7=inv(id=0) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=0000000m fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=mmmmmmmm fp-384=mmmmmmmm fp-392=mmmmmmmm fp-400=mmmmmmmm fp-408=mmmmmmmm fp-416=mmmmmmmm fp-424=fp fp-432=fp 51: (c7) r3 s>>= 56

vbavireddi · 2024-01-08T07:32:50Z

@RonFed for restapi ("name":"net/http/server") also we are getting similar error.

Version:
/app # uname -a Linux <cutom-image> 5.4.249-163.359.amzn2.x86_64 #1 SMP Wed Jul 12 18:58:58 UTC 2023 x86_64 Linux
Base img: golang:1.20.5-alpine3.18
ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.10.0-alpha

Error:
`{"level":"info","ts":1704698393.3934438,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:94","msg":"building OpenTelemetry Go instrumentation ...","globalImpl":false}
{"level":"info","ts":1704698395.394087,"logger":"Instrumentation.Analyzer","caller":"process/discover.go:69","msg":"found process","pid":7}
{"level":"error","ts":1704698395.394404,"logger":"Instrumentation.Analyzer","caller":"process/module.go:195","msg":"error parsing module info","error":"Malformed version: (devel)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/process.parseModules\n\t/app/internal/pkg/process/module.go:195\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).getModuleDetails\n\t/app/internal/pkg/process/module.go:50\ngo.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).Analyze\n\t/app/internal/pkg/process/analyze.go:83\ngo.opentelemetry.io/auto.NewInstrumentation\n\t/app/instrumentation.go:122\nmain.main\n\t/app/cli/main.go:101\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"info","ts":1704698395.3945549,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:113","msg":"No symbols found in binary, trying to find functions using .gosymtab"}
{"level":"info","ts":1704698395.4090765,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"net/http.serverHandler.ServeHTTP","Offset":4352192,"ReturnOffsets":[4353032,4353076]}}
{"level":"info","ts":1704698395.4091427,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"net/http.(*Transport).roundTrip","Offset":4404448,"ReturnOffsets":[4404956,4405034,4405520,4405757,4405851,4406026,4406044,4406790,4406877,4406945,4407040,4407118,4407186,4407612,4407828]}}
{"level":"info","ts":1704698395.4091587,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*loopyWriter).headerHandler","Offset":4749184,"ReturnOffsets":[4749500,4749544,4749573,4749614,4749731,4749873]}}
{"level":"info","ts":1704698395.409165,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*http2Client).NewStream","Offset":4804128,"ReturnOffsets":[4804596,4805751,4805837,4805959,4806061,4806551,4806669]}}
{"level":"info","ts":1704698395.409169,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders","Offset":4844608,"ReturnOffsets":[4845501,4845635,4852641,4853281,4853457,4853558,4853793,4854129,4854263,4854977,4856648,4856894]}}
{"level":"info","ts":1704698395.409186,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(*ClientConn).Invoke","Offset":5251584,"ReturnOffsets":[5252146,5252212]}}
{"level":"info","ts":1704698395.4091961,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(*Server).handleStream","Offset":5358112,"ReturnOffsets":[5359252,5360544,5360602,5360672,5360741]}}
{"level":"info","ts":1704698395.4092786,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":13}
{"level":"info","ts":1704698395.4093184,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":14}
{"level":"info","ts":1704698395.4093506,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":15}
{"level":"info","ts":1704698395.409393,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":16}
{"level":"info","ts":1704698395.4094226,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":17}
{"level":"info","ts":1704698395.4094503,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":18}
{"level":"info","ts":1704698395.409479,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":19}
{"level":"info","ts":1704698395.4095564,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":20}
{"level":"info","ts":1704698395.4095926,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":21}
{"level":"info","ts":1704698395.4096258,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":22}
{"level":"info","ts":1704698395.4096704,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":23}
{"level":"info","ts":1704698395.409714,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":24}
{"level":"info","ts":1704698395.4097433,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":44}
{"level":"info","ts":1704698395.409792,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":45}
{"level":"info","ts":1704698395.4098186,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":7}
{"level":"info","ts":1704698395.4102046,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:234","msg":"Detected linux kernel version","version":"5.4.0"}
{"level":"info","ts":1704698395.4103284,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:246","msg":"mlock ret","ret":0}
{"level":"info","ts":1704698395.4103427,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:64","msg":"Detaching from process","pid":7}
{"level":"info","ts":1704698395.410408,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:43","msg":"mmaped remote memory","start_addr":"7F7CB0D73000","end_addr":"7F7CB0E73000"}
{"level":"info","ts":1704698395.4104197,"logger":"Instrumentation","caller":"app/instrumentation.go:134","msg":"target process analysis completed","pid":7,"go_version":"1.20.5","dependencies":{"-buildmode=exe":null,"-compiler=gc":null,"-ldflags="-s":null,"-tags=musl":null,"CGO_CFLAGS=":null,"CGO_CPPFLAGS=":null,"CGO_CXXFLAGS=":null,"CGO_ENABLED=1":null,"CGO_LDFLAGS=":null,"GOAMD64=v1":null,"GOARCH=amd64":null,"GOOS=linux":null,"command-line-arguments":null,"github.com/ZscalerCWP/cwp-gloo-custom-auth-server":null,"github.com/ZscalerCWP/zpc-api-key-management":"1.0.1","github.com/ZscalerCWP/zpc-common/cache":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-common/http/middlewares/error":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-common/http/writer":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-common/prometheus":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-common/tracing":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-common/viper":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-common/zerolog":"0.0.0-20230531135819-02ed9b0e7fef","github.com/ZscalerCWP/zpc-framework":"1.0.6","github.com/alecthomas/kong":"0.7.1","github.com/auth0/go-jwt-middleware/v2":"2.1.0","github.com/beorn7/perks":"1.0.1","github.com/cenkalti/backoff/v4":"4.2.0","github.com/cespare/xxhash/v2":"2.2.0","github.com/dgryski/go-rendezvous":"0.0.0-20200823014737-9f7001d12a5f","github.com/fsnotify/fsnotify":"1.6.0","github.com/go-logr/logr":"1.2.3","github.com/go-logr/stdr":"1.2.2","github.com/golang/protobuf":"1.5.3","github.com/google/uuid":"1.3.0","github.com/grpc-ecosystem/grpc-gateway/v2":"2.16.0","github.com/hashicorp/hcl":"1.0.0","github.com/labstack/echo/v4":"4.10.2","github.com/labstack/gommon":"0.4.0","github.com/magiconair/properties":"1.8.7","github.com/mattn/go-colorable":"0.1.13","github.com/mattn/go-isatty":"0.0.17","github.com/matttproud/golang_protobuf_extensions":"1.0.4","github.com/mitchellh/mapstructure":"1.5.0","github.com/openzipkin/zipkin-go":"0.4.1","github.com/pelletier/go-toml/v2":"2.0.7","github.com/pkg/errors":"0.9.1","github.com/prometheus/client_golang":"1.14.0","github.com/prometheus/client_model":"0.3.0","github.com/prometheus/common":"0.41.0","github.com/prometheus/procfs":"0.9.0","github.com/redis/go-redis/extra/rediscmd/v9":"9.0.2","github.com/redis/go-redis/v9":"9.0.2","github.com/rs/zerolog":"1.29.0","github.com/spf13/afero":"1.9.5","github.com/spf13/cast":"1.5.1","github.com/spf13/jwalterweatherman":"1.1.0","github.com/spf13/pflag":"1.0.5","github.com/spf13/viper":"1.15.0","github.com/subosito/gotenv":"1.4.2","github.com/valyala/bytebufferpool":"1.0.0","github.com/valyala/fasttemplate":"1.2.2","go.opentelemetry.io/otel":"1.14.0","go.opentelemetry.io/otel/exporters/jaeger":"1.14.0","go.opentelemetry.io/otel/exporters/otlp/internal/retry":"1.14.0","go.opentelemetry.io/otel/exporters/otlp/otlptrace":"1.14.0","go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc":"1.14.0","go.opentelemetry.io/otel/exporters/stdout/stdouttrace":"1.14.0","go.opentelemetry.io/otel/exporters/zipkin":"1.14.0","go.opentelemetry.io/otel/sdk":"1.14.0","go.opentelemetry.io/otel/trace":"1.14.0","go.opentelemetry.io/proto/otlp":"0.19.0","go.uber.org/atomic":"1.11.0","go.uber.org/dig":"1.16.1","go.uber.org/fx":"1.19.3","go.uber.org/multierr":"1.11.0","go.uber.org/zap":"1.24.0","golang.org/x/crypto":"0.9.0","golang.org/x/exp":"0.0.0-20230522175609-2e198f4a06a1","golang.org/x/net":"0.10.0","golang.org/x/sys":"0.8.0","golang.org/x/text":"0.9.0","google.golang.org/genproto/googleapis/api":"0.0.0-20230530153820-e85fd2cbaebc","google.golang.org/genproto/googleapis/rpc":"0.0.0-20230530153820-e85fd2cbaebc","google.golang.org/grpc":"1.55.0","google.golang.org/protobuf":"1.30.0","gopkg.in/go-jose/go-jose.v2":"2.6.1","gopkg.in/ini.v1":"1.67.0","gopkg.in/square/go-jose.v2":"2.6.0","gopkg.in/yaml.v3":"3.0.1","std":"1.20.5"},"total_functions_found":7}
{"level":"info","ts":1704698395.4105577,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:107","msg":"no functions found for probe, removing","name":"github.com/gin-gonic/gin"}
{"level":"info","ts":1704698395.410568,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:107","msg":"no functions found for probe, removing","name":"database/sql"}
{"level":"info","ts":1704698395.410572,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:107","msg":"starting instrumentation..."}
{"level":"info","ts":1704698395.4106522,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:177","msg":"Mounting bpffs","allocations_details":{"StartAddr":140173519564800,"EndAddr":140173520613376}}
{"level":"info","ts":1704698395.4108272,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:162","msg":"loading probe","name":"net/http/server"}
{"level":"error","ts":1704698395.7103634,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:165","msg":"error while loading probes, cleaning up","name":"net/http/server","error":"field UprobeHandlerFuncServeHTTP: program uprobe_HandlerFunc_ServeHTTP: load program: no space left on device: 496: (71) r1 = *(u8 *)(r10 -45): 497: R0=in (truncated, 32578 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:165\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:120\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:152\nmain.main\n\t/app/cli/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"info","ts":1704698395.710505,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:190","msg":"Cleaning bpffs"}
{"level":"error","ts":1704698395.710586,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:109","msg":"instrumentation crashed","error":"field UprobeHandlerFuncServeHTTP: program uprobe_HandlerFunc_ServeHTTP: load program: no space left on device: 496: (71) r1 = *(u8 *)(r10 -45): 497: R0=in (truncated, 32578 line(s) omitted)","stacktrace":"main.main\n\t/app/cli/main.go:109\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
Verifier log: load program: no space left on device:
(32480 line(s) omitted)
propagating r8
propagating fp-176

from 497 to 467: safe
propagating r8
propagating fp-176

from 495 to 467: safe
propagating r8
propagating fp-176

from 493 to 467: safe
propagating r8
propagating fp-176

from 492 to 515: safe
propagating r8
propagating fp-176

from 485 to 468: safe
propagating r8
propagating fp-176

from 483 to 468: safe
propagating r4
last_idx 478 first_idx 535
regs=10 stack=0 before 541: (6d) if r4 s> r0 goto pc-64
regs=11 stack=0 before 540: (b7) r4 = 0
regs=1 stack=0 before 539: (85) call bpf_probe_read#4

from 541 to 478: safe
467: R0=inv(id=0) R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6=map_value(id=0,off=0,ks=4,vs=736,imm=0) R7=map_value(id=0,off=208,ks=4,vs=336,imm=0) R8=invP3 R9=map_value(id=0,off=64,ks=4,vs=336,imm=0) R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=m??????? fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmm? fp-64=mmmmmmmm fp-72=mmmmmmmm fp-144=mmmmmmmm fp-152=mmmm???? fp-160=map_value fp-168=mmmmmmmm fp-176=invP3 fp-184=mmmmmmmm fp-192=map_value fp-200=map_value
;
467: (79) r2 = *(u64 *)(r10 -160)
468: R0=inv(id=0) R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=map_value(id=0,off=0,ks=4,vs=336,imm=0) R6=map_value(id=0,off=0,ks=4,vs=736,imm=0) R7=map_value(id=0,off=208,ks=4,vs=336,imm=0) R8=invP3 R9=map_value(id=0,off=64,ks=4,vs=336,imm=0) R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=m??????? fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmm? fp-64=mmmmmmmm fp-72=mmmmmmmm fp-144=mmmmmmmm fp-152=mmmm???? fp-160=map_value fp-168=mmmmmmmm fp-176=invP3 fp-184=mmmmmmmm fp-192=map_value fp-200=map_value
; for (u64 i = 0; i < 8; i++)`

@vasiliy-grinko Can you please share if you find any workaround for this issue.

Co0ker · 2024-04-02T13:34:46Z

Hi!
Any updates on this issue?
I tested the latest version (v0.11.0-alpha) and got a similar error:

{"level":"info","ts":1712064476.9977193,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:94","msg":"building OpenTelemetry Go instrumentation ...","globalImpl":false}
{"level":"info","ts":1712064479.021917,"logger":"Instrumentation.Analyzer","caller":"process/discover.go:77","msg":"found process","pid":13}
{"level":"error","ts":1712064479.0235422,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"error parsing module version","error":"Malformed version: (devel)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).Analyze\n\t/app/internal/pkg/process/analyze.go:98\ngo.opentelemetry.io/auto.NewInstrumentation\n\t/app/instrumentation.go:122\nmain.main\n\t/app/cli/main.go:101\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}
{"level":"info","ts":1712064479.2411113,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"net/http.serverHandler.ServeHTTP","Offset":3181888,"ReturnOffsets":[3182728,3182772]}}
{"level":"info","ts":1712064479.2412572,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"net/http.(*Transport).roundTrip","Offset":3234144,"ReturnOffsets":[3234652,3234730,3235216,3235453,3235547,3235722,3235740,3236486,3236573,3236641,3236736,3236814,3236882,3237308,3237524]}}
{"level":"info","ts":1712064479.2412677,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*loopyWriter).headerHandler","Offset":7689856,"ReturnOffsets":[7690172,7690216,7690245,7690286,7690451,7690791]}}
{"level":"info","ts":1712064479.2412744,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*http2Client).NewStream","Offset":7740896,"ReturnOffsets":[7741268,7742406,7742491,7742617,7742720,7743193,7743221]}}
{"level":"info","ts":1712064479.2412806,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders","Offset":7780576,"ReturnOffsets":[7781436,7781555,7787818,7788481,7788642,7788728,7788848,7789584,7791129,7791569]}}
{"level":"info","ts":1712064479.2412875,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(*ClientConn).Invoke","Offset":8194752,"ReturnOffsets":[8195314,8195380]}}
{"level":"info","ts":1712064479.241293,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:110","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(*Server).handleStream","Offset":8298528,"ReturnOffsets":[8299668,8300960,8301018,8301088,8301157]}}
{"level":"info","ts":1712064479.2414215,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:51","msg":"Requesting memory allocation","size":262144,"page size":4096,"cpu count":8}
{"level":"info","ts":1712064479.2416844,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":13}
{"level":"info","ts":1712064479.2417872,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":14}
{"level":"info","ts":1712064479.3204803,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":15}
{"level":"info","ts":1712064479.3207407,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":16}
{"level":"info","ts":1712064479.3207974,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":17}
{"level":"info","ts":1712064479.3208456,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":18}
{"level":"info","ts":1712064479.3209286,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":19}
{"level":"info","ts":1712064479.3211067,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":20}
{"level":"info","ts":1712064479.3211799,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":21}
{"level":"info","ts":1712064479.321278,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":22}
{"level":"info","ts":1712064479.3213801,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":23}
{"level":"info","ts":1712064479.3214848,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":24}
{"level":"info","ts":1712064479.3215709,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:138","msg":"attach successfully","tid":25}
{"level":"info","ts":1712064479.321801,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:94","msg":"Set memlock on process successfully"}
{"level":"info","ts":1712064479.3221786,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:250","msg":"Detected linux kernel version","version":"5.15.0"}
{"level":"info","ts":1712064479.322465,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:262","msg":"mlock ret","ret":0}
{"level":"info","ts":1712064479.322523,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:84","msg":"Detaching from process","pid":13}
{"level":"info","ts":1712064479.3226507,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:62","msg":"mmaped remote memory","start_addr":"0x7fcae8da0000","end_addr":"0x7fcae8de0000"}
{"level":"info","ts":1712064479.322679,"logger":"Instrumentation","caller":"app/instrumentation.go:133","msg":"target process analysis completed","pid":13,"go_version":"1.20.14","dependencies":{"cloud.google.com/go":"0.97.0","github.com/Azure/azure-sdk-for-go":"32.4.0+incompatible","github.com/Azure/go-autorest/autorest":"0.11.27","github.com/Azure/go-autorest/autorest/adal":"0.9.20","github.com/Azure/go-autorest/autorest/azure/auth":"0.5.11","github.com/Azure/go-autorest/autorest/azure/cli":"0.4.5","github.com/Azure/go-autorest/autorest/date":"0.3.0","github.com/Azure/go-autorest/autorest/to":"0.4.0","github.com/Azure/go-autorest/autorest/validation":"0.3.1","github.com/Azure/go-autorest/logger":"0.2.1","github.com/Azure/go-autorest/tracing":"0.6.0","github.com/OpenDNS/vegadns2client":"0.0.0-20180418235048-a3fa4a771d87","github.com/PuerkitoBio/purell":"1.1.1","github.com/PuerkitoBio/urlesc":"0.0.0-20170810143723-de5bf2ad4578","github.com/akamai/AkamaiOPEN-edgegrid-golang":"1.2.1","github.com/aliyun/alibaba-cloud-sdk-go":"1.61.1755","github.com/andres-erbsen/clock":"0.0.0-20160526145045-9e14626cd129","github.com/armon/go-metrics":"0.3.9","github.com/armon/go-radix":"1.0.0","github.com/aws/aws-sdk-go":"1.39.0","github.com/beorn7/perks":"1.0.1","github.com/boombuler/barcode":"1.0.1-0.20190219062509-6c824513bacc","github.com/cenkalti/backoff/v3":"3.0.0","github.com/cenkalti/backoff/v4":"4.1.3","github.com/census-instrumentation/opencensus-proto":"0.3.0","github.com/cespare/xxhash/v2":"2.1.2","github.com/civo/civogo":"0.3.11","github.com/cloudflare/cloudflare-go":"0.49.0","github.com/cncf/xds/go":"0.0.0-20220314180256-7f1daf1720fc","github.com/cpu/goacmedns":"0.1.1","github.com/davecgh/go-spew":"1.1.1","github.com/deepmap/oapi-codegen":"1.9.1","github.com/dimchansky/utfbom":"1.1.1","github.com/dnsimple/dnsimple-go":"0.71.1","github.com/emicklei/go-restful/v3":"3.8.0","github.com/envoyproxy/go-control-plane":"0.10.3","github.com/envoyproxy/protoc-gen-validate":"0.6.7","github.com/evanphx/json-patch/v5":"5.6.0","github.com/exoscale/egoscale":"0.90.0","github.com/fatih/color":"1.13.0","github.com/fatih/structs":"1.1.0","github.com/fsnotify/fsnotify":"1.5.4","github.com/ghodss/yaml":"1.0.0","github.com/go-acme/lego/v4":"4.9.0","github.com/go-errors/errors":"1.0.1","github.com/go-logr/logr":"1.2.3","github.com/go-logr/zapr":"1.2.3","github.com/go-openapi/jsonpointer":"0.19.5","github.com/go-openapi/jsonreference":"0.19.5","github.com/go-openapi/swag":"0.19.14","github.com/go-resty/resty/v2":"2.1.1-0.20191201195748-d7b97669fe48","github.com/gogo/protobuf":"1.3.2","github.com/golang-jwt/jwt/v4":"4.2.0","github.com/golang/groupcache":"0.0.0-20210331224755-41bb18bfe9da","github.com/golang/protobuf":"1.5.2","github.com/golang/snappy":"0.0.4","github.com/google/gnostic":"0.5.7-v3refs","github.com/google/go-cmp":"0.5.8","github.com/google/go-querystring":"1.1.0","github.com/google/gofuzz":"1.1.0","github.com/google/uuid":"1.3.0","github.com/googleapis/gax-go/v2":"2.1.0","github.com/gophercloud/gophercloud":"1.0.0","github.com/gophercloud/utils":"0.0.0-20210216074907-f6de111f2eae","github.com/hashicorp/errwrap":"1.1.0","github.com/hashicorp/go-cleanhttp":"0.5.2","github.com/hashicorp/go-hclog":"1.2.0","github.com/hashicorp/go-immutable-radix":"1.3.1","github.com/hashicorp/go-multierror":"1.1.1","github.com/hashicorp/go-plugin":"1.4.3","github.com/hashicorp/go-retryablehttp":"0.7.1","github.com/hashicorp/go-rootcerts":"1.0.2","github.com/hashicorp/go-secure-stdlib/mlock":"0.1.1","github.com/hashicorp/go-secure-stdlib/parseutil":"0.1.6","github.com/hashicorp/go-secure-stdlib/strutil":"0.1.2","github.com/hashicorp/go-sockaddr":"1.0.2","github.com/hashicorp/go-uuid":"1.0.2","github.com/hashicorp/go-version":"1.6.0","github.com/hashicorp/golang-lru":"0.5.4","github.com/hashicorp/hcl":"1.0.0","github.com/hashicorp/vault/api":"1.8.0","github.com/hashicorp/vault/sdk":"0.6.0","github.com/hashicorp/yamux":"0.0.0-20180604194846-3520598351bb","github.com/iij/doapi":"0.0.0-20190504054126-0bbf12d6d7df","github.com/imdario/mergo":"0.3.12","github.com/infobloxopen/infoblox-go-client":"1.1.1","github.com/jmespath/go-jmespath":"0.4.0","github.com/josharian/intern":"1.0.0","github.com/json-iterator/go":"1.1.12","github.com/k0kubun/go-ansi":"0.0.0-20180517002512-3bf9e2903213","github.com/kolo/xmlrpc":"0.0.0-20200310150728-e0350524596b","github.com/labbsr0x/bindman-dns-webhook":"1.0.2","github.com/labbsr0x/goh":"1.0.1","github.com/linode/linodego":"1.9.1","github.com/liquidweb/go-lwApi":"0.0.5","github.com/liquidweb/liquidweb-cli":"0.6.9","github.com/liquidweb/liquidweb-go":"1.6.3","github.com/mailru/easyjson":"0.7.6","github.com/mattn/go-colorable":"0.1.12","github.com/mattn/go-isatty":"0.0.16","github.com/matttproud/golang_protobuf_extensions":"1.0.2-0.20181231171920-c182affec369","github.com/miekg/dns":"1.1.50","github.com/mimuret/golang-iij-dpf":"0.7.1","github.com/mitchellh/copystructure":"1.0.0","github.com/mitchellh/go-homedir":"1.1.0","github.com/mitchellh/go-testing-interface":"1.0.0","github.com/mitchellh/mapstructure":"1.5.0","github.com/mitchellh/reflectwalk":"1.0.0","github.com/modern-go/concurrent":"0.0.0-20180306012644-bacd9c7ef1dd","github.com/modern-go/reflect2":"1.0.2","github.com/munnerz/goautoneg":"0.0.0-20191010083416-a7dc8b61c822","github.com/namedotcom/go":"0.0.0-20180403034216-08470befbe04","github.com/nrdcg/auroradns":"1.1.0","github.com/nrdcg/desec":"0.6.0","github.com/nrdcg/dnspod-go":"0.4.0","github.com/nrdcg/freemyip":"0.2.0","github.com/nrdcg/goinwx":"0.8.1","github.com/nrdcg/namesilo":"0.2.1","github.com/nrdcg/porkbun":"0.1.1","github.com/oklog/run":"1.0.0","github.com/oracle/oci-go-sdk":"24.3.0+incompatible","github.com/ovh/go-ovh":"1.1.0","github.com/patrickmn/go-cache":"2.1.0+incompatible","github.com/pierrec/lz4":"2.5.2+incompatible","github.com/pkg/errors":"0.9.1","github.com/pmezard/go-difflib":"1.0.0","github.com/pquerna/otp":"1.3.0","github.com/prometheus/client_golang":"1.13.0","github.com/prometheus/client_model":"0.2.0","github.com/prometheus/common":"0.37.0","github.com/prometheus/procfs":"0.8.0","github.com/ryanuber/go-glob":"1.0.0","github.com/sacloud/api-client-go":"0.2.1","github.com/sacloud/go-http":"0.1.2","github.com/sacloud/iaas-api-go":"1.3.2","github.com/sacloud/packages-go":"0.0.5","github.com/scaleway/scaleway-sdk-go":"1.0.0-beta.9","github.com/sirupsen/logrus":"1.8.1","github.com/smartystreets/go-aws-auth":"0.0.0-20180515143844-0c1422d1fdb9","github.com/softlayer/softlayer-go":"1.0.6","github.com/softlayer/xmlrpc":"0.0.0-20200409220501-5f089df7cb7e","github.com/spf13/cast":"1.3.1","github.com/spf13/pflag":"1.0.5","github.com/stretchr/objx":"0.4.0","github.com/stretchr/testify":"1.8.0","github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common":"1.0.490","github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/dnspod":"1.0.490","github.com/transip/gotransip/v6":"6.17.0","github.com/vinyldns/go-vinyldns":"0.9.16","github.com/vultr/govultr/v2":"2.17.2","github.com/yandex-cloud/go-genproto":"0.0.0-20220805142335-27b56ddae16f","github.com/yandex-cloud/go-sdk":"0.0.0-20220805164847-cf028e604997","go.opencensus.io":"0.23.0","go.uber.org/atomic":"1.9.0","go.uber.org/multierr":"1.6.0","go.uber.org/ratelimit":"0.2.0","go.uber.org/zap":"1.21.0","golang.org/x/crypto":"0.1.0","golang.org/x/net":"0.1.0","golang.org/x/oauth2":"0.0.0-20220909003341-f21342109be1","golang.org/x/sys":"0.1.0","golang.org/x/term":"0.1.0","golang.org/x/text":"0.4.0","golang.org/x/time":"0.0.0-20220609170525-579cf78fd858","gomodules.xyz/jsonpatch/v2":"2.2.0","google.golang.org/api":"0.57.0","google.golang.org/genproto":"0.0.0-20220502173005-c8bf987b8c21","google.golang.org/grpc":"1.49.0","google.golang.org/protobuf":"1.28.1","gopkg.in/inf.v0":"0.9.1","gopkg.in/ini.v1":"1.66.6","gopkg.in/ns1/ns1-go.v2":"2.6.5","gopkg.in/square/go-jose.v2":"2.6.0","gopkg.in/yaml.v2":"2.4.0","gopkg.in/yaml.v3":"3.0.1","k8s.io/api":"0.25.2","k8s.io/apiextensions-apiserver":"0.25.0","k8s.io/apimachinery":"0.25.2","k8s.io/client-go":"0.25.2","k8s.io/component-base":"0.25.0","k8s.io/klog/v2":"2.70.1","k8s.io/kube-openapi":"0.0.0-20220803162953-67bda5d908f1","k8s.io/utils":"0.0.0-20220728103510-ee6ede2d64ed","sigs.k8s.io/controller-runtime":"0.13.0","sigs.k8s.io/json":"0.0.0-20220713155537-f223a00ba0e2","sigs.k8s.io/structured-merge-diff/v4":"4.2.3","sigs.k8s.io/yaml":"1.3.0","std":"1.20.14"},"total_functions_found":7}
{"level":"info","ts":1712064479.323352,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:112","msg":"no functions found for probe, removing","name":"github.com/gin-gonic/gin/server"}
{"level":"info","ts":1712064479.3234062,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:112","msg":"no functions found for probe, removing","name":"database/sql/client"}
{"level":"info","ts":1712064479.323424,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:107","msg":"starting instrumentation..."}
{"level":"info","ts":1712064479.323667,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:190","msg":"Mounting bpffs","allocations_details":{"StartAddr":140509466722304,"EndAddr":140509466984448,"NumCPU":8}}
{"level":"info","ts":1712064479.325669,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:176","msg":"loading probe","name":"google.golang.org/grpc/client"}
{"level":"info","ts":1712064479.964876,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:176","msg":"loading probe","name":"google.golang.org/grpc/server"}
Verifier log: load program: no space left on device:
	(14109 line(s) omitted)
	428: (7b) *(u64 *)(r10 -200) = r6
	429: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=mmmmmmmm fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	429: (7b) *(u64 *)(r10 -208) = r6
	430: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216=mmmmmmmm fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=mmmmmmmm fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	430: (7b) *(u64 *)(r10 -216) = r6
	431: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=mmmmmmmm fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	431: (7b) *(u64 *)(r10 -224) = r6
	432: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232=mmmmmmmm fp-240=mmmmmmmm fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	432: (7b) *(u64 *)(r10 -232) = r6
	433: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240=mmmmmmmm fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	433: (7b) *(u64 *)(r10 -240) = r6
	434: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	434: (7b) *(u64 *)(r10 -248) = r6
	435: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	435: (7b) *(u64 *)(r10 -256) = r6
	436: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264=mmmmmmmm fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	436: (7b) *(u64 *)(r10 -264) = r6
	437: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272=mmmmmmmm fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	437: (7b) *(u64 *)(r10 -272) = r6
	438: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280=mmmmmmmm fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	438: (7b) *(u64 *)(r10 -280) = r6
	439: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288=mmmmmmmm fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	439: (7b) *(u64 *)(r10 -288) = r6
	440: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296=mmmmmmmm fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	440: (7b) *(u64 *)(r10 -296) = r6
	441: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304=mmmmmmmm fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	441: (7b) *(u64 *)(r10 -304) = r6
	442: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312=mmmmmmmm fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	442: (7b) *(u64 *)(r10 -312) = r6
	443: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320=mmmmmmmm fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	443: (7b) *(u64 *)(r10 -320) = r6
	444: R0=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	444: (79) r1 = *(u64 *)(r10 -424)
	445: R0=inv(id=0) R1_w=fp-126 R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	445: (05) goto pc-397
	49: R0=inv(id=0) R1_w=fp-126 R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; char ch1 = str[2 * i + 1];
	49: (71) r3 = *(u8 *)(r1 +0)
	50: R0=inv(id=0) R1_w=fp-126 R3_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	50: (67) r3 <<= 56
	51: R0=inv(id=0) R1_w=fp-126 R3_w=inv(id=0,smax_value=9151314442816847872,umax_value=18374686479671623680,var_off=(0x0; 0xff00000000000000),s32_min_value=0,s32_max_value=0,u32_max_value=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	51: (c7) r3 s>>= 56
	52: R0=inv(id=0) R1_w=fp-126 R3_w=inv(id=0,smin_value=-128,smax_value=127) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib1 = (ch1 & 0xF) + (ch1 >> 6) | ((ch1 >> 3) & 0x8);
	52: (bf) r2 = r3
	53: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=603,smin_value=-128,smax_value=127) R3_w=inv(id=603,smin_value=-128,smax_value=127) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	53: (57) r2 &= 15
	54: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R3_w=inv(id=603,smin_value=-128,smax_value=127) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; char ch1 = str[2 * i + 1];
{"level":"error","ts":1712064481.3294368,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:179","msg":"error while loading probes, cleaning up","name":"google.golang.org/grpc/server","error":"field UprobeHttp2ServerOperateHeader: program uprobe_http2Server_operateHeader: load program: no space left on device: 73: (4f) r5 |= r4: 74: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R5_w=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmm (truncated, 14207 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:179\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:126\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:151\nmain.main\n\t/app/cli/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}
	54: (67) r3 <<= 32
	55: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R3_w=inv(id=0,smax_value=9223372032559808512,umax_value=18446744069414584320,var_off=(0x0; 0xffffffff00000000),s32_min_value=0,s32_max_value=0,u32_max_value=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	55: (77) r3 >>= 32
	56: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R3_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib1 = (ch1 & 0xF) + (ch1 >> 6) | ((ch1 >> 3) & 0x8);
	56: (bf) r4 = r3
	57: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R3_w=inv(id=604,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4_w=inv(id=604,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	57: (77) r4 >>= 6
	58: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R3_w=inv(id=604,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib1 = (ch1 & 0xF) + (ch1 >> 6) | ((ch1 >> 3) & 0x8);
	58: (0f) r2 += r4
	59: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=67108878,var_off=(0x0; 0x7ffffff)) R3_w=inv(id=604,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib1 = (ch1 & 0xF) + (ch1 >> 6) | ((ch1 >> 3) & 0x8);
	59: (77) r3 >>= 3
	60: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=67108878,var_off=(0x0; 0x7ffffff)) R3_w=inv(id=0,umax_value=536870911,var_off=(0x0; 0x1fffffff)) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib1 = (ch1 & 0xF) + (ch1 >> 6) | ((ch1 >> 3) & 0x8);
	60: (57) r3 &= 8
	61: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0,umax_value=67108878,var_off=(0x0; 0x7ffffff)) R3_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; out[i] = (nib0 << 4) | nib1;
	61: (4f) r2 |= r3
	62: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; char ch0 = str[2 * i];
	62: (71) r3 = *(u8 *)(r1 -1)
	63: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	63: (67) r3 <<= 56
	64: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=0,smax_value=9151314442816847872,umax_value=18374686479671623680,var_off=(0x0; 0xff00000000000000),s32_min_value=0,s32_max_value=0,u32_max_value=0) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	64: (c7) r3 s>>= 56
	65: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=0,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	65: (bf) r4 = r3
	66: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=605,smin_value=-128,smax_value=127) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	66: (67) r4 <<= 32
	67: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,smax_value=9223372032559808512,umax_value=18446744069414584320,var_off=(0x0; 0xffffffff00000000),s32_min_value=0,s32_max_value=0,u32_max_value=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	67: (77) r4 >>= 32
	68: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib0 = (ch0 & 0xF) + (ch0 >> 6) | ((ch0 >> 3) & 0x8);
	68: (bf) r5 = r4
	69: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=606,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R5_w=inv(id=606,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	69: (77) r5 >>= 6
	70: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=606,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R5_w=inv(id=0,umax_value=67108863,var_off=(0x0; 0x3ffffff)) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib0 = (ch0 & 0xF) + (ch0 >> 6) | ((ch0 >> 3) & 0x8);
	70: (0f) r5 += r3
	71: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=606,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R5_w=inv(id=0,smin_value=-128,smax_value=67108990) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib0 = (ch0 & 0xF) + (ch0 >> 6) | ((ch0 >> 3) & 0x8);
	71: (77) r4 >>= 3
	72: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=536870911,var_off=(0x0; 0x1fffffff)) R5_w=inv(id=0,smin_value=-128,smax_value=67108990) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; u8 nib0 = (ch0 & 0xF) + (ch0 >> 6) | ((ch0 >> 3) & 0x8);
	72: (57) r4 &= 8
	73: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R5_w=inv(id=0,smin_value=-128,smax_value=67108990) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmmmmmmm fp-360=mmmmmmmm fp-368=invP6 fp-376=inv fp-384=inv fp-392=inv fp-400=inv fp-408=inv fp-416=mmmmmmmm fp-424=fp fp-432=fp
	; out[i] = (nib0 << 4) | nib1;
	73: (4f) r5 |= r4
	74: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R5_w=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmm
	(truncated)
{"level":"info","ts":1712064481.3792746,"logger":"Instrumentation.Manager.google.golang.org/grpc/client","caller":"probe/probe.go:213","msg":"Closed","Probe":"google.golang.org/grpc/client"}
{"level":"info","ts":1712064481.3793674,"logger":"Instrumentation.Manager.google.golang.org/grpc/server","caller":"probe/probe.go:213","msg":"Closed","Probe":"google.golang.org/grpc/server"}
{"level":"info","ts":1712064481.3793843,"logger":"Instrumentation.Manager.net/http/server","caller":"probe/probe.go:213","msg":"Closed","Probe":"net/http/server"}
{"level":"info","ts":1712064481.3793988,"logger":"Instrumentation.Manager.net/http/client","caller":"probe/probe.go:213","msg":"Closed","Probe":"net/http/client"}
{"level":"info","ts":1712064481.3794107,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:204","msg":"Cleaning bpffs"}
{"level":"error","ts":1712064481.3795562,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:109","msg":"instrumentation crashed","error":"field UprobeHttp2ServerOperateHeader: program uprobe_http2Server_operateHeader: load program: no space left on device: 73: (4f) r5 |= r4: 74: R0=inv(id=0) R1_w=fp-126 R2_w=inv(id=0) R3_w=inv(id=605,smin_value=-128,smax_value=127) R4_w=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R5_w=inv(id=0) R6=invP0 R7=inv(id=602) R8=fp-280 R9=invP6 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mm?????? fp-144=mmmmmmmm fp-152=mmmm???? fp-160_w=00000000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=00000000 fp-232_w=00000000 fp-240_w=00000000 fp-248_w=00000000 fp-256_w=00000000 fp-264_w=00000000 fp-272_w=00000000 fp-280_w=00000000 fp-288_w=00000000 fp-296_w=00000000 fp-304_w=00000000 fp-312_w=00000000 fp-320_w=00000000 fp-328=mmmmmmmm fp-336=mmmmmmmm fp-344=mmmmmmmm fp-352=mmm (truncated, 14207 line(s) omitted)","stacktrace":"main.main\n\t/app/cli/main.go:109\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}

grcevski · 2024-04-02T13:45:19Z

I think we need to increase the verifier log buffer during load, so that we can see what the actual error is. The not enough space error is hiding the actual verifier issue. @Co0ker what's your kernel version?

Co0ker · 2024-04-02T14:10:26Z

@grcevski
"Detected linux kernel version","version":"5.15.0"

grcevski · 2024-04-26T21:59:01Z

Hi @Co0ker, I tried various ways to reproduce this but I can't see the error on my side. I tried on kernel version 5.15.0-101. Can you please set OTEL_GO_AUTO_SHOW_VERIFIER_LOG=true and provide the log after a re-run? Thanks!

vasiliy-grinko added the bug Something isn't working label Oct 9, 2023

grcevski mentioned this issue Apr 22, 2024

REQUEST: New membership for grcevski open-telemetry/community#2067

Closed

6 tasks

mike2194 mentioned this issue May 2, 2024

error building instrumentation with apps defining GOEXPERIMENT (Malformed version X:<experiment>) #812

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error while running instrumentors #367

error while running instrumentors #367

vasiliy-grinko commented Oct 9, 2023 •

edited

Loading

RonFed commented Oct 16, 2023

zsdyx commented Nov 28, 2023 •

edited

Loading

RonFed commented Nov 30, 2023

zsdyx commented Dec 2, 2023 •

edited

Loading

zsdyx commented Dec 2, 2023

zsdyx commented Dec 4, 2023

RonFed commented Dec 9, 2023

vbavireddi commented Jan 7, 2024 •

edited

Loading

RonFed commented Jan 7, 2024

vbavireddi commented Jan 7, 2024

vbavireddi commented Jan 8, 2024 •

edited

Loading

Co0ker commented Apr 2, 2024

grcevski commented Apr 2, 2024

Co0ker commented Apr 2, 2024

grcevski commented Apr 26, 2024

error while running instrumentors #367

error while running instrumentors #367

Comments

vasiliy-grinko commented Oct 9, 2023 • edited Loading

Describe the bug

Environment

Expected behavior

Logs

RonFed commented Oct 16, 2023

zsdyx commented Nov 28, 2023 • edited Loading

RonFed commented Nov 30, 2023

zsdyx commented Dec 2, 2023 • edited Loading

zsdyx commented Dec 2, 2023

zsdyx commented Dec 4, 2023

RonFed commented Dec 9, 2023

vbavireddi commented Jan 7, 2024 • edited Loading

Describe the bug

Environment

To Reproduce

RonFed commented Jan 7, 2024

vbavireddi commented Jan 7, 2024

vbavireddi commented Jan 8, 2024 • edited Loading

Co0ker commented Apr 2, 2024

grcevski commented Apr 2, 2024

Co0ker commented Apr 2, 2024

grcevski commented Apr 26, 2024

vasiliy-grinko commented Oct 9, 2023 •

edited

Loading

zsdyx commented Nov 28, 2023 •

edited

Loading

zsdyx commented Dec 2, 2023 •

edited

Loading

vbavireddi commented Jan 7, 2024 •

edited

Loading

vbavireddi commented Jan 8, 2024 •

edited

Loading