Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic when setting dont_fragment #685

Closed
mem opened this issue Aug 27, 2020 · 3 comments · Fixed by #686
Closed

panic when setting dont_fragment #685

mem opened this issue Aug 27, 2020 · 3 comments · Fixed by #686

Comments

@mem
Copy link
Contributor

mem commented Aug 27, 2020

Host operating system: output of uname -a

Linux clara 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

blackbox_exporter version: output of blackbox_exporter -version

blackbox_exporter, version 0.17.0 (branch: master, revision: 7913a15d3c1b225f7427e9240c63ab5075281845)
  build user:       marcelo@clara
  build date:       20200827-17:06:30
  go version:       go1.14.7

What is the blackbox.yml module config.

modules:
  ping_df:
    prober: icmp
    icmp:
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false
      dont_fragment: true
  ping_f:
    prober: icmp
    icmp:
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false
      dont_fragment: false

What is the prometheus.yml scrape config.

none

What logging output did you get from adding &debug=true to the probe URL?

none, BBE crashes.

What did you do that produced an error?

$ make
$ sudo setcap cap_net_raw+ep ./blackbox_exporter
$ getcap ./blackbox_exporter
./blackbox_exporter = cap_net_raw+ep

### request http://localhost:9115/probe?target=localhost&module=ping_df&debug=true

$ ./blackbox_exporter --config.file=config.yml --log.level=debug
level=info ts=2020-08-27T17:10:23.860Z caller=main.go:212 msg="Starting blackbox_exporter" version="(version=0.17.0, branch=master, revision=7913a15d3c1b225f7427e9240c63ab5075281845)"
level=info ts=2020-08-27T17:10:23.860Z caller=main.go:213 msg="Build context" (gogo1.14.7,usermarcelo@clara,date20200827-17:06:30)=(MISSING)
level=info ts=2020-08-27T17:10:23.860Z caller=main.go:225 msg="Loaded config file"
level=debug ts=2020-08-27T17:10:23.860Z caller=main.go:233 externalURL=http://clara:9115
level=debug ts=2020-08-27T17:10:23.860Z caller=main.go:247 routePrefix=/
level=info ts=2020-08-27T17:10:23.860Z caller=main.go:369 msg="Listening on address" address=:9115
ts=2020-08-27T17:10:41.197Z caller=main.go:169 module=ping_df target=localhost level=debug msg="Beginning probe" probe=icmp timeout_seconds=119.5
ts=2020-08-27T17:10:41.197Z caller=main.go:169 module=ping_df target=localhost level=debug msg="Resolving target address" ip_protocol=ip4
ts=2020-08-27T17:10:41.197Z caller=main.go:169 module=ping_df target=localhost level=debug msg="Resolved target address" ip=127.0.0.1
ts=2020-08-27T17:10:41.197Z caller=main.go:169 module=ping_df target=localhost level=debug msg="Creating socket"
2020/08/27 11:10:41 http: panic serving [::1]:60312: interface conversion: *icmp.PacketConn is not net.Conn: missing method Read
goroutine 49 [running]:
net/http.(*conn).serve.func1(0xc000211d60)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:1800 +0x139
panic(0xaa8ca0, 0xc00030abd0)
        /home/marcelo/devel/go/go-1.14/src/runtime/panic.go:975 +0x3e3
golang.org/x/net/ipv4.NewRawConn(0xc77fc0, 0xc0003446c0, 0xc000308530, 0x7, 0xc0003446c0)
        /home/marcelo/devel/grafana/blackbox_exporter/vendor/golang.org/x/net/ipv4/endpoint.go:169 +0x45
github.com/prometheus/blackbox_exporter/prober.ProbeICMP(0xc70860, 0xc000310060, 0xc000316012, 0x9, 0xc000035760, 0x4, 0x0, 0x0, 0x0, 0x0, ...)
        /home/marcelo/devel/grafana/blackbox_exporter/prober/icmp.go:164 +0x1f7c
main.probeHandler(0xc6eda0, 0xc00032a000, 0xc000314000, 0xc00000e328, 0xc63a60, 0xc000227140, 0xc000094c80)
        /home/marcelo/devel/grafana/blackbox_exporter/main.go:119 +0x977
main.run.func4(0xc6eda0, 0xc00032a000, 0xc000314000)
        /home/marcelo/devel/grafana/blackbox_exporter/main.go:304 +0xb1
net/http.HandlerFunc.ServeHTTP(0xc00000dc00, 0xc6eda0, 0xc00032a000, 0xc000314000)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:2041 +0x44
net/http.(*ServeMux).ServeHTTP(0x113e5a0, 0xc6eda0, 0xc00032a000, 0xc000314000)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:2416 +0x1a5
net/http.serverHandler.ServeHTTP(0xc000262000, 0xc6eda0, 0xc00032a000, 0xc000314000)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:2836 +0xa3
net/http.(*conn).serve(0xc000211d60, 0xc707e0, 0xc00030c000)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:1924 +0x86c
created by net/http.(*Server).Serve
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:2962 +0x35c
ts=2020-08-27T17:10:41.197Z caller=main.go:169 module=ping_df target=localhost level=debug msg="Beginning probe" probe=icmp timeout_seconds=119.5
ts=2020-08-27T17:10:41.197Z caller=main.go:169 module=ping_df target=localhost level=debug msg="Resolving target address" ip_protocol=ip4
ts=2020-08-27T17:10:41.198Z caller=main.go:169 module=ping_df target=localhost level=debug msg="Resolved target address" ip=127.0.0.1
ts=2020-08-27T17:10:41.198Z caller=main.go:169 module=ping_df target=localhost level=debug msg="Creating socket"
2020/08/27 11:10:41 http: panic serving [::1]:60314: interface conversion: *icmp.PacketConn is not net.Conn: missing method Read
goroutine 50 [running]:                                                                                                                                                                [1/9380]
net/http.(*conn).serve.func1(0xc000211e00)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:1800 +0x139
panic(0xaa8ca0, 0xc00030b380)
        /home/marcelo/devel/go/go-1.14/src/runtime/panic.go:975 +0x3e3
golang.org/x/net/ipv4.NewRawConn(0xc77fc0, 0xc000344c40, 0xc0003087b0, 0x7, 0xc000344c40)
        /home/marcelo/devel/grafana/blackbox_exporter/vendor/golang.org/x/net/ipv4/endpoint.go:169 +0x45
github.com/prometheus/blackbox_exporter/prober.ProbeICMP(0xc70860, 0xc000310840, 0xc000316792, 0x9, 0xc000035760, 0x4, 0x0, 0x0, 0x0, 0x0, ...)
        /home/marcelo/devel/grafana/blackbox_exporter/prober/icmp.go:164 +0x1f7c
main.probeHandler(0xc6eda0, 0xc00032a0e0, 0xc000202500, 0xc00000e328, 0xc63a60, 0xc000227140, 0xc000094c80)
        /home/marcelo/devel/grafana/blackbox_exporter/main.go:119 +0x977
main.run.func4(0xc6eda0, 0xc00032a0e0, 0xc000202500)
        /home/marcelo/devel/grafana/blackbox_exporter/main.go:304 +0xb1
net/http.HandlerFunc.ServeHTTP(0xc00000dc00, 0xc6eda0, 0xc00032a0e0, 0xc000202500)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:2041 +0x44
net/http.(*ServeMux).ServeHTTP(0x113e5a0, 0xc6eda0, 0xc00032a0e0, 0xc000202500)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:2416 +0x1a5
net/http.serverHandler.ServeHTTP(0xc000262000, 0xc6eda0, 0xc00032a0e0, 0xc000202500)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:2836 +0xa3
net/http.(*conn).serve(0xc000211e00, 0xc707e0, 0xc000099600)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:1924 +0x86c
created by net/http.(*Server).Serve
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:2962 +0x35c
ts=2020-08-27T17:10:41.198Z caller=main.go:169 module=ping_df target=localhost level=debug msg="Beginning probe" probe=icmp timeout_seconds=119.5
ts=2020-08-27T17:10:41.198Z caller=main.go:169 module=ping_df target=localhost level=debug msg="Resolving target address" ip_protocol=ip4
ts=2020-08-27T17:10:41.198Z caller=main.go:169 module=ping_df target=localhost level=debug msg="Resolved target address" ip=127.0.0.1
ts=2020-08-27T17:10:41.198Z caller=main.go:169 module=ping_df target=localhost level=debug msg="Creating socket"
2020/08/27 11:10:41 http: panic serving [::1]:60316: interface conversion: *icmp.PacketConn is not net.Conn: missing method Read
goroutine 81 [running]:
net/http.(*conn).serve.func1(0xc000356140)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:1800 +0x139
panic(0xaa8ca0, 0xc00027c4e0)
        /home/marcelo/devel/go/go-1.14/src/runtime/panic.go:975 +0x3e3
golang.org/x/net/ipv4.NewRawConn(0xc77fc0, 0xc00027e1a0, 0xc000035c10, 0x7, 0xc00027e1a0)
        /home/marcelo/devel/grafana/blackbox_exporter/vendor/golang.org/x/net/ipv4/endpoint.go:169 +0x45
github.com/prometheus/blackbox_exporter/prober.ProbeICMP(0xc70860, 0xc000047c80, 0xc00003abd2, 0x9, 0xc000035760, 0x4, 0x0, 0x0, 0x0, 0x0, ...)
        /home/marcelo/devel/grafana/blackbox_exporter/prober/icmp.go:164 +0x1f7c
main.probeHandler(0xc6eda0, 0xc0002620e0, 0xc000202600, 0xc00000e328, 0xc63a60, 0xc000227140, 0xc000094c80)
        /home/marcelo/devel/grafana/blackbox_exporter/main.go:119 +0x977
main.run.func4(0xc6eda0, 0xc0002620e0, 0xc000202600)
        /home/marcelo/devel/grafana/blackbox_exporter/main.go:304 +0xb1
net/http.HandlerFunc.ServeHTTP(0xc00000dc00, 0xc6eda0, 0xc0002620e0, 0xc000202600)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:2041 +0x44
net/http.(*ServeMux).ServeHTTP(0x113e5a0, 0xc6eda0, 0xc0002620e0, 0xc000202600)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:2416 +0x1a5
net/http.serverHandler.ServeHTTP(0xc000262000, 0xc6eda0, 0xc0002620e0, 0xc000202600)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:2836 +0xa3
net/http.(*conn).serve(0xc000356140, 0xc707e0, 0xc000099700)
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:1924 +0x86c
created by net/http.(*Server).Serve
        /home/marcelo/devel/go/go-1.14/src/net/http/server.go:2962 +0x35c

What did you expect to see?

not crash

What did you see instead?

crash
@mem
Copy link
Contributor Author

mem commented Aug 27, 2020 •
edited
Loading

3007522 changed the type of the argument that is passed to ipv4.NewRawConn. This function starts by doing cc, err := socket.NewConn(c.(net.Conn)) (in golang.org/x/net/ipv4/endpoint.go), where net.Conn is an interface, and the instance that is passed to NewRawConn does not implement this interface, therefore this panic.

Setting the capability is not necessary, you can run BBE as root and you get the same result. The capability (or root) is necessary for being able to set IP header level options.

mem added a commit to mem/blackbox_exporter that referenced this issue Aug 27, 2020
@mem
Fix panic when running ICMPv4 probe with DontFragment
97dff69 
A recent change modified the way an IPv4 raw socket is created and BBE
is panicing when ICMPv4 is used with DontFragment set.

golang.org/x/net doesn't seem to have a way to create the necessary
socket, so fall back to the previous method for that case in particular
(ICMP, IPv4, DontFragment=true).

Fixes prometheus#685

Signed-off-by: Marcelo E. Magallon <[email protected]>
@mem mem mentioned this issue Aug 27, 2020
Merged
mem added a commit to mem/blackbox_exporter that referenced this issue Aug 28, 2020
@mem
Fix panic when running ICMPv4 probe with DontFragment
6f7caa5 
A recent change modified the way an IPv4 raw socket is created and BBE
is panicing when ICMPv4 is used with DontFragment set.

golang.org/x/net doesn't seem to have a way to create the necessary
socket, so fall back to the previous method for that case in particular
(ICMP, IPv4, DontFragment=true).

Fixes prometheus#685

Signed-off-by: Marcelo E. Magallon <[email protected]>
brian-brazil pushed a commit that referenced this issue Aug 29, 2020
@mem
Fix panic when running ICMPv4 probe with DontFragment (#686)
04d136f 
A recent change modified the way an IPv4 raw socket is created and BBE
is panicing when ICMPv4 is used with DontFragment set.

golang.org/x/net doesn't seem to have a way to create the necessary
socket, so fall back to the previous method for that case in particular
(ICMP, IPv4, DontFragment=true).

Fixes #685

Signed-off-by: Marcelo E. Magallon <[email protected]>
@sgoodliff
Copy link

Hi,

Im not sure of your release cycle but any ideas when this is likely to get released ?

Thanks

@brian-brazil
Copy link
Contributor

There's another PR currently open, once that's in I'll probably release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix panic when running ICMPv4 probe with DontFragment mem/blackbox_exporter
3 participants
@mem @sgoodliff @brian-brazil