Missing header error while trying to run a trace

[achanda]

I tried to run a trace on a cluster built with kops, here is what I see in trace logs

# kubectl-trace logs kubectl-trace-c2405ead-a4b3-11e9-890b-0232f65d1adc
if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
/bpftrace/include/asm_goto_workaround.h:14:10: fatal error: 'linux/types.h' file not found

Relevant info

# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.9", GitCommit:"16236ce91790d4c75b79f6ce96841db1c843e7d2", GitTreeState:"clean", BuildDate:"2019-03-25T06:40:24
Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.9", GitCommit:"16236ce91790d4c75b79f6ce96841db1c843e7d2", GitTreeState:"clean", BuildDate:"2019-03-25T06:30:48
Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
# kubectl-trace version
git commit:
build date: undefined
# uname -a
Linux ip-10-65-134-126 4.9.0-7-amd64 #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x86_64 GNU/Linux

[leodido]

Hi @achanda thanks for opening this!

Can you provide the OS info for the node running this?

[achanda]

@leodido sorry, forgot that bit 😅 Update now. I got the same error when I ran the cli from MacOS, I assumed that it will work only on Linux.

[leodido]

Ok, thanks. Will take a look.

The target k8s cluster should be running on Linux. The client machine can be anything.

I'm not sure if it works on docker for mac to be honest (because of this).

If you want to share any other details (e.g, the minimal steps to recreate a similar cluster with kops would be really helpful) please do! :)

[achanda]

I ran this from a master node, so all other nodes in the cluster should have the same kernel and Linux version. We do, however, use our own custom AMI that adds a bunch of other tools on top of Kops' AMI. I can try this on a vanilla kops cluster.

[rikatz]

Same here. Running master and nodes with ubuntu 19.04.

Trying installing the build-essentials in all of them, but the error still happens. The cluster is running with Cilium correctly, so it might be some other thing.

The same happens while running against Node or Pod, as the following:

kubectl trace run -f opensnoop.bt pod/nginx-7c45b84548-9wl6j -a
trace 8b0e3f4d-a66b-11e9-b0f3-000c29cf9885 created

if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
/bpftrace/include/asm_goto_workaround.h:14:10: fatal error: 'linux/types.h' file not found
exit status 1

Tks

[rikatz]

So I've digging a little bit and found another error:

$ docker run -it --rm -v /lib/modules:/lib/modules -v /sys:/sys --privileged --pid=host --entrypoint /bin/sh quay.io/iovisor/kubectl-trace-bpftrace:latest

# bpftrace teste.bt
No error information: couldn't set RLIMIT_MEMLOCK for bpftrace. If your program is not loading, you can try "ulimit -l 8192" to fix the problem
/bpftrace/include/asm_goto_workaround.h:14:10: fatal error: 'linux/types.h' file not found
# apk add --update linux-headers
bpftrace teste.bt
definitions.h:10:3: error: unknown type name 's64'
definitions.h:11:3: error: unknown type name 'umode_t'
definitions.h:20:3: error: unknown type name 's64'
definitions.h:22:3: error: unknown type name 's64'
definitions.h:23:3: error: unknown type name 'umode_t'

So I've mounted the /usr/src from the host and worked fine

docker run -it --rm -v /lib/modules:/lib/modules -v /usr/src/:/usr/src -v /sys:/sys --privileged --pid=host  --entrypoint /bin/sh quay.io/iovisor/kubectl-trace-bpftrace:latest

Don't know if this is an Ubuntu issue or general one, but I'll keep digging to see if just mounting /usr/src (and putting linux-headers as a dependency to kubectl-trace) is all we need.

[rikatz]

It seems I've to force the --fetch-headers to kubectl-trace run, but it still doesn't work.

I've recompiled kubectl-trace to force the container mount/map between host /usr/src and container's /usr/src and this did worked fine, but don't know if this is the right path to follow.

Waiting for further directions :)

[fntlnz]

@rikatz oh good catch! Can you give the output of:

ls -la /lib/modules/$(uname -r)

I'm curious to see the symlinks there. Maybe we need to change the strategy to detect headers in different distros.

[rikatz]

@fntlnz for sure!

 ls -la /lib/modules/$(uname -r)
total 5540
drwxr-xr-x  5 root root    4096 Jul  7 20:57 .
drwxr-xr-x  3 root root    4096 Jul  7 20:57 ..
lrwxrwxrwx  1 root root      39 Jun 24 08:09 build -> /usr/src/linux-headers-5.0.0-20-generic
drwxr-xr-x  2 root root    4096 Jun 24 08:09 initrd
drwxr-xr-x 17 root root    4096 Jul  7 20:57 kernel
-rw-r--r--  1 root root 1334328 Jul  7 20:57 modules.alias
-rw-r--r--  1 root root 1312239 Jul  7 20:57 modules.alias.bin
-rw-r--r--  1 root root    7744 Jun 24 08:09 modules.builtin
-rw-r--r--  1 root root    9784 Jul  7 20:57 modules.builtin.bin
-rw-r--r--  1 root root  597481 Jul  7 20:57 modules.dep
-rw-r--r--  1 root root  832653 Jul  7 20:57 modules.dep.bin
-rw-r--r--  1 root root     317 Jul  7 20:57 modules.devname
-rw-r--r--  1 root root  213933 Jun 24 08:09 modules.order
-rw-r--r--  1 root root     633 Jul  7 20:57 modules.softdep
-rw-r--r--  1 root root  594841 Jul  7 20:57 modules.symbols
-rw-r--r--  1 root root  724184 Jul  7 20:57 modules.symbols.bin
drwxr-xr-x  3 root root    4096 Jul  7 20:57 vdso

It seems that the symlink there created by Ubuntu might be an issue.

[rikatz]

Tks for the fix @fntlnz and @joestringer :)

[jdel12]

I just encountered this as well on a vanilla GKE cluster. I haven't done any significant exploring to be clear, just bringing it up if it's a recurring issue.

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T23:41:55Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-gke.14", GitCommit:"56d89863d1033f9668ddd6e1c1aea81cd846ef88", GitTreeState:"clean", BuildDate:"2019-11-07T19:12:22Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

planet-express$ #kubectl trace run -e 'tracepoint:block:block_rq_issue { @ = hist(args->bytes); }' pod/esclus02-es-nodeset-default-0 -a -n default

planet-express$ k logs kubectl-trace-730dad7e-3654-11ea-8b16-acde48001122-r8rm2
if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
/bpftrace/include/asm_goto_workaround.h:14:10: fatal error: 'linux/types.h' file not found
exit status 1

planet-express$ kubectl trace version
git commit: d34d1d586b110af718aeadc9f3213c78e543a961
build date: 2019-09-19 10:00:13 -0500 CDT

### this one was with the --fetch-headers flag
planet-express$ k logs kubectl-trace-be21a85a-3655-11ea-bafa-acde48001122-qhnnz
if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
Attaching 1 probe...
ioctl(PERF_EVENT_IOC_SET_BPF): File exists
Error attaching probe: tracepoint:block:block_rq_issue
exit status 255

[stafot]

Also fails on AWS EKS 1.16 with Amazon Linux 2 OS on nodes.

2020-06-07T17:16:27.304465751Z if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
2020-06-07T17:16:27.327396294Z /bpftrace/include/asm_goto_workaround.h:14:10: fatal error: 'linux/types.h' file not found
2020-06-07T17:16:27.330727621Z exit status 1

[joestringer]

@stafot I think you also need to install the kernel-headers package on the node to get this working on Amazon Linux 2.

[stafot]

@joestringer thanks I will let you know if this fixes the issue.

[stafot]

@joestringer Unfortunately it doesn't work.

 sudo yum install kernel-headers
Loaded plugins: priorities, update-motd
amzn2-core                                                                                                                                                                           | 3.7 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package kernel-headers.x86_64 0:4.14.181-140.257.amzn2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================================================
 Package                                           Arch                                      Version                                                    Repository                                     Size
============================================================================================================================================================================================================
Installing:
 kernel-headers                                    x86_64                                    4.14.181-140.257.amzn2                                     amzn2-core                                    1.1 M

Transaction Summary
============================================================================================================================================================================================================
Install  1 Package

Total download size: 1.1 M
Installed size: 4.2 M
Is this ok [y/d/N]: y
Downloading packages:
kernel-headers-4.14.181-140.257.amzn2.x86_64.rpm                                                                                                                                     | 1.1 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kernel-headers-4.14.181-140.257.amzn2.x86_64                                                                                                                                             1/1 
  Verifying  : kernel-headers-4.14.181-140.257.amzn2.x86_64                                                                                                                                             1/1 

Installed:
  kernel-headers.x86_64 0:4.14.181-140.257.amzn2                                                                                                                                                            

Complete!

And the error remains the same.

[stafot]

For anyone in the future having the same problem on Amazon Linux 2 the solution to run successfully kubectl trace is to have installed on node the kernel-devel package.

sudo  yum install kernel-devel-$(uname -r)

relevant resource

Thanks, @joestringer for the initial advice that help me to find the fix for my use case.

[fluffysquirrels]

Similarly, sudo apt-get install linux-headers-amd64 fixed this on Debian 10.

[tuannvm]

For me it's still not working. I need to append the --fetch-headers to the kubectl trace command

[stafot]

@tuannvm on what node type are you running out of interest?

[FingerLiu]

Any guy had luck with CentOS?
Still not work on CentOS.
Even if after adding --fetch-headers.
I inspect into the init container, and found logs below.
Found something suspicious:on centos kernel version(uname -r) is 5.15.4-1.el7.elrepo.x86_64
but the install command is yum install kernel-ml-headers.x86_64

WARNING: Cannot find distro-specific headers for "CentOS Linux". Fetching generic headers.
+ BUILD_DIR=/linux-generic-5.15.4-1.el7.elrepo.x86_64
+ SOURCES_DIR=/usr/src/linux-generic-5.15.4-1.el7.elrepo.x86_64
+ [[ ! -e /usr/src/linux-generic-5.15.4-1.el7.elrepo.x86_64/.installed ]]
+ echo 'Installing kernel headers for generic kernel'
Installing kernel headers for generic kernel
+ fetch_generic_linux_sources
++ awk '-vFS=[-+]' '{ print $1 }'
++ echo 5.15.4-1.el7.elrepo.x86_64
+ kernel_version=5.15.4
++ echo 5.15.4-1.el7.elrepo.x86_64
++ awk -vFS=. '{ print $1 }'
+ major_version=5
+ [[ 5.15.4 == *.0 ]]
+ echo 'Fetching upstream kernel sources for 5.15.4.'
Fetching upstream kernel sources for 5.15.4.
+ mkdir -p /linux-generic-5.15.4-1.el7.elrepo.x86_64
+ curl -sL https://www.kernel.org/pub/linux/kernel/v5.x/linux-5.15.4.tar.gz
+ tar --strip-components=1 -xzf - -C /linux-generic-5.15.4-1.el7.elrepo.x86_64
tar: invalid magic
tar: short read
real	0m1.344s
user	0m0.035s
sys	0m0.017s

[FingerLiu]

I can confirm that I have already install header and devel on my node, why should I --fetch-headers on pod again.