Skip to content

Commit c7e686f

Browse files
committed
feat(kubectl-trace): allow passing a service account to enable usage of pod security policies
Signed-off-by: Lorenzo Fontana <[email protected]>
1 parent 9e635f2 commit c7e686f

File tree

3 files changed

+140
-32
lines changed

3 files changed

+140
-32
lines changed

README.md

+111-7
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ This will download and compile `kubectl-trace` so that you can use it as a kubec
2929
You don't need to setup anything on your cluster before using it, please don't use it already
3030
on a production system, just because this isn't yet 100% ready.
3131

32-
**Run a program from string literal:**
32+
### Run a program from string literal
3333

3434
In this case we are running a program that probes a tracepoint
3535
on the node `ip-180-12-0-152.ec2.internal`.
@@ -39,15 +39,15 @@ kubectl trace run ip-180-12-0-152.ec2.internal -e "tracepoint:syscalls:sys_enter
3939
```
4040

4141

42-
**Run a program from file:**
42+
### Run a program from file
4343

4444
Here we run a program named `read.bt` against the node `ip-180-12-0-152.ec2.internal`
4545

4646
```
4747
kubectl trace run ip-180-12-0-152.ec2.internal -f read.bt
4848
```
4949

50-
**Run a program against a Pod**
50+
### Run a program against a Pod
5151

5252
![Screenshot showing the read.bt program for kubectl-trace](docs/img/pod.png)
5353

@@ -83,6 +83,114 @@ So, running against a pod **doesn't mean** that your bpftrace program will be co
8383
knowledge of the context of a container, in this case only the root process id is supported via the `$container_pid` variable.
8484

8585

86+
### Using a custom service account
87+
88+
By default `kubectl trace` will use the `default` service account in the target namespace (that is also `default`), to schedule the pods needed for your bpftrace program.
89+
90+
If you need to pass a service account you can use the `--serviceaccount` flag.
91+
92+
```bash
93+
kubectl trace run --serviceaccount=kubectltrace ip-180-12-0-152.ec2.internal -f read.bt
94+
```
95+
96+
### Executing in a cluster using Pod Security Policies
97+
98+
If your cluster has pod security policies you will need to make so that `kubectl trace` can
99+
use a service account that can run privileged containers.
100+
101+
That service account, then will need to be in a group that uses the proper privileged `PodSecurityPolicy`.
102+
103+
First, create the service account that you will use with `kubectl trace`,
104+
you can use a different namespace other than `default`, just remember to pass that namespace to the `run` command when you will use `kubectl trace`:
105+
106+
```yaml
107+
apiVersion: v1
108+
kind: ServiceAccount
109+
metadata:
110+
name: kubectltrace
111+
namespace: default
112+
```
113+
114+
Now that we have a `kubectltrace` service account let's create a Pod Security Policy:
115+
116+
```yaml
117+
apiVersion: policy/v1beta1
118+
kind: PodSecurityPolicy
119+
metadata:
120+
name: kubectltrace
121+
spec:
122+
fsGroup:
123+
rule: RunAsAny
124+
privileged: true
125+
runAsUser:
126+
rule: RunAsAny
127+
seLinux:
128+
rule: RunAsAny
129+
supplementalGroups:
130+
rule: RunAsAny
131+
volumes:
132+
- '*'
133+
allowedCapabilities:
134+
- '*'
135+
hostPID: true
136+
hostIPC: true
137+
hostNetwork: true
138+
hostPorts:
139+
- min: 1
140+
max: 65536
141+
```
142+
143+
Ok, this `PodSecurityPolicy` will allow users assigned to it to run privileged containers,
144+
`kubectl trace` needs that because of the extended privileges eBPF programs need to run with
145+
to trace your kernel and programs running in it.
146+
147+
Now with a `ClusterRoleBinding` you bind the `ClusterRole` with the `ServiceAccount`, so that
148+
they can work together with the `PodSecurityPolicy` we just created.
149+
150+
You can change the `namespace: default` here if you created the service account in a namespace other than `default`.
151+
152+
```yaml
153+
apiVersion: rbac.authorization.k8s.io/v1
154+
kind: ClusterRole
155+
metadata:
156+
name: kubectltrace-psp
157+
rules:
158+
- apiGroups:
159+
- policy
160+
resources:
161+
- podsecuritypolicies
162+
resourceNames:
163+
- kubectltrace
164+
verbs:
165+
- use
166+
---
167+
apiVersion: rbac.authorization.k8s.io/v1
168+
kind: ClusterRoleBinding
169+
metadata:
170+
name: kubectltrace-psp
171+
subjects:
172+
- kind: ServiceAccount
173+
name: kubectltrace
174+
namespace: default
175+
roleRef:
176+
apiGroup: rbac.authorization.k8s.io
177+
kind: ClusterRole
178+
name: kubectltrace-psp
179+
```
180+
181+
OK! Now that we are all set we can just run the program by specifying the service account
182+
we just created and it will use our pod security policy!
183+
184+
```bash
185+
kubectl trace run --serviceaccount=kubectltrace ip-180-12-0-152.ec2.internal -f read.bt
186+
```
187+
188+
If you used a different namespace other than default for your service account, you will want to specify the namespace too, like this:
189+
190+
```bash
191+
kubectl trace run --namespace=mynamespace --serviceaccount=kubectltrace ip-180-12-0-152.ec2.internal -f read.bt
192+
```
193+
86194
### More bpftrace programs
87195

88196
Need more programs? Look [here](https://github.com/iovisor/bpftrace/tree/master/tools).
@@ -119,10 +227,6 @@ kubectl trace run pod/<pod-name> -c <container> f read.bt
119227
120228
So I would say, the next thing is to run bpftrace programs at a pod scope other than at node scope.</strike>
121229
122-
**bpftrace work**
123-
124-
I also plan to contribute some IO functions to bpftrace to send data to a backend database like InfluxDB instead of only stdout
125-
because that would enable having things like graphs showing
126230
127231
## Contributing
128232

pkg/cmd/run.go

+18-15
Original file line numberDiff line numberDiff line change
@@ -54,13 +54,14 @@ type RunOptions struct {
5454
explicitNamespace bool
5555

5656
// Local to this command
57-
container string
58-
eval string
59-
program string
60-
resourceArg string
61-
attach bool
62-
isPod bool
63-
podUID string
57+
container string
58+
eval string
59+
program string
60+
resourceArg string
61+
attach bool
62+
isPod bool
63+
podUID string
64+
serviceAccount string
6465

6566
nodeName string
6667

@@ -103,6 +104,7 @@ func NewRunCommand(factory factory.Factory, streams genericclioptions.IOStreams)
103104
cmd.Flags().BoolVarP(&o.attach, "attach", "a", o.attach, "Wheter or not to attach to the trace program once it is created")
104105
cmd.Flags().StringVarP(&o.eval, "eval", "e", "", "Literal string to be evaluated as a bpftrace program")
105106
cmd.Flags().StringVarP(&o.program, "filename", "f", "", "File containing a bpftrace program")
107+
cmd.Flags().StringVar(&o.serviceAccount, "serviceaccount", "default", "Service account to use to set in the pod spec of the kubectl-trace job")
106108

107109
return cmd
108110
}
@@ -265,14 +267,15 @@ func (o *RunOptions) Run() error {
265267
}
266268

267269
tj := tracejob.TraceJob{
268-
Name: fmt.Sprintf("%s%s", meta.ObjectNamePrefix, string(juid)),
269-
Namespace: o.namespace,
270-
ID: juid,
271-
Hostname: o.nodeName,
272-
Program: o.program,
273-
PodUID: o.podUID,
274-
ContainerName: o.container,
275-
IsPod: o.isPod,
270+
Name: fmt.Sprintf("%s%s", meta.ObjectNamePrefix, string(juid)),
271+
Namespace: o.namespace,
272+
ServiceAccount: o.serviceAccount,
273+
ID: juid,
274+
Hostname: o.nodeName,
275+
Program: o.program,
276+
PodUID: o.podUID,
277+
ContainerName: o.container,
278+
IsPod: o.isPod,
276279
}
277280

278281
job, err := tc.CreateJob(tj)

pkg/tracejob/job.go

+11-10
Original file line numberDiff line numberDiff line change
@@ -24,14 +24,15 @@ type TraceJobClient struct {
2424
}
2525

2626
type TraceJob struct {
27-
Name string
28-
ID types.UID
29-
Namespace string
30-
Hostname string
31-
Program string
32-
PodUID string
33-
ContainerName string
34-
IsPod bool
27+
Name string
28+
ID types.UID
29+
Namespace string
30+
ServiceAccount string
31+
Hostname string
32+
Program string
33+
PodUID string
34+
ContainerName string
35+
IsPod bool
3536
}
3637

3738
// WithOutStream setup a file stream to output trace job operation information
@@ -101,7 +102,6 @@ func (t *TraceJobClient) findConfigMapsWithFilter(nf TraceJobFilter) ([]apiv1.Co
101102
}
102103

103104
func (t *TraceJobClient) GetJob(nf TraceJobFilter) ([]TraceJob, error) {
104-
105105
jl, err := t.findJobsWithFilter(nf)
106106
if err != nil {
107107
return nil, err
@@ -218,7 +218,8 @@ func (t *TraceJobClient) CreateJob(nj TraceJob) (*batchv1.Job, error) {
218218
Template: apiv1.PodTemplateSpec{
219219
ObjectMeta: commonMeta,
220220
Spec: apiv1.PodSpec{
221-
HostPID: true,
221+
HostPID: true,
222+
ServiceAccountName: nj.ServiceAccount,
222223
Volumes: []apiv1.Volume{
223224
apiv1.Volume{
224225
Name: "program",

0 commit comments

Comments
 (0)