Proposal: kubectl extension #122

smarterclayton · 2016-12-01T18:10:19Z

pmorie · 2016-12-02T01:09:55Z

I have a strong need for this feature. For the service-catalog, there is a need to build macros and other extensions for resources in a not-Kubernetes API server. It would be tremendously useful if there were a way to register these with 'normal' kubectl instead of publishing a wholly different binary.

deads2k · 2016-12-02T13:04:24Z

We need it for API servers providing new resource types as well.

I think that starting really small like executing your binary with all the flags, is the minimal useful thing we could provide and it doesn't preclude doing something fancy with proxies later.

rootfs · 2016-12-02T15:04:02Z

contributors/design-proposals/kubectl-extension.md

+Proposal
+--------
+
+Define a system for `kubectl` that allows new subcommands and subcommand trees to be added by placing an executable in a specific


I prefer to support both containerized command and in-directory command. It is more extensible to for kubectl to respond to specific handler definitions (e.g. similar to how clink works)

Yes - IMHO the primary should be on containerized extensions.

We have a use-case where we are shipping TPRs and require custom commands for them. Thus shipping kubectl extensions in containers (or DaemonSets then) would be very convenient. And would fit much bette rinto the broader story.

Just my 2ct

I love containers as much as anyone but I feel that plugins ala git are much simpler thing to tackle on all of the platforms kubectl supports today.

rootfs · 2016-12-02T15:05:06Z

I love this feature to extend kubectl to search out tree subcommands and support certain volume features.

bgrant0607 · 2016-12-02T20:09:26Z

cc @kubernetes/sig-cli

fabianofranz · 2016-12-02T20:27:27Z

So far from the discussions in the original PR, I'm seeing 3 potential kinds of extensions:

kubectl binary extensions: the ability to extend kubectl by adding more subcommands. For this extension kind the top command can either expose access to remote calls (e.g. RPC) or the current context to access the API (e.g. a client proxy), or completely leave it to the extension to handle (e.g. Git-style "extensions"). Use cases include:

Add more views or dashboard-like features to the existing data. Examples:
- live kubectl status with termui
Subcommands to access third-party APIs. Examples:
- kubectl jenkins
- kubectl heapster
- kubectl git or kubectl github
New commands that for some reason we don't want to be part of the core binary. Examples:
- kubectl rsync
- kubectl plugins and kubectl plugin install
- kubectl autoupdate

extend existing commands for TPR: the ability to extend existing commands to add support to new resources. Examples:

kubectl create my-third-party-object
kubectl set image third_party/object new_image

extend existing commands for existing types: the ability to extend existing commands to add more features to existing types. Examples:

kubectl set session-affinity <service>
kubectl set app-label

For binary extensions, I believe there is great value in allowing contributors to write plugins in other languages (not only Go). Plugins could be other compiled languages, Ruby scripts, bash scripts, etc; if we define an integration point not specific to Go (e.g. RPC instead of gorpc, or even REST with YAML/JSON, etc). Interesting related project: https://npf.io/2015/05/pie/.

fabianofranz · 2016-12-02T20:32:35Z

That said, I wrote a PoC for item 1 (binary extensions). There are potential security issues like mentioned, but it's interesting to see what can be done. The architecture is pretty similar to Helm plugins.

kubernetes/kubernetes#37499

Give kubectl aging a try (plugin written in Ruby script, uses data from the API).

I have a strong need for this feature.

@pmorie can you take a look at the PoC above? How good/bad that addresses your need?

davidopp · 2016-12-04T00:19:31Z

I like this idea; it would be great to add some concrete examples to the doc.

One use case I think is particularly important is to allow the creator of a Third Party Resource to control what "get" and "describe" show for their resource (e.g. what data, how it is formatted in the screen output, etc.).

pmorie · 2016-12-04T01:20:11Z

I think it would be awesome for API servers to be able to indicate formatted get and describe output of any resource - would get you most of the way there on new API servers, you would just need to implement macros as extensions. Might not be what you would always want to do, but would be handy for prototyping and getting new things in front of people with lower complexity.

…

On Sat, Dec 3, 2016 at 7:19 PM David Oppenheimer ***@***.***> wrote: I like this idea; it would be great to add some concrete examples to the doc. One use case I think is particularly important is to allow the creator of a Third Party Resource to control what "get" and "describe" show for their resource (e.g. what data, how it is formatted in the screen output, etc.). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#122 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAWXmN74v1CSGhvkBibPlxtnFKwjoIE8ks5rEgcVgaJpZM4LBtnz> .

adohe-zz · 2016-12-04T07:44:59Z

I think it would be awesome for API servers to be able to indicate
formatted get and describe output of any resource

yes, there is just another proposal describe this.

soltysh · 2016-12-05T10:04:23Z

contributors/design-proposals/kubectl-extension.md

+behavior (such as complex inference of commands) may not be supported in order to reduce the complexity of the lookup.
+
+Kubectl would lazily include the appropriate commands in preference to the internal command structure if detected (a user asking for
+`kubectl a b c` would *first* check for `kubectl-a-b-c`, `kubectl-a-b`, or `kubectl-a` before loading the internal command).


Is this the solution to address the usecase of providing my own getter (or other existing command) for TPR, iow. if I want to have kubectl get mynewtpr/foo I need to define kubectl-get-mynewtpr?

get/edit/delete by resource/name, create/update from file should work without custom extensions

if you want commands like kubectl create mytype some-name --custom-option-1=something, you would need custom extensions

I'd want to start with a limited number of places to hook in custom extensions (like kubectl create x, kubectl set x). We also shouldn't differentiate between thirdpartyresources and other types unknown to kubectl (they should behave the same externally from the API)

The lookup scheme from @soltysh above looks okay to me. The only drawback is that the extensions needs to have the boilerplate to do nice output.
It would work in my case wher I'd like to output some useful informations for a TPR.

But that is a problem which can be addressed later by making the default kubectl output functionality reusable.

Is kubectl describe <resource-type> another case targeted here?

Is kubectl describe another case targeted here?

I don't think so... I think that was supposed to be addressed by enabling returning descriptive info about a resource from the server (#123)

For Helm, we backed off this approach because it got very confusing to the user. It was nearly impossible to tell whether a command was executing a plugin or not.

Envision this from the user's perspective. User A has command kubectl foo bar baz. User B tries to execute the same command, and it doesn't work. User A doesn't know which of their many installed plugins provides this behavior. How do they resolve the situation?

Our solution was to only expose a top level subcommand to plugins. This leads to a flat top-level namespace, but also makes it very clear which plugin is being invoked.

As a side benefit, it made the help system much simpler.

pwittrock · 2016-12-14T21:18:26Z

contributors/design-proposals/kubectl-extension.md

+
+All kubectl command extensions MUST:
+
+* Support the `-h` and `--help` flags to display a help page


We could probably make extensions authors live's easier by refactoring kubectl and providing pieces of it as libraries.

We made this same requirement, and also used the plugin metadata file to load help text into helm at runtime. The consistency is a HUGE boon to users. 👍

We also required that certain flags (like --debug) follow the same conventions as Helm.

Implementation-wise, what we actually did was parse these flags in the top-level binary, swallow the flags, and then converted them into environment variables. The benefit here (aside from being closer to Cobra's intent) is that the top-level CLI enforces consistency and the actual toggling of flags on and off is an implementation detail:

https://github.com/kubernetes/helm/blob/master/docs/plugins.md#a-note-on-flag-parsing

pwittrock · 2016-12-14T21:24:04Z

contributors/design-proposals/kubectl-extension.md

+
+All kubectl command extensions SHOULD:
+
+* Follow the display and output conventions of normal kubectl commands.


There will probably be some details to work out around how extension versioning is handled:

different versions of the extensions themselves (are these versioned independently from k8s versions?)

extension kubernetes client (e.g. golang client) version vs server version skew

extension kubernetes client version vs kubectl version skew

extensions version should be independently from k8s version.

But may have a dependency on the k8s version? e.g. "requires 1.7"?

Yeah I think support defining compatible k8s version is must have. Like >1.4.0 & <1.5.0 (or other syntax).

this is definitely necessary, extension plugin should be able to define k8s version requirement, and itself version should be independent.

@adohe indeed.

pwittrock · 2016-12-14T21:27:15Z

contributors/design-proposals/kubectl-extension.md

+location on disk, like Git.  Allow third parties to extend kubectl by placing their extensions in that directory. Ensure that help
+and other logic correctly includes those extensions.
+
+A kubectl command extension would be an executable located in `EXEC_PATH` (an arbitrary directory to be defined that follows similar


Non-goal: Provide extensions as new flags to non-extension commands. E.g. kubectl get pod -o <my-own-extension-format>

Another non-goal that is often requested: new auth mechanisms

Hrm - this one comes up a lot, simply because we have no way other than "recompile the world". Are you just skeptical of how we can solve this cleanly?

technosophos · 2017-01-19T20:18:31Z

contributors/design-proposals/kubectl-extension.md

+
+A kubectl command extension would be an executable located in `EXEC_PATH` (an arbitrary directory to be defined that follows similar
+conventions in Linux) with a name pattern like `kubectl-COMMAND[-SUBCOMMAND[...]]` with one or many sub parts. The presence of
+a command extension overrides any built in command.


Speaking for Helm, we started with a similar naming convention and later backed off. While it worked for Git, we felt like placing requirements on the naming convention of the plugin executable was limiting to plugin authors. It also required lots of name munging in code.

So we went from that requirement to having a single source location in $HELM_HOME (analogous to ~/.kube where plugin metadata could go. The metadata included information on the name of the actual binary.

That model got us the additional benefit of being able to read in the metadata file in order to generate top-level help text. In that way, helm -h could read the Usage: field in the plugin metadata for each plugin, and then display plugin help text alongside built-in help text.

The presence of a command extension overrides any built in command.

This seems like asking for trouble.

Letting plugins override core commands is very fragmenting (examples wouldn't work consistently)

Multiple plugins could target overriding the same core command

I would expect commands like kubectl get and kubectl describe to be high-value targets for plugins trying to add in pretty-printing for their own types, and those are extremely difficult to wrap accurately

Even if plugins are trying to behave well and pick names that don't conflict, future kubectl additions could be rendered inert by a plugin that had claimed the name for itself

It makes support really tough ("run kubectl get foo. wait, what plugins do you have installed?")

technosophos · 2017-01-19T20:50:03Z

I like this approach. Based on Helm's implementation, we have seen people build plugins ranging from simple (more like Git aliases) to highly complex (multi-subcommand Python or Go programs). And I think kubectl will benefit from an even richer plugin ecosystem than we've seen.

If there are any questions about the pros and cons of Helm's approach, what we've learned, or how we did particular things, I'm happy to answer.

bgrant0607 · 2017-03-03T18:55:39Z

ref kubernetes/kubernetes#13606

pwittrock · 2017-04-05T15:49:48Z

@smarterclayton @monopole

WDYT of merging this document as-is, and updating it to reflect changes to design as needed?

pwittrock · 2017-04-05T20:56:06Z

@fabianofranz

WDYT of merging this document as-is, and updating it to reflect changes to design as needed?

fabianofranz · 2017-04-05T21:17:22Z

@pwittrock works for me.

monopole · 2017-04-06T22:03:43Z

@pwittrock works for me too.

However, along with @liggitt, i wouldn't mind seeing short verbiage about command name collisions, e.g.

Plugins with a command name that collides with a previously loaded command name
(native or from some other plugin) will not be loaded. In later kubectl releases, new native
commands may mean plugins that once loaded will no longer load. Avoid the need
to rename by namespacing plugin command names. E.g. if Acme Co. writes a plugin that
gets something, specify the command name acme-get rather than simply get.

But also fine with it being added later. Plugins are well plowed ground (git, helm etc.) and see no reason to do anything more exotic here.

Automatic merge from submit-queue kubectl binary plugins **What this PR does / why we need it**: Introduces the ability to extend `kubectl` by adding third-party plugins that will be exposed through `kubectl`. Plugins are executable commands written in any language. To be included as a plugin, a binary or script file has to 1. be located under one of the supported plugin path locations: 1.1 `~/.kubectl/plugins` dir 1.2. one or more directory set in the `KUBECTL_PLUGINS_PATH` env var 1.3. the `kubectl/plugins` dir under one or more directory set in the `XDG_DATA_DIRS` env var, which defaults to `/usr/local/share:/usr/share` 2. in any of the plugin path above, have a subfolder with the plugin file(s) 3. in the subfolder, contain at least a `plugin.yaml` file that describes the plugin Example: ``` $ cat ~/.kube/plugins/myplugin/plugin.yaml name: "myplugin" shortDesc: "My plugin's short description" command: "echo Hello plugins!" $ kubectl myplugin Hello plugins! ``` ~~In case the plugin declares `tunnel: true`, the plugin engine will pass the `KUBECTL_PLUGIN_API_HOST` env var when calling the plugin binary. Plugins can then access the Kube REST API in "http://$KUBECTL_PLUGIN_API_HOST/api" using the same context currently in use by `kubectl`.~~ Test plugins are provided in `pkg/kubectl/plugins/examples`. Just copy (or symlink) the files to `~/.kube/plugins` to test. **Which issue this PR fixes**: Related to the discussions in the proposal document: #30086 and kubernetes/community#122. **Release note**: ```release-note Introduces the ability to extend kubectl by adding third-party plugins. Developer preview, please refer to the documentation for instructions about how to use it. ```

Proposal: kubectl extension

Proposal: kubectl extension

4323d90

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 1, 2016

smarterclayton mentioned this pull request Dec 1, 2016

Add a proposal for kubectl extension kubernetes/kubernetes#30086

Closed

rootfs reviewed Dec 2, 2016

View reviewed changes

soltysh reviewed Dec 5, 2016

View reviewed changes

pwittrock reviewed Dec 14, 2016

View reviewed changes

pigmej mentioned this pull request Dec 21, 2016

Better CLI UX Mirantis/k8s-AppController#156

Open

bgrant0607 assigned fabianofranz and pwittrock Jan 18, 2017

technosophos reviewed Jan 19, 2017

View reviewed changes

pigmej mentioned this pull request Feb 21, 2017

Way to describe dependency/definition content from kubectl Mirantis/k8s-AppController#16

Open

bgrant0607 mentioned this pull request Mar 3, 2017

kubectl plugins kubernetes/kubernetes#13606

Closed

fabianofranz mentioned this pull request Mar 16, 2017

kubectl binary plugins kubernetes/kubernetes#37499

Merged

liggitt mentioned this pull request Apr 5, 2017

add kubectl subcommand to issue/revoke keystone tokens kubernetes/kubernetes#41179

Closed

pwittrock merged commit 65f5293 into kubernetes:master Apr 5, 2017

pmorie mentioned this pull request May 8, 2017

Make it possible to add porcelain commands to kubectl for service-catalog kubernetes-retired/service-catalog#806

Closed

bryk mentioned this pull request Jun 2, 2017

Support for user-customizeable links in Pods, Service and StatefulSets through Annotations kubernetes/dashboard#1989

Open

surajssd mentioned this pull request Jun 16, 2017

Hook into kubectl commandline as extension kedgeproject/kedge#28

Open

MadhavJivrajani pushed a commit to MadhavJivrajani/community that referenced this pull request Nov 30, 2021

Merge pull request kubernetes#122 from smarterclayton/kubectl_extension

93ff61d

Proposal: kubectl extension

danehans pushed a commit to danehans/community that referenced this pull request Jul 18, 2023

Added title to org membership form (kubernetes#122)

69bd005


		All kubectl command extensions MUST:

		* Support the `-h` and `--help` flags to display a help page


		All kubectl command extensions SHOULD:

		* Follow the display and output conventions of normal kubectl commands.

Proposal: kubectl extension #122

Proposal: kubectl extension #122

Conversation

smarterclayton commented Dec 1, 2016

pmorie commented Dec 2, 2016

deads2k commented Dec 2, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rootfs commented Dec 2, 2016

bgrant0607 commented Dec 2, 2016

fabianofranz commented Dec 2, 2016 • edited Loading

fabianofranz commented Dec 2, 2016 • edited Loading

davidopp commented Dec 4, 2016

pmorie commented Dec 4, 2016 via email

adohe-zz commented Dec 4, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pigmej Dec 15, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

technosophos commented Jan 19, 2017

bgrant0607 commented Mar 3, 2017

pwittrock commented Apr 5, 2017

pwittrock commented Apr 5, 2017

fabianofranz commented Apr 5, 2017

monopole commented Apr 6, 2017 • edited Loading

fabianofranz commented Dec 2, 2016 •

edited

Loading

fabianofranz commented Dec 2, 2016 •

edited

Loading

pigmej Dec 15, 2016 •

edited

Loading

monopole commented Apr 6, 2017 •

edited

Loading