Let the commands store flagComp functions internally (and avoid global state) #2012

maxlandon · 2023-08-08T02:21:15Z

Problem

Currently, flag completion functions are a global map, preventing garbage collection
to work properly, especially if the commands using these funcs are garbage-collected themselves.
This might happen if commands are declared, bound and used through yielder functions.

This is also produce more coherent API usage in different cases:

Declare flag completions close to where you declare the flags themselves,
When a code path happens to have access to a given command, it knows how to bind flags to it (in a more dynamic manner, for that respect).

In addition, and while in theory nothing prevents a given command cmd1 to be called with
RegisterFlagsCompletion(cmd2Flag), and still store and work correctly with another command
flag.

Changes

Add the old flagCompletionFunctions map and its mutex as unexported fields of the Command.

CLAassistant · 2023-08-08T02:21:20Z

All committers have signed the CLA.

marckhouzam · 2023-08-08T11:48:39Z

Thanks @maxlandon for trying to improve things. However, this is actually how the original implementation tried to do it in #1423 which broke flag registration as described in #1437 and fixed in #1438

maxlandon · 2023-08-08T13:03:21Z

Thanks for pointing those to me.
After examination and comparing with my approach, it seems that there is a very small - but very important - difference between the failed attempts. See following/preceding code comments.
Sincerely,

maxlandon · 2023-08-08T13:09:39Z

@marckhouzam thanks for the quick reply in any case, I see that you guys are very busy on this repo.

I am also sorry for unearthing this thing, and doubly sorry to insist on the need for this to be solved out a way or another.
This PR has been opened in relation to a larger aim, which is to be able to use the carapace engine as a library, and the latter project is on track to provide exceptional completion possibilities and cross-shell portability.

I will open a thread dedicated to this, to expose more things in detail about that.

In any case, I would be very, very, very grateful if it was possible to gather a little bit of attention once again on the present issue.

Sincerely,

Luap99

This approach work for most cases but it will not work for persistent flags right now.
If you have a persistent flag defined on the root cmd you do not get that completion on a child command. In order to support that you would need to walk the tree up to root command when you lookup the completion function.

command.go

maxlandon · 2023-09-14T17:51:25Z

True, I completely forgot this initialization problem.

I will look at it this weekend, but normally I just need to add initialization in the related functions.

maxlandon · 2023-09-30T16:32:47Z

Normally the initialization problem should be fixed with the last commit I pushed.

Not sure how to solve for persistent root command' flags, but working on this.

maxlandon · 2023-09-30T18:52:47Z

Hello @marckhouzam, sorry to ping but I think this is very urgent:
I forgot to warn when you merged #1943 that we are still actually trying to avoid global state, and the corresponding PR did not take this into account.

So in this PR and in the last 3-4 commits, the following changes/fixes have been made:

Flag comp registration & getters are accessible through any command as a method, and when not found on it, its parents (if any) are looked up recursively. The function signatures don't change at all, they are just methods now.
Ensure flag completion maps are always initialized before use.
Merged with cobra master latest.

You would be very, very kind to review and/or test this as quick as possible, so that we can avoid too many people using the global flag comp getters in their code.
Thanks a lot !

Related to #2019
Also pinging @Luap99 , since you seem to be very in touch with what should and what should not be done. It seems you are able to verify very quickly if recursive/parent-root flag completions do work in this setup or not.
Thanks for any help you'll bring on this !

Ping me ruthlessly if needed for anything, at any time.

completions.go

command.go

completions.go

marckhouzam · 2023-10-14T01:51:02Z

I missed this. Sorry.
We’re going to have to look at this before the next release

completions.go

marckhouzam · 2023-10-22T20:44:33Z

@maxlandon This does not compile

maxlandon · 2023-10-22T22:56:32Z

@marckhouzam i'm going to fix this later today and will ping back

marckhouzam · 2023-10-23T14:08:07Z

@maxlandon I have added extra tests in #2053 to protect us against regressions.
Please make sure your changes pass on top of #2053.

maxlandon · 2023-10-23T14:09:44Z

@maxlandon I have added extra tests in #2053 to protect us against regressions.

Please make sure your changes pass on top of #2053.

Thanks a lot @marckhouzam , could you compile the code in this PR ? Or do you still need a fix ?

Away from my laptop today unfortunately, but can be available to help for the rest of the week and beyond.

Edit: I can take care of merging the various branches on which tests have been added, as well any other for that matter.

Edit 2: I can probably do the whole thing in a new PR, with everything gathered/fixed/documented in one place.
Branch name like "completion-state-no-global"

marckhouzam · 2023-10-23T14:25:55Z

Thanks a lot @marckhouzam , could you compile the code in this PR ? Or do you still need a fix ?

I got it to compile (see small diff at bottom [*]), but I get a panic running the tests.

Edit: I can take care of merging the various branches on which tests have been added, as well any other for that matter.

Just rebase your PR on top of #2053 as two commits.

Edit 2: I can probably do the whole thing in a new PR, with everything gathered/fixed/documented in one place. Branch name like "completion-state-no-global"

No need for a new PR. Please just squash all your changes into one commit and force-push.

[*] patch to fix compilation:

diff --git a/completions.go b/completions.go
index 0ee4f28..c566268 100644
--- a/completions.go
+++ b/completions.go
@@ -190,10 +190,8 @@ func (c *Command) initializeCompletionStorage() {
 		c.flagCompletionMutex = new(sync.RWMutex)
 	}

-	var completionFn func(cmd *Command, args []string, toComplete string) ([]string, ShellCompDirective)
-
 	if c.flagCompletionFunctions == nil {
-		c.flagCompletionFunctions = make(map[*flag.Flag]completionFn, 0)
+		c.flagCompletionFunctions = make(map[*flag.Flag]func(*Command, []string, string) ([]string, ShellCompDirective), 0)
 	}
 }

marckhouzam · 2023-10-23T14:30:53Z

@maxlandon Can you clarify (I still don't understand, sorry), why we need to avoid global state.

Currently, flag completion functions are a global map, preventing garbage collection to work properly, especially if the commands using these funcs are garbage-collected themselves. This might happen if commands are declared, bound and used through yielder functions.

Ok, that make sense, but it is not dramatic, I think.

This is also produce more coherent API usage in different cases:

Declare flag completions close to where you declare the flags themselves,

This is already how it is done. A developer registers the flag completion right next to the flag declaration. They don't know how the registration is stored in cobra.

When a code path happens to have access to a given command, it knows how to bind flags to it (in a more dynamic manner, for that respect).

I don't get this one. Can you give an example.

In addition, and while in theory nothing prevents a given command cmd1 to be called with RegisterFlagsCompletion(cmd2Flag), and still store and work correctly with another command flag.

I can't grok that sentence. Do you want to be able to do this, or are you trying to avoid it?
Currently, it can happen but only if cmd1 has a flag also named cmd2Flag.

maxlandon · 2023-10-23T15:07:53Z

@marckhouzam below is the main, summed up reason for this, followed by the 3 points aiming to illustrate /clarify your last reply.

Main reason: in short, having global state (for completions or else) is in effect making an assumption about the lifecycle of commands and their completions. This assumption is impractical, useless and risky, while not being improving clarity of code related to our present matter.

1) Garbage collection (ex: closed loop applications)
In the current state of things, it is always assumed that commands will be declared/allocated/executed only once in the lifetime of the program (traditional, single-exec CLI).
However, if you consider this console library and the way it uses yielder functions to cobra commands (for execution and completion, you will notice that having global flagcomp maps is preventing correct garbage collection.
See this section of the documentation for a quick example.
See also this issue for another example where this is a problem (and related to the other carapace proposal of mine)

2) Flags compFuncs binding close to command declaration
Yes it's true that people will bind the compFuncs close to the command declaration. So consider the following:

// Current (global state)
cmd := cobra.Command{}
cobra.RegisterFlagCompletion(myFunc)

// Without global state, directly through the command method.
cmd := cobra.Command{}
cmd.RegisterFlagCompletion(myFunc)

From above you can see that:

Point 1) is solved
We don't need to use the cobra package reference for binding our compFunc.
See 3) for how this is clearer on more complex scenarii.

3) Parent/child commands and their compFuncs
The advantage of binding compFuncs directly to commands is that we get a clearer/more granular way of binding completions so specific commands. Easier to read, and no impact on the completions lookup when these are being served (because the flagComp getters will look up to the command's parent when it does not have the target compFunc).

Consider this:

// root command has a flag and its completion.
parent := cobra.Command{}
parent.RegisterFlagCompletion("myflag", flagFuncParent)

// Child command has other flags and completions
child := cobra.Command{}
child.RegisterFlagCompletion("childflag", childCompFunc)

// No let's query the compfuncs
parentComp := child.GetFlagComp("myflag") // recursively looks up to the parent until found
childComp := child.GetFlagComp("childflag") // immediately found

notFoundComp := parent.GetFlagComp("childflag") // not working since lookup is always up the tree. This is conforming with how cobra looks up the command trees when devising what to complete.

4) the sentence you can't grok
Forget about this, the three points above are meant to clarify everything. Hopefully they do...

maxlandon · 2023-10-23T16:32:18Z

Thanks a lot @marckhouzam , could you compile the code in this PR ? Or do you still need a fix ?

I got it to compile (see small diff at bottom [*]), but I get a panic running the tests.

Edit: I can take care of merging the various branches on which tests have been added, as well any other for that matter.

Just rebase your PR on top of #2053 as two commits.

Edit 2: I can probably do the whole thing in a new PR, with everything gathered/fixed/documented in one place. Branch name like "completion-state-no-global"

No need for a new PR. Please just squash all your changes into one commit and force-push.

[*] patch to fix compilation:
diff --git a/completions.go b/completions.go

index 0ee4f28..c566268 100644

--- a/completions.go

+++ b/completions.go

@@ -190,10 +190,8 @@ func (c *Command) initializeCompletionStorage() {

 		c.flagCompletionMutex = new(sync.RWMutex)

 	}



-	var completionFn func(cmd *Command, args []string, toComplete string) ([]string, ShellCompDirective)

-

 	if c.flagCompletionFunctions == nil {

-		c.flagCompletionFunctions = make(map[*flag.Flag]completionFn, 0)

+		c.flagCompletionFunctions = make(map[*flag.Flag]func(*Command, []string, string) ([]string, ShellCompDirective), 0)

 	}

 }

I did not see this message (i'm on my phone atm). Will do as indicated !

maxlandon · 2023-11-02T13:28:45Z

Hello @marckhouzam , I was actually working on this at this moment !

So to "answer your answer":
1 - Perfect, glad to have made my point successfully.
2 - Yes indeed, the RegisterFlagCompletion is a method using the global map, and therefore we're about to change the global map from a per-command one.
3 - Quite the same as 2), and the present/to-come changes are about recursively checking commands/parents flagComp maps.

More commits to solve all of this in a few minutes/hours !

maxlandon · 2023-11-02T13:35:20Z

Actually, to answer your third point (GetCompletionFlagByName() methods):
You will notice that in the present state of things we have one method and one function:

Global function, querying the global map thanks to a flag pointer:

cobra/completions.go

Line 149 in 890302a

    
           func GetFlagCompletion(flag *pflag.Flag) (func(cmd *Command, args []string, toComplete string) ([]string, ShellCompDirective), bool) {

A command method, querying the global map thanks to a flag name:

cobra/completions.go

Line 158 in 890302a

    
           func (c *Command) GetFlagCompletionByName(flagName string) (func(cmd *Command, args []string, toComplete string) ([]string, ShellCompDirective), bool) {

So with the changes in the current PR, the global function GetFlagCompletion(flag *pflag.Flag) becomes a method of the command, so that in effect people have two different ways of querying compFuncs.
Internally, the logic is the same for both the GetFlagCompletion(pointer) and GetFlagCompletionByName(): they will recursively walk up their parents' tree until the corresponding flagCompFunc is found.

code.

maxlandon · 2023-11-02T14:00:32Z

Okay normally this should be good. Summarizing the work done and steps taken below:

Merge the main branch into this one, to get all new completion test code.
Change GetFlagCompletion(flagPtr *flag.Flag) to a method of the Command type.
For both GetFlagCompletion(ptr) and GetFlagCompletionByName(string), recursively walk the parents until the corresponding flagCompFunc is found. Return it along with a boolean.
Add some internal utility functions for ensuring all private Command maps/mutexes are initialized before being used.
Adapt the cmd.getCompletions() function (used when the __complete command is called) to actually use the cmd.GetFlagCompletion(ptr) method, which works in way consistent with how cobra traverses/looks up the command tree when devising what to complete.
Comment all the new functions, internal or external-facing.
Pass all tests.

Will pinpoint these numbers in the commits code, if you want to have visual confirmation for all of them.

completions.go

maxlandon · 2023-11-02T14:06:43Z

completions.go

-func GetFlagCompletion(flag *pflag.Flag) (func(cmd *Command, args []string, toComplete string) ([]string, ShellCompDirective), bool) {
-	flagCompletionMutex.RLock()
-	defer flagCompletionMutex.RUnlock()
+func (c *Command) GetFlagCompletion(flag *pflag.Flag) (func(cmd *Command, args []string, toComplete string) ([]string, ShellCompDirective), bool) {


Change this to a command method.

maxlandon · 2023-11-02T14:07:56Z

completions.go

-	completionFunc, exists := flagCompletionFunctions[flag]
-	return completionFunc, exists
+	// Or walk up the command tree.
+	return c.Parent().GetFlagCompletion(flag)


Recursively walk up the command parents' tree until flagComp is found.
Behavior identical regardless of if the flagComp is searched with a flag pointer, or a flag name.

completions.go

maxlandon · 2023-11-02T14:08:59Z

completions.go

-		flagCompletionMutex.RLock()
-		completionFn = flagCompletionFunctions[flag]
-		flagCompletionMutex.RUnlock()
+		completionFn, _ = finalCmd.GetFlagCompletion(flag)


Use our new recursive methods to lookup what to complete when __complete is called.
Passes all tests

marckhouzam · 2023-11-02T14:26:41Z

Thanks for the update @maxlandon !
Following the issues you describe here I have posted #2063 to fix the API of #1943 (and fix a bug).
This should avoid us painting ourselves in a corner while waiting for this PR here to get merged. You may want to have a look to confirm it addresses your immediate worries about propagating a global state.

As a release might be cut in the next hours, I prefer to play it safe and start with #2063 to avoid rushing your PR (which is more complicated and requires a more detailed review, as you saw from the repeated required rework.)

maxlandon · 2023-11-02T14:29:31Z

Thanks for the update @maxlandon ! Following the issues you describe here I have posted #2063 to fix the API of #1943 (and fix a bug). This should avoid us painting ourselves in a corner while waiting for this PR here to get merged. You may want to have a look to confirm it addresses your immediate worries about propagating a global state.

As a release might be cut in the next hours, I prefer to play it safe and start with #2063 to avoid rushing your PR (which is more complicated and requires a more detailed review, as you saw from the repeated required rework.)

Doing the review on the other PRs at the moment.
In the meantime may I request @jpmcb review on this one as well ?
I feel this PR is not that complicated, as shown by the number of additions/deletions.

I would still pretend that it is better to review/accept/reject this one before any release.
But I might be overly pretentious here...

maxlandon · 2023-11-02T14:41:14Z

@marckhouzam could you just allow the workflows to run once on this one ? just to make sure I'm not defending something that doesn't pass the bar :) !
Thanks a lot

EDIT: Nevermind, just saw that you ran them 15 minutes ago !

jpmcb · 2023-11-02T15:07:44Z

In the meantime may I request @jpmcb review on this one as well ?

Thanks for the ping: I'd be happy to help get this through. Looking now

jpmcb

Overall, this looks solid. I'm also going to spend some time testing this against kubectl and it's completions to ensure we're not missing / breaking anything

marckhouzam · 2023-11-02T15:23:35Z

@maxlandon can you rebase please

github-actions · 2023-11-02T16:23:19Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

moment.

github-actions · 2023-11-02T16:44:13Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

maxlandon · 2023-11-02T16:46:48Z

@marckhouzam (and @jpmcb for that matter here)

Even better than a rebase, I renamed GetFlagCompletionByName() func to GetFlagCompletionFuncByName() for the sake of consistency, and suddenly, running the tests raised a significant logic mistake in the way this function would perform the lookup.

Now everything is solved, logic is good for both functions.
Following is the comment of this new little change.

Hopeful to see this go through ! I appreciate the attention you all have given to this thing recently ! Thanks a lot !

maxlandon · 2024-08-09T04:03:43Z

Hello @marckhouzam, any chances to see this merged soon ?

Don't use global map for flag completions.

f1f260e

maxlandon mentioned this pull request Aug 8, 2023

API: Add a CacheF() function to control completion cache refresh carapace-sh/carapace#887

Closed

maxlandon mentioned this pull request Aug 19, 2023

Proposal: Freeze the completion API #2019

Open

Luap99 reviewed Sep 14, 2023

View reviewed changes

command.go Show resolved Hide resolved

command.go Show resolved Hide resolved

Ensure all completion maps are initialized before use

53fb4eb

Merge master, and adapt flag completion getters

4d3a6e1

github-actions bot added the area/github For changes to Github specific things not shipped in the library label Sep 30, 2023

Merge branch 'main' into completion

d6df128

github-actions bot removed the area/github For changes to Github specific things not shipped in the library label Sep 30, 2023

Avoid infinitely walking up the command tree

5dad425

maxlandon commented Oct 4, 2023

View reviewed changes

completions.go Outdated Show resolved Hide resolved

command.go Show resolved Hide resolved

command.go Show resolved Hide resolved

completions.go Outdated Show resolved Hide resolved

marckhouzam added this to the 1.8.0 milestone Oct 14, 2023

marckhouzam reviewed Oct 14, 2023

View reviewed changes

completions.go Outdated Show resolved Hide resolved

marckhouzam mentioned this pull request Oct 23, 2023

Add tests for flag completion registration #2053

Merged

Merge branch 'main' into completion

13d042d

maxlandon added 2 commits November 2, 2023 14:39

Fix compilation issues and remove commented code

340cb4c

Pass all tests by using correct flagComp lookups in completion command

69578f2

code.

maxlandon commented Nov 2, 2023

View reviewed changes

completions.go Show resolved Hide resolved

maxlandon commented Nov 2, 2023

View reviewed changes

completions.go Show resolved Hide resolved

maxlandon commented Nov 2, 2023

View reviewed changes

completions.go Show resolved Hide resolved

maxlandon commented Nov 2, 2023

View reviewed changes

maxlandon mentioned this pull request Nov 2, 2023

Improve API to get flag completion function #2063

Merged

Remove useless duplicate import

1548134

jpmcb reviewed Nov 2, 2023

View reviewed changes

maxlandon added 2 commits November 2, 2023 16:20

Merge branch 'main' into completion

e7f1370

Rename GetFlagCompletionFuncByName() for flag names lookup

f5848b6

Fix a logic lookup bug that was kind enough to surface at the good

5aadb0d

moment.

marckhouzam modified the milestones: 1.8.0, 1.9.0 Nov 15, 2023

marckhouzam removed this from the 1.8.1 milestone Nov 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Let the commands store flagComp functions internally (and avoid global state) #2012

Let the commands store flagComp functions internally (and avoid global state) #2012

maxlandon commented Aug 8, 2023

CLAassistant commented Aug 8, 2023 •

edited

Loading

marckhouzam commented Aug 8, 2023

maxlandon commented Aug 8, 2023

maxlandon commented Aug 8, 2023

Luap99 left a comment

maxlandon commented Sep 14, 2023

maxlandon commented Sep 30, 2023

maxlandon commented Sep 30, 2023 •

edited

Loading

marckhouzam commented Oct 14, 2023

marckhouzam commented Oct 22, 2023

maxlandon commented Oct 22, 2023 •

edited

Loading

marckhouzam commented Oct 23, 2023

maxlandon commented Oct 23, 2023 •

edited

Loading

marckhouzam commented Oct 23, 2023

marckhouzam commented Oct 23, 2023

maxlandon commented Oct 23, 2023 •

edited

Loading

maxlandon commented Oct 23, 2023

maxlandon commented Nov 2, 2023

maxlandon commented Nov 2, 2023

maxlandon commented Nov 2, 2023

maxlandon Nov 2, 2023

maxlandon Nov 2, 2023

maxlandon Nov 2, 2023

marckhouzam commented Nov 2, 2023

maxlandon commented Nov 2, 2023

maxlandon commented Nov 2, 2023 •

edited

Loading

jpmcb commented Nov 2, 2023

jpmcb left a comment

marckhouzam commented Nov 2, 2023

github-actions bot commented Nov 2, 2023

github-actions bot commented Nov 2, 2023

maxlandon commented Nov 2, 2023

maxlandon commented Aug 9, 2024

Let the commands store flagComp functions internally (and avoid global state) #2012

Are you sure you want to change the base?

Let the commands store flagComp functions internally (and avoid global state) #2012

Conversation

maxlandon commented Aug 8, 2023

Problem

Changes

CLAassistant commented Aug 8, 2023 • edited Loading

marckhouzam commented Aug 8, 2023

maxlandon commented Aug 8, 2023

maxlandon commented Aug 8, 2023

Luap99 left a comment

Choose a reason for hiding this comment

maxlandon commented Sep 14, 2023

maxlandon commented Sep 30, 2023

maxlandon commented Sep 30, 2023 • edited Loading

marckhouzam commented Oct 14, 2023

marckhouzam commented Oct 22, 2023

maxlandon commented Oct 22, 2023 • edited Loading

marckhouzam commented Oct 23, 2023

maxlandon commented Oct 23, 2023 • edited Loading

marckhouzam commented Oct 23, 2023

marckhouzam commented Oct 23, 2023

maxlandon commented Oct 23, 2023 • edited Loading

maxlandon commented Oct 23, 2023

maxlandon commented Nov 2, 2023

maxlandon commented Nov 2, 2023

maxlandon commented Nov 2, 2023

maxlandon Nov 2, 2023

Choose a reason for hiding this comment

maxlandon Nov 2, 2023

Choose a reason for hiding this comment

maxlandon Nov 2, 2023

Choose a reason for hiding this comment

marckhouzam commented Nov 2, 2023

maxlandon commented Nov 2, 2023

maxlandon commented Nov 2, 2023 • edited Loading

jpmcb commented Nov 2, 2023

jpmcb left a comment

Choose a reason for hiding this comment

marckhouzam commented Nov 2, 2023

github-actions bot commented Nov 2, 2023

github-actions bot commented Nov 2, 2023

maxlandon commented Nov 2, 2023

maxlandon commented Aug 9, 2024

CLAassistant commented Aug 8, 2023 •

edited

Loading

maxlandon commented Sep 30, 2023 •

edited

Loading

maxlandon commented Oct 22, 2023 •

edited

Loading

maxlandon commented Oct 23, 2023 •

edited

Loading

maxlandon commented Oct 23, 2023 •

edited

Loading

maxlandon commented Nov 2, 2023 •

edited

Loading