Context: add three new experimental retriever strategies

[hitesh-1997]

Context

The PR adds three major context sources useful for autocomplete and next-edit-suggestion.

Recent Copy retriever.

Developers often copy/cut and paste context and before pasting we show autocomplete suggestions to the user. The PR leverages the copied content on the clipboard by the user as a context source.
I wasn't able to find any vscode api event exposed which triggers when user copy or cut text in the editor. This seems like an open issue: microsoft/vscode#30066

Another alternative I think of is to use a keybinding of Ctrl+x and Ctrl+c, but not fully sure about its implications, since this is one of the most common shortcuts.

As a workaround, The way current PR accomplishes the same is:

1. Tracks the selection made by the user in the last 1 minutes and keeps tracks of upto 100 most recent selections.
2. At the time of retrieval, checks if the current clipboard content matches the selection items in the list.

Recent View Ports

This context source captures and utilizes the recently viewed portions of code in the editor. It keeps track of the visible areas of code that the developer has scrolled through or focused on within a specified time frame.

Diagnostics

The Diagnostics context source leverages the diagnostic information provided by VS Code and language servers. It collects and utilizes information about errors, for a file as a context source.

Test plan

1. Automated test

• Added CI tests for each of the retrievers

2. Manual test

• Override the setting "cody.autocomplete.experimental.graphContext": "recent-copy" in vscode settings.
• Observe the context events using Autocomplete Trace View

[valerybugakov]

Let's use an object here to make the constructor call readable new RecentCopyRetriever(60 * 1000, 100).

[hitesh-1997]

done

[valerybugakov]

Should we use > here to find outdates items?

Suggested change

	const firstOutdatedChange = this.trackedSelections.findIndex(selection => {
	return now - selection.timestamp < this.maxAgeMs
	})
	const firstOutdatedChange = this.trackedSelections.findIndex(selection => {
	return now - selection.timestamp > this.maxAgeMs
	})

[valerybugakov]

Let's add basic unit tests to account for such cases. Sonnet 3.5 can probably cover 90% of cases here.

[hitesh-1997]

Thanks for the suggestion Valery :)
Added the test case for all the retrievers.

[valerybugakov]

This assigns the removed element to trackedSelections instead of removing it. It should be:

Suggested change

	// Remove the selection from the tracked selections
	this.trackedSelections = this.trackedSelections.splice(existingSelectionIndex, 1)
	// Remove the selection from the tracked selections
	this.trackedSelections.splice(existingSelectionIndex, 1)

[hitesh-1997]

corrected

[valerybugakov]

Suggested change

	const selectionItem = this.getSelectionItemIfExist(clipboardContent)
	const selectionItem = this.trackedSelections.find(ts => ts.content === text)

[hitesh-1997]

done

[valerybugakov]

Do we need an async function here?

Suggested change

	private async onDidChangeTextEditorSelection(
	event: vscode.TextEditorSelectionChangeEvent
	): Promise<void> {
	const editor = event.textEditor
	this.addSelectionForTracking(editor.document, editor.selection)
	}
	private onDidChangeTextEditorSelection(event: vscode.TextEditorSelectionChangeEvent): void {
	const editor = event.textEditor
	this.addSelectionForTracking(editor.document, editor.selection)
	}

[hitesh-1997]

oh right, we dont need async, removed it

[valerybugakov]

Suggested change

	this.trackedSelections = []
	for (const disposable of this.disposables) {
	disposable.dispose()
	}
	}
	this.trackedSelections = []
	for (const disposable of this.disposables) {
	disposable.dispose()
	}
	this.disposables = []
	}

[hitesh-1997]

done

[olafurpg]

LGTM 👍🏻 Only have nitpick comments on the coding style

Do I understand correctly that these context sources are disabled by default, and they can only be configured to run one at a time (not mixed)? Either way, not a blocking comment, just unclear to me how we will mix these

[olafurpg]

nit: I personally avoid writing test cases with hardcoded line numbers like these. I find it's better to compute the position based on markers in the original file like this

            function foo() {
                console.log(<DIAGNOSTIC_START/>'foo'<DIAGNOSTIC_END/>)
            }

The benefit of doing it this way is that you can more easily refactor the input code while having the start/end positions update automatically for you.

My experience is that hardcoded numbers like these quickly become out of sync and it's a nightmare to update them.

[hitesh-1997]

that's a nice technique, will address in the follow up PR :)

[olafurpg]

nit: it's better to use toplevel functions instead of class methods when you don't access this inside the method. In this case, you can probably just inline the call to vscode.env.clipboard.readText()

[olafurpg]

Aaah, I see we're mocking it out in the tests, which is why it's a method here. Fine by me, but might be worth adding a comment explaining the motivation

[hitesh-1997]

added the comment

[olafurpg]

nit, see #5648

Suggested change

	import { debounce } from 'lodash'
	import debounce from 'lodash/debounce'

[hitesh-1997]

done

[olafurpg]

nit: when using string keys, it can help with clarity to explain what the keys actually are either in the variable name or with a comment

Suggested change

	private trackedViewPorts: Map<string, TrackedViewPort> = new Map()
	private viewportsByDocumentUri: Map<string, TrackedViewPort> = new Map()

[hitesh-1997]

done

[olafurpg]

Might be simpler to use LRUCache, which we are already using in the repo for tree-sitter trees.

Docs https://www.npmjs.com/package/lru-cache

[hitesh-1997]

ah yes, this is much simpler now, changed to lru cache.

[hitesh-1997]

LGTM 👍🏻 Only have nitpick comments on the coding style

Do I understand correctly that these context sources are disabled by default, and they can only be configured to run one at a time (not mixed)? Either way, not a blocking comment, just unclear to me how we will mix these

A great point Olaf.
I am logging the data collected for these context sources here in the PR: #5661
This will help us how to combine these context sources to improve the model.