Refactor and unify source code analysis functionality

[jitseniesen]

This is only an idea that I wanted to record, prompted by #3491, and thus quite vague. There are the number of places where we need to parse Python code, including:

• automatic indentation
• automatic insertion of colon at end of line
• highlighting of matching brackets
• generating outline
• syntax highlighting (via Pygments)
• folding (when we implement that)

We do this sometimes ad hoc, leading to bugs (e.g.#1448, #1673, #3214) and duplication of effort. I think we should try to do all these things in one place. This may also help with making Spyder language agnostic.

A first step may be to do tokenizing in one place. Can we perhaps make use of Pygment's tokenizer?

[ccordoba12]

@jitseniesen, unfortunately we can't use Pygments to tokenize all file types because it needs to parse the entire file instead of doing it line by line, which is what QSyntaxHighlighter expects (and it's also more efficient for big files).

So for some file types (e.g. Python) we need to define our own tokenization routines.

---

About your other points, I'm all for them. Let's wait until @rlaverde finishes its refactoring of CodeEditor to discuss them :-)

[GauravGaikwad94]

Is code refactor feature available in spyder? @ccordoba12 @rlaverde

[rlaverde]

No, code refactoring isn't avalaible, @ccordoba12 is refering to an internal refactoring of spyder code (Editor extensions), about code refactoring feature you could take a look to #1937 and #4751

[goanpeca]

This is related to work I will probably implement in the plugin new API side. After all plugins have been migrated, I plan to create a Python plugin that should extend the base plugins with Python Specific functionality. In doing this I will probably achieve what @jitseniesen is proposing.