Refactor highlighting

[gwenn]

Introduce Style and StyledBlock traits
And their ansi-str implementations

• Make ansi-str optional
• Fix Highlighter trait
• Fix rendering accordingly
• Check that new API works with an hardcoded continuation prompt.

See #793, #372
See https://github.com/gwenn/rustyline-notes/blob/master/src/design.md

[zao111222333]

It seems that we can change the ansi_str::get_blocks form fn get_blocks(text: &str) to fn get_blocks<'s>(text: Cow<'s, str>). And then all those error here 6b34d32 will be solved.
(In this case we have to create a new crate rather than use ansi_str since I don't know the origin author's opinion with this change).
Shall I have a try on this approach and make a commit?

[zao111222333]

I find that once we define

fn highlight_line<'l>(&self, line: &'l str, pos: usize) -> impl Iterator<Item = impl 'l+StyledBlock>

The new highlight_line function signature conflicts to the Option<&dyn Highligher> due to https://doc.rust-lang.org/reference/items/traits.html#object-safety . So use generic <H: Highligher> + Option<&H> to replace Option<&dyn Highligher>

And I have maked a pull request, see #799

[gwenn]

Maybe we should not try to implement highlight_line for ansi_str.
See 92bcae0

[zao111222333]

Looks good! And how do you think Vec<(Self::Style, &'l str)> or impl Iterator<(Self::Style, &'l str)>?
The cons and pros of impl Iterator<(Self::Style, &'l str)>:
pros:

• More flexible, a superset of Vec since Vec::<Foo>::into_iter will return impl Iterator<Foo>
• Can be a lazy iterators (e.g., Map, FilterMap, etc.), since let v: Vec<_> = iter().map(...).collect() require a duplicate collect function with O(n)

cons:

• Not straightforward for beignner (compare to Vec)
• Conflicts to the Option<&dyn Highligher> due to https://doc.rust-lang.org/reference/items/traits.html#object-safety, but we can use <H: Highligher> + Option<&H> to overcome this issue

[gwenn]

Alternatives use Vec:
https://docs.rs/syntect/5.2.0/syntect/easy/struct.HighlightLines.html#method.highlight
https://docs.rs/reedline/latest/reedline/trait.Highlighter.html#tymethod.highlight + https://docs.rs/reedline/latest/reedline/struct.StyledText.html
https://docs.rs/termwiz/latest/termwiz/lineedit/trait.LineEditorHost.html#method.highlight_line

[gwenn]

Highlighter implementations are becoming messy.
See 9e5c0e1
Maybe we should remove the type Style associated type and support only () or anstyle::Style ?
Knowing that we also need to fix highlight_hint...

[zao111222333]

Thanks a lot for your hard working, I believe impl Iterator<Item = impl 'l + StyledBlock> is the best function signature. And what can I do for this feature?

[gwenn]

And what can I do for this feature?

See above:
[ ] Fix rendering accordingly
[ ] Check that new API works with an hardcoded continuation prompt.

[zao111222333]

I seem to find a decent way to finish the task "Check that new API works with an hardcoded continuation prompt",
with the pull request gwenn#3

[zao111222333]

After weeks of struggling, here is my solution, pls have a look at gwenn#3

[zao111222333]

Let me briefly summarize my commits

1. Use DisplayOnce as a consistent & back-compatible highlight output type

fn highlight<'b, 's: 'b, 'l: 'b>(
    &'s mut self,
    line: &'l str,
    pos: usize,
) -> impl 'b + DisplayOnce;

the back-compatiblity is that, the old highlight function even needn't to change their signatures, since we will auto-impl DisplayOnce for all types that impl Display. for example

fn highlight<'b, 's: 'b, 'l: 'b>(
    &'s mut self,
    line: &'l str,
    pos: usize,
) -> std::borrow::Cow<'b, str>

And it also supports split-highlight, since we impl DisplayOnce for StyledBlocks, a wrapper of StyledBlock iterator.

pub struct StyledBlocks<'l, B, I>
where
    B: 'l + StyledBlock,
    I: Iterator<Item = B>,
{
    iter: I,
    _marker: PhantomData<&'l ()>,
}

2. Reduce tokenizer/parser overhead
   I don't get the point why we need helper: Option<H> in Editor, and I change it to helper: H. Then we can avoid lifetime issue when using &mut self in highlight, validate, complete, ...
   as you have mentioned at
   https://github.com/gwenn/rustyline/blob/highlight/src/completion.rs#L93
   https://github.com/gwenn/rustyline/blob/highlight/src/lib.rs#L543
   Now the Helper becomes https://github.com/zao111222333/rustyline/blob/496d2767ef5168d1629ae7f7cc712f8e6d8000cf/src/lib.rs#L548-L584
   User can put the tokenizer/parser at update_after_edit, which will be called just after the line is modified, to update context. Then use the context in highlight, validate, complete, ...

fn update_after_edit(&mut self, line: &str, pos: usize, forced_refresh: bool)

And only one thing that user have to change is the Editor creating.
From

let helper = MyHelper::new();
let mut rl = Editor::new()?;
rl.set_helper(helper);

To

let helper = MyHelper::new();
let mut rl = Editor::new(helper)?;

And use () for default helper:

let mut rl = Editor::new(())?;

3. hardcoded continuation prompt
   User can implement the continuation prompt in highlight function to render continuation prompt. And then set the continuation prompt width for shifting cursor when editing, by Helper::continuation_prompt_width