Fixes #5291: add ImageView support with Sixel rendering and optimize re-rendering

[mrazza]

Creates a ImageView view and uses it in UICatalog.

This view takes no new dependencies in Terminal.Gui so it operates on raw Color[,] arrays. The performance for resizing these is poor. As a result, it may make sense to take an optional lambda to allow the constructor to provide a more efficient resizing implementation using their image library of choice.

Fixes

• Fixes General ImageView control and Sixel performance improvements #5291

Proposed Changes/Todos

Pull Request checklist:

• I've named my PR in the form of "Fixes #issue. Terse description."
• My code follows the style guidelines of Terminal.Gui - if you use Visual Studio, hit CTRL-K-D to automatically reformat your files before committing.
• My code follows the Terminal.Gui library design guidelines
• I ran dotnet test before commit
• I have made corresponding changes to the API documentation (using /// style comments)
• My changes generate no new warnings
• I have checked my code and corrected any poor grammar or misspellings
• I conducted basic QA to assure all features are working

[oca-agent]

Thanks for the PR! I have a few minor technical suggestions after reviewing the implementation:

1. Dictionary vs. ConcurrentDictionary: In ImageView.cs, _attributeCache uses a ConcurrentDictionary. Since rendering in Terminal.Gui typically occurs on the main UI thread, would a standard Dictionary be sufficient? This could slightly reduce overhead.

2. Scaling Logic Math: In ScaleNearestNeighbor, the calculation (double)y / newHeight * srcHeight works well, but for very frequent resizes or large images, switching to integer arithmetic might provide a small performance boost by avoiding double conversions.

3. Memory Management: During window resizes, GetScaledImage allocates new Color[,] arrays frequently. To reduce GC pressure in these scenarios, we might want to consider using an ArrayPool or a reusable buffer for the scaling operations in a future iteration.

Overall, the Sixel optimization logic and the new ImageView are great additions!

[BDisp]

I see some regression in the Images scenario:

• The "Supports Sixel" checkbox does not appear enabled even though the terminal supports it. In the develop branch it appears enabled.

• The dialog box does not overlap the Sixel image. The ImageView.cs should be responsible for rendering the Sixel image in its clipping area instead of OutputBase.cs. Normally, the drawing is processed by the runnable modal from the top of the stack to the last runnable, with the respective clipping areas already drawn being excluded in subsequent renders. At least, this is the new criterion used to avoid drawing areas that would later be redrawn again. Since this PR now has a new ImageView.cs control, it should be used to handle its own drawing.

• Another problem I've noticed is that it doesn't simultaneously display the output of Sixel and Fire, as the develop branch allows. In this PR, after clicking the "Output Sixel" button, the Sixel output appears, but when clicking the "Start Fire" button, the Sixel image disappears.

[mrazza]

Thank you!

I'm aware of the first two. I tested the last bullet earlier and did not experience it. Maybe I broke something recently; will take a look.

Are folks aligned with this direction? I can work to address all three of these (although the middle one is not a regression; it's the current behavior) if this seems like a reasonable direction. We should really move the rendering logic out of the ConcurrentQueue in OutputBase and into the view itself but I figured we could do that iteratively so I would maybe suggest ignoring the middle improvement for now.

[BDisp]

To ignore the middle one for now then just render the sixel after the dialog box is closed.

[Copilot]

Pull request overview

This PR introduces a new ImageView control for rendering Color[,] pixel buffers (cell-based fallback or sixel when available) and extends the driver/output pipeline to detect sixel support and avoid re-emitting sixel data every frame by tracking a dirty flag.

Changes:

• Added Terminal.Gui.Views.ImageView with scaling and sixel/cell-based rendering paths.
• Added driver-level sixel capability detection (IDriver.SixelSupport) and output-level SixelToRender.IsDirty/AlwaysRender behavior to skip redundant sixel writes.
• Updated UICatalog’s Images scenario and added/updated tests for the new behavior.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
Tests/UnitTestsParallelizable/Views/ImageViewTests.cs	New unit tests covering ImageView scaling/rendering and sixel helpers.
Tests/UnitTestsParallelizable/Drivers/Output/OutputBaseTests.cs	New tests verifying OutputBase.Write skips/clears dirty sixels and DriverImpl.SixelSupport storage.
Tests/UnitTestsParallelizable/Application/MainLoopCoordinatorTests.cs	Adds coverage around sixel detection startup behavior (but one test name/behavior mismatch).
Terminal.Gui/Views/Markdown/MarkdownView.Drawing.cs	Queues markdown sixels with AlwaysRender=true to keep them visible under the new dirty model.
Terminal.Gui/Views/ImageView.cs	New ImageView implementation with sixel/cell rendering and scaling.
Terminal.Gui/Drivers/Output/OutputBase.cs	Skips sixel writes when not dirty (unless AlwaysRender) and clears IsDirty after writing.
Terminal.Gui/Drivers/IDriver.cs	Adds SixelSupport to the public driver interface.
Terminal.Gui/Drivers/DriverImpl.cs	Implements SixelSupport storage and internal setter.
Terminal.Gui/Drawing/Sixel/SixelToRender.cs	Adds IsDirty and AlwaysRender flags.
Terminal.Gui/App/MainLoop/MainLoopCoordinator.cs	Runs sixel detection during driver initialization for non-legacy consoles.
Examples/UICatalog/Scenarios/Images.cs	Refactors scenario to use the new ImageView and driver-level sixel detection.

[mrazza]

@BDisp While I was not able to repro your third issue, I suspect it was caused by removing a sixel from the queue during Fire rendering. I've addressed that risk. This is more evidence that the ConcurrentQueue of sixels rendered at the end of the draw pipeline is very very hacky but hoping to punt a fix for that into a future PR. I also addressed the other two regressions you identified. Thanks!

Supports Sixel now correctly displays resolved value:

Dialog renders before image:

Fire renders at the same time as the sixel:

Also addressed copilot comments.

[BDisp]

The only regression I see now is that the sixel output is constantly redrawing, causing a lot of oscillation. This would be normal if the fire output were being drawn, but not just when the sixel is being displayed.

[mrazza]

The only regression I see now is that the sixel output is constantly redrawing, causing a lot of oscillation. This would be normal if the fire output were being drawn, but not just when the sixel is being displayed.

Interesting. Not noticing that on foot on Linux; maybe the redraw is so fast it's not noticeable or maybe something else is causing that area of the screen to not invalidate on my machine. Thanks for the powershell validation. Not sure what the solution is yet.

Importantly, it's not clear to me why this would be different. I suspect this is caused by something marking that area of the OutputBuffer as IsDirty and there being a delay between when it is re-rendered (rendering nothing -

Terminal.Gui/Terminal.Gui/Drivers/Output/OutputBase.cs

Line 109 in aa2c41b

if (!buffer.Contents! [row, col].IsDirty)

) and the Sixel fully flushing at the end of the draw cycle (

Terminal.Gui/Terminal.Gui/Drivers/Output/OutputBase.cs

Line 219 in aa2c41b

Write (new StringBuilder (s.SixelData));

). However, this existed before my change.

[mrazza]

I have an ugly solution involving reaching into the content buffer and marking the impacted cells as not dirty. I'll push this but I don't feel good about it. Something like this:

            // Mark the content buffer for the area we will draw as not dirty.
            // This will avoid redrawing the area of the screen that will
            // eventually be overwritten by the sixel anyway.
            if (ScreenContents is { } contents && Driver is { } driver)
            {
                for (int y = dirtyRect.Y; y < dirtyRect.Bottom; y++)
                {
                    for (int x = dirtyRect.X; x < dirtyRect.Right; x++)
                    {
                        if (x >= 0 && y >= 0 && x < driver.Cols && y < driver.Rows)
                        {
                            contents [y, x].IsDirty = false;
                        }
                    }
                }
            }

[mrazza]

As I struggle to repro this myself, @BDisp would appreciate if you could pull this down and give it a test.

[tznind]

I added metrics and internal logs when i was optimising layout and redraw.

Theres a markdown file on it.

You could add sixel redraws to it.

It used dotnet counters tool

[BDisp]

It's much better than the development branch now. The flickering is only visible on the left side due to the constant redrawing of the border.

At least the flickering isn't noticeable across the entire sixel image, as the develop branch below demonstrates. Thanks.

[BDisp]

Looking more closely at the first GIF, it's not just the left side that's flickering, but also the bottom side. I suspect it's not due to the border, but rather another reason I haven't yet been able to figure out. But it's worth investigating because it might even be a completely different problem unrelated to the changes in this PR. Do you agree?

[mrazza]

@BDisp Thank you! I could probably resolve the remaining flickering by scaling and positioning the image so that its bounds are on exact cell boundaries that I clear the IsDirty flag on... either by effectively shrinking the image within the Viewport or by expanding the IsDirty clearing by one cell in all directions. The former is likely preferred between these two options otherwise the borders would never draw. Even so, I'm not sure it's worth doing. The flickering is most common when doing rapid buffer refreshes like in the fire animation which may be uncommon and defaulting to rendering smaller images seems less than ideal (in particular if the renderable area is small; which is the case in my app --- constraining by an entire cell width/height will result in an extremely small image). I believe, in the situations where the flickering is problematic, a consumer could resolve this themselves by adding Padding.Thickness of (1, 1, 1, 1) to the ImageView.

TL;DR: This is an existing problem and I believe the flickering on the edges we're seeing is because the bounds of the sixel do not cleanly align with character Cells used when rendering on-sixel graphics and so portions of the sixel extend beyond the Cells I clear the IsDirty flag on (i.e. partially into another cell).

[tig]

Please update ImageSharp from 3.1.12 to 4.0.0 as part of this.

[mrazza]

It seems, as of 4.0, you are now required to have a license for ImageSharp?

https://sixlabors.com/pricing/#plans

/home/razza/.nuget/packages/sixlabors.imagesharp/4.0.0/build/SixLabors.ImageSharp.targets(27,5): error : No Six Labors license found. Set $(SixLaborsLicenseKey), set $(SixLaborsLicenseFile), or add a 'sixlabors.lic' file to the project/workspace. [/home/razza/sources/mrazza/Terminal.Gui/Examples/UICatalog/UICatalog.csproj]
/home/razza/.nuget/packages/sixlabors.imagesharp/4.0.0/build/SixLabors.ImageSharp.targets(27,5): error : Please obtain a license from https://sixlabors.com/pricing/

Starting with ImageSharp 4.0.0, projects that directly depend on ImageSharp require a valid Six Labors license at build time. This enforcement applies to direct dependencies only.

https://docs.sixlabors.com/articles/imagesharp/index.html?tabs=tabid-1#license
https://sixlabors.com/posts/licence-enforcement-changes/

Not sure what you want to do with that @tig

[tig]

I just applied for their oss license.

[mrazza]

Any next steps here?

[tig]

Any next steps here?

I think we have to wait until they provide the license. I just submitted a 2nd request.

[mrazza]

ACK. I got a license in less than a day. You should immediately see a 2 month license and within 24hrs I got a 1 year license in my email.