Debug library discussion

[JayFoxRox]

The current debug library has various issues:

• Only allows sending messages to the framebuffer.
• It doesn't support many display modes and needs to guess the internal framebuffer state (see vmCurrent uninitialized before XVideoSetMode call #149).
• Considers XVideoGetFB the framebuffer; this is bad. See XVideoInit never allocated the framebuffer memory that it advertises via XVideoGetFB #169 and XVideoGetFB can return outdated address #170.
• It is called from various parts of nxdk (audio driver, network driver, pbkit, legacy-fileio) and can't be disabled; given the XVideoGetFB issues, this causes random side-effects and random writes to memory that the user doesn't expect.
• It has dead code, such as constants for colored output. See debug.h pollutes namespace #171.
• Existing code goes out of its way to support debug.h, like pbkit having explicit functions to switch framebuffers between GPU (self-allocated) and debug.h (XVideoGetFB); even if done properly (2 allocated framebuffers) that just means a higher memory consumption and complexity.
• It's not a debug library, but a more generic "console-output" library.
• It ships 2 fonts (terminal + sasteroids2), but only 1 of them is used (terminal).
• The fonts can be hard to read on small / bad monitors.
• The paging sometimes hides valuable information and can be confusing.

All of this makes debug.h hacky and error prone.

---

Ideally we'll replace it with a dedicated debugging lib: debug/debug.h (or similar).

This new debug lib would provide tools for the user to install a local debugger (as supported by the kernel); potentially:

• Crash handlers (SIGSEGV handling).
• Debug input / output handler ( DbgPrompt / DbgPrint).
• Memory status watchdog (Kernel hooks for guarding allocations or warning about low-memory condition).
• Explicit API for debug input / output, in case the kernel debugger is not used.

Debugger output could be forwarded to callbacks, another kernel debugger, explicit serial connection, a file, or a character buffer.

If the new API also has a framebuffer implementation, an explicit function to redraw the framebuffer would probably be used instead (over implicitly updating); in a callback based API, the user could still decide to explicitly redraw on each message.

However, I'd argue that we don't even need a framebuffer implementation. If it's easy enough to retrieve information from the debug API, then the debug log could be rendered using pbkit, SDL or other code, from within the application.
- There could still be an independent "console-application" lib which replaces the existing framebuffer-based debug.h

[JayFoxRox]

For exposure and because it's slightly related: I found this catch handler for OutputDebugString on thrimbors repository: thrimbor@098e2fb