docs(design): rewrite and expand architecture design documents

docs/en/design/command.md

@@ -0,0 +1,97 @@
+# Command Processing
+
+## Background
+
+A language server needs to know how to compile every file. This information comes from the compilation database (`compile_commands.json`), generated by build systems (CMake, Bazel, Meson, etc.). However, the raw compilation commands in the CDB cannot be fed directly to the Clang frontend for several reasons:
+
+**Compilation commands are driver-level, not frontend-level.** The CDB records the commands that the build system uses to invoke the compiler (e.g., `g++ -std=c++17 -O2 -c foo.cpp`), but the language server needs Clang frontend (cc1) arguments. Converting from driver commands to cc1 arguments requires querying the compiler toolchain to obtain the target triple, system header search paths, and other information.
+
+**Compilation commands contain semantically irrelevant options.** Build systems pass options that only affect code generation (e.g., `-fPIC`, `-fomit-frame-pointer`), which do not affect semantic analysis but add complexity. They also include build-artifact-related options (e.g., `-o foo.o`, `-emit-pch`), which are entirely meaningless to a language server.
+
+**Compilation commands lack implicit information.** System header paths, default language standards, compiler built-in macro definitions, and similar details do not appear explicitly in the CDB -- they are implicitly provided by the compiler toolchain. The language server must probe the toolchain to fill in this information.
+
+**Large projects have highly redundant compilation commands.** In a project with tens of thousands of files, the vast majority share the same compiler and semantic options (`-std=c++17`, `-Wall`, etc.), differing only in include paths and macro definitions. Without deduplication, this wastes significant memory and startup time. In clangd issues, users have reported Bazel-generated `compile_commands.json` files exceeding 10GB, with individual entries reaching 250KB.
+
+## Design
+
+### Overall Pipeline
+
+Command processing is a multi-stage pipeline:
+
+```text
+compile_commands.json (raw commands)
+    | load and parse
+Argument classification (codegen-only / discarded / user-content / semantic)
+    | two-level separation
+CanonicalCommand (shared semantic options) + Patch (per-file user content)
+    | toolchain probing
+cc1 arguments (complete frontend-consumable compilation command)
+    | search path extraction
+SearchConfig (four-tier header search paths)
+```
+
+### Argument Classification
+
+When loading the CDB, each compilation option is classified into one of four categories:
+
+**Discarded**: Options related to build artifacts that the language server does not need. Examples include `-o` (output file), `-c` (compilation mode), `-M` (dependency scanning), `-emit-pch` (PCH building), etc. These are simply dropped.
+
+**Codegen-only**: Options that only affect the code generation backend and do not affect semantic analysis. Examples include `-fPIC`, `-fomit-frame-pointer`, `-funwind-tables`, debug info option groups (`-g*`), etc. These do not change the AST or diagnostic output and are dropped. Note that `-O` and `-fsanitize=` may appear to be code-generation-related, but they define macros (such as `__OPTIMIZE__` and `__has_feature(address_sanitizer)`), so they are retained.
+
+**User-content**: Options that may differ per file but do not affect toolchain probing results. Primarily include paths (`-I`, `-isystem`, `-iquote`, `-idirafter`) and macro definitions (`-D`, `-U`). These are extracted as patches and reattached after toolchain probing.
+
+**Semantic**: All remaining options. These affect compilation semantics and play a role in toolchain probing. Examples include `-std=c++17`, `-Wall`, `-target`, `-march=`, etc.
+
+### Two-level Separation: Canonical and Patch
+
+After classification, the compilation command is split into two parts:
+
+- **CanonicalCommand**: The driver name + all semantic options. Represents "the semantic aspects of how to invoke the compiler."
+- **Patch**: All user-content options. Represents "what this particular file additionally needs."
+
+**Why this separation?**
+
+The core reason is **toolchain probing cache efficiency**. Toolchain probing requires actually invoking the compiler driver (e.g., `g++ -v`), which is an expensive operation (typically 100ms+). The probing result depends only on the driver and semantic options -- user-content options (`-I`, `-D`) do not affect the cc1 arguments output by the driver. Therefore, as long as the semantic options are the same, regardless of what different `-I` paths files may have, they can share the same probing result.
+
+In real projects, tens of thousands of files may have only 5-50 distinct CanonicalCommands. This means the toolchain only needs to be probed 5-50 times instead of tens of thousands.
+
+This also enables memory deduplication of compilation commands. Identical CanonicalCommand and Patch combinations are merged into the same CompilationInfo instance and shared via pointers.
+
+### Path Absolutization
+
+During the loading phase, all include path options in user content are absolutized -- if a path is relative, it is resolved to an absolute path based on the CDB entry's `directory` field. This ensures consistency in subsequent usage, regardless of the current working directory.
+
+### Toolchain Probing
+
+Toolchain probing converts driver-level commands to cc1 arguments:
+
+**Probing process**: For a given CanonicalCommand, the corresponding compiler driver (GCC, Clang, MSVC, etc.) is invoked to obtain the complete cc1 arguments -- including the target triple, system header search paths, default macro definitions, and all other implicit information.
+
+**Caching strategy**: Probing results are cached by (driver, file extension, non-user-content flags). The file extension is part of the cache key because `.c` and `.cpp` files may trigger different driver rules. User-content options are not part of the cache key because they do not affect driver output.
+
+**Negative caching**: Failed toolchain probes (e.g., a nonexistent GCC version) also cache the failure result, avoiding repeated attempts for every file.
+
+**Startup warm-up**: At server startup, probing is initiated in parallel for all unique cache keys, pre-populating the cache. By the time LSP requests start arriving, all toolchain probing results are already available.
+
+**Compiler adaptation**: Different compiler families (GCC, Clang, MSVC/ClangCL, etc.) have different driver invocation methods and output parsing logic. Some compiler families (NVCC, Intel, Zig) are recognized but fall through to a generic Clang-driver path rather than having fully dedicated handling.
+
+### Search Path Extraction
+
+After toolchain probing, the cc1 arguments contain the complete header search paths. SearchConfig extracts and organizes these paths into a four-tier structure:
+
+1. **Quoted** (`-iquote` directories): Search paths for `#include "foo.h"`
+2. **Angled** (`-I` directories): Search paths for `#include <foo.h>`
+3. **System** (`-isystem` directories): System header paths where diagnostics are suppressed
+4. **After** (`-idirafter` directories): Searched after system directories
+
+This four-tier model is largely consistent with Clang's internal search logic (some less common options like `-cxx-isystem`, `-iwithsysroot`, and Framework search paths are not yet supported). Paths within each tier are deduplicated (starting from the Angled tier, matching Clang's `RemoveDuplicates` algorithm), ensuring search behavior matches the actual compiler.
+
+SearchConfig is the foundational input for include path resolution, include path completion, dependency graph construction, and other features.
+
+## Design Decisions and Trade-offs
+
+**Why classify by option ID rather than string matching?** Classification uses Clang's own option table (`OptTable`), ensuring all options are correctly identified, including those with complex syntax (e.g., `-Wno-error=deprecated`, `-isystem=/usr/include`). String matching is prone to missing edge cases.
+
+**Why don't user-content options participate in the toolchain cache key?** This is the core of the two-level separation design. `-I` and `-D` do not change the system paths, target triple, or other information output by the driver. Excluding them reduces the number of cache keys from "one per file" to "one per configuration," yielding an order-of-magnitude improvement in large projects.
+
+**Why must search path deduplication match Clang's algorithm?** If the language server's search path order differs from the actual compiler's, include resolution might find different files, producing confusing inconsistencies. Strict matching ensures behavioral consistency.