Docs: Add models page

dictionary.txt

@@ -79,6 +79,8 @@ mkdocs
 fnllm
 typer
 spacy
+kwargs
+ollama
 
 # Library Methods
 iterrows
@@ -190,6 +192,7 @@ Arxiv
 kwds
 jsons
 txts
+byog
 
 # Dulce
 astrotechnician

docs/config/init.md

@@ -29,4 +29,4 @@ The `init` command will create the following files in the specified directory:
 
 ## Next Steps
 
-After initializing your workspace, you can either run the [Prompt Tuning](../prompt_tuning/auto_prompt_tuning.md) command to adapt the prompts to your data or even start running the [Indexing Pipeline](../index/overview.md) to index your data. For more information on configuring GraphRAG, see the [Configuration](overview.md) documentation.
+After initializing your workspace, you can either run the [Prompt Tuning](../prompt_tuning/auto_prompt_tuning.md) command to adapt the prompts to your data or even start running the [Indexing Pipeline](../index/overview.md) to index your data. For more information on configuration options available, see the [YAML details page](yaml.md).

docs/config/models.md

@@ -0,0 +1,101 @@
+# Language Model Selection and Overriding
+
+This page contains information on selecting a model to use and options to supply your own model for GraphRAG. Note that this is not a guide to finding the right model for your use case.
+
+## Default Model Support
+
+GraphRAG was built and tested using OpenAI models, so this is the default model set we support. This is not intended to be a limiter or statement of quality or fitness for your use case, only that it's the set we are most familiar with for prompting, tuning, and debugging.
+
+GraphRAG also utilizes a language model wrapper library used by several projects within our team, called fnllm. fnllm provides two important functions for GraphRAG: rate limiting configuration to help us maximize throughput for large indexing jobs, and robust caching of API calls to minimize consumption on repeated indexes for testing, experimentation, or incremental ingest. fnllm uses the OpenAI Python SDK under the covers, so OpenAI-compliant endpoints are a base requirement out-of-the-box.
+
+## Model Selection Considerations
+
+GraphRAG has been most thoroughly tested with the gpt-4 series of models from OpenAI, including gpt-4 gpt-4-turbo, gpt-4o, and gpt-4o-mini. Our [arXiv paper](https://arxiv.org/abs/2404.16130), for example, performed quality evaluation using gpt-4-turbo.
+
+Versions of GraphRAG before 2.2.0 made extensive use of `max_tokens` and `logit_bias` to control generated response length or content. The introduction of the o-series of models added new, non-compatible parameters because these models include a reasoning component that has different consumption patterns and response generation attributes than non-reasoning models. GraphRAG 2.2.0 now supports these models, but there are important differences that need to be understood before you switch.
+
+- Previously, GraphRAG used `max_tokens` to limit responses in a few locations. This is done so that we can have predictable content sizes when building downstream context windows for summarization. We have now switched from using `max_tokens` to use a prompted approach, which is working well in our tests. We suggest using `max_tokens` in your language model config only for budgetary reasons if you want to limit consumption, and not for expected response length control. We now also support the o-series equivalent `max_completion_tokens`, but if you use this keep in mind that there may be some unknown fixed reasoning consumption amount in addition to the response tokens, so it is not a good technique for response control.
+- Previously, GraphRAG used a combination of `max_tokens` and `logit_bias` to strictly control a binary yes/no question during gleanings. This is not possible with reasoning models, so again we have switched to a prompted approach. Our tests with gpt-4o, gpt-4o-mini, and o1 show that this works consistently, but could have issues if you have an older or smaller model.
+- The o-series models are much slower and more expensive. It may be useful to use an asymmetric approach to model use in your config: you can define as many models as you like in the `models` block of your settings.yaml and reference them by key for every workflow that requires a language model. You could use gpt-4o for indexing and o1 for query, for example. Experiment to find the right balance of cost, speed, and quality for your use case.
+- The o-series models contain a form of native native chain-of-thought reasoning that is absent in the non-o-series models. GraphRAG's prompts sometimes contain CoT because it was an effective technique with the gpt-4* series. It may be counterproductive with the o-series, so you may want to tune or even re-write large portions of the prompt templates (particularly for graph and claim extraction).
+
+Example config with asymmetric model use:
+
+```yaml
+models:
+  extraction_chat_model:
+    api_key: ${GRAPHRAG_API_KEY}
+    type: openai_chat
+    auth_type: api_key
+    model: gpt-4o
+    model_supports_json: true
+  query_chat_model:
+    api_key: ${GRAPHRAG_API_KEY}
+    type: openai_chat
+    auth_type: api_key
+    model: o1
+    model_supports_json: true
+
+...
+
+extract_graph:
+  model_id: extraction_chat_model
+  prompt: "prompts/extract_graph.txt"
+  entity_types: [organization,person,geo,event]
+  max_gleanings: 1
+
+...
+
+
+global_search:
+  chat_model_id: query_chat_model
+  map_prompt: "prompts/global_search_map_system_prompt.txt"
+  reduce_prompt: "prompts/global_search_reduce_system_prompt.txt"
+  knowledge_prompt: "prompts/global_search_knowledge_system_prompt.txt"
+```
+
+Another option would be to avoid using a language model at all for the graph extraction, instead using the `fast` [indexing method](../index/methods.md) that uses NLP for portions of the indexing phase in lieu of LLM APIs.
+
+## Using Non-OpenAI Models
+
+As noted above, our primary experience and focus has been on OpenAI models, so this is what is supported out-of-the-box. Many users have requested support for additional model types, but it's out of the scope of our research to handle the many models available today. There are two approaches you can use to connect to a non-OpenAI model:
+
+### Proxy APIs
+
+Many users have used platforms such as [ollama](https://ollama.com/) to proxy the underlying model HTTP calls to a different model provider. This seems to work reasonably well, but we frequently see issues with malformed responses (especially JSON), so if you do this please understand that your model needs to reliably return the specific response formats that GraphRAG expects. If you're having trouble with a model, you may need to try prompting to coax the format, or intercepting the response within your proxy to try and handle malformed responses.
+
+### Model Protocol
+
+As of GraphRAG 2.0.0, we support model injection through the use of a standard chat and embedding Protocol and an accompanying ModelFactory that you can use to register your model implementation. This is not supported with the CLI, so you'll need to use GraphRAG as a library.
+
+- Our Protocol is [defined here](https://github.com/microsoft/graphrag/blob/main/graphrag/language_model/protocol/base.py)
+- Our base implementation, which wraps fnllm, [is here](https://github.com/microsoft/graphrag/blob/main/graphrag/language_model/providers/fnllm/models.py)
+- We have a simple mock implementation in our tests that you can [reference here](https://github.com/microsoft/graphrag/blob/main/tests/mock_provider.py)
+
+Once you have a model implementation, you need to register it with our ModelFactory:
+
+```python
+class MyCustomModel:
+    ...
+    # implementation
+
+# elsewhere...
+ModelFactory.register_chat("my-custom-chat-model", lambda **kwargs: MyCustomModel(**kwargs))
+```
+
+Then in your config you can reference the type name you used:
+
+```yaml
+models:
+  default_chat_model:
+    type: my-custom-chat-model
+
+
+extract_graph:
+  model_id: default_chat_model
+  prompt: "prompts/extract_graph.txt"
+  entity_types: [organization,person,geo,event]
+  max_gleanings: 1
+```
+
+Note that your custom model will be passed the same params for init and method calls that we use throughout GraphRAG. There is not currently any ability to define custom parameters, so you may need to use closure scope or a factory pattern within your implementation to get custom config values.

docs/config/overview.md

@@ -4,8 +4,8 @@ The GraphRAG system is highly configurable. This page provides an overview of th
 
 ## Default Configuration Mode
 
-The default configuration mode is the simplest way to get started with the GraphRAG system. It is designed to work out-of-the-box with minimal configuration. The primary configuration sections for the Indexing Engine pipelines are described below. The main ways to set up GraphRAG in Default Configuration mode are via:
+The default configuration mode is the simplest way to get started with the GraphRAG system. It is designed to work out-of-the-box with minimal configuration. The main ways to set up GraphRAG in Default Configuration mode are via:
 
-- [Init command](init.md) (recommended)
-- [Using YAML for deeper control](yaml.md)
+- [Init command](init.md) (recommended first step)
+- [Edit settings.yaml for deeper control](yaml.md)
 - [Purely using environment variables](env_vars.md) (not recommended)

docs/config/yaml.md

@@ -60,12 +60,14 @@ models:
 - `concurrent_requests` **int** The number of open requests to allow at once.
 - `async_mode` **asyncio|threaded** The async mode to use. Either `asyncio` or `threaded`.
 - `responses` **list[str]** - If this model type is mock, this is a list of response strings to return.
-- `max_tokens` **int** - The maximum number of output tokens.
-- `temperature` **float** - The temperature to use.
-- `top_p` **float** - The top-p value to use.
 - `n` **int** - The number of completions to generate.
-- `frequency_penalty` **float** - Frequency penalty for token generation.
-- `presence_penalty` **float** - Frequency penalty for token generation.
+- `max_tokens` **int** - The maximum number of output tokens. Not valid for o-series models.
+- `temperature` **float** - The temperature to use. Not valid for o-series models.
+- `top_p` **float** - The top-p value to use. Not valid for o-series models.
+- `frequency_penalty` **float** - Frequency penalty for token generation. Not valid for o-series models.
+- `presence_penalty` **float** - Frequency penalty for token generation. Not valid for o-series models.
+- `max_completion_tokens` **int** - Max number of tokens to consume for chat completion. Must be large enough to include an unknown amount for "reasoning" by the model. o-series models only.
+- `reasoning_effort` **low|medium|high** - Amount of "thought" for the model to expend reasoning about a response. o-series models only.
 
 ## Input Files and Chunking
 
@@ -212,7 +214,6 @@ Tune the language model-based graph extraction process.
 - `prompt` **str** - The prompt file to use.
 - `entity_types` **list[str]** - The entity types to identify.
 - `max_gleanings` **int** - The maximum number of gleaning cycles to use.
-- `encoding_model` **str** - The text encoding model to use. Default is to use the encoding model aligned with the language model (i.e., it is retrieved from tiktoken if unset). This is only used for gleanings during the logit_bias check.
 
 ### summarize_descriptions
 
@@ -221,6 +222,7 @@ Tune the language model-based graph extraction process.
 - `model_id` **str** - Name of the model definition to use for API calls.
 - `prompt` **str** - The prompt file to use.
 - `max_length` **int** - The maximum number of output tokens per summarization.
+- `max_input_length` **int** - The maximum number of tokens to collect for summarization (this will limit how many descriptions you send to be summarized for a given entity or relationship).
 
 ### extract_graph_nlp
 
@@ -274,7 +276,6 @@ These are the settings used for Leiden hierarchical clustering of the graph to c
 - `prompt` **str** - The prompt file to use.
 - `description` **str** - Describes the types of claims we want to extract.
 - `max_gleanings` **int** - The maximum number of gleaning cycles to use.
-- `encoding_model` **str** - The text encoding model to use. Default is to use the encoding model aligned with the language model (i.e., it is retrieved from tiktoken if unset). This is only used for gleanings during the logit_bias check.
 
 ### community_reports
 
@@ -329,11 +330,7 @@ Indicates whether we should run UMAP dimensionality reduction. This is used to p
 - `conversation_history_max_turns` **int** - The conversation history maximum turns.
 - `top_k_entities` **int** - The top k mapped entities.
 - `top_k_relationships` **int** - The top k mapped relations.
-- `temperature` **float | None** - The temperature to use for token generation.
-- `top_p` **float | None** - The top-p value to use for token generation.
-- `n` **int | None** - The number of completions to generate.
-- `max_tokens` **int** - The maximum tokens.
-- `llm_max_tokens` **int** - The LLM maximum tokens.
+- `max_context_tokens` **int** - The maximum tokens to use building the request context.
 
 ### global_search
 
@@ -346,20 +343,14 @@ Indicates whether we should run UMAP dimensionality reduction. This is used to p
 - `map_prompt` **str | None** - The global search mapper prompt to use.
 - `reduce_prompt` **str | None** - The global search reducer to use.
 - `knowledge_prompt` **str | None** - The global search general prompt to use.
-- `temperature` **float | None** - The temperature to use for token generation.
-- `top_p` **float | None** - The top-p value to use for token generation.
-- `n` **int | None** - The number of completions to generate.
-- `max_tokens` **int** - The maximum context size in tokens.
-- `data_max_tokens` **int** - The data llm maximum tokens.
-- `map_max_tokens` **int** - The map llm maximum tokens.
-- `reduce_max_tokens` **int** - The reduce llm maximum tokens.
-- `concurrency` **int** - The number of concurrent requests.
-- `dynamic_search_llm` **str** - LLM model to use for dynamic community selection.
+- `max_context_tokens` **int** - The maximum context size to create, in tokens.
+- `data_max_tokens` **int** - The maximum tokens to use constructing the final response from the reduces responses.
+- `map_max_length` **int** - The maximum length to request for map responses, in words.
+- `reduce_max_length` **int** - The maximum length to request for reduce responses, in words.
 - `dynamic_search_threshold` **int** - Rating threshold in include a community report.
 - `dynamic_search_keep_parent` **bool** - Keep parent community if any of the child communities are relevant.
 - `dynamic_search_num_repeats` **int** - Number of times to rate the same community report.
 - `dynamic_search_use_summary` **bool** - Use community summary instead of full_context.
-- `dynamic_search_concurrent_coroutines` **int** - Number of concurrent coroutines to rate community reports.
 - `dynamic_search_max_level` **int** - The maximum level of community hierarchy to consider if none of the processed communities are relevant.
 
 ### drift_search
@@ -370,11 +361,9 @@ Indicates whether we should run UMAP dimensionality reduction. This is used to p
 - `embedding_model_id` **str** - Name of the model definition to use for Embedding calls.
 - `prompt` **str** - The prompt file to use.
 - `reduce_prompt` **str** - The reducer prompt file to use.
-- `temperature` **float** - The temperature to use for token generation.",
-- `top_p` **float** - The top-p value to use for token generation.
-- `n` **int** - The number of completions to generate.
-- `max_tokens` **int** - The maximum context size in tokens.
 - `data_max_tokens` **int** - The data llm maximum tokens.
+- `reduce_max_tokens` **int** - The maximum tokens for the reduce phase. Only use if a non-o-series model.
+- `reduce_max_completion_tokens` **int** - The maximum tokens for the reduce phase. Only use for o-series models.
 - `concurrency` **int** - The number of concurrent requests.
 - `drift_k_followups` **int** - The number of top global results to retrieve.
 - `primer_folds` **int** - The number of folds for search priming.
@@ -388,7 +377,8 @@ Indicates whether we should run UMAP dimensionality reduction. This is used to p
 - `local_search_temperature` **float** - The temperature to use for token generation in local search.
 - `local_search_top_p` **float** - The top-p value to use for token generation in local search.
 - `local_search_n` **int** - The number of completions to generate in local search.
-- `local_search_llm_max_gen_tokens` **int** - The maximum number of generated tokens for the LLM in local search.
+- `local_search_llm_max_gen_tokens` **int** - The maximum number of generated tokens for the LLM in local search. Only use if a non-o-series model.
+- `local_search_llm_max_gen_completion_tokens` **int** - The maximum number of generated tokens for the LLM in local search. Only use for o-series models.
 
 ### basic_search
 
@@ -397,13 +387,4 @@ Indicates whether we should run UMAP dimensionality reduction. This is used to p
 - `chat_model_id` **str** - Name of the model definition to use for Chat Completion calls.
 - `embedding_model_id` **str** - Name of the model definition to use for Embedding calls.
 - `prompt` **str** - The prompt file to use.
-- `text_unit_prop` **float** - The text unit proportion. 
-- `community_prop` **float** - The community proportion.
-- `conversation_history_max_turns` **int** - The conversation history maximum turns.
-- `top_k_entities` **int** - The top k mapped entities.
-- `top_k_relationships` **int** - The top k mapped relations.
-- `temperature` **float | None** - The temperature to use for token generation.
-- `top_p` **float | None** - The top-p value to use for token generation.
-- `n` **int | None** - The number of completions to generate.
-- `max_tokens` **int** - The maximum tokens.
-- `llm_max_tokens` **int** - The LLM maximum tokens.
+- `k` **int | None** - Number of text units to retrieve from the vector store for context building.