add option to add metadata into text chunks

[dayesouza]

Description

change metadata column in input table
add new default column into final_documents called creation_date
add option to prepend metadata into text chunks (default False)
add option to count metadata tokens in max tokens (default False)

Proposed Changes

• Added new configuration options prepend_metadata and count_tokens_with_metadata to the ChunkingConfig class in graphrag/config/models/chunking_config.py. These options allow metadata to be prepended to chunks and included in token counts.

• Updated the create_base_text_units function to include the new prepend_metadata and count_tokens_with_metadata parameters, and adjusted the aggregation logic to handle metadata.

• Modified the chunk_text function to support the new metadata options, including updating the run_strategy function to prepend metadata and count tokens if specified

• Introduced a new method get_creation_date in BlobPipelineStorage and FilePipelineStorage to retrieve the creation date of files. This method is now used in various parts of the code to add the creation date to data frames.

• Updated the load_file function in csv.py and text.py to include the file creation date in the data frame.

• Added new integration tests and unit

Checklist

• I have tested these changes locally.
• I have reviewed the code changes.
• I have updated the documentation (if necessary).
• I have added appropriate unit tests (if applicable).