Handle big text ingestion

[pankajastro]

Recently, I saw below error and feel that we need a mechanism to handle large docs

[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:439} WARNING - Error during upsert. Rolling back all inserts for docs with errors.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:328} INFO - Removing id f9c660b6-a393-54ef-9f95-20a616c3edfe for rollback.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:333} INFO - UUID 767b916a-f6ef-51e1-bf23-cea43a801baa does not exist. Skipping deletion during rollback.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:328} INFO - Removing id c7ccb038-c35a-501e-8404-1ce26782f0f7 for rollback.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:328} INFO - Removing id 8a084e49-5e06-5827-8165-69c05d29716e for rollback.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:328} INFO - Removing id 9a413399-6b68-5660-99d8-4bda030f9496 for rollback.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:467} ERROR - Errors encountered during ingest.
[2023-12-19, 05:42:30 UTC] {ask_astro_weaviate_hook.py:468} ERROR - <generator object AskAstroWeaviateHook.ingest_data.<locals>.<genexpr> at 0x7fc00df27e00>
[2023-12-19, 05:42:31 UTC] {taskinstance.py:1937} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/airflow/decorators/base.py", line 221, in execute
    return_value = super().execute(context)
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/operators/python.py", line 192, in execute
    return_value = self.execute_callable()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/operators/python.py", line 209, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/include/tasks/extract/utils/weaviate/ask_astro_weaviate_hook.py", line 469, in ingest_data
    raise AirflowException("Errors encountered during ingest.")
airflow.exceptions.AirflowException: Errors encountered during ingest.

https://clmkpsyfc000301kcjie00t1l.astronomer.run/d2uulcqu/dags/ask_astro_load_airflow_docs/grid?dag_run_id=scheduled__2023-12-18T05%3A00%3A00%2B00%3A00&task_id=ingest_data&tab=mapped_tasks&map_index=0

[pankajastro]

I handle in case forum by truncating it. not a perfect solution just in case want to check

ask-astro/airflow/include/tasks/extract/astro_forum_docs.py

Lines 88 to 109 in 441def6

	def truncate_tokens(text: str, encoding_name: str, max_length: int = 8192) -> str:
	"""
	Truncates a text string based on the maximum number of tokens.
	param string (str): The input text string to be truncated.
	param encoding_name (str): The name of the encoding model.
	param max_length (int): The maximum number of tokens allowed. Default is 8192.
	"""
	import tiktoken
	try:
	encoding = tiktoken.encoding_for_model(encoding_name)
	except ValueError as e:
	raise ValueError(f"Invalid encoding_name: {e}")
	encoded_string = encoding.encode(text)
	num_tokens = len(encoded_string)
	if num_tokens > max_length:
	text = encoding.decode(encoded_string[:max_length])
	return text

[davidgxue]

On a related note, I also noticed some of our inserts/upserts have issues with having too long of a chunk being attempted to be ingested. see error log below. This needs further investigation as it could mean some docs aren't properly ingested

[2024-01-16, 05:02:04 UTC] {weaviate.py:430} INFO - Error occurred in batch process for 00d97bbe-cec4-51bc-b0d2-11f1326f167e with error {'error': [{'message': "update vector: connection to: OpenAI API failed with status: 400 error: This model's maximum context length is 8192 tokens, however you requested 20585 tokens (20585 in your prompt; 0 for the completion). Please reduce your prompt; or completion length."}]}
[2024-01-16, 05:02:04 UTC] {weaviate.py:430} INFO - Error occurred in batch process for d1eace29-205f-53bf-9fc5-6645f133877c with error {'error': [{'message': "update vector: connection to: OpenAI API failed with status: 400 error: This model's maximum context length is 8192 tokens, however you requested 28319 tokens (28319 in your prompt; 0 for the completion). Please reduce your prompt; or completion length."}]}
[2024-01-16, 05:02:04 UTC] {weaviate.py:430} INFO - Error occurred in batch process for a82bb7bc-f2fc-5698-ad79-c28185a33e82 with error {'error': [{'message': "update vector: connection to: OpenAI API failed with status: 400 error: This model's maximum context length is 8192 tokens, however you requested 13851 tokens (13851 in your prompt; 0 for the completion). Please reduce your prompt; or completion length."}]}
[2024-01-16, 05:02:04 UTC] {weaviate.py:430} INFO - Error occurred in batch process for 9075559f-b0ce-5d8f-8162-28fd974a99e1 with error {'error': [{'message': "update vector: connection to: OpenAI API failed with status: 400 error: This model's maximum context length is 8192 tokens, however you requested 15193 tokens (15193 in your prompt; 0 for the completion). Please reduce your prompt; or completion length."}]}
[2024-01-16, 05:02:04 UTC] {weaviate.py:440} INFO - Total Objects 6066 / Objects 6009 successfully inserted and Objects 22 had errors.