Skip to content

Commit

Permalink
Remove slack reference from README (#214)
Browse files Browse the repository at this point in the history
We do not include slack data in our production because PII restriction 
so I feel it makes sense to remove this from README too
  • Loading branch information
pankajastro authored Dec 19, 2023
1 parent ad905e0 commit 9c1aadc
Showing 1 changed file with 0 additions and 1 deletion.
1 change: 0 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ Ask Astro uses a set of Airflow DAGs that: ingest data from a source via an API
- [Apache Airflow GitHub](https://github.com/apache/airflow) issues and pull requests
- [OpenLineage GitHub](https://github.com/OpenLineage/OpenLineage)
- [OpenLineage GitHub docs](https://github.com/OpenLineage/docs)
- Apache Airflow Slack's [#troubleshooting channel](https://app.slack.com/client/TCQ18L22Z/CCQ7EGB1P)
- [StackOverflow's Stack Exchange Data Dump](https://archive.org/details/stackexchange)

Generally, each of these sources has a DAG that handles the ingestion flow. We use LangChain's built-in text splitters for processing Markdown, RST, and Python code into smaller chunks to ensure each document is small enough to give accurate results when doing embeddings. We then use a Weaviate provider that we've built (and plan to publish) to both embed and store each document as a vector in Weaviate using OpenAI's embedding model.
Expand Down

0 comments on commit 9c1aadc

Please sign in to comment.