Fix and improve site search on docs.langflow.org

[eric-schneider]

Title

Fix and improve site search on docs.langflow.org

Type

Improvement

Description

There are a number of issues with the Mendable-powered search component on docs.langflow.com.

tl;dr the Mendable search index is wildly out of date, search results link to Markdown files in GitHub (breaks context and modality) and those links are 404'ing

Current state

You can access the Mendable UI by clicking the floating button component in the bottom-right. Typing into the search box gets you instant keyword results (nice). Optionally, you can select “Ask Mendable” to generate an AI response (also nice).

Problems:

• ⚠️ The Mendable search index is sourced from a static snapshot of Docusaurus .mdx files from August 18, 2023.
• ⚠️ Keyword search results and AI response citations are all links to the indexed .mdx files.
• ⚠️ All of these links are currently 404'ing due to the outdated index.

Proposed changes

1. Change the search index data source to a web crawl of docs.langflow.org, configured to auto-sync every 24hrs.

   • With the updated data source, keyword search results and AI response citations will be proper links to pages on docs.langflow.org.
   • Auto-syncing will also ensure that search results stay up to date and not 404.

2. Keep the floating button component, but also add a search bar component to the top nav bar.

   • Engaging with either component opens the same Mendable UI.
   • The additional search bar will provide a more visible, potentially more accessible method for users to discover and launch the Mendable UI.

Use Case

No response

Implementation Plan

1. Add Mendable search bar component to SearchBar.js.

2. Update site url in docusaurus.config.js.

   • The Docusaurus site URL is misconfigured with an old GitHub Pages address, which is causing incorrect URLs to appear in the generated sitemap.xml.

   • Mendable seems to heavily rely on the sitemap to crawl the site. The sitemap needs to be fixed before we can ingest it into Mendable.

3. Standardize the implementation of the Mendable anon_key.

   • The key was sourced from an environment variable at some point (MENDABLE_ANON_KEY), but has since been hard-coded directly into Footer.js. I’ll need someone to advise on whether we should re-introduce this environment variable or continue to hard-code the key.