Standardize DocumentLoader docstrings and integration docs

[isahers1]

Privileged issue

• I am a LangChain maintainer, or was asked directly by a LangChain maintainer to create an issue here.

Issue Content

Issue

To make our document loader integrations as easy to use as possible we need to make sure the docs for them are thorough and standardized. There are two parts to this: updating the document loader docstrings and updating the actual integration docs.

This needs to be done for each DocumentLoader integration, ideally with one PR per DocumentLoader.

Related to broader issues #21983 and #22005.

Docstrings

Each DocumentLoader class docstring should have the sections shown in the Appendix below. The sections should have input and output code blocks when relevant. See RecursiveUrlLoader docstrings and corresponding API reference for an example.

Doc Pages

Each DocumentLoader docs page should follow this template. See RecursiveUrlLoader for an example.

You can use the langchain-cli to quickly get started with a new document loader integration docs page (run from root of repo):

poetry run pip install -e libs/cli
poetry run langchain-cli integration create-doc --name "foo-bar" --name-class FooBar --component-type "DocumentLoader" --destination-dir ./docs/docs/integrations/document_loaders/

where --name is the integration package name without the "langchain-" prefix and --name-class is the class name without the "Loader" suffix. This will create a template doc with some autopopulated fields at docs/docs/integrations/document_loaders/foo_bar.ipynb.

To build a preview of the docs you can run (from root):

make docs_clean
make docs_build
cd docs/build/output-new
yarn
yarn start

Appendix

"""
ModuleName document loader integration

# TODO: Replace with relevant packages, env vars.
Setup:
    Install ``__package_name__`` and set environment variable ``__MODULE_NAME___API_KEY``.

    .. code-block:: bash

        pip install -U __package_name__
        export __MODULE_NAME___API_KEY="your-api-key"

# TODO: Replace with relevant init params.
Instantiate:
    .. code-block:: python

        from __module_name__ import __ModuleName__Loader

        loader = __ModuleName__Loader(
            url = "https://docs.python.org/3.9/",
            # otherparams = ...
        )

Load:
    .. code-block:: python

        docs = loader.load()
        print(docs[0].page_content[:100])
        print(docs[0].metadata)

    .. code-block:: python

        TODO: Example output

# TODO: Delete if async load is not implemented
Async load:
    .. code-block:: python

        docs = await loader.aload()
        print(docs[0].page_content[:100])
        print(docs[0].metadata)

    .. code-block:: python

        TODO: Example output

Lazy load:
    .. code-block:: python

        docs = []
        docs_lazy = loader.lazy_load()

        # async variant:
        # docs_lazy = await loader.alazy_load()

        for doc in docs_lazy:
            docs.append(doc)
        print(docs[0].page_content[:100])
        print(docs[0].metadata)

    .. code-block:: python

        TODO: Example output  
"""

[lucas-tucker]

I will work on this today

[dosubot]

Hi, @isahers1. I'm Dosu, and I'm helping the LangChain team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• The issue aims to improve documentation for DocumentLoader integrations by standardizing docstrings and integration docs.
• Each DocumentLoader class should have detailed docstrings with specific sections and code examples.
• Integration docs should follow a given template.
• User "lucas-tucker" volunteered to work on this task, and you reacted positively.

Next Steps:

• Please let me know if this issue is still relevant to the latest version of the LangChain repository. If so, you can keep the discussion open by commenting on the issue.
• Otherwise, the issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!