JSON mode

[pseudotensor]

Fixes #1526

• Can pass in response_format='json_object' at CLI or API or UI to get json. Works for most models even if don't support json mode directly, except Danube (really bad) and to some extent Gemma (one character mistakes).
• Can pass guided_json to specify the schema that should be a spec form with type and properties. The actual json spec is inside properties. https://github.com/vllm-project/vllm/blob/c64cf38673780544087af5ad5d3baf879a29220b/tests/entrypoints/test_openai_server.py#L28-L73
• If pass guided_json for vLLM >=0.4.0 instances, then strictly follows format including keys, types, etc.
• Can pass separately guided_regex, guided_choice, guided_grammar for similar control.
• Handle old vLLM and other models that do not have json mode by using code blocks. Only Danube and Gemma have issues.
• Handle mistral and openai directly for json mode
• New test covers 138 combinations of all models our "internal" system uses for LLM or MyData an either Query or Summarize. All work except Danube/Gemma.
• Upgrade langchain components and migrate.
• 'json' as new model bool for model list
• Support llama.cpp or transformers with outlines for broader support. Unlikely soon. https://outlines-dev.github.io/outlines/reference/models/llamacpp/
• Support AWQ, etc. Unlikely, depends upon those maintainers and outlines maintainer. vLLM is best solution.
• Make template from example (json_code -> guided_json template) using https://github.com/wolverdude/GenSON

example guided_json, guided_regex, guided_choice schemas to be passed in as string to h2oGPT.

TEST_SCHEMA = {
    "type": "object",
    "properties": {
        "name": {
            "type": "string"
        },
        "age": {
            "type": "integer"
        },
        "skills": {
            "type": "array",
            "items": {
                "type": "string",
                "maxLength": 10
            },
            "minItems": 3
        },
        "work history": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "company": {
                        "type": "string"
                    },
                    "duration": {
                        "type": "string"
                    },
                    "position": {
                        "type": "string"
                    }
                },
                "required": ["company", "position"]
            }
        }
    },
    "required": ["name", "age", "skills", "work history"]
}

TEST_REGEX = (r"((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)\.){3}"
              r"(25[0-5]|(2[0-4]|1\d|[1-9]|)\d)")

TEST_CHOICE = [
    "Python", "Java", "JavaScript", "C++", "C#", "PHP", "TypeScript", "Ruby",
    "Swift", "Kotlin"
]

[pseudotensor]

[pseudotensor]

[pseudotensor]

[pseudotensor]

Working for all except Cohere, which should be native vLLM json