handle max tokens

[krrishdholakia]

Anthropic has:

completion = anthropic.completions.create(
    model="claude-2",
    max_tokens_to_sample=300,
    prompt=f"{HUMAN_PROMPT} how does a court case get to the Supreme Court? {AI_PROMPT}",
)

Replicate has:

    output = replicate.run(
      "replicate/flan-t5-xl:eec2f71c986dfa3b7a5d842d22e1130550f015720966bec48beaae059b19ef4c",
      # "replicate/llama-7b:2014ee1247354f2e81c0b3650d71ca715bc1e610189855f134c30ecb841fae21",
      input={
        "prompt": prompt,
        "max_length": 256
      })

i believe they both achieve similar purposes.

Link to docs: Anthropic

Replicate

[krrishdholakia]

we now handle this like openai does with a max_token input

[krrishdholakia]

for more details - https://litellm.readthedocs.io/en/latest/input/