Solar-Fullstack-LLM-101/06_PDF_CAG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/UpstageAI/cookbook/blob/main/Solar-Fullstack-LLM-101/06_PDF_CAG.ipynb\">\n",
    "<img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. PDF CAG(Credibility-Aware Generation)\n",
    "\n",
    "## Overview  \n",
    "In this exercise, we will explore how to apply Credibility-Aware Generation (CAG) techniques to process PDF documents using the Solar framework. This involves extracting text from PDFs, assessing the credibility of the content, and generating reliable outputs based on the extracted information. This notebook will guide you through the steps needed to integrate CAG with PDF handling.\n",
    " \n",
    "## Purpose of the Exercise\n",
    "The purpose of this exercise is to demonstrate the integration of Credibility-Aware Generation with PDF document processing. By the end of this tutorial, users will be able to extract text from PDFs, evaluate its credibility, and generate credible outputs using the Solar framework, enhancing the accuracy and trustworthiness of the information derived from PDF sources.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip3 install -qU langchain-upstage pypdf python-dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @title set API key\n",
    "import os\n",
    "import getpass\n",
    "from pprint import pprint\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "from IPython import get_ipython\n",
    "\n",
    "if \"google.colab\" in str(get_ipython()):\n",
    "    # Running in Google Colab. Please set the UPSTAGE_API_KEY in the Colab Secrets\n",
    "    from google.colab import userdata\n",
    "    os.environ[\"UPSTAGE_API_KEY\"] = userdata.get(\"UPSTAGE_API_KEY\")\n",
    "else:\n",
    "    # Running locally. Please set the UPSTAGE_API_KEY in the .env file\n",
    "    from dotenv import load_dotenv\n",
    "\n",
    "    load_dotenv()\n",
    "\n",
    "if \"UPSTAGE_API_KEY\" not in os.environ:\n",
    "    os.environ[\"UPSTAGE_API_KEY\"] = getpass.getpass(\"Enter your Upstage API key: \")\n"
]

  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SolarSample](figures/solar_sample.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Ignoring wrong pointing object 8 0 (offset 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Size Type H6 (Avg.) ARC HellaSwag MMLU TruthfulQA Winogrande GSM8KSOLAR 10.7B-Instruct⇠11BAlignment-tuned74.2071.0888.1666.2171.4383.5864.75Qwen 72B⇠72B Pretrained 73.60 65.19 85.9477.3760.19 82.4870.43Mixtral 8x7B-Instruct-v0.1⇠47B Instruction-tuned 72.62 70.22 87.63 71.16 64.58 81.37 60.73Yi 34B-200K⇠34B Pretrained 70.81 65.36 85.58 76.06 53.64 82.56 61.64Yi 34B⇠34B Pretrained 69.42 64.59 85.69 76.35 56.23 83.03 50.64Mixtral 8x7B-v0.1⇠47B Pretrained 68.42 66.04 86.49 71.82 46.78 81.93 57.47Llama 2 70B⇠70B Pretrained 67.87 67.32 87.33 69.83 44.92 83.74 54.06Falcon 180B⇠180B Pretrained 67.85 69.4588.8670.50 45.4786.9045.94SOLAR 10.7B⇠11BPretrained66.0461.9584.6065.4845.0483.6655.50Qwen 14B⇠14B Pretrained 65.86 58.28 83.99 67.70 49.43 76.80 58.98Mistral 7B-Instruct-v0.2⇠7B Instruction-tuned 65.71 63.14 84.88 60.78 68.26 77.19 40.03Yi 34B-Chat⇠34B Instruction-tuned 65.32 65.44 84.16 74.90 55.37 80.11 31.92Mistral 7B⇠7B Pretrained 60.97 59.98 83.31 64.16 42.15 78.37 37.83Table 2: Ev\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "loader = PyPDFLoader(\"pdfs/solar_sample.pdf\")\n",
    "docs = loader.load()  # or layzer.lazy_load()\n",
    "print(docs[0].page_content[:1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import PromptTemplate\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_upstage import ChatUpstage\n",
    "\n",
    "\n",
    "llm = ChatUpstage()\n",
    "\n",
    "prompt_template = PromptTemplate.from_template(\n",
    "    \"\"\"\n",
    "    Please provide most correct answer from the following context. \n",
    "    If the answer is not present in the context, please write \"The information is not present in the context.\"\n",
    "    ---\n",
    "    Question: {question}\n",
    "    ---\n",
    "    Context: {Context}\n",
    "    \"\"\"\n",
    ")\n",
    "chain = prompt_template | llm | StrOutputParser()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Table 2 provides evaluation results in the Open LLM Leaderboard for SOLAR 10.7B and SOLAR 10.7B-Instruct, along with other top-performing models. The scores for six tasks mentioned in Sec. 4.1, along with the H6 score (average of six tasks), are reported. The table also includes the size of the models in units of billions of parameters and the training stage of the model, which is chosen from {Pretrained, Instruction-tuned, Alignment-tuned}. Models based on SOLAR 10.7B are colored purple, and the best scores for H6 and the individual tasks are shown in bold.'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"question\": \"Explain Table 2?\", \"Context\": docs})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The MMLU scores of SOLAR 10.7B is 45.04.'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"question\": \"What is MMLU scores of SOLAR 10.7B?\", \"Context\": docs})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The information is not present in the context.'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"question\": \"What is ARC of Mistral?\", \"Context\": docs})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Excercise \n",
    "\n",
    "How can we easily get information from complicated tables for LLMs?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}