002_Alto-XML-Dateien_einlesen_und_Texte_extrahieren.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6409d458-89cd-464b-9e97-ca7269dbefc8",
   "metadata": {},
   "source": [
    "# Summer School: Digitale Methoden der Zeitungsanalyse\n",
    "\n",
    "## Teil 2: Alto-XML-Dateien einlesen und Text extrahieren"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af9bb1e9-8c44-4aeb-9bf0-5906c332051e",
   "metadata": {},
   "source": [
    "In diesem Teil werden zunächst die XML-Dateien eingelesen und der reine Text extrahiert. \n",
    "\n",
    "Um die Arbeitsumgebung für die folgenden Schritte passend einzurichten, sollten zunächst die benötigten Python-Biblitoheken importiert werden.\n",
    "\n",
    "- `pandas`: Bibliothek zur Datenanalyse.\n",
    "- `lxml`: Bibliothek zur schnellen und flexiblen Verarbeitung von XML- und HTML-Dokumenten\n",
    "- `BeautifulSoup`: Bibliothek zum einfachen Parsen und Scrapen von HTML- und XML-Dokumenten\n",
    "- `unicodedata`: Standardbibliothek zur Handhabung und Normalisierung von Unicode-Daten\n",
    "- `pathlib`: Biobliothek zur Arbeit mit Dateisystempfaden in Python\n",
    "- `tqdm`: Zur Erstellung von Fortschrittsbalken in Python-Schleifen\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e038d51d-89f1-45e2-b769-52b4f794931f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from lxml import etree\n",
    "from tqdm import tqdm\n",
    "import unicodedata\n",
    "from bs4 import BeautifulSoup\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "057d5e7c-fc38-4652-93eb-20584607986f",
   "metadata": {},
   "source": [
    "Optional: Lokale Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1688f38d-d5f4-4f38-b2f9-03844d7792e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q tqdm\n",
    "%pip install -q bs4\n",
    "%pip install -q pathlib\n",
    "%pip install -q matplotlib\n",
    "%pip install -q numpy\n",
    "%pip install -q wordcloud\n",
    "%pip install -q plotly.express \n",
    "%pip install -q spacy\n",
    "%pip install -q nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "080c9716-0884-42cd-a802-92a99b32a99c",
   "metadata": {},
   "source": [
    "#### Exemplarisches Laden **einer** XML-Datei: "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d7cdea9-35f6-48be-a4a8-b24fb033132f",
   "metadata": {},
   "source": [
    "Direktes laden einer Datei: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e97cc70-4ed3-4d78-a925-41490daca404",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open (\"La_Otra_Alemania/1942-01-01_BWSVNEKFTM7SPW4SQAWQQNGIHHTRFDS4/DDB_FULLTEXT/5.xml\") as f: \n",
    "    content = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdcf27e7-9ed3-450e-ae0b-09f8fde61637",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a92ee86-b829-42bf-ac21-d2d4ad24e506",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(content))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7ec3d68-a1a7-4620-baa6-76531e2dc14b",
   "metadata": {},
   "source": [
    "Datei direkt als XML laden: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18da9813-16f9-411b-8fc5-34b03f3ec710",
   "metadata": {},
   "outputs": [],
   "source": [
    "xml = BeautifulSoup(open('La_Otra_Alemania/1942-01-01_BWSVNEKFTM7SPW4SQAWQQNGIHHTRFDS4/DDB_FULLTEXT/5.xml'),'lxml-xml') \n",
    "print(xml.prettify())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b5a2813-3152-4c2e-9c7a-0969f27d5658",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(xml))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8b4cada-572d-4cd5-afee-b952f2b5be0a",
   "metadata": {},
   "source": [
    "#### Laden **aller** .xml-Dateien aus einem Unterordner\n",
    "\n",
    "Laden aller Dateien aus dem Unterverzeichnis \"La_Otra_Alemania/1942-01-01_BWSVNEKFTM7SPW4SQAWQQNGIHHTRFDS4\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eed3331c-9399-48a0-b566-c8716dbf65da",
   "metadata": {},
   "outputs": [],
   "source": [
    "folder_xml = []\n",
    "\n",
    "for filepath in Path('./La_Otra_Alemania/1942-01-01_BWSVNEKFTM7SPW4SQAWQQNGIHHTRFDS4').glob('*/*.XML'):\n",
    "    with filepath.open() as f:\n",
    "        soup = BeautifulSoup(f,'lxml-xml')\n",
    "        folder_xml.append(soup) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f767b99-5c31-4dcc-90a4-46d1dda2cd79",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(folder_xml))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40cf60ca-229b-4d1a-adbc-0c470bb39d8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(folder_xml)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "680a5a80-bd97-4beb-be72-01ccfe323bd2",
   "metadata": {},
   "source": [
    "#### Laden ALLER .xml-Dateien im Unterverzeichnis \"La_Otra_Alemania\"\n",
    "\n",
    "Statt einer Liste wird nun ein Dictionary erzeugt, welches zur besseren Nachvollziehbarkeit als Schlüssel auch den jeweiligen Dateinamen enthält:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59426f21-e4cd-496a-971a-15cb300b43e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_xml = {}\n",
    "\n",
    "for filepath in Path('./La_Otra_Alemania').glob('**/*.XML'):\n",
    "    with filepath.open(encoding='utf-8') as f:\n",
    "        # read as string: \n",
    "        xml_string = f.read()\n",
    "        #Dateiname inklusive der Namen der beiden übergeordneten Ordner als Schlüssel:\n",
    "        key = f\"{filepath.parent.parent.name}/{filepath.parent.name}/{filepath.name}\"\n",
    "        all_xml[key] = xml_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57771e6e-b5c6-49da-a93a-cc68d619c4fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(all_xml))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4342950f-fa4d-49fd-88da-a98379c723c5",
   "metadata": {},
   "source": [
    "#### Kurze Erkärung zu .glob: \n",
    "\n",
    "- Mustererkennung: Die glob-Methode wird verwendet, um Dateipfade mit einem bestimmten Muster abzugleichen. In diesem Fall ist das Muster '\\*/\\*.XML'.\n",
    "- Musterdetails:\n",
    "    - '\\*' entspricht einer beliebigen Anzahl von Zeichen, einschließlich keinem.\n",
    "    - '\\*.XML' entspricht jeder Datei mit der Erweiterung '.XML'.\n",
    "    - Das Muster '\\*/\\*.XML' sucht speziell nach '.XML'-Dateien, die sich eine Verzeichnisebene unterhalb des angegebenen Pfads ('./Folder') befinden.\n",
    "- Rekursive Suche: Das Muster '\\*/\\*.XML' sucht nach XML-Dateien in allen unmittelbaren Unterverzeichnissen von './Folder'. Um rekursiv durch alle Unterverzeichnisse in beliebiger Tiefe zu suchen, wird das Muster '\\*\\*/\\*.XML' verwendet.\n",
    "\n",
    "\n",
    "#### Umwandlung in ein Pandas-Dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccef7d68-1810-4ef4-b6bf-79a44d7a8899",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(list(all_xml.items()), columns=['filename', 'content'])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4695bd76-2e9a-4f44-a763-83a5ddbb347c",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### Suchen von möglichen Problemen (optional): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b58cb63-664b-451b-85bc-def909e4265e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sucht nach dem Index von Reihen, in denen die Spalte \"text\" keinen Inhalt hat:\n",
    "empty_text_indices = df[df['text'] == \"\"].index\n",
    "\n",
    "# Ausgabe der gefundenen Zeilen ohne Text: \n",
    "for index in empty_text_indices:\n",
    "    print(f\"Index: {index}, Content: {df.loc[index, 'Filename']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b0066d8-e0de-44c3-a14b-9f5fb2c6f798",
   "metadata": {},
   "source": [
    "#### Text aus XML extrahieren: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6104b29-5dc6-4242-a29c-a7bb1366beaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_text(content):\n",
    "    # Remove XML declaration if present\n",
    "    if content.startswith('<?xml'):\n",
    "        content = content.split('?>', 1)[1]\n",
    "\n",
    "    NS = {'alto': 'http://www.loc.gov/standards/alto/ns-v2#'}\n",
    "    tree = etree.fromstring(unicodedata.normalize(\"NFC\", content))\n",
    "    \n",
    "    text_lines = []  # Initialize as an empty list to store text lines\n",
    "    \n",
    "    for line in tree.xpath('//alto:TextLine', namespaces=NS):\n",
    "        text = \" \".join(\n",
    "            word for word in line.xpath('alto:String/@CONTENT', namespaces=NS))\n",
    "        text_lines.append(text)  # Append each extracted text line to the list\n",
    "    \n",
    "    return \" \".join(text_lines)  # Return all text as a single string"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd66949e-35f7-4ccf-9050-4ec3ebf0e21d",
   "metadata": {},
   "source": [
    "Erstellen einer neuen Spalte \"text\". In diese wird das jeweilige Ergebnis der Anwendung der Funktion \"extract_text\" auf das korrespondierende Element in der Spalte \"content\" des Dataframes \"df\" geschrieben: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6832a392-f865-41dc-96c4-5340b08c2138",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['text'] = df['content'].apply(extract_text)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83bdbff7-d347-4bad-92f8-6e6aca419b09",
   "metadata": {},
   "source": [
    "Löschen der Spalte \"content\" um Speicherplatz zu sparen: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54dafaba-6a0f-4e6f-b0a1-fb1e84eb0327",
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = df.drop(columns=['content'])\n",
    "df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2212facb-5caa-47d3-8563-3a05c76434b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Zeilen löschen, in denen die Spalte 'Text' leer ist\n",
    "df2 = df2[df2['text'].notna() & (df2['text'] != '')]\n",
    "df2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e6e8ece-ff63-4bd2-aa60-b3b7231f8e9a",
   "metadata": {},
   "source": [
    "Speichern als CSV: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41c00e96-8429-4b90-88fb-b2e8a67a19cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.to_csv(\"otra_alemania_content.csv\", encoding = \"UTF-8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9811df28-01e0-4f98-968f-705a5e15cc92",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}