Check Kingfisher Collect log file

check_for_errors.ipynb

@@ -4,8 +4,7 @@
   "metadata": {
     "colab": {
       "name": "data_collection_and_processing_errors",
-      "provenance": [],
-      "authorship_tag": "ABX9TyPffcgelqfpE7r9y+mDKzan"
+      "provenance": []
     },
     "kernelspec": {
       "name": "python3",
@@ -22,6 +21,43 @@
         "## Check for data collection and processing errors"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Kingfisher Collect Log"
+      ],
+      "metadata": {
+        "id": "DWcRuKnZt--_"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Print the crawler statistics from the log file specified in the setup section. If `downloader/response_status_count/{code}` is non-zero and `{code}` is an HTTP error code (400-599), then the collection may be incomplete. Where possible, you should check the total number of releases and/or contracting processes against the front-end of the data source."
+      ],
+      "metadata": {
+        "id": "YoxNFk17uFZe"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "if log_url != '':\n",
+        "\n",
+        "  response = requests.get(log_url, auth=('scrape', scrapy_password))\n",
+        "\n",
+        "  with open('log_file', 'wb') as f:\n",
+        "    f.write(response.content)\n",
+        "  \n",
+        "  log = ScrapyLogFile('log_file').logparser\n",
+        "  pprint(dict(log['crawler_stats']))"
+      ],
+      "metadata": {
+        "id": "kfzwh_ExuEVX"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -409,4 +445,4 @@
       "outputs": []
     }
   ]
-}
+}

setup_environment.ipynb

@@ -5,7 +5,7 @@
     "colab": {
       "name": "setup_notebook",
       "provenance": [],
-      "authorship_tag": "ABX9TyPpPT0FsHcqUr/d/JNLgAmy"
+      "authorship_tag": "ABX9TyPajfocnRv6KzHqBnH2Lywx"
     },
     "kernelspec": {
       "name": "python3",
@@ -44,13 +44,39 @@
       "execution_count": null,
       "outputs": []
     },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "To check that the Kingfisher Collect scraper for your collection completed sucessfully, set the URL of the log file for the job from [Kingfisher Collect](https://collect.kingfisher.open-contracting.org), e.g. `https://collect.kingfisher.open-contracting.org/logs/kingfisher/united_kingdom_contracts_finder_releases/2ef35b40b93b11ec88920c9d92c523cb.log`."
+      ],
+      "metadata": {
+        "id": "CEmXhx6wn_Ft"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import getpass\n",
+        "\n",
+        "log_url = ''\n",
+        "\n",
+        "if log_url != '':\n",
+        "\n",
+        "  scrapy_password = getpass.getpass('Enter the password for collect.kingfisher.open-contracting.org:')"
+      ],
+      "metadata": {
+        "id": "bhtlijjgmBU2"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
     {
       "cell_type": "markdown",
       "metadata": {
         "id": "F8fwq_F6XMLs"
       },
       "source": [
-        "Enter credentials:"
+        "Enter your Kingfisher Process credentials:"
       ]
     },
     {
@@ -59,9 +85,6 @@
         "id": "27yRx-oaXPtd"
       },
       "source": [
-        "import getpass\n",
-        "\n",
-        "print('Enter your Kingfisher credentials')\n",
         "user = input('Username:')\n",
         "password = getpass.getpass('Password:')"
       ],
@@ -86,7 +109,13 @@
         "# Install Kingfisher Colab and required packages\n",
         "%shell pip install --upgrade 'ocdskingfishercolab<0.4' psycopg2-binary > pip.log\n",
         "\n",
+        "# Install Scrapy Log Analyzer\n",
+        "!pip install git+https://github.com/open-contracting/scrapy-log-analyzer.git\n",
+        "\n",
         "# Import libraries and functions\n",
+        "import pandas as pd\n",
+        "import requests\n",
+        "\n",
         "from google.colab.data_table import DataTable\n",
         "from ocdskingfishercolab import (\n",
         "    list_source_ids,\n",
@@ -96,7 +125,8 @@
         "    save_dataframe_to_sheet,\n",
         "    save_dataframe_to_spreadsheet,\n",
         "    set_search_path)\n",
-        "import pandas as pd\n",
+        "from pprint import pprint\n",
+        "from scrapyloganalyzer import ScrapyLogFile\n",
         "\n",
         "# Load https://pypi.org/project/ipython-sql/\n",
         "%load_ext sql \n",
@@ -142,6 +172,17 @@
       ],
       "execution_count": null,
       "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!"
+      ],
+      "metadata": {
+        "id": "sQL1XchVxUlp"
+      },
+      "execution_count": null,
+      "outputs": []
     }
   ]
-}
+}