24_MongoDB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MongoDB简介\n",
    "[官网](https://www.mongodb.com/) | [文档入口](https://docs.mongodb.com/) | [MongoDB Python Drivers文档](https://docs.mongodb.com/ecosystem/drivers/python/) | [MongoDB 大学](https://university.mongodb.com/courses/M001/about)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 安装\n",
    "\n",
    "1. 下载[MongoDB Server](https://www.runoob.com/mongodb/mongodb-osx-install.html) | [PyMongo](https://www.runoob.com/python3/python-mongodb.html)\n",
    "* 先创建数据存储地址 `mkdir -p ./data/db`\n",
    "* 再启动 mongo 服务端 `mongod --dbpath=./data/db`\n",
    "* 最后`import pymongo` 就可以操作了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 概念\n",
    "数据结构\n",
    "* Databases 数据库\n",
    "* Collections 表\n",
    "* Documents JSON格式的数据\n",
    "\n",
    "数据操作\n",
    "* CRUD (Create Read Update Delete) 增删改查 \n",
    "\n",
    "工具\n",
    "* MongoDB Atlas 云服务\n",
    "* MongoDB Compass 数据管理客户端"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据操作\n",
    "[PyMongo文档](https://pymongo.readthedocs.io/en/stable/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pymongo import MongoClient,DESCENDING\n",
    "client = MongoClient('mongodb://127.0.0.1:27017')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 新增"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "db = client.test_db\n",
    "users = db.users\n",
    "result = users.insert_one({'name':'hawk'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result.inserted_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 查询"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.list_database_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "db.list_collection_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "users.find_one()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for item in users.find({'name': 'hawk'}):\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 计数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "users.count_documents({'name': 'hawk'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for item in users.find().sort('_id', DESCENDING):\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 其他"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for item in users.find().sort('_id', DESCENDING).skip(1).limit(2):\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 删除"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "users.delete_one({'name':'hawk'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "db.drop_collection('users')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.drop_database('test_db')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 修改\n",
    "[Update Operator文档](https://docs.mongodb.com/manual/reference/operator/update/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "users.update_one({'name':'hawk'},{'$set':{'age':30}})\n",
    "users.update_one({'age':30},{'$inc':{'age':2}})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for item in users.find():\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 筛选条件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "<table><thead><tr><th>符号</th><th>含义</th><th>示例</th></tr></thead><tbody><tr><td><code>$lt</code></td><td>小于</td><td><code>{'age': {'$lt': 20}}</code></td></tr><tr><td><code>$gt</code></td><td>大于</td><td><code>{'age': {'$gt': 20}}</code></td></tr><tr><td><code>$lte</code></td><td>小于等于</td><td><code>{'age': {'$lte': 20}}</code></td></tr><tr><td><code>$gte</code></td><td>大于等于</td><td><code>{'age': {'$gte': 20}}</code></td></tr><tr><td><code>$ne</code></td><td>不等于</td><td><code>{'age': {'$ne': 20}}</code></td></tr><tr><td><code>$in</code></td><td>在范围内</td><td><code>{'age': {'$in': [20, 23]}}</code></td></tr><tr><td><code>$nin</code></td><td>不在范围内</td><td><code>{'age': {'$nin': [20, 23]}}</code></td></tr></tbody></table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "<table><thead><tr><th>符号</th><th>含义</th><th>示例</th><th>示例含义</th></tr></thead><tbody><tr><td><code>$regex</code></td><td>匹配正则表达式</td><td><code>{'name': {'$regex': '^M.*'}}</code></td><td><code>name</code>以M开头</td></tr><tr><td><code>$exists</code></td><td>属性是否存在</td><td><code>{'name': {'$exists': True}}</code></td><td><code>name</code>属性存在</td></tr><tr><td><code>$type</code></td><td>类型判断</td><td><code>{'age': {'$type': 'int'}}</code></td><td><code>age</code>的类型为<code>int</code></td></tr><tr><td><code>$mod</code></td><td>数字模操作</td><td><code>{'age': {'$mod': [5, 0]}}</code></td><td>年龄模5余0</td></tr><tr><td><code>$text</code></td><td>文本查询</td><td><code>{'$text': {'$search': 'Mike'}}</code></td><td><code>text</code>类型的属性中包含<code>Mike</code>字符串</td></tr><tr><td><code>$where</code></td><td>高级条件查询</td><td><code>{'$where': 'obj.fans_count == obj.follows_count'}</code></td><td>自身粉丝数等于关注数</td></tr></tbody></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "详细规则 https://docs.mongodb.com/manual/reference/operator/query/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 合并操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find_one_and_delete\n",
    "# find_one_and_replace\n",
    "users.find_one_and_update({'age':32},{'$inc':{'age':2}})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 实践案例，爬人人都是产品经理网站\n",
    "> MongoDB的存储位置为./data/db\n",
    "\n",
    "现存问题：\n",
    "- [x] 爬取url太慢：用代理并行去爬\n",
    "- [ ] 爬url，经常overtry，因为网站经常会卡死，设定workers少一点"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# default_exp RRPM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "import requests,re,time,sys\n",
    "from bs4 import BeautifulSoup\n",
    "from crawler_from_scratch import IpPool\n",
    "from pymongo import MongoClient\n",
    "from datetime import datetime\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "client = MongoClient('mongodb://127.0.0.1:27017')\n",
    "db = client.crawler\n",
    "art_coll = db.articals\n",
    "\n",
    "category = 'pd'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "def simple_request(url) -> object:\n",
    "    res = requests.get(url,headers={'user-agent':'Mozilla/5.0'})\n",
    "    soup = BeautifulSoup(res.text)\n",
    "    return soup.body"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成待爬url list\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def crawl_artical_url_v1(category_page_url) -> iter:\n",
    "    res = requests.get(category_page_url)\n",
    "    soup = BeautifulSoup(res.text,features='lxml').body\n",
    "    for s in soup.find_all('h2','post-title'):\n",
    "        url = s.a['href']\n",
    "        if 'woshipm' in url: yield url\n",
    "    \n",
    "def get_urls_v1(start,end) -> list:\n",
    "    global category\n",
    "    urls = []\n",
    "    category_url = f'http://www.woshipm.com/category/{category}'\n",
    "    for i in range(start,end+1):\n",
    "        page_url = f'{category_url}/page/{i}'\n",
    "        print(page_url)\n",
    "        for url in crawl_artical_url_v1(page_url):\n",
    "            urls.append(url)\n",
    "    print('urls:',len(urls),'per_page:',len(urls)/(end-start+1))\n",
    "    urls = list(set(urls))\n",
    "    return urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_urls_v1(20,30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def crawl_artical_url(category_page_url) -> list:\n",
    "    urls = []\n",
    "    res = IpPool.proxy_get(category_page_url)\n",
    "    soup = BeautifulSoup(res.text,features='lxml').body\n",
    "    for s in soup.find_all('h2','post-title'):\n",
    "        url = s.a['href']\n",
    "        if 'woshipm' in url: urls.append(url)\n",
    "    \n",
    "    # 获取失败的情况\n",
    "    if len(urls) == 0: \n",
    "        print('\\n\\n!no urls:',category_page_url, 'retrying......')\n",
    "        urls = crawl_artical_url(category_page_url)\n",
    "    print('crawl urls:',category_page_url,'got:',len(urls))\n",
    "    return urls\n",
    "\n",
    "def parallel_crawl_artical_url(start,end) -> list:\n",
    "    global category\n",
    "    urls = []\n",
    "    page_urls = [f'http://www.woshipm.com/category/{category}/page/{i}' for i in range(start,end+1)]\n",
    "    for data in IpPool.parallel_task(crawl_artical_url,page_urls,10):\n",
    "        urls += data\n",
    "    urls = list(set(urls))\n",
    "    print('urls:',len(urls))\n",
    "    return urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.7yip.cn/free/ 新增：0，库存更新为：2122个\n",
      "http://www.ip3366.net/free/ 新增：4，库存更新为：2126个\n",
      "http://www.nimadaili.com/gaoni/ 新增：37，库存更新为：2163个\n",
      "http://www.xiladaili.com/gaoni/ 新增：3，库存更新为：2166个\n",
      "https://www.kuaidaili.com/free/inha/ 新增：0，库存更新为：2167个\n",
      "https://ip.jiangxianli.com/?anonymity=2 新增：7，库存更新为：2173个\n",
      "http://proxyslist.com/ 新增：23，库存更新为：2196个\n",
      "https://list.proxylistplus.com/Fresh-HTTP-Proxy-List-1 新增：1，库存更新为：2197个\n",
      "移除： 0 个IP\n",
      "success: http://www.woshipm.com/category/it/page/2 try times: 1\n",
      "crawl urls: http://www.woshipm.com/category/it/page/2 got: 12\n"
     ]
    }
   ],
   "source": [
    "crawl_artical_url('http://www.woshipm.com/category/it/page/2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "category = 'it'\n",
    "urls = parallel_crawl_artical_url(1,100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['http://www.woshipm.com/it/3690544.html',\n",
       " 'http://www.woshipm.com/it/3690303.html',\n",
       " 'http://www.woshipm.com/it/3679467.html',\n",
       " 'http://www.woshipm.com/it/3689204.html',\n",
       " 'http://www.woshipm.com/it/3688145.html',\n",
       " 'http://www.woshipm.com/it/3658393.html',\n",
       " 'http://www.woshipm.com/it/3685361.html',\n",
       " 'http://www.woshipm.com/it/3685687.html',\n",
       " 'http://www.woshipm.com/it/3684275.html',\n",
       " 'http://www.woshipm.com/it/3679947.html',\n",
       " 'http://www.woshipm.com/it/3684034.html',\n",
       " 'http://www.woshipm.com/it/3678488.html',\n",
       " 'http://www.woshipm.com/it/3678488.html',\n",
       " 'http://www.woshipm.com/it/3686449.html',\n",
       " 'http://www.woshipm.com/it/3686307.html',\n",
       " 'http://www.woshipm.com/it/3683307.html',\n",
       " 'http://www.woshipm.com/it/3679936.html',\n",
       " 'http://www.woshipm.com/it/3684661.html',\n",
       " 'http://www.woshipm.com/it/3682891.html',\n",
       " 'http://www.woshipm.com/it/3678595.html',\n",
       " 'http://www.woshipm.com/it/3678962.html',\n",
       " 'http://www.woshipm.com/it/3677359.html',\n",
       " 'http://www.woshipm.com/it/3677782.html',\n",
       " 'http://www.woshipm.com/it/3679192.html']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 代理访问&保存html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "def save_artical(response) -> int:\n",
    "    global category\n",
    "    url = response.url\n",
    "    _id = int(re.search(r'\\d+',url).group())\n",
    "    \n",
    "    soup = BeautifulSoup(response.text,features='lxml')\n",
    "    content = str(soup.find('div','single-wrapper'))\n",
    "\n",
    "    try:\n",
    "        artical = art_coll.insert_one({'_id':_id,\n",
    "                                       'category':category,\n",
    "                                       'url':url,\n",
    "                                       'html':content})\n",
    "        return artical.inserted_id\n",
    "    except:\n",
    "        print('\\n\\n!error',url,sys.exc_info())\n",
    "    \n",
    "def crawl_artical(url):\n",
    "    global category,art_coll\n",
    "    _id = int(re.search(r'\\d+',url).group())\n",
    "    \n",
    "    # 爬过，则结束\n",
    "    if art_coll.find_one({'_id':_id}):\n",
    "        print('exist:',url)\n",
    "        return\n",
    "    \n",
    "    # 重复爬10次，直到res=200，不然就log\n",
    "    res = IpPool.proxy_get(url) \n",
    "    if res: save_artical(res)          "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "exist: http://www.woshipm.com/rp/3660512.html\n"
     ]
    }
   ],
   "source": [
    "crawl_artical('http://www.woshipm.com/rp/3660512.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 批量操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "def crawl_artical_url_and_html(category_page_url):\n",
    "    urls = crawl_artical_url(category_page_url)\n",
    "    IpPool.parallel_task(crawl_artical,urls,10,False)\n",
    "    print('complete:',category_page_url)\n",
    "    return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "def crawl_artical_by_category():\n",
    "    global category\n",
    "    category = 'blockchain'\n",
    "    start,end = 1,20\n",
    "    category_page_urls = [f'http://www.woshipm.com/category/{category}/page/{i}' for i in range(start,end+1)]\n",
    "    IpPool.parallel_task(crawl_artical_url_and_html,category_page_urls,20)\n",
    "#     urls = parallel_crawl_artical_url(1,133)    \n",
    "# #     urls = get_urls_v1(1,257)\n",
    "#     IpPool.parallel_task(crawl_artical,urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 完善数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_no_html() -> list:\n",
    "    no_html_urls = []\n",
    "#     for a in art_coll.find({'html':{'$exists':True},'$where':\"(this.html.length < 100)\"}):\n",
    "    for a in art_coll.find({'html':{'$exists':False}}):\n",
    "        no_html_urls.append(a['url'])\n",
    "    print(len(no_html_urls))\n",
    "    return no_html_urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_html(url):\n",
    "    res = IpPool.proxy_get(url)\n",
    "    soup = BeautifulSoup(res.text,features='lxml')\n",
    "    content = str(soup.find('div','single-wrapper'))\n",
    "    result = art_coll.update_one({'url':url},{'$set':{'html':content}})\n",
    "    print('update:',url,result.modified_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "check_no_html()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IpPool.parallel_task(update_html,check_no_html())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 解析内容字段"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "def parse_ymd(s):\n",
    "    year_s, mon_s, day_s = s.split('-')\n",
    "    return datetime(int(year_s), int(mon_s), int(day_s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "def parse_html(_id=3614430) -> object:\n",
    "    update_content = {}\n",
    "    try:\n",
    "        soup = BeautifulSoup(art_coll.find_one({'_id':_id})['html'],features='lxml')\n",
    "        update_content['title'] = soup.h2.text\n",
    "        author_soup = soup.find('div','postMetaLockup--authorWithBio u-flex')\n",
    "        update_content['author_link'] = author_soup.a['href']\n",
    "\n",
    "        author_info = [s for s in author_soup.stripped_strings]\n",
    "        update_content['author_name'] = author_info[0]\n",
    "        update_content['likes'] = int(author_info[-1])\n",
    "        update_content['stars'] = int(author_info[-2])\n",
    "        update_content['views'] = int(author_info[-3])\n",
    "        update_content['publish_date'] = parse_ymd(author_info[-4])\n",
    "        update_content['tags'] = [a.text for a in soup.find('div','taglist')] if soup.find('div','taglist') else []\n",
    "    except:\n",
    "        print('parse error:',_id)\n",
    "    return update_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "def update_artical_info(_id) -> int: \n",
    "    result = art_coll.update_one({'_id':_id},{'$set':parse_html(_id)})\n",
    "    return result.modified_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "parse error: 1633891\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'title': '如何报名'}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parse_html(1633891)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "update_artical_info(3614430)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "# http://www.woshipm.com/it/793639.html\n",
    "# http://www.woshipm.com/it/1633891.html\n",
    "# 这两个是培训课程，需删掉\n",
    "def update_all_artical_info():\n",
    "    ids = [i['_id'] for i in art_coll.find({'likes':{'$exists':False}})]\n",
    "    task_results = IpPool.parallel_task(update_artical_info,ids,1)\n",
    "    modify_count = len([i for i in task_results if i> 0])\n",
    "    print('modify:',modify_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 检查数据完整性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "def correct_field_type(_id=3614430):\n",
    "    art = art_coll.find_one({'_id':_id})\n",
    "    if type(art['views']) == str:\n",
    "        if '万' in art['views']:\n",
    "            art['views'] = float(art['views'].replace('万',''))*10000\n",
    "        elif 'm' in art['views']:\n",
    "            art['views'] = float(art['views'].replace('m',''))*1000000\n",
    "            \n",
    "        result = art_coll.update_one({'_id':_id},\n",
    "                                     {'$set':{'stars':int(art['stars']),\n",
    "                                              'likes':int(art['likes']),\n",
    "                                              'views':int(art['views']),\n",
    "                                              'publish_date':parse_ymd(art['publish_date'])}})\n",
    "        return result.modified_count\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_field_type(3624725)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converted 24_MongoDB.ipynb.\r\n"
     ]
    }
   ],
   "source": [
    "!nbdev_build_lib --fname 24_MongoDB.ipynb\n",
    "!mv crawler_from_scratch/RRPM.py RRPM.py "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "progress:20.00%\n",
      "time cost: 0 s 5 tasks\n"
     ]
    }
   ],
   "source": [
    "# export\n",
    "if __name__ == '__main__': \n",
    "    ids = [i['_id'] for i in art_coll.find()]\n",
    "    IpPool.parallel_task(correct_field_type,ids,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据库导出备份\n",
    "https://cloud.tencent.com/document/product/240/5321"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 导出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_coll = db.test\n",
    "test_coll.insert_many(art_coll.find().limit(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-04-17T09:17:11.211+0800\tconnected to: mongodb://127.0.0.1:27017/\r\n",
      "2020-04-17T09:17:11.227+0800\texported 10 records\r\n"
     ]
    }
   ],
   "source": [
    "# mongoexport --host 10.66.187.127:27017 -u mongouser -p thepasswordA1 --authenticationDatabase=admin --db=testdb --collection=testcollection -o /data/export_testdb_testcollection.json\n",
    "!mongoexport --host 127.0.0.1:27017 --db=crawler --collection=test -o ./data/export_test_coll.json\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 导入\n",
    "可以识别date格式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-04-17T09:19:12.501+0800\tconnected to: mongodb://127.0.0.1:27017/\n",
      "2020-04-17T09:19:12.573+0800\t10 document(s) imported successfully. 0 document(s) failed to import.\n"
     ]
    }
   ],
   "source": [
    "# mongoimport --host 10.66.187.127:27017 -u mongouser -p thepasswordA1 --authenticationDatabase=admin --db=testdb --collection=testcollection2 --file=/data/export_testdb_testcollection.json\n",
    "!mongoimport --host 127.0.0.1:27017 --db=crawler --collection=test2 --file=./data/export_test_coll.json\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### rrpm文章数据备份"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 所有文章的html源文件\n",
    "rrpm_raw_coll = db.rrpm_raw\n",
    "for i in art_coll.find():\n",
    "# for i in art_coll.find().limit(5):\n",
    "    rrpm_raw_coll.insert_one({'html':i['html'],'url':i['url']})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mongoexport --host 127.0.0.1:27017 --db=crawler --collection=rrpm_raw -o ./data/export_rrpm_raw.json\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 重新建立字段数据库\n",
    "把有用的都保存下来，删除原始数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "def parse_ymd(s):\n",
    "    year_s, mon_s, day_s = s.split('-')\n",
    "    return datetime(int(year_s), int(mon_s), int(day_s))\n",
    "\n",
    "def parse_num(string):\n",
    "    if '万' in string:\n",
    "        return int(float(string.replace('万',''))*10000)\n",
    "    elif 'm' in string:\n",
    "        return int(float(string.replace('m',''))*1000000)\n",
    "    else:\n",
    "        try:\n",
    "            return int(string)\n",
    "        except:\n",
    "            print('error:',string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_comment(soup) -> object:\n",
    "    cmt = {}\n",
    "    comment = soup\n",
    "    comment_author = comment.find('div','comment-author')\n",
    "    if comment_author.a:\n",
    "        cmt['author_name'] = comment_author.a.text.strip()\n",
    "        cmt['author_link'] = comment_author.a['href']\n",
    "    cmt['content'] = comment.find('div','comment-content').text.strip()\n",
    "    cmt['comment_time'] = datetime.strptime(comment.find('span','comment-time')['datetime'],\"%Y-%m-%dT%H:%M:%S+08:00\")\n",
    "    return cmt\n",
    "\n",
    "def parse_comments(soup) -> list:\n",
    "    cmt_list = []\n",
    "    for item in soup.find_all('li','comment'): \n",
    "        cmt_list.append(parse_comment(item))\n",
    "    return cmt_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_artical(soup) -> object:\n",
    "    artical = {}\n",
    "    \n",
    "    artical['title'] = soup.h2.text\n",
    "    artical['publish_date'],artical['views'],artical['stars'],artical['likes'] = [item.text.strip() for item in soup.find_all('span','post-meta-item')]\n",
    "    \n",
    "    author_info = soup.find('div','postMetaLockup--authorWithBio')    \n",
    "    artical['author_name'] = author_info.find_all('a')[1].text.strip()    \n",
    "    artical['author_link'] = author_info.find_all('a')[1]['href']    \n",
    "    artical['author_desc'] = author_info.find('div','des').text.strip()  \n",
    "    \n",
    "    main = soup.find('div','grap')\n",
    "    artical['contents'] = list(main.stripped_strings)\n",
    "    artical['imgs'] = [item['src'] for item in main.find_all('img') if 'src' in item.attrs]\n",
    "    \n",
    "    artical['tags'] = [item.text for item in soup.find('div','taglist')] if soup.find('div','taglist') else []\n",
    "    \n",
    "    artical['publish_date'] = parse_ymd(artical['publish_date'])\n",
    "    artical['views'] = parse_num(artical['views'])\n",
    "    artical['stars'] = parse_num(artical['stars'])\n",
    "    artical['likes'] = parse_num(artical['likes'])\n",
    "    \n",
    "    \n",
    "    artical['comments'] = parse_comments(soup)\n",
    "    return artical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_artical_to_mongodb(raw_item):\n",
    "    try:\n",
    "        soup = BeautifulSoup(raw_item['html'],features='lxml')\n",
    "        _id = int(re.search(r'\\d+',raw_item['url']).group())\n",
    "        artical = parse_artical(soup)\n",
    "        artical['_id'] = _id\n",
    "        artical['url'] = raw_item['url']\n",
    "        if not rrpm_articals_coll.insert_one(artical):\n",
    "            print('save failed:',raw_item['url'])\n",
    "            with open('failed_urls.text','a') as f:\n",
    "                f.write(raw_item['url']+'\\n')\n",
    "\n",
    "    except:\n",
    "        print('parse error:',raw_item['url'])\n",
    "        with open('failed_urls.text','a') as f:\n",
    "            f.write(raw_item['url']+'\\n')\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rrpm_articals_coll = db.rrpm_articals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "items = rrpm_raw_coll.find()\n",
    "IpPool.parallel_task(save_artical_to_mongodb,items)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 发布"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[master b5eeff2] restruct database\r\n",
      " 1 file changed, 491 insertions(+), 10 deletions(-)\r\n"
     ]
    }
   ],
   "source": [
    "!git add 24_MongoDB.ipynb\n",
    "!git commit -m \"restruct database\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nbdev_build_lib --fname 24_MongoDB.ipynb\n",
    "!mv crawler_from_scratch/RRPM.py RRPM.py "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}