{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [],
      "source": [
        "def hint(string): return print(\u0027\u0027.join(map(chr, map(lambda x: int(x, 16), string.split(\u0027.\u0027)))))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## Environment\n",
        "+ Pycharm\n",
        "+ python3.6\n",
        "+ networkx\n",
        "+ jieba\n",
        "+ numpy, pandas, matplotlib\n",
        "+ gensim"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## References"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "+ https://nlp.stanford.edu/pubs/see2017get.pdf\n",
        "+ https://web.eecs.umich.edu/~mihalcea/papers/mihalcea.emnlp04.pdf\n",
        "+ https://openreview.net/pdf?id\u003dSyK00v5xx\n",
        "+ https://arxiv.org/abs/1409.0473\n",
        "+ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## Background Knowledge"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q1. 简述自动摘要技术的三个应用场景？"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答：\n",
        "1. 新闻简报（对长新闻的自动提取）\n",
        "2. 微信公众号文章的自动摘要\n",
        "3. paper文章的自动摘要\n",
        "4. 谷歌登搜索引擎搜索后看到的简短摘要"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "##### 提示，如果需要帮助，可以运行以下的提示"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "# remove the # before hint(), to show the example sulution\n",
        "#hint(\u0027a.641c.7d22.5f15.64ce.ff1b.a.32.2e.20.89c2.70b9.62bd.53d6.ff1b.a.33.2e.20.65b0.95fb.ff1b.a.34.2e.20.6c47.62a5.6587.6863.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q2. 目前的摘要技术分为哪两类？"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答：\n",
        "1. Extraction 抽取式\n",
        "2. Abstraction 生成式\n",
        "\n",
        "\u003e 目前Extraction抽取式的主要方法：\n",
        "\n",
        "\u003e\u003e - 基于统计：统计词频，位置等信息，计算句子权值，再简选取权值高的句子作为文摘，特点：简单易用，但对词句的使用大多仅停留在表面信息。\n",
        "\n",
        "\u003e\u003e - 基于图模型：构建拓扑结构图，对词句进行排序。例如，TextRank/LexRank。\n",
        "\n",
        "\u003e\u003e - 基于潜在语义：使用主题模型，挖掘词句隐藏信息。例如，采用LDA，HMM。\n",
        "\n",
        "\u003e\u003e - 基于线路规划：将摘要问题转为线路规划，求全局最优解。\n",
        "\n",
        "\u003e\u003e 在python语言中用于文本摘要自动提取的库包括goose，SnowNLP，TextTeaser，sumy，TextRank等。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.6458.8981.20.41.62.73.74.72.61.63.74.69.76.65.3a.20.91cd.65b0.6574.7406.4fe1.606f.a.62bd.53d6.20.45.78.74.72.61.63.74.69.76.65.3a.20.4ece.539f.6587.4e2d.63d0.53d6.90e8.5206.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q3. 目前实现自动摘要的难点在哪里？列出三点。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答：\n",
        "1. 生成字数简短、精炼，读起来通顺的摘要，如何评价摘要比较难\n",
        "2. 找到权重最高的句子\n",
        "3. 中文多文档摘要"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.31.2e.20.7f3a.4e4f.6807.6ce8.8bad.7ec3.6570.636e.ff1b.a.32.2e.20.57fa.4e8e.6587.672c.7406.89e3.ff0c.63d0.53d6.91cd.8981.5185.5bb9.96be.ff1b.a.33.2e.20.4eba.7c7b.5bf9.6587.672c.903b.8f91.654f.611f.ff0c.6458.8981.7ed3.679c.5bb9.9519.7387.4f4e.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## Basic Technologies"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q4. 什么是PageRank？ PageRank 用在什么地方？用 $Latex$ 写出其方程"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答： \n",
        "\n",
        "（简单）**Pagerank算法是Google的网页排名算法，通过网页间的互相‘投票’得到网页的重要性分数。**\n",
        "\n",
        "（详细）PageRank 是一种通过网页之间的超链接来计算网页重要性的技术，以 Google 创办人 Larry Page 之姓来命名，Google 用它来体现网页的相关性和重要性。PageRank 通过网络浩瀚的超链接关系来确定一个页面的等级，把从 A 页面到 B 页面的链接解释为 A 页面给 B 页面投票，Google 根据 A 页面（甚至链接到 A 的页面）的等级和投票目标的等级来决定 B 的等级。简单的说，一个高等级的页面可以使其他低等级页面的等级提升。\n",
        "\n",
        "整个互联网可以看作是一张有向图图，网页是图中的节点，网页之间的链接就是图中的边。如果网页 A 存在到网页 B 的链接，那么就有一条从网页 A 指向网页 B 的有向边。\n",
        "\n",
        "构造完图后，使用下面的公式来计算网页 i 的重要性（PR值）：\n",
        "\n",
        "$$ S(V_{i}) \u003d (1 - d) + d \\cdot \\sum_{j \\in In(V_{i})} \\frac{1}{\\vert {Out(V_{j})} \\vert} S(V_{j}) $$\n",
        "\n",
        "- d是阻尼系数，一般设置为 0.85。\n",
        "- $In(V_{i})$是存在指向网页 i 的链接的网页集合。\n",
        "- $Out(V_{j})$是网页 j中的链接指向的网页的集合。\n",
        "- $\\vert {Out(V_{j})} \\vert$是集合中元素的个数。\n",
        "\n",
        "PageRank 需要使用上面的公式多次迭代才能得到结果。初始时，可以设置每个网页的重要性为 1。\n",
        " "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.31.2e.20.65b9.7a0b.3a.a.20.20.20.20.50.52.28.70.5f.69.29.20.3d.20.a.72.61.63.7b.31.2d.64.7d.7b.4e.7d.20.2b.20.64.20.5c.73.75.6d.5f.7b.70.5f.6a.20.5c.69.6e.20.4d.28.70.5f.69.29.7d.20.a.72.61.63.7b.50.52.20.28.70.5f.6a.29.7d.7b.4c.28.70.5f.6a.29.7d.a.32.2e.20.53c2.8003.ff1a.a.20.20.20.20.68.74.74.70.73.3a.2f.2f.77.77.77.2e.77.69.6b.69.77.61.6e.64.2e.63.6f.6d.2f.65.6e.2f.50.61.67.65.52.61.6e.6b.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q5. PageRank和TextRank的关系是什么，TextRank的背景原理是什么？ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答：\n",
        "\n",
        "（简单）**TextRank是PageRank算法在文本关键词提取和摘要抽取上的应用。**\n",
        "\n",
        "（简单）**TextRank把文章中的每一个词视为图上的一个节点，在一定窗口范围内的词之间建立连接关系，通过PageRank的迭代算法得到每个词的重要性，进而排序提取前几个词作为关键词。**\n",
        "\n",
        "（详细）TextRank 公式在 PageRank 公式的基础上，为图中的边引入了权值的概念：\n",
        "\n",
        "$$ WS(V_{i}) \u003d (1 - d) + d \\cdot \\sum_{j \\in In(V_{i})} \\frac{w_{ji}}{\\sum_{V_{k} \\in {Out(V_{j})}} w_{ji} } WS(V_{j}) $$\n",
        "\n",
        "- $w_{ji}$就是是为图中节点 $V_{i}$到 $V_{j}$ 的边的权值 。\n",
        "- d 依然为阻尼系数，代表从图中某一节点指向其他任意节点的概率，一般取值为0.85。\n",
        "- $In(V_{i})$和 $Out(V_{j})$也和 PageRank 类似，分别为指向节点 V_{i}的节点集合和从节点 V_{i}出发的边指向的节点集合。\n",
        "\n",
        "在 TextRank 构建的图中，默认节点就是句子，权值 $w_{ji}$就是两个句子 $S_{i}$和 $S_{j}$的相似程度。两个句子的相似度使用下面的公式来计算：\n",
        "\n",
        "$$Similarity(S_{i}, S_{j}) \u003d \\frac{\\vert {w_{k} \\vert w_{k} \\in S_{i} \\\u0026  w_{k} \\in S_{j} }\\vert }{log(\\vert S_{i}) + log(\\vert S_{j})} $$ \n",
        "\n",
        "分子是在两个句子中都出现的单词的数量，|Si|是句子 i 中的单词数。\n",
        "使用 TextRank 算法计算图中各节点的得分时，同样需要给图中的节点指定任意的初值，通常都设为1。然后递归计算直到收敛，即图中任意一点的误差率小于给定的极限值时就可以达到收敛，一般该极限值取 0.0001。\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      },
      "source": [
        "## 使用 TextRank 提取摘要\n",
        "自动摘要，就是从文章中自动抽取关键句。人类对关键句的理解通常是能够概括文章中心的句子，而机器只能模拟人类的理解，即拟定一个权重的评分标准，给每个句子打分，之后给出排名靠前的几个句子。基于 TextRank 的自动文摘属于自动摘录，通过选取文本中重要度较高的句子形成文摘。\n",
        "\n",
        "依然使用 TextRank 公式：\n",
        "$$ WS(V_{i}) \u003d (1 - d) + d \\cdot \\sum_{j \\in In(V_{i})} \\frac{w_{ji}}{\\sum_{V_{k} \\in {Out(V_{j})}} w_{ji} } WS(V_{j}) $$\n",
        "\n",
        "等式左边表示一个句子的权重（WS 是 weight_sum 的缩写），右侧的求和表示每个相邻句子对本句子的贡献程度。与提取关键字的时候不同，一般认为全部句子都是相邻的，不再通过窗口提取。\n",
        "\n",
        "边的权值 Wij代表句子 Si 和 Sj 的相似度，既可以使用上面介绍过的基于句子间内容覆盖率的方法计算，也可以使用基于编辑距离，基于语义词典，余弦相似度，BM25 算法等等。\n",
        "因为我们是要抽取关键句，因而是以句子为基本单位。使用 TextRank 提取摘要的整个过程如下：\n",
        "\n",
        "1. 处理：将文本分割成句子 S1,S2,⋯,Sm，以句子为节点构建图\n",
        "2. 计算句子相似度：对句子进行分词、取停用词等处理，以便于计算任意两个句子之间的相似度。将计算好的句子相似度作为两个句子构成的边的权值\n",
        "3. 句子权重：根据公式，迭代传播权重计算各句子的得分。\n",
        "4. 抽取文摘句：得到的句子得分进行倒序排序，抽取重要度最高的 N 个句子作为候选文摘句。\n",
        "5. 形成文摘：根据字数或句子数要求，从候选文摘句中抽取句子组成文摘。\n",
        "\n",
        "\u003e 使用 TextRank 提取关键词（未整理进来），参考：https://xiaosheng.me/2017/04/08/article49/\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.53c2.8003.8bba.6587.ff1a.20.68.74.74.70.73.3a.2f.2f.77.65.62.2e.65.65.63.73.2e.75.6d.69.63.68.2e.65.64.75.2f.7e.6d.69.68.61.6c.63.65.61.2f.70.61.70.65.72.73.2f.6d.69.68.61.6c.63.65.61.2e.65.6d.6e.6c.70.30.34.2e.70.64.66.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q6. 在存在大量原始文本以及该文本对应的摘要内容时，我们是否可以使用 Seq2Seq 进行生成？ Sequence2Sequence的原理是什么？ Attention的机制原理是什么？ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "答：可以。\n",
        "seq2seq的原理是将输入序列进行编码，然后再解码得到输出序列的过程。\n",
        "训练编码器将输入的原文向量提取出关键的语义，作为解码器的输入；训练的答案是摘要文本的向量。原理就是将LSTM分别作为encoder和decoder来进行信息的提取与还原生成。\n",
        "\n",
        "\n",
        "attention的机制是对encoder的每一个隐藏状态有一个权值的打分评判，分数越高的，说明输出的结果越依赖当前encoder的输入，而让模型达到更加‘注意’这一个输入点的结果。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "参考地址：[完全图解RNN、RNN变体、Seq2Seq、Attention机制](https://zhuanlan.zhihu.com/p/28054589)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "seq2seq 是一个Encoder–Decoder 结构的网络，它的输入是一个序列，输出也是一个序列， Encoder 中将一个可变长度的信号序列变为固定长度的向量表达，Decoder 将这个固定长度的向量变成可变长度的目标的信号序列。\n",
        "这个结构最重要的地方在于输入序列和输出序列的长度是可变的，可以用于翻译，聊天机器人，句法分析，文本摘要等。\n",
        "![想象一下翻译任务，input是一段英文，output是一段中文。](https://pic1.zhimg.com/80/v2-d90b7f4c45a9641296ce0862fe8ce0ac_hd.jpg)\n",
        "\n",
        "\u003e [完全图解RNN、RNN变体、Seq2Seq、Attention机制](https://zhuanlan.zhihu.com/p/28054589)\n",
        "\n",
        "### Extractive, Abstractive \n",
        "+ Extractive是从文中找出关键信息，然后拼接进行结果输出\n",
        "    + 确定重要信息\n",
        "+ Abstracrtive: 依据文本的输入，生产单词（可能是新的单词）进行结果输出\n",
        "    + Seq2Seq的问题\n",
        "    + Transfomer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## Data Pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "我们首先加载数据"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import jieba\n",
        "import os\n",
        "import pandas as pd\n",
        "from collections import Counter\n",
        "import numpy as np\n",
        "# from opencc import OpenCC\n",
        "import jieba\n",
        "import random\n",
        "from gensim.models import FastText\n",
        "from gensim.models.word2vec import LineSentence\n",
        "\n",
        "def cut(string): return \u0027 \u0027.join(jieba.cut(string))\n",
        "path_root \u003d \u0027C:/AI-NLP/learn-NLP-luhuibo/data\u0027\n",
        "news_file \u003d os.path.join(path_root, \u0027sqlResult_1558435.csv\u0027) \n",
        "# this file could downloaded from github group\u0027s : \n",
        "#  https://github.com/Artificial-Intelligence-for-NLP-Chinese/datasource "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "\u0027C:/AI-NLP/learn-NLP-luhuibo/data\\\\sqlResult_1558435.csv\u0027"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "news_file"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border\u003d\"1\" class\u003d\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style\u003d\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eid\u003c/th\u003e\n",
              "      \u003cth\u003eauthor\u003c/th\u003e\n",
              "      \u003cth\u003esource\u003c/th\u003e\n",
              "      \u003cth\u003econtent\u003c/th\u003e\n",
              "      \u003cth\u003efeature\u003c/th\u003e\n",
              "      \u003cth\u003etitle\u003c/th\u003e\n",
              "      \u003cth\u003eurl\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e89617\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e快科技@http://www.kkj.cn/\u003c/td\u003e\n",
              "      \u003ctd\u003e此外，自本周（6月12日）起，除小米手机6等15款机型外，其余机型已暂停更新发布（含开发版/...\u003c/td\u003e\n",
              "      \u003ctd\u003e{\"type\":\"科技\",\"site\":\"cnbeta\",\"commentNum\":\"37\"...\u003c/td\u003e\n",
              "      \u003ctd\u003e小米MIUI 9首批机型曝光：共计15款\u003c/td\u003e\n",
              "      \u003ctd\u003ehttp://www.cnbeta.com/articles/tech/623597.htm\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e89616\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e快科技@http://www.kkj.cn/\u003c/td\u003e\n",
              "      \u003ctd\u003e骁龙835作为唯一通过Windows 10桌面平台认证的ARM处理器，高通强调，不会因为只考...\u003c/td\u003e\n",
              "      \u003ctd\u003e{\"type\":\"科技\",\"site\":\"cnbeta\",\"commentNum\":\"15\"...\u003c/td\u003e\n",
              "      \u003ctd\u003e骁龙835在Windows 10上的性能表现有望改善\u003c/td\u003e\n",
              "      \u003ctd\u003ehttp://www.cnbeta.com/articles/tech/623599.htm\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e2\u003c/td\u003e\n",
              "      \u003ctd\u003e89615\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e快科技@http://www.kkj.cn/\u003c/td\u003e\n",
              "      \u003ctd\u003e此前的一加3T搭载的是3400mAh电池，DashCharge快充规格为5V/4A。\\r\\n...\u003c/td\u003e\n",
              "      \u003ctd\u003e{\"type\":\"科技\",\"site\":\"cnbeta\",\"commentNum\":\"18\"...\u003c/td\u003e\n",
              "      \u003ctd\u003e一加手机5细节曝光：3300mAh、充半小时用1天\u003c/td\u003e\n",
              "      \u003ctd\u003ehttp://www.cnbeta.com/articles/tech/623601.htm\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e3\u003c/td\u003e\n",
              "      \u003ctd\u003e89614\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e新华社\u003c/td\u003e\n",
              "      \u003ctd\u003e这是6月18日在葡萄牙中部大佩德罗冈地区拍摄的被森林大火烧毁的汽车。新华社记者张立云摄\\r\\n\u003c/td\u003e\n",
              "      \u003ctd\u003e{\"type\":\"国际新闻\",\"site\":\"环球\",\"commentNum\":\"0\",\"j...\u003c/td\u003e\n",
              "      \u003ctd\u003e葡森林火灾造成至少62人死亡 政府宣布进入紧急状态（组图）\u003c/td\u003e\n",
              "      \u003ctd\u003ehttp://world.huanqiu.com/hot/2017-06/10866126....\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e4\u003c/td\u003e\n",
              "      \u003ctd\u003e89613\u003c/td\u003e\n",
              "      \u003ctd\u003e胡淑丽_MN7479\u003c/td\u003e\n",
              "      \u003ctd\u003e深圳大件事\u003c/td\u003e\n",
              "      \u003ctd\u003e（原标题：44岁女子跑深圳约会网友被拒，暴雨中裸身奔走……）\\r\\n@深圳交警微博称：昨日清...\u003c/td\u003e\n",
              "      \u003ctd\u003e{\"type\":\"新闻\",\"site\":\"网易热门\",\"commentNum\":\"978\",...\u003c/td\u003e\n",
              "      \u003ctd\u003e44岁女子约网友被拒暴雨中裸奔 交警为其披衣相随\u003c/td\u003e\n",
              "      \u003ctd\u003ehttp://news.163.com/17/0618/00/CN617P3Q0001875...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "      id      author                  source  \\\n",
              "0  89617         NaN  快科技@http://www.kkj.cn/   \n",
              "1  89616         NaN  快科技@http://www.kkj.cn/   \n",
              "2  89615         NaN  快科技@http://www.kkj.cn/   \n",
              "3  89614         NaN                     新华社   \n",
              "4  89613  胡淑丽_MN7479                   深圳大件事   \n",
              "\n",
              "                                             content  \\\n",
              "0  此外，自本周（6月12日）起，除小米手机6等15款机型外，其余机型已暂停更新发布（含开发版/...   \n",
              "1  骁龙835作为唯一通过Windows 10桌面平台认证的ARM处理器，高通强调，不会因为只考...   \n",
              "2  此前的一加3T搭载的是3400mAh电池，DashCharge快充规格为5V/4A。\\r\\n...   \n",
              "3    这是6月18日在葡萄牙中部大佩德罗冈地区拍摄的被森林大火烧毁的汽车。新华社记者张立云摄\\r\\n   \n",
              "4  （原标题：44岁女子跑深圳约会网友被拒，暴雨中裸身奔走……）\\r\\n@深圳交警微博称：昨日清...   \n",
              "\n",
              "                                             feature  \\\n",
              "0  {\"type\":\"科技\",\"site\":\"cnbeta\",\"commentNum\":\"37\"...   \n",
              "1  {\"type\":\"科技\",\"site\":\"cnbeta\",\"commentNum\":\"15\"...   \n",
              "2  {\"type\":\"科技\",\"site\":\"cnbeta\",\"commentNum\":\"18\"...   \n",
              "3  {\"type\":\"国际新闻\",\"site\":\"环球\",\"commentNum\":\"0\",\"j...   \n",
              "4  {\"type\":\"新闻\",\"site\":\"网易热门\",\"commentNum\":\"978\",...   \n",
              "\n",
              "                           title  \\\n",
              "0           小米MIUI 9首批机型曝光：共计15款   \n",
              "1     骁龙835在Windows 10上的性能表现有望改善   \n",
              "2      一加手机5细节曝光：3300mAh、充半小时用1天   \n",
              "3  葡森林火灾造成至少62人死亡 政府宣布进入紧急状态（组图）   \n",
              "4       44岁女子约网友被拒暴雨中裸奔 交警为其披衣相随   \n",
              "\n",
              "                                                 url  \n",
              "0     http://www.cnbeta.com/articles/tech/623597.htm  \n",
              "1     http://www.cnbeta.com/articles/tech/623599.htm  \n",
              "2     http://www.cnbeta.com/articles/tech/623601.htm  \n",
              "3  http://world.huanqiu.com/hot/2017-06/10866126....  \n",
              "4  http://news.163.com/17/0618/00/CN617P3Q0001875...  "
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "news_content \u003d pd.read_csv(news_file, encoding\u003d\u0027gb18030\u0027)\n",
        "news_content.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "然后进行：\n",
        "    1. 把content这一列全部拿出来\n",
        "    2. 对content进行切词\n",
        "    3. 统计单词出现的频率格式\n",
        "    4. 其他的分析\n",
        "`提示`：相关代码可以在Github Group的Slides里边的Lecture18，自动摘要里边看到。 https://github.com/Computing-Intelligence/jupyters_and_slides/blob/master/Lecuter-18-AutoSummarizaiton.pdf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "# 业内习惯性将加载数据用函数def load_data()来处理\n",
        "# 加载数据，并将处理的数据存到pureContent.csv中\n",
        "def load_data():\n",
        "    pure_file \u003d os.path.join(path_root, \u0027pureContent.csv\u0027)\n",
        "    if not os.path.exists(pure_file):\n",
        "        news_file \u003d os.path.join(path_root, \u0027sqlResult_1558435.csv\u0027)\n",
        "        news_content \u003d pd.read_csv(news_file, encoding\u003d\u0027gb18030\u0027)\n",
        "        #fillna()会将DataFrame中nan数据的数据填充为想要的数据，并返回填充后的结果。这里讲NANt填充为空\n",
        "        news_content[\u0027content\u0027] \u003d news_content[\u0027content\u0027].fillna(\u0027\u0027)\n",
        "        pure_content \u003d pd.DataFrame()\n",
        "        pure_content[\u0027content\u0027] \u003d news_content[\u0027content\u0027]\n",
        "        pure_content \u003d pure_content.fillna(\u0027\u0027)\n",
        "        pure_content[\u0027tokenized_content\u0027] \u003d pure_content[\u0027content\u0027].apply(cut)\n",
        "        pure_content.to_csv(pure_file, encoding\u003d\u0027gb18030\u0027)\n",
        "    else:\n",
        "        print(\u0027File found! \u0027)\n",
        "        pure_content \u003d pd.read_csv(pure_file, encoding\u003d\u0027gb18030\u0027)\n",
        "        pure_content \u003d pure_content.fillna(\u0027\u0027)\n",
        "    return pure_content"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Building prefix dict from the default dictionary ...\n",
            "Loading model from cache C:\\Users\\壹心理\\AppData\\Local\\Temp\\jieba.cache\n",
            "Loading model cost 0.748 seconds.\n",
            "Prefix dict has been built succesfully.\n"
          ]
        }
      ],
      "source": [
        "pure_content \u003d load_data()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "File found! \n"
          ]
        }
      ],
      "source": [
        "# 再次确认文件是否加载进来\n",
        "pure_content \u003d load_data()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border\u003d\"1\" class\u003d\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style\u003d\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eUnnamed: 0\u003c/th\u003e\n",
              "      \u003cth\u003econtent\u003c/th\u003e\n",
              "      \u003cth\u003etokenized_content\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e此外，自本周（6月12日）起，除小米手机6等15款机型外，其余机型已暂停更新发布（含开发版/...\u003c/td\u003e\n",
              "      \u003ctd\u003e此外 ， 自 本周 （ 6 月 12 日 ） 起 ， 除 小米 手机 6 等 15 款 机型...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e骁龙835作为唯一通过Windows 10桌面平台认证的ARM处理器，高通强调，不会因为只考...\u003c/td\u003e\n",
              "      \u003ctd\u003e骁龙 835 作为 唯一 通过 Windows   10 桌面 平台 认证 的 ARM 处理...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e2\u003c/td\u003e\n",
              "      \u003ctd\u003e2\u003c/td\u003e\n",
              "      \u003ctd\u003e此前的一加3T搭载的是3400mAh电池，DashCharge快充规格为5V/4A。\\r\\n...\u003c/td\u003e\n",
              "      \u003ctd\u003e此前 的 一加 3T 搭载 的 是 3400mAh 电池 ， DashCharge 快充 规...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e3\u003c/td\u003e\n",
              "      \u003ctd\u003e3\u003c/td\u003e\n",
              "      \u003ctd\u003e这是6月18日在葡萄牙中部大佩德罗冈地区拍摄的被森林大火烧毁的汽车。新华社记者张立云摄\\r\\n\u003c/td\u003e\n",
              "      \u003ctd\u003e这是 6 月 18 日 在 葡萄牙 中部 大 佩德罗 冈 地区 拍摄 的 被 森林 大火 烧...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e4\u003c/td\u003e\n",
              "      \u003ctd\u003e4\u003c/td\u003e\n",
              "      \u003ctd\u003e（原标题：44岁女子跑深圳约会网友被拒，暴雨中裸身奔走……）\\r\\n@深圳交警微博称：昨日清...\u003c/td\u003e\n",
              "      \u003ctd\u003e（ 原 标题 ： 44 岁 女子 跑 深圳 约会 网友 被 拒 ， 暴雨 中 裸身 奔走 …...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "   Unnamed: 0                                            content  \\\n",
              "0           0  此外，自本周（6月12日）起，除小米手机6等15款机型外，其余机型已暂停更新发布（含开发版/...   \n",
              "1           1  骁龙835作为唯一通过Windows 10桌面平台认证的ARM处理器，高通强调，不会因为只考...   \n",
              "2           2  此前的一加3T搭载的是3400mAh电池，DashCharge快充规格为5V/4A。\\r\\n...   \n",
              "3           3    这是6月18日在葡萄牙中部大佩德罗冈地区拍摄的被森林大火烧毁的汽车。新华社记者张立云摄\\r\\n   \n",
              "4           4  （原标题：44岁女子跑深圳约会网友被拒，暴雨中裸身奔走……）\\r\\n@深圳交警微博称：昨日清...   \n",
              "\n",
              "                                   tokenized_content  \n",
              "0  此外 ， 自 本周 （ 6 月 12 日 ） 起 ， 除 小米 手机 6 等 15 款 机型...  \n",
              "1  骁龙 835 作为 唯一 通过 Windows   10 桌面 平台 认证 的 ARM 处理...  \n",
              "2  此前 的 一加 3T 搭载 的 是 3400mAh 电池 ， DashCharge 快充 规...  \n",
              "3  这是 6 月 18 日 在 葡萄牙 中部 大 佩德罗 冈 地区 拍摄 的 被 森林 大火 烧...  \n",
              "4  （ 原 标题 ： 44 岁 女子 跑 深圳 约会 网友 被 拒 ， 暴雨 中 裸身 奔走 …...  "
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pure_content.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "# Train the word vector"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "# 加载模型函数，如果不存在则gensim自带的fastText进行训练，\n",
        "# 语料是all_corpus.txt，用pure_content中处理过的内容\n",
        "def load_model(pure_content):\n",
        "    model_file \u003d os.path.join(path_root, \u0027wordmodel_50.model\u0027)\n",
        "    savedContent \u003d os.path.join(path_root, \u0027all_corpus.txt\u0027)\n",
        "\n",
        "    if not os.path.exists(model_file):\n",
        "        if not os.path.exists(savedContent):\n",
        "            with open(savedContent , \u0027w\u0027, encoding\u003d\u0027utf-8\u0027) as f:\n",
        "                f.write(\u0027 \u0027.join(pure_content[\u0027tokenized_content\u0027].tolist()))\n",
        "        model \u003d FastText(LineSentence(savedContent), workers\u003d8,window\u003d5, size\u003d50, iter\u003d10, min_count\u003d1)\n",
        "        model.save(model_file)\n",
        "    else:\n",
        "        print(\u0027model Found!\u0027)\n",
        "        model \u003d FastText.load(model_file)\n",
        "    return model\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\python36\\lib\\site-packages\\smart_open\\smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
            "  \u0027See the migration notes for details: %s\u0027 % _MIGRATION_NOTES_URL\n",
            "c:\\python36\\lib\\site-packages\\smart_open\\smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
            "  \u0027See the migration notes for details: %s\u0027 % _MIGRATION_NOTES_URL\n"
          ]
        }
      ],
      "source": [
        "model \u003d load_model(pure_content)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "model Found!\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\python36\\lib\\site-packages\\smart_open\\smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
            "  \u0027See the migration notes for details: %s\u0027 % _MIGRATION_NOTES_URL\n"
          ]
        }
      ],
      "source": [
        "# 检查模型是否训练好\n",
        "model \u003d load_model(pure_content)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "from functools import reduce"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "tokeners \u003d [t for l in pure_content[\u0027tokenized_content\u0027].tolist() for t in l.split()]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\u0027此外\u0027, \u0027，\u0027, \u0027自\u0027, \u0027本周\u0027, \u0027（\u0027, \u00276\u0027, \u0027月\u0027, \u002712\u0027, \u0027日\u0027, \u0027）\u0027]"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokeners[:10]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "# Counts the words occurence"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "from collections import Counter\n",
        "tokener_counter \u003d Counter(tokeners)\n",
        "frequence \u003d {w: count/len(tokeners) for w, count in tokener_counter.items()}\n",
        "occurences_frequences \u003d sorted(list(frequence.values()), reverse\u003dTrue)\n",
        "X \u003d range(len(occurences_frequences))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\u003cmatplotlib.lines.Line2D at 0x1f33505abe0\u003e]"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 10778 characters]\u003d\u003d\n",
            "text/plain": [
              "\u003cFigure size 432x288 with 1 Axes\u003e"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.plot(X[:50], occurences_frequences[:50])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Counter({\u0027此外\u0027: 4239,\n",
              "         \u0027，\u0027: 1360799,\n",
              "         \u0027自\u0027: 4829,\n",
              "         \u0027本周\u0027: 486,\n",
              "         \u0027（\u0027: 212529,\n",
              "         \u00276\u0027: 32709,\n",
              "         \u0027月\u0027: 189525,\n",
              "         \u002712\u0027: 10468,\n",
              "         \u0027日\u0027: 169953,\n",
              "         \u0027）\u0027: 212782,\n",
              "         \u0027起\u0027: 5266,\n",
              "         \u0027除\u0027: 1362,\n",
              "         \u0027小米\u0027: 287,\n",
              "         \u0027手机\u0027: 2997,\n",
              "         \u0027等\u0027: 58386,\n",
              "         \u002715\u0027: 8582,\n",
              "         \u0027款\u0027: 702,\n",
              "         \u0027机型\u0027: 139,\n",
              "         \u0027外\u0027: 3291,\n",
              "         \u0027其余\u0027: 482,\n",
              "         \u0027已\u0027: 21046,\n",
              "         \u0027暂停\u0027: 834,\n",
              "         \u0027更新\u0027: 623,\n",
              "         \u0027发布\u0027: 7840,\n",
              "         \u0027含\u0027: 710,\n",
              "         \u0027开发\u0027: 3911,\n",
              "         \u0027版\u0027: 925,\n",
              "         \u0027/\u0027: 33916,\n",
              "         \u0027体验版\u0027: 1,\n",
              "         \u0027内测\u0027: 6,\n",
              "         \u0027稳定版\u0027: 1,\n",
              "         \u0027暂不受\u0027: 1,\n",
              "         \u0027影响\u0027: 8135,\n",
              "         \u0027以\u0027: 38171,\n",
              "         \u0027确保\u0027: 2854,\n",
              "         \u0027工程师\u0027: 523,\n",
              "         \u0027可以\u0027: 13287,\n",
              "         \u0027集中\u0027: 2910,\n",
              "         \u0027全部\u0027: 3790,\n",
              "         \u0027精力\u0027: 261,\n",
              "         \u0027进行\u0027: 25986,\n",
              "         \u0027系统优化\u0027: 1,\n",
              "         \u0027工作\u0027: 20722,\n",
              "         \u0027。\u0027: 636522,\n",
              "         \u0027有人\u0027: 1847,\n",
              "         \u0027猜测\u0027: 209,\n",
              "         \u0027这\u0027: 22754,\n",
              "         \u0027也\u0027: 47399,\n",
              "         \u0027是\u0027: 100283,\n",
              "         \u0027将\u0027: 61656,\n",
              "         \u0027主要\u0027: 9804,\n",
              "         \u0027用到\u0027: 41,\n",
              "         \u0027MIUI\u0027: 5,\n",
              "         \u00279\u0027: 11344,\n",
              "         \u0027的\u0027: 703820,\n",
              "         \u0027研发\u0027: 3216,\n",
              "         \u0027之中\u0027: 654,\n",
              "         \u00278\u0027: 12192,\n",
              "         \u0027去年\u0027: 6330,\n",
              "         \u00275\u0027: 60104,\n",
              "         \u0027距今已有\u0027: 22,\n",
              "         \u0027一年\u0027: 3203,\n",
              "         \u0027有余\u0027: 30,\n",
              "         \u0027时候\u0027: 3067,\n",
              "         \u0027更新换代\u0027: 34,\n",
              "         \u0027了\u0027: 121903,\n",
              "         \u0027当然\u0027: 880,\n",
              "         \u0027关于\u0027: 4597,\n",
              "         \u0027确切\u0027: 68,\n",
              "         \u0027信息\u0027: 7725,\n",
              "         \u0027我们\u0027: 19982,\n",
              "         \u0027还是\u0027: 5133,\n",
              "         \u0027等待\u0027: 882,\n",
              "         \u0027官方消息\u0027: 10,\n",
              "         \u0027骁龙\u0027: 55,\n",
              "         \u0027835\u0027: 21,\n",
              "         \u0027作为\u0027: 10385,\n",
              "         \u0027唯一\u0027: 1367,\n",
              "         \u0027通过\u0027: 18167,\n",
              "         \u0027Windows\u0027: 49,\n",
              "         \u002710\u0027: 14454,\n",
              "         \u0027桌面\u0027: 109,\n",
              "         \u0027平台\u0027: 8298,\n",
              "         \u0027认证\u0027: 664,\n",
              "         \u0027ARM\u0027: 4,\n",
              "         \u0027处理器\u0027: 105,\n",
              "         \u0027高通\u0027: 73,\n",
              "         \u0027强调\u0027: 3181,\n",
              "         \u0027不会\u0027: 3786,\n",
              "         \u0027因为\u0027: 5593,\n",
              "         \u0027只\u0027: 4573,\n",
              "         \u0027考虑\u0027: 2345,\n",
              "         \u0027性能\u0027: 437,\n",
              "         \u0027而\u0027: 17404,\n",
              "         \u0027去\u0027: 7291,\n",
              "         \u0027屏蔽掉\u0027: 2,\n",
              "         \u0027小\u0027: 6576,\n",
              "         \u0027核心\u0027: 3311,\n",
              "         \u0027相反\u0027: 230,\n",
              "         \u0027他们\u0027: 13878,\n",
              "         \u0027正\u0027: 4777,\n",
              "         \u0027联手\u0027: 413,\n",
              "         \u0027微软\u0027: 340,\n",
              "         \u0027找到\u0027: 1826,\n",
              "         \u0027一种\u0027: 4035,\n",
              "         \u0027适合\u0027: 812,\n",
              "         \u0027、\u0027: 296644,\n",
              "         \u0027兼顾\u0027: 203,\n",
              "         \u0027和\u0027: 134220,\n",
              "         \u0027功耗\u0027: 22,\n",
              "         \u0027完美\u0027: 458,\n",
              "         \u0027方案\u0027: 2844,\n",
              "         \u0027报道\u0027: 9687,\n",
              "         \u0027称\u0027: 7460,\n",
              "         \u0027已经\u0027: 12235,\n",
              "         \u0027拿到\u0027: 1059,\n",
              "         \u0027一些\u0027: 9377,\n",
              "         \u0027新\u0027: 23942,\n",
              "         \u0027源码\u0027: 2,\n",
              "         \u0027以便\u0027: 402,\n",
              "         \u0027更好\u0027: 2858,\n",
              "         \u0027地\u0027: 12766,\n",
              "         \u0027理解\u0027: 1283,\n",
              "         \u0027big\u0027: 2,\n",
              "         \u0027.\u0027: 6965,\n",
              "         \u0027little\u0027: 1,\n",
              "         \u0027架构\u0027: 217,\n",
              "         \u0027资料\u0027: 1904,\n",
              "         \u0027显示\u0027: 8471,\n",
              "         \u0027一款\u0027: 598,\n",
              "         \u0027集成\u0027: 198,\n",
              "         \u0027CPU\u0027: 57,\n",
              "         \u0027GPU\u0027: 14,\n",
              "         \u0027基带\u0027: 3,\n",
              "         \u0027蓝牙\u0027: 28,\n",
              "         \u0027Wi\u0027: 12,\n",
              "         \u0027-\u0027: 18797,\n",
              "         \u0027Fi\u0027: 12,\n",
              "         \u0027SoC\u0027: 2,\n",
              "         \u0027比\u0027: 23335,\n",
              "         \u0027传统\u0027: 7068,\n",
              "         \u0027Wintel\u0027: 3,\n",
              "         \u0027节省\u0027: 273,\n",
              "         \u0027至少\u0027: 2482,\n",
              "         \u002730%\u0027: 492,\n",
              "         \u0027PCB\u0027: 14,\n",
              "         \u0027空间\u0027: 3598,\n",
              "         \u0027按计划\u0027: 193,\n",
              "         \u0027今年\u0027: 13936,\n",
              "         \u0027Q4\u0027: 8,\n",
              "         \u0027华硕\u0027: 11,\n",
              "         \u0027惠普\u0027: 20,\n",
              "         \u0027联想\u0027: 222,\n",
              "         \u0027首发\u0027: 489,\n",
              "         \u0027Win10\u0027: 9,\n",
              "         \u0027电脑\u0027: 854,\n",
              "         \u0027预计\u0027: 4389,\n",
              "         \u0027均\u0027: 4423,\n",
              "         \u0027二合一\u0027: 1,\n",
              "         \u0027形态\u0027: 363,\n",
              "         \u0027产品\u0027: 8906,\n",
              "         \u0027只是\u0027: 2397,\n",
              "         \u0027个\u0027: 23043,\n",
              "         \u0027开始\u0027: 11177,\n",
              "         \u0027未来\u0027: 8199,\n",
              "         \u0027也许\u0027: 532,\n",
              "         \u0027还\u0027: 21786,\n",
              "         \u0027能\u0027: 12159,\n",
              "         \u0027见到\u0027: 507,\n",
              "         \u0027三星\u0027: 340,\n",
              "         \u0027Exynos\u0027: 4,\n",
              "         \u0027联发科\u0027: 17,\n",
              "         \u0027华为\u0027: 319,\n",
              "         \u0027麒麟\u0027: 84,\n",
              "         \u0027澎湃\u0027: 254,\n",
              "         \u0027进入\u0027: 9193,\n",
              "         \u0027此前\u0027: 3154,\n",
              "         \u0027一加\u0027: 51,\n",
              "         \u00273T\u0027: 1,\n",
              "         \u0027搭载\u0027: 504,\n",
              "         \u00273400mAh\u0027: 1,\n",
              "         \u0027电池\u0027: 444,\n",
              "         \u0027DashCharge\u0027: 1,\n",
              "         \u0027快充\u0027: 7,\n",
              "         \u0027规格\u0027: 256,\n",
              "         \u0027为\u0027: 58104,\n",
              "         \u00275V\u0027: 2,\n",
              "         \u00274A\u0027: 47,\n",
              "         \u0027至于\u0027: 403,\n",
              "         \u0027缩水\u0027: 91,\n",
              "         \u0027可能\u0027: 9902,\n",
              "         \u0027与\u0027: 58317,\n",
              "         \u0027刘作\u0027: 3,\n",
              "         \u0027虎\u0027: 236,\n",
              "         \u0027所说\u0027: 293,\n",
              "         \u0027要\u0027: 21407,\n",
              "         \u0027做\u0027: 8635,\n",
              "         \u0027市面\u0027: 6,\n",
              "         \u0027最\u0027: 8964,\n",
              "         \u0027轻薄\u0027: 18,\n",
              "         \u0027大屏\u0027: 35,\n",
              "         \u0027旗舰\u0027: 90,\n",
              "         \u0027设定\u0027: 380,\n",
              "         \u0027有关\u0027: 6292,\n",
              "         \u0027按照\u0027: 4632,\n",
              "         \u0027目前\u0027: 16852,\n",
              "         \u0027掌握\u0027: 1293,\n",
              "         \u0027拥有\u0027: 3178,\n",
              "         \u00275.5\u0027: 117,\n",
              "         \u0027寸\u0027: 23,\n",
              "         \u00271080P\u0027: 2,\n",
              "         \u0027AMOLED\u0027: 41,\n",
              "         \u0027显示屏\u0027: 110,\n",
              "         \u00276G\u0027: 1,\n",
              "         \u00278GB\u0027: 9,\n",
              "         \u0027RAM\u0027: 15,\n",
              "         \u002764GB\u0027: 10,\n",
              "         \u0027128GB\u0027: 9,\n",
              "         \u0027ROM\u0027: 14,\n",
              "         \u0027双\u0027: 904,\n",
              "         \u00271600\u0027: 130,\n",
              "         \u0027万\u0027: 3390,\n",
              "         \u0027摄像头\u0027: 393,\n",
              "         \u0027备货\u0027: 15,\n",
              "         \u0027量\u0027: 1298,\n",
              "         \u0027“\u0027: 239982,\n",
              "         \u0027惊喜\u0027: 278,\n",
              "         \u0027”\u0027: 239981,\n",
              "         \u0027根据\u0027: 6920,\n",
              "         \u0027京东\u0027: 683,\n",
              "         \u0027泄露\u0027: 390,\n",
              "         \u0027售价\u0027: 288,\n",
              "         \u0027xx99\u0027: 1,\n",
              "         \u0027元\u0027: 6836,\n",
              "         \u0027应该\u0027: 2792,\n",
              "         \u0027在\u0027: 264634,\n",
              "         \u00272799\u0027: 1,\n",
              "         \u00272899\u0027: 3,\n",
              "         \u00272999\u0027: 3,\n",
              "         \u0027中\u0027: 77942,\n",
              "         \u0027某个\u0027: 188,\n",
              "         \u0027这是\u0027: 11356,\n",
              "         \u002718\u0027: 8040,\n",
              "         \u0027葡萄牙\u0027: 1224,\n",
              "         \u0027中部\u0027: 1063,\n",
              "         \u0027大\u0027: 17646,\n",
              "         \u0027佩德罗\u0027: 87,\n",
              "         \u0027冈\u0027: 6,\n",
              "         \u0027地区\u0027: 10979,\n",
              "         \u0027拍摄\u0027: 5481,\n",
              "         \u0027被\u0027: 25288,\n",
              "         \u0027森林\u0027: 1288,\n",
              "         \u0027大火\u0027: 147,\n",
              "         \u0027烧毁\u0027: 39,\n",
              "         \u0027汽车\u0027: 5223,\n",
              "         \u0027新华社\u0027: 142462,\n",
              "         \u0027记者\u0027: 62274,\n",
              "         \u0027张立\u0027: 120,\n",
              "         \u0027云摄\u0027: 114,\n",
              "         \u0027原\u0027: 2855,\n",
              "         \u0027标题\u0027: 1408,\n",
              "         \u0027：\u0027: 102478,\n",
              "         \u002744\u0027: 419,\n",
              "         \u0027岁\u0027: 8731,\n",
              "         \u0027女子\u0027: 3715,\n",
              "         \u0027跑\u0027: 2773,\n",
              "         \u0027深圳\u0027: 2741,\n",
              "         \u0027约会\u0027: 99,\n",
              "         \u0027网友\u0027: 1821,\n",
              "         \u0027拒\u0027: 125,\n",
              "         \u0027暴雨\u0027: 687,\n",
              "         \u0027裸身\u0027: 3,\n",
              "         \u0027奔走\u0027: 66,\n",
              "         \u0027…\u0027: 12053,\n",
              "         \u0027@\u0027: 405,\n",
              "         \u0027交警\u0027: 347,\n",
              "         \u0027微博称\u0027: 12,\n",
              "         \u0027昨日\u0027: 796,\n",
              "         \u0027清晨\u0027: 590,\n",
              "         \u0027发现\u0027: 8559,\n",
              "         \u0027有\u0027: 46555,\n",
              "         \u0027一\u0027: 11369,\n",
              "         \u0027赤裸\u0027: 15,\n",
              "         \u0027上身\u0027: 53,\n",
              "         \u0027行走\u0027: 549,\n",
              "         \u0027南坪\u0027: 30,\n",
              "         \u0027快速\u0027: 2526,\n",
              "         \u0027上\u0027: 48837,\n",
              "         \u0027期间\u0027: 5085,\n",
              "         \u0027轻生\u0027: 59,\n",
              "         \u0027年头\u0027: 136,\n",
              "         \u0027一辅警\u0027: 2,\n",
              "         \u0027后\u0027: 28680,\n",
              "         \u0027赶紧\u0027: 198,\n",
              "         \u0027其\u0027: 11646,\n",
              "         \u0027披上\u0027: 64,\n",
              "         \u0027黄衣\u0027: 7,\n",
              "         \u0027并\u0027: 26574,\n",
              "         \u0027一路\u0027: 19209,\n",
              "         \u0027劝说\u0027: 105,\n",
              "         \u0027她\u0027: 12951,\n",
              "         \u0027那么\u0027: 2006,\n",
              "         \u0027事发\u0027: 635,\n",
              "         \u0027时\u0027: 19000,\n",
              "         \u0027到底\u0027: 642,\n",
              "         \u0027都\u0027: 28041,\n",
              "         \u0027发生\u0027: 8362,\n",
              "         \u0027些\u0027: 177,\n",
              "         \u0027什么\u0027: 3491,\n",
              "         \u0027呢\u0027: 1402,\n",
              "         \u0027？\u0027: 13642,\n",
              "         \u0027南\u0027: 1079,\n",
              "         \u0027带\u0027: 3057,\n",
              "         \u0027您\u0027: 694,\n",
              "         \u0027一起\u0027: 4931,\n",
              "         \u0027还原\u0027: 178,\n",
              "         \u0027现场\u0027: 5881,\n",
              "         \u0027龙岗\u0027: 29,\n",
              "         \u0027大队\u0027: 906,\n",
              "         \u0027坂田\u0027: 6,\n",
              "         \u0027中队\u0027: 210,\n",
              "         \u0027辅警\u0027: 55,\n",
              "         \u0027刘青\u0027: 31,\n",
              "         \u0027女生\u0027: 286,\n",
              "         \u0027一位\u0027: 3058,\n",
              "         \u0027外表\u0027: 91,\n",
              "         \u0027高大\u0027: 178,\n",
              "         \u0027帅气\u0027: 39,\n",
              "         \u0027说话\u0027: 342,\n",
              "         \u0027略带\u0027: 25,\n",
              "         \u0027腼腆\u0027: 55,\n",
              "         \u002790\u0027: 1030,\n",
              "         \u0027青年\u0027: 3857,\n",
              "         \u0027介绍\u0027: 7292,\n",
              "         \u002716\u0027: 7776,\n",
              "         \u0027早上\u0027: 457,\n",
              "         \u00277\u0027: 14055,\n",
              "         \u002736\u0027: 933,\n",
              "         \u0027分\u0027: 7690,\n",
              "         \u0027他\u0027: 37118,\n",
              "         \u0027正在\u0027: 7686,\n",
              "         \u0027环城\u0027: 42,\n",
              "         \u0027南路\u0027: 90,\n",
              "         \u0027附近\u0027: 3620,\n",
              "         \u0027值勤\u0027: 15,\n",
              "         \u0027接到\u0027: 923,\n",
              "         \u0027机动车\u0027: 427,\n",
              "         \u0027危险\u0027: 947,\n",
              "         \u0027警情\u0027: 34,\n",
              "         \u0027随后\u0027: 2399,\n",
              "         \u0027骑着\u0027: 105,\n",
              "         \u0027铁骑\u0027: 40,\n",
              "         \u0027沿路\u0027: 20,\n",
              "         \u0027寻找\u0027: 1402,\n",
              "         \u0027大概\u0027: 342,\n",
              "         \u0027花\u0027: 1232,\n",
              "         \u0027十多分钟\u0027: 14,\n",
              "         \u0027大道\u0027: 729,\n",
              "         \u0027出口\u0027: 2200,\n",
              "         \u0027往\u0027: 508,\n",
              "         \u0027方向\u0027: 2429,\n",
              "         \u0027逆行\u0027: 23,\n",
              "         \u0027辅道\u0027: 5,\n",
              "         \u0027该\u0027: 9781,\n",
              "         \u0027身上\u0027: 963,\n",
              "         \u0027一丝不挂\u0027: 4,\n",
              "         \u0027地逆\u0027: 2,\n",
              "         \u0027车流\u0027: 64,\n",
              "         \u0027而行\u0027: 125,\n",
              "         \u0027走时\u0027: 10,\n",
              "         \u0027停\u0027: 457,\n",
              "         \u0027坐\u0027: 533,\n",
              "         \u0027躺\u0027: 292,\n",
              "         \u0027险象环生\u0027: 8,\n",
              "         \u0027刘青停\u0027: 2,\n",
              "         \u0027好\u0027: 11871,\n",
              "         \u0027另外\u0027: 2021,\n",
              "         \u0027一名\u0027: 8107,\n",
              "         \u0027巡防\u0027: 21,\n",
              "         \u0027员\u0027: 899,\n",
              "         \u0027追\u0027: 228,\n",
              "         \u0027上去\u0027: 131,\n",
              "         \u0027情绪\u0027: 913,\n",
              "         \u0027很\u0027: 9207,\n",
              "         \u0027低落\u0027: 24,\n",
              "         \u0027话\u0027: 791,\n",
              "         \u0027不\u0027: 28419,\n",
              "         \u0027多\u0027: 18153,\n",
              "         \u0027尝试\u0027: 988,\n",
              "         \u0027交流\u0027: 5354,\n",
              "         \u0027离开\u0027: 1486,\n",
              "         \u0027可\u0027: 11077,\n",
              "         \u0027愿意\u0027: 1350,\n",
              "         \u0027接受\u0027: 5826,\n",
              "         \u0027继续\u0027: 6952,\n",
              "         \u0027缓慢\u0027: 315,\n",
              "         \u0027地往\u0027: 8,\n",
              "         \u0027快速路\u0027: 60,\n",
              "         \u0027主干道\u0027: 56,\n",
              "         \u0027走\u0027: 4719,\n",
              "         \u0027此时\u0027: 448,\n",
              "         \u0027路边\u0027: 252,\n",
              "         \u0027聚集\u0027: 726,\n",
              "         \u0027市民\u0027: 3429,\n",
              "         \u0027围观\u0027: 132,\n",
              "         \u0027为了\u0027: 4643,\n",
              "         \u0027刺激\u0027: 524,\n",
              "         \u0027一边\u0027: 976,\n",
              "         \u0027盯\u0027: 131,\n",
              "         \u0027着\u0027: 13980,\n",
              "         \u0027驱赶\u0027: 42,\n",
              "         \u0027群众\u0027: 5297,\n",
              "         \u0027从\u0027: 29697,\n",
              "         \u0027警方\u0027: 4265,\n",
              "         \u0027提供\u0027: 12182,\n",
              "         \u0027一份\u0027: 1888,\n",
              "         \u0027视频\u0027: 2929,\n",
              "         \u0027了解\u0027: 7623,\n",
              "         \u0027到\u0027: 30452,\n",
              "         \u002725\u0027: 6884,\n",
              "         \u0027出现\u0027: 7608,\n",
              "         \u0027坂雪岗\u0027: 4,\n",
              "         \u0027监控\u0027: 945,\n",
              "         \u0027穿着\u0027: 330,\n",
              "         \u0027白色\u0027: 474,\n",
              "         \u0027内裤\u0027: 24,\n",
              "         \u0027沿着\u0027: 382,\n",
              "         \u0027当时\u0027: 3175,\n",
              "         \u0027正值\u0027: 417,\n",
              "         \u0027上班\u0027: 367,\n",
              "         \u0027高峰期\u0027: 174,\n",
              "         \u0027十字路口\u0027: 80,\n",
              "         \u0027排起\u0027: 33,\n",
              "         \u0027长队\u0027: 74,\n",
              "         \u0027当\u0027: 3397,\n",
              "         \u0027路上\u0027: 902,\n",
              "         \u0027纷纷\u0027: 1366,\n",
              "         \u0027驻足\u0027: 146,\n",
              "         \u0027观望\u0027: 271,\n",
              "         \u0027不少\u0027: 4548,\n",
              "         \u0027车辆\u0027: 2199,\n",
              "         \u0027放慢\u0027: 41,\n",
              "         \u0027速度\u0027: 1784,\n",
              "         \u0027但\u0027: 23362,\n",
              "         \u0027停下\u0027: 76,\n",
              "         \u0027脚步\u0027: 325,\n",
              "         \u0027依然\u0027: 2130,\n",
              "         \u0027行进\u0027: 261,\n",
              "         \u0027中间\u0027: 441,\n",
              "         \u0027一辆\u0027: 1276,\n",
              "         \u0027货车\u0027: 522,\n",
              "         \u0027挡住\u0027: 72,\n",
              "         \u0027镜头\u0027: 367,\n",
              "         \u0027但是\u0027: 3871,\n",
              "         \u0027再次出现\u0027: 54,\n",
              "         \u0027没\u0027: 2591,\n",
              "         \u0027穿\u0027: 742,\n",
              "         \u0027全身\u0027: 134,\n",
              "         \u0027裸露\u0027: 113,\n",
              "         \u0027朝着\u0027: 233,\n",
              "         \u0027周围\u0027: 641,\n",
              "         \u0027没有\u0027: 13997,\n",
              "         \u0027人\u0027: 30582,\n",
              "         \u0027尾随\u0027: 34,\n",
              "         \u0027或者\u0027: 3079,\n",
              "         \u0027上前\u0027: 134,\n",
              "         \u0027劝止\u0027: 2,\n",
              "         \u0027大清早\u0027: 8,\n",
              "         \u0027看到\u0027: 4399,\n",
              "         \u0027这样\u0027: 5822,\n",
              "         \u0027情况\u0027: 9444,\n",
              "         \u0027恐怕\u0027: 270,\n",
              "         \u0027大家\u0027: 3297,\n",
              "         \u0027办法\u0027: 1838,\n",
              "         \u0027淡定\u0027: 55,\n",
              "         \u0027面对\u0027: 2432,\n",
              "         \u0027这一\u0027: 5346,\n",
              "         \u0027表示\u0027: 22175,\n",
              "         \u0027根本\u0027: 977,\n",
              "         \u0027不敢\u0027: 478,\n",
              "         \u0027看\u0027: 5723,\n",
              "         \u0027心里\u0027: 643,\n",
              "         \u0027挺\u0027: 446,\n",
              "         \u0027别扭\u0027: 16,\n",
              "         \u0027感觉\u0027: 1558,\n",
              "         \u0027尴尬\u0027: 286,\n",
              "         \u0027跟随\u0027: 440,\n",
              "         \u0027作出\u0027: 2602,\n",
              "         \u0027让\u0027: 18757,\n",
              "         \u0027意想不到\u0027: 53,\n",
              "         \u0027举动\u0027: 265,\n",
              "         \u0027突然\u0027: 1034,\n",
              "         \u0027靠近\u0027: 342,\n",
              "         \u0027护栏\u0027: 80,\n",
              "         \u0027上面\u0027: 362,\n",
              "         \u0027跳下去\u0027: 10,\n",
              "         \u0027赶忙\u0027: 19,\n",
              "         \u0027冲上去\u0027: 21,\n",
              "         \u0027拉住\u0027: 4,\n",
              "         \u0027手\u0027: 920,\n",
              "         \u0027控制\u0027: 2707,\n",
              "         \u0027住\u0027: 1617,\n",
              "         \u0027远离\u0027: 309,\n",
              "         \u0027碍于\u0027: 21,\n",
              "         \u0027穿衣服\u0027: 18,\n",
              "         \u0027刘青递\u0027: 2,\n",
              "         \u0027衣服\u0027: 397,\n",
              "         \u0027把\u0027: 11853,\n",
              "         \u0027扔\u0027: 189,\n",
              "         \u0027排水沟\u0027: 22,\n",
              "         \u0027里\u0027: 10077,\n",
              "         \u0027往前走\u0027: 41,\n",
              "         \u0027只能\u0027: 1621,\n",
              "         \u0027紧紧\u0027: 183,\n",
              "         \u0027拉\u0027: 1601,\n",
              "         \u0027一只\u0027: 1350,\n",
              "         \u0027跟\u0027: 2058,\n",
              "         \u0027后面\u0027: 329,\n",
              "         \u0027一路上\u0027: 96,\n",
              "         \u0027耐心\u0027: 404,\n",
              "         \u0027开导\u0027: 21,\n",
              "         \u0027安慰\u0027: 129,\n",
              "         \u0027听到\u0027: 603,\n",
              "         \u0027不断\u0027: 6213,\n",
              "         \u0027重复\u0027: 372,\n",
              "         \u0027一句\u0027: 558,\n",
              "         \u0027要是\u0027: 179,\n",
              "         \u0027你\u0027: 6959,\n",
              "         \u0027遭遇\u0027: 1703,\n",
              "         \u0027我\u0027: 21713,\n",
              "         \u0027事\u0027: 2098,\n",
              "         \u0027会\u0027: 17310,\n",
              "         \u0027不时\u0027: 309,\n",
              "         \u0027试图\u0027: 824,\n",
              "         \u0027挣脱\u0027: 25,\n",
              "         \u0027手要\u0027: 8,\n",
              "         \u0027冲向\u0027: 99,\n",
              "         \u0027往下跳\u0027: 4,\n",
              "         \u0027就\u0027: 26690,\n",
              "         \u0027牵\u0027: 74,\n",
              "         \u0027天\u0027: 3329,\n",
              "         \u0027下起\u0027: 4,\n",
              "         \u0027大暴雨\u0027: 172,\n",
              "         \u0027雨大\u0027: 2,\n",
              "         \u0027连\u0027: 1380,\n",
              "         \u0027眼睛\u0027: 460,\n",
              "         \u0027睁不开\u0027: 5,\n",
              "         \u0027说\u0027: 40082,\n",
              "         \u0027瞬间\u0027: 568,\n",
              "         \u0027雨\u0027: 365,\n",
              "         \u0027透\u0027: 58,\n",
              "         \u0027帮助\u0027: 5162,\n",
              "         \u0027冒\u0027: 175,\n",
              "         \u0027大雨\u0027: 255,\n",
              "         \u0027四十分钟\u0027: 4,\n",
              "         \u0027吧\u0027: 1250,\n",
              "         \u0027来说\u0027: 2870,\n",
              "         \u0027想\u0027: 3717,\n",
              "         \u0027回家\u0027: 768,\n",
              "         \u0027然后\u0027: 1218,\n",
              "         \u0027递过来\u0027: 11,\n",
              "         \u0027微博上\u0027: 106,\n",
              "         \u0027照片\u0027: 52271,\n",
              "         \u0027披着\u0027: 2,\n",
              "         \u0027小心翼翼\u0027: 87,\n",
              "         \u0027旁边\u0027: 305,\n",
              "         \u0027场景\u0027: 754,\n",
              "         \u0027南平\u0027: 67,\n",
              "         \u0027下来\u0027: 843,\n",
              "         \u0027带到\u0027: 258,\n",
              "         \u0027派出所\u0027: 670,\n",
              "         \u0027那\u0027: 2590,\n",
              "         \u0027姑娘\u0027: 462,\n",
              "         \u0027什么样\u0027: 410,\n",
              "         \u0027事情\u0027: 1461,\n",
              "         \u0027才\u0027: 3918,\n",
              "         \u0027据\u0027: 10742,\n",
              "         \u0027透露\u0027: 1575,\n",
              "         \u0027姓\u0027: 65,\n",
              "         \u0027陈\u0027: 1073,\n",
              "         \u0027系\u0027: 660,\n",
              "         \u0027湖北\u0027: 2478,\n",
              "         \u0027家属\u0027: 735,\n",
              "         \u0027反映\u0027: 1547,\n",
              "         \u0027其有\u0027: 10,\n",
              "         \u0027精神病\u0027: 53,\n",
              "         \u0027史\u0027: 436,\n",
              "         \u0027三天\u0027: 431,\n",
              "         \u0027前\u0027: 13660,\n",
              "         \u0027陈某\u0027: 291,\n",
              "         \u0027老家\u0027: 228,\n",
              "         \u0027来\u0027: 10977,\n",
              "         \u0027受挫\u0027: 60,\n",
              "         \u0027导致\u0027: 3834,\n",
              "         \u0027异常\u0027: 422,\n",
              "         \u0027遂\u0027: 155,\n",
              "         \u0027产生\u0027: 3063,\n",
              "         \u0027念头\u0027: 97,\n",
              "         \u0027送往\u0027: 416,\n",
              "         \u0027某\u0027: 1584,\n",
              "         \u0027医院\u0027: 4146,\n",
              "         \u0027治疗\u0027: 2117,\n",
              "         \u0027大大\u0027: 395,\n",
              "         \u0027君\u0027: 159,\n",
              "         \u0027希望\u0027: 7863,\n",
              "         \u0027早点\u0027: 41,\n",
              "         \u0027康复\u0027: 586,\n",
              "         \u0027其实\u0027: 1756,\n",
              "         \u0027真爱\u0027: 14,\n",
              "         \u0027到来\u0027: 1088,\n",
              "         \u0027存在\u0027: 5138,\n",
              "         \u0027年龄\u0027: 910,\n",
              "         \u0027限制\u0027: 1628,\n",
              "         \u0027你们\u0027: 733,\n",
              "         \u0027因\u0027: 4901,\n",
              "         \u0027善良\u0027: 84,\n",
              "         \u0027原因\u0027: 3562,\n",
              "         \u0027一众\u0027: 39,\n",
              "         \u0027暖\u0027: 226,\n",
              "         \u0027男点\u0027: 2,\n",
              "         \u0027ZAN\u0027: 2,\n",
              "         \u0027弓常\u0027: 2,\n",
              "         \u0027yan\u0027: 3,\n",
              "         \u0027桦\u0027: 7,\n",
              "         \u0027问\u0027: 1095,\n",
              "         \u0027这个\u0027: 8688,\n",
              "         \u0027哥哥\u0027: 300,\n",
              "         \u0027女票\u0027: 2,\n",
              "         \u0027吗\u0027: 1784,\n",
              "         \u0027原谅\u0027: 24,\n",
              "         \u0027一辈子\u0027: 241,\n",
              "         \u0027浪荡\u0027: 2,\n",
              "         \u0027不羁\u0027: 16,\n",
              "         \u0027爱\u0027: 1501,\n",
              "         \u0027萨摩耶\u0027: 2,\n",
              "         \u0027过不去\u0027: 16,\n",
              "         \u0027嘛\u0027: 94,\n",
              "         \u0027想想\u0027: 81,\n",
              "         \u0027自己\u0027: 14006,\n",
              "         \u0027家人\u0027: 1291,\n",
              "         \u0027同时\u0027: 10798,\n",
              "         \u0027感谢\u0027: 671,\n",
              "         \u0027蜀黍\u0027: 12,\n",
              "         \u0027火心\u0027: 2,\n",
              "         \u0027聆听\u0027: 198,\n",
              "         \u0027心灵\u0027: 226,\n",
              "         \u0027点赞\u0027: 354,\n",
              "         \u0027中華云盾\u0027: 2,\n",
              "         \u0027警察\u0027: 1769,\n",
              "         \u0027就是\u0027: 8003,\n",
              "         \u0027需时\u0027: 3,\n",
              "         \u0027申出\u0027: 2,\n",
              "         \u0027援手\u0027: 49,\n",
              "         \u0027Tomchlee\u0027: 2,\n",
              "         \u0027帅\u0027: 27,\n",
              "         \u0027！\u0027: 6638,\n",
              "         \u0027SJ\u0027: 2,\n",
              "         \u0027李赫海\u0027: 2,\n",
              "         \u0027i\u0027: 48,\n",
              "         \u0027木有\u0027: 4,\n",
              "         \u0027男子\u0027: 3965,\n",
              "         \u0027迷奸\u0027: 1,\n",
              "         \u0027拍\u0027: 742,\n",
              "         \u0027418\u0027: 15,\n",
              "         \u0027女方\u0027: 36,\n",
              "         \u002720\u0027: 11010,\n",
              "         \u002750\u0027: 2594,\n",
              "         \u0027号\u0027: 6633,\n",
              "         \u0027上午\u0027: 2172,\n",
              "         \u0027淮安市\u0027: 71,\n",
              "         \u0027涟水县\u0027: 2,\n",
              "         \u0027公安局\u0027: 835,\n",
              "         \u0027刑警大队\u0027: 17,\n",
              "         \u0027一个\u0027: 21937,\n",
              "         \u0027奇怪\u0027: 106,\n",
              "         \u0027报警\u0027: 489,\n",
              "         \u0027电话\u0027: 1365,\n",
              "         \u0027言语\u0027: 85,\n",
              "         \u0027不清\u0027: 61,\n",
              "         \u0027声称\u0027: 322,\n",
              "         \u0027遭到\u0027: 832,\n",
              "         \u0027侵害\u0027: 323,\n",
              "         \u0027几个\u0027: 1330,\n",
              "         \u0027关键词\u0027: 1433,\n",
              "         \u0027令\u0027: 1318,\n",
              "         \u0027民警\u0027: 1427,\n",
              "         \u0027紧张\u0027: 1123,\n",
              "         \u0027起来\u0027: 2656,\n",
              "         \u0027受到\u0027: 3667,\n",
              "         \u0027A股\u0027: 1973,\n",
              "         \u0027纳入\u0027: 2269,\n",
              "         \u0027MSCI\u0027: 1292,\n",
              "         \u0027指数\u0027: 3957,\n",
              "         \u0027利好\u0027: 460,\n",
              "         \u0027消息\u0027: 3605,\n",
              "         \u0027市场\u0027: 19029,\n",
              "         \u0027周三\u0027: 197,\n",
              "         \u0027再度\u0027: 749,\n",
              "         \u0027上演\u0027: 1330,\n",
              "         \u0027龙马\u0027: 23,\n",
              "         \u0027行情\u0027: 906,\n",
              "         \u0027周四\u0027: 145,\n",
              "         \u0027金融股\u0027: 141,\n",
              "         \u0027白马股\u0027: 104,\n",
              "         \u0027表现\u0027: 2997,\n",
              "         \u0027喜人\u0027: 50,\n",
              "         \u0027尾盘\u0027: 240,\n",
              "         \u0027跳水\u0027: 638,\n",
              "         \u0027之后\u0027: 4770,\n",
              "         \u0027仅\u0027: 3079,\n",
              "         \u0027金融\u0027: 5616,\n",
              "         \u0027板块\u0027: 1378,\n",
              "         \u0027仍\u0027: 5630,\n",
              "         \u0027维系\u0027: 46,\n",
              "         \u0027红盘\u0027: 23,\n",
              "         \u0027状态\u0027: 1956,\n",
              "         \u0027分析\u0027: 3961,\n",
              "         \u0027人士\u0027: 3337,\n",
              "         \u0027认为\u0027: 11639,\n",
              "         \u0027受益\u0027: 746,\n",
              "         \u0027于\u0027: 17298,\n",
              "         \u0027低\u0027: 2236,\n",
              "         \u0027估值\u0027: 529,\n",
              "         \u0027重获\u0027: 92,\n",
              "         \u0027资金\u0027: 6566,\n",
              "         \u0027青睐\u0027: 428,\n",
              "         \u0027存量\u0027: 366,\n",
              "         \u0027博弈\u0027: 248,\n",
              "         \u0027格局\u0027: 1170,\n",
              "         \u0027下\u0027: 18128,\n",
              "         \u0027风格\u0027: 817,\n",
              "         \u0027交替\u0027: 116,\n",
              "         \u0027震荡\u0027: 698,\n",
              "         \u0027料\u0027: 122,\n",
              "         \u0027延续\u0027: 1212,\n",
              "         \u0027流动性\u0027: 989,\n",
              "         \u0027改善\u0027: 2716,\n",
              "         \u0027经济\u0027: 22208,\n",
              "         \u0027悲观\u0027: 86,\n",
              "         \u0027预期\u0027: 2482,\n",
              "         \u0027修正\u0027: 134,\n",
              "         \u0027有助于\u0027: 1113,\n",
              "         \u0027支撑\u0027: 1950,\n",
              "         \u0027继而\u0027: 89,\n",
              "         \u0027大盘\u0027: 374,\n",
              "         \u0027逐步\u0027: 1834,\n",
              "         \u0027向\u0027: 15203,\n",
              "         \u0027九\u0027: 240,\n",
              "         \u0027再现\u0027: 372,\n",
              "         \u0027未能\u0027: 1010,\n",
              "         \u0027上行\u0027: 417,\n",
              "         \u0027态势\u0027: 1185,\n",
              "         \u0027两市\u0027: 518,\n",
              "         \u0027成交\u0027: 1469,\n",
              "         \u0027小幅\u0027: 488,\n",
              "         \u0027放量\u0027: 143,\n",
              "         \u002729\u0027: 4886,\n",
              "         \u0027中信\u0027: 140,\n",
              "         \u0027一级\u0027: 557,\n",
              "         \u0027行业\u0027: 6515,\n",
              "         \u0027收盘\u0027: 661,\n",
              "         \u0027银行\u0027: 5991,\n",
              "         \u0027非银行\u0027: 46,\n",
              "         \u0027两个\u0027: 4158,\n",
              "         \u0027收红\u0027: 39,\n",
              "         \u0027分别\u0027: 5488,\n",
              "         \u0027上涨\u0027: 3992,\n",
              "         \u00271.80%\u0027: 6,\n",
              "         \u00270.20%\u0027: 30,\n",
              "         \u0027二级\u0027: 432,\n",
              "         \u0027来看\u0027: 2608,\n",
              "         \u0027股份制\u0027: 119,\n",
              "         \u0027城商行\u0027: 53,\n",
              "         \u0027涨幅\u0027: 2242,\n",
              "         \u0027最高\u0027: 3452,\n",
              "         \u0027达到\u0027: 5390,\n",
              "         \u00272.22%\u0027: 2,\n",
              "         \u0027国有银行\u0027: 37,\n",
              "         \u00270.82%\u0027: 6,\n",
              "         \u0027信托\u0027: 384,\n",
              "         \u0027及其\u0027: 1807,\n",
              "         \u00270.64%\u0027: 10,\n",
              "         \u0027保险\u0027: 1528,\n",
              "         \u00270.34%\u0027: 16,\n",
              "         \u0027证券\u0027: 1486,\n",
              "         \u00270.06%\u0027: 19,\n",
              "         \u0027成分股\u0027: 180,\n",
              "         \u0027共有\u0027: 2059,\n",
              "         \u002721\u0027: 7406,\n",
              "         \u0027其中\u0027: 10635,\n",
              "         \u0027招商银行\u0027: 94,\n",
              "         \u0027最大\u0027: 6078,\n",
              "         \u00276.66%\u0027: 9,\n",
              "         \u0027贵阳\u0027: 861,\n",
              "         \u00273.65%\u0027: 8,\n",
              "         \u0027上海银行\u0027: 29,\n",
              "         \u0027华夏银行\u0027: 46,\n",
              "         \u0027浦发银行\u0027: 41,\n",
              "         \u0027兴业银行\u0027: 67,\n",
              "         \u0027超过\u0027: 7201,\n",
              "         \u00271.50%\u0027: 7,\n",
              "         \u0027共\u0027: 4581,\n",
              "         \u002717\u0027: 7486,\n",
              "         \u0027个股\u0027: 827,\n",
              "         \u0027安信\u0027: 49,\n",
              "         \u0027中国\u0027: 71181,\n",
              "         \u0027太保\u0027: 30,\n",
              "         \u0027居前\u0027: 316,\n",
              "         \u0027两名\u0027: 1573,\n",
              "         \u00274.57%\u0027: 3,\n",
              "         \u00273.04%\u0027: 4,\n",
              "         \u0027西水股份\u0027: 4,\n",
              "         \u0027华安\u0027: 41,\n",
              "         \u0027人寿\u0027: 118,\n",
              "         \u0027新华\u0027: 6665,\n",
              "         \u00272%\u0027: 268,\n",
              "         \u0027相对而言\u0027: 54,\n",
              "         \u0027券商\u0027: 587,\n",
              "         \u0027股多\u0027: 1,\n",
              "         \u0027下跌\u0027: 2849,\n",
              "         \u0027近期\u0027: 2594,\n",
              "         \u0027对\u0027: 50582,\n",
              "         \u0027消息面\u0027: 47,\n",
              "         \u0027宣布\u0027: 5117,\n",
              "         \u00272018\u0027: 1641,\n",
              "         \u0027年\u0027: 123532,\n",
              "         \u0027新兴\u0027: 1203,\n",
              "         \u0027占\u0027: 4953,\n",
              "         \u0027群体\u0027: 1186,\n",
              "         \u0027国金\u0027: 24,\n",
              "         \u0027李立峰\u0027: 7,\n",
              "         \u0027团队\u0027: 4258,\n",
              "         \u0027指出\u0027: 5061,\n",
              "         \u0027最新\u0027: 2318,\n",
              "         \u0027包含\u0027: 577,\n",
              "         \u0027222\u0027: 110,\n",
              "         \u0027剔除\u0027: 105,\n",
              "         \u0027中等\u0027: 130,\n",
              "         \u0027市值\u0027: 498,\n",
              "         \u0027非\u0027: 2844,\n",
              "         \u0027互联互通\u0027: 1137,\n",
              "         \u0027交易\u0027: 3728,\n",
              "         \u0027股票\u0027: 1227,\n",
              "         \u0027以及\u0027: 10208,\n",
              "         \u0027停牌\u0027: 262,\n",
              "         \u0027标的\u0027: 430,\n",
              "         \u0027由于\u0027: 5419,\n",
              "         \u0027很多\u0027: 5800,\n",
              "         \u0027A\u0027: 1026,\n",
              "         \u0027H股\u0027: 52,\n",
              "         \u0027EM\u0027: 6,\n",
              "         \u0027权重\u0027: 223,\n",
              "         \u0027由\u0027: 13081,\n",
              "         \u00270.5%\u0027: 120,\n",
              "         \u0027上升\u0027: 2114,\n",
              "         \u00270.73%\u0027: 56,\n",
              "         \u002740.11%\u0027: 1,\n",
              "         \u0027泛\u0027: 83,\n",
              "         \u0027消费\u0027: 3114,\n",
              "         \u0027次之\u0027: 17,\n",
              "         \u002724.26%\u0027: 1,\n",
              "         \u0027涵盖\u0027: 698,\n",
              "         \u0027大部分\u0027: 1275,\n",
              "         \u0027权重股\u0027: 80,\n",
              "         \u0027动态\u0027: 901,\n",
              "         \u0027加入\u0027: 2036,\n",
              "         \u0027指\u0027: 1521,\n",
              "         \u0027比高\u0027: 52,\n",
              "         \u0027金融公司\u0027: 45,\n",
              "         \u0027增加\u0027: 5500,\n",
              "         \u0027近一半\u0027: 77,\n",
              "         \u0027其他\u0027: 6419,\n",
              "         \u0027稀释\u0027: 38,\n",
              "         \u0027尽管\u0027: 2086,\n",
              "         \u0027事件\u0027: 5187,\n",
              "         \u0027短期\u0027: 1066,\n",
              "         \u0027有所\u0027: 1601,\n",
              "         \u0027提振\u0027: 342,\n",
              "         \u0027中长期\u0027: 347,\n",
              "         \u0027海外\u0027: 2835,\n",
              "         \u0027增量\u0027: 349,\n",
              "         \u0027升温\u0027: 324,\n",
              "         \u0027短期内\u0027: 264,\n",
              "         \u0027尚\u0027: 546,\n",
              "         \u0027不能\u0027: 4328,\n",
              "         \u0027有效\u0027: 4002,\n",
              "         \u0027放大\u0027: 213,\n",
              "         \u0027独\u0027: 64,\n",
              "         \u0027乐乐\u0027: 15,\n",
              "         \u0027情景\u0027: 211,\n",
              "         \u0027难以\u0027: 1256,\n",
              "         \u0027持续\u0027: 6932,\n",
              "         \u0027难\u0027: 1358,\n",
              "         \u0027改变\u0027: 2763,\n",
              "         \u0027光大\u0027: 51,\n",
              "         \u0027并未\u0027: 1157,\n",
              "         \u0027引起\u0027: 1395,\n",
              "         \u0027太大\u0027: 90,\n",
              "         \u0027热情\u0027: 877,\n",
              "         \u0027成交量\u0027: 528,\n",
              "         \u0027较为\u0027: 979,\n",
              "         \u0027平淡\u0027: 71,\n",
              "         \u0027较大\u0027: 1592,\n",
              "         \u0027变化\u0027: 2960,\n",
              "         \u0027成长\u0027: 1718,\n",
              "         \u0027股\u0027: 1011,\n",
              "         \u0027题材股\u0027: 50,\n",
              "         \u0027全天\u0027: 301,\n",
              "         \u0027低迷\u0027: 397,\n",
              "         \u0027表明\u0027: 1385,\n",
              "         \u0027很少\u0027: 493,\n",
              "         \u0027不同\u0027: 5431,\n",
              "         \u0027之间\u0027: 3805,\n",
              "         \u0027腾挪\u0027: 23,\n",
              "         \u0027跷跷板\u0027: 19,\n",
              "         \u0027使得\u0027: 1099,\n",
              "         \u0027难有\u0027: 51,\n",
              "         \u0027突破\u0027: 3090,\n",
              "         \u0027中期\u0027: 399,\n",
              "         \u0027依旧\u0027: 629,\n",
              "         \u0027偏空\u0027: 12,\n",
              "         \u0027维持\u0027: 1185,\n",
              "         \u0027虽然\u0027: 4223,\n",
              "         \u0027至今\u0027: 1600,\n",
              "         \u0027夏普\u0027: 133,\n",
              "         \u0027智能手机\u0027: 296,\n",
              "         \u0027无法\u0027: 2745,\n",
              "         \u0027排得\u0027: 2,\n",
              "         \u0027完全\u0027: 2445,\n",
              "         \u0027没落\u0027: 8,\n",
              "         \u00272013\u0027: 1557,\n",
              "         \u0027退出\u0027: 1574,\n",
              "         \u00273\u0027: 30625,\n",
              "         \u0027月份\u0027: 3190,\n",
              "         \u0027官方\u0027: 1207,\n",
              "         \u0027回归\u0027: 952,\n",
              "         \u0027预示\u0027: 133,\n",
              "         \u0027很快\u0027: 969,\n",
              "         \u0027新机\u0027: 27,\n",
              "         \u0027登场\u0027: 236,\n",
              "         \u0027第一款\u0027: 48,\n",
              "         \u0027登陆\u0027: 389,\n",
              "         \u0027又\u0027: 8702,\n",
              "         \u0027怎么样\u0027: 131,\n",
              "         \u0027近日\u0027: 4490,\n",
              "         \u0027型号\u0027: 222,\n",
              "         \u0027FS8016\u0027: 1,\n",
              "         \u0027神秘\u0027: 390,\n",
              "         \u0027悄然\u0027: 283,\n",
              "         \u0027GeekBench\u0027: 1,\n",
              "         \u0027分库\u0027: 16,\n",
              "         \u0027相关\u0027: 11730,\n",
              "         \u0027机子\u0027: 5,\n",
              "         \u0027并非\u0027: 1034,\n",
              "         \u0027定位\u0027: 1107,\n",
              "         \u0027所\u0027: 6083,\n",
              "         \u0027660\u0027: 21,\n",
              "         \u0027配备\u0027: 486,\n",
              "         \u00274GB\u0027: 14,\n",
              "         \u0027内存\u0027: 75,\n",
              "         \u0027最受\u0027: 187,\n",
              "         \u0027瞩目\u0027: 124,\n",
              "         \u0027芯片\u0027: 569,\n",
              "         \u0027之一\u0027: 5815,\n",
              "         \u0027采用\u0027: 1965,\n",
              "         \u002714\u0027: 7781,\n",
              "         \u0027纳米\u0027: 145,\n",
              "         \u0027工艺\u0027: 765,\n",
              "         \u0027八个\u0027: 106,\n",
              "         \u0027Kryo\u0027: 1,\n",
              "         \u0027260\u0027: 95,\n",
              "         \u0027设计\u0027: 5724,\n",
              "         \u0027Adreno\u0027: 1,\n",
              "         \u0027512\u0027: 14,\n",
              "         \u0027X12\u0027: 1,\n",
              "         \u0027LTE\u0027: 26,\n",
              "         \u0027调制解调器\u0027: 7,\n",
              "         \u0027当前\u0027: 3000,\n",
              "         \u0027市面上\u0027: 85,\n",
              "         \u0027只有\u0027: 3515,\n",
              "         \u0027上市\u0027: 1794,\n",
              "         \u0027销售\u0027: 3842,\n",
              "         \u0027OPPO\u0027: 49,\n",
              "         \u0027R11\u0027: 8,\n",
              "         \u0027核新能\u0027: 1,\n",
              "         \u0027820\u0027: 22,\n",
              "         \u0027强\u0027: 2276,\n",
              "         \u0027单核\u0027: 6,\n",
              "         \u0027改进\u0027: 609,\n",
              "         \u0027明显\u0027: 3638,\n",
              "         \u0027所以\u0027: 2380,\n",
              "         \u0027放在\u0027: 740,\n",
              "         \u0027变成\u0027: 1264,\n",
              "         \u0027高端\u0027: 1602,\n",
              "         \u0027机\u0027: 562,\n",
              "         \u0027不过\u0027: 4040,\n",
              "         \u0027签署\u0027: 2479,\n",
              "         \u0027排他性\u0027: 19,\n",
              "         \u0027协议\u0027: 4190,\n",
              "         \u0027独占\u0027: 27,\n",
              "         ...})"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokener_counter"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## 使用TextRank进行自动摘要提取"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "#### Task 1 使用neteworkx包，建立一个随机图，对该图进行PageRank分析\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hints: 代码可以在Github Group 中看到"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "import networkx\n",
        "import random"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "def genrate_name(length\u003d3):  return \u0027\u0027.join(chr(random.randint(65, 73)) for _ in range(length))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "\u0027ADD\u0027"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "genrate_name()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "graph \u003d {\n",
        "    \u0027Yao\u0027: [\u0027Guo\u0027, \u0027Wang\u0027, \u0027Tian\u0027, \u0027Tim\u0027] + [genrate_name() for _ in range(6)], \n",
        "    \u0027Guo\u0027: [\u0027Li\u0027] + [genrate_name() for _ in range(5)], \n",
        "    \u0027Wang\u0027: [\u0027Li_2\u0027] + [genrate_name() for _ in range(5)], \n",
        "    \u0027Li\u0027: [genrate_name() for _ in range(3)], \n",
        "    \u0027Li_2\u0027: [genrate_name() for _ in range(1)]\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{\u0027Yao\u0027: [\u0027Guo\u0027,\n",
              "  \u0027Wang\u0027,\n",
              "  \u0027Tian\u0027,\n",
              "  \u0027Tim\u0027,\n",
              "  \u0027FEA\u0027,\n",
              "  \u0027EEI\u0027,\n",
              "  \u0027BDB\u0027,\n",
              "  \u0027GCI\u0027,\n",
              "  \u0027DGF\u0027,\n",
              "  \u0027CDH\u0027],\n",
              " \u0027Guo\u0027: [\u0027Li\u0027, \u0027GHG\u0027, \u0027GGA\u0027, \u0027DIE\u0027, \u0027FDC\u0027, \u0027EFF\u0027],\n",
              " \u0027Wang\u0027: [\u0027Li_2\u0027, \u0027GAI\u0027, \u0027HGE\u0027, \u0027EII\u0027, \u0027AEB\u0027, \u0027CEE\u0027],\n",
              " \u0027Li\u0027: [\u0027IBA\u0027, \u0027CDF\u0027, \u0027CEB\u0027],\n",
              " \u0027Li_2\u0027: [\u0027HGA\u0027]}"
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "graph"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "social_network \u003d networkx.graph.Graph()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "help(social_network.add_edges_from)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "for vertex, edge in graph.items():\n",
        "    for v in edge:\n",
        "        social_network.add_edges_from([(vertex, v)])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "# jupter notebook中需要添加这行代码才能显示图\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\python36\\lib\\site-packages\\networkx\\drawing\\nx_pylab.py:579: MatplotlibDeprecationWarning: \n",
            "The iterable function was deprecated in Matplotlib 3.1 and will be removed in 3.3. Use np.iterable instead.\n",
            "  if not cb.iterable(width):\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 28171 characters]\u003d\n",
            "text/plain": [
              "\u003cFigure size 432x288 with 1 Axes\u003e"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "networkx.draw_networkx(social_network)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "ranking_result \u003d networkx.pagerank(social_network)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[(\u0027Yao\u0027, 0.1747698412634544),\n",
              " (\u0027Wang\u0027, 0.12616099221117885),\n",
              " (\u0027Guo\u0027, 0.12434209154337311),\n",
              " (\u0027Li\u0027, 0.07600909782512963),\n",
              " (\u0027Li_2\u0027, 0.04007465697366682),\n",
              " (\u0027HGA\u0027, 0.022586982681035123),\n",
              " (\u0027IBA\u0027, 0.021706946858994478),\n",
              " (\u0027CDF\u0027, 0.021706946858994478),\n",
              " (\u0027CEB\u0027, 0.021706946858994478),\n",
              " (\u0027GAI\u0027, 0.020875426652805028),\n",
              " (\u0027HGE\u0027, 0.020875426652805028),\n",
              " (\u0027EII\u0027, 0.020875426652805028),\n",
              " (\u0027AEB\u0027, 0.020875426652805028),\n",
              " (\u0027CEE\u0027, 0.020875426652805028),\n",
              " (\u0027GHG\u0027, 0.020654738077342236),\n",
              " (\u0027GGA\u0027, 0.020654738077342236),\n",
              " (\u0027DIE\u0027, 0.020654738077342236),\n",
              " (\u0027FDC\u0027, 0.020654738077342236),\n",
              " (\u0027EFF\u0027, 0.020654738077342236),\n",
              " (\u0027Tian\u0027, 0.020410584159305276),\n",
              " (\u0027Tim\u0027, 0.020410584159305276),\n",
              " (\u0027FEA\u0027, 0.020410584159305276),\n",
              " (\u0027EEI\u0027, 0.020410584159305276),\n",
              " (\u0027BDB\u0027, 0.020410584159305276),\n",
              " (\u0027GCI\u0027, 0.020410584159305276),\n",
              " (\u0027DGF\u0027, 0.020410584159305276),\n",
              " (\u0027CDH\u0027, 0.020410584159305276)]"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sorted(ranking_result.items(), key\u003dlambda x: x[1], reverse\u003dTrue)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{\u0027Yao\u0027: 0.1747698412634544,\n",
              " \u0027Guo\u0027: 0.12434209154337311,\n",
              " \u0027Wang\u0027: 0.12616099221117885,\n",
              " \u0027Tian\u0027: 0.020410584159305276,\n",
              " \u0027Tim\u0027: 0.020410584159305276,\n",
              " \u0027FEA\u0027: 0.020410584159305276,\n",
              " \u0027EEI\u0027: 0.020410584159305276,\n",
              " \u0027BDB\u0027: 0.020410584159305276,\n",
              " \u0027GCI\u0027: 0.020410584159305276,\n",
              " \u0027DGF\u0027: 0.020410584159305276,\n",
              " \u0027CDH\u0027: 0.020410584159305276,\n",
              " \u0027Li\u0027: 0.07600909782512963,\n",
              " \u0027GHG\u0027: 0.020654738077342236,\n",
              " \u0027GGA\u0027: 0.020654738077342236,\n",
              " \u0027DIE\u0027: 0.020654738077342236,\n",
              " \u0027FDC\u0027: 0.020654738077342236,\n",
              " \u0027EFF\u0027: 0.020654738077342236,\n",
              " \u0027Li_2\u0027: 0.04007465697366682,\n",
              " \u0027GAI\u0027: 0.020875426652805028,\n",
              " \u0027HGE\u0027: 0.020875426652805028,\n",
              " \u0027EII\u0027: 0.020875426652805028,\n",
              " \u0027AEB\u0027: 0.020875426652805028,\n",
              " \u0027CEE\u0027: 0.020875426652805028,\n",
              " \u0027IBA\u0027: 0.021706946858994478,\n",
              " \u0027CDF\u0027: 0.021706946858994478,\n",
              " \u0027CEB\u0027: 0.021706946858994478,\n",
              " \u0027HGA\u0027: 0.022586982681035123}"
            ]
          },
          "execution_count": 30,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "networkx.pagerank(social_network)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "#### Task 2 使用PageRank 对文本进行关联分析"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hints: 代码可以在Github Group 中看到"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border\u003d\"1\" class\u003d\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style\u003d\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eUnnamed: 0\u003c/th\u003e\n",
              "      \u003cth\u003econtent\u003c/th\u003e\n",
              "      \u003cth\u003etokenized_content\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e此外，自本周（6月12日）起，除小米手机6等15款机型外，其余机型已暂停更新发布（含开发版/...\u003c/td\u003e\n",
              "      \u003ctd\u003e此外 ， 自 本周 （ 6 月 12 日 ） 起 ， 除 小米 手机 6 等 15 款 机型...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e骁龙835作为唯一通过Windows 10桌面平台认证的ARM处理器，高通强调，不会因为只考...\u003c/td\u003e\n",
              "      \u003ctd\u003e骁龙 835 作为 唯一 通过 Windows   10 桌面 平台 认证 的 ARM 处理...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e2\u003c/td\u003e\n",
              "      \u003ctd\u003e2\u003c/td\u003e\n",
              "      \u003ctd\u003e此前的一加3T搭载的是3400mAh电池，DashCharge快充规格为5V/4A。\\r\\n...\u003c/td\u003e\n",
              "      \u003ctd\u003e此前 的 一加 3T 搭载 的 是 3400mAh 电池 ， DashCharge 快充 规...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e3\u003c/td\u003e\n",
              "      \u003ctd\u003e3\u003c/td\u003e\n",
              "      \u003ctd\u003e这是6月18日在葡萄牙中部大佩德罗冈地区拍摄的被森林大火烧毁的汽车。新华社记者张立云摄\\r\\n\u003c/td\u003e\n",
              "      \u003ctd\u003e这是 6 月 18 日 在 葡萄牙 中部 大 佩德罗 冈 地区 拍摄 的 被 森林 大火 烧...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003ctd\u003e4\u003c/td\u003e\n",
              "      \u003ctd\u003e4\u003c/td\u003e\n",
              "      \u003ctd\u003e（原标题：44岁女子跑深圳约会网友被拒，暴雨中裸身奔走……）\\r\\n@深圳交警微博称：昨日清...\u003c/td\u003e\n",
              "      \u003ctd\u003e（ 原 标题 ： 44 岁 女子 跑 深圳 约会 网友 被 拒 ， 暴雨 中 裸身 奔走 …...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "   Unnamed: 0                                            content  \\\n",
              "0           0  此外，自本周（6月12日）起，除小米手机6等15款机型外，其余机型已暂停更新发布（含开发版/...   \n",
              "1           1  骁龙835作为唯一通过Windows 10桌面平台认证的ARM处理器，高通强调，不会因为只考...   \n",
              "2           2  此前的一加3T搭载的是3400mAh电池，DashCharge快充规格为5V/4A。\\r\\n...   \n",
              "3           3    这是6月18日在葡萄牙中部大佩德罗冈地区拍摄的被森林大火烧毁的汽车。新华社记者张立云摄\\r\\n   \n",
              "4           4  （原标题：44岁女子跑深圳约会网友被拒，暴雨中裸身奔走……）\\r\\n@深圳交警微博称：昨日清...   \n",
              "\n",
              "                                   tokenized_content  \n",
              "0  此外 ， 自 本周 （ 6 月 12 日 ） 起 ， 除 小米 手机 6 等 15 款 机型...  \n",
              "1  骁龙 835 作为 唯一 通过 Windows   10 桌面 平台 认证 的 ARM 处理...  \n",
              "2  此前 的 一加 3T 搭载 的 是 3400mAh 电池 ， DashCharge 快充 规...  \n",
              "3  这是 6 月 18 日 在 葡萄牙 中部 大 佩德罗 冈 地区 拍摄 的 被 森林 大火 烧...  \n",
              "4  （ 原 标题 ： 44 岁 女子 跑 深圳 约会 网友 被 拒 ， 暴雨 中 裸身 奔走 …...  "
            ]
          },
          "execution_count": 31,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pure_content.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "lengthes_of_text \u003d map(len, pure_content[\u0027content\u0027].tolist())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[177,\n",
              " 329,\n",
              " 216,\n",
              " 45,\n",
              " 1674,\n",
              " 1056,\n",
              " 657,\n",
              " 4093,\n",
              " 3665,\n",
              " 755,\n",
              " 169,\n",
              " 1467,\n",
              " 487,\n",
              " 774,\n",
              " 1137,\n",
              " 331,\n",
              " 293,\n",
              " 495,\n",
              " 289,\n",
              " 1707]"
            ]
          },
          "execution_count": 33,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "list(lengthes_of_text)[:20]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\u0027c:\\\\python36\\\\lib\\\\site-packages\\\\networkx\u0027]"
            ]
          },
          "execution_count": 34,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "networkx.__path__"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "import matplotlib.font_manager as fm\n",
        "fp1 \u003d fm.FontProperties(fname\u003d\"../data/SourceHanSerifSC-Regular.otf\")\n",
        "# how to set chinese display: \n",
        "# https://knowlab.wordpress.com/2016/05/25/networkx-%E7%B9%AA%E5%9C%96%E9%A1%AF%E7%A4%BA%E4%B8%AD%E6%96%87%E7%9A%84%E8%A7%A3%E6%B1%BA%E6%96%B9%E6%B3%95/\n",
        "\n",
        "def get_connect_graph_by_text_rank(tokenized_text: str, window\u003d3):\n",
        "    keywords_graph \u003d networkx.Graph()\n",
        "    tokeners \u003d tokenized_text.split()\n",
        "    for ii, t in enumerate(tokeners):\n",
        "        word_tuples \u003d [(tokeners[connect], t) \n",
        "                       for connect in range(ii-window, ii+window+1) \n",
        "                       if connect \u003e\u003d 0 and connect \u003c len(tokeners)]\n",
        "        keywords_graph.add_edges_from(word_tuples)\n",
        "\n",
        "    return keywords_graph"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "first_content \u003d pure_content.iloc[1][\u0027tokenized_content\u0027]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "first_content_continous \u003d pure_content.iloc[4][\u0027content\u0027]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\u0027（原标题：44岁女子跑深圳约会网友被拒，暴雨中裸身奔走……）\u0027,\n",
              " \u0027@深圳交警微博称：昨日清晨交警发现有一女子赤裸上身，行走在南坪快速上，期间还起了轻生年头，一辅警发现后赶紧为其披上黄衣，并一路劝说她。\u0027,\n",
              " \u0027那么事发时\u0027,\n",
              " \u0027到底都发生了些什么呢？\u0027,\n",
              " \u0027南都记者带您一起还原现场\u0027,\n",
              " \u0027南都记者在龙岗大队坂田中队见到了辅警刘青（发现女生的辅警），一位外表高大帅气，说话略带些腼腆的90后青年。\u0027,\n",
              " \u0027刘青介绍，6月16日早上7时36分，他正在环城南路附近值勤，接到中队关于一位女子裸身进入机动车可能有危险的警情，随后骑着小铁骑开始沿路寻找，大概花了十多分钟在南坪大道坂田出口往龙岗方向的逆行辅道上发现该女子。\u0027,\n",
              " \u0027女子身上一丝不挂地逆车流而行，时走时停，时坐时躺，险象环生。刘青停好小铁骑，和另外一名巡防员追了上去，发现女子的情绪很低落，话不多，刘青尝试和女子交流，劝说女子离开，可女子并不愿意接受，继续缓慢地往南坪快速路的主干道上走去。\u0027,\n",
              " \u0027此时路边上已经聚集了很市民围观，为了不刺激女子的情绪，刘青和巡防员一边盯着女子一边驱赶着围观的群众。\u0027,\n",
              " \u0027现场还原\u0027,\n",
              " \u0027从警方提供的一份视频了解到，16日早上7时25分，女子出现在坂雪岗大道与环城南路的监控视频中，此时女子还穿着白色的内裤，正沿着坂雪岗大道往南坪快速的方向缓慢地走着。\u0027,\n",
              " \u0027当时正值上班高峰期，十字路口的车流已经排起了长队。当女子出现时，路上的市民纷纷驻足观望，不少车辆也放慢了速度，但女子并不为市民观望停下脚步，依然缓慢走着。当女子行进到十字路口中间时，一辆大货车挡住了镜头，但是当女子再次出现镜头时，可以发现女子已经没穿内裤了，全身裸露继续朝着南坪快速方向走去。记者发现，视频中女子周围并没有人尾随或者上前劝止的市民。\u0027,\n",
              " \u0027一大清早路上看到这样的情况\u0027,\n",
              " \u0027恐怕大家都没办法淡定\u0027,\n",
              " \u0027面对这一情况\u0027,\n",
              " \u0027刘青表示，“一开始根本不敢看她，心里挺别扭，感觉很尴尬”，但当刘青跟随女子上了南坪快速路主干道时，女子作出了让人意想不到的举动，她突然靠近护栏要从上面跳下去，刘青赶忙冲上去拉住了女子的手，将其控制住并远离护栏。碍于女子没有穿衣服，刘青递上衣服，女子没接受还把衣服扔到排水沟里，继续往前走，没办法刘青只能紧紧拉着她的一只手跟在后面。\u0027,\n",
              " \u0027刘青一路上耐心地开导安慰她，但只听到她不断地重复着一句话“要是你也遭遇我的事，你也会这样的”，期间她还不时试图挣脱刘青的手要冲向护栏往下跳。\u0027,\n",
              " \u0027就这样，我被牵着走了大概十多分钟，天突然下起了大暴雨，雨大的连眼睛都睁不开”刘青继续说着，瞬间他们就被雨透了，但女子依然不愿意接受刘青的帮助，就继续冒着大雨往前走。\u0027,\n",
              " \u0027大概走了有四十分钟吧，女子突然停下来说“我想回家了”，然后女子也接受了刘青递过来的小黄衣，就出现了深圳微博上的照片，女子披着小黄衣，刘青小心翼翼地在旁边走着的场景。从南平快速下来后，刘青和巡防员将女子带到了附近的坂田派出所。\u0027,\n",
              " \u0027那姑娘到底是遭遇了什么样的事情\u0027,\n",
              " \u0027才会说\u0027,\n",
              " \u0027“要是你也遭遇我的事，你也会这样”\u0027,\n",
              " \u0027据警方透露，该女子姓陈，系湖北人，今年44岁，据家属反映其有精神病史。三天前，陈某从老家来深圳约会网友，但约会受挫导致情绪异常，女子遂产生轻生念头。\u0027,\n",
              " \u0027目前\u0027,\n",
              " \u0027陈某已经被送往深圳某精神病医院进行治疗\u0027,\n",
              " \u0027大大君只希望姑娘能早点康复\u0027,\n",
              " \u0027其实真爱的到来并不存在年龄的限制\u0027,\n",
              " \u0027你们说呢？\u0027,\n",
              " \u0027因善良的原因\u0027,\n",
              " \u0027一众网友纷纷为\u0027,\n",
              " \u0027交警暖男点ZAN\u0027,\n",
              " \u0027@弓常yan桦：就想问这个小哥哥有女票吗\u0027,\n",
              " \u0027@原谅我这一辈子浪荡不羁爱萨摩耶：有什么过不去的要轻生嘛？\u0027,\n",
              " \u0027想想自己的家人。同时也感谢交警蜀黍\u0027,\n",
              " \u0027@火心聆听心灵：点赞交警\u0027,\n",
              " \u0027@中華云盾：警察……警察就是群众最需时申出援手\u0027,\n",
              " \u0027@Tomchlee：蜀黍帅！\u0027,\n",
              " \u0027@SJ-李赫海i：这个交警很暖有木有！\u0027,\n",
              " \u0027男子迷奸网友拍418个视频\u0027,\n",
              " \u0027女方从20岁到50岁不等\u0027,\n",
              " \u0027去年6月7号上午，淮安市涟水县公安局刑警大队突然接到了一个奇怪的报警电话，一名女子言语不清，声称自己遭到了侵害。女子、被侵害、言语不清，几个关键词令接到电话的民警瞬间紧张起来。\u0027]"
            ]
          },
          "execution_count": 38,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "first_content_continous.replace(\u0027,\u0027, \u0027 \u0027).split()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "import re"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "pattern \u003d re.compile(\u0027[。，,.]\u0027)\n",
        "token_by_sentence_first \u003d pattern.sub(\u0027 \u0027, first_content_continous).split()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\u0027（原标题：44岁女子跑深圳约会网友被拒\u0027,\n",
              " \u0027暴雨中裸身奔走……）\u0027,\n",
              " \u0027@深圳交警微博称：昨日清晨交警发现有一女子赤裸上身\u0027,\n",
              " \u0027行走在南坪快速上\u0027,\n",
              " \u0027期间还起了轻生年头\u0027,\n",
              " \u0027一辅警发现后赶紧为其披上黄衣\u0027,\n",
              " \u0027并一路劝说她\u0027,\n",
              " \u0027那么事发时\u0027,\n",
              " \u0027到底都发生了些什么呢？\u0027,\n",
              " \u0027南都记者带您一起还原现场\u0027]"
            ]
          },
          "execution_count": 41,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "token_by_sentence_first[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 13604 characters]\n",
            "text/plain": [
              "\u003cFigure size 432x288 with 1 Axes\u003e"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence_graph \u003d get_connect_graph_by_text_rank(\u0027 \u0027.join(token_by_sentence_first))\n",
        "networkx.draw(sentence_graph)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "ranking_sentence \u003d networkx.pagerank(sentence_graph)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "def split_sentence(sentence):\n",
        "    pattern \u003d re.compile(\u0027[。，,.]：\u0027)\n",
        "    split \u003d pattern.sub(\u0027 \u0027, sentence).split()  # split sentence\n",
        "    return split"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q6. 切句子是如何实现的？如何使用正则表达式进行句子分割？ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#分割句子，将句子按照逗号和句号分隔开。\n",
        "def split_sentence(sentence):\n",
        "    pattern \u003d re.compile(\u0027[(, 。\\s)！？。.]\u0027)\n",
        "    split \u003d pattern.sub(\u0027 \u0027, sentence).split()  # split sentence\n",
        "    return split"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\u0027我生气了\u0027, \u0027我不开心\u0027]"
            ]
          },
          "execution_count": 51,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "split_sentence(\u0027我生气了,我不开心。\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q7. 如果对切词之后的句子的标点符号进行恢复？"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "\u0027原来他们每天偷偷摸摸干这事。 原来他们每天偷偷摸摸干这事。 大家还记得今年4月光谷曾发生的一件大事吗。 抢购高峰期，网页打开慢，网银交易出现比较迟钝现象。 有不法分子会冒充客服或店主，以单子出现“调单”“卡单”等说法为幌子，故意说没有收到款项欺诈消费者，取得信任后要求消费者以其他办法进行转账。 识骗方法：消费者要注意，调单、卡单、激活订单等多为诈骗术语，见此类词语基本可以断定对方是骗子。 没有收到款项这样的问题，是不会在东京、天猫这样的大型网络购物平台出现的。 抢购高峰期，网页打开慢，网银交易出现比较迟钝现象。 有不法分子会冒充客服或店主，以单子出现“调单”“卡单”等说法为幌子，故意说没有收到款项欺诈消费者，取得信任后要求消费者以其他办法进行转账。 识骗方法：消费者要注意，调单、卡单、激活订单等多为诈骗术语，见此类词语基本可以断定对方是骗子。 没有收到款项这样的问题，是不会在东京、天猫这样的大型网络购物平台出现的。 同时，提醒大家：保护好自己账户信息，遇到情况一定及时报警。 武汉破特大网络诈骗案抓808人:以\"内幕消息\"骗人\u0027"
            ]
          },
          "execution_count": 52,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "def get_summarization_simple_with_text_rank(text, constraint\u003d200):\n",
        "    return get_summarization_simple(text, sentence_ranking_by_text_ranking, constraint)\n",
        "\n",
        "\n",
        "#建立句子和标点符号之间的关系，例如，建立一个字典\n",
        "def get_summarization_simple(text, score_fn, constraint\u003d200):\n",
        "    sub_sentence \u003d split_sentence(text)\n",
        "    ranking_sentence \u003d score_fn(sub_sentence)\n",
        "    selected_text \u003d set()\n",
        "    current_text \u003d \u0027\u0027\n",
        "    \n",
        "    for sen, _ in ranking_sentence:\n",
        "        if len(current_text) \u003c constraint:\n",
        "            current_text +\u003d sen\n",
        "            selected_text.add(sen)\n",
        "        else:\n",
        "            break\n",
        "\n",
        "    summarized \u003d []\n",
        "    for sen in sub_sentence:  # print the selected sentence by sequent\n",
        "        if sen in selected_text:\n",
        "            summarized.append(sen)\n",
        "    return summarized\n",
        "\n",
        "\n",
        "def sentence_ranking_by_text_ranking(split_sentence):\n",
        "    sentence_graph \u003d get_connect_graph_by_text_rank(\u0027 \u0027.join(split_sentence))\n",
        "    ranking_sentence \u003d networkx.pagerank(sentence_graph)\n",
        "    ranking_sentence \u003d sorted(ranking_sentence.items(), key\u003dlambda x: x[1], reverse\u003dTrue)\n",
        "    return ranking_sentence\n",
        "\n",
        "\n",
        "sharp_news \u003d pure_content.iloc[8][\u0027content\u0027]\n",
        "\u0027。 \u0027.join(get_summarization_simple_with_text_rank(sharp_news, constraint\u003d250))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "1296----\n",
            "江南华南强降雨持续 北方旱区多雷阵雨 这几天，暴雨成了天气关键词 其中，广东、福建、台湾等地一天下了四五百毫米的超级暴雨；江浙沪则是连绵不绝的“梅雨”；还有东北以及京津冀等地将会有冷涡雷雨 当它在低纬度以较强形式出现时，就是我们熟知的台风；而在中高纬度出现时，就是雨雪“制造机”了 历史上，从蒙古国来的低涡也曾造成多次暴雨事件 比如就北京来说，据统计，1956年-2000年蒙古低涡共出现131次，占北京暴雨过程总次数386次的34%，是北京地区出现频率较高的暴雨天气系统 不过单纯低涡并不足以形成降雨，需要水汽条件，近日京津冀等地气温持续偏高，水汽条件从南方北上，所以降雨一触即发\n"
          ]
        }
      ],
      "source": [
        "for index in random.sample(list(range(len(pure_content))), 30):\n",
        "    content \u003d pure_content.iloc[index][\u0027content\u0027]\n",
        "    if len(content) \u003e 1000:\n",
        "        print(\u0027{}----\u0027.format(index))\n",
        "        print(\u0027 \u0027.join(get_summarization_simple_with_text_rank(content, constraint\u003d250)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "metadata": {
        "scrolled": false,
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[(\u0027一名女子言语不清\u0027, 0.008761809701694283),\n",
              " (\u0027行走在南坪快速上\u0027, 0.008761809701694281),\n",
              " (\u0027期间还起了轻生年头\u0027, 0.00853806788025346),\n",
              " (\u0027淮安市涟水县公安局刑警大队突然接到了一个奇怪的报警电话\u0027, 0.00853806788025346),\n",
              " (\u0027一辅警发现后赶紧为其披上黄衣\u0027, 0.008371719895408159),\n",
              " (\u0027去年6月7号上午\u0027, 0.008371719895408159),\n",
              " (\u0027并一路劝说她\u0027, 0.008245759268329758),\n",
              " (\u0027女方从20岁到50岁不等\u0027, 0.008245759268329758),\n",
              " (\u0027那么事发时\u0027, 0.00815029910456809),\n",
              " (\u0027男子迷奸网友拍418个视频\u0027, 0.00815029910456809),\n",
              " (\u0027@SJ-李赫海i：这个交警很暖有木有！\u0027, 0.008078401421739694),\n",
              " (\u0027到底都发生了些什么呢？\u0027, 0.008078401421739692),\n",
              " (\u0027南都记者带您一起还原现场\u0027, 0.008024087516959199),\n",
              " (\u0027@Tomchlee：蜀黍帅！\u0027, 0.008024087516959199),\n",
              " (\u0027南都记者在龙岗大队坂田中队见到了辅警刘青（发现女生的辅警）\u0027, 0.007983207314774927),\n",
              " (\u0027@中華云盾：警察……警察就是群众最需时申出援手\u0027, 0.007983207314774927),\n",
              " (\u0027一位外表高大帅气\u0027, 0.007952624377193676),\n",
              " (\u0027@火心聆听心灵：点赞交警\u0027, 0.007952624377193674),\n",
              " (\u0027说话略带些腼腆的90后青年\u0027, 0.007929884597094566),\n",
              " (\u0027同时也感谢交警蜀黍\u0027, 0.007929884597094566),\n",
              " (\u0027刘青介绍\u0027, 0.007913142240506744),\n",
              " (\u0027想想自己的家人\u0027, 0.007913142240506744),\n",
              " (\u0027@原谅我这一辈子浪荡不羁爱萨摩耶：有什么过不去的要轻生嘛？\u0027, 0.007900964870248895),\n",
              " (\u00276月16日早上7时36分\u0027, 0.007900964870248892),\n",
              " (\u0027他正在环城南路附近值勤\u0027, 0.007892232774304273),\n",
              " (\u0027@弓常yan桦：就想问这个小哥哥有女票吗\u0027, 0.007892232774304273),\n",
              " (\u0027接到中队关于一位女子裸身进入机动车可能有危险的警情\u0027, 0.00788607704919873),\n",
              " (\u0027交警暖男点ZAN\u0027, 0.00788607704919873),\n",
              " (\u0027一众网友纷纷为\u0027, 0.007881820981426759),\n",
              " (\u0027随后骑着小铁骑开始沿路寻找\u0027, 0.007881820981426757),\n",
              " (\u0027因善良的原因\u0027, 0.007878941618434662),\n",
              " (\u0027大概花了十多分钟在南坪大道坂田出口往龙岗方向的逆行辅道上发现该女子\u0027, 0.00787894161843466),\n",
              " (\u0027你们说呢？\u0027, 0.007877040414333012),\n",
              " (\u0027女子身上一丝不挂地逆车流而行\u0027, 0.00787704041433301),\n",
              " (\u0027时走时停\u0027, 0.007875818243366639),\n",
              " (\u0027其实真爱的到来并不存在年龄的限制\u0027, 0.007875818243366639),\n",
              " (\u0027时坐时躺\u0027, 0.007875055326830038),\n",
              " (\u0027大大君只希望姑娘能早点康复\u0027, 0.007875055326830038),\n",
              " (\u0027险象环生\u0027, 0.007874594202822473),\n",
              " (\u0027陈某已经被送往深圳某精神病医院进行治疗\u0027, 0.007874594202822473),\n",
              " (\u0027目前\u0027, 0.007874325165885149),\n",
              " (\u0027刘青停好小铁骑\u0027, 0.007874325165885147),\n",
              " (\u0027女子遂产生轻生念头\u0027, 0.00787417418096783),\n",
              " (\u0027和另外一名巡防员追了上去\u0027, 0.007874174180967829),\n",
              " (\u0027但约会受挫导致情绪异常\u0027, 0.007874093011541456),\n",
              " (\u0027发现女子的情绪很低落\u0027, 0.007874093011541454),\n",
              " (\u0027话不多\u0027, 0.007874051412730365),\n",
              " (\u0027陈某从老家来深圳约会网友\u0027, 0.007874051412730365),\n",
              " (\u0027刘青尝试和女子交流\u0027, 0.007874031209340058),\n",
              " (\u0027三天前\u0027, 0.007874031209340058),\n",
              " (\u0027劝说女子离开\u0027, 0.007874021979828482),\n",
              " (\u0027据家属反映其有精神病史\u0027, 0.007874021979828482),\n",
              " (\u0027今年44岁\u0027, 0.00787401805200212),\n",
              " (\u0027可女子并不愿意接受\u0027, 0.007874018052002118),\n",
              " (\u0027继续缓慢地往南坪快速路的主干道上走去\u0027, 0.007874016514985274),\n",
              " (\u0027系湖北人\u0027, 0.007874016514985274),\n",
              " (\u0027此时路边上已经聚集了很市民围观\u0027, 0.007874015971942193),\n",
              " (\u0027该女子姓陈\u0027, 0.007874015971942193),\n",
              " (\u0027为了不刺激女子的情绪\u0027, 0.007874015803169621),\n",
              " (\u0027据警方透露\u0027, 0.007874015803169621),\n",
              " (\u0027你也会这样”\u0027, 0.007874015758793634),\n",
              " (\u0027刘青和巡防员一边盯着女子一边驱赶着围观的群众\u0027, 0.007874015758793632),\n",
              " (\u0027现场还原\u0027, 0.007874015749519759),\n",
              " (\u0027“要是你也遭遇我的事\u0027, 0.007874015749519759),\n",
              " (\u0027从警方提供的一份视频了解到\u0027, 0.00787401574814256),\n",
              " (\u0027才会说\u0027, 0.00787401574814256),\n",
              " (\u002716日早上7时25分\u0027, 0.007874015748031496),\n",
              " (\u0027女子出现在坂雪岗大道与环城南路的监控视频中\u0027, 0.007874015748031496),\n",
              " (\u0027此时女子还穿着白色的内裤\u0027, 0.007874015748031496),\n",
              " (\u0027正沿着坂雪岗大道往南坪快速的方向缓慢地走着\u0027, 0.007874015748031496),\n",
              " (\u0027当时正值上班高峰期\u0027, 0.007874015748031496),\n",
              " (\u0027十字路口的车流已经排起了长队\u0027, 0.007874015748031496),\n",
              " (\u0027当女子出现时\u0027, 0.007874015748031496),\n",
              " (\u0027路上的市民纷纷驻足观望\u0027, 0.007874015748031496),\n",
              " (\u0027不少车辆也放慢了速度\u0027, 0.007874015748031496),\n",
              " (\u0027但女子并不为市民观望停下脚步\u0027, 0.007874015748031496),\n",
              " (\u0027依然缓慢走着\u0027, 0.007874015748031496),\n",
              " (\u0027当女子行进到十字路口中间时\u0027, 0.007874015748031496),\n",
              " (\u0027一辆大货车挡住了镜头\u0027, 0.007874015748031496),\n",
              " (\u0027但是当女子再次出现镜头时\u0027, 0.007874015748031496),\n",
              " (\u0027可以发现女子已经没穿内裤了\u0027, 0.007874015748031496),\n",
              " (\u0027全身裸露继续朝着南坪快速方向走去\u0027, 0.007874015748031496),\n",
              " (\u0027记者发现\u0027, 0.007874015748031496),\n",
              " (\u0027视频中女子周围并没有人尾随或者上前劝止的市民\u0027, 0.007874015748031496),\n",
              " (\u0027一大清早路上看到这样的情况\u0027, 0.007874015748031496),\n",
              " (\u0027恐怕大家都没办法淡定\u0027, 0.007874015748031496),\n",
              " (\u0027面对这一情况\u0027, 0.007874015748031496),\n",
              " (\u0027刘青表示\u0027, 0.007874015748031496),\n",
              " (\u0027“一开始根本不敢看她\u0027, 0.007874015748031496),\n",
              " (\u0027心里挺别扭\u0027, 0.007874015748031496),\n",
              " (\u0027感觉很尴尬”\u0027, 0.007874015748031496),\n",
              " (\u0027但当刘青跟随女子上了南坪快速路主干道时\u0027, 0.007874015748031496),\n",
              " (\u0027女子作出了让人意想不到的举动\u0027, 0.007874015748031496),\n",
              " (\u0027她突然靠近护栏要从上面跳下去\u0027, 0.007874015748031496),\n",
              " (\u0027刘青赶忙冲上去拉住了女子的手\u0027, 0.007874015748031496),\n",
              " (\u0027将其控制住并远离护栏\u0027, 0.007874015748031496),\n",
              " (\u0027碍于女子没有穿衣服\u0027, 0.007874015748031496),\n",
              " (\u0027刘青递上衣服\u0027, 0.007874015748031496),\n",
              " (\u0027女子没接受还把衣服扔到排水沟里\u0027, 0.007874015748031496),\n",
              " (\u0027继续往前走\u0027, 0.007874015748031496),\n",
              " (\u0027没办法刘青只能紧紧拉着她的一只手跟在后面\u0027, 0.007874015748031496),\n",
              " (\u0027刘青一路上耐心地开导安慰她\u0027, 0.007874015748031496),\n",
              " (\u0027但只听到她不断地重复着一句话“要是你也遭遇我的事\u0027, 0.007874015748031496),\n",
              " (\u0027你也会这样的”\u0027, 0.007874015748031496),\n",
              " (\u0027期间她还不时试图挣脱刘青的手要冲向护栏往下跳\u0027, 0.007874015748031496),\n",
              " (\u0027就这样\u0027, 0.007874015748031496),\n",
              " (\u0027我被牵着走了大概十多分钟\u0027, 0.007874015748031496),\n",
              " (\u0027天突然下起了大暴雨\u0027, 0.007874015748031496),\n",
              " (\u0027雨大的连眼睛都睁不开”刘青继续说着\u0027, 0.007874015748031496),\n",
              " (\u0027瞬间他们就被雨透了\u0027, 0.007874015748031496),\n",
              " (\u0027但女子依然不愿意接受刘青的帮助\u0027, 0.007874015748031496),\n",
              " (\u0027就继续冒着大雨往前走\u0027, 0.007874015748031496),\n",
              " (\u0027大概走了有四十分钟吧\u0027, 0.007874015748031496),\n",
              " (\u0027女子突然停下来说“我想回家了”\u0027, 0.007874015748031496),\n",
              " (\u0027然后女子也接受了刘青递过来的小黄衣\u0027, 0.007874015748031496),\n",
              " (\u0027就出现了深圳微博上的照片\u0027, 0.007874015748031496),\n",
              " (\u0027女子披着小黄衣\u0027, 0.007874015748031496),\n",
              " (\u0027刘青小心翼翼地在旁边走着的场景\u0027, 0.007874015748031496),\n",
              " (\u0027从南平快速下来后\u0027, 0.007874015748031496),\n",
              " (\u0027刘青和巡防员将女子带到了附近的坂田派出所\u0027, 0.007874015748031496),\n",
              " (\u0027那姑娘到底是遭遇了什么样的事情\u0027, 0.007874015748031496),\n",
              " (\u0027声称自己遭到了侵害\u0027, 0.007760391420088102),\n",
              " (\u0027@深圳交警微博称：昨日清晨交警发现有一女子赤裸上身\u0027, 0.007760391420088101),\n",
              " (\u0027暴雨中裸身奔走……）\u0027, 0.006743520732558356),\n",
              " (\u0027女子、被侵害、言语不清\u0027, 0.006743520732558355),\n",
              " (\u0027（原标题：44岁女子跑深圳约会网友被拒\u0027, 0.005706295418150805),\n",
              " (\u0027几个关键词令接到电话的民警瞬间紧张起来\u0027, 0.005706295418150804)]"
            ]
          },
          "execution_count": 54,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sorted(ranking_sentence.items(), key\u003dlambda x: x[1], reverse\u003dTrue)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "keywords_graph \u003d get_connect_graph_by_text_rank(first_content, window\u003d5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "ranking_words \u003d networkx.pagerank(keywords_graph)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 39553 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 40857 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 39553 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 40857 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20316 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20026 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20316 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20026 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21807 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 19968 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 21807 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 19968 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 36890 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 36807 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 36890 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 36807 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26700 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 38754 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26700 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 38754 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24179 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21488 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24179 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 21488 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 35748 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 35777 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 35748 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 35777 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 30340 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 30340 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 22788 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 29702 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 22120 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 22788 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 29702 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 22120 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 65292 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 65292 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 39640 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 39640 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24378 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 35843 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24378 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 35843 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 19981 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20250 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 19981 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20250 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 22240 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 22240 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21482 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 21482 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 32771 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 34385 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 32771 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 34385 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24615 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 33021 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24615 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 33021 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 32780 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 32780 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21435 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 21435 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 23631 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 34109 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 25481 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 23631 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 34109 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 25481 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 23567 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 23567 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26680 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24515 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26680 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24515 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 12290 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 12290 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 30456 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21453 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 30456 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 21453 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20182 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20204 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20182 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20204 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 27491 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 27491 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 32852 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 25163 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 32852 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 25163 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24494 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 36719 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24494 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 36719 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 25214 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21040 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 25214 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 21040 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 31181 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 31181 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 36866 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21512 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 36866 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 21512 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 12289 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 12289 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20860 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 39038 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20860 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 39038 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21644 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 21644 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21151 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 32791 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 21151 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 32791 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 23436 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 32654 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 23436 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 32654 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26041 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26696 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26041 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26696 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 25253 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 36947 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 25253 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 36947 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 31216 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 31216 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24050 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 32463 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24050 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 32463 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 25343 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 25343 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20102 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 20102 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20123 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20123 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26032 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 26032 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 28304 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 30721 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 28304 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 30721 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20197 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20415 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20197 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20415 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26356 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 22909 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26356 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 22909 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 22320 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 22320 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 35299 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 35299 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26550 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26500 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26550 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26500 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 36164 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26009 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 36164 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26009 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26174 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 31034 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26174 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 31034 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 27454 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 27454 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 38598 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 25104 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 38598 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 25104 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 22522 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24102 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 22522 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24102 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 34013 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 29273 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 34013 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 29273 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 27604 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 27604 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20256 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 32479 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20256 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 32479 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21487 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 21487 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 33410 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 30465 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 33410 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 30465 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 33267 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 23569 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 33267 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 23569 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 31354 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 38388 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 31354 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 38388 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 25353 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 35745 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21010 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 25353 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 35745 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 21010 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20170 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24180 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20170 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24180 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21326 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 30805 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 21326 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 30805 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24800 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26222 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24800 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26222 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24819 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24819 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 23558 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 23558 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 39318 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21457 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 39318 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 21457 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 30005 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 33041 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 30005 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 33041 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 39044 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 39044 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 22343 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 22343 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26159 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 26159 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20108 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20108 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24418 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24577 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24418 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24577 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20135 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 21697 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20135 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 21697 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24403 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 28982 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24403 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 28982 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 21482 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26159 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20010 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 20010 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 24320 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 22987 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 24320 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 22987 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26410 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26469 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26410 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26469 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20063 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 35768 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20063 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 35768 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 36824 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 36824 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 33021 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 35265 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 35265 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 19977 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 26143 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 19977 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 26143 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 31185 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 31185 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 40594 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 40607 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 40594 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 40607 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 31859 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 23567 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 31859 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 28558 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 28227 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 28558 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 28227 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 31561 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:176: RuntimeWarning: Glyph 31561 missing from current font.\n",
            "  font.load_char(ord(s), flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 36827 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:211: RuntimeWarning: Glyph 20837 missing from current font.\n",
            "  font.set_text(s, 0.0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 36827 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n",
            "c:\\python36\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py:180: RuntimeWarning: Glyph 20837 missing from current font.\n",
            "  font.set_text(s, 0, flags\u003dflags)\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 92151 characters]\u003d\n",
            "text/plain": [
              "\u003cFigure size 432x288 with 1 Axes\u003e"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from matplotlib import rcParams\n",
        "rcParams[\u0027font.family\u0027] \u003d \u0027../data/SourceHanSerifSC-Regular.otf\u0027\n",
        "networkx.draw_networkx(keywords_graph)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[(\u0027，\u0027, 0.04971796909107254),\n",
              " (\u0027的\u0027, 0.032169208603727614),\n",
              " (\u0027。\u0027, 0.02860228225909264),\n",
              " (\u0027、\u0027, 0.02771502328849811),\n",
              " (\u0027骁龙\u0027, 0.017759850248184172),\n",
              " (\u0027Windows\u0027, 0.015115542829056361),\n",
              " (\u002710\u0027, 0.014661459543306039),\n",
              " (\u0027835\u0027, 0.01417195081727616),\n",
              " (\u0027桌面\u0027, 0.013938442660657342),\n",
              " (\u0027平台\u0027, 0.012843338290542764)]"
            ]
          },
          "execution_count": 58,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sorted(ranking_words.items(), key\u003dlambda x: x[1], reverse\u003dTrue)[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      },
      "source": [
        "一、文本预处理的常见步骤\n",
        "\n",
        "1.使用正则表达式去除文本多余的换行符及空格，数字及英文字母；\n",
        "\n",
        "2.文章分句：分句的依据为文章中出现句子结束符“.!?。！？”从而实现句子之间的切分；\n",
        "\n",
        "3.使用python的中文分词库jieba进行句子分词；\n",
        "\n",
        "4.使用停用词表去除停用词；\n",
        "\n",
        "5.TF_IDF特征词提取\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "#### Task3 在Pycharm里边合并代码，合并成一个单独的函数，实现输入一篇文章，输出该文本的摘要流程"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "`Jupyter Notebook`里边适合做中间结果的记录以及想法验证，真正开发还是要在Pycharm等集成IDE里边进行"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## 使用词向量进行自动摘要"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q8. 使用词向量进行自动摘要的基本假设是什么？ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "**句子向量能够衡量句子之间的相似度，故而可以通过该方法判断某个句子是否和文本含义相关**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "#### Task4. 使用普林斯顿2017年的sentence embedding方法，进行句子embedding的构建"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q9. 该句子embedding的原理是什么，有什么优势？ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答："
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "第一步，将句子里所有的词向量都乘以一个权重，这个权重是（a）/（a+frequency），其中a是一个常数；然后把所有的词向量相加，得到一个简单的句向量。\n",
        "\n",
        "第二步：将所有的句向量减去他们在句向量矩阵的主成分U的投影，有点类似PCA，然后得到的就是该句子的句向量。\n",
        "\n",
        "优点：快，无监督，性能好，运算简单"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "提示：频率，重要性，快速，性能好"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q10. embedding是什么？ 用数学的方式解释embedding。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答：将数据中个体原本之间的关系能够在低维空间上表示，同时能够用数学的方法来保留这些个体之间的相互关系"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "提示：假设两个entity保持具有某种“偏序”特性，即，可以比较，那么，将这两个 entity 投影到另外的空间中，在该空间中，这种“偏序”依然能够保持"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.63d0.793a.ff1a.5047.8bbe.4e24.4e2a.65.6e.74.69.74.79.4fdd.6301.5177.6709.67d0.79cd.201c.504f.5e8f.201d.7279.6027.ff0c.5373.ff0c.53ef.4ee5.6bd4.8f83.ff0c.90a3.4e48.ff0c.5c06.8fd9.4e24.4e2a.20.65.6e.74.69.74.79.20.6295.5f71.5230.53e6.5916.7684.7a7a.95f4.4e2d.ff0c.5728.8be5.7a7a.95f4.4e2d.ff0c.8fd9.79cd.201c.504f.5e8f.201d.4f9d.7136.80fd.591f.4fdd.6301.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q11. 除了word embedding， sentence embedding， embedding还可以应用到哪些场景？"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答：用户、商品。对于之间的关系描述性非常强，更加准确。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "例如社交网络里边人员的embedding（进行好友推荐）， 购物网站商品的embedding（进行商品推荐）\n",
        "另外，可以思考一下基于这种embedding的推荐的优点在哪里？ "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.4f8b.5982.793e.4ea4.7f51.7edc.91cc.8fb9.4eba.5458.7684.65.6d.62.65.64.64.69.6e.67.ff08.8fdb.884c.597d.53cb.63a8.8350.ff09.ff0c.20.8d2d.7269.7f51.7ad9.5546.54c1.7684.65.6d.62.65.64.64.69.6e.67.ff08.8fdb.884c.5546.54c1.63a8.8350.ff09.a.53e6.5916.ff0c.53ef.4ee5.601d.8003.4e00.4e0b.57fa.4e8e.8fd9.79cd.65.6d.62.65.64.64.69.6e.67.7684.63a8.8350.7684.4f18.70b9.5728.54ea.91cc.ff1f.20.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q12. 这种embedding的方法和tfidf有何异同？ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答：tf-idf通过计算某个单字在一段文字中出现的频率，与在全部文档中出现的频率，来计算这个词出现的频率，通常用来表示这个词对于一段文字的重要程度； embedding方法是计算这个词上下文中的词出现的概率，通过这个矩阵来表示这个词的语义。"
      ]
    },
    {
      "cell_type": "raw",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "提示：tfidf考虑的频率其 tf 和 idf 分别是什么？ 该embedding的频率类似于什么？ "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "hint(\u0027a.63d0.793a.ff1a.74.66.69.64.66.8003.8651.7684.9891.7387.5176.20.74.66.20.548c.20.69.64.66.20.5206.522b.662f.4ec0.4e48.ff1f.20.8be5.65.6d.62.65.64.64.69.6e.67.7684.9891.7387.7c7b.4f3c.4e8e.4ec0.4e48.ff1f.20.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q13. 构建句子想了的过程中，面对未知的单词，如果解决？"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答:省略、随机值、古德-图灵法"
      ]
    },
    {
      "cell_type": "raw",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "提示：开发型问题，可以省略，可以赋予随机值，可以给一特定值，具体情况需要经过试验进行分析"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.5f00.53d1.578b.95ee.9898.ff0c.53ef.4ee5.7701.7565.ff0c.53ef.4ee5.8d4b.4e88.968f.673a.503c.ff0c.53ef.4ee5.7ed9.4e00.7279.5b9a.503c.ff0c.5177.4f53.60c5.51b5.9700.8981.7ecf.8fc7.8bd5.9a8c.8fdb.884c.5206.6790.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q14. 构建完成句子向量之后，句子相似度如何判断？ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答：计算两个矩阵的余弦距离(cosine，余弦距离)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.63.6f.73.69.6e.65.ff0c.20.4f59.5f26.8ddd.79bb.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q15. 为什么要用cosine距离进行判断，为何不用欧几里得？ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答：\n"
      ]
    },
    {
      "cell_type": "raw",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "提示：请参考word2vec训练时候的的概率分布的定义，可以看到其loss当时优化的就是 exp(v1 . v2), 那么当 v1 . v2最大的时候，\n",
        "就是这两个向量方向一致的时候，所以，该词向量在训练的时候的loss的衡量，就是依据方向进行的。 "
      ]
    },
    {
      "cell_type": "raw",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "另外，也不是不能用欧几里得，但是依据上一提示之分析，欧几里得是不能很好保持语义相似性的。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "def cosine_similarity(vec1, vec2):\n",
        "    \u0027\u0027\u0027\n",
        "    计算两个向量之间的余弦相似度\n",
        "    :param vec1:\n",
        "    :param vec2:\n",
        "    :return:\n",
        "    \u0027\u0027\u0027\n",
        "    tx \u003d np.array(vec1)\n",
        "    ty \u003d np.array(vec2)\n",
        "    cos1 \u003d np.sum(tx * ty)\n",
        "    cos21 \u003d np.sqrt(sum(tx ** 2))\n",
        "    cos22 \u003d np.sqrt(sum(ty ** 2))\n",
        "    cosine_value \u003d cos1 / float(cos21 * cos22)\n",
        "    return cosine_value\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.63d0.793a.ff1a.8bf7.53c2.8003.77.6f.72.64.32.76.65.63.8bad.7ec3.65f6.5019.7684.7684.6982.7387.5206.5e03.7684.5b9a.4e49.ff0c.53ef.4ee5.770b.5230.5176.6c.6f.73.73.5f53.65f6.4f18.5316.7684.5c31.662f.20.65.78.70.28.76.31.20.2e.20.76.32.29.2c.20.90a3.4e48.5f53.20.76.31.20.2e.20.76.32.6700.5927.7684.65f6.5019.ff0c.a.5c31.662f.8fd9.4e24.4e2a.5411.91cf.65b9.5411.4e00.81f4.7684.65f6.5019.ff0c.6240.4ee5.ff0c.8be5.8bcd.5411.91cf.5728.8bad.7ec3.7684.65f6.5019.7684.6c.6f.73.73.7684.8861.91cf.ff0c.5c31.662f.4f9d.636e.65b9.5411.8fdb.884c.7684.3002.20.a\u0027)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.53e6.5916.ff0c.4e5f.4e0d.662f.4e0d.80fd.7528.6b27.51e0.91cc.5f97.ff0c.4f46.662f.4f9d.636e.4e0a.4e00.63d0.793a.4e4b.5206.6790.ff0c.6b27.51e0.91cc.5f97.662f.4e0d.80fd.5f88.597d.4fdd.6301.8bed.4e49.76f8.4f3c.6027.7684.3002.20.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q16. cosine距离的时候，如果两个向量的长度区别很大，但是方向类似，cosine的距离是小还是大？ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "回答：cosine只考虑“方向“是否一样"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u002763.6f.73.69.6e.65.53ea.8003.8651.201c.65b9.5411.201c.662f.5426.4e00.6837\u0027)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#先将所有的词向量都乘以一个常数alpha，然后将它们相加，之后再通过PCA提取他们的句向量\n",
        "\n",
        "def sentence_embedding(sentence, cut\u003dcut):\n",
        "    # weight \u003d alpah/(alpah + p) \n",
        "    # alpha is a parameter, 1e-3 ~ 1e-5\n",
        "    alpha \u003d 1e-4\n",
        "    \n",
        "    global frequence\n",
        "    \n",
        "    max_fre \u003d max(frequence.values())\n",
        "    \n",
        "    words \u003d cut(sentence).split()\n",
        "    \n",
        "    sentence_vec \u003d np.zeros_like(model.wv[\u0027测试\u0027])\n",
        "    \n",
        "    words \u003d [w for w in words if w in model]\n",
        "    \n",
        "    for w in words:\n",
        "        weight \u003d alpha / (alpha + frequence.get(w, max_fre))\n",
        "        sentence_vec +\u003d weight * model.wv[w]\n",
        "    \n",
        "    sentence_vec /\u003d len(words)\n",
        "    # Skip the PCA\n",
        "    return sentence_vec"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 63,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "from scipy.spatial.distance import cosine\n",
        "from functools import partial\n",
        "def get_corrlations(text, cut_fn):\n",
        "    if isinstance(text, list): text \u003d \u0027 \u0027.join(text)\n",
        "        \n",
        "    sub_sentences \u003d split_sentence(text)\n",
        "    sentence_vector \u003d sentence_embedding(text, cut_fn)\n",
        "        \n",
        "    correlations \u003d {}\n",
        "    \n",
        "    for sub_sentence in sub_sentences:\n",
        "        sub_sen_vec \u003d sentence_embedding(sub_sentence, cut_fn)\n",
        "        correlation \u003d cosine(sentence_vector, sub_sen_vec)\n",
        "        correlations[sub_sentence]\u003d correlation\n",
        "\n",
        "    return sorted(correlations.items(), key\u003dlambda x: x[1], reverse\u003dTrue)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 64,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "corrlations_with_global_cut \u003d partial(get_corrlations, cut_fn\u003dcut)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 65,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "def get_summarization_simple_by_sen_embedding(text, constraint\u003d200):\n",
        "    return get_summarization_simple(text, corrlations_with_global_cut, constraint)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 127,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\python36\\lib\\site-packages\\ipykernel_launcher.py:16: DeprecationWarning: Call to deprecated `__contains__` (Method will be removed in 4.0.0, use self.wv.__contains__() instead).\n",
            "  app.launch_new_instance()\n"
          ]
        }
      ],
      "source": [
        "corrlations \u003d get_corrlations(sharp_news, cut)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 128,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "\u0027虽然至今夏普智能手机在市场上无法排得上号，已经完全没落，并于 2013 年退出中国市场，但是今年 3 月份官方突然宣布回归中国，预示着很快就有夏普新机在中国登场了。那么，第一款夏普手机什么时候登陆中国呢？又会是怎么样的手机呢？\\r\\n近日，一款型号为 FS8016 的夏普神秘新机悄然出现在 GeekBench 的跑分库上。从其中相关信息了解到，这款机子并非旗舰定位，所搭载的是高通骁龙 660 处理器，配备有 4GB 的内存。骁龙 660 是高通今年最受瞩目的芯片之一，采用 14 纳米工艺，八个 Kryo 260 核心设计，集成 Adreno 512 GPU 和 X12 LTE 调制解调器。\\r\\n当前市面上只有一款机子采用了骁龙 660 处理器，那就是已经上市销售的 OPPO R11。骁龙 660 尽管并非旗舰芯片，但在多核新能上比去年骁龙 820 强，单核改进也很明显，所以放在今年仍可以让很多手机变成高端机。不过，由于 OPPO 与高通签署了排他性协议，可以独占两三个月时间。\\r\\n考虑到夏普既然开始测试新机了，说明只要等独占时期一过，夏普就能发布骁龙 660 新品了。按照之前被曝光的渲染图了解，夏普的新机核心竞争优势还是全面屏，因为从 2013 年推出全球首款全面屏手机 EDGEST 302SH 至今，夏普手机推出了多达 28 款的全面屏手机。\\r\\n在 5 月份的媒体沟通会上，惠普罗忠生表示：“我敢打赌，12 个月之后，在座的各位手机都会换掉。因为全面屏时代的到来，我们怀揣的手机都将成为传统手机。”\\r\\n\u0027"
            ]
          },
          "execution_count": 128,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sharp_news"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 129,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[(\u002728\u0027, 1.016034135594964),\n",
              " (\u00275\u0027, 1.0141534013673663),\n",
              " (\u0027Kryo\u0027, 0.9813904333859682),\n",
              " (\u002714\u0027, 0.9812515676021576),\n",
              " (\u0027X12\u0027, 0.944944329559803),\n",
              " (\u00273\u0027, 0.9362230375409126),\n",
              " (\u0027Adreno\u0027, 0.9205908104777336),\n",
              " (\u0027R11\u0027, 0.8780125975608826),\n",
              " (\u0027GeekBench\u0027, 0.8449447751045227),\n",
              " (\u0027512\u0027, 0.8348383456468582),\n",
              " (\u0027和\u0027, 0.8271111249923706),\n",
              " (\u0027260\u0027, 0.755718469619751),\n",
              " (\u00272013\u0027, 0.7396700978279114),\n",
              " (\u0027的跑分库上\u0027, 0.7396254241466522),\n",
              " (\u0027在\u0027, 0.7283863425254822),\n",
              " (\u0027FS8016\u0027, 0.722855269908905),\n",
              " (\u0027302SH\u0027, 0.7159470021724701),\n",
              " (\u0027新品了\u0027, 0.6967576444149017),\n",
              " (\u0027”\u0027, 0.6781797707080841),\n",
              " (\u0027660\u0027, 0.6705841720104218),\n",
              " (\u0027调制解调器\u0027, 0.6603162884712219),\n",
              " (\u0027820\u0027, 0.656488835811615),\n",
              " (\u0027EDGEST\u0027, 0.644380509853363),\n",
              " (\u0027月份的媒体沟通会上，惠普罗忠生表示：“我敢打赌，12\u0027, 0.6281294524669647),\n",
              " (\u0027不过，由于\u0027, 0.5738721489906311),\n",
              " (\u0027又会是怎么样的手机呢\u0027, 0.5254146158695221),\n",
              " (\u0027LTE\u0027, 0.4806997776031494),\n",
              " (\u0027与高通签署了排他性协议，可以独占两三个月时间\u0027, 0.46851474046707153),\n",
              " (\u0027个月之后，在座的各位手机都会换掉\u0027, 0.4561747908592224),\n",
              " (\u00274GB\u0027, 0.4518991708755493),\n",
              " (\u0027纳米工艺，八个\u0027, 0.4441490173339844),\n",
              " (\u0027GPU\u0027, 0.43543368577957153),\n",
              " (\u0027的夏普神秘新机悄然出现在\u0027, 0.4312923550605774),\n",
              " (\u0027年退出中国市场，但是今年\u0027, 0.42829495668411255),\n",
              " (\u0027核心设计，集成\u0027, 0.39412999153137207),\n",
              " (\u0027骁龙\u0027, 0.3783636689186096),\n",
              " (\u0027OPPO\u0027, 0.3694418668746948),\n",
              " (\u0027处理器，配备有\u0027, 0.36644911766052246),\n",
              " (\u0027月份官方突然宣布回归中国，预示着很快就有夏普新机在中国登场了\u0027, 0.3652356266975403),\n",
              " (\u0027的内存\u0027, 0.34972482919692993),\n",
              " (\u0027那么，第一款夏普手机什么时候登陆中国呢\u0027, 0.30330872535705566),\n",
              " (\u0027近日，一款型号为\u0027, 0.2991504669189453),\n",
              " (\u0027是高通今年最受瞩目的芯片之一，采用\u0027, 0.29484617710113525),\n",
              " (\u0027至今，夏普手机推出了多达\u0027, 0.28605204820632935),\n",
              " (\u0027年推出全球首款全面屏手机\u0027, 0.24424141645431519),\n",
              " (\u0027因为全面屏时代的到来，我们怀揣的手机都将成为传统手机\u0027, 0.24134373664855957),\n",
              " (\u0027虽然至今夏普智能手机在市场上无法排得上号，已经完全没落，并于\u0027, 0.24057024717330933),\n",
              " (\u0027强，单核改进也很明显，所以放在今年仍可以让很多手机变成高端机\u0027, 0.2193922996520996),\n",
              " (\u0027尽管并非旗舰芯片，但在多核新能上比去年骁龙\u0027, 0.19502663612365723),\n",
              " (\u0027当前市面上只有一款机子采用了骁龙\u0027, 0.18982809782028198),\n",
              " (\u0027款的全面屏手机\u0027, 0.18976688385009766),\n",
              " (\u0027从其中相关信息了解到，这款机子并非旗舰定位，所搭载的是高通骁龙\u0027, 0.15708070993423462),\n",
              " (\u0027按照之前被曝光的渲染图了解，夏普的新机核心竞争优势还是全面屏，因为从\u0027, 0.1554616093635559),\n",
              " (\u0027处理器，那就是已经上市销售的\u0027, 0.14474469423294067),\n",
              " (\u0027考虑到夏普既然开始测试新机了，说明只要等独占时期一过，夏普就能发布骁龙\u0027, 0.13174527883529663)]"
            ]
          },
          "execution_count": 129,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "corrlations"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 130,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "# 将列表元组转换为字典形式\n",
        "# corrlations \u003d dict(corrlations)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 131,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\python36\\lib\\site-packages\\ipykernel_launcher.py:16: DeprecationWarning: Call to deprecated `__contains__` (Method will be removed in 4.0.0, use self.wv.__contains__() instead).\n",
            "  app.launch_new_instance()\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "\u00272013。 年退出中国市场，但是今年。 3。 又会是怎么样的手机呢。 FS8016。 的夏普神秘新机悄然出现在。 GeekBench。 的跑分库上。 660。 4GB。 660。 14。 纳米工艺，八个。 Kryo。 260。 核心设计，集成。 Adreno。 512。 GPU。 和。 X12。 LTE。 调制解调器。 660。 R11。 660。 820。 不过，由于。 与高通签署了排他性协议，可以独占两三个月时间。 660。 新品了。 2013。 EDGEST。 302SH。 28。 在。 5。 月份的媒体沟通会上，惠普罗忠生表示：“我敢打赌，12。 个月之后，在座的各位手机都会换掉。 ”\u0027"
            ]
          },
          "execution_count": 131,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sharp_news \u003d pure_content.iloc[6][\u0027content\u0027]\n",
        "\u0027。 \u0027.join(get_summarization_simple_by_sen_embedding(sharp_news, constraint\u003d200))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 134,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\python36\\lib\\site-packages\\ipykernel_launcher.py:16: DeprecationWarning: Call to deprecated `__contains__` (Method will be removed in 4.0.0, use self.wv.__contains__() instead).\n",
            "  app.launch_new_instance()\n"
          ]
        }
      ],
      "source": [
        "corrlations \u003d get_corrlations(sharp_news, cut)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 143,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "corrlations_with_global_cut \u003d partial(get_corrlations, cut_fn\u003dcut)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 144,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "def split_sentence(sentence):\n",
        "    pattern \u003d re.compile(\u0027[。，,.]：\u0027)\n",
        "    split \u003d pattern.sub(\u0027 \u0027, sentence).split()  # split sentence\n",
        "    return split"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 145,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "def get_sequent_correlation(text):\n",
        "    sub_sentence \u003d split_sentence(text)\n",
        "    corrleations \u003d corrlations_with_global_cut(sub_sentence)\n",
        "    correlation_with_sequent \u003d [(sen, corrlations[sen]) for sen in sub_sentence]\n",
        "    return correlation_with_sequent"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 146,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\python36\\lib\\site-packages\\ipykernel_launcher.py:16: DeprecationWarning: Call to deprecated `__contains__` (Method will be removed in 4.0.0, use self.wv.__contains__() instead).\n",
            "  app.launch_new_instance()\n"
          ]
        },
        {
          "ename": "AttributeError",
          "evalue": "\u0027list\u0027 object has no attribute \u0027items\u0027",
          "output_type": "error",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[1;32m\u003cipython-input-146-1b5f0a80968e\u003e\u001b[0m in \u001b[0;36m\u003cmodule\u003e\u001b[1;34m\u001b[0m\n\u001b[1;32m----\u003e 1\u001b[1;33m \u001b[0msub_sentence\u001b[0m \u001b[1;33m\u003d\u001b[0m \u001b[0mget_sequent_correlation\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msharp_news\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0msub_sentence\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;32m\u003cipython-input-145-eb59d122b975\u003e\u001b[0m in \u001b[0;36mget_sequent_correlation\u001b[1;34m(text)\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mget_sequent_correlation\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m     \u001b[0msub_sentence\u001b[0m \u001b[1;33m\u003d\u001b[0m \u001b[0msplit_sentence\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----\u003e 3\u001b[1;33m     \u001b[0mcorrleations\u001b[0m \u001b[1;33m\u003d\u001b[0m \u001b[0mcorrlations_with_global_cut\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msub_sentence\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m     \u001b[0mcorrelation_with_sequent\u001b[0m \u001b[1;33m\u003d\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msen\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcorrlations\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0msen\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0msen\u001b[0m \u001b[1;32min\u001b[0m \u001b[0msub_sentence\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0mcorrelation_with_sequent\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;32m\u003cipython-input-141-9aea96e7ce9e\u003e\u001b[0m in \u001b[0;36mget_corrlations\u001b[1;34m(text, cut_fn)\u001b[0m\n\u001b[0;32m     12\u001b[0m         \u001b[0mcorrelations\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0msub_sentence\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u003d\u001b[0m \u001b[0mcorrelation\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     13\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---\u003e 14\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0msorted\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcorrlations\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[1;33m\u003d\u001b[0m\u001b[1;32mlambda\u001b[0m \u001b[0mx\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mx\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mreverse\u001b[0m\u001b[1;33m\u003d\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[1;31mAttributeError\u001b[0m: \u0027list\u0027 object has no attribute \u0027items\u0027"
          ]
        }
      ],
      "source": [
        "sub_sentence \u003d get_sequent_correlation(sharp_news)\n",
        "sub_sentence"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "#### Task15 基于Sentene Embedding 在pycharm里边完成句子摘要的完整过程"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q17. 找出几个句子进行plot， 解释其与全文的相关性的排序"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "请阐述："
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q18. 不同类型的文章，例如时政，财经，社会，故事，其相关性分布有何区别？ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "请阐述："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.63d0.793a.ff1a.53ef.4ee5.4ece.5176.5206.5e03.7684.5bc6.5ea6.ff0c.5206.5e03.53d8.5316.7b49.8d8b.52bf.8fdb.884c.89c2.5bdf.3002.a\u0027)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## More Added Feature"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "### KNN思想"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "\u003e 对于一个sub_sentence来说，它的重要性，取决于他本身的重要性和周围的句子(neighbors)的重要性的综合；"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "例如，当我们有一个列表是 [1, 1, 2, 3, 8, 1, 2]的时候，其中的 8 数值太大，这在我们的摘要中的表现为，该句子周围的句子都是不那么相关的，但是该句子单独的相关性很高。 那么，如果把这个句子摘录进来，就会导致“不通顺”. 我们可以采用KNN的思想，将这个列表进行重新计算，让它每一个元素的值，等于自己的值和周围的几个值的一个计算结果。 "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q19. 如何使用KNN思想进行“平滑”操作? 请代码实现"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "回答：\n\n我们会发现，句子的评分分布是这样起伏很大的尖锐曲线，这样抽取的句子会断断续续，显得很突兀，因此我们需要根据句子自身的重要性和周围句子的重要性，结合KNN算法使得结果更加平滑。\n\n`请补全代码：`\n```\nL \u003d []\ndef knn_smooth(): pass\n\n```"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.23.4e00.4e2a.53ef.4ee5.7684.73.6f.6c.75.74.69.6f.6e.662f.ff1a.a.a.4c.20.3d.20.5b.31.2c.20.32.2c.20.33.2c.20.34.2c.20.31.30.2c.20.31.2c.20.32.5d.a.77.69.6e.64.6f.77.20.3d.20.33.a.a.4c.20.3d.20.5b.6e.70.2e.6d.65.61.6e.28.4c.5b.69.2d.77.69.6e.64.6f.77.3a.69.2b.77.69.6e.64.6f.77.5d.29.20.66.6f.72.20.69.20.69.6e.20.72.61.6e.67.65.28.6c.65.6e.28.4c.29.29.5d.a.23.20.5f53.7136.ff0c.8fd9.91cc.7684.6e.70.2e.6d.65.61.6e.662f.4e00.79cd.6700.7b80.5355.7684.5e73.5747.5316.65b9.6cd5.ff0c.6211.4eec.8fd8.53ef.4ee5.8003.8651.5176.4ed6.7684.4f8b.5982.8ddd.79bb.52a0.6743.ff0c.7b49.65b9.6cd5.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "### Title 标题"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "\u003e 如果一个文章有标题，那么，其实这个标题已经包含了很多它的摘要信息， 在考虑语义建模的时候，就应该对标题更加重要的考虑；"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "如果该文本有标题，那么标题可以帮助我们很多。在之前，我们计算每个句子与文章整体的相似度是对每个子句与整体文章进行相似度距离计算，那么，我们这个时候，就可以把标题的embedding结果拿出来，那么每句话的相似度就是这句话与整体文章的相似度和标题的相似度的一个“结合”。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Q20. 请代码实现这一思路。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "回答：\n\n如果你文本存在标题，那么标题已经包含了其中的重要信息，摘要时应当把标题考虑进去。\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "#hint(\u0027a.74.69.74.6c.65.20.3d.20.27.73.6f.6d.65.20.77.6f.72.64.73.27.a.63.6f.6e.74.65.6e.74.20.3d.20.27.6d.6f.72.65.20.61.6e.64.20.6d.6f.72.65.20.77.6f.72.64.73.27.a.73.65.6e.74.65.6e.65.5f.76.65.63.5f.74.69.74.6c.65.20.3d.20.67.65.74.5f.73.65.6e.74.65.6e.63.65.5f.76.65.63.28.74.69.74.6c.65.29.a.73.65.6e.74.65.6e.65.5f.76.65.63.5f.63.6f.6e.74.65.6e.74.20.3d.20.67.65.74.5f.73.65.6e.74.65.6e.63.65.5f.76.65.63.28.63.6f.6e.74.65.6e.74.29.a.23.5bf9.4e8e.4e00.4e2a.5b50.53e5.20.73.75.62.5f.73.65.6e.5f.6e.2c.20.4ee5.524d.7684.73.69.6d.69.6c.61.72.69.74.79.662f.20.63.6f.73.69.6e.65.28.67.65.74.5f.73.65.6e.74.65.6e.65.5f.76.65.63.28.73.75.62.5f.73.65.6e.5f.6e.29.2c.20.73.65.6e.74.65.6e.65.5f.76.65.63.5f.63.6f.6e.74.65.6e.74.29.a.23.73b0.5728.53ef.4ee5.662f.a.70.20.3d.20.30.2e.35.a.73.65.6e.5f.76.65.63.20.3d.20.67.65.74.5f.73.65.6e.74.65.6e.65.5f.76.65.63.28.73.75.62.5f.73.65.6e.5f.6e.29.a.73.69.6d.69.6c.61.72.69.74.79.20.3d.20.70.20.2a.20.63.6f.73.69.6e.65.28.73.65.6e.5f.76.65.63.2c.20.73.65.6e.74.65.6e.65.5f.76.65.63.5f.74.69.74.6c.65.29.20.2b.20.28.31.20.2d.20.70.29.20.2a.20.63.6f.73.69.6e.65.28.73.65.6e.5f.76.65.63.2c.20.73.65.6e.74.65.6e.65.5f.76.65.63.5f.63.6f.6e.74.65.6e.74.29.a.23.20.5f53.7136.ff0c.8fd9.91cc.7684.70.4ee5.53ca.70.548c.63.6f.73.69.6e.65.7684.6784.5efa.90fd.662f.53ef.4ee5.53d8.5316.7684.3002.20.70.20.548c.20.31.2d.70.662f.7ebf.6027.5173.7cfb.ff0c.53ef.4ee5.662f.5176.4ed6.7684.5173.7cfb.3002.81ea.5df1.5b9a.5373.53ef.a\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "### Keywords 关键字"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "\u003e 关键字是很重要的，如果我们能够比较准确的提取出来关键词/字，然后对关键字/词包含的句子增加其权重；"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "对整篇文章提取关键字，该关键字可以通过TextRank或者tfidf或者gensim自带的包进行提取。 然后对包含了关键词的句子进行手动加权"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "#### Task 请代码实现\n\n在文本中，存在关键词的句子往往包含了比较重要的信息，当一个句子包含比关键词时，应适当增加此句子的评分。\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "### Position 位置信息"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "对于句子而言，如果其出现在段落开端，结尾，是否是重要的？ 请做实验证明之，并且代码进行改进。 "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "\u003e 对于一些文本，文章，例如一个故事的这种文章，那么他的textRank， sentence embedding 会发现，并不会出现很明显有些句子是重要的，有些句子不重要的，如果plot他的曲线的话，我们这个时候就要考虑他的位置，开头，结尾，增加一些权重；"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "#### Task 加入位置信息的考量\n\n在阅读理解时，开头、结尾、段首和段尾等几乎都起着总领全文、总结或承上启下的作用，在摘要时，必须也要把句子的位置考虑进去。\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "### Topic 主题信息"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "自学Gensim LDA的使用方法，对于文章获得其主题，然后依据主题对每个句子进行判断，每个句子是否和该主题相关。 \n参考网站： \n1. Google Search： Gensim LDA\n2. https://github.com/xiaoyichao/-python-gensim-LDA-/blob/master/topicmodel3.py\n\n【参考地址】https://blog.csdn.net/Yellow_python/article/details/83097994\n![gensim_LDA流程](https://img-blog.csdn.net/20181021090249778?watermark/2/text/aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1llbGxvd19weXRob24\u003d/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA\u003d\u003d/dissolve/70)\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "\u003e LDA主题模型，得到的主题分布，我们可以用每句话和的出来的这些主题进行相似度对比，我们不仅仅是是考虑他的整个的text， 我们还有考虑主题；\n\n根据LDA总结出来的话题，我们可以用分句与其对比，把话题也考虑到摘要中。\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "#### Task 合并以上功能，实现一个单独的函数，该函数接受一个长文本和字数限制，输出一个短文中"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## Questions 思考题："
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "1. 除了Sentence Embedding和TextRank, 还有什么方案？"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "2. 为什么这个课题不用Seq2Seq?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "3. 深度学习、机器学习与NLP之间的关系是什么？ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "4. 你还有什么其他的思考和心得？ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## 最后一步： 使用Flask、Bottle、Bootstrap变成一个网络应用并且部署在服务器上，这样别人就可以直接通过网址访问你的应用啦。"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}