1. B站博人传评论数据爬取简介
今天想了半天不知道抓啥，去B站看跳舞的小姐姐，忽然看到了评论，那就抓取一下B站的评论数据，视频动画那么多，也不知道抓取哪个，选了一个博人传跟火影相关的，抓取看看。网址： https://www.bilibili.com/bangumi/media/md5978/?from=search&seid=16013388136765436883#short 在这个网页看到了18560条短评，数据量也不大，抓取看看，使用的还是scrapy。
2. B站博人传评论数据案例---获取链接
从开发者工具中你能轻易的得到如下链接，有链接之后就好办了，如何创建项目就不在啰嗦了，我们直接进入主题。 
我在代码中的parse函数中，设定了两个yield一个用来返回items 一个用来返回requests。 然后实现一个新的功能，每次访问切换UA，这个点我们需要使用到中间件技术。
1class BorenSpider(scrapy.Spider): 2 BASE_URL = "https://bangumi.bilibili.com/review/web_api/short/list?media_id=5978&folded=0&page_size=20&sort=0&cursor={}" 3 name = 'Boren' 4 allowed_domains = ['bangumi.bilibili.com'] 5 6 start_urls = [BASE_URL.format("76742479839522")] 7 8 def parse(self, response): 9 print(response.url) 10 resdata = json.loads(response.body_as_unicode()) 11 12 if resdata["code"] == 0: 13 # 获取最后一个数据 14 if len(resdata["result"]["list"]) > 0: 15 data = resdata["result"]["list"] 16 cursor = data[-1]["cursor"] 17 for one in data: 18 item = BorenzhuanItem() 19 20 item["author"] = one["author"]["uname"] 21 item["content"] = one["content"] 22 item["ctime"] = one["ctime"] 23 item["disliked"] = one["disliked"] 24 item["liked"] = one["liked"] 25 item["likes"] = one["likes"] 26 item["user_season"] = one["user_season"]["last_ep_index"] if "user_season" in one else "" 27 item["score"] = one["user_rating"]["score"] 28 yield item 29 30 yield scrapy.Request(self.BASE_URL.format(cursor),callback=self.parse) 31
3. B站博人传评论数据案例---实现随机UA
第一步， 在settings文件中添加一些UserAgent,我从互联网找了一些
1USER_AGENT_LIST=[ 2 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1", 3 "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11", 4 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6", 5 "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6", 6 "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1", 7 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5", 8 "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5", 9 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", 10 "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", 11 "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)", 12 "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", 13 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", 14 "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)", 15 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", 16 "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", 17 "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3", 18 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24", 19 "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24" 20] 21
第二步，在settings文件中设置 “DOWNLOADER_MIDDLEWARES”
1# Enable or disable downloader middlewares 2# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html 3DOWNLOADER_MIDDLEWARES = { 4 #'borenzhuan.middlewares.BorenzhuanDownloaderMiddleware': 543, 5 'borenzhuan.middlewares.RandomUserAgentMiddleware': 400, 6} 7
第三步，在 middlewares.py 文件中导入 settings模块中的 USER_AGENT_LIST 方法
1from borenzhuan.settings import USER_AGENT_LIST # 导入中间件 2import random 3 4class RandomUserAgentMiddleware(object): 5 def process_request(self, request, spider): 6 rand_use = random.choice(USER_AGENT_LIST) 7 if rand_use: 8 request.headers.setdefault('User-Agent', rand_use) 9
好了，随机的UA已经实现，你可以在parse函数中编写如下代码进行测试
print(response.request.headers)
4. B站博人传评论数据----完善item
这个操作相对简单，这些数据就是我们要保存的数据了。！
1 author = scrapy.Field() 2 content = scrapy.Field() 3 ctime = scrapy.Field() 4 disliked = scrapy.Field() 5 liked = scrapy.Field() 6 likes = scrapy.Field() 7 score = scrapy.Field() 8 user_season = scrapy.Field()
5. B站博人传评论数据案例---提高爬取速度
在settings.py中设置如下参数：
1# Configure maximum concurrent requests performed by Scrapy (default: 16) 2CONCURRENT_REQUESTS = 32 3# Configure a delay for requests for the same website (default: 0) 4# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay 5# See also autothrottle settings and docs 6DOWNLOAD_DELAY = 1 7# The download delay setting will honor only one of: 8CONCURRENT_REQUESTS_PER_DOMAIN = 16 9CONCURRENT_REQUESTS_PER_IP = 16 10# Disable cookies (enabled by default) 11COOKIES_ENABLED = False
解释说明
一、降低下载延迟
DOWNLOAD_DELAY = 0
将下载延迟设为0，这时需要相应的防ban措施，一般使用user agent轮转，构建user agent池，轮流选择其中之一来作为user agent。
二、多线程
CONCURRENT_REQUESTS = 32 CONCURRENT_REQUESTS_PER_DOMAIN = 16 CONCURRENT_REQUESTS_PER_IP = 16
scrapy网络请求是基于Twisted，而Twisted默认支持多线程，而且scrapy默认也是通过多线程请求的，并且支持多核CPU的并发，我们通过一些设置提高scrapy的并发数可以提高爬取速度。
三、禁用cookies
COOKIES_ENABLED = False
6. B站博人传评论数据案例---保存数据
最后在pipelines.py 文件中，编写保存代码即可
1import os 2import csv 3 4class BorenzhuanPipeline(object): 5 6 7 def __init__(self): 8 store_file = os.path.dirname(__file__)+'/spiders/bore.csv' 9 self.file = open(store_file,"a+",newline="",encoding="utf-8") 10 self.writer = csv.writer(self.file) 11 12 def process_item(self, item, spider): 13 try: 14 15 self.writer.writerow(( 16 item["author"], 17 item["content"], 18 item["ctime"], 19 item["disliked"], 20 item["liked"], 21 item["likes"], 22 item["score"], 23 item["user_season"] 24 )) 25 26 except Exception as e: 27 print(e.args) 28 29 def close_spider(self, spider): 30 self.file.close() 31
运行代码之后，发现过了一会报错了 
去看了一眼，原来是数据爬取完毕~！！！
<div align="center"> <img src="https://user-gold-cdn.xitu.io/2019/2/13/168e437eb9b832e0?w=94&h=108&f=gif&s=9304" /> </div>