大家好，我是皮皮。
一、前言
还是昨天的那个网络爬虫问题，大佬们，帮忙看看这个网络爬虫代码怎么修改？那个粉丝说自己不熟悉pandas，用pandas做的爬虫，虽然简洁，但是自己不习惯，想要在他自己的代码基础上进行修改，获取数据的代码已经写好了，就差存储到csv中去了。
他的原始代码如下：
1import requests 2from lxml import etree 3import csv 4import time 5import pandas as pd 6 7 8def gdpData(page): 9 url = f'https://www.hongheiku.com/category/gdjsgdp/page/{page}' 10 headers ={'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'} 11 resp = requests.get(url,headers = headers) 12# print(resp.text) 13 data(resp.text) 14file = open('data.csv',mode='a',encoding='utf-8',newline='') 15csv_write=csv.DictWriter(file,fieldnames=['排名','地区','GDP','年份']) 16csv_write.writeheader() 17def data(text): 18 e = etree.HTML(text) 19 lst = e.xpath('//*[@id="tablepress-48"]/tbody/tr[@class="even"]') 20 for l in lst: 21 no = l.xpath('./td[1]/center/span/text()') 22 name = l.xpath('./td[2]/a/center/text()') 23 team = l.xpath('./td[3]/center/text()') 24 year = l.xpath('./td[4]/center/text()') 25 data_dict = { 26 '排名':no, 27 '地区':name, 28 'GDP':team, 29 '年份':year 30 } 31 print(f'排名：{no} 地区:{name} GDP:{team} 年份:{year} ') 32 csv_write.writerow(data_dict) 33file.close() 34url = 'https://www.hongheiku.com/category/gdjsgdp' 35headers ={'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'} 36resp = requests.get(url,headers = headers) 37# print(resp.text) 38data(resp.text) 39e = etree.HTML(resp.text) 40#//*[@id="tablepress-48"]/tbody/tr[192]/td[3]/center 41count = e.xpath('//div[@class="pagination pagination-multi"][last()]/ul/li[last()]/span/text()')[0].split(' ')[1] 42for index in range(int(count) - 1): 43 gdpData(index + 2)
二、实现过程
这里粉丝给了一瓶冰红茶的费用，一个热心市民给了一份代码，在他的代码基础上进行修改的，代码如下：
1import requests 2from lxml import etree 3import csv 4import time 5import pandas as pd 6 7 8def gdpData(page): 9 url = f'https://www.hongheiku.com/category/gdjsgdp/page/{page}' 10 headers ={'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'} 11 resp = requests.get(url,headers = headers) 12# print(resp.text) 13 data(resp.text) 14 15def data(text): 16 file = open('data.csv', mode='a', encoding='utf-8', newline='') 17 csv_write = csv.DictWriter(file, fieldnames=['排名', '地区', 'GDP', '年份']) 18 csv_write.writeheader() 19 e = etree.HTML(text) 20 lst = e.xpath('//*[@id="tablepress-48"]/tbody/tr[@class="even"]') 21 for l in lst: 22 no = ''.join(l.xpath('./td[1]/center/span/text()')) 23 name = ''.join(l.xpath('./td[2]/a/center/text()')[0]) 24 team = ''.join(l.xpath('./td[3]/center/text()')) 25 year = ''.join(l.xpath('./td[4]/center/text()')) 26 data_dict = { 27 '排名':no, 28 '地区':name, 29 'GDP':team, 30 '年份':year 31 } 32 print(f'排名：{no} 地区:{name} GDP:{team} 年份:{year} ') 33 csv_write.writerow(data_dict) 34 file.close() 35 36url = 'https://www.hongheiku.com/category/gdjsgdp' 37headers ={'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'} 38resp = requests.get(url,headers = headers) 39# print(resp.text) 40data(resp.text) 41 42e = etree.HTML(resp.text) 43#//*[@id="tablepress-48"]/tbody/tr[192]/td[3]/center 44count = e.xpath('//div[@class="pagination pagination-multi"][last()]/ul/li[last()]/span/text()')[0].split(' ')[1] 45for index in range(int(count) - 1): 46 gdpData(index + 2)
代码运行之后，数据就存储到csv中去了。
顺利地解决了粉丝的问题！
三、总结
大家好，我是皮皮。这篇文章主要盘点了一个Python网络爬虫后数据存储的问题，文中针对该问题，给出了具体的解析和代码实现，帮助粉丝顺利解决了问题。
最后感谢粉丝【蓝桉】提问，感谢【热心市民】给出的思路和代码解析，感谢【eric】等人参与学习交流。