部分商业网站对爬虫程序限制较多，在数据采集的过程中对爬虫请求进行了多种验证，导致爬虫程序需要深入分析目标网站的反爬策略，定期更新和维护爬虫程序，增加了研发的时间和投入成本。这种情况下，使用无头浏览器例如Selenium，模拟用户的请求进行数据采集是更加方便快捷的方式。同时为了避免目标网站出现IP限制，配合爬虫代理，实现每次请求自动切换IP，能够保证长期稳定的数据采集。以python的demo为例：
1from selenium import webdriver 2 import string 3 import zipfile 4 # 代理服务器(产品官网 www.16yun.cn) 5 proxyHost = "t.16yun.cn" 6 proxyPort = "31111" 7 # 代理验证信息 8 proxyUser = "username" 9 proxyPass = "password" 10 def create_proxy_auth_extension(proxy_host, proxy_port, 11 proxy_username, proxy_password, 12 scheme='http', plugin_path=None): 13 if plugin_path is None: 14 plugin_path = r'D:/{}_{}@t.16yun.zip'.format(proxy_username, proxy_password) 15 manifest_json = """ 16 { 17 "version": "1.0.0", 18 "manifest_version": 2, 19 "name": "16YUN Proxy", 20 "permissions": [ 21 "proxy", 22 "tabs", 23 "unlimitedStorage", 24 "storage", 25 "", 26 "webRequest", 27 "webRequestBlocking" 28 ], 29 "background": { 30 "scripts": ["background.js"] 31 }, 32 "minimum_chrome_version":"22.0.0" 33 } 34 """ 35 background_js = string.Template( 36 """ 37 var config = { 38 mode: "fixed_servers", 39 rules: { 40 singleProxy: { 41 scheme: "${scheme}", 42 host: "${host}", 43 port: parseInt(${port}) 44 }, 45 bypassList: ["foobar.com"] 46 } 47 }; 48 chrome.proxy.settings.set({value: config, scope: "regular"}, function() {}); 49 function callbackFn(details) { 50 return { 51 authCredentials: { 52 username: "${username}", 53 password: "${password}" 54 } 55 }; 56 } 57 chrome.webRequest.onAuthRequired.addListener( 58 callbackFn, 59 {urls: [""]}, 60 ['blocking'] 61 ); 62 """ 63 ).substitute( 64 host=proxy_host, 65 port=proxy_port, 66 username=proxy_username, 67 password=proxy_password, 68 scheme=scheme, 69 ) 70 with zipfile.ZipFile(plugin_path, 'w') as zp: 71 zp.writestr("manifest.json", manifest_json) 72 zp.writestr("background.js", background_js) 73 return plugin_path 74 proxy_auth_plugin_path = create_proxy_auth_extension( 75 proxy_host=proxyHost, 76 proxy_port=proxyPort, 77 proxy_username=proxyUser, 78 proxy_password=proxyPass) 79 option = webdriver.ChromeOptions() 80 option.add_argument("--start-maximized") 81 # 如报错 chrome-extensions 82 # option.add_argument("--disable-extensions") 83 option.add_extension(proxy_auth_plugin_path) 84 # 关闭webdriver的一些标志 85 # option.add_experimental_option('excludeSwitches', ['enable-automation']) 86 driver = webdriver.Chrome(chrome_options=option) 87 # 修改webdriver get属性 88 # script = ''' 89 # Object.defineProperty(navigator, 'webdriver', { 90 # get: () => undefined 91 # }) 92 # ''' 93 # driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {"source": script}) 94 driver.get("http://httpbin.org/ip")
要注意必须保证plugin_path参数下的文件存放目录是存在的，同时程序拥有该目录的读写权限，否则浏览器会出现代理认证信息读取失败的情况，就会强制弹出认证窗口，要求输入代理用户名和密码，出现程序运行中断的情况。