Java爬虫——常用的maven依赖

java实现爬虫常用的第三方包：

• httpclient，for http
• jsoup，for dom
• rhino，for js
• jackson，for json

pom.xml摘录

1<dependencies>
2
3        <!-- simulate web browser -->
4        <dependency>
5            <groupId>org.apache.httpcomponents</groupId>
6            <artifactId>httpclient</artifactId>
7            <version>4.5.7</version>
8        </dependency>
9
10        <!-- parse DOM -->
11        <dependency>
12            <groupId>org.jsoup</groupId>
13            <artifactId>jsoup</artifactId>
14            <version>1.11.3</version>
15        </dependency>
16
17        <!-- jackson -->
18        <dependency>
19            <groupId>com.fasterxml.jackson.core</groupId>
20            <artifactId>jackson-databind</artifactId>
21            <version>2.9.8</version>
22        </dependency>
23
24        <!-- parse javascript -->
25        <dependency>
26            <groupId>org.mozilla</groupId>
27            <artifactId>rhino</artifactId>
28            <version>1.7.10</version>
29        </dependency>
30
31        <!-- simulate client action -->
32        <dependency>
33            <groupId>net.sourceforge.htmlunit</groupId>
34            <artifactId>htmlunit</artifactId>
35            <version>2.33</version>
36        </dependency>
37
38        <!-- upgrade junit to junit4 -->
39        <dependency>
40            <groupId>junit</groupId>
41            <artifactId>junit</artifactId>
42            <version>4.12<!-- default is v3.8.1 --></version>
43            <scope>test</scope>
44        </dependency>
45
46        <!-- log -->
47        <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-api -->
48        <dependency>
49            <groupId>org.slf4j</groupId>
50            <artifactId>slf4j-api</artifactId>
51            <version>1.7.25</version>
52        </dependency>
53        <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12 -->
54        <dependency>
55            <groupId>org.slf4j</groupId>
56            <artifactId>slf4j-log4j12</artifactId>
57            <version>1.7.25</version>
58            <!-- <scope>test</scope> -->
59        </dependency>
60
61    </dependencies>

启用log4j基本配置，在main方法中加入语句：

1public static void main(String[] args) {
2
3    //启用log4j基本配置
4    //不想去写配置文件，可以用Java基本配置
5    BasicConfigurator.configure();
6    //...
7}