Druid时序数据库升级流程

目前Druid集群版本为0.11.0，新版本0.12.1已支持Druid SQL和Redis，考虑到Druid新特性以及性能的提升，因此需要将Druid从0.11.0版本升级到0.12.1版本，下面将对Druid升级步骤做详细的介绍，升级时请严格按照此步骤进行升级，以免出现一些不可预知的问题。

1. Druid升级包

Druid官网下载druid-0.12.1-bin.tar.gz和mysql-metadata-storage-0.12.1.tar.gz

2. 配置Druid-0.12.1

• 解压druid-0.12.1-bin.tar.gz

  [work@druid]$ tar -zxvf druid-0.12.1-bin.tar.gz [work@druid]$ rm -rf druid-0.12.1-bin.tar.gz

• 解压mysql-metadata-storage-0.12.1.tar.gz

  [work@druid]$ tar -zxvf mysql-metadata-storage-0.12.1.tar.gz -C druid-0.12.1/extensions/ [work@druid]$ rm -rf mysql-metadata-storage-0.12.1.tar.gz

3. 配置common.runtime.properties

1[work@druid druid-0.11.0]$ cd conf/druid/_common
2[work@druid _common]$ vi common.runtime.properties
3
4
5# If you specify `druid.extensions.loadList=[]`, Druid won't load any extension from file system.
6# If you don't specify `druid.extensions.loadList`, Druid will load all the extensions under root extension directory.
7# More info: http://druid.io/docs/latest/operations/including-extensions.html
8druid.extensions.loadList=["druid-kafka-eight", "druid-hdfs-storage", "druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "mysql-metadata-storage"]
9
10# If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory
11# and uncomment the line below to point to your directory.
12#druid.extensions.hadoopDependenciesDir=/my/dir/hadoop-dependencies
13
14#
15# Logging
16#
17
18# Log all runtime properties on startup. Disable to avoid logging properties on startup:
19druid.startup.logging.logProperties=true
20
21#
22# Zookeeper
23#
24
25druid.zk.service.host=172.16.XXX.XXX:2181
26druid.zk.paths.base=/druid
27
28#
29# Metadata storage
30#
31
32# For Derby server on your Druid Coordinator (only viable in a cluster with a single Coordinator, no fail-over):
33#druid.metadata.storage.type=derby
34#druid.metadata.storage.connector.connectURI=jdbc:derby://localhost:1527/var/druid/metadata.db;create=true
35#druid.metadata.storage.connector.host=localhost
36#druid.metadata.storage.connector.port=1527
37
38# For MySQL:
39druid.metadata.storage.type=mysql
40druid.metadata.storage.connector.connectURI=jdbc:mysql://172.16.XXX.XXX:3306/druid
41druid.metadata.storage.connector.user=root
42druid.metadata.storage.connector.password=123456
43
44# For PostgreSQL:
45#druid.metadata.storage.type=postgresql
46#druid.metadata.storage.connector.connectURI=jdbc:postgresql://db.example.com:5432/druid
47#druid.metadata.storage.connector.user=...
48#druid.metadata.storage.connector.password=...
49
50#
51# Deep storage
52#
53
54# For local disk (only viable in a cluster if this is a network mount):
55#druid.storage.type=local
56#druid.storage.storageDirectory=var/druid/segments
57
58# For HDFS:
59druid.storage.type=hdfs
60druid.storage.storageDirectory=/druid/segments
61
62# For S3:
63#druid.storage.type=s3
64#druid.storage.bucket=your-bucket
65#druid.storage.baseKey=druid/segments
66#druid.s3.accessKey=...
67#druid.s3.secretKey=...
68
69#
70# Indexing service logs
71#
72
73# For local disk (only viable in a cluster if this is a network mount):
74#druid.indexer.logs.type=file
75#druid.indexer.logs.directory=var/druid/indexing-logs
76
77# For HDFS:
78druid.indexer.logs.type=hdfs
79druid.indexer.logs.directory=/druid/indexing-logs
80
81# For S3:
82#druid.indexer.logs.type=s3
83#druid.indexer.logs.s3Bucket=your-bucket
84#druid.indexer.logs.s3Prefix=druid/indexing-logs
85
86#
87# Service discovery
88#
89
90druid.selectors.indexing.serviceName=druid/overlord
91druid.selectors.coordinator.serviceName=druid/coordinator
92
93#
94# Monitoring
95#
96
97druid.monitoring.monitors=["io.druid.java.util.metrics.JvmMonitor"]
98druid.emitter=logging
99druid.emitter.logging.logLevel=info
100
101# Storage type of double columns
102# ommiting this will lead to index double as float at the storage layer
103
104druid.indexing.doubleStorage=double

4. 复制HDFS配置文件

1[work@druid _common]$ cp core-site.xml /alidata/server/druid-0.12.1/conf/druid/_common/
2[work@druid _common]$ cp hdfs-site.xml /alidata/server/druid-0.12.1/conf/druid/_common/
3[work@druid _common]$ cp mapred-site.xml /alidata/server/druid-0.12.1/conf/druid/_common/
4[work@druid _common]$ cp yarn-site.xml /alidata/server/druid-0.12.1/conf/druid/_common/

5.启用Druid SQL功能

1[work@druid broker]$ vi runtime.properties
2
3
4druid.service=druid/broker
5druid.host=172.16.XXX.XXX
6druid.port=8082
7
8# HTTP server threads
9druid.broker.http.numConnections=5
10druid.server.http.numThreads=9
11
12# Processing threads and buffers
13druid.processing.buffer.sizeBytes=256000000
14druid.processing.numThreads=2
15
16# Query cache (we use a small local cache)
17druid.broker.cache.useCache=true
18druid.broker.cache.populateCache=true
19druid.cache.type=local
20druid.cache.sizeInByte=2000000000
21
22# enable druid sql and http
23druid.sql.enable=true
24druid.sql.avatica.enable=true
25druid.sql.http.enable=true

备注：broker、overlord、coordinator、historical、middleManager等目录下的runtime.properties新增属性druid.host=ipAddress

6. 更新MiddleManager任务执行数capacity

1work@druid middleManager]$ vi runtime.properties
2
3
4druid.service=druid/middleManager
5druid.host=172.16.XXX.XXX
6druid.port=8091
7
8# Number of tasks per middleManager
9druid.worker.capacity=20
10
11# Task launch parameters
12druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
13druid.indexer.task.baseTaskDir=var/druid/task
14
15# HTTP server threads
16druid.server.http.numThreads=9
17
18# Processing threads and buffers on Peons
19druid.indexer.fork.property.druid.processing.buffer.sizeBytes=256000000
20druid.indexer.fork.property.druid.processing.numThreads=2
21
22# Hadoop indexing
23druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp
24druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.7.3"]

7. 升级Historical

[work@druid druid-0.12.1]$ nohup >> nohuphistorical.out java `cat conf/druid/historical/jvm.config | xargs` -cp conf/druid/_common:conf/druid/historical:lib/* io.druid.cli.Main server historical &

8. 升级Overlord

[work@druid druid-0.12.1]$ nohup >> logs/nohupoverlord.out java `cat conf/druid/overlord/jvm.config | xargs` -cp conf/druid/_common:conf/druid/overlord:lib/* io.druid.cli.Main server overlord &

9. 升级MiddleManager

• 禁止Overlor再向指定服务的MiddleManager分配任务

  http://<MiddleManager_IP:PORT>/druid/worker/v1/disable

• 查看指定MiddleManager任务列表

  http://<MiddleManager_IP:PORT>/druid/worker/v1/tasks

• 启动MiddleManager

  [work@druid druid-0.12.1]$ nohup >> logs/nohupmiddleManager.out java cat conf/druid/middleManager/jvm.config | xargs -cp conf/druid/_common:conf/druid/middleManager:lib/* io.druid.cli.Main server middleManager &

• 启用Overlord向指定MiddleManager分配任务

  http://<MiddleManager_IP:PORT>/druid/worker/v1/enable

10. 升级Broker

[work@druid druid-0.12.1]$ nohup >> nohupbroker.out java `cat conf/druid/broker/jvm.config | xargs` -cp conf/druid/_common:conf/druid/broker:lib/* io.druid.cli.Main server broker &

11. 升级Coordinator

[work@druid druid-0.12.1]$ nohup >> nohupcoordinator.out java `cat conf/druid/coordinator/jvm.config | xargs` -cp conf/druid/_common:conf/druid/coordinator:lib/* io.druid.cli.Main server coordinator &

至此，Druid就完成从0.11.0版本升级到0.12.1版本。