都9102年，还用properties配置文件吗？非也非也。
Azkaban flow 2.0使用yaml进行作业配置：
上传的文件夹中，可以包含多个project 的yml配置文件。
Flow YAML File
关于Flow的文件有如下定义:
- 每个Flow对应一个yml文件.
- Flow的名称以yml文件的名称， 如: my-flow-name.flow.
- 包含所有的执行DAG节点.
- 每个执行节点可以是job或者是flow.
- 每个执行节点可以有一下属性：name, type, config, dependsOn 和节点.
- Node dependencies are specified by listing the parent nodes in dependsOn list.
- 包含其他的flow配置、.
- 所有的properties配置文件，将会迁移到YAML文件里面
Demo：acceptance-test.flow
1--- 2config: 3 user.to.proxy: azktest 4 param.hadoopOutData: /tmp/wordcounthadoopout 5 param.inData: /tmp/wordcountpigin 6 param.outData: /tmp/wordcountpigout 7 8# This section defines the list of jobs 9# A node can be a job or a flow 10# In this example, all nodes are jobs 11nodes: 12 # Job definition 13 # The job definition is like a YAMLified version of properties file 14 # with one major difference. All custom properties are now clubbed together 15 # in a config section in the definition. 16 # The first line describes the name of the job 17 - name: AZTest 18 type: noop 19 # The dependsOn section contains the list of parent nodes the current 20 # node depends on 21 dependsOn: 22 - hadoopWC1 23 - NoOpTest1 24 - hive2 25 - java1 26 - jobCommand2 27 28 - name: pigWordCount1 29 type: pig 30 # The config section contains custom arguments or parameters which are 31 # required by the job 32 config: 33 pig.script: src/main/pig/wordCountText.pig 34 35 - name: hadoopWC1 36 type: hadoopJava 37 dependsOn: 38 - pigWordCount1 39 config: 40 classpath: ./* 41 force.output.overwrite: true 42 input.path: ${param.inData} 43 job.class: com.linkedin.wordcount.WordCount 44 main.args: ${param.inData} ${param.hadoopOutData} 45 output.path: ${param.hadoopOutData} 46 47 - name: hive1 48 type: hive 49 config: 50 hive.script: src/main/hive/showdb.q 51 52 - name: NoOpTest1 53 type: noop 54 55 - name: hive2 56 type: hive 57 dependsOn: 58 - hive1 59 config: 60 hive.script: src/main/hive/showTables.sql 61 62 - name: java1 63 type: javaprocess 64 config: 65 Xms: 96M 66 java.class: com.linkedin.foo.HelloJavaProcessJob 67 68 - name: jobCommand1 69 type: command 70 config: 71 command: echo "hello world from job_command_1" 72 73 - name: jobCommand2 74 type: command 75 dependsOn: 76 - jobCommand1 77 config: 78 command: echo "hello world from job_command_2"
然后打包的zip文件的架构如下：
1project_root 2├── sample_project.project 3├── flow1.flow 4├── flow2.flow 5├── ... 6├── flown.flow 7├── lib 8│ ├── ... 9│ └── paranamer-2.4.1.jar 10└── src 11 └── main 12 ├── hive 13 │ └── query.q 14 └── pig 15 └── pig1.pig
在此之上，还可以做条件flow：
本人了解的Azkaban的传参方式，目前的解决方案有2种。
-
改源码，提供想要的EL表达式，比如${yesterday-ymd} 这种 (比较推荐，但是稍微麻烦)
-
使用 python props.py > $JOB_OUTPUT_PROP_FILE 方式。
将一个JSON 数据，输出到一个 $JOB_OUTPUT_PROP_FILE 环境变量中，这个环境变量，只能将参数传递一层依赖。
比如 init_step里面是>JOB_OUTPUT_PROP_FILE，那么依赖 init_step -> second_step -> last_step 中，second_step可以读取init_step中的环境变量。last_step读取不了。