storm客户端提交topology失败：
1java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException 2 at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141) 3 at backtype.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:176) 4 at backtype.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:158) 5 at cn.com.tiza.dataquality.service.service.JobService.start(JobService.java:93) 6 at cn.com.tiza.dataquality.service.service.JobService.submit(JobService.java:149) 7 at cn.com.tiza.dataquality.service.resource.JobResource.execute(JobResource.java:33) 8 at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source) 9 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 10 at java.lang.reflect.Method.invoke(Method.java:498) 11 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) 12 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) 13 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) 14 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) 15 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) 16 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) 17 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) 18 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) 19 at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
nimbus.log:
12017-11-17T16:12:27.038+0800 b.s.d.nimbus [WARN] Topology submission exception. (topology name='dq23') #<IllegalArgumentException java.lang.IllegalArgumentException: storm-local/nimbus/inb 2ox/stormjar-bc004b80-5f6e-4b94-9505-911511e5cc1f.jar to copy to storm-local/nimbus/stormdist/dq23-94-1510906347 does not exist!> 32017-11-17T16:12:27.038+0800 o.a.t.s.TNonblockingServer [ERROR] Unexpected exception while invoking! 4java.lang.IllegalArgumentException: storm-local/nimbus/inbox/stormjar-bc004b80-5f6e-4b94-9505-911511e5cc1f.jar to copy to storm-local/nimbus/stormdist/dq23-94-1510906347 does not exist! 5 at backtype.storm.daemon.nimbus$fn__4364.invoke(nimbus.clj:1173) ~[storm-core-0.9.7.jar:0.9.7] 6 at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na] 7 at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307) ~[storm-core-0.9.7.jar:0.9.7] 8 at backtype.storm.daemon.nimbus$fn__4261$exec_fn__1104__auto__$reify__4274.submitTopologyWithOpts(nimbus.clj:953) ~[storm-core-0.9.7.jar:0.9.7] 9 at backtype.storm.daemon.nimbus$fn__4261$exec_fn__1104__auto__$reify__4274.submitTopology(nimbus.clj:966) ~[storm-core-0.9.7.jar:0.9.7] 10 at backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240) ~[storm-core-0.9.7.jar:0.9.7] 11 at backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228) ~[storm-core-0.9.7.jar:0.9.7] 12 at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32) ~[storm-core-0.9.7.jar:0.9.7] 13 at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34) ~[storm-core-0.9.7.jar:0.9.7] 14 at org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632) ~[storm-core-0.9.7.jar:0.9.7] 15 at org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201) [storm-core-0.9.7.jar:0.9.7] 16 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144] 17 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144] 18 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
经过分析nimbus有个clean-inbox的机制来定时清理inbox中的jar文件，并有两个配置项来设置定时策略：
1/** 2 * How often nimbus should wake the cleanup thread to clean the inbox. 3 * @see NIMBUS_INBOX_JAR_EXPIRATION_SECS 4 */ 5public static final String NIMBUS_CLEANUP_INBOX_FREQ_SECS = "nimbus.cleanup.inbox.freq.secs"; 6 7/** 8 * The length of time a jar file lives in the inbox before being deleted by the cleanup thread. 9 * 10 * Probably keep this value greater than or equal to NIMBUS_CLEANUP_INBOX_JAR_EXPIRATION_SECS. 11 * Note that the time it takes to delete an inbox jar file is going to be somewhat more than 12 * NIMBUS_CLEANUP_INBOX_JAR_EXPIRATION_SECS (depending on how often NIMBUS_CLEANUP_FREQ_SECS 13 * is set to). 14 * @see NIMBUS_CLEANUP_FREQ_SECS 15 */ 16public static final String NIMBUS_INBOX_JAR_EXPIRATION_SECS = "nimbus.inbox.jar.expiration.secs";
NIMBUS_CLEANUP_INBOX_FREQ_SECS： 表示nimbus多久唤醒一次清理线程去进行清理；
NIMBUS_INBOX_JAR_EXPIRATION_SECS：表示jar文件在inbox中存活的时长，在清理线程清理之前如果到期了就会被清理
另一方面，通过storm-core提供的StormSubmitter.submitTopology的方法进行提交任务时，上传jar包的逻辑如下：
1 private static String submittedJar = null; 2 3 private static void submitJar(Map conf, ProgressListener listener) { 4 if(submittedJar==null) { 5 LOG.info("Jar not uploaded to master yet. Submitting jar..."); 6 String localJar = System.getProperty("storm.jar"); 7 submittedJar = submitJar(conf, localJar, listener); 8 } else { 9 LOG.info("Jar already uploaded to master. Not submitting jar."); 10 } 11 }
只要客户端进程不停，jar包就只上传一次。
所以等一个小时后，jar会被清除，重新提交任务就找不到inbox中的jar文件。