ActiveMQ producer 流量控制

http://activemq.apache.org/producer-flow-control.html

翻译：

流量控制是指：如果broker检测到destination的内存限制、temp文件限制、file store限制被超过了，就会减慢消息的流动。producer会被阻塞直到有可用资源，或者收到一个JMSException：这些行为是可以配置的。

值得一提的是：默认的<systemUsage>设置会导致producer阻塞，当达到memoryLimit或<systemUsage>的限制值时，这种阻塞行为有时被误认为是一个"挂起的producer"，而事实上producer只是在等待可用空间。

同步发送的消息会自动使用按producer的流量控制；通常会应用到同步发送持久化消息，除非你开启useAsyncSend标志。

使用异步发送的producer（通常讲，发送非持久化消息的producer）不需要等待来自broker的确认；所以，当达到memory limit时，你不会被通知到。如果你想知道broker的限制值达到了，你需要配置 ProducerWindowSize 连接参数，然后异步的消息也能按producer控制流量了。

ActiveMQConnectionFactory connctionFactory = ... connctionFactory.setProducerWindowSize(1024000);

 ProducerWindowSize 是producer发送的最大字节数，在等待来自broker的消息确认前。 如果你在发送非持久化消息（默认异步发送），并且希望知道queue或topic的memory limit是否达到了。那么你需要设置connection factory 为 'alwaysSyncSend'。但是，这会降低速度，它能保证你的producer马上知道内存问题。 如果你喜欢，你可以关掉指定jms queue和topic的流量控制，例如：

1<destinationPolicy>  
2    <policyMap>    
3        <policyEntries>      
4            <policyEntry topic="FOO.>" producerFlowControl="false"/> 
5            </policyEntries>  
6    </policyMap>
7</destinationPolicy>

注意，在ActiveMQ 5.x中引入了新的file cursor，非持久化消息会被刷到临时文件存储中来减少内存使用量。所以，你会发现queue的memoryLimit永远达不到，因为file cursor花不了多少内存，如果你真的要把所有非持久化消息保存在内存中，并且当memoryLimit达到时停止producer，你应该配置<vmQueueCursor>。

1<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">      
2    <pendingQueuePolicy>    
3        <vmQueueCursor/>  
4    </pendingQueuePolicy>
5</policyEntry>

上面的片段能保证，所有的消息保存在内存中，并且每一个队列只有1Mb的限制。

How Producer Flow Control works

如果你在发送持久化消息，broker会发送一个ProducerAck消息给producer，它告知producer前一个发送窗口已经被处理了，所以producer现在可以发送下一个窗口。这和consumer的消息确认很像。当没有空间可用时，调用send()操作会无限阻塞，另一种方法是在客户端抛出异常。通过设置sendFailIfNoSpace为true，broker会导致send()抛出javax.jms.ResourceAllocationException，异常会传播到客户端。下面是配置示例：

1<systemUsage> 
2    <systemUsage sendFailIfNoSpace="true">   
3        <memoryUsage>     
4            <memoryUsage limit="20 mb"/>   
5        </memoryUsage> 
6    </systemUsage>
7</systemUsage>

 这种做法的好处是，客户端会捕获一个javax.jms.ResourceAllocationException异常，等一会然后重试send()，而不再是无限地等待。

从5.3.1开始，加入了sendFailIfNoSpaceAfterTimeout属性，如果broker在配置的时间内仍然没有空余空间，此时send()才会失败，并且把异常传递到客户端，下面是配置：

1<systemUsage> 
2    <systemUsage sendFailIfNoSpaceAfterTimeout="3000">   
3        <memoryUsage>     
4            <memoryUsage limit="20 mb"/>   
5        </memoryUsage> 
6    </systemUsage>
7</systemUsage>

关闭流量控制

通常的需求是关闭流量控制，这样消息分发可以一直进行直到磁盘空间被 pending messages 耗尽。

通过配置<systemUsage>元素的某些属性，你可以降低producer的速率。

1<systemUsage>  
2    <systemUsage>    
3        <memoryUsage>      
4            <memoryUsage limit="64 mb" />    
5        </memoryUsage>    
6        <storeUsage>      
7            <storeUsage limit="100 gb" />    
8        </storeUsage>    
9        <tempUsage>      
10            <tempUsage limit="10 gb" />    
11        </tempUsage>  
12    </systemUsage>
13</systemUsage>

<memoryUsage>对应NON_PERSISTENT消息的内存容量，<storeUsage> 对应PERSITENT消息的磁盘容量，<tempUsage>对应临时文件的磁盘容量。

结合代码分析：

client-side： org.apache.activemq.ActiveMQMessageProducer.send

broker-side：从 org.apache.activemq.broker.region.Queue.send 开始

1//org.apache.activemq.broker.region.Queue
2public void send(final ProducerBrokerExchange producerExchange, final Message message) throws Exception {
3    final ConnectionContext context = producerExchange.getConnectionContext();
4    // There is delay between the client sending it and it arriving at the
5    // destination.. it may have expired.
6    message.setRegionDestination(this);
7    ProducerState state = producerExchange.getProducerState();
8    if (state == null) {
9        LOG.warn("Send failed for: {}, missing producer state for: {}", message, producerExchange);
10        throw new JMSException("Cannot send message to " + getActiveMQDestination() + " with invalid (null) producer state");
11    }
12    final ProducerInfo producerInfo = producerExchange.getProducerState().getInfo();
13    //是否发送ProducerAck
14    final boolean sendProducerAck = !message.isResponseRequired() && producerInfo.getWindowSize() > 0
15            && !context.isInRecoveryMode();
16    if (message.isExpired()) {
17        // message not stored - or added to stats yet - so check here
18        broker.getRoot().messageExpired(context, message, null);
19        if (sendProducerAck) {
20            ProducerAck ack = new ProducerAck(producerInfo.getProducerId(), message.getSize());
21            context.getConnection().dispatchAsync(ack);
22        }
23        return;
24    }
25    if (memoryUsage.isFull()) { //如果内存耗尽
26        // 尽管这里有大段代码，但是调试没进这儿
27    }
28    doMessageSend(producerExchange, message);
29    if (sendProducerAck) {
30        ProducerAck ack = new ProducerAck(producerInfo.getProducerId(), message.getSize());
31        context.getConnection().dispatchAsync(ack);
32    }
33}

把消息放进 broker 的 pendingList 之前，会检查可用空间：

1//org.apache.activemq.broker.region.Queue
2void doMessageSend(final ProducerBrokerExchange producerExchange, final Message message) 
3    throws IOException,    Exception {
4    final ConnectionContext context = producerExchange.getConnectionContext();
5    ListenableFuture<Object> result = null;
6    boolean needsOrderingWithTransactions = context.isInTransaction();
7
8    producerExchange.incrementSend();
9    //检查使用空间
10    checkUsage(context, producerExchange, message);
11    sendLock.lockInterruptibly();
12    try {
13        if (store != null && message.isPersistent()) {
14            try {
15                message.getMessageId().setBrokerSequenceId(getDestinationSequenceId());
16                if (messages.isCacheEnabled()) {
17                    result = store.asyncAddQueueMessage(context, message, isOptimizeStorage());
18                    result.addListener(new PendingMarshalUsageTracker(message));
19                } else {
20                    store.addMessage(context, message);
21                }
22                if (isReduceMemoryFootprint()) {
23                    message.clearMarshalledState();
24                }
25            } catch (Exception e) {
26                // we may have a store in inconsistent state, so reset the cursor
27                // before restarting normal broker operations
28                resetNeeded = true;
29                throw e;
30            }
31        }
32        // did a transaction commit beat us to the index?
33        synchronized (orderIndexUpdates) {
34            needsOrderingWithTransactions |= !orderIndexUpdates.isEmpty();
35        }
36        if (needsOrderingWithTransactions ) {
37            // If this is a transacted message.. increase the usage now so that
38            // a big TX does not blow up
39            // our memory. This increment is decremented once the tx finishes..
40            message.incrementReferenceCount();
41
42            registerSendSync(message, context);
43        } else {
44            // Add to the pending list, this takes care of incrementing the
45            // usage manager.
46            sendMessage(message);
47        }
48    } finally {
49        sendLock.unlock();
50    }
51    if (!needsOrderingWithTransactions) {
52        messageSent(context, message);
53    }
54    if (result != null && message.isResponseRequired() && !result.isCancelled()) {
55        try {
56            result.get();
57        } catch (CancellationException e) {
58            // ignore - the task has been cancelled if the message
59            // has already been deleted
60        }
61    }
62}

对持久化消息和非持久化消息分类检查：

1// org.apache.activemq.broker.region.Queue
2private void checkUsage(ConnectionContext context,ProducerBrokerExchange producerBrokerExchange, Message message) 
3    throws ResourceAllocationException, IOException, InterruptedException {
4    if (message.isPersistent()) { // 持久化消息
5        if (store != null && systemUsage.getStoreUsage().isFull(getStoreUsageHighWaterMark())) {
6            final String logMessage = "Persistent store is Full, " + getStoreUsageHighWaterMark() + "% of "
7                + systemUsage.getStoreUsage().getLimit() + ". Stopping producer ("
8                + message.getProducerId() + ") to prevent flooding "
9                + getActiveMQDestination().getQualifiedName() + "."
10                + " See http://activemq.apache.org/producer-flow-control.html for more info";
11
12            waitForSpace(context, producerBrokerExchange, systemUsage.getStoreUsage(), getStoreUsageHighWaterMark(), logMessage);
13        }
14    } else if (messages.getSystemUsage() != null && systemUsage.getTempUsage().isFull()) {
15        // 非持久化消息
16        final String logMessage = "Temp Store is Full ("
17                + systemUsage.getTempUsage().getPercentUsage() + "% of " + systemUsage.getTempUsage().getLimit()
18                +"). Stopping producer (" + message.getProducerId()
19            + ") to prevent flooding " + getActiveMQDestination().getQualifiedName() + "."
20            + " See http://activemq.apache.org/producer-flow-control.html for more info";
21
22        waitForSpace(context, producerBrokerExchange, messages.getSystemUsage().getTempUsage(), logMessage);
23    }
24}

最后进入具体的处理逻辑：

1// org.apache.activemq.broker.region.BaseDestination
2protected final void waitForSpace(ConnectionContext context, ProducerBrokerExchange producerBrokerExchange, 
3        Usage<?> usage, int highWaterMark, String warning) 
4        throws IOException, InterruptedException, ResourceAllocationException {
5    // 如果配置了sendFailIfNoSpace="true"
6    if (!context.isNetworkConnection() && systemUsage.isSendFailIfNoSpace()) {
7        getLog().debug("sendFailIfNoSpace, forcing exception on send, usage: {}: {}", usage, warning);
8        throw new ResourceAllocationException(warning);
9    }
10    if (!context.isNetworkConnection() && systemUsage.getSendFailIfNoSpaceAfterTimeout() != 0) {
11        if (!usage.waitForSpace(systemUsage.getSendFailIfNoSpaceAfterTimeout(), highWaterMark)) {
12            getLog().debug("sendFailIfNoSpaceAfterTimeout expired, forcing exception on send, usage: {}: {}", usage, warning);
13            throw new ResourceAllocationException(warning);
14        }
15    } else {
16        long start = System.currentTimeMillis();
17        long nextWarn = start;
18        producerBrokerExchange.blockingOnFlowControl(true);
19        destinationStatistics.getBlockedSends().increment();
20        while (!usage.waitForSpace(1000, highWaterMark)) {
21            if (context.getStopping().get()) {
22                throw new IOException("Connection closed, send aborted.");
23            }
24
25            long now = System.currentTimeMillis();
26            if (now >= nextWarn) {
27                getLog().info("{}: {} (blocking for: {}s)", new Object[]{ usage, warning, new Long(((now - start) / 1000))});
28                nextWarn = now + blockedProducerWarningInterval;
29            }
30        }
31        long finish = System.currentTimeMillis();
32        long totalTimeBlocked = finish - start;
33        destinationStatistics.getBlockedTime().addTime(totalTimeBlocked);
34        producerBrokerExchange.incrementTimeBlocked(this,totalTimeBlocked);
35        producerBrokerExchange.blockingOnFlowControl(false);
36    }
37}

如果配置了sendFailIfNoSpace="true"，并且抛出异常了，处理异常的调用栈如下：

1//org.apache.activemq.broker.TransportConnection
2public void serviceException(Throwable e) {
3    if (...) {
4        ...
5    }
6    else if (!stopping.get() && !inServiceException) {
7        inServiceException = true;
8        try {
9            SERVICELOG.warn("Async error occurred: ", e);
10            ConnectionError ce = new ConnectionError();
11            ce.setException(e);
12            if (pendingStop) {
13                dispatchSync(ce);
14            } else {
15                dispatchAsync(ce);
16            }
17        } finally {
18            inServiceException = false;
19        }
20    }
21}

那么 producer 是如何处理 ConnectionError 消息呢？

在org.apache.activemq.ActiveMQConnection.onCommand(Object o) 方法中：

1public Response processConnectionError(final ConnectionError error) throws Exception {
2    executor.execute(new Runnable() {
3        @Override
4        public void run() {
5            onAsyncException(error.getException());
6        }
7    });
8    return null;
9}