前言
在Tomcat集群中，当一个节点出现故障，其他节点该如何接管故障节点的Session信息呢？本文带来的解决方案是基于MSM+Memcached实现Session共享。
相关介绍
MSM
MSM–Memcached Session Manager是一个高可用的Tomcat Session共享解决方案，除了可以从本机内存快速读取Session信息(仅针对黏性Session)外，同时可使用Memcached存取Session，以实现高可用。
工作原理
Sticky Session（黏性） 模式下的工作原理
#Tomcat本地Session为主Session，Memcached 中的Session为备Session
安装在Tomcat上的MSM使用本机内存保存Session，当一个请求执行完毕之后，如果对应的Session在本地不存在（即某用户的第一次请求），则将该Session复制一份至Memcached；当该Session的下一个请求到达时，会使用Tomcat的本地Session，请求处理结束之后，Session的变化会同步更新到 Memcached，保证数据一致。
当集群中的一个Tomcat挂掉，下一次请求会被路由到其他Tomcat上。负责处理此此请求的Tomcat并不清楚Session信息，于是从Memcached查找该Session，更新该Session并将其保存至本机。此次请求结束，Session被修改，送回Memcached备份。
Non-sticky Session （非黏性）模式下的工作原理
#Tomcat本地Session为中转Session，Memcached为主备Session
收到请求，加载备Session至本地容器，若备Session加载失败则从主Session加载
请求处理结束之后，Session的变化会同步更新到Memcached，并清除Tomcat本地Session
实现过程
实验拓扑
#系统环境：CentOS6.6
nginx安装配置
1解决依赖关系 2[root@scholar ~]# yum groupinstall "Development Tools" "Server Platform Deveopment" -y 3[root@scholar ~]# yum install openssl-devel pcre-devel -y 4[root@scholar ~]# groupadd -r nginx 5[root@scholar ~]# useradd -r -g nginx nginx 6[root@scholar ~]# tar xf nginx-1.6.3.tar.gz 7[root@scholar ~]# cd nginx-1.6.3 8[root@scholar nginx-1.6.3]# ./configure \ 9> --prefix=/usr/local/nginx \ 10> --sbin-path=/usr/sbin/nginx \ 11> --conf-path=/etc/nginx/nginx.conf \ 12> --error-log-path=/var/log/nginx/error.log \ 13> --http-log-path=/var/log/nginx/access.log \ 14> --pid-path=/var/run/nginx/nginx.pid \ 15> --lock-path=/var/lock/nginx.lock \ 16> --user=nginx \ 17> --group=nginx \ 18> --with-http_ssl_module \ 19> --with-http_flv_module \ 20> --with-http_stub_status_module \ 21> --with-http_gzip_static_module \ 22> --http-client-body-temp-path=/usr/local/nginx/client/ \ 23> --http-proxy-temp-path=/usr/local/nginx/proxy/ \ 24> --http-fastcgi-temp-path=/usr/local/nginx/fcgi/ \ 25> --http-uwsgi-temp-path=/usr/local/nginx/uwsgi \ 26> --http-scgi-temp-path=/usr/local/nginx/scgi \ 27> --with-pcre 28[root@scholar nginx-1.6.3]# make && make install 29为nginx提供SysV init脚本 30[root@scholar ~]# vim /etc/rc.d/init.d/nginx 31#新建文件/etc/rc.d/init.d/nginx，内容如下： 32#!/bin/sh 33# 34# nginx - this script starts and stops the nginx daemon 35# 36# chkconfig: - 85 15 37# description: Nginx is an HTTP(S) server, HTTP(S) reverse \ 38# proxy and IMAP/POP3 proxy server 39# processname: nginx 40# config: /etc/nginx/nginx.conf 41# config: /etc/sysconfig/nginx 42# pidfile: /var/run/nginx.pid 43# Source function library. 44. /etc/rc.d/init.d/functions 45# Source networking configuration. 46. /etc/sysconfig/network 47# Check that networking is up. 48[ "$NETWORKING" = "no" ] && exit 0 49nginx="/usr/sbin/nginx" 50prog=$(basename $nginx) 51NGINX_CONF_FILE="/etc/nginx/nginx.conf" 52[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx 53lockfile=/var/lock/subsys/nginx 54make_dirs() { 55# make required directories 56user=`nginx -V 2>&1 | grep "configure arguments:" | sed's/[^*]*--user=\([^ ]*\).*/\1/g' -` 57options=`$nginx -V 2>&1 | grep 'configure arguments:'` 58for opt in $options; do 59if [ `echo $opt | grep '.*-temp-path'` ]; then 60value=`echo $opt | cut -d "=" -f 2` 61if [ ! -d "$value" ]; then 62# echo "creating" $value 63mkdir -p $value && chown -R $user $value 64fi 65fi 66done 67} 68start() { 69[ -x $nginx ] || exit 5 70[ -f $NGINX_CONF_FILE ] || exit 6 71make_dirs 72echo -n $"Starting $prog: " 73daemon $nginx -c $NGINX_CONF_FILE 74retval=$? 75echo 76[ $retval -eq 0 ] && touch $lockfile 77return $retval 78} 79stop() { 80echo -n $"Stopping $prog: " 81killproc $prog -QUIT 82retval=$? 83echo 84[ $retval -eq 0 ] && rm -f $lockfile 85return $retval 86} 87restart() { 88configtest || return $? 89stop 90sleep 1 91start 92} 93reload() { 94configtest || return $? 95echo -n $"Reloading $prog: " 96killproc $nginx -HUP 97RETVAL=$? 98echo 99} 100force_reload() { 101restart 102} 103configtest() { 104$nginx -t -c $NGINX_CONF_FILE 105} 106rh_status() { 107status $prog 108} 109rh_status_q() { 110rh_status >/dev/null 2>&1 111} 112case "$1" in 113start) 114rh_status_q && exit 0 115$1 116;; 117stop) 118rh_status_q || exit 0 119$1 120;; 121restart|configtest) 122$1 123;; 124reload) 125rh_status_q || exit 7 126$1 127;; 128force-reload) 129force_reload 130;; 131status) 132rh_status 133;; 134condrestart|try-restart) 135rh_status_q || exit 0 136;; 137*) 138echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force- 139reload|configtest}" 140exit 2 141esac 142 143配置nginx 144[root@scholar ~]# vim /etc/nginx/nginx.conf 145upstream www.scholar.com { 146server 172.16.10.123:8080; 147server 172.16.10.124:8080; 148} 149server { 150listen 80; 151server_name www.scholar.com; 152location / { 153proxy_pass http://www.scholar.com; 154index index.jsp index.html index.htm; 155} 156} 157[root@scholar ~]# service nginx start 158Starting nginx: [ OK ] 159测试页面 160[root@node1 local]# cd tomcat/webapps/ 161[root@node1 webapps]# mkdir -pv test/WEB-INF/{classes,lib} 162[root@node1 webapps]# cd test/ 163[root@node1 test]# vim index.jsp 164<%@ page language="java" %> 165<html> 166<head><title>TomcatA</title></head> 167<body> 168<h1><font color="red">TomcatA.scholar.com</font></h1> 169<table align="centre" border="1"> 170<tr> 171<td>Session ID</td> 172<% session.setAttribute("scholar.com","scholar.com"); %> 173<td><%= session.getId() %></td> 174</tr> 175<tr> 176<td>Created on</td> 177<td><%= session.getCreationTime() %></td> 178</tr> 179</table> 180</body> 181</html> 182#另一个节点将TomcatA替换为TomcatB，颜色设为蓝色 183[root@node1 test]# service tomcat start
此时Session信息并不一致，接下来我们通过配置MSM实现Session共享
memcached安装
1#解决依赖关系 2[root@scholar ~]# yum groupinstall "Development Tools" "Server Platform Deveopment" -y 3#安装libevent 4#memcached依赖于libevent API，因此要事先安装之 5[root@scholar ~]# tar xf libevent-2.0.22-stable.tar.gz 6[root@scholar ~]# cd libevent-2.0.22-stable 7[root@scholar libevent-2.0.22-stable]# ./configure --prefix=/usr/local/libevent 8[root@scholar libevent-2.0.22-stable]# make && make install 9[root@scholar ~]# echo "/usr/local/libevent/lib" > /etc/ld.so.conf.d/libevent.conf 10[root@scholar ~]# ldconfig 11#安装配置memcached 12[root@scholar ~]# tar xf memcached-1.4.24.tar.tar 13[root@scholar ~]# cd memcached-1.4.24 14[root@scholar memcached-1.4.24]# ./configure --prefix=/usr/local/memcached --with-libevent=/usr/local/libevent 15[root@scholar memcached-1.4.24]# make && make install 16 17[root@scholar ~]# vim /etc/init.d/memcached 18#!/bin/bash 19# 20# Init file for memcached 21# 22# chkconfig: - 86 14 23# description: Distributed memory caching daemon 24# 25# processname: memcached 26# config: /etc/sysconfig/memcached 27. /etc/rc.d/init.d/functions 28## Default variables 29PORT="11211" 30USER="nobody" 31MAXCONN="1024" 32CACHESIZE="64" 33RETVAL=0 34prog="/usr/local/memcached/bin/memcached" 35desc="Distributed memory caching" 36lockfile="/var/lock/subsys/memcached" 37start() { 38echo -n $"Starting $desc (memcached): " 39daemon $prog -d -p $PORT -u $USER -c $MAXCONN -m $CACHESIZE 40RETVAL=$? 41[ $RETVAL -eq 0 ] && success && touch $lockfile || failure 42echo 43return $RETVAL 44} 45stop() { 46echo -n $"Shutting down $desc (memcached): " 47killproc $prog 48RETVAL=$? 49[ $RETVAL -eq 0 ] && success && rm -f $lockfile || failure 50echo 51return $RETVAL 52} 53restart() { 54stop 55start 56} 57reload() { 58echo -n $"Reloading $desc ($prog): " 59killproc $prog -HUP 60RETVAL=$? 61[ $RETVAL -eq 0 ] && success || failure 62echo 63return $RETVAL 64} 65case "$1" in 66start) 67start 68;; 69stop) 70stop 71;; 72restart) 73restart 74;; 75condrestart) 76[ -e $lockfile ] && restart 77RETVAL=$? 78;; 79reload) 80reload 81;; 82status) 83status $prog 84RETVAL=$? 85;; 86*) 87echo $"Usage: $0 {start|stop|restart|condrestart|status}" 88RETVAL=1 89esac 90exit $RETVAL 91 92将所需jar包放入各tomcat节点的tomcat安装目录下的lib目录中 93[root@node1 ~]# cd msm/ 94[root@node1 msm]# ls 95javolution-5.4.3.1.jar msm-javolution-serializer-1.8.1.jar 96memcached-session-manager-1.8.1.jar spymemcached-2.10.2.jar 97memcached-session-manager-tc7-1.8.1.jar 98[root@node1 msm]# cp * /usr/local/tomcat/lib/ 99#各tomcat节点都需执行以上操作 100[root@node1 msm]# vim /usr/local/tomcat/conf/server.xml 101<?xml version='1.0' encoding='utf-8'?> 102<Server port="8005" shutdown="SHUTDOWN"> 103<Listener className="org.apache.catalina.startup.VersionLoggerListener" /> 104<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> 105<Listener className="org.apache.catalina.core.JasperListener"/> 106<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> 107<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" /> 108<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" /> 109<GlobalNamingResources> 110<Resource name="UserDatabase" auth="Container" 111type="org.apache.catalina.UserDatabase" 112description="User database that can be updated and saved" 113</GlobalNamingResources> 114<Service name="Catalina"> 115<Connector port="8080" protocol="HTTP/1.1" 116connectionTimeout="20000" 117redirectPort="8443" /> 118<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"/> 119<Engine name="Catalina" defaultHost="localhost"> 120<Realm className="org.apache.catalina.realm.LockOutRealm"> 121<Realm className="org.apache.catalina.realm.UserDatabaseRealm" 122resourceName="UserDatabase"/> 123</Realm> 124<Host name="www.scholar.com" appBase="webapps" 125unpackWARs="true" autoDeploy="true"> 126<Context path="/test" docBase="/usr/local/tomcat/webapps/test/" reloadable="true"> 127<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager" 128memcachedNodes="n1:172.16.10.126:11211,n2:172.16.10.212:11211" 129failoverNodes="n1" 130requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$" 131transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.Javolu 132tionTranscoderFactory" 133/> 134</Context> 135<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" 136prefix="scholar_access_log." suffix=".txt" 137pattern="%h %l %u %t "%r" %s %b" /> 138</Host> 139<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" 140prefix="scholar_access_log." suffix=".txt" 141pattern="%h %l %u %t "%r" %s %b" /> 142</Host> 143<Host name="localhost" appBase="webapps" 144unpackWARs="true" autoDeploy="true"> 145<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" 146prefix="localhost_access_log." suffix=".txt" 147pattern="%h %l %u %t "%r" %s %b" /> 148</Host> 149</Engine> 150</Service> 151</Server>
访问测试
由此可见，Session共享已实现，下面我们模拟TomcatB节点故障，看一下Session是否会发生改变
虽然因为TomcatB故障，导致用户请求被调度到了TomcatA节点上，但Session ID并未发生改变，即Session集群内的所有节点都保存有全局的Session信息，很好的实现了用户访问的不中断
当n2（memcached）节点发生故障，Session信息会不会转移到其他memcached节点呢，我们来试一下
Session已转移到n1上，而且Session ID并未发生改变，至此，Tomcat基于MSM+Memcached实现Session共享目的已实现
The end
Tomcat基于MSM+Memcached实现Session共享实验就说到这里了