mudas Blog

The World According to …

Apache 2.2.3 with mod_jk (1.2.31) configuration example

with 13 comments

1. Download mod_jk from http://tomcat.apache.org/download-connectors.cgi and compile it, on my Centos 5.5 (64bit) i do this:

./configure --with-apxs=/usr/sbin/apxs
When you have problems with apxs try this:
yum install httpd-devel
./configure --with-apxs=yes

2. Copy the mod_jk.so to /etc/httpd/modules

3. Create in /etc/httpd/conf.d/ a file called mod-jk.conf with this content (example):

# Configuration Example for mod_jk
# used in combination with Apache 2.2.x

# Change the path and file name of the module, in case
# you have installed it outside of httpd, or using
# a versioned file name.
LoadModule jk_module modules/mod_jk.so

# We need a workers file exactly once
# and in the global server
JkWorkersFile conf.d/workers.properties

# If you want to put all mounts into an external file
# that gets reloaded automatically after changes
# (with a default latency of 1 minute),
# you can define the name of the file here.
JkMountFile conf.d/uriworkermap.properties

# Our JK error log
# You can (and should) use rotatelogs here
JkLogFile logs/mod_jk.log

# Our JK log level (trace,debug,info,warn,error)
JkLogLevel info

# Define the log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

# JkRequestLogFormat
JkRequestLogFormat "%w %V %T"

# Add shared memory.
# This directive is present with 1.2.10 and
# later versions of mod_jk, and is needed for
# for load balancing to work properly
JkShmFile /var/cache/httpd/jk.shm

# Define a new log format you can use in any CustomLog in order
# to add mod_jk specific information to your access log.
#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{Cookie}i\" \"%{Set-Cookie}o\" %{pid}P %{tid}P %{JK_LB_FIRST_NAME}n %{JK_LB_LAST_NAME}n ACC %{JK_LB_LAST_ACCESSED}n ERR %{JK_LB_LAST_ERRORS}n BSY %{JK_LB_LAST_BUSY}n %{JK_LB_LAST_STATE}n %D" extended_jk

# JkOptions indicates to send SSK KEY SIZE
# Since: 1.2.26
JkOptions +ForwardKeySize +ForwardURIProxy -ForwardDirectories

# IMPORTANT: Mounts and virtual hosts
# If you are using VirtualHost elements, you
# - can put mounts only used in some virtual host into its VirtualHost element
# - can copy all global mounts to it using "JkMountCopy On" inside the VirtualHost
# - can copy all global mounts to all virtual hosts by putting
# "JkMountCopy All" into the global server
# Since: 1.2.26
JkMountCopy All

# Start a separate thread for internal tasks like
# idle connection probing, connection pool resizing
# and load value decay.
# Run these tasks every JkWatchdogInterval seconds.
# Since: 1.2.27
JkWatchdogInterval 60

# Configure access to jk-status and jk-manager
# If you want to make this available in a virtual host,
# either move this block into the virtual host
# or copy it logically there by including "JkMountCopy On"
# in the virtual host.
# Add an appropriate authentication method here!

Location /jkstatus INFO: don't forget the angle bracket before and after 'Location /jkstatus' (wordpress has a problem with it)
# Inside Location we can omit the URL in JkMount
JkMount jkstatus
Order deny,allow
Deny from all
Allow from 127.0.0.1
Allow from 172.1.1.110
/Location INFO: don't forget the angle bracket before and after '/Location' (wordpress has a problem with it)

Location /jkmanager INFO: don't forget the angle bracket before and after 'Location /jkmanager' (wordpress has a problem with it)
# Inside Location we can omit the URL in JkMount
JkMount jkmanager
Order deny,allow
Deny from all
Allow from 127.0.0.1
Allow from 172.1.1.110

AuthType Basic
AuthName "jkmanager logon"
AuthUserFile /etc/httpd/security/htuser-webbox
AuthGroupFile /etc/httpd/security/htgroup-webbox
Require group jkadmingroup
Require user jkchecker
/Location INFO: don't forget the angle bracket before and after '/Location' (wordpress has a problem with it)

4. Create in /etc/httpd/conf.d/ a file called workers.properties with this content (example):

# workers.properties

###
### !!! special configuration for jk status !!!
###

# Define list of workers that will be used for mapping requests.
# This directive can be used multiple times for the same load balancer.
#
worker.list=jkstatus,jkmanager

# Define two status worker for managing load balancer:
# - jkstatus for read-only use
# - jkmanager for read/write use

worker.jkstatus.type=status
worker.jkstatus.read_only=true

worker.jkmanager.type=status

###
### !!! default !!!
### ! reference settings at the bottom !
###

#
# Now we configure the first rule set

# Define some defaults for the first set - default (access via port 8009)
#
worker.default.port=8009
worker.default.reference=worker.template

# And we use them in one group
#
worker.lbdefault.domain=dom1
worker.lbdefault.distance=0
worker.lbdefault.reference=worker.default

# Define list of workers that will be used for mapping requests.
# This directive can be used multiple times for the same load balancer.
#
worker.list=lb-app1,lb-app2

# app1-test
#
worker.lb-app1.type=lb
worker.lb-app1.method=R
worker.lb-app1.balance_workers=w207,w209,w210,w218
worker.lb-app1.sticky_session=True
worker.lb-app1.sticky_session_force=False

worker.w207.host=10.14.14.207
worker.w207.reference=worker.lbdefault
# Activation allows to configure
# whether this node should actually be used
# A: active (use node fully)
# D: disabled (only use, if sticky session needs this node)
# S: stopped (do not use)
# Since: 1.2.19
worker.w207.activation=A

worker.w209.host=10.14.14.209
worker.w209.reference=worker.lbdefault
# Activation allows to configure
# whether this node should actually be used
# A: active (use node fully)
# D: disabled (only use, if sticky session needs this node)
# S: stopped (do not use)
# Since: 1.2.19
worker.w209.activation=A

worker.w210.host=10.14.14.210
worker.w210.reference=worker.lbdefault
# Activation allows to configure
# whether this node should actually be used
# A: active (use node fully)
# D: disabled (only use, if sticky session needs this node)
# S: stopped (do not use)
# Since: 1.2.19
worker.w210.activation=S

worker.w218.host=10.14.14.218
worker.w218.reference=worker.lbdefault
# Activation allows to configure
# whether this node should actually be used
# A: active (use node fully)
# D: disabled (only use, if sticky session needs this node)
# S: stopped (do not use)
# Since: 1.2.19
worker.w218.activation=S

# app2-test
#
worker.lb-app2.type=lb
worker.lb-app2.method=R
worker.lb-app2.balance_workers=w248,w249,w236
worker.lb-app2.sticky_session=True
worker.lb-app2.sticky_session_force=False

worker.w248.host=10.14.14.248
worker.w248.reference=worker.lbdefault
# Activation allows to configure
# whether this node should actually be used
# A: active (use node fully)
# D: disabled (only use, if sticky session needs this node)
# S: stopped (do not use)
# Since: 1.2.19
worker.w248.activation=S

worker.w249.host=10.14.14.249
worker.w249.reference=worker.lbdefault
# Activation allows to configure
# whether this node should actually be used
# A: active (use node fully)
# D: disabled (only use, if sticky session needs this node)
# S: stopped (do not use)
# Since: 1.2.19
worker.w249.activation=S

worker.w236.host=10.14.14.236
worker.w236.reference=worker.lbdefault
# Activation allows to configure
# whether this node should actually be used
# A: active (use node fully)
# D: disabled (only use, if sticky session needs this node)
# S: stopped (do not use)
# Since: 1.2.19
worker.w236.activation=S

# Define list of workers that will be used for mapping requests.
# This directive can be used multiple times for the same load balancer.
#
worker.list=w022,w023

# webservices-1-test
#
worker.w022.host=10.14.32.22
worker.w022.reference=worker.lbdefault
# Activation allows to configure
# whether this node should actually be used
# A: active (use node fully)
# D: disabled (only use, if sticky session needs this node)
# S: stopped (do not use)
# Since: 1.2.19
worker.w022.activation=A

# webservices-2-test
#
worker.w023.host=10.14.32.23
worker.w023.reference=worker.lbdefault
# Activation allows to configure
# whether this node should actually be used
# A: active (use node fully)
# D: disabled (only use, if sticky session needs this node)
# S: stopped (do not use)
# Since: 1.2.19
worker.w023.activation=A

###
### !!! special configuration for another app !!!
### ! reference settings at the bottom !
###

#
# Now we configure the second rule set

# Define some defaults for the second set - another app special (access via port 7909)
#
worker.another.port=7909
worker.another.reference=worker.template

# And we use them in one group
#
worker.lbanother.domain=dom1
worker.lbanother.distance=0
worker.lbanother.reference=worker.another

# Define list of workers that will be used for mapping requests.
# This directive can be used multiple times for the same load balancer.
#
worker.list=wa022,wa023

# another-app1-test
#
worker.wa022.host=10.14.32.22
worker.wa022.reference=worker.another
# Activation allows to configure
# whether this node should actually be used
# A: active (use node fully)
# D: disabled (only use, if sticky session needs this node)
# S: stopped (do not use)
# Since: 1.2.19
worker.wa022.activation=A

# another-app2-test
#
worker.wa023.host=10.14.32.23
worker.wa023.reference=worker.another
# Activation allows to configure
# whether this node should actually be used
# A: active (use node fully)
# D: disabled (only use, if sticky session needs this node)
# S: stopped (do not use)
# Since: 1.2.19
worker.wa023.activation=A

###
### !!! global settings / reference template !!!
###

# worker.maintain: seconds, default=60
# Worker connection pool maintain interval in seconds.
# If set to the positive value JK will scan all connections for all workers
# specified in worker.list directive and check if connections needs to be recycled.
worker.maintain=60

# We put the parameters
# which should apply to all our ajp13
# workers into the referenced template
# - Type is ajp13
worker.template.type=ajp13

# - socket_timeout: seconds, default=0
# Socket timeout in seconds used for the communication channel between JK and remote host.
# If the remote host does not respond inside the timeout specified, JK will generate an error, and retry again.
# - alternatively you can use since: 1.2.27 'template.socket_connect_timeout' milliseconds, default=0
worker.template.socket_timeout=10

# - socket_keepalive: boolean, default=false
# Should we send TCP keepalive packets
# when connection is idle (socket option)?
worker.template.socket_keepalive=true

# - connection_pool_timeout: seconds, default=0
# Idle time, before a connection is eligible
# for being closed (pool shrinking).
# This should be the same value as connectionTimeout
# in the Tomcat AJP connector, but there it is
# milliseconds, here seconds.
# This value must equal tomcat server.xml's connectionTimeout of 10 minutes.
worker.template.connection_pool_timeout=600

# - recovery_options: number, default=0
# Bit mask to configure, if a request, which was send
# to a backend successfully, should be retried on another backend
# in case there's a problem with the response.
# Value "3" disables retries, whenever a part of the request was
# successfully send to the backend.
# This attribute is a bit mask. The following bits are allowed:
# 1: don't recover if Tomcat failed after getting the request
# 2: don't recover if Tomcat failed after sending the headers to client
# 4: close the connection to Tomcat, if we detect an error when writing back the answer to the client (browser)
# 8: always recover requests for HTTP method HEAD (even if Bits 1 or 2 are set)
# 16: always recover requests for HTTP method GET (even if Bits 1 or 2 are set)
worker.template.recovery_options=7

# - ping_mode: Character, default=none
# When should we use cping/cpong connection probing?
# C = directly after establishing a new connection
# P = directly before sending each request
# I = in regular intervals for idle connections
# using the watchdog thread
# A = all of the above
# Since: 1.2.27
worker.template.ping_mode=A

# - ping_timeout: milliseconds, default=10000
# Wait timeout for cpong after cping
# Can be overwritten for modes C and P
# Using connect_timeout and prepost_timeout.
# Since: 1.2.27
worker.template.ping_timeout=5000

# - reply_timeout: milliseconds, default=0
# Any pause longer than this timeout during waiting
# for a part of the reply will abort handling the request
# in mod_jk. The request will proceed running in
# Tomcat, but the web server resources will be freed
# and an error is send to the client.
# For individual requests, the timeout can be overwritten
# by the Apache environment variable JK_REPLY_TIMEOUT.
# JK_REPLY_TIMEOUT since: 1.2.27
worker.template.reply_timeout=300000

#
# The next two properties allow mod_jk to efficiently determine,
# if a connection is in error or not responding fast enough due to whatever reason.
#
# - connect_timeout: milliseconds, default=0
# Connect timeout property told webserver to send a PING request
# on ajp13 connection after connection is established.
# The parameter is the delay in milliseconds to wait for the PONG reply.
worker.template.connect_timeout=10000

# - prepost_timeout: milliseconds, default=0
# Prepost timeout property told webserver to send a PING request
# on ajp13 connection before forwarding to it a request.
# The parameter is the delay in milliseconds to wait for the PONG reply.
worker.template.prepost_timeout=10000

5. Create in /etc/httpd/conf.d/ a file called uriworkermap.properties with this content (example):

# uriworkermap.properties
#
# This file provides sample mappings for example wlb
# worker defined in workermap.properties.minimal
# The general syntax for this file is:
# [URL]=[Worker name]

#####
# configuration for all services
#####

#####
# app1-test
#####
/foo=lb-app1
/foo/*=lb-app1

#####
# app2-test
#####
/bar=lb-app2
/bar/*=lb-app2

#####
# webservices-1-test
#####
/abc/performance/*=w022
/abc-webstorage/performance/*=w022
/abc-ws/performance/*=w022

#####
# webservices-2-test
#####
/abc/en/performance/*=w023
/abc-webstorage/en/performance/*=w023
/abc-ws/en/performance/*=w023

#####
# another-app1-test
#####
/xyz/research/*=wa022
/xyz-webstorage/research/*=wa022
/xyz-ws/research/*=wa022

#####
# another-app2-test
#####
/xyz/en/research/*=wa023
/xyz-webstorage/en/research/*=wa023
/xyz-ws/en/research/*=wa023

6. Configure your jboss or tomact and set the proper jvm_route and don’t forget to set in the server.xml file the connectionTimeout to 10 minutes for the ajp connector (eg: connectionTimeout=”600000″)

About these ads

Written by David Murko

January 13, 2011 at 5:42 pm

Posted in Apache, JBoss

Tagged with , ,

13 Responses

Subscribe to comments with RSS.

  1. Hi David!

    Initially want to thank you for the great setting and their explanations.
    It helped me a lot this article.

    1º) I have two servers Xeon 3.8GHz Quad Core 2 with 8GB RAM, installed with Linux Debian Squeeze and I have both Apache 2.2.16-6, Tomcat 6.0.32 and mod_jk 1.2.31 (load balance).

    2º) I have a web application that receives the maximum 250 concurrent users.

    3º) The settings file apache.conf are basically like this:

    Timeout 300
    KeepAlive On
    MaxKeepAliveRequests 250
    KeepAliveTimeout 15

    StartServers 5
    MinSpareServers 10
    MaxSpareServers 25
    MaxClients 250
    MaxRequestsPerChild 1

    StartServers 5
    MinSpareThreads 5
    MaxSpareThreads 25
    ThreadLimit 64
    ThreadsPerChild 25
    MaxClients 250
    MaxRequestsPerChild 1

    StartServers 5
    MinSpareThreads 5
    MaxSpareThreads 45
    ThreadLimit 64
    ThreadsPerChild 25
    MaxClients 250
    MaxRequestsPerChild 1

    4º) The settings file server.xml are basically like this:

    5º) The file is sysconf.conf with the default Debian installation.

    6º) My configuration of the JVM this way:

    JAVA_OPTS=” -server -Xss1024k -Xms1536m -Xmx1536m -XX:MaxPermSize=128m -XX:NewSize=128m -XX:MaxNewSize=128m -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0 -XX:+UseTLAB -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled ”

    I would like to ask you some questions:

    1º) How are the settings of the JVM on your server?

    2º) Have you made any configuration in sysconf.conf such as:
    kernel.shmmni
    kernel.shmmax
    kernel.shmall
    vm.overcommit_memory

    3º) You can see some misconfiguration on the information that you passed up?

    Already thank you for all your help and God bless you.

    Dennys

    March 17, 2011 at 6:23 pm

  2. I forgot to mention that the settings are basically the server.xml file like this:

    Connector port=”8009″ protocol=”AJP/1.3″ redirectPort=”8443″ connectionTimeout=”600000″ enableLookups=”false” maxThreads=”250″ minSpareThreads=”50″ maxSpareThreads=”250″

    Excuse me and thanks.

    Dennys

    March 17, 2011 at 6:30 pm

  3. I reevaluate the parameters and maxSpareThreads minSpareThreads and decided to reduce them as defined in the file apache.conf

    PS: I hope a new atigo your talking about the Jetty Server.

    Again thank you and congratulations.

    Dennys

    March 18, 2011 at 5:32 pm

  4. Hi David,
    Great post. I would like to know the difference betwee JkWatchDogInterval and ping_mode=I, looks like both will just probe idle connections.

    Chaitanya Gogineni

    April 13, 2011 at 10:02 pm

    • Hello Chaitanya,

      It seems you are right … i found this at the mod_jk homepage:

      JkWatchdogInterval:
      This directive configures the watchdog thread interval in seconds. The workers are maintained periodically by a background thread running periodically every watchdog_interval seconds. Worker maintenance checks for idle connections, corrects load status and is able to detect backend health status.
      The maintenance only happens, if since the last maintenance at least worker.maintain seconds have passed. So setting the JkWatchdogInterval much smaller than worker.maintain is not useful.
      The default value is 0 seconds, meaning the watchdog thread will not be created, and the maintenance is done in combination with normal requests instead.
      This directive is only allowed once. It must be put into the global part of the configuration.
      This directive has been added in version 1.2.27 of mod_jk. It is available only for httpd 2.x and above using APR libraries including thread support.

      ping_mode:
      I (interval): If set, the connection will be probed during the regular internal maintenance cycle, but only if it is idle longer than connection_ping_interval. The timeout can be set by ping_timeout.

      But to be honest, i am not a great expert in this case.
      If you are interested to get further/better information i would suggest to post your question to this mailing list: users[AT]tomcat.apache.org
      They respond really very quick and helped me also in past (thx to r.jung) when i had a problem with an older mod_jk version.

      Regards, david

      David Murko

      April 14, 2011 at 10:46 am

  5. Hi
    using apache2.2.17,mod_jk_1.2.30,tomcat5.5.27..
    my normal LB working fine….(24 RAM on linux) but i want 1500 concurent request to handle..What all i have to configure at apache(prefork.mpm),ajp and tomcat configuration ?

    i confused with max clients and (ajp connector maxprocesses,maxthread in server.xml..

    simu

    April 16, 2011 at 2:01 pm

    • Hi simu,

      I suggest to start with default settings at apache + jboss/tomcat side and check the current and max connection counter a both sides.
      I have good experience with this kind of setup for a performant ha solution:

      internet
      |
      loadbalancer (active/passive)
      |
      apache (active/active) with mod_jk
      |
      jboss/tomcat (active/active)
      |
      db cluster (active/active)

      You can use for ‘pound’ for load-balancing (with is easy to configure and do also the ssl termination) in active/passive mode with linux ha tools with is a good starting point for a working solution. The loadbalancer also act as a squid proxy for all outside requests (also for jboss/tomcat request) using a virtual interface with is controlled via the linux ha tools.

      If you really running in a out of connection situation between apache and jboss/tomcat rise up the values for server limit/max client at apache side and max threats at jboss/tomcat side _but_ make really sure you need this.

      Try to check the values with nagios and twiddle and set some thresholds when you go to production.

      1500 concurrent requests should not be permanent open like a stream … normally it should be a lot of short get/post request so assume you should start with default values and measure your peak values.

      I hope this helps …

      David Murko

      April 18, 2011 at 2:05 pm

  6. hi David
    Thanks for Reply.

    ./apachectl -V
    Server version: Apache/2.2.17 (Unix)
    Server built: Apr 19 2011 08:35:28
    Server’s Module Magic Number: 20051115:25
    Server loaded: APR 1.4.2, APR-Util 1.3.10
    Compiled using: APR 1.4.2, APR-Util 1.3.10
    Architecture: 64-bit
    Server MPM: Worker
    threaded: yes (fixed thread count)
    forked: yes (variable process count)
    Server compiled with….
    -D APACHE_MPM_DIR=”server/mpm/worker”
    -D APR_HAS_SENDFILE
    -D APR_HAS_MMAP
    -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
    -D APR_USE_SYSVSEM_SERIALIZE
    -D APR_USE_PTHREAD_SERIALIZE
    -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
    -D APR_HAS_OTHER_CHILD
    -D AP_HAVE_RELIABLE_PIPED_LOGS
    -D DYNAMIC_MODULE_LIMIT=128
    -D HTTPD_ROOT=”/applications/balancer”
    -D SUEXEC_BIN=”/applications/balancer/bin/suexec”
    -D DEFAULT_SCOREBOARD=”logs/apache_runtime_status”
    -D DEFAULT_ERRORLOG=”logs/error_log”
    -D AP_TYPES_CONFIG_FILE=”conf/mime.types”
    -D SERVER_CONFIG_FILE=”conf/httpd.conf”

    See with above setup i am checking load with SOAPUI.I replaced my setup from prefork to worker MPM.with Default setting i am getting “max client error in error.log”.So i have made following change.please look and validate things if am on track:

    APACHE->
    http.conf

    ServerLimit 30
    StartServers 20
    MaxClients 1500
    MinSpareThreads 40
    MaxSpareThreads 100
    ThreadsPerChild 50
    MaxRequestsPerChild 0

    ——
    —–
    worker.properities
    ——————–
    worker.list=balancer

    #For tomcat 1
    worker.list=worker1
    #worker.worker1.domain=worker1
    worker.worker1.type=ajp13
    worker.worker1.host=
    worker.worker1.port=8009
    worker.worker1.lbfactor=1
    .
    .
    #For tomcat 4
    worker.list=worker4
    #worker.worker4.domain=worker4
    worker2.worker4.type=ajp13
    worker.worker4.host=
    worker.worker4.port=8012
    worker.worker4.lbfactor=1

    worker.balancer.type=lb
    worker.balancer.balance_workers=worker1,worker2,worker3,worker4
    worker.balancer.local_worker_only=1
    worker.balancer.sticky_session=1

    4 TOMCAT->

    —–

    server.xml(tomcat 1)

    i have few questions..:-)

    1)i am hitting with 1500 concurrent user.is this configuration fine?

    2)AJP conector shold unique or not?

    3)I HAVE SET startserver 20(means 20 child processes with 50 thread per child) .but y am getting 2 with following command

    “ps aux | grep http | grep -v “\(root\|grep\)” | wc -l”–>2

    4)Any other configuration need to do for worker MPM?

    THanks a lot in advance david

    regards
    sim

    simu

    April 19, 2011 at 2:05 pm

    • hi
      Missed tomcat details

      simu

      April 19, 2011 at 2:07 pm

  7. Define an AJP 1.3 Connector on port 8009
    Connector port=”8009″
    enableLookups=”false” redirectPort=”8443″
    minProcessors=”25″ maxProcessors=”2000″
    maxThreads=”500″ minSpareThreads=”25″
    maxSpareThreads=”75″ protocol=”AJP/1.3″

    simu

    April 19, 2011 at 2:09 pm

    • Hi simu,

      You have very specific questions :)
      I am so sorry, because i am not an expert for answering this in a proper manner …

      The only advise i can give is to post your use case to this group:

      users[AT]tomcat.apache.org

      This guys are really good and helped me also when i had problems with my config … more brains, better knowledge :)

      Regards, david

      David Murko

      April 19, 2011 at 5:33 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: