Tag: java

Distributed Testing with JMeter on EC2

Here’s a little quickie. I wanted to do some distributed testing with JMeter utilizing EC2 instances as slaves, while running the master/UI locally on my laptop. There are several challenges around this all due to the fact that JMeter chose to use Java RMI as the communications protocol. If you thought the FTP protocol was bad Java RMI is worse. I’ll spare you the gory details and get to what worked for me.

EC2 Instances and Security Group

Ok so first, off you’ll need to spin up one or more EC2 instances, I just chose the Amazon Linux distro.

The key here is the Security Group settings, I simply greated four rules,

Type Protocol Ports Source
Custom TCP Rule TCP 1099
Custom TCP Rule TCP 2048
Custom UDP Rule UDP 1099
Custom UDP Rule UDP 2048

There are only two ports involved here, 1099 is the Java RMI Registry port, and 2048 will be the service port that the JMeter server binds to.

Starting the JMeter Slaves

Install JMeter on the EC2 instances somewhere. When you start it up, use the following:

jmeter-server -Dserver.rmi.localport=2048 -Djava.rmi.server.hostname=`curl -s`

This is doing a couple of things, we’re telling the server to tell Java RMI to make our server bind to a well known port (port 2048), ahh, that’s much better for firewalls. And we’re also telling RMI to forge the server name to be the EC2 Public IP address.

SSH Tunnel to the Slaves

Probably should have mentioned it earlier, but now you’ll need to set up some port forwarding-fu between you and the slaves. This ‘ll work. I’ll tell ya why later.

ssh -R11099:localhost:11099 ......

Starting the JMeter Master/UI

The last bit of trickery is in starting the JMeter client. Here’s what I did

bin/jmeter -Dclient.rmi.localport=11099 -Djava.rmi.server.hostname=localhost -Jremote_hosts=ec2-ip-address,ec2-ip-address

What we’re doing here is a few things, telling the “client” (aka Master (aka UI)) to start its own RMI registry on port 11099, and fooling the registry into believing our local hostname is ‘localhost’.

When the client connects to a slave, it will now say, “hey talk to my RMI server on localhost:11099”. It’ll try to do that on the EC2 instance, and when it does the SSH Tunnel will forward the traffic back to the local machine, and everything will be happy.

Finally the last part of the line, enter the Public IPs of all the EC2 slaves you got.

Good Luck

And that is all I did to get it working for me, hope this helps you too.


Of Tomcat, OutOfMemory Errors & Continuous Development

So you are working in your favorite IDE on some sort of web application or web service, and you are continuously deploying to Tomcat but ocassionally the deployment fails, you see the CPU of the JVM process spike to 100% and you see something like java.lang.OutOfMemoryError: PermGen space show you in your catalina.out file.

The root cause is arguably either a memory leak in Tomcat or something that your application is not cleaning up when its being undeployed/shut down.

But never mind that, you just are annoyed by having to find the process ID of the Tomcat JVM and kill it so you can restart Tomcat.

Instead of doing all that by hand, you can have the JVM kill itself when it encounters this condition, with the use of the “-XX:OnOutOfMemoryError” JVM option. It is as simple as starting up the JVM with the option

-XX:OnOutOfMemoryError="kill -9 %p"

Now, I use NetBeans, and it’s actually not that trivial to try to specify this option correctly in the NetBeans tomcat config, so I punted to just tweaking the catalina.sh script, here’s what I did.

First, I made a backup of catalina.sh

cp catalina.sh catalina.sh-orig

Next, I added the following lines just after the initial comment block in the file.

OOM_OPTS="-XX:OnOutOfMemoryError=kill -9 %p"
export OOM_OPTS

Finally, I found every location where the JVM was being invoked and added “$OOM_OPTS” to the argument list. Its actually pretty easy to do with a sed command

sed -e 's/\($JAVA_OPTS $CATALINA_OPTS\)/\1 "$OOM_OPTS"/g' catalina.sh

And voila. Next time you restart Tomcat, when it encounters an OutOfMemory condition it will kill itself automatically.

Adding custom Class-Path entries to a Netbeans Java Application Project

I want to jot this down before I forget. I was working in a Netbeans Java Application project, but I wanted to augment the Class-Path of the MANIFEST.MF in the Jar file to include an additional location.

If you’ve looked at a project, you’ll notice that Netbeans does create a MANIFEST.MF itself, but if you attempt to add a Class-Path property by hand, it will get bundled verbatim, so any additional libraries that Netbeans would have added for you don’t get added.

I ended up with this solution. I added this to the build.xml file

<target name="-pre-jar">
    <property location="${build.classes.dir}" name="build.classes.dir.resolved"/>
    <pathconvert property="run.classpath.without.build.classes.dir">
        <path path="${run.classpath}"/>
        <map from="${build.classes.dir.resolved}" to=""/>
    <pathconvert pathsep=" " property="jar.classpath">
        <path path="${run.classpath.without.build.classes.dir}"/>
            <globmapper from="*" to="lib/*"/>
    <manifest file="MANIFEST.MF">
        <attribute name="Class-Path" value="${jar.classpath} lib/"/>

Most of the meat of the target is taken from nbproject/build-impl.xml, the only thing I have really done is add the desired paths to the “Class-Path” line, in this case I just added “lib/” in there.