Category: Computing

Distributed Testing with JMeter on EC2

Here’s a little quickie. I wanted to do some distributed testing with JMeter utilizing EC2 instances as slaves, while running the master/UI locally on my laptop. There are several challenges around this all due to the fact that JMeter chose to use Java RMI as the communications protocol. If you thought the FTP protocol was bad Java RMI is worse. I’ll spare you the gory details and get to what worked for me.

EC2 Instances and Security Group

Ok so first, off you’ll need to spin up one or more EC2 instances, I just chose the Amazon Linux distro.

The key here is the Security Group settings, I simply greated four rules,

Type Protocol Ports Source
Custom TCP Rule TCP 1099
Custom TCP Rule TCP 2048
Custom UDP Rule UDP 1099
Custom UDP Rule UDP 2048

There are only two ports involved here, 1099 is the Java RMI Registry port, and 2048 will be the service port that the JMeter server binds to.

Starting the JMeter Slaves

Install JMeter on the EC2 instances somewhere. When you start it up, use the following:

jmeter-server -Dserver.rmi.localport=2048 -Djava.rmi.server.hostname=`curl -s`

This is doing a couple of things, we’re telling the server to tell Java RMI to make our server bind to a well known port (port 2048), ahh, that’s much better for firewalls. And we’re also telling RMI to forge the server name to be the EC2 Public IP address.

SSH Tunnel to the Slaves

Probably should have mentioned it earlier, but now you’ll need to set up some port forwarding-fu between you and the slaves. This ‘ll work. I’ll tell ya why later.

ssh -R11099:localhost:11099 ......

Starting the JMeter Master/UI

The last bit of trickery is in starting the JMeter client. Here’s what I did

bin/jmeter -Dclient.rmi.localport=11099 -Djava.rmi.server.hostname=localhost -Jremote_hosts=ec2-ip-address,ec2-ip-address

What we’re doing here is a few things, telling the “client” (aka Master (aka UI)) to start its own RMI registry on port 11099, and fooling the registry into believing our local hostname is ‘localhost’.

When the client connects to a slave, it will now say, “hey talk to my RMI server on localhost:11099”. It’ll try to do that on the EC2 instance, and when it does the SSH Tunnel will forward the traffic back to the local machine, and everything will be happy.

Finally the last part of the line, enter the Public IPs of all the EC2 slaves you got.

Good Luck

And that is all I did to get it working for me, hope this helps you too.


Of Tomcat, OutOfMemory Errors & Continuous Development

So you are working in your favorite IDE on some sort of web application or web service, and you are continuously deploying to Tomcat but ocassionally the deployment fails, you see the CPU of the JVM process spike to 100% and you see something like java.lang.OutOfMemoryError: PermGen space show you in your catalina.out file.

The root cause is arguably either a memory leak in Tomcat or something that your application is not cleaning up when its being undeployed/shut down.

But never mind that, you just are annoyed by having to find the process ID of the Tomcat JVM and kill it so you can restart Tomcat.

Instead of doing all that by hand, you can have the JVM kill itself when it encounters this condition, with the use of the “-XX:OnOutOfMemoryError” JVM option. It is as simple as starting up the JVM with the option

-XX:OnOutOfMemoryError="kill -9 %p"

Now, I use NetBeans, and it’s actually not that trivial to try to specify this option correctly in the NetBeans tomcat config, so I punted to just tweaking the script, here’s what I did.

First, I made a backup of


Next, I added the following lines just after the initial comment block in the file.

OOM_OPTS="-XX:OnOutOfMemoryError=kill -9 %p"
export OOM_OPTS

Finally, I found every location where the JVM was being invoked and added “$OOM_OPTS” to the argument list. Its actually pretty easy to do with a sed command

sed -e 's/\($JAVA_OPTS $CATALINA_OPTS\)/\1 "$OOM_OPTS"/g'

And voila. Next time you restart Tomcat, when it encounters an OutOfMemory condition it will kill itself automatically.

Quickie to show IP Addresses on console login

Here’s a simple script to put into /etc/rc3.d/ to add the IP addresses to the /etc/issue file so that they appear on the console login screen. Handy for environments like Virtual Box or VMWare, to display the IP addresses that were configured via DHCP so I can SSH directly into the system.

# /etc/rc3.d/S99issue

cp /etc/issue /tmp/issue-$$
head -3 /tmp/issue-$$ > /etc/issue
ifconfig | awk '/^[a-z]/{ i=$1 } /inet addr/{print i,"\t",$1,$2}' >> /etc/issue
echo >> /etc/issue
rm -f /tmp/issue-$$

Adding custom Class-Path entries to a Netbeans Java Application Project

I want to jot this down before I forget. I was working in a Netbeans Java Application project, but I wanted to augment the Class-Path of the MANIFEST.MF in the Jar file to include an additional location.

If you’ve looked at a project, you’ll notice that Netbeans does create a MANIFEST.MF itself, but if you attempt to add a Class-Path property by hand, it will get bundled verbatim, so any additional libraries that Netbeans would have added for you don’t get added.

I ended up with this solution. I added this to the build.xml file

<target name="-pre-jar">
    <property location="${build.classes.dir}" name="build.classes.dir.resolved"/>
    <pathconvert property="">
        <path path="${run.classpath}"/>
        <map from="${build.classes.dir.resolved}" to=""/>
    <pathconvert pathsep=" " property="jar.classpath">
        <path path="${}"/>
            <globmapper from="*" to="lib/*"/>
    <manifest file="MANIFEST.MF">
        <attribute name="Class-Path" value="${jar.classpath} lib/"/>

Most of the meat of the target is taken from nbproject/build-impl.xml, the only thing I have really done is add the desired paths to the “Class-Path” line, in this case I just added “lib/” in there.

Compiling Java-Readline with GNU readline for OSX

I had previously wrote some instructions for compiling the libreadline-java on OSX with the built-in readline libary on OS X (for use with Henplus). Its time to revisit that, cause it had some issues.

  • OS X Lion, and maybe other versions, doesn’t ship with GNU readline at all. Instead, it ships with a BSD-licensed alternative called libedit, which has a compatibility layer to emulate GNU readline, but has some weird quirks that for those used to GNU readline get annoyed with.
  • libeadline-java I have since found already supports libedit. I didn’t realize that at the time, and so the patches I made do essentially the same thing as if libreadline-java were configured to compile against libedit in the first place.
  • And also, this guy Andre keeps bugging me, I don’t know him, but don’t want to let him down none the less.

The new strategy is this, compile your own GNU readline, then point libreadline-java at it and compile against your new GNU readline library. There’s still some tweaks that need to be done, but there are less and they are more straight forward.

Compiling GNU readline

There’s only one small tweak you have to make to the build scripts to get it to compile correctly, at least for Lion. The latest version of GNU readline, version 6.2, doesn’t have support for compiling against Lion, but thanks to some other sleuthing, the tweak is pretty simple:

In the file support/shobj-conf,

  • Find the line darwin[89]*|darwin10*) and change it to darwin[89]*|darwin10*|darwin11*)

Now, configure, build and install as usual, we’ll target /usr/local as the destination

configure --prefix=/usr/local
make intall

Ok, done with that. I’ve made a readline-6.2-osx-lion.patch for those of you interested.

Tweaking Java-Readline

Ok, now on to compiling libreadline-java with the freshly build libreadline. First off, the list of tweaks you need to make

In the source root, edit Makefile and make the following changes:

  • Below the line # Operating system dependent
    Add JAVA_HOME = `/usr/libexec/java_home`
  • Change the JAVAINCLUDE variable
    to /System/Library/Frameworks/JavaVM.framework/Headers
    (This is needed to find jni.h)
  • Change the JAVANATINC variable to /usr/local/include

Next, edit src/native/Makefile and change the following:

  • Change the  LIBPATH variable to -L/usr/local/lib
  • Change the CFLAGS to -fno-common
  • Change $(CC) -shared (OBJECTS) $(LIBPATH) $($(TG)_LIBS) -o $@ to
    $(CC) -bundle -flat_namespace $(OBJECTS) $(LIBPATH) $($(TG)_LIBS) -o $@

Now, you can run make, and you will end up with and libreadline-java.jar.

Rename to libJavaReadline.dylib, and you are set. You can now go on to use libreadline-java in your target application, like Henplus, as you expect.

I’ve made a libreadline-java-0.8.0-osx-lion.patch script for those who would rather use the patch command rather than doing it by hand.

These are a few of my favorite apps…

I’m sure everyone’s got ’em. When you get past the facebook an twitter clients, what are the iOS apps you really find useful? Here’s a bunch of mine. What’s yours?

Air Video – This is a really cool app and accompanying server to let you play your own videos, anywhere. No iCloud necessary! Dup your DVD collection and watch it where ever an whenever you want. I’ve got about 2Mbit outbound bandwidth which is more than enough for nice looking video on any iOS device.

Xfinity TV – for as much as we all hate Comcast, this app fills some glaring voids in their on-screen UI. Search listings easier. Better organization to on-demand offerings and even control your TV from the app.

IM+ Pro, Touch Term, Pocket Cloud – this trio of apps is a pretty good set to have for any system administrator, giving you instant messaging, SSH, VNC and RDP clients.

Wikinvest, Mint – both great tools to manage your investments and personal finances. Tie in pretty well with their respective sites.

Mobile Mouse – another client/server pair. Gives you a virtual keyboard & mouse so you can control your Windows an Mac OS systems.

Reeder – nicely packaged Google Reader enabled feed reader. This replaces Mobile RSS which, up until iOS 5 was my reader of choice.

Push Mail – great app/service to allow you to send Notifications to your phone via email. You can configure different profiles for different messages to display different email fields in the Notifications.

IProxy – happen to still have an unlimited data plan and a developer device? Use iProxy instead of enabling full fledged tethering.

Flipbook – yeah the UI I’d kind of gimmicky and hard-cores will probably get their news elsewhere but it’s a nice read for standing in the grocery line.

MySQL user@localhost and user@%

Its always bugged me that I have to create two user accounts in MySQL, one with a wildcard host (%) and another for localhost.

Finally investigated it enough to understand why. Briefly, three things are going on.

  1. First, MySQL takes a non-wildcard host as precedence when looking for user accounts
  2. The mysql_install_db script create an anonymous user account ”@’localhost’
  3. The anonymous use account takes precedence when the hostname matches.

So the net effect is that when you connect locally, in general MySQL will use the anonymous account to authenticate to, no matter what you specify as a username, if the only account that matches the username also has a wildcard host field.

This is explained in the MySQL reference manual:

It is necessary to have both accounts for monty to be able to connect from anywhere as monty. Without the localhost account, the anonymous-user account for localhost that is created by mysql_install_db would take precedence when monty connects from the local host. As a result, monty would be treated as an anonymous user. The reason for this is that the anonymous-user account has a more specific Host column value than the 'monty'@'%' account and thus comes earlier in the user table sort order.

I’m not quite sure why an anonymous user is created by default, some documentation suggests that on Windows, its used as the ‘root’ account.

Interestingly, although you need both user@localhost and user@%, you don’t also need to grant privileges to both users. It appears to be sufficient to grant privileges to user@% and those privileges will also be used for user@localhost (this was another source of confusion for me.)

If it bugs you so much, there are other options

  • remove the anonynous account(s) entirely. I’ve seen references in a couple of places suggesting this is an okay thing to do
    delete from mysql.user where user = ''
  • rename the anonymous account(s). If deleting them seems a little too drastic
    update mysql.user set user = 'visitor' where user = ''

I’m not sure which option I like, but now I finally understand MySQL authentication better.