MySQL user@localhost and user@%

November 2, 2011 Leave a comment

Its always bugged me that I have to create two user accounts in MySQL, one with a wildcard host (%) and another for localhost.

Finally investigated it enough to understand why. Briefly, three things are going on.

  1. First, MySQL takes a non-wildcard host as precedence when looking for user accounts
  2. The mysql_install_db script create an anonymous user account ”@’localhost’
  3. The anonymous use account takes precedence when the hostname matches.

So the net effect is that when you connect locally, in general MySQL will use the anonymous account to authenticate to, no matter what you specify as a username, if the only account that matches the username also has a wildcard host field.

This is explained in the MySQL reference manual:

It is necessary to have both accounts for monty to be able to connect from anywhere as monty. Without the localhost account, the anonymous-user account for localhost that is created by mysql_install_db would take precedence when monty connects from the local host. As a result, monty would be treated as an anonymous user. The reason for this is that the anonymous-user account has a more specific Host column value than the 'monty'@'%' account and thus comes earlier in the user table sort order.

I’m not quite sure why an anonymous user is created by default, some documentation suggests that on Windows, its used as the ‘root’ account.

Interestingly, although you need both user@localhost and user@%, you don’t also need to grant privileges to both users. It appears to be sufficient to grant privileges to user@% and those privileges will also be used for user@localhost (this was another source of confusion for me.)

If it bugs you so much, there are other options

  • remove the anonynous account(s) entirely. I’ve seen references in a couple of places suggesting this is an okay thing to do
    delete from mysql.user where user = ''
  • rename the anonymous account(s). If deleting them seems a little too drastic
    update mysql.user set user = 'visitor' where user = ''

I’m not sure which option I like, but now I finally understand MySQL authentication better.

Categories: Computing

NetBeans code templates for Builder Pattern

October 29, 2011 Leave a comment

Here’s a couple of handy code templates (short cuts) for developing classes that use the Builder pattern.

This first one creates the Builder inner class itself. I’ve used the abbreviation “pscB” for this (Public Static Class Builder) –

public static class Builder {
    public Builder() {
    public ${Type} build() {
        ${Type} result = new ${Type}();
        return result;

And this one will simplify creating a “setter” field within the Builder, I use the abbreviation “psBB” for this –

public Builder ${field}(${Type} ${field}) {
    this.${field} = ${field};    
    return this;
Categories: Computing

IOps calculations

September 19, 2011 Leave a comment

So I got a disk array and I want to know what sort of IOps workload it can support.

Articles like IOps? provide a pretty standard formula I have seen before, but only answers the question “Given a required workload IOps for an application, how big does the array need to be to support it?” This equation is

Array IOPS = (Workload IOPS * Read Workload Percent) + (Workload IOPS * Write Workload Percent * RAID IO Penalty)

However, sometimes you want to answer the opposite question “Given a total number of IOps for an array, how much workload IOps can it support?”

The answer is really just to solve for Workload IOPS in the above equation. To save you the math,

Workload IOPS = Array IOPS / (Read Workload Percent + Write Workload Percent * RAID IO Penalty)

This could have been pretty handy a couple of years ago when I was trying to build a private cloud, to help me understand how much workload the arrays we used would support  – and therefore figure out how many virtual machines we could run.

Categories: Computing

Henplus with Java-Readline for OS X

September 14, 2011 9 comments

UPDATE: The procedure below is not entirely correct. It will work, but its not the most straight forward, and it probably won’t get you what you are really looking for. I have written a new procedure for building Java-Readline and GNU readline for OS X. Please use these instructions instead.


So, I found it time to revisit an old friend, HenPlus since I’ve been mucking around in a few databases lately.

The wrinkle this time, OS X. The main problem being the JNI interface to the GNU Readline library Java-Readline does not compile all that well on OS X.

Doing a bit of searching I found a few articles describing the steps to compile it, but they are slightly out of date for OS X 10.7 Lion, so I figured I’d put in my two cents.

For starters, Nathan Toone’s article Java-Readline on Mac OS X Update gets pretty close, but things have changed since he wrote it in 2006. So here’s my redux based on it.

Note that in my case, I did not care about building a universal binary (if that’s even possible on Lion) so I didn’t have to make as many changes as he did.

Tweaking Java-Readline


In the source root, edit Makefile and make the following changes:

  • Below the line # Operating system dependent
    Add JAVA_HOME = `/usr/libexec/java_home`
  • Clear out the JAVANATINC variable
  • Change the JAVAINCLUDE variable
    to /System/Library/Frameworks/JavaVM.framework/Headers
    (This is needed to find jni.h)

Next, edit src/native/Makefileand change the following:

  • Clear out the LIBPATH variable
  • Change the CFLAGS to -fno-common -DMAC_OS
  • Change $(CC) -shared (OBJECTS) $(LIBPATH) $($(TG)_LIBS) -o $@ to
    $(CC) -bundle -flat_namespace $(OBJECTS) $(LIBPATH) $($(TG)_LIBS) -o $@
  • Change the JavaReadline_LIBS variable to be -lreadline -ltermcap

Now, you need to edit src/native/org_gnu_readline_Readline.cand make the following changes:

  • On lines 98, 114, 213, and 224,
    change #ifdef JavaReadline to #if defined JavaReadline && !defined MAC_OS
  • On lines 216, 235, and 475,
    change #ifdef JavaEditline to #if defined JavaEditline || defined MAC_OS

Now, you can run make, and you will end up with and libreadline-java.jar.

Rename to libJavaReadline.jnilib.

I’ve made a patch script of all the changes to save anyone some effort in editing all the files by hand. Simply run

  • patch -p1 < libreadline-java-0.8.0-osx.patch

HenPlus Notes

Once you have compiled Java-Readline, you can compile HenPlus as described. A couple of things to note though

  • Copy libreadline-java.jar from Java-Readline into <INSTALLDIR>/share/henplus
  • Make <INSTALLDIR>/lib and copy libJavaReadline.jnilib into it
  • Change all occurrences of LD_LIBRARY_PATH in bin/henplus to DYLD_LIBRARY_PATH

And from there you should be set.

Categories: Computing

Sending email as multiple identities with Gmail & Thunderbird

August 22, 2011 1 comment

I just figured this one out, and before I forget it.

I’ve been a user of Thunderbird’s Multiple Identities feature for a while. It allows you to send mail from a single account using multiple identities. So, for example, I have my personal email address for the majority of my email, but I have a second address which is really an alias for my wife and I which we give to day care, schools, etc. And when communicating with said entities I want the email to look as if it came from the alias.

I’m not sure when it changed, but recently even though I have set everything up the same way in Thunderbird, when the email was delivered to the receiver, it would show up as my primary address. Not good.

I immediately suspected the recent releases of Thunderbird, 5.0 & 6.0, but a few searches and I found nothing.

The only other change was switching over to Google Apps to host my personal domain email, I had done that a while ago though. But still poked around anyways.

And wouldn’t you know I found two settings, either one or both of them fixed the problem.

These are both in Mail Settings -> Accounts (note Mail Settings is under the Gear icon in the upper right)

  • First I added the email alias to my account by clicking on Add another email address you own link.
  • Second I set When replying to a message to Reply from the same address the message was sent to.

Not sure which of these fixed the problem, I suspect the second, but either way, now the receiver correctly sees the alias when I choose  to send mail as that identity.

Categories: Computing

SSL Termination, Load Balancers & Java

August 22, 2011 5 comments

So you got this killer Java web app you are developing, or you are deploying some super cool off the shelf application, and you decide to use some sort of SSL Accelerator to offload your HTTPS traffic, and it looks something like this:

Only things aren’t going so well. Maybe all the generated URLs for HTTPS connections are coming out as plain HTTP. Or your web app is attempting something clever and ntify when specific requests are plain HTTP and redirect them over to HTTPS, however even after redirection the app still tries to redirect again.

What Gives?

As you have probably guessed, the introduction of the SSL Accelerator has caused a problem because now all the incoming requests to your web app are coming in as plain HTTP, because HTTPS was terminated upstream somewhere and so your application server never knows about it.  In Java web applications, this generally manifests itself as HttpServletRequest.isSecure() always returning false, wreaking havoc in its wake. The Servlet machinery makes lots of decisions based on the value of isSecure(). All the way down to when you are defining security-constraint elements in web.xml and setting transport-guarantee to CONFIDENTIAL.

So What’s the Fix?

I’ve encountered this quite often, with various SSL Accelerators, Load Balancers and App & Web Servers. In general, the idea is to maintain separate paths for the requests all the way through to the app servers, like so:

In doing so, you can typically configure the back end web or app server to correctly identify the originating scheme of the request (HTTP or HTTPS).

Any Examples?

The Easy – Plain Ol’ Tomcat Backend.

By far the easiest container to work with is Apache Tomcat. The options available in a Connector clearly had this type of configuration in mind. Specifically these are

  • scheme – this tells Tomcat which protocol to use to create links, either http or https.
  • proxyPort – this tells Tomcat what to use as the real originating port, instead of the port the connector is listening on.
  • proxyHost – similarly, this tells Tomcat what to use as the real host name, instead of the host Tomcat is running on.
  • secure – this tells Tomcat whether the Connector is secure or not.
  • SSLEnabled – whether to configure the connector for SSL/TLS or simply Plain HTTP.

So, in the case where you are only using Apache Tomcat in your platform,

The correct configuration can be accomplished by creating two Connectors, one for port 8080 and one for port 8081. These would look like

     Define the connector on port 8080 to handle
     originating plain HTTP requests. Nothing
     special here. Make sure to set proxyPort
     and proxyHost to the appropriate values
     for your web site.
<Connector port="8080" protocol="HTTP/1.1"
           maxThreads="150" clientAuth="false"
           scheme="http" secure="false"
           proxyPort="80" proxyHost=""

     Define the connector on port 8080 to handle
     originating HTTPS requests. Here we
     set scheme to https and secure to true. Tomcat
     will still serve this as plain HTTP because
     SSLEnabled is set to false.
<Connector port="8081" protocol="HTTP/1.1"
           maxThreads="150" clientAuth="false"
           scheme="https" secure="true"
           proxyPort="443" proxyHost=""

And then, of course, configure your load balancer (and SSL Accelerator) to direct traffic to the appropriate ports.

A Little Harder – Tomcat with Apache Httpd

Introducing Apache Httpd into the mix really all that much harder than using simply Tomcat. Here’s the desired traffic pattern.

You may notice that HTTP protocol stops at the Httpd and then from Httpd to Tomcat the protocol switches to AJP. And then there’s the two streams coming into Httpd, but lets tackle Tomcat first. Its actually not very different at all from straight Tomcat.  The only difference is switching to AJP/1.3 for the protocol.

<Connector port="8009" protocol="AJP/1.3"
           maxThreads="150" clientAuth="false"
           scheme="http" secure="false"
           proxyPort="80" proxyHost=""

<Connector port="8010" protocol="AJP/1.3"
           maxThreads="150" clientAuth="false"
           scheme="https" secure="true"
           proxyPort="443" proxyHost=""

Now, for Httpd. This is solved by setting up a separate virtual servers for each port, and passing them off to the appropriate Tomcat Connector using mod_jk and separate worker pools for each Tomcat Connector.

JkWorkersFile /etc/httpd/conf/

<VirtualHost *:80>
    DocumentRoot /var/www/html

    JkMount /webapp/* tomcat-8009


<VirtualHost *:81>
    DocumentRoot /var/www/html

    JkMount /webapp/* tomcat-8010


And now the file.

worker.list= tomcat-8009, tomcat-8010



And that’s it. The one caveat here, is that although originating from HTTPS and plain-HTTP follow different paths even through Httpd, Httpd doesn’t really know that. So if you are doing anything interesting other than serving static content from Httpd, you may have an issue, for example if you are running PHP or something else that may need to care, you may be out of luck.

The Difficult – Less Configurable Application Servers

The Tomcat team as really taken a lot of time to provide a highly configurable platform, but what if this is not your target deployment platform, and you are faced with attempting to do SSL Termination on a platform that isn’t so helpful. One example that I’ve got a bunch of experience with is what was called Sun Java System Web Server and now Oracle iPlanet Web Server. Although I did enjoy working with it, the platform did not provide such an easy method of integrating an external SSL Accelerator.

But all is not lost. Fortunately Java Servlets version 2.3 and newer provide some facilities that can help. Servlets 2.3 provide two interesting features, Filters and Wrappers.

Filters provide a method for you to do some pre-processing of requests before your Servlets, JSP and anything else is invoked (they also can do post-processing but we don’t need that here.)

Wrappers provide a method for you enclose a Request or Response in your own object.

The two, together can help you fake the Java machinery into setting HttpServletRequest.isSecure() and HttpServletRequest.getScheme() into returning the correct values depending on the original protocol.

Lets look at creating a Wrapper first. Its as simple as

public class FakeSecurityRequestWrapper extends HttpServletRequestWrapper {
    public FakeSecurityRequestWrapper(HttpServletRequest request) {

    public boolean isSecure() {
        return true;

    public String getScheme() {
        return "https";

Pretty trivial. Now to create a Filter to inject the wrapped Request into the machinery.

public class FakeSecurityFilter implements Filter {

    public void doFilter(ServletRequest req, ServletResponse res,
        FilterChain chain) throws IOException, ServletException {

        FakeSecurityRequestWrapper fakereq =
            new FakeSecurityRequestWrapper(req);
        chain.doFilter(fakereq, res);

Now just plumb this up in your Web App’s web.xml as the first filter in the chain, and you are there. Well, most of the way there. there’s the opposite problem now. Every request, no matter what the originating protocol, appear to be HTTPS. In a mixed protocol environment, that is still a problem.

No worries, we can do it! Yes, with a little more tweaking, it should be possible to support a mixed protocol environment.

And its with a little trick from the load balancer or SSL Accelerator. Most of them that I have run into allow you to modify the request as its passing through the appliance. To the idea is to insert a Header into the HTTP request for request that have gone through the SSL Accelerator. A typical flow would look something like this.

I won’t go into the configuration on the SSL Accelerator/Load Balancer end, but lets look at enhancing the Filter to handle mixed protocols. In this example, we’ll assume the Accelerator/LB sets the header X-Original-Protocol to https for requests coming through it. The doFilter() method becomes

    public void doFilter(ServletRequest req, ServletResponse res,
        FilterChain chain) throws IOException, ServletException {

        String origProto = req.getHeader("X-Original-Protocol");
        if (origProto != null && origProto.equals("https")) {
            FakeSecurityRequestWrapper fakereq =
                new FakeSecurityRequestWrapper(req);
            chain.doFilter(fakereq, res);
        } else {
            chain.doFilter(req, res);

Voila. Now the Java Servlet machinery should be handling a mixed protocol environment just fine. This technique should work with just about any web container, be it WebLogic, JBoss, SJSWS/IWS or GlassFish.

Its arguable whether this method is more straight forward than the other previous methods, but it does have it down sides, which for me, make it less desirable. It does require a little programming, and needs actively integrated into each web app you are deploying and I don’t like having to muck with 3rd party web.xml files if I don’t need to, and for that I leave this as a last resort option.

I’m sure there are other techniques for working with SSL Accelerators, these are just the methods that worked for me so far. Hope others find them useful.

Categories: Computing

The monkey on your back – Disaster Recovery

May 25, 2011 Leave a comment

Its seems that more and more projects lately have had the mandatory line item “Disaster Recovery” as part of their requirements. Whenever it comes up I try only to pay as much attention to it as necessary because to me it is one of those problems that are not solvable within the specified constraints – that is, the budget you are given. It can become incredibly frustrating if you take it too seriously because, although everyone requires a Disaster Recovery plan, no one can quite define it beyond that. Ask a Business Analyst or VP and they’ll spout off some impossible number of 9’s of availability, period. And if you allow us engineers to define it, we can come up with a myriad of opinions, none of which is “achievable”.

That said, I want to put down some thoughts on the subject. And perhaps some final suggestions/recommendations.

What is Disaster Recovery?

Seems like a good place to start, with a definition. In my book, DR has always been a plan for what to do when something, incredibly improbable, that is out of your control happens, which effects the ability of the masses to reach your applications/services. These are generally classified as problems with “location”, power, cooling & connectivity. Since most folks no longer run their own data center, they are at the mercy of the hosting provider they select. Unforeseeable acts of some sort, include things such as an act of nature – a tornado obliterating the data center, or someone zapping the power grid causing you to go dark, and even to the back-hoe tearing up a bundle of fiber. These are all extraordinary events but none the less, it would seem that they are something we need to plan for, and is the basis for DR.

Its worth it to note that Disaster Recovery typically excludes everything within your control of your piece of the data center. So local networking issues, local hardware issues, local storage issues, local application issues are not part of the plan. After all, this all should be built highly available in some fashion anyways, already.

Disaster Recovery Models

Lets go into some details about the different options I’ve encountered when discussing DR. In general, the idea is for you to maintain a presence in more than one geographically dispersed data center, with the thought that any extraordinary event in one data center won’t occur in the other. Because they are geographically dispersed, the weather conditions should be different enough not to cause a simultaneous act of nature. And  same goes since they are on different power grids, and have different network connectivity into the sites. So what are our options?


This is the most operationally expensive and architecturally complex, so lets look at this first. The idea is that you maintain your applications/services in two locations and have them both actively serving production requests. Although this gives you the most return on your investment, since you are using all your assets all the time, it does present a bunch of challenges.

Operationally, your team must maintain the applications on essentially twice the amount of systems as needed to handle all your production traffic, such that each data center is running at only 50% capacity. This is necessary since you have to deal with the extraordinary case when an “event” occurs and the remaining data center that is functioning begins to receive 100% of the traffic. So, since you are maintaining double the machines, you’ll probably need to increase your operational staff to be able to maintain those additional assets.

Architecturally, you must now deal with a whole slew of data management issues. If your application is any bit interesting, it’ll be receiving data, and to both data centers. How do you do that and still maintain some coherence? You can decide that one data center is the master if its available, and always update and query data from it, while syncing the data from the master to the slave. When a DR event occurs, the application must have some logic to decide it now must fail over to the secondary data providers. But now you must be careful not to automatically fail back with out making sure the original master data source was up to date. An ugly problem.

Another option would be to build a data services tier that updates both data sources with the same data no matter where the tier is, and similarly retrieves data from both sites, combines it intelligently somehow, and returns it back to the caller. You still have the same problem of what to do when you recover from the event, and now you have to deal with ensuring connectivity between data centers at all times. Oh, and since they are geographically dispersed, you’ll probably have latency issues to deal with, limiting how quick you can update and retrieve data, and possibly reducing customer satisfaction.


This option can alleviate much of the architectural complexity of attempting an Active/Active configuration, but is still pretty operationally expensive. As the name implies, this model has all requests being serviced out of only one active data center at a time. The standby data center is typically configured so that there is a continuous synchronization (or push) of data into it. There are lots of technologies available for synchronizing data so that shouldn’t incur too much operational overhead.

However, operationally, you still have to maintain two instances of your applications/services in two data centers and if you want to be able to handle 100% of the traffic, twice as many systems. This is nominally less expensive than the active/active model since your operations staff really has to critically deal with only one live data center at a time. But that is the double edged sword. Since they are focused on  the live data center, it is likely, with out right set of disciplined folks, that the standby data center will grow mold. The application won’t be upgraded at the same rate as the live instance, and so all hell breaks loose during a DR event.

And since its a standby data center 99.9… (ad nauseum) percent of the time, it is costing you quite a bit to just heat and cool air. You could look at recapturing some of that investment by looking for other uses of the standby, for example, putting non-production instances of the application, like test and staging instances in the standby. If you’ve built things around some virtualization technology like VMWare vSphere, or XEN, or (gulp) HyperV, you could cleverly manage your computing resources so the standby production virtual machines are given the resources they need when even occurs. Depending on the chosen technology, this could be easier or harder to do.

Active/Hot Backups

This model is the least expensive, and probably the least complex, but as a result there is less possibility of providing illusive N 9’s of availability.

With this option you do two things. First, as with active/standby, you build full data sources at each data center, and configure a synchronization from one to the other. This gets your most critical piece of the picture in two place always. Or with minimum data loss at least.

You also periodically backup your systems and push the images to the backup data center. You do this in such a way that you minimize the number of systems you need to keep the backup data. Perhaps all you need is a replication of your shared storage, and one or two systems to manage the incoming backups. Using a virtualization technology can help here too. The right technology can help you snapshot, and copy images to your backup datacenter.

Along with all this, you work on agreements with your hosting provider to have systems available for installation and configuration within a specific time period. With this, you build an activation plan. What to when a DR event occurs. This would things like working with your provider to setup and configure the hardware, re-hydrating all those backed up virtual machines, testing and go live procedures.

Clearly, this is not a model to maintain 9’s no matter what the reason, but it is a cost effective option which limits the amount of daily operational complexity a team must deal with.

Events Classes

I think its worthwhile to look at the different event classes in a little more detail as this can provide input on which model to choose.

Network Events

This is the case where data center connectivity goes dark, for whatever reason. A back-hoe takes out some fiber. A backbone router crashes. A system administrator funks up some routing tables. China hijacks the Internet Traffic.

But here’s the deal, just as you have build your data center space so its network is highly available, any good hosting provider should do the same though multiple peering arrangements, fiber entering multiple locations in the facility, and so forth. And they should be able to redirect traffic if such an event occurs, and maybe so quickly you don’t even notice it, but even if you do, certainly quicker than you can do something about it with any of the models except perhaps the active/active model.

Power Events

These can occur when a transformer at the municipal utilities blows up, and so forth, which would cause loss of power to your hosting provider. However, same deal, hopefully you have chosen a hosting provider which backup power capabilities. There is still no guarantee that backup power actually goes online appropriately, as happened at 365Main in 2007. As mentioned in the report, power was restored 45 minutes later.

Which begs the question, except for the active/active model, when to you decide to invoke an DR activation plan? Is it time based, governed by the 9’s? Is it by the feel of the severity of the event? Something else? And how do you weigh that against the cost of performing a fail-back or re-synchronization after the primary data center is back online?

Cooling Events

Is this really something to worry about? Any facility should have an N+1, N+2 or better set of chillers. If they don’t you shouldn’t even consider hosting with them.

Environmental Events

These are the act of nature type of events. From earthquakes and tornadoes, to floods, fallen trees, and so forth. These are events where you have very little probability that you will even be able to get any data back from the facility, ever. This, I think, is the heart of the necessity for Disaster Recovery.

It means, minimally, you must have all your data, your crown jewels, at another location, whether its at another data center, on your laptop, in a thumb drive, or something else. It just needs to exist in two places, minimally, period, no matter how remote the possibility.

There are things you can do to mitigate the possibility of environmental events, too, which can help you decide on an appropriate DR model. Survey the environmental risks of the locations of the hosting providers. For example, Amazon has a huge data center in Oregon which is free from tornado threats, and probably earthquakes too. Flooding could be a concern, however. Then there’s Switch, with a SuperNAP in Nevada, although cooling could be a non-problem, its likely free of the majority of acts of nature. There’s Chicago, San Antonio, Utah, lots of other locations with providers, too.

Some Thoughts

I started this out by mentioning why, besides all the technical challenges, Disaster Recovery is such a frustrating topic. Its generally poorly defined, and everyone has their opinions on what it really means. So I think the first thing to do if you are stuck in this situation is to define your own very specific and well defined meaning to it and make sure everyone agrees, before you even consider any implementation plan.

It is my contention that DR is something quite extraordinary, and so perhaps one should deal with it in an exceptional way. For example, maybe, instead of thinking about one string of 9’s for uptime, you consider two. Under normal operation, you impose a tighter tolerance for downtime, or say 99.999% uptime, which would allow only 5 minutes of downtime over the year. However, if you are ever so unlucky to encounter a event, perhaps agree on something a little more lenient, even say only 99.9% uptime gives you almost 9 hours of downtime. It seems like a lot. It is a lot. But considering, in my definition DR is about dealing with environmental events, some leeway may be in order. Especially considering the mean time between events, which depending on the data center location, could be decades.

Once you have agreed on availability numbers, this, more than anything else, drives your implementation plan. If you are allowed only a small amount of downtime, clearly you must bite the bullet and go with an active/active solution. But you’ll need to budget for it. On the other hand, if you are given more leeway, perhaps an active/back up strategy makes more sense. You need to do this minimally anyways, as the data must be duplicated geographically.

Don’t forget to run the numbers to compare the lost revenue during downtime against the cost of the solution. You may find that the cost of implementing a sophisticated DR solution is more than the revenue that would have been lost during the downtime.

There are many reasons for building complex geographically dispersed, clustered applications. Whether disaster recovery is a good reason enough is questionable but it really depends on the specifics. Hopefully this will help folks begin those discussions.

Categories: Computing