We did some bits and pieces automation on one of our projects over the last few months. It impressed some managers at our customer, who then asked to start a separate project to do end-to-end automation.
We just finished the first iteration. Now, once you have stuff set-up and configured, the following things happen every night:

• the application is built and packaged.
• The install elements: war, database dump, sql scripts, install instructions, release notes get copied to a release folder. The release is named by the timestamp.
• The application is installed and configured on a remote machine.
• The application is started on the remote machine.
• Some automated code review tools are run, and reports generated in a nice consolidated format.
• Junit tests are run on the deployed application, and code coverage reports are generated.
  

So, the next morning, when the team comes in, they know if some things broke. They can also spend 15 minutes running through the code review reports and test the application for sanity.

If a formal release needs to be made, that is also a single command : you just specify the tag name, and all of the above happens. In addition, source also gets tagged.

What's next: integrating canoo web test or some other similar tool with the deployed application, so that the nightly build can also do the sanity tests.

Finally, we are getting close to becoming the cool dudes of automation!

An excellent post on how to investigate these most tricky of bugs: ones that take place once in a while, and you cannot figure out how to reproduce them. Basically for testers, but very useful for developers as well.

Of late, as I've started reading blogs, I've seen a number of discussions on checked v/s unchecked exceptions, including in the context of C# v/s Java. Some well known people thinking Java, such as Bruce Eckel, also think checked excptions bring more trouble than they are worth.

Yesterday I was talking to my new colleague who comes from Microsoft, and he pointed me to an interview with Anders Hejlsberg, who created the C# language. Here he talks about why they chose not to include checked exceptions in C#. A number of other major arguments are similar. ( I'm too lazy to go back and find the links to all of them, so believe me, or search yourself :-) ). Here is a summary of my understanding of the argument, which is essentially a practical rather than conceptual argument:

• As you move up the hierarchy of methods in an app, or if you are building a huge system, the number of checked exceptions you need to catch keep increasing.
• You don't care about most of these exceptions.
• As a result, writing catch blocks for each of them irritates one no end.
• Therefore, most people just write

catch(Exception e){
\\ error handling
}
with the intent to come back later, which never happens.

• Therefore, checked exceptions serve little useful purpose, hence they should go.

If that's really a correct summary, (which I hope it is not), then I don't think I need to state where they are wrong. But let me do it anyway:

• If it's a feature that most people ignore, that doesn't mean it is not useful. Without the feature, those people won't do a better job of handling exceptions. With the feature, at least some people are doing a good job. Even they will not be able to do a good job with unchecked exceptions.
  
• If your methods are throwing a large number of exceptions, and/or you catch them with the block above, you are doing sloppy programming. Sloppy programming can't be used as a reason against a feature. ( Yes, it can be used as an argument to try and improve that feature.) Moreover, if your programmers are sloppy, they will probably do a worse job without checked exceptions.
• If people won't use exceptions properly, which the compiler partly forces them to, they certainly won't document their code properly, which is the only thing you can rely on when you don't have checked exceptions.

Having said that, the point about the empty catch block is one that I can identify with - have seen lots of people do that. But I think the solution is rather simple, although one that cannot be enforced by the compiler:

• for your application, create two base exceptions: MyAppApplicationException as a checked exception, and MyAppSystemException as an unchecked exception.
• subclass MyAppApplicationException when you have a new handlable error situation.
• subclass, or use MyAppSystemException when you have a new non-handlable error situation. Also use it to abstract out all lower level system exceptions.
• Client code: catch the specific instances of MyAppApplicationException that you can handle, and then, if there are any more, catch MyAppApplicationException. These two together mean that you catch only those that you care about, and can ignore the rest.
• Enforce this through the use of automated and manual code reviews.

Another argument Anders used is that checked exceptions break versionability - that is, if I need to add an Exception in a later version of my API, then I will break backwards compatibility.
I'm a much lesser programmer than Anders, so I'd take it as given, except that I don't see his point at all. Checked exceptions are part of your method signature, because they are part of your contract with client code. If you come across a need to add or remove a checked exception, you are altering behavior of your method. How is that conceptually different from adding or removing a parameter?
Let's assume for a second that there might be situations where you can alter the Exception list without altering behavior. You are probably doing it based on complaints from your users, so they will probably need the change anyway. Of course, they will need to modify code, but that would be for a real pressing need, right?
Even if you do need to add Exceptions that should not require changing client code, using the BaseException approach allows you to do it: all client code only needs to always have caught the BaseException.

Assuming that we use the BaseException approach, are there still any problems with Checked Exceptions?

I got into a bit of an animated discussion at work last week about Load Testing, and what it means. The other people involved were senior QA and delivery folk. We couldn't agree on a common understanding of the concepts, forget terminology. A search onWikipedia also failed to provide a clear explanation.

Here's what I think about it. To me, there are three concepts, and between them, they cover all aspects of testing a system under load. Everything else is a combination of these. The terms I use for them may not be the same as the terms other people use, or perhaps they use other terms for these concepts, but let's focus on the concepts. (I'm using web based application terminology here, but I guess it should be similar in other situations.)

• Performance or Throughput: You load your server to various number of users, and measure the throughput, or response. Again, normally done in pages served per second, or average page response time.
  
• Scalability: Define the lowest throughput that is acceptable to you. Then load the server (for instance, by increasing the number of users) till the throughput starts going below the defined one. The number of users logged on at this point define the scalability of your system in terms of users.
  
• Reliability: Let the system run under extreme load ( same as above), for a large number of computations. In the webapp world, use a scenario that visits all your pages, and let it run for a long long time: time really meaning pages served in this case. Look at two different measures:
• number of faults occurred. This gives you a reliability measure by the standard definition.
  
• amount of server resources consumed. ( RAM, disk space, DB connections, file handles etc.) This should stabilise after some time, and stay there. If any of these keep growing, there's a resource leak.
  

Here are a couple of (horribly drawn) pictures. But I hope they are allowing me to put my point across:-)





These are the basic concepts. A number of variables vary based on your goals and your application, but the concepts remain the same. Here are some examples of things that will vary:

• number of pages in reliability: for an intranet performance review app for a 200 people company, 50,000 requests would probably be a lot. For an amazon.com type of application, we're probably talking hundreds of millions or billions.
• the server resources you measure.
• the metric you use to measure througput.
• How do you measure 'concurrent users'. For instance, you might have your users wait 1 minute between actions, or not wait at all.
• Your acceptable throughput threshold.
• Number of concurrent users you want to measure throughput for.
• Server configuration
  

When executing these tests, typically you will do multiple iterations, across multiple installations, and do all of these at the same time.

One of the item the others did not agree on was the y-axis in the reliability test. They didn't agree that number of pages served was the right number there. I believe that number of pages is the only term one can use there - the x-axis is upto you. And here's why:

Ultimately, all your measures are relative to the agent driving the application, because the agent is responsible for getting your functions called. In a webapp where everything is done via a web page, this agent is your end user, who only makes page requests. Hence the only true measure of 'life' of the application is the number of pages served.

I find quite a few people talking about load testing, including our customers, without really being clear about what they need. In this particular case, they started saying 'load test'. I put across this definition, ( or rather, had the on-site guy do it) and they came out with: well, we can do those as well, as long as they are useful! I doubt he had any better idea of what he wanted even then.

At one of my previous jobs, at an enterprise product development organisation, we did all three - for reliability, we ran through 500,000 pages, which took us a couple of days. For throughput, we wanted at least 0.1 seconds of page response time. Scalability we just measured in throughput terms.

Does this sound like a reasonable definition of load testing? Am I missing something?