Lasse Westh's Idle Thoughts

Awesome Gmail

2011-08-11T05:54:00.001-07:00

HTTP tidbits

2011-01-20T04:00:00.000-08:00

Reading up on RFC 2616, section 6, some interesting details that I want to keep:

reason phrases [...] MAY be replaced by local equivalents without affecting the protocol

Could come in handy for debugging?

HTTP applications are not required to understand the meaning of all registered status codes, [...]. However, applications MUST understand the class of any status code, as indicated by the first digit, and treat any unrecognized response as being equivalent to the x00 status code of that class

Interesting edge case. And HTTP status codes are arithmetic types, not enums.

Ninite

2010-12-08T03:09:00.000-08:00

Note to self: http://ninite.com/ works really well.

Thanks Carlos!

Brief is not simple

2009-10-06T15:09:00.000-07:00

I am always keen to reduce the world into simple metaphors and proverbs. Big fan of "the emperor has no clothes" for example (and not only because H. C. Andersen was danish!), as it can be applied to so many things. Particularly, while we battle for the hearts and minds of the IT industry, advocating simplicity and denouncing ceremony, it seems to come in handy time and again.

Anyway, I was reading about Guava (thanks for the link Simon!), and stumbled upon one that was missing from my library: "brevity is the soul of wit". In the context of source code, you can imagine a snooty Perl hacker uttering this phrase after writing a particularly pleasing regular expression that no-one will ever be able to decode. However, I cannot think of anything less appealing in source code that brevity or wit. As a way of communicating, code is really best expressed simply, not briefly or cleverly.

Of course, outside the world of code I am a huge fan of folks who can be concise, thus.

Watch your dependencies

2009-07-14T03:44:00.000-07:00

Dependencies are a headache. They cause all sorts of problems. So you should keep aware of them.

Tools are a-plenty, for many languages, but here is a good starter for your favourite open platform: Byecycle

Other tools do more, but are also more complicated and hard to interpret. If you want a gentle introduction, start with this little visual tool.

It can help you find dependency cycles

It can help you spot classes that do too much

It simply gives you a 30.000ft view of how your code looks, something which is difficult to get from text files on even a medium sized code base.

I like Apache HTTP Client over java.net.HttpURLConnection

2009-05-11T01:49:00.000-07:00

My last project involved CAS and a lot of integration points, so we ended up with lots of HTTP traffic as part of our integration tests.

We started having problems with our build, spurious exceptions would break it, but we couldn't reproduce the problem. It was not until we did some rudimentary load testing that the problem surfaced: Persistent HTTP connections.

Problem was, we had written our tests using java.net.HttpURLConnection like this:

try {
  HttpURLConnection connection = (HttpURLConnection) new URL("http://localhost:8080").openConnection();
  connection.connect();
  assertEquals(HttpURLConnection.HTTP_OK, connection.getResponseCode());
} catch (...){
  ...
}
finally {
  connection.disconnect();
  connection.close();
}

Turns out, because we didn't care about the response body and thus never bothered reading it, and that would tie up TCP connections. The more tests we ran, the more likely it became that we would see an exception like this:

Exception in thread "main" java.net.BindException: Address already in use: connect

So while we thought we were being diligent by writing a finally block to disconnect and close the connection, it turns out that is completely superfluous. What we would have to write was this:

try {
  HttpURLConnection connection = (HttpURLConnection) new URL("http://localhost:8080").openConnection();
  connection.connect();
  assertEquals(HttpURLConnection.HTTP_OK, connection.getResponseCode());
} catch (...){
  ...
}
finally {
  while (connection.getInputStream().read() != -1) ;

  connection.disconnect();
  connection.close();
}

(Actually, that just works for happy HTTP responses, as this library will throw an exception on 4xx and 5xx's!? But lets not digress...)

Now that's a lot of code for something so simple - look at Apache HTTP Client for the same task:

HttpClient httpClient = new HttpClient();
GetMethod method = new GetMethod("http://localhost:8080");
assertEquals(HttpStatus.SC_OK, httpClient.executeMethod(method));

Notice, no cleanup code! And even better, comparing peak use of TCP connections as a function of number of connections opened (measured with netstat -a on my Windows box):

And look at the number of exceptions we get on a sample run - HTTP Client cleans up the input buffer automatically up and gives zero exceptions:

So while the API library can be made to work, I don't think I will be using it in future. On top being verbose, the way it handles "unhappy" response codes and the fact that it will insert a Content-Type behind your back makes it a poor choice.

Waldo: Excellent!

2009-05-05T04:12:00.000-07:00

Being currently beached and briefly free of family life, I had some time to catch up on my reading. This one really filled a gap in my knowledge of distributed computing and its history.

I was in a conversation once that only now makes proper sense. The phrase "Waldo taught is that" was used in a discussion about REST, which puzzled me - I hadn't heard of Waldo except for the fellow in the striped sweater. Later I was reading some notes from QCon via Steve Vinoski's blog, and again the mysterious Wally was mentioned. And finally a quick googling turns up a reference from Uncle Bob and his friends on their list of reading recommendations. Now I can't ignore Wally any longer.

Waldo et al. described the impedance mismatch of language abstractions and distributed computing in this 1994 thesis. And I am regretting it took 15 years before I read it, as it makes some very straight forward arguments - things I wish I had been more aware of in past. In fact, thinking back, in 2002 I took a course which featured CORBA, and I was very impressed with it at the time.

Qouting from the abstract: "We argue that objects that interact in a distributed system need to be dealt with in ways that are intrinsically different from objects that interact in a single address space. These differences are required because distributed systems require that the programmer be aware of latency, have a different model of memory access, and take into account issues of concurrency and partial failure".

So if you are doing any web/ SOA/ remoting at all, you really should go find Wally and get some extra context.

Manual Deployment Antipattern

2009-02-19T03:05:00.000-08:00

It always felt wrong and now I found that it is actually a named antipattern! Yay! Now I can take that and use it to knock people on the head. Intellectual bullying - I'm lovin' it!

Alright, the article is about deployment automation, and this one in particular brings back memories. Hours and hours spent, manually deploying and redeploying a system, trying to ensure all dependencies were met.

Oh and of course, the reason I am loving this is the Antipatterns book, which I thought was brilliant. The whole concept of common pitfalls is just great - read it and you already know things you should be avoiding, you don't have to learn the hard way.

Digital signatures in Java

2009-02-16T14:58:00.000-08:00

We needed this for work, so while learning I wrote this very small program.

package digisign;

import java.security.InvalidKeyException;

import java.security.KeyFactory;

import java.security.KeyPair;

import java.security.KeyPairGenerator;

import java.security.NoSuchAlgorithmException;

import java.security.PrivateKey;

import java.security.PublicKey;

import java.security.Signature;

import java.security.SignatureException;

import java.security.spec.InvalidKeySpecException;

import java.security.spec.X509EncodedKeySpec;

* http://java.sun.com/docs/books/tutorial/security/apisign/index.html

public class Main {

public static void main(String[] args) throws NoSuchAlgorithmException, InvalidKeyException, SignatureException, InvalidKeySpecException {

// create some keys

KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance("RSA");

keyPairGenerator.initialize(1024);

KeyPair keyPair = keyPairGenerator.generateKeyPair();

PrivateKey privateKey = keyPair.getPrivate();

PublicKey publicKey = keyPair.getPublic();

// sign some data

byte[] data = "Hello, world!\n".getBytes();

Signature signature = Signature.getInstance("SHA1withRSA");

signature.initSign(privateKey);

signature.update(data);

byte[] sign = signature.sign();

// serialize public key and hand it over

byte[] pubKey = publicKey.getEncoded();

// store to disk, read in etc...

X509EncodedKeySpec encodedKeySpec = new X509EncodedKeySpec(pubKey);

KeyFactory keyFactory = KeyFactory.getInstance("RSA");

publicKey = keyFactory.generatePublic(encodedKeySpec);

// verify signature

signature = Signature.getInstance("SHA1withRSA");

signature.initVerify(publicKey);

signature.update(data);

boolean b = signature.verify(sign);

System.out.println("signature " + (b ? "ok." : "invalid!"));

}

Software Sloth

2009-01-05T13:54:00.000-08:00

I know I should be babysitting, but I saw Ian's nuanced post about technical debt and thought he left one out, and it is a cardinal one.

The fundamental problem is you either do not care enough and/ or you are too proud to change your was. The blogosphere is full of good advice, there are loads of great textbooks available, and peer learning is catching on - so help is available. So why do some still not "get it" and start producing good software?

Technical debt can be excused if it is backed up my conscious, well thought through decisions. There can be justification for skimping on quality. However, I contend that most technical debt is accrued through sloth. You either do not know what quality is, because you do not care enough to go and find out. You do not know the price of short sightedness. You cannot be bothered to read good books, or participate in collaborative peer efforts to better yourself. Or you are sinister, arrogant and not interested in improving your ways, not receptive to better ideas - even when you see mounting technical debt all around you.

I am a firm believer in evolution, and if we frame the problem thus, it is clear what the real problem is: because there isn't enough talent to go around, to fill enough posts and attain critical mass, sloth is allowed to thrive. Evolutionary pressures only work when resources are scarce, and sloth is apparently a smaller problem than getting bums on seats and poor code out the door.

Or maybe I am mistaken, and quantity is the real measure of success?

Hello, Nagios\n

2008-11-04T02:53:00.000-08:00

Nagios is a generic monitoring tool, and I want to use it to monitor my own application. It has a lot of built-in and offical plugins. However, it is always good to try a Hello, World-type example.

Firstly, my very interesting application. It follows the Nagios Plugin guidelines: return a value in {0, 1, 2, 3} and a status message of less than 4k to standard out. And it gives you some random sysadmin-action:

(np.c, compiles into an executable np)

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

int main(int argc, char* argv[]) {
 srand(time(NULL));

 int r = rand() % 10;

 int status = -1;
 char* message = NULL;

 switch(r){
  case 0:
  case 1:
  case 2:
  case 3:
   status = 0;
   message = "Everything is alright :-)";
   break;
  case 4:
  case 5:
  case 6:
   status = 1;
   message = "You have been warned.";
   break;
  case 7:
  case 8:
   status = 2;
   message = "She is gonna blow!";
   break;
  case 9:
   status = 3;
   message = "Oh shit...";
   break;
 }

 printf("%s\n", message);

 return status;
}

Nagios is easy to install with this quickstart guide. You just need to configure it:

1) In $NAGIOS_HOME/etc/nagios.cfg, add a line cfg_file=$NAGIOS_HOME/etc/objects/np.cfg

2) In $NAGIOS_HOME/etc/nagios.cfg, set interval_length to a suitably low number, e.g. 5 seconds, so you can see changes quickly

3) In $NAGIOS_HOME/etc/objects/np.cfg, make it look something like this:

define hostgroup {
  hostgroup_name        app_hosts
  alias                 Application Hosts
}

define host {
  use                   generic-host
  host_name             app_host
  alias                 Application Host
  address               localhost
  hostgroups            app_hosts
  max_check_attempts    10
}

define service {
  use                   local-service
  host_name             app_host
  service_description   My Application
  check_command         monitor_np
  normal_check_interval 1
  retry_check_interval  1
}

define command {
  command_name          monitor_np
  command_line          $APPLICATION_HOME/bin/np
}

The configuration is quite fidgety. Lots of required attributes, no sensible defaults. Luckily you can reuse example configuration that comes with Nagios. And there is a handy pre-flight-check feature, you just run /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg and it will give you errors and warnings against your configuration files, with line numbers.

Anyway, by now Bob is your uncle and you can restart nagios to pick up changes. Then point your browser to http://localhost/nagios and watch the action!

Any Fool Can Sysadmin

2008-11-03T03:06:00.000-08:00

I have Windows XP on my laptop, but I needed a Linux environment for client work. The last time I played around with dual booting was in 2000 when I had just bought my self-builder home computer. I installed Windows 2000 then RedHat 6, but the whole thing failed (I suspect my then very new motherboard didn't have drivers, or something) and I had to wipe everything (using my home sysadmin skills: if anything goes wrong, wipe and reinstall!).

So, it is fair to say I am a non-sysadmin, and maybe that's why I am sometimes accused of being a girly drinker. But with VMware I am suddenly sucessful and confident - in fact I am so confident I might just run as root - it is a vm after all, what could go wrong?

I found this guide extremely helpful. VMware Player is free and easy, the guide helps to hook things up, Ubuntu is quite user friendly. Why was I so scared of sysadmining for so long?

Well, these are early days, and I might just revert to my old self if I get a problem with Linux - wipe and reinstall. But since it is a vm, that is actually a painless solution...

Actually I forget; I first tried installing Damn Small Linux using a VMware Appliance, since it was only a 50MB download. However, I didn't manage to get the network working. And my colleagues and I didn't have a clue how to fix it. So I ended up with Ubuntu. Morale of the story though, sysadmining is no fun if you can't easily wipe and reinstall.

Software Apprentice

2008-10-03T05:32:00.000-07:00

Kerry is always talking about Dreyfus and nursing competencies.

That got me thinking: is there a book you can give to a software developer that he can follow and use to improve himself through the five levels?

Well, I haven't read it yet but Elizabeth Keogh recommended me Apprenticeship Patterns: Guidance For The Aspiring Software Craftsman. I shall put it on my list right after finishing the DDD book and several volumes on how to raise babies.

More on this topic: http://dannorth.net/2008/06/learning-to-lean - loved the Bruce Lee quote at the bottom.

Annoying Eclipse Configuration

2008-10-01T03:22:00.000-07:00

I needed to get Eclipse to use a JDK instead of a JRE so I could play around with Maven.

Easy peasy? Well, no, not exactly...

"Just specify a -vm option in your eclipse.ini, and Bob is your uncle", they stated. Hah!

I put the option in with a blank line seperator. Doesn't work.
I delete the blank line after reading a blog entry. Doesn't work.
Hmm, it keeps defaulting to my JRE, maybe I will uninstall that so I can "debug". Que slow Windows uninstallation...
Now I get error messages. Maybe it is the space in Program Files? I'll put a %20 in there. Doesn't work.
I discover option values go onto a new line (how obscure). Combinatorials ensue. Still doesn't work.

Turns out it won't work if you don't -vm option before the -vmargs option. Why??? Worse, it will just silently die/ ignore/ not warn you. Perhaps they should work on their error messages?

But i'm not ashamed to have wasted half an hour. Others have wasted time on this stuff too.

Isn't there a standard or convention for text-based config files, and couldn't someone get Eclipse to use it?

Unit Testing With Threads

2008-09-30T03:50:00.000-07:00

I saw this and started thinking back on a calculus course at university.

Wouldn't we expect a theorem like this to be true:

"for all codebases C and features F, there exists a set of refactorings R and unit test U, such that U(F) is safe under R(C)"

In other words, can we not manipulate any piece of untestable thread code into a testable one, given enough refactorings and inversion of control levers (and effort!)? Seems reasonable to me.

In a situation with 2 threads and one interaction, it is easy to imagine the combinatorial complexity of test cases. It of course explodes when the numbers go up - but perhaps we could argue that for any number of threads and interactions, we can reduce the complexity of our tests by looking at each in isolation...

If my theorem holds, you can of course turn it around and use it to knock people over their heads when they screw up thread testing - hmm, how productive that would be :-)

Interesting thought, anyway.

Surprisingly Good Error Messages!

2008-09-25T08:58:00.000-07:00

I was stunned recently by finding a piece of software that does something reasonably complicated, but still manages to give good error messages.

Just look at this beauty from Spring IOC:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'magiceightball' defined in file [C:\...\random\magiceightball.test.xml]: Cannot resolve reference to bean 'randomsource' while setting constructor argument; nested exception is org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [foo.Foo] for bean with name 'randomsource' defined in file [C:\...\random\magiceightball.test.xml]; nested exception is java.lang.ClassNotFoundException: foo.Foo at [...]

That is actually perfectly readable! Alright, I am a nerd, and I have some experience with debugging Java code, but I bet a 9-year-old child could deduce that they forgot to implement a RandomSource called Foo.

I was stunned when I saw it, perhaps because it is so rare to get useful error messages from clever frameworks. As soon as reflection enters the picture, people just give up it seems. Look at this example as a contrast:

com.google.inject.CreationException: Guice configuration errors: 1) Error at foo.BarTestModule$TestBar.(BarTest.java:18): Could not find a suitable constructor in foo.BarTestModule$TestBar. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor. 1 error[s] at com.google.inject.BinderImpl.createInjector(BinderImpl.java:277)
[blah blah]

Hmm, that's funny, I distinctly remember creating a zero-argument constructor in TestBar??? It turns out, Guice does not support injection of inner classes. And it only took me an hour to figure that out... So is this error message helpful, or rather harmful?

Hibernate is equally bad and will give you a lot of grief if you do not tread carefully. And these are popular, high-profile pieces of software, not random open-source amateur junk off of SourceForge. So how come Spring IOC got it right and these guys got it so wrong?

T-shirt Size Estimation

2008-09-22T04:38:00.000-07:00

We have all been through painfil estimation sessions. What makes them painful is formality.

The Living Room Architect talks about a Story Matrix, as a way of facilitating a smoother flow for estimating a bunch of stories. Good idea.

One problem still persists though, is the formality of it, with the whole points debate: do we use Fibonacci, linear, exponential, what? What do the numbers mean? And the apprehention on the part of us developers to be specific with stories they have only just begun to think about.

Someone today introduced me to T-shirt Sized Estimation - you basically cluster your stories into S, M and L. Sizes cover plain "lines of code", complexity and risk. But it has an implied light touch which seems to solve some problems I think, it seemed to go quickly and easily, and we did get a result out of it.

No numbers, so nothing to get too hung up about
Only 3 categories, so we don't have to discuss things for very long - it is implied that this isn't the final clustering
We still got to communicate about the stories, so Agile Tick Mark™ right there
And we got a rough set of estimates

Now, that isn't of course the final word in the estimation process, and we all know that, so no point in getting defensive. We will need numbers at some point. But it got us perhaps half way. We did talk about our understanding of the stories, and we did get a picture of how hard the project might be.

But most importantly, we didn't get hung up on details - instead we split the process into 2 (or more) phases. A bit like when you look at algorithms, you know that when you need to traverse a graph and do something complicated, you can either write the mother of all recursive functions, or you can split up into separate stages (search and modify stages for example). In the end, we all know that O(n) + O(n) = O(n) - i.e. the amount of effort will be the same, but it will be easier to manage the two things separately.

We will certainly need another card session, and we will need some numbers at some point, but for now we have fleshed out some things, in a very short time and noone has run off crying. And that has to be considered progress.

Schema Spy

2008-09-15T09:18:00.000-07:00

While reverse-engineering a customer's database, my pair-mate introduced me to Schema Spy. It looks quite good: lightweight cross-db reporting and visualization, runs from the command line and gives you an HTML folder structure.

http://schemaspy.sourceforge.net/

Top 10 Books On Software

2008-09-11T15:28:00.000-07:00

A friend of mine blogged about books. His question was: "What are, in your opinion, the top 10 books that every software engineer/developer should read?". Well, here are a few to get you started:

Gamma et al.: "Design Patterns". An absolute classic. After reading it I immediately went into pattern fever (as everyone does?). Interestingly, the patterns themselves aren't that useful for the stuff I do day to day, but the idea is powerful. And of course, this book started the whole Patterns Movement...

Møller & Schwartzbach: "An Introduction to XML and Web Technologies". I am slightly biased, having taken an XML course from the authors. But, this book really puts things in perspective, it is thorough and clear, and offers opinion. The web is full of simple examples, re-hashes, cheaply written stuff without depth. So this was a great and welcome read.

Larman: "Applying UML and Patterns". This one really hit the spot. I was idling somewhat, lacking direction, and this one gave me a spark of inspiration. The way it formulates those simple, old-fashioned OO values, it takes you back to first principles and reminds you why it is we do what we do. It is easy to get lost in new technology, tools, methodology and process - but what we really need is to remember those few, simple things: model your domain, separate concerns, an Bob is your uncle!

All The Cool Kids Have A Blog

2008-09-11T15:18:00.000-07:00

All the cool kids have a blog, so I think I should have one too. That is about as much reasoning as I can muster right now.

I am always impressed with blogger's boldness, the way they reveal their hand to the world and take sides. I wonder if I will be as brave? I like to ask questions. Do questions belong in a blog? Or does it only work if you have that blogger-swagger?

Anyway, we shall see what comes of it!