A scavailable Squid proxy cache because Maven!

I had opportunity to use Squid recently at work and want to share - also because Google didn't just hand me this blueprint as I had expected.

Actually it was Julian Simpson, he of ThoughtWorks/ Neo4j/ DevOps New Zealand fame, who inspired me down the pub one night (John Scott's Lund, lovely place).

The problem we had was Maven Central's infrastructure not being good enough for our needs; we had too many failing builds because of connectivity issues - artifact downloads. Basically HTTP GET. Between Maven-the-client and Sonatype's CDN things just weren't working well enough. We tried bintray.com too, with similar disappointing results. Something had to be done. It is worth noting here that Maven-the-client does do retries by default, so this isn't entirely basic stuff.

Background

We run a bunch of AWS instances on demand as part of CI, and they in turn run Maven - hundreds of concurrent jobs (and therefore machines) during office hours. Machines come up blank and go away when not needed, so most of the time we have to bootstrap the local Maven repository from somewhere. In fact if a bunch of these come up at the same time they conduct a form of DOS attack on Maven Central's CDN - oops. Anyway, Julian outlined a solution where we build a proxy and fiddle with artifact caching headers, to form a shared service for these cloud build agents. Intra-AWS HTTP downloads FTW.

Maven-the-client insists on sending some vicious headers: Cache-store: no-store, Cache-control: no-cache, Expires: 0. This forces requests to go all the way to origin server, and I guess they do it to gather usage statistics - certainly Maven Central artifacts never change, or the internetz would break.

Therefore first item of business would be to stop respecting those headers and have Squid cache objects anyway, and for a long time. Yes, this breaks the HTTP contract. No, that isn't a problem.

Secondly, we do not want to replace one problem with another one so this needs to run as a 24/7/365 shared-nothing highly-available stateless 0.99999nines99 service, pretty much. You know, text book scavailable stuff. AWS can help us there, think an ELB in front of an auto-scaling group full of instances running Squid: one machine fails and auto-scaling replaces it. Data loss is no problem, the new instance is just a cold cache which will work, just more slowly.

One-page scavailability

The blueprint, such as it is; not pretty, but neat to not have more bits: a CloudFormation template, instances running Ubuntu 16.04, service sitting on port 1080, bit of security and auth.

{
  "Description": "Maven proxy service",
  "Parameters": {
    "keyname": { "Type": "AWS::EC2::KeyPair::KeyName" }
  },
  "Resources": {
    "loadbalancer": {
      "Type": "AWS::ElasticLoadBalancing::LoadBalancer",
      "Properties": {
        "AvailabilityZones": { "Fn::GetAZs": "" },
        "Listeners": [
          {
            "LoadBalancerPort": "1080",
            "InstancePort": "3128",
            "Protocol": "TCP"
          }
        ]
      }
    },
    "securitygroup": {
      "Type": "AWS::EC2::SecurityGroup",
      "Properties": {
        "GroupDescription": "SSH and Squid",
        "SecurityGroupIngress": [
          {
            "CidrIp": "0.0.0.0/0",
            "FromPort": "22",
            "IpProtocol": "tcp",
            "ToPort": "22"
          },
          {
            "FromPort": "3128",
            "IpProtocol": "tcp",
            "SourceSecurityGroupOwnerId": {
              "Fn::GetAtt": [ "loadbalancer", "SourceSecurityGroup.OwnerAlias" ]
            },
            "SourceSecurityGroupName": {
              "Fn::GetAtt": [ "loadbalancer", "SourceSecurityGroup.GroupName" ]
            },
            "ToPort": "3128"
          }
        ]
      }
    },
    "waithandle": {
      "Type": "AWS::CloudFormation::WaitConditionHandle"
    },
    "waitcondition": {
      "Type": "AWS::CloudFormation::WaitCondition",
      "Properties": {
        "Count": "3",
        "Handle": {
          "Ref": "waithandle"
        },
        "Timeout": "500"
      }
    },
    "launchconfiguration": {
      "Type": "AWS::AutoScaling::LaunchConfiguration",
      "Properties": {
        "ImageId": "ami-c1167eb8",
        "InstanceType": "m3.large",
        "KeyName": { "Ref": "keyname" },
        "SecurityGroups": [ { "Ref": "securitygroup" } ],
        "UserData": {
          "Fn::Base64": {
            "Fn::Join": [
              "",
              [
                "#!/bin/bash -eux", "\n",
                "\n",
                "apt update", "\n",
                "apt install --yes squid3 heat-cfntools apache2-utils", "\n",
                "\n",
                "echo '# Maven proxy service'                                                                  >  /etc/squid/squid.conf", "\n",
                "echo 'maximum_object_size_in_memory 30 MB'                                                    >> /etc/squid/squid.conf", "\n",
                "echo 'acl mavencentral dstdomain repo.maven.apache.org uk.maven.org'                          >> /etc/squid/squid.conf", "\n",
                "echo 'cache deny !mavencentral'                                                               >> /etc/squid/squid.conf", "\n",
                "echo 'refresh_pattern repo.maven.apache.org 288000 100% 576000 ignore-reload ignore-no-store' >> /etc/squid/squid.conf", "\n",
                "echo 'refresh_pattern uk.maven.org          288000 100% 576000 ignore-reload ignore-no-store' >> /etc/squid/squid.conf", "\n",
                "echo 'http_port 3128'                                                                         >> /etc/squid/squid.conf", "\n",
                "echo 'auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwd'              >> /etc/squid/squid.conf", "\n",
                "echo 'auth_param basic realm Maven proxy service'                                             >> /etc/squid/squid.conf", "\n",
                "echo 'acl ncsa_users proxy_auth REQUIRED'                                                     >> /etc/squid/squid.conf", "\n",
                "echo 'http_access allow ncsa_users'                                                           >> /etc/squid/squid.conf", "\n",
                "\n",
                "htpasswd -b -c /etc/squid/passwd someone something",
                "\n",
                "systemctl reload squid", "\n",
                "\n",
                "export SIGNAL_URL='", { "Ref" : "waithandle" }, "'\n",
                "echo 'signal url: ${SIGNAL_URL}'", "\n",
                "cfn-signal ${SIGNAL_URL}", "\n",
                "\n"
              ]
            ]
          }
        }
      }
    },
    "autoscalinggroup": {
      "Type": "AWS::AutoScaling::AutoScalingGroup",
      "Properties": {
        "AvailabilityZones": { "Fn::GetAZs": ""  },
        "LaunchConfigurationName": { "Ref": "launchconfiguration" },
        "LoadBalancerNames": [ { "Ref": "loadbalancer" } ],
        "MinSize": "3",
        "MaxSize": "3"
      }
    }
  },
  "Outputs" : {
    "dnsname" : {
      "Description": "DNS name for load balancer",
      "Value" : { "Fn::GetAtt" : [ "loadbalancer", "DNSName" ] }
    }
  }
}

So how did it go?

Well actually, it didn't go particularly well. We did succeed in reducing the frequency of failures by a factor 100 or something, yay! But the business problem persisted, just too many broken builds and people kept complaining.

The next approach we took was to start caching artifacts on machines: mvn dependency:go-offline with a twist because that plugin is broken and abandoned (thanks Cosmin et al for fixing/ forking!). In fact seeding machines when we build AMIs, then patching them up just before running builds - all because Maven! That works well enough in conjunction with some system properties we divined, system.maven.wagon.http.retryHandler.class=default and system.maven.wagon.http.retryHandler.nonRetryableClasses=java.io.InterruptedIOException,java.net.UnknownHostException,java.net.ConnectException (don't ask). But what a mess! In future surely we will all come around to sticking dependencies in source control eh? Not sure why that has become a religious topic, it's not the 90s anymore, we have bandwidth and disk space.

Also, while working on this I kept thinking Mark Nottingham's stale-while-revalidate would be the perfect fit here, but it wasn't in Squid 3.5 (or I couldn't get it working). Meh. Something to solve if doing this again.

Bootstrapping Windows for programmatic fun & profit

At work we do a bunch of work to ensure our database product works as expected. You're familiar with unit tests and integration tests of course, but take that up a notch and consider the domain - a database product - and you just need a lot more engineering around the thing. A full blown database is a different beast than your bog standard mom & pop web shop, and product is another level of complexity over common website type of stuff, in that we send out binary artifacts to customers, and they run them for years. I trust you are suitably impressed by now. The sharp end of the software industry innit.

Anyway one thing we do to keep a high standard for our product is, we run soak tests regularly. Soak testing involves testing a system with a typical production load, over a continuous availability period, to validate system behaviour under production use. For us that means days or weeks of (synthetic) production workloads run on production grade hardware, with injected faults, backup events and whatnot. We have been doing it for years, and we have become exceedingly efficient at it. It is a very effective method for catching problems before customers do.

In practice it is a case of programmatically setting up things like a database cluster, a load generator, reporting etc. into a resource-oriented, hypermedia-driven set of servers, agents and client-side APIs. Programmatically because we need economics and automation and all those other good bits. We are at a point where starting a soak test is a case of running a single (albeit rather long) command line. Pretty sweet.

We use cloud machines to scale, and therefore one crux we had to solve was bootstrapping machines. On AWS that's done by injecting a script via user data, and that script unfolds into more scripts downloaded from S3, software installed via packages, and our custom services and workloads kicked off. Untouched by human hand and it really is beautiful - on Linux.

Windows!

As awesome as the above is, it is qualified by "on Linux" because the other platforms we support are just not as easy to work with. Until now anyway. In past when we looked, through the prism of non-Windows developers/ having other priorities, we never managed to crack this problem of bootstrapping and it was left to one side. But I looked again when I had to pick up Windows Server 2019 recently for other work, and I am happy to report it is mature enough for what we need. So what do we need?

Well we effectively just need a script to run automatically on a Windows box at launch that can install Java (we are a 4j company after all!), download an executable jar and launch it. Plus a few details. That initial jar blooms into all the other bits we need, and is well-known shared code that we run all the time on Linux (Write Once, Run Everywhere, remember?)

A minimal (PowerShell) user data script

Here is just enough PowerShell to get going. You surround it with <powershell>...</powershell> (more detail in the official documentation) and stick it in as user data when launching the machine:

# bootstrap-script.ps1

# find a home
$workspace = $env:LocalAppData + "\myworkspace"
New-Item -ItemType "directory" -Path $workspace

# download the business script
$scriptfile = $workspace + "\business-script.ps1"
Read-S3Object
  -BucketName "mycloudhome"
  -Key "business-script.ps1"
  -File $scriptfile

# execute it
$logfile = $workspace + "\bootstrap.log"
Push-Location $workspace
& $scriptfile *> $logfile

Your business script now runs as Administrator in its own workspace. The possibilities are endless.

Why/ how does that work?

Well for starters, the Windows Server 2019 base image on AWS comes with certain AWS cmdlets pre-installed. Firstly Read-S3Object which you can use to call home effectively, you can specify an S3 bucket + key from your own account and ensure access using IAM instance profiles. That makes it a single line of code to get at the business script you want. Here is an example IAM policy:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::mycloudhome/*"
    }
  ]
}

Data in that bucket can be static or dynamic, meaning you can change the business script ahead of time. You could for example stick versioned scripts in there, think business-script-$UUID.ps1, but then you'd need to inject that UUID somehow. An exercise for the reader, and there are several options: dynamic bootstrap script or maybe reading instance metadata leading to reading instance tags - be creative.

You could inline everything into a gigantic bootstrap script in principle, but I think for robustness' sake better to separate into one script focused on bootstrap, which never changes, and which is small enough to get under the cap on user data length (16K); and business scripts that are specialized into various business scenarios.

In the end you launch your business script, pipe output to a logfile, and let that take over. Easy peasy.

A minimal script for starting a Java service

Coming back to our motivation, getting tools and services running for e.g. soak testing, there are a few more handy bits to consider:

• We need Java, that does not come pre-installed
• We need the business software - executable jars are nice here
• We need a port opened so that our client side orchestration scripts can take over

You could do something like this. First, download and unzip a JDK from our magic bucket:

# business-script.ps1

# download JDK

$jdkzip = "jdk-11.0.5_windows-x64_bin.zip"

Read-S3Object

  -BucketName "mycloudhome"

  -Key "jdk-11.0.5_windows-x64_bin.zip"

  -File $jdkzip

Expand-Archive -LiteralPath $jdkzip -DestinationPath jdk

Now let's get the JDK onto path. Here that's a case of getting the first sub-folder of a folder - simple and effective:

# "install" JDK

$jdkfoldername = (Get-ChildItem -Directory jdk)[0].name

$jdkfolderpath = Resolve-Path ("jdk\" + $jdkfoldername + "\bin")

$env:Path += (";" + $jdkfolderpath)

Time for magic bucket to shine again. I guess I'd recommend injecting a single correlation (uu)id and otherwise isolating reference data from dynamic data using a bucket named using that (uu)id, for robustness. But the mechanics are in essence this:

# download business jar

Read-S3Object

  -BucketName "mycloudhome"

  -Key "business.jar"

  -File "business.jar"

Remember the Windows firewall, we need to open it from the inside:

# open up firewall

New-NetFirewallRule

  -DisplayName "Allow inbound port 20000"

  -LocalPort 20000

  -Protocol TCP

  -Action Allow

And now it is just Java software which we know and love (and which we share across Linux too):

# go time!

& java -jar business.jar <parameters go here>

Now you sit down and poll for your service to be ready.

The client end

Here's some example client code. It gets more advanced when you start injecting more variability, and of course you need to create auxiliary bits like security groups also. We make use of CloudFormation to create isolated experiments:

public void createServiceOnWindows() throws Exception {

    // refresh business script in magic bucket

    PutObjectRequest putObjectRequest = new PutObjectRequest("mycloudhome", "business-script.ps1", /* the actual business-script.ps1 file */);

    AmazonS3ClientBuilder.defaultClient().putObject(putObjectRequest);

    // launch instance using bootstrap script

    String userData = /* <powershell> contents of bootstrap-script.ps1 <powershell>, base64 encoded */

    RunInstancesRequest runInstancesRequest = new RunInstancesRequest()

            .withIamInstanceProfile(

                    new IamInstanceProfileSpecification()

                            .withArn(/* something to give access to magic bucket */)

            )

            .withImageId(/* tested with Windows Server 2019 AMI */)

            .withSecurityGroupIds(/* open port 20000 for our example, maybe RDP for debugging */)

            .withUserData(userData)

            [...]

            );

    AmazonEC2ClientBuilder.defaultClient().runInstances(runInstancesRequest);

}

Parting words

There are no technical blockers to extending our extensive Java-based tooling to also cover Windows. And with minimal scaffolding too.

I am not fond of PowerShell's model and error handling, which is a good reason for getting out from PowerShell and into our Java tooling as quickly as possible - remember, our tooling initially is just empty shell and we need client driven orchestration to achieve business goals like installing our database product, seeding data, changing config, yada yada. But I guess PowerShell is just good enough, and the AWS command line tools in general are just rock solid, on Windows or otherwise.

Installation Testing @ Neo4j

At work we produce a database product which we deliver to customers in a variety of formats. Tarballs and zips you can just extract and use on your home computer, be it Linux, Mac or Windows; A Docker image for the hipsters; and old school Debian and RPM packages for the slow movers^H^H^H^H^H^H^H^H^H^H^Hstable enterprises. Oh and shameless, out of context plug.

We had all those packages sans Docker when I joined years ago, and they weren't great. Testing was manual and not thorough at all. They have improved, just by having skilled people work on them - so far, so regular every day software. These are critical components actually, they are a core mechanism for getting our software to customers, and customers in turn rely on them for their critical business operations. So no pressure.

Anyway, one day I had an epiphany: why don't we just write automated tests around these packages, so that we can have confidence they work as expected when making changes and adding features? Duh!

We do that for everything else, and on multiple levels. When you pare it back this is deterministic stuff about file locations and permissions, really. Turns out we already had all the building blocks and blueprints, but I still feel this was a tiny local game changer.

Asking around the place it was clear this was not something anyone had seen before, and I do think we are quite a testing-forward place too. Some spitballing by the water cooler with my mate Steven Baker helped clarify things - he's had the actual package knowledge, I was just the ideas man - and we were ready to go. And so the very first Installation Testing framework was birthed.

So what is it?

Well what is a package? It's a thing that runs on a computer, it puts files in places when you install, in our case it starts a service that you can poke with a few ancillary tools, and you can uninstall and even purge it when you lose interest.

So there are the contours of the framework: exercising the package installer is a case of running it and observing files appear where expected, with expected permissions, and verifying a service is started automatically and becomes available. That the right dependencies are installed too. Uninstalling and purging is the reverse. A bit of poking at the ancillary tools to see they are also working. Easy peasy!

Once you get into it, a first challenge is sandboxing and isolating: you want a clean slate for every run, so that you have reproducibility and an aura of science about it. We already do this in other areas, using throw-away AWS instances. Spot priced too because economics, and these days you are charged by the minute so very little waste. Indeed the catalogue of AWS instance OSes helps us reach different OSes (Debian, Ubuntu, RHEL, Amazon Linux, ...) at different versions, spanning the space our customer base lives in. So much winning there.

The tests are really a series of commands sent via SSH, there are calls and waits and assertions like you would use in similar system testing, classic stuff. It gets frameworky once you realise the same high level script applies to each platform, yay reuse. But really it is basic stuff when you think about it.

Now, there were stumbles. Internal ones like picking technology, where we tried and discarded Vagrant+VirtualBox as the container for the different platforms, we couldn't make it work reliably for us. We also discarded Cucumber for the high level scripting, because really, we don't have non-coders looking at this and Ruby isn't a core skill here (we're a 4j shop remember!), so a bunch of hassle with no payoff.

External stumbles currently include Zypper which is a PITA to work with, and of course all the little niggles and of course bugs found+fixed. But external stumbles is exactly what we hunting for here, so that's just grand.

Status

Current incarnation of Installation Testing was implemented by my good colleague Jenny Owen as Java + JUnit with Maven, using the standard AWS client libraries and JSch. It works great, we have readable code and running this in CI gives us so much clarity and confidence which we never had before. There are thoughts about open sourcing it as it might be useful to others. Do you think it is? I'd love to hear experience reports if you faced similar problems.

So, you launch AWS instances, commandeer them via SSH, exercise and evaluate the software, terminate the instance. In parallel across the 12 and growing platforms multiplied by currently 2 editions of Neo4j we are interested in, for that low latency feedback. Textbook. What's not to like?

The International E-road Network and Neo4j

I was having fun recently with some E-road data that I found. E-roads are highly intuitive for spatial/ graph-y stuff: you will be on one regularly, and they will lead you to Rome, eventually. Or Aarhus. And because it is graph-y and spatial at the same time, it is obvious to try some shortest path queries on it, which Neo4j has built-in. Think route planning.

This particular dataset has 895 reference places and 1250 sections of road between them. Roads have a distance attribute, which will come in handy. And there is some more metadata to play with like country codes and whether a road is a water crossing.

The data needed some massaging into CSV format before it can be imported into Neo4j. I ended up with one line of CSV per section of E-road, so some duplication of reference places, but meh - Neo4j can merge them back together:


Oh and please go ahead and use the dataset if you find it interesting.

Let's go from Århus to Rome:

Cypher has a feature called variable length pattern matching. Here is the simplest possible Cypher query for finding the path we want:

MATCH p=((aarhus {name: "Århus"})-[:EROAD*]-(rome {name: "Roma"}))
RETURN p

Well, that didn't work, Spinner-of-Death™. The dataset is too large, or the query is too broad, or my laptop is too small. Aha! But I happen to know from playing with the dataset that there exists a part of length 28 between Aarhus and Rome, so we can give the path finder a maximum:

MATCH p=((aarhus {name: "Århus"})-[:EROAD*28]-(rome {name: "Roma"}))
RETURN p

Meh, Spinner-of-Death™ again...

Waypoints! Let's make it even bit easier and constrain the query by inserting waypoints. The path I found while exploring goes via Stuttgart and Milan:

MATCH p=((aarhus {name: "Århus"})-[:EROAD*11]-(stuttgart {name: "Stuttgart"})-[:EROAD*11]-(milan {name: "Milano"})-[:EROAD*6]-(rome {name: "Roma"}))
RETURN p

Result!

We can even find the length of the paths:

MATCH p=(aarhus {name: "Århus"})-[:EROAD*11]-(stuttgart {name: "Stuttgart"})-[:EROAD*11]-(milan {name: "Milano"})-[:EROAD*6]-(rome {name: "Roma"})
RETURN REDUCE(s = 0, r IN relationships(p) | s + TOINT(r.distance)) AS total_distance ORDER BY total_distance ASC

It comes up to 2329 km for the shortest path, 3491 km for the longest, and there are 48 paths that fit the pattern.

We need a shortest path algorithm

Alright, the trouble with the above approach is, the waypoints I chose are probably be sub-optimal, and therefore the path isn't as short as it could be. Also it is a jumble to look at, there are several paths between Stuttgart and Milan of length 11 for example, and it is hard to get an intuition of this intuitive spatial/ graph-y data.

Luckily, Cypher has a shortest path algorithm built in:

MATCH p=shortestPath((aarhus {name: "Århus"})-[rels:EROAD*]-(rome {name: "Roma"}))
RETURN p, length(p), REDUCE(s = 0, r IN rels | s + TOINT(r.distance)) AS total_distance

So there is the shortest path from Aarhus to Rome - in terms of hops. 22 hops and 2948 km, including sailing across the Adriatic Sea. Let's call it the scenic route.

We know that is suboptimal, but at least the query looks sane now without the waypoints. Oh and some graphics skills, we can edit the graph in Neo4j Browser, drag and arrange the nodes so they neatly overlay locations on a map.

Weighted shortest path FTW!

Right. Last refinement, promise. Weighted shortest path is supported in Neo4j using Dijkstra from the APOC procedure library plugin, and we need that so we can minimise distance instead of just hops:

MATCH (aarhus {name: 'Århus'}), (rome {name: 'Roma'})
CALL apoc.algo.dijkstra(aarhus, rome, 'EROAD', 'distance') YIELD path, weight
RETURN path, length(path), weight

Neat and simple query giving us a path with 26 hops, 2147 km, and quite straight-looking on the map. We have a winner.

Divide and conquer your merge hell with Git

I'm no Git wizard, and it is a scary powerful tool. But from today I absolutely love it!

I was doing the dreaded master merge, facing merge hell, and feeling kinda glum. But I found this as a way of breaking the problem up and merging only a manageable chunk at a time - say merge master as of 5 days ago:

$ git merge "master@{5 days ago}"

or master as it was 3 commits ago:

$ git merge master^^^

More work, sure, you have to potentially re-visit files, but it helps triangulate the hard spots. Indeed, you can do a backwards binary search for a manageable chunk.

More revision-vocabulary here: http://git-scm.com/docs/gitrevisions.html

Versioning

In my new job as a product engineer - as opposed to being on the application side - this is a good, concise read about semantic versioning.

The E in ELB

From http://aws.amazon.com/articles/1636185810492479:

There are a variety of load testing tools, and most of the tools are designed to address the question of how many servers a business must procure based on the amount of traffic the servers are able to handle. To test server load in this situation, it was logical to quickly ramp up the traffic to determine when the server became saturated, and then to try iterations of the tests based on request and response size to determine the factors affecting the saturation point.

When you create an elastic load balancer, a default level of capacity is allocated and configured. As Elastic Load Balancing sees changes in the traffic profile, it will scale up or down. The time required for Elastic Load Balancing to scale can range from 1 to 7 minutes, depending on the changes in the traffic profile. When Elastic Load Balancing scales, it updates the DNS record with the new list of IP addresses. To ensure that clients are taking advantage of the increased capacity, Elastic Load Balancing uses a TTL setting on the DNS record of 60 seconds. It is critical that you factor this changing DNS record into your tests. If you do not ensure that DNS is re-resolved or use multiple test clients to simulate increased load, the test may continue to hit a single IP address when Elastic Load Balancing has actually allocated many more IP addresses. Because your end users will not all be resolving to that single IP address, your test will not be a realistic sampling of real-world behavior.

Can you live with O(minutes) responsiveness?

Awesome Gmail

HTTP tidbits

Reading up on RFC 2616, section 6, some interesting details that I want to keep:

reason phrases [...] MAY be replaced by local equivalents without affecting the protocol

Could come in handy for debugging?

HTTP applications are not required to understand the meaning of all registered status codes, [...]. However, applications MUST understand the class of any status code, as indicated by the first digit, and treat any unrecognized response as being equivalent to the x00 status code of that class

Interesting edge case. And HTTP status codes are arithmetic types, not enums.

Ninite

Note to self: http://ninite.com/ works really well.

Thanks Carlos!

Brief is not simple

I am always keen to reduce the world into simple metaphors and proverbs. Big fan of "the emperor has no clothes" for example (and not only because H. C. Andersen was danish!), as it can be applied to so many things. Particularly, while we battle for the hearts and minds of the IT industry, advocating simplicity and denouncing ceremony, it seems to come in handy time and again.

Anyway, I was reading about Guava (thanks for the link Simon!), and stumbled upon one that was missing from my library: "brevity is the soul of wit". In the context of source code, you can imagine a snooty Perl hacker uttering this phrase after writing a particularly pleasing regular expression that no-one will ever be able to decode. However, I cannot think of anything less appealing in source code that brevity or wit. As a way of communicating, code is really best expressed simply, not briefly or cleverly.

Of course, outside the world of code I am a huge fan of folks who can be concise, thus.

Watch your dependencies

Dependencies are a headache. They cause all sorts of problems. So you should keep aware of them.

Tools are a-plenty, for many languages, but here is a good starter for your favourite open platform: Byecycle

Other tools do more, but are also more complicated and hard to interpret. If you want a gentle introduction, start with this little visual tool.

It can help you find dependency cycles



It can help you spot classes that do too much



It simply gives you a 30.000ft view of how your code looks, something which is difficult to get from text files on even a medium sized code base.

I like Apache HTTP Client over java.net.HttpURLConnection

My last project involved CAS and a lot of integration points, so we ended up with lots of HTTP traffic as part of our integration tests.

We started having problems with our build, spurious exceptions would break it, but we couldn't reproduce the problem. It was not until we did some rudimentary load testing that the problem surfaced: Persistent HTTP connections.

Problem was, we had written our tests using java.net.HttpURLConnection like this:

try {
  HttpURLConnection connection = (HttpURLConnection) new URL("http://localhost:8080").openConnection();
  connection.connect();
  assertEquals(HttpURLConnection.HTTP_OK, connection.getResponseCode());
} catch (...){
  ...
}
finally {
  connection.disconnect();
  connection.close();
}

Turns out, because we didn't care about the response body and thus never bothered reading it, and that would tie up TCP connections. The more tests we ran, the more likely it became that we would see an exception like this:

Exception in thread "main" java.net.BindException: Address already in use: connect

So while we thought we were being diligent by writing a finally block to disconnect and close the connection, it turns out that is completely superfluous. What we would have to write was this:

try {
  HttpURLConnection connection = (HttpURLConnection) new URL("http://localhost:8080").openConnection();
  connection.connect();
  assertEquals(HttpURLConnection.HTTP_OK, connection.getResponseCode());
} catch (...){
  ...
}
finally {
  while (connection.getInputStream().read() != -1) ;

  connection.disconnect();
  connection.close();
}

(Actually, that just works for happy HTTP responses, as this library will throw an exception on 4xx and 5xx's!? But lets not digress...)

Now that's a lot of code for something so simple - look at Apache HTTP Client for the same task:

HttpClient httpClient = new HttpClient();
GetMethod method = new GetMethod("http://localhost:8080");
assertEquals(HttpStatus.SC_OK, httpClient.executeMethod(method));

Notice, no cleanup code! And even better, comparing peak use of TCP connections as a function of number of connections opened (measured with netstat -a on my Windows box):



And look at the number of exceptions we get on a sample run - HTTP Client cleans up the input buffer automatically up and gives zero exceptions:



So while the API library can be made to work, I don't think I will be using it in future. On top being verbose, the way it handles "unhappy" response codes and the fact that it will insert a Content-Type behind your back makes it a poor choice.

Waldo: Excellent!

Being currently beached and briefly free of family life, I had some time to catch up on my reading. This one really filled a gap in my knowledge of distributed computing and its history.

I was in a conversation once that only now makes proper sense. The phrase "Waldo taught is that" was used in a discussion about REST, which puzzled me - I hadn't heard of Waldo except for the fellow in the striped sweater. Later I was reading some notes from QCon via Steve Vinoski's blog, and again the mysterious Wally was mentioned. And finally a quick googling turns up a reference from Uncle Bob and his friends on their list of reading recommendations. Now I can't ignore Wally any longer.

Waldo et al. described the impedance mismatch of language abstractions and distributed computing in this 1994 thesis. And I am regretting it took 15 years before I read it, as it makes some very straight forward arguments - things I wish I had been more aware of in past. In fact, thinking back, in 2002 I took a course which featured CORBA, and I was very impressed with it at the time.

Qouting from the abstract: "We argue that objects that interact in a distributed system need to be dealt with in ways that are intrinsically different from objects that interact in a single address space. These differences are required because distributed systems require that the programmer be aware of latency, have a different model of memory access, and take into account issues of concurrency and partial failure".

So if you are doing any web/ SOA/ remoting at all, you really should go find Wally and get some extra context.

Manual Deployment Antipattern

It always felt wrong and now I found that it is actually a named antipattern! Yay! Now I can take that and use it to knock people on the head. Intellectual bullying - I'm lovin' it!

Alright, the article is about deployment automation, and this one in particular brings back memories. Hours and hours spent, manually deploying and redeploying a system, trying to ensure all dependencies were met.

Oh and of course, the reason I am loving this is the Antipatterns book, which I thought was brilliant. The whole concept of common pitfalls is just great - read it and you already know things you should be avoiding, you don't have to learn the hard way.

Digital signatures in Java

We needed this for work, so while learning I wrote this very small program.

package digisign;

import java.security.InvalidKeyException;

import java.security.KeyFactory;

import java.security.KeyPair;

import java.security.KeyPairGenerator;

import java.security.NoSuchAlgorithmException;

import java.security.PrivateKey;

import java.security.PublicKey;

import java.security.Signature;

import java.security.SignatureException;

import java.security.spec.InvalidKeySpecException;

import java.security.spec.X509EncodedKeySpec;

/*

 * http://java.sun.com/docs/books/tutorial/security/apisign/index.html

 */

public class Main {

    public static void main(String[] args) throws NoSuchAlgorithmException, InvalidKeyException, SignatureException, InvalidKeySpecException {

// create some keys

KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance("RSA");

keyPairGenerator.initialize(1024);

KeyPair keyPair = keyPairGenerator.generateKeyPair();

PrivateKey privateKey = keyPair.getPrivate();

PublicKey publicKey = keyPair.getPublic();

// sign some data

byte[] data = "Hello, world!\n".getBytes();

Signature signature = Signature.getInstance("SHA1withRSA");

signature.initSign(privateKey);

signature.update(data);

byte[] sign = signature.sign();

// serialize public key and hand it over

byte[] pubKey = publicKey.getEncoded();

// store to disk, read in etc...

X509EncodedKeySpec encodedKeySpec = new X509EncodedKeySpec(pubKey);

KeyFactory keyFactory = KeyFactory.getInstance("RSA");

publicKey = keyFactory.generatePublic(encodedKeySpec);

// verify signature

signature = Signature.getInstance("SHA1withRSA");

signature.initVerify(publicKey);

signature.update(data);

boolean b = signature.verify(sign);

System.out.println("signature " + (b ? "ok." : "invalid!"));

    }

}

Software Sloth

I know I should be babysitting, but I saw Ian's nuanced post about technical debt and thought he left one out, and it is a cardinal one.

The fundamental problem is you either do not care enough and/ or you are too proud to change your was. The blogosphere is full of good advice, there are loads of great textbooks available, and peer learning is catching on - so help is available. So why do some still not "get it" and start producing good software?

Technical debt can be excused if it is backed up my conscious, well thought through decisions. There can be justification for skimping on quality. However, I contend that most technical debt is accrued through sloth. You either do not know what quality is, because you do not care enough to go and find out. You do not know the price of short sightedness. You cannot be bothered to read good books, or participate in collaborative peer efforts to better yourself. Or you are sinister, arrogant and not interested in improving your ways, not receptive to better ideas - even when you see mounting technical debt all around you.

I am a firm believer in evolution, and if we frame the problem thus, it is clear what the real problem is: because there isn't enough talent to go around, to fill enough posts and attain critical mass, sloth is allowed to thrive. Evolutionary pressures only work when resources are scarce, and sloth is apparently a smaller problem than getting bums on seats and poor code out the door.

Or maybe I am mistaken, and quantity is the real measure of success?

Hello, Nagios\n

Nagios is a generic monitoring tool, and I want to use it to monitor my own application. It has a lot of built-in and offical plugins. However, it is always good to try a Hello, World-type example.

Firstly, my very interesting application. It follows the Nagios Plugin guidelines: return a value in {0, 1, 2, 3} and a status message of less than 4k to standard out. And it gives you some random sysadmin-action:

(np.c, compiles into an executable np)

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

int main(int argc, char* argv[]) {
 srand(time(NULL));

 int r = rand() % 10;

 int status = -1;
 char* message = NULL;

 switch(r){
  case 0:
  case 1:
  case 2:
  case 3:
   status = 0;
   message = "Everything is alright :-)";
   break;
  case 4:
  case 5:
  case 6:
   status = 1;
   message = "You have been warned.";
   break;
  case 7:
  case 8:
   status = 2;
   message = "She is gonna blow!";
   break;
  case 9:
   status = 3;
   message = "Oh shit...";
   break;
 }

 printf("%s\n", message);

 return status;
}

Nagios is easy to install with this quickstart guide. You just need to configure it:

1) In $NAGIOS_HOME/etc/nagios.cfg, add a line cfg_file=$NAGIOS_HOME/etc/objects/np.cfg

2) In $NAGIOS_HOME/etc/nagios.cfg, set interval_length to a suitably low number, e.g. 5 seconds, so you can see changes quickly

3) In $NAGIOS_HOME/etc/objects/np.cfg, make it look something like this:

define hostgroup {
  hostgroup_name        app_hosts
  alias                 Application Hosts
}

define host {
  use                   generic-host
  host_name             app_host
  alias                 Application Host
  address               localhost
  hostgroups            app_hosts
  max_check_attempts    10
}

define service {
  use                   local-service
  host_name             app_host
  service_description   My Application
  check_command         monitor_np
  normal_check_interval 1
  retry_check_interval  1
}

define command {
  command_name          monitor_np
  command_line          $APPLICATION_HOME/bin/np
}

The configuration is quite fidgety. Lots of required attributes, no sensible defaults. Luckily you can reuse example configuration that comes with Nagios. And there is a handy pre-flight-check feature, you just run /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg and it will give you errors and warnings against your configuration files, with line numbers.

Anyway, by now Bob is your uncle and you can restart nagios to pick up changes. Then point your browser to http://localhost/nagios and watch the action!

Any Fool Can Sysadmin

I have Windows XP on my laptop, but I needed a Linux environment for client work. The last time I played around with dual booting was in 2000 when I had just bought my self-builder home computer. I installed Windows 2000 then RedHat 6, but the whole thing failed (I suspect my then very new motherboard didn't have drivers, or something) and I had to wipe everything (using my home sysadmin skills: if anything goes wrong, wipe and reinstall!).

So, it is fair to say I am a non-sysadmin, and maybe that's why I am sometimes accused of being a girly drinker. But with VMware I am suddenly sucessful and confident - in fact I am so confident I might just run as root - it is a vm after all, what could go wrong?

I found this guide extremely helpful. VMware Player is free and easy, the guide helps to hook things up, Ubuntu is quite user friendly. Why was I so scared of sysadmining for so long?

Well, these are early days, and I might just revert to my old self if I get a problem with Linux - wipe and reinstall. But since it is a vm, that is actually a painless solution...

Actually I forget; I first tried installing Damn Small Linux using a VMware Appliance, since it was only a 50MB download. However, I didn't manage to get the network working. And my colleagues and I didn't have a clue how to fix it. So I ended up with Ubuntu. Morale of the story though, sysadmining is no fun if you can't easily wipe and reinstall.

Software Apprentice

Kerry is always talking about Dreyfus and nursing competencies.

That got me thinking: is there a book you can give to a software developer that he can follow and use to improve himself through the five levels?

Well, I haven't read it yet but Elizabeth Keogh recommended me Apprenticeship Patterns: Guidance For The Aspiring Software Craftsman. I shall put it on my list right after finishing the DDD book and several volumes on how to raise babies.

More on this topic: http://dannorth.net/2008/06/learning-to-lean - loved the Bruce Lee quote at the bottom.