Joost Blog

After a month of hard work from many, several all-nighters, and with the release of beta 0.7.2, we are now serving users from our first and main production data-centre in Luxembourg City. Within only a few hours of flipping the switch and bringing on-line our new Long Tail Servers, we're already seeing it take over 75% of the long-tail traffic. We now have over 10 times the capacity we had just one week ago, and we'll be doubling and then tripling that in just the next 2 weeks, which is a great help to us as we ramp up the amount of invites going out.

To help us manage the beta release, along with controlling invitations, each release has a time limit on its certificate and 0.7.1's certificate is about to run out. That's why you'll get a message telling you you have to upgrade to 0.7.2. But we have put a number of updates into 0.7.2 as well; to support the new data centres, fixing a number of start up problems and letting you run the program on Vista (as administrator).

We had a few small hiccups along the way to Luxembourg, we even got lost in Brussels just trying to drive to there (we're spending the extra €5 a day for a GPS unit next time), but it's a huge deal for us to have our first site operational. It also means I can sleep a lot more soundly, knowing we are no longer hovering close to our limits on the Leiden bandwidth! As Dirk mentioned before, we had a router fail on us just before departing. After a very long day testing everything, connecting everything together and shipping a few dozen terabytes of test traffic, it was pretty gutting to have a router fail just before I was ready to leave and get some sleep. At first, I thought it was the console cable, but no - it was definitely the router. Although we had a backup-plan, it was an enormous relief that after contacting the Cisco Technical Advice Centre at 03:06, they had a replacement on the way to Luxembourg by only 05:30 (in fact, it was there before us!). We've had no problems with the replacement unit, and this is why we test equipment before putting it into production.

Although Luxembourg is our first and main site, others are scheduled to come on-line early next year, we want our network latency to you to be as low as possible, so that when you change channel it happens as quickly as we can make it. We're paying close attention to how things go now, as we'll be repeating the same design many times. We're also paying close attention to our usage statistics, to decide where best to place our sites.

Over the next month or so, I hope you'll forgive us any network blips. We're still working hard to get true resilience in place, but once done we want to have geographic, network and power resilience with carrier-grade availability, just like TV. Right now we have 25 Sun x2200's (some of which will be replaced with more powerful dual power-supply machines in the near future), spread across 4 power feeds, to serve content. Each server is connected to at least 2 different switches, we're using Cisco 3750's and all of our traffic is routed by a Cisco 7301. We'll be adding a second router shortly, but in the meantime we're using the switches very healthy routing capabilities to provide equipment resilience.

Athbhliain faoi mhaise daoibh!