So for the last few weeks as part of my learning python journey I've been knocking together bits of a url shortener service which is complete and functional. I've just been toying with things and basically learning through trying, and a bit of benchmarking I guess. As part of getting it ready to deploy (on app engine) is to find a name. Obviously a short name. So I came up with quite a few that I really liked, two were pure gold and another not far off. Of course the domains were squatted on so I had to move on and keep thinking. That really gives me the shits. Two of them even didn't bother to have dns hosted. If you buy a domain at least have the decency to host the dns and redirect the site to something.

Perhaps the flip side to all this silliness, is I still think url shorteners are stupid. Remember goatse or more recently the rick rolling phase. Link shorteners were another way of fooling people to visit links they'd otherwise know not to, purely by them looking at the url before clicking. By using link shortening services you're taking away someone's ability to control where their browser goes (unless they know that services way of previewing), and worse, you're making a statement that you know where they want to/should go to for them. As for reducing typing at the expense of communication speed it's even more terrible; big a little b number 7; bloody slow as if you ask me. If someone is linking to your site it should be more accessible than that, they shouldn't need to make short links. I realise some will use analytics off their shortened links, or they're shortened for a reason like twitter and fitting into 140 character messages (ug). The other reason why they're stupid, which was highlighted when various url shorteners closed (trim, cligs) is all the broken links when the company goes bust. The Internet Archive is keeping a copy of them for some of the shorteners but not all, and without the domain name it'd still be a broken link. This is not a new problem, but nor are url shorteners either. As most of them introduce a single point of failure, it's good to see google offering one too. Theirs performs well and has a pretty awesome api too. Seems like it's now a solved problem. Perhaps I should have picked another very simple web service to build instead.

As for trying to build an app using new(old) technology to solve a mostly trivial problem while learning a new language, even that was slightly miss aimed. Most of them seem to just use standard technology with some decent caching on the front (which is an obvious performance boost). Example: until very recently is.gd used php+mysql for it all, though now that page has moved (thus archive link) and their replacement page says it's mysql+mongodb+php, which is where I was going (though using python instead of php). Based on the sheer number shorteners out there, I'm not the only one who knocked one up as something to do.

So that's sort of on the chopping block before leaving the starting gates – unless I think of a cool name soon.

Completely unrelated to that, I called up telstra recently to activate a prepaid mobile broadband service. I'll start by saying they haven't changed anything since the last time I called up to activate a prepaid wireless broadband service. Changed n-o-t-h-i-n-g. Nada. As I knew it would be a long and painful call I called from a land line (non telstra). After being probed for enough id to get a car loan and supplying an existing customer account number (in matching name - yes it was my already existing account) I was asked if the service 0400 xxxxxx was my land line phone. Wow. I mean, wow. Not only is that totally unrelated to the purpose of the call, it also showed to me that the person I was speaking to was clearly not in or from Australia (for the overseas reader all mobile/cell phones here start with 04xx). The script being followed was just about read out word for word too. What a pain in the ass. It's not like I'm phoning up with a complex problem, I'm not requesting the plans to build a drug lab or nuclear warhead delivery system, nor am I asking for large prime numbers to be factored. Why is this so difficult? That's even beside the point; the prepaid service doesn't need anything account wise on their end. If you run out of credit it stops working, you can use the service to recharge. There's no line of credit there, so why is it so hard. I'm sure they just want a name attached to the account should you do something dodgy and law enforcement comes a knocking - what other reason is there for them needing to know who's on the end of an activated service, it's so the man can track you. Obviously no burners or trac phones for us. So we'll have to make our own.

Clearly the man is out to get us, all of us, even if we've done nothing wrong.

Billy said something about needing more pictures, and I completely agree.

I noticed recently that a google apps account now is a real google account, and can log into most (nearly all) services that a normal google account can log into. So this kicked off the idea of migrating from gmail to google apps mail again. Originally I didn't because of features not present in the apps version, but now I look at it, nearly all the features I care about are across (new contacts tool isn't but I don't use it that often anyway).

Fortunately syncronising most things in and out of google is easy, Data Liberation made all the instructions easy to follow and located in one place. I also only use a few of google's applications anyway; reader, gmail, docs and more recently picasa web. All except mail were very easy to export/import between the accounts, but mail still proves to be a challenge.

This time around (circular reference) I noticed gmail-backup doesn't actually back up all of the email in the account (it misses sent items, and doesn't copy/apply stars). It's linux version needed Python 2.5, I have 2.6 and 3 (Fedora 13) and that wasn't good enough. Running it under wine backed up ok, but resulted in an unhandled exception part way through the restore, so it too was useless. As a last resort I ran it on windows and once it finished restoring, the message counts were wrong. What the hell? BackupGoo wasn't immediately clear that it could only backup and was trial-ware. Lame.

Then I went to consider doing it manually, thunderbirds away! Well that was destined for failure too. It had no way to mass import .eml files (from the backups) and drag-n-droping them in didn't work (on linux – might have on windows). I found an add-in ImportExportTools which was actually quite useful, it exported the whole mailbox to a series of mbox files, which I could then import. Great I was thinking, until I noticed again the message counts were wrong. It wasn't exporting all messages. What the hell?

So then I poked around looking for an imap copy application, and after looking at several, downloaded imapcopy which looked like it was going to be fine. Alas, another red herring, half way through the restore, it dumped an unhandled exception and a stack dump. So I grabbed the code (java) and fixed that bug, ran it again, found a few more issues, fixed them. After a few cycles of that, adding more debug code and logging to help track down where the messages were being duplicated, it's now complete. I'll have to split out the bug fixes from the hacky debug code and submit a patch.

Next up on the geeky bits. Ages ago I replaced my old WRT54GS router with an even older SnapGear router due to the WRT not handling the throughput of my adsl line (22mbit). Then around Christmas last year I bought an Ubiquity Networks RouterStation Pro (board). Due to power supply related difficulties, lack of a case, the need to build OpenWRT trunk (and me failing at that this time) it sat in an anti-static bag for about 10 months.

However one day after coming back from Timor I sat down and decided to get it going. I checked out a clean build tree of OpenWRT (not my ages old one from ye old days which was the cause of my failure) and built a very basic image, successfully. In cleaning up the house and chucking out stuff, I found a power supply which happened to have the same plug (and polarity) as the board needed, but only 12v. Everything I read said the board needs 24v or more for the mini-pci wireless cards to work right. So I checked the power supply's output, and it was about 15v, nice. Even better when the board booted up. Although it already comes with OpenWRT on the board, it is a pretty outdated trunk version which is why I started with building a new image. So I had a quick poke around the now old firmware, noted a few things, tar'd up the root image and flashed my new one to it.

Flashing it was, umm, tricky at first. The board didn't come with much in the way of documentation and it's sort of spread around the net rather than being located in a few obvious places. Fortunately again, the OpenWRT site's guide for this board was mostly spot on.

Quick guide for the lost/interested out there.

1. Grab the image from the OpenWRT site, you'll want the one for the RouterStation PRO. Snapshot or latest RC (right now, Backfire 10.03.1-rc3) If you're flashing with tftp you'll want the factory image, and I recommend the squashfs version (unless you have specific needs and know what you're doing) So grab: openwrt-ar71xx-ubnt-rspro-squashfs-factory.bin

2. Connect your computer to the WAN port (or the WAN port to your lan etc)

3. Configure your network like this: 192.168.1.2/255.255.255.0 (anything but 192.168.1.20 and 192.168.1.1) On linux all I do is bring up an aliased network device: ifconfig eth0:0 192.168.1.2 netmask 255.255.255.0 up This will not affect your normal eth0 connection. YMMV if you have a firewall setup locally.

4. For peace of mind, leave a ping window running, pinging 192.168.1.20 (use -t if on windows, so it keeps running).

5. Here's the tricky bit. While holding the reset button, plug in the power to your RouterStation Pro and hold the reset button for at LEAST 10 seconds. You should see the third LED flash a bit and the second one is supposed to come on, but I don't think mine did.

6. The pinging should succeed here.

7. Send the firmware image with tftp in binary mode tftp -m binary 192.168.1.20 -c put openwrt-ar71xx-ubnt-rspro-squashfs-factory.bin In theory after flashing the router will reboot and provide dhcp on the switch (the other 3 ports). You could now telnet to 192.168.1.1 as usual, root with no password on the LAN ports. SSH won't work until you set a password, and when a password is set, telnet won't start on boot anymore.

If you're like me, and slightly uncoordinated (or drunk), you can set tftp to retry and leave it running like the ping window as follows (linux only most likely): echo -e "binary\nrexmt 1\ntimeout 60\ntrace\nput openwrt-ar71xx-ubnt-rspro-squashfs-factory.bin\n" | tftp 192.168.1.20

This way as soon as the board is ready for the image, it will be sent.

Once you're on a half recent build of OpenWRT you can flash using the sysupgrade image either from the web gui or from command line, which saves all this mucking around with tftp. Not that I'm knocking tftp, I did build a client and server in C# as an exercise – and it performed quite well.

After some fairly basic setup I was able to swap my routers over and it's been great ever since. I now have wireless again, and now it's 802.11n. Yay.

I guess that'll do for now.

Ok, so this was a bit of a work in progress. Basically a collection of screwups lead to an entertaining side effect, and then as these things do, swings and round-a-abouts later bit me in the ass.

So after the great Perth storm, I had no power for 2 days but when it came back I basically had free quota (after a power cycle to reconnect at a decent sync speed.) Not being one to jump to conclusions I initially just assumed it was temporary and would correct itself with the whole backlog of usage.

After a few days and finding no one else had the issue I decided to hammer it. And hammer it I did, to the tune of at least 1.25TB over 10 days. I say at least because since I stopped bothering with traffic accounting at home it's really hard to say. My Usenet account shows some pretty amazing stats, but that wasn't the only source. Roughly 120GB or so per day for about 10 days later, and including lots of PSN content and other stuff. All my Linux and Solaris iso's are up to date and so on.

Due to this, my forecasted storage upgrade had to be brought forward a month or two, and with my 1.1TB snapshot which I still couldn't delete (zfs dedupe related bug) I had to build a whole new pool. Sigh. So after some browsing all the local sites for the cheapest 2TB options, and emailing my usual drive provider who didn't have the model I wanted (unfortunately) I bought some from a store I don't usually frequent. I had to pay cash as the reason I don't frequent them is a stupid 5% surcharge for using common modern technology (Credit cards). Also I didn't really want such a large purchase attributed to me by name. I know the types of people who work in these places, and I'd rather they don't spread that information around. Also funny was the receipt had no serial numbers listed. Also adding to my dodgy opinion of such store.

So after mounting the new disks in my backup filer I did some initial disk testing to make sure the disks were ok (dodgy store etc) and then commenced the snapshot sync. 2 days later it was done, and it had 250gb more to sync. Whee. Once it was all up to date I had to swap the disks between the 2 cases. Only then was I able to create the new backup pool (bigger disks now) and sync back the other way. Fortunately ZFS makes it very simple to move stuff around with snapshots and send/recv. There is no other way to handle datasets of this size as nicely. Though when btrfs matures, it could become my choice due to being on Linux (only time will tell there, ZFS still has a pile of odd quirks). So I've now got an upgraded storage subsystem, still on Solaris/ZFS and still with a full mirror on the backup box.

Now for the first karma bit.

I knew this error couldn't continue for ever, and after a while you do run out of things to download. There were day long gaps in there before I thought, "why don't I get that". Ohh 250gb thats too big. Click. It was actually quite draining (emotionally more than sleep time). The thought of unlimited and then not being able to use it. Of course this only comes from living in a quota'd world, if you never had quotas and could get anything when you wanted it, you wouldn't store anything either, where as here you store everything and max out your quota every month because you use it or lose it. Ok so when my connection dropped and it reconnected, usage started counting... obviously that would be in the middle of the night, so right around 1am. By the time I woke up at 6am to stop it, I'd done 24gb of my 30gb quota. Whoops. Adding to that, of the 250gb queued, I had just over half, and being mildly obsessive compulsive, incomplete things are not good. So now I had enough to not want to chuck it away, but not enough to call it quits. Alas a dilemma.

Interesting sidebar: The connection history log showed "Admin Reset" rather than "Normal Disconnect" so I know it was knocked off deliberately.

I basically did nothing for the next 2 weeks until nearly the end of month. So having stuff queued up and a need to complete that, I proceeded to follow my usual trick. (Details not posted laa laa laa). So when I upgraded my plan, I clicked submit and the progress thing came up, and then nothing. Basically what happened is they had a major fault and the upgrade didn't work (no details posted on the fault beyond a thread on whingepuddle "Major service disruption", but it took out all their corporate sites, mail, streaming radio and lots more). The following morning the fault was fixed (according to the post on whingepuddle and the outage notification site) yet my plan change was not effective, so I did it again. Ah ha, now it all goes pear shaped.

Karma bite number 2.

Plan change claimed to take effect, but by 8am I was shaped. How could that be, not only should my off peak now end at 9am, I had plenty of quota unused. Disconnect, reconnect, fixed. Whatever, go to work. Few hours later I check on my queue. Hmmm, shaped again. Reconnecting didn't help this time, member tools of course show heaps of quota free and of course no shaping. Not being one to call the call centre (see rant below) I asked around why this might be and asked for my connection to be booted from their end to see if that fixed it (since the tool to do this in member tools was not loading). It did for a bit, until I got home at least.

When I got home, being the constant tinkerer I am, I noticed my transfer speed was going slower than it should, so I figured now was as good a time as any to swap modems (from a loaner to my new one). So I swapped them over, bit of down time aside for obvious reasons, and it reconnects probably 10 mins later. Shaped again. Assholes.

In many ways dealing with ISPs are like dealing with the government or simply the man. You can't talk to anyone who can really change anything, but you can talk to someone who might be able to pass a message along to someone who can influence something. That something might be related or not, but it usually can't make things any worse. So being generally annoyed and feeling that bite digging in I was all meh and decided to play some Rock Band and then watch some stuff before doing anything. Eventually I emailed in the fault, under the assumption (rightly or wrongly) that it might be fixed by a) Midnight, b) 1am, c) 2am without any involvement anyway. I prefer to think of things like my problems as non critical, so not to bother the call centre with it. But it's worth getting it logged because if it escalates you at least have that to fall back on when you've had no net for 2 weeks and they claim to just hear about it.

So I snoozed and at midnight it was still shaped, at 1am it was still shaped and sometime between then and 4am when I woke up it was unshaped. Kicked things off again, more snoozing, shaped again. Reconnect, all ok again. Hmmm. I noticed my email had been logged at 3am as an issue, so maybe it was "fixed" then. After talking myself into calling the call centre (quite a feat) to add to the issue that it was up and down again I called up. Informed of a 6 minute wait, logged a callback. While waiting for the callback I got shaped again.

Call centre rant.

I was prepared for having to follow the script, because to get anywhere you need to fit into a pre-defined customer mould otherwise the person on the other end, lets call him/her Sam (not the real name in this case) would not be able to help you one bit and could even get timid or scared from the unfamiliar situation. Sam was very caring and read the notes from the email the night before, and then I explained that it was pretty much the same thing and I wanted to add that it had continued on. Of course that’s not one of Sam's scripts, so we went down the power cycle modem, what modem, check cabling, username, password path, swapped back to the ISP provided (crap) modem. He/She couldn't quite get the modem-router two device idea, so I quickly glossed over that. I knew the problem was not my end - after all I could still drag data down off the ISP's ftp at full speed. Sam did port resets and so on, eventually they wanted to try my account on their test bench, which they did and reported (after 20mins on hold) that yes it appeared that I was shaped. Great, 50mins on the phone after a 10min wait to be told what I'd emailed in the night before. Excited voice: It was even ok to plug my modem-router back in ... they got me to unplug that while they tested my account.

A few hours later I got a call saying it was fixed and to power cycle my modem (grin). Of course being at work I was unable to do that, but was able to reboot the connection remotely, which I did and things were ok again. Fired up the queue again. By this stage my second chance at off peak was long gone and my 50gb of planned usage was wasted. Sigh. Also of interest; the issue was closed before confirmation was received. That was the end of Sam, bless his/her heart. Of course my stories never end that well. Sam's caring voice managed to reassure me that not only do they do an excellent job and offer great products at competitive prices; they just aren't always able to get things going smoothly without some further tweaking. Perhaps in this case treating the symptoms rather than the cause. About 20 minutes later another new problem started.

More karma?

Now the connection was dieing every 20mins or so. The logs from my router suggested the connection was just falling over and then a reconnection was failing causing further delay before it reconnected successfully. A friend of mine who I was chatting to on msn thought this was quite funny after a few hours but also realised that even though the NOC had been involved, they wouldn't know what to do to fix it and realised that the issue would continue beyond the quota month until fixed. He managed to pass a message of this situation to someone still at ii (clearly they don’t let all their good staff leave) and still in a position to be able to look at and fix the issue. From what we've been able to figure out, the traffic accounting system was not able to identify which account to assign the traffic counts emitted from my connection to, so it killed the connection. This process happens periodically so that explains the delay. The duplicate account in LDAP was probably caused by the failed plan change when their "major service disruption" occurred. Which is funny because databases have had ACID commits for years, perhaps the plan change developers were on acid while working. As soon as this error was fixed, the connection stayed up without issue, no power cycle needed.

The to add salt to the wound, while shopping around for prices on some other hardware, I spot the 2TB disks of the model I want are now available from my preferred supplier, for $50 cheaper (per drive) than I paid 3 weeks prior. That wouldn't be such a problem if I'd paid with credit card, but alas I did not so no opportunity to claim the price drop back. The irony here is when I upgraded to 1TB disks nearly 3 years ago they too dropped 20% within 2 weeks of my purchase. So to recap, if it wasn't for the start of month freebie, I might have saved myself nearly $500. Karma baby.

So what did I learn?

1. Against my best judgement I gave the call centre a go, only to be let down again for multiple reasons. Pity I didn't get sent the customer feedback survey thing, because the guy/girl did everything expected of them. No more, no less and would get top marks from me in nearly every category. Unfortunately they were not able to identify let alone fix the root cause due to it not being in their script book.

2. It's never totally safe to change your plan.

3. Taking advantage of free quota is perfectly fine and encouraged, because you'll get screwed over in the long run anyway, may as well lift the average.

4. Next time I invest in new drives, wait 4 weeks no matter what. (Though that could just infuriate me more if it then drops within 3 weeks of that again.)

Next post coming alot sooner than this one did, it's all about TV and stuff.

I recently had the opportunity to play with a pair of Netcomm homeplug networking devices. The idea was to use some of them to eliminate the two long cat6 cables in the house, of which one crosses a high traffic area (kitchen and passageway), the other just runs around the wall out of the way. Unfortunately for me, despite being labelled 200Mbps and being told I'd get real world 80Mbps I actually saw closer to 15Mbps. Yep, that slow. A single pair of devices on a single power circuit and doing a single large file copy through them sat on 15% of a 100Mbps link. The tool for the devices clearly indicating 170/143Mbps (TX/RX). And my house was built in the last 5 years by some most likely unqualified teenagers... but that's another story. At first I thought maybe it was my netbook being crap, over wireless it gets 31%/54Mbps (16Mbps), so that sounds fair, but then I tried the cable, and well 85Mbps blew that theory out of the water fast. So at least for me, it seems that my equipment is capable of 85Mbps and putting the homeplug devices inline reduces that to 15Mbps. Shocking.

But in the spirit of still trying what I started, I put the homeplug devices in place for the theatre's needs, after all thats only running 100meg now. Then I looked at the average bitrate needed to play various 1080p x264 files, and most were around 6-8Mbit, so that was promising. However, that was 3 second averages I was watching, so it was still possible for spikes in bandwidth required for high action scenes. In theory buffering should take care of that, but we know the WDTV lacks decent buffering (it does assume local storage after all) and most PC applications that are decent lack buffer options (vlc has them, but fails at the decent bit). Previousy I worked on needing 75+Mbps for smooth playback, which was a stick your finger in the air and take a guess figure, and with the usb nic on the wdtv it all played back smooth. With the homeplug it started out fine, and I plugged the netbook in on that side of the network to drag more data over concurrently, hopefully giving me a rough idea on the throughput being used for the playback (15 - what I'm getting = what it's using. Very complex). It was fine until, as expected, a high action scene and then the video started chunking up and my transfer had already hit 0bytes/sec several seconds before the first thunk. So basically unusable for 1080P, despite being the NP201AV edition. AV = audio video? Yeah right.

This was also ignoring the general incompetence of their installer/software cd. Ok I have 1 windows PC, a netbook, with no cdrom. I copy the contents of the cd to my nas using my linux box. I try to run it from there, I get an error, I copy it locally and try, same error. I copy it to the root of a drive and I get the same error. It couldn't find some file, of course not naming the file doesn't help. I considered making an iso of it but I didn't have any iso mounting tools on the netbook and didn't want to bog it down with more rarely used drivers. So I proceed to hang a usb cdrom off it, just to install a configuration tool. That worked.

Oh but it doesn't end there, the installer COPIES the dotnet framework installer to it's directory. Yep, despite it already being installed, it copies the installer over to my hard drive, and not one installer, three. The normal 1.1 installer and both x86 and x64 builds of 2.0. Obviously they're planning for my PC to magically reboot into XP64. Fortunately disk space isn't at a premium, but this totally unnecessary extra 100meg isn't warranted for a 1500kb configuration tool. If you considered that both 1.1 and 2.0 were already present (along with full visual studio 2005), and my cpu (N280) is totally incapable of running 64 bit code, it's a waste.