Do it on the device, or do it on the server?

This weekend, I thought I'd extend my little Android usage tracking application to work on more ISPs than the one (Internode) that it already does. As my phone is (sadly) on Optus, I thought I'd write one for that.

Internode was easy to add, as they have a documented API for accessing usage counters which are ideal for computer consumption. Optus on the other hand only provide a web application interface to check usage, necessitating the use of a web scraper. A web scraper is an application that pretends to be a user on a web page, makes all the appropriate calls (and fudges any javascript calls that are necessary) to get the results it needs. It then parses the (often non-compliant) HTML results that come back to get data. I have no problem doing this, and have done so on several occasions before, but it is not easy work and can be quite fiddly.  Parsing the HTML is often the most difficult part, as it is usually not well formed XML so you can't just use DOM to parse it.

In short order, I had a working prototype that used JTidy to clean up the HTML into something that I could use properly and then XPath to extract the elements of the document I needed.  It works great, except that the document clean up and parsing into DOM takes a really long time on a resource constrained device such as a phone.  It takes about 20 seconds to clean up and parse the document on my development emulator, which is too slow to produce a good mobile experience, especially if you have to parse multiple documents as I do.

So now I'm faced with a choice.  I could write a man in the middle service that the phone sends the user's login details to which then performs the parsing on behalf of the user and sends the results on to the phone, but there are a number of drawbacks to this:

1. This means that the user is sending his login details to a 3rd party, which is a security no no.
2. It introduces a single point of failure into the equation.  If my app gets popular the middle man service could get slammed.  If Optus decides that they don't like what I'm doing, they could easily block it.
3. It means I need to host a service, which means additional expense.

I don't want to do this, so what I'm left with is more hacky solutions, using regular expressions to find what I want in the HTML documents retrieved from the provider.  This will take me longer to code, will be more prone to failure, and is just generally nasty.  I'm not happy.  Devices these days are very powerful, and there should not be the need for intermediary servers to help with processing.

Of course this would all be much easier if the providers published web services interfaces to their data, rather than just web applications.  This has been the mantra of SOA and internet connected businesses since the terms were coined.  It doesn't even cost them that much more to do it, and would lead to better designed web applications, but thats a subject for another rant.  Optus doesn't do this because there's no economic incentive for them to do so.  They gain nothing directly from publishing a usable web interface, so they can't be bothered... bah!

To be fair to Optus, they aren't the only ones that don't get it.  No ISPs and telcos provide any decent interfaces, other than Internode.

I wrote an android application

I purchased an Android phone a few weeks ago. Part of the reason that I got it was that I wanted to see what the differences were between Android and iPhone.  This extends out to how to program them as well, so I had to write an application, just like I did for iPhone.

Last night, I released my little application.  Its a usage meter for my ISP, Internode. I deliberately chose something quite simple so that I could cut my teeth on the platform, and I must say that I'm very impressed. I found it very easy to write the application for Android, especially because it uses the same tools and libraries that I am used to using for my day job.  One other difference that I notice is that it is much more obvious what is going on inside an Android application.  The documentation describes things clearly and gives you full visibility.  Apple on the other hand like to keep their platform a little more mysterious.  There are plenty of good documents on how to do things, but you still get the impression that there's something going on under the hood that you don't quite understand.  Perhaps its just that I'm more familiar with the Java ecosystem.

If you'd like to have a look at it, check out its site at 8-bit cloud.  I've also been playing with the web site, and I will be improving it and hopefully making it more fun.

PS: I was tempted to call the application NodePony after a recent meme that Internode has got going with its cute little plush toys.  In the end, I decided that I shouldn't try to cash in on it.  It would spoil the meme...

What do I do now?

I've been trying to work out what I want to do with my career recently.  When I've been speaking with mentors and colleagues, the first question that comes up is quite reasonably always "What is it you want to do?".  I have to admit that this question has had me stumped for some time.  For the last three years, I've been working as a principal consultant in the system integration space for medium to large size businesses, and I have not found it satisfying.  I do the work well enough (some say even exemplary), but I haven't been able to summon the passion that I can put into work which allows me to excel.  I find myself compelled to seek more responsibility and higher pay, but its just not working for me.

Yesterday, I went to see Stephen Fry speak.  It was a very entertaining journey through Stephen's life, and the things that make him tick.  He called it his personal WWW, the things that inspire and drive him that for the contrivance of his topic all start with the letter W.  One of the Ws that he spoke of was Writing.  He doesn't just like to write.  Its not that he is paid to write.  He is compelled to write.  His love of language and how it can move people is his passion, his reason for getting up in the morning.

For many people, working in IT is just a job.  They have other things in their life that they call their passions, and they just come to the office to get a pay cheque.  I have many interests and hobbies, but my central passion is programming computers.  I view it as a creative pursuit.  Not art per se, but a craft that can produce elegant and useful things.  When I come home at night, more often than not I end up still camped in front of a computer.  So for me, computers are my passion, my raison d'être.  As with all passions, this gives me an edge.

It goes beyond being a code monkey however. Sometimes when I discuss the topic, people say "oh, you'd get bored just doing programming", because I've worked as a solution architect for so long, and I can operate at a business level as well as a technical one.  This is true, but not for the reason that people think.  I'm a bloody good programmer, an nothing gives me more pleasure than solving a technical challenge, but that's not enough to make something that works.  Creating modern software is a complex task, and to do it correctly there is a mix of communication, leadership and technical expertise required. I'd arrogantly like to think that I can perform all of these tasks.  I like the challenge of setting up simple, streamlined processes and teams that get jobs done.  It'd be nice if I got to do a bit of hands on work, because a true leader is a doer as well as a manager, but its more of an oversight, training and review function.

Which technology I work with is largely irrelevant.  There are some that are more interesting to work with than others, and some I have more experience with than others, but all are interesting to me.  Because technology is my passion, I pick new techniques and languages up very easily.  My company recently had a need to develop an iPhone application.  It just so happened I had been playing with iPhone programming in my spare time.  Where other engineers would have said "no, I haven't done that before" or asked to go on training, I relished the opportunity to pick up something new and got stuck in.  I trained a small team and let it to success.

So much of consulting consists of going into organisations that do not operate effectively.  They are rendered moribund by internal politics, people of limited talent in positions of power, and the sheer difficulty of organising a large workforce (usually too large) that is quite often afraid of change.  This is the reason that they bring in outside expert help.  These organisations get the job done, quite often a boring one, and they turn a profit.  But they do not produce exceptional results.

There are people that are very good at going into dysfunctional situations and turning them around.  I have immense admiration for these people because they do a very difficult and often thankless task.  They turn failure into a bare pass.  They can take pride in the fact that they put the hard yards in to get a result, but the end product is rarely anything to rave about.

I have performed project recovery work successfully before, but I am rarely given the opportunity to effect any real change.  I'm brought in as a technical specialist, often very late in the piece when many of the decisions have been made, to solve a particular problem, often within ludicrous constraints that don't make any sense in the context of producing something that works.  I've lost count of how many times I've been asked to work on a "platform project" to put in place tools and procedures for an organisation that does not have any function to put on that platform yet. How pointless.

So here's what I want from my job:  I want to create good software that does useful things. To do this, I want the freedom to be a good chief nerd (or lead engineer, or architect, or whatever else you want to call it).  I know what I'm doing.  Just get out of my way and let me do it.

My challenge now is to engineer the opportunity to do this.  To quote a review of yesterday's performance that appeared in The Age, "Fry proves that we can take power over, and joy in, the role that is ourselves.".  Time to prove all the outrageous claims that I've just made :)

Samsung Galaxy S - 3 weeks in

My ageing iPhone 3G didn't take the update to iOS 4.0 very well. It still worked, but it seemed clunky and slow after the upgrade. Six weeks ago, I (conveniently) took this as a sign that I needed a new phone, and started considering my options. I could wait for the new shiny iPhone 4, or I could look at Android. It has been reported that Android has caught up to Apple now, both in terms of hardware and software. Of course, the biggest thing that everyone touted as the biggest advantage of moving to android was: freedom. Being a developer and nerd, this appealed to me.

So I went out and bought a Samsung Galaxy S I9000 (now there's a mouthful of a name...). Its got a big super amoled 4 inch screen, a very fast processor (reportedly faster than that in any other current phone), and came for free on a very reasonably priced Optus plan. I've had the phone for three weeks now, long enough for me to sort out any learning curves and kinks in the system, so I thought I'd post my impressions.

The biggest thing I noticed buying this phone was that it didn't "Just work" like an iPhone does. It turns out that there was a performance problem which detracted from the new phone feel that I expected. In order to get it working properly, I needed to go onto a bunch of technical forum sites and work out how to flash a new (leaked beta) firmware to fix the problem. Flashing was made even more difficult because they were windows only flashing tools, and they wouldn't work in a virtual machine on my mac. In the end, I needed to drag an old clunker machine out of the cupboard to get flashing to work.

The headphone jack also uses a different configuration to iPhones.  This means that I can't use my existing Sennheiser headset on the phone.  If I do, the sound ends up muddy and the lyrics almost disappear.  The supplied headphones are ok, but I'd still prefer to use my aftermarket headset.  Apparently Samsung have used the same configuration as Nokia, which makes some sense, but it still pisses me off that I need to use an adapter to get my headphones working.

Now that I've passed those problems, I can start to see why this is such a great phone. I now get twitter & yammer notifications on the phone in an unobtrusive way,  and there's no waiting for updates either.  On the iPhone, when there was a new message, a notification would appear, and when you clicked on "view" it took you to the application which then loaded the message (or whatever) from the internet.  This could take a few (up to 10) seconds.  On android, it loads the message first, then shows you the notification.  No waiting!

There's also widgets on the home screen. If I don't like the default home screen that Samsung Provides (TouchWiz 3.0) I can replace it with a number of open source alternatives that are much whizzier and glitzy. There's a cool little active background that makes the screen look like you can see through to a circuit board. There's just so much stuff!!!

Of course, there's also a downside to this. I installed a couple of different music players to try them out. When I pressed the "play music" button on my headset, all of them started at once :S. I worked out how to fix this, but its another example of not "Just working".  I also need to do a bit of research when I want an application, because there isn't just one way of doing it.

For work, there's exchange support built in, so it works just as well as an iPhone in that department.  There's also an office docs reader built in, and all the applications we are used to such as dropbox work as well, if not better than on the iPhone.

Battery life is okay.  Its taken a while to settle down into a pattern, but I'm now getting almost two days between charges, including moderate usage.  The firmware upgrade has been a big help here.

Finally, there's the looks. I have to admit that one reason for moving away from iPhone was that everyone has one now. Sadly, the I9000 doesn't help here, because it looks exactly like an iPhone 3GS.  They even copied the packaging with the phone coming in a little black cardboard box.   Its so similar I'm surprised apple didn't sue them. Oh well, its a pretty vain reason to want to change.

PROS:

1. The screen is gorgeous.
   
2. There's a lot of choice of applications
   
3. lots of storage - 16Gb internal, which I have since augmented with a 32Gb MicroSD card to give me a total of 48Gb.
   
4. SWYPE is a great text input method
   
5. Google integration is great
6. When Android 2.2 comes out, it will get much faster.

CONS:

• Performance problem - Now Solved!
• Doesn't like OS X very much
• There's a lot of choice of applications
• The Games aren't as good.
• Only available on Optus at the moment.  I would have preferred to stay on Telstra Prepaid

All in all, now that I've sorted out my performance problem, I'm very happy with my new phone. I like the choice that I get, and I'm willing to invest the time it takes to tweak the experience to my taste. In the end, I suppose thats the difference between the masses and the nerds: Some are willing to be told the best way to do something, whilst others prefer to tread their own path (or at least the path that is shown them by a bunch of other nerds on a forum)

Now, to work out how to program the thing!

I'm giving a lecture on Cloud Computing

This post is basically just to try out embedding a google wave post in a blog post :) There is some news however.

I am giving a lecture to a Masters of IT class at Monash. This wave is the place where I am gathering information about what I will present. I will also provide access to the students during the class, if they wish to participate or continue the discussion at a later date.

Date and Time: Friday 4th June 2010,
Location: Rooom H125, Monash University, Caulfield Campus

EDIT: Hmmm, it seems that Disqus, the service I use for managing comments on this site, includes the script that is used to view google wave. This means that there will be an unfortunate doubling of the content below.

Google Reader also strips off the google wave component, which makes sense considering the content is added by javascript. We wouldn't want nasty javascript content sneaking out there into reader land, would we?

Good to know.

Distributed social networking

With Facebook's recent stumbles in online privacy, a lot of people are now calling for distribtued information sharing systems, which people can use to interoperate between different servers, different providers, or run your own if you so choose. The Diaspora project has been started by a bunch of grad students to do exactly that, and it looks like they've gone viral.  At the time of writing, they had raised over $160,000, when all they were looking for is $10,000.  Not bad for a first round of funding, especially considering all they've done so far is produce a video.

After talking about this at the pub on friday (the pub is where I do most of my best theorising :)), a friend suggested that I look at the Friend of a Friend Project. He suggested that distributed social networking has been around for ages.  Unfortunately, foaf does not deal properly with security or webs of trust, so I feel that it is useless for what most people would want.

There's no technical impediment to stop distributed social networking from working.  In the end, it would be a much more secure, robust and scalable solution.  As we've seen a number of times, Twitter sometimes struggles with stability and when it goes down, all of Twitter goes down.  These are the original principles upon which the Internet was founded.  The problem is that modern, slick systems take a lot of resources to put together.  Many people and companies don't view the internet as a way of speaking together in an interoperable fashion any more; they want to make money which means not being interoperable.   Google, Yahoo, Twitter and Facebook provide these services so that they can make money of the information we give them.  We make a choice to trade our privacy for the service that we are provided.

I wish the Diaspora guys good luck.  The biggest hurdle they will face is getting the masses to care.  In order to get enough take up of their solution, they will need to either convince your average user to host their own server, or find companies that are willing to host the service but not get access to any of the data. I can't think of any big company that would want to do that.   There's nothing in it for them if they can't mine all of your data.  Most ISPs will host email for you, but they do it as a method of lock in.  once you're on me@localisp.com, it provides a reason to stay with that ISP rather than taking your email with you.  Besides, email was well entrenched before the rise of the ISP.  Every ISP had to provide email. It was just expected.  There's no reason to start hosting a social networking server unless everyone else does, which means you've got a chicken and egg problem.

Its something I've been thinking about a lot this weekend.  How do we make the general populace care enough about their personal data to protect it?  I don't know the answer.

Easy Exchange Email Extraction

Okay, that should be forwarding, but I wanted some alliteration :)

Some time ago, I posted a very technical approach to forwarding email properly using Microsoft Exchange, which is useful if you work at an organisation that uses it. At the time, I was aware that the steps involved were to technical for some people, so I didn't really expect too many people to take it up.

Some colleagues at work expressed some interest in using themselves, but didn't want to go to the effort of setting up their own man-in-the-middle server to fix the emails. To combat this, and to make it easy enough for anyone to set up email forwarding for exchange, I updated my forwarder to work for any email address. I now present this work to the public, in the form of the Crimson Cactus Exchange Redirector. If you would like to perform exchange redirection easily, this might be of some use to you.

In fact, I set this up quite some time ago, but I've been too frightened to publicise it until now because I was concerned that it might become a source of spam or too much traffic on my web server. I've finally gotten up enough courage to try it out now, but please consider it a beta service. It may have to be turned of at short notice if it is abused.

Please give me your feedback if you find this service useful.

Bruce.