Blog Migration

Earlier this month, I posted a new website for my hobby company 8bitcloud.com. As part of this change, I have now decided to host my personal blog here as well. brucecooper.net will henceforth redirect to 8bitcloud.com/blog.html. All URLs will continue to retrieve the correct posts, but with the new layout.  There shouldn't be any disruption to service, but who knows. RSS feeds should automatically switch over as well.  Please let me know if I cock it up somehow, although if I do, you won't see this :)

So why have I done this? Blogger is a great platform, but I've been experimenting with jekyll as a website creation method, and thought I'd try it out for a bit.  If I don't like it, I think I'll move to wordpress, which I've been mulling over for a while, so perhaps this is a simple way of testing it out.  In the end, I did it so that I had something new to play with.  Yeah, I know... I'm sad...

I appreciate your patience.

New version of NodeDroid, now with Optus

I'm the author of a usage tracking utility on Android called NodeDroid. Originally it only supported Internode, but I am now expanding it to support other ISPs and telcos. The first one I want to try out is Optus. I've just uploaded the new version of the application, and I'm hoping some of you guys would like to try it out.

The application works by screen-scraping the optus web site, and presenting it in a better format, along with usage graphs and the normal sort of thing you would expect from a usage meter. In the future, I hope to add the ability to see bills, as well as provide warnings when your quota is being reached. That sort of thing.

Because there are a large number of plans, and they all have different rules, I would imagine that I will need to take into account a large number of factors. At the moment, I've only been able to work from my own plan, which is an Extreme Cap. It should work quite well for other Caps, but it will probably break on prepaid and other account types.

If you would like to participate, you must already have a log in to the optus account page. If my application can't read your usage, it would be useful for me to see what your usage page on the optus portal looks like, along with all the usage lines expanded. If you can provide a screenshot or source, that would be beneficial (remember to black out the phone numbers first though!!)

If you would like to participate, please fetch the application from the market by searching for NodeDroid, or through AppBrain Here

I will also be expanding the program to support other providers in the near future. Vodafone Mobile prepaid broadband will be easiest for me, as I have an account, as well as Telstra prepaid, but if you are extra keen on getting something, please send me a whim.

I would appreciate any feedback you can give, either here or on my website at http://www.8bitcloud.com/nodedroid.html. The website will be updated soon with details of the new beta.  I have also started a thread on Whirlpool where people can discuss it (or not...)

Do it on the device, or do it on the server?

This weekend, I thought I'd extend my little Android usage tracking application to work on more ISPs than the one (Internode) that it already does. As my phone is (sadly) on Optus, I thought I'd write one for that.

Internode was easy to add, as they have a documented API for accessing usage counters which are ideal for computer consumption. Optus on the other hand only provide a web application interface to check usage, necessitating the use of a web scraper. A web scraper is an application that pretends to be a user on a web page, makes all the appropriate calls (and fudges any javascript calls that are necessary) to get the results it needs. It then parses the (often non-compliant) HTML results that come back to get data. I have no problem doing this, and have done so on several occasions before, but it is not easy work and can be quite fiddly.  Parsing the HTML is often the most difficult part, as it is usually not well formed XML so you can't just use DOM to parse it.

In short order, I had a working prototype that used JTidy to clean up the HTML into something that I could use properly and then XPath to extract the elements of the document I needed.  It works great, except that the document clean up and parsing into DOM takes a really long time on a resource constrained device such as a phone.  It takes about 20 seconds to clean up and parse the document on my development emulator, which is too slow to produce a good mobile experience, especially if you have to parse multiple documents as I do.

So now I'm faced with a choice.  I could write a man in the middle service that the phone sends the user's login details to which then performs the parsing on behalf of the user and sends the results on to the phone, but there are a number of drawbacks to this:

1. This means that the user is sending his login details to a 3rd party, which is a security no no.
2. It introduces a single point of failure into the equation.  If my app gets popular the middle man service could get slammed.  If Optus decides that they don't like what I'm doing, they could easily block it.
3. It means I need to host a service, which means additional expense.

I don't want to do this, so what I'm left with is more hacky solutions, using regular expressions to find what I want in the HTML documents retrieved from the provider.  This will take me longer to code, will be more prone to failure, and is just generally nasty.  I'm not happy.  Devices these days are very powerful, and there should not be the need for intermediary servers to help with processing.

Of course this would all be much easier if the providers published web services interfaces to their data, rather than just web applications.  This has been the mantra of SOA and internet connected businesses since the terms were coined.  It doesn't even cost them that much more to do it, and would lead to better designed web applications, but thats a subject for another rant.  Optus doesn't do this because there's no economic incentive for them to do so.  They gain nothing directly from publishing a usable web interface, so they can't be bothered... bah!

To be fair to Optus, they aren't the only ones that don't get it.  No ISPs and telcos provide any decent interfaces, other than Internode.

I wrote an android application

I purchased an Android phone a few weeks ago. Part of the reason that I got it was that I wanted to see what the differences were between Android and iPhone.  This extends out to how to program them as well, so I had to write an application, just like I did for iPhone.

Last night, I released my little application.  Its a usage meter for my ISP, Internode. I deliberately chose something quite simple so that I could cut my teeth on the platform, and I must say that I'm very impressed. I found it very easy to write the application for Android, especially because it uses the same tools and libraries that I am used to using for my day job.  One other difference that I notice is that it is much more obvious what is going on inside an Android application.  The documentation describes things clearly and gives you full visibility.  Apple on the other hand like to keep their platform a little more mysterious.  There are plenty of good documents on how to do things, but you still get the impression that there's something going on under the hood that you don't quite understand.  Perhaps its just that I'm more familiar with the Java ecosystem.

If you'd like to have a look at it, check out its site at 8-bit cloud.  I've also been playing with the web site, and I will be improving it and hopefully making it more fun.

PS: I was tempted to call the application NodePony after a recent meme that Internode has got going with its cute little plush toys.  In the end, I decided that I shouldn't try to cash in on it.  It would spoil the meme...

What do I do now?

I've been trying to work out what I want to do with my career recently.  When I've been speaking with mentors and colleagues, the first question that comes up is quite reasonably always "What is it you want to do?".  I have to admit that this question has had me stumped for some time.  For the last three years, I've been working as a principal consultant in the system integration space for medium to large size businesses, and I have not found it satisfying.  I do the work well enough (some say even exemplary), but I haven't been able to summon the passion that I can put into work which allows me to excel.  I find myself compelled to seek more responsibility and higher pay, but its just not working for me.

Yesterday, I went to see Stephen Fry speak.  It was a very entertaining journey through Stephen's life, and the things that make him tick.  He called it his personal WWW, the things that inspire and drive him that for the contrivance of his topic all start with the letter W.  One of the Ws that he spoke of was Writing.  He doesn't just like to write.  Its not that he is paid to write.  He is compelled to write.  His love of language and how it can move people is his passion, his reason for getting up in the morning.

For many people, working in IT is just a job.  They have other things in their life that they call their passions, and they just come to the office to get a pay cheque.  I have many interests and hobbies, but my central passion is programming computers.  I view it as a creative pursuit.  Not art per se, but a craft that can produce elegant and useful things.  When I come home at night, more often than not I end up still camped in front of a computer.  So for me, computers are my passion, my raison d'être.  As with all passions, this gives me an edge.

It goes beyond being a code monkey however. Sometimes when I discuss the topic, people say "oh, you'd get bored just doing programming", because I've worked as a solution architect for so long, and I can operate at a business level as well as a technical one.  This is true, but not for the reason that people think.  I'm a bloody good programmer, an nothing gives me more pleasure than solving a technical challenge, but that's not enough to make something that works.  Creating modern software is a complex task, and to do it correctly there is a mix of communication, leadership and technical expertise required. I'd arrogantly like to think that I can perform all of these tasks.  I like the challenge of setting up simple, streamlined processes and teams that get jobs done.  It'd be nice if I got to do a bit of hands on work, because a true leader is a doer as well as a manager, but its more of an oversight, training and review function.

Which technology I work with is largely irrelevant.  There are some that are more interesting to work with than others, and some I have more experience with than others, but all are interesting to me.  Because technology is my passion, I pick new techniques and languages up very easily.  My company recently had a need to develop an iPhone application.  It just so happened I had been playing with iPhone programming in my spare time.  Where other engineers would have said "no, I haven't done that before" or asked to go on training, I relished the opportunity to pick up something new and got stuck in.  I trained a small team and let it to success.

So much of consulting consists of going into organisations that do not operate effectively.  They are rendered moribund by internal politics, people of limited talent in positions of power, and the sheer difficulty of organising a large workforce (usually too large) that is quite often afraid of change.  This is the reason that they bring in outside expert help.  These organisations get the job done, quite often a boring one, and they turn a profit.  But they do not produce exceptional results.

There are people that are very good at going into dysfunctional situations and turning them around.  I have immense admiration for these people because they do a very difficult and often thankless task.  They turn failure into a bare pass.  They can take pride in the fact that they put the hard yards in to get a result, but the end product is rarely anything to rave about.

I have performed project recovery work successfully before, but I am rarely given the opportunity to effect any real change.  I'm brought in as a technical specialist, often very late in the piece when many of the decisions have been made, to solve a particular problem, often within ludicrous constraints that don't make any sense in the context of producing something that works.  I've lost count of how many times I've been asked to work on a "platform project" to put in place tools and procedures for an organisation that does not have any function to put on that platform yet. How pointless.

So here's what I want from my job:  I want to create good software that does useful things. To do this, I want the freedom to be a good chief nerd (or lead engineer, or architect, or whatever else you want to call it).  I know what I'm doing.  Just get out of my way and let me do it.

My challenge now is to engineer the opportunity to do this.  To quote a review of yesterday's performance that appeared in The Age, "Fry proves that we can take power over, and joy in, the role that is ourselves.".  Time to prove all the outrageous claims that I've just made :)

Samsung Galaxy S - 3 weeks in

My ageing iPhone 3G didn't take the update to iOS 4.0 very well. It still worked, but it seemed clunky and slow after the upgrade. Six weeks ago, I (conveniently) took this as a sign that I needed a new phone, and started considering my options. I could wait for the new shiny iPhone 4, or I could look at Android. It has been reported that Android has caught up to Apple now, both in terms of hardware and software. Of course, the biggest thing that everyone touted as the biggest advantage of moving to android was: freedom. Being a developer and nerd, this appealed to me.

So I went out and bought a Samsung Galaxy S I9000 (now there's a mouthful of a name...). Its got a big super amoled 4 inch screen, a very fast processor (reportedly faster than that in any other current phone), and came for free on a very reasonably priced Optus plan. I've had the phone for three weeks now, long enough for me to sort out any learning curves and kinks in the system, so I thought I'd post my impressions.

The biggest thing I noticed buying this phone was that it didn't "Just work" like an iPhone does. It turns out that there was a performance problem which detracted from the new phone feel that I expected. In order to get it working properly, I needed to go onto a bunch of technical forum sites and work out how to flash a new (leaked beta) firmware to fix the problem. Flashing was made even more difficult because they were windows only flashing tools, and they wouldn't work in a virtual machine on my mac. In the end, I needed to drag an old clunker machine out of the cupboard to get flashing to work.

The headphone jack also uses a different configuration to iPhones.  This means that I can't use my existing Sennheiser headset on the phone.  If I do, the sound ends up muddy and the lyrics almost disappear.  The supplied headphones are ok, but I'd still prefer to use my aftermarket headset.  Apparently Samsung have used the same configuration as Nokia, which makes some sense, but it still pisses me off that I need to use an adapter to get my headphones working.

Now that I've passed those problems, I can start to see why this is such a great phone. I now get twitter & yammer notifications on the phone in an unobtrusive way,  and there's no waiting for updates either.  On the iPhone, when there was a new message, a notification would appear, and when you clicked on "view" it took you to the application which then loaded the message (or whatever) from the internet.  This could take a few (up to 10) seconds.  On android, it loads the message first, then shows you the notification.  No waiting!

There's also widgets on the home screen. If I don't like the default home screen that Samsung Provides (TouchWiz 3.0) I can replace it with a number of open source alternatives that are much whizzier and glitzy. There's a cool little active background that makes the screen look like you can see through to a circuit board. There's just so much stuff!!!

Of course, there's also a downside to this. I installed a couple of different music players to try them out. When I pressed the "play music" button on my headset, all of them started at once :S. I worked out how to fix this, but its another example of not "Just working".  I also need to do a bit of research when I want an application, because there isn't just one way of doing it.

For work, there's exchange support built in, so it works just as well as an iPhone in that department.  There's also an office docs reader built in, and all the applications we are used to such as dropbox work as well, if not better than on the iPhone.

Battery life is okay.  Its taken a while to settle down into a pattern, but I'm now getting almost two days between charges, including moderate usage.  The firmware upgrade has been a big help here.

Finally, there's the looks. I have to admit that one reason for moving away from iPhone was that everyone has one now. Sadly, the I9000 doesn't help here, because it looks exactly like an iPhone 3GS.  They even copied the packaging with the phone coming in a little black cardboard box.   Its so similar I'm surprised apple didn't sue them. Oh well, its a pretty vain reason to want to change.

PROS:

1. The screen is gorgeous.
   
2. There's a lot of choice of applications
   
3. lots of storage - 16Gb internal, which I have since augmented with a 32Gb MicroSD card to give me a total of 48Gb.
   
4. SWYPE is a great text input method
   
5. Google integration is great
6. When Android 2.2 comes out, it will get much faster.

CONS:

• Performance problem - Now Solved!
• Doesn't like OS X very much
• There's a lot of choice of applications
• The Games aren't as good.
• Only available on Optus at the moment.  I would have preferred to stay on Telstra Prepaid

All in all, now that I've sorted out my performance problem, I'm very happy with my new phone. I like the choice that I get, and I'm willing to invest the time it takes to tweak the experience to my taste. In the end, I suppose thats the difference between the masses and the nerds: Some are willing to be told the best way to do something, whilst others prefer to tread their own path (or at least the path that is shown them by a bunch of other nerds on a forum)

Now, to work out how to program the thing!

I'm giving a lecture on Cloud Computing

This post is basically just to try out embedding a google wave post in a blog post :) There is some news however.

I am giving a lecture to a Masters of IT class at Monash. This wave is the place where I am gathering information about what I will present. I will also provide access to the students during the class, if they wish to participate or continue the discussion at a later date.

Date and Time: Friday 4th June 2010,
Location: Rooom H125, Monash University, Caulfield Campus

EDIT: Hmmm, it seems that Disqus, the service I use for managing comments on this site, includes the script that is used to view google wave. This means that there will be an unfortunate doubling of the content below.

Google Reader also strips off the google wave component, which makes sense considering the content is added by javascript. We wouldn't want nasty javascript content sneaking out there into reader land, would we?

Good to know.

Distributed social networking

With Facebook's recent stumbles in online privacy, a lot of people are now calling for distribtued information sharing systems, which people can use to interoperate between different servers, different providers, or run your own if you so choose. The Diaspora project has been started by a bunch of grad students to do exactly that, and it looks like they've gone viral.  At the time of writing, they had raised over $160,000, when all they were looking for is $10,000.  Not bad for a first round of funding, especially considering all they've done so far is produce a video.

After talking about this at the pub on friday (the pub is where I do most of my best theorising :)), a friend suggested that I look at the Friend of a Friend Project. He suggested that distributed social networking has been around for ages.  Unfortunately, foaf does not deal properly with security or webs of trust, so I feel that it is useless for what most people would want.

There's no technical impediment to stop distributed social networking from working.  In the end, it would be a much more secure, robust and scalable solution.  As we've seen a number of times, Twitter sometimes struggles with stability and when it goes down, all of Twitter goes down.  These are the original principles upon which the Internet was founded.  The problem is that modern, slick systems take a lot of resources to put together.  Many people and companies don't view the internet as a way of speaking together in an interoperable fashion any more; they want to make money which means not being interoperable.   Google, Yahoo, Twitter and Facebook provide these services so that they can make money of the information we give them.  We make a choice to trade our privacy for the service that we are provided.

I wish the Diaspora guys good luck.  The biggest hurdle they will face is getting the masses to care.  In order to get enough take up of their solution, they will need to either convince your average user to host their own server, or find companies that are willing to host the service but not get access to any of the data. I can't think of any big company that would want to do that.   There's nothing in it for them if they can't mine all of your data.  Most ISPs will host email for you, but they do it as a method of lock in.  once you're on me@localisp.com, it provides a reason to stay with that ISP rather than taking your email with you.  Besides, email was well entrenched before the rise of the ISP.  Every ISP had to provide email. It was just expected.  There's no reason to start hosting a social networking server unless everyone else does, which means you've got a chicken and egg problem.

Its something I've been thinking about a lot this weekend.  How do we make the general populace care enough about their personal data to protect it?  I don't know the answer.

Easy Exchange Email Extraction

Okay, that should be forwarding, but I wanted some alliteration :)

Some time ago, I posted a very technical approach to forwarding email properly using Microsoft Exchange, which is useful if you work at an organisation that uses it. At the time, I was aware that the steps involved were to technical for some people, so I didn't really expect too many people to take it up.

Some colleagues at work expressed some interest in using themselves, but didn't want to go to the effort of setting up their own man-in-the-middle server to fix the emails. To combat this, and to make it easy enough for anyone to set up email forwarding for exchange, I updated my forwarder to work for any email address. I now present this work to the public, in the form of the Crimson Cactus Exchange Redirector. If you would like to perform exchange redirection easily, this might be of some use to you.

In fact, I set this up quite some time ago, but I've been too frightened to publicise it until now because I was concerned that it might become a source of spam or too much traffic on my web server. I've finally gotten up enough courage to try it out now, but please consider it a beta service. It may have to be turned of at short notice if it is abused.

Please give me your feedback if you find this service useful.

Bruce.

Presentation

I'm presenting a session at EJA tomorrow, entitled "Choose Your Cloud". I'll be comparing the different cloud vendors, with a particular focus on Google App Engine.

As an experiment, I've done my presentation in Prezi. I'm not sure if its cool or distracting. I'll check with my audience tomorrow to find out :)

Choose your Cloud on Prezi

How predictable are you?

Ars Technica has posted about research that has been conducted about prediction your location, based upon data gathered from mobile phone towers.  The research shows that it is possible to predict where you will be at any given time with 93% accuracy, even before additional information such as calendars is taken into consideration.  That is spectacularly good, and just goes to show how good the information that is being gathered by our gadgets really is.  What this article makes me think about is how well adaptive user interfaces, once provided with the appropriate data, will be.  I want it... now.

How I would design a forms workflow system for google wave

Yesterday, I posted a quick reply to a post on the google wave developer blog about creating a form based workflow system for google wave.  I was quite busy at the time and i think in my haste I may have been a bit brusque.    By way of making amends, here is how I would set up such a system.

A forms workflow engine needs a number of components in order to operate correctly.  In particular:

1. A way of specifying relationships between people.  The Manager-Employee relationship is probably the most important here, but there are others too (like whom is in the HR department).
2. A way of specifying what information is needed in the form.  What fields, how they are validated, that sort of thing.  Calculated fields would be important here.
3. A way of specifying the workflow for the form, and working out which relationships to use and which form to use in that workflow.
4. A way for a participant in a group to select which form he wants to fill in.  A particular person may be engaged in multiple organisations, so having a single form library to choose from isn't enough

All of this could be managed in Google Wave, but it would not be trivial.  If set up correctly, it could work quite smoothly however.  First, we need a bot that performs multiple functions, or multiple bots that co-operate to provide the functions listed above.  I think it would be more understandable for users to just have one bot, so lets use that model.  For the purposes of this discussion, I'll call it FormBot.

Next, we need a way of creating the models that are specified above.  An operator would create a wave, tag it with its function (so that the bot can tell what purpose the wave serves) and then adds FormBot to the wave.  FormBot adds a gadget to the wave which allows the operator to edit the data.

For the relationship diagram, the gadget/bot combo would be able to work out which users they were dealing with by looking at the participants of the wave.  This effectively becomes your personnel database, in lieu of having it hooked up to an LDAP server.  The gadget would then allow the operator to drop people into relationships.  The robot would detect any changes, and then suck up the information into its own database (this bit is quite important, as it is how the bot, when processing an instance of a form, will be able to find the information in needs).

The Form & Workflow models are linked together in a one-one relationship, so they can be added on another wave, this time tagged with workflowmodel.  There would be a couple of gadgets to allow the models to be edited, which are then sucked into the database again.

So finally, how does the bot process an instance of a workflow form?  A user would have an extension installed on his client which adds an entry to the "new wave" menu which would say "New Form".  This would create a new wave and add FormBot.  FormBot would check its database to see what Form Waves it has which are linked to relationship models with the user as a participant.  It would then present these as a list to the user to allow him to choose which form he wanted to fill in.  The Bot would then create the form and allow the user to edit it and progress through the workflow.  

In summary, what we've done here is create a couple of setting waves that the bot uses to configure the application, and to allow it to provide different form libraries to different groups of people.  I think its quite elegant and would work really well.  There'd be a lot of work to get it implemented though.  The gadgets to allow the editing of the models would be the key bit.  I'd love to do it but I doubt I'll find enough time to do it myself.  I did create a proof of technology some time ago, but it wasn't configurable, nor did it hook up to a proper user database.

Google wants a workflow engine/robot for wave too.

Google just posted on their Wave Developer Blog that their "wishlist" would have on it a way to process document workflow using google wave.  This reminds me of a post I wrote a while back on what google wave could be used for.  I did a proof of concept, but taking the concept to production would take too much effort for one guy in his spare time.  Probably the biggest barrier is integrating with HR systems.  How does the system plug into your corporate HR system to work out who your manager is when you post?

Either way, its good to see that other people see the same value in wave that I do.

Toshiba announces 'digital secretary' functionality

Last week, I ranted about how our cellphones will start adapting to what we are doing based upon the information they can gather on our behaviour.  Right on cue, Toshiba have now announced that they are building exactly this technology, and it will be available by the end of the year.  I'm impressed.  I wonder if they will introduce it only into the Japanese market, or more widely on Android or something similar.

My google wave post just broke :( but google fixed it :)

About 3 months ago, I wrote a google wave gadget, which allows users to collaboratively work on a google wave gadget within google wave.  I've gotten a bit of publicity out of it and its all been great.   Sadly, tonight, the wave that I created to document my gadget has crapped itself, collapsing under the publicity of its own popularity, with more than 200 blips and 500 participants.   The main blip in the wave now has no content whatsoever, and I can't view its history to repair its view.

This is a bad sign.  I like google wave, and I want to support it as a new way of communicating that is easy, effective and efficient.  The problem is that I can't trust it.  If its going to destroy information on me, how can I rely upon it?  Sure, its in beta at the moment, and I probably shouldn't complain, but I'm disappointed.  Its got a truckload of participants now, and I understand that that brings a lot of complexity, but I don't want to loose data.  If I could restore its state it would be fine... The really sad thing is that I'm due to deliver a presentation on my gadget tomorrow night, espousing how good google wave is and how it can be used.   What am I going to do now?

Time to address reliability issues Google.

Update: I've asked google to look into the problem, and it looks like somebody accidentally deleted all of the content in the wave.  This could have been a simple PEBKAC, or a more complex interaction of network latency causing the google wave software to do something unexpected.  It is beta software after all, so these things are to be expected.  It wouldn't be a problem at all, except I can't get access to playback to retrieve an older copy.

Never fear however, the dynamic chaps at google have had a look at my wave, and apparently there are over 30000 revisions to it.  This was a bit much for the javascript engine's tiny little brain and it broke.  They've fixed it now, and all is back as it was.  Thanks Christian!

The time is now for inference engines in user interfaces

I've been thinking the last couple of days about the future of phone user interfaces, and I suppose the future of interfaces in general.  At the moment we have fairly static interfaces, with a scrolling list of applications with the occasional widget to tell us the weather forecast or what not.  We set it up how we like it and thats it.  Some user interfaces (in particular, I'm thinking of Android's pages) provide different screens to cater for different use cases, but it is still a manual affair.

What I'm looking forward to is the day when my phone can infer things about what I'm doing.  Our phones (and our extended computer networks)  know an incredible amount of information about us.  They know where we are, and if we are moving or not, and they can remember where we have been before and at what times.  They have our calendar, they know what time it is now.  That there is enough information to begin to infer things.

For example, I tend to finish work at roughly the same time every day.  When I finish, I walk to Flinders St train station and catch a train home.  During the day, I couldn't give a tinkers cuss what the train timetables look like, but when I walk out the door, all of a sudden I'm keen.  My phone, based upon my previous history of movement, detecting when I start moving from the building I've been in for the last few hours, could easily infer that I am leaving, and that my most likely destination is home.  Wouldn't it be cool if it could alter its home page to show timetable information because thats where it thinks I'm going.

We can get more sophisticated here by adding in additional information too.  If I throw my calendar into the equation, the destination guesser has more information available to it.  If I have an appointment at 5:30pm in a different part of town, it can logically infer that I'm not going home, but rather that I'm going to this location.  Instead of showing me the train timetables for my home line, it could show me a tram route to my meeting, or the location of the nearest taxi rank.  It could even choose which option to show me based upon the time until my meeting.  If I've left myself lots of time then I can take public transport, but if time is running short perhaps I should take a taxi.  My phone becomes a true digital assistant, rather than a window on to information that I have to instruct how to operate.

I want my phone to do this now, and there's no reason it couldn't be done.  I'm sorely tempted to get venture capital funding and go do this.  Its where the action is going to be in the near future, in my opinion.  I've got a whole bunch of ideas about what sorts of information could be fed into an inference engine.

We do need be careful to remember the lesson of Microsoft's clippy however.  In order for an inference engine to work, it needs to be accurate.  It needs to provide value to the user.  It should also be unobtrusive.  if the user simply wants to get to his email, or his web browser, it should be no more difficult to get to than it is on today's interfaces.  Clippy failed on both counts here, and was widely lampooned and hated for it.  The amount of information that the clippy inference engine had to work with was limited, so therefore the assistance it could provide was worthless.

No inference engine will be perfect.  If I walk out the door and start walking towards the train station, I might not actually be going home.  Instead, I might be going to a bar on the way to the train station to meet a friend that just phoned me up.  There's no way my phone is going to be able to guess that (unless it monitors my phone conversation, and understands what we are talking about.  We can't do that yet...).  The secret here is to be unobtrusive.  The phone should offer information on what it thinks I'm doing, but in such a way that if I want to do something completely different it doesn't get in the way.  Phone user interfaces are getting sophisticated now(e.g. SlideScreen) so I see no reason why inferred information can not be incorporated.

One final thought about this is privacy. In order for this to work properly, our computer/phone needs to collect, store (for pattern analysis) and cross-check a lot of really private information.  Where does this information get stored and analysed?  at Google/Yahoo/Facebook?  The inference engine will then construct a model of my behaviour to predict what I'm doing.  I'm not sure I want a big corporation like Google to be able to make these sorts of inferences for me...  but thats naive.  Big corporations like Google already construct models of our behaviour, and every time we sign up to a new service like Google calendar or gmail we give them more information to model us.  I'd prefer to run such a service on my own hardware, but that isn't the way that the industry is going.

But as I ranted recently, perhaps that isn't an issue any more.  Are we willing to give up fundamental privacy in order to get the advantages?  I'm not sure...

I'm part of a presentation on developing Google Wave extensions, Wed 27th of Jan at RMIT

The local Melbourne chapter of GTUG is hosting a meeting around developing extensions for Google Wave this wednesday at RMIT.  The big presenter there will be Pamela Fox, Google's developer relations person, but I've been asked to give a quick run down at the end on my experiences developing a mind map gadget.  If I get time, I might also theorise about how wave could be used to provide ad-hoc communications focused tools inside business.

If you'd like to come along, please register here.  Its open to everyone.

On Privacy

Facebook's CEO, Mark Zuckerberg recently said that privacy is no longer a social norm, Google's Eric Schmidt has also said that if you want to do things online that you want to keep private, then you really shouldn't use online services such as Google, due to laws requiring identification and retention of data.

There are benefits to treating your privacy as a commodity.  The most obvious examples are the personalities that have achieved fame and riches through living their lives in a very public fashion, such as Paris Hilton or Kim Kardashian, but each and every one of us makes a decision to trade our privacy each time we go online, whether we do it knowingly or not.

I recently showed my Father in law how to use Picasa web-albums feature as a way that he could easily catalogue his photos from a trip, upload them to the web and then share them with people.  He was delighted and immediately started pestering his friends to view the web album, which of course involves creating a google account (if you want to restrict viewing to a group of people).  One of his friends refused, citing that he didn't want Google to know about his every move online.

This goes to show the difference between not caring about privacy online (my Father in law) and having enough knowledge to be scared but not enough to fully understand (his friend).  My Father in law knows in an abstract sense that google can track his activities, but he doesn't care.  He has made an implicit (and some would say uninformed) decision to trade some of his privacy for the additional features that Picasa gives him.   His friend doesn't want to share this information, and understands that an account is a tracking mechanism.  What he doesn't understand is that sites like google routinely issue web cookies which are almost as good at tracking people as an account when tied to server logs. Even if you don't sign up to google they will be storing information about you and your browsing habits.  It won't be as easy to pin the usage directly to you as in a person with a name, but it definitely can be, and is, done.  Google do this so that they can target adverts that are tailored to you to your screen, but that doesn't mean that the info can't be used for other means too.

In order to be completely private online, a user needs to go to extreme lengths, using cookie blocking software and IP anonymising routing.  If you do this however, many features that we have come to rely upon no longer work.  You can't browse online email.  You can't one-click share photos with friends, you can't use social networking sites.  Even if you could, there would no longer be any incentive for software giants to produce cool software for us to use, because they could no longer make money off us. The vast majority of us seem to be quite happy to make this trade.

If privacy becomes a commodity, then I would like to have control of that information.  I tend to agree with both Mark and Eric, and I am quite open on the internet, but I want to be able to control what information is used, and what isn't.  If my privacy is a commodity,  that means it has value, and I'd like to see whether I am getting value for privacy.  It is possible to envisage an architecture to the net whereby all personal information is stored locally, on a server that each of us controls, or encrypted on central servers in such a way that only people we allow can have access to that information.  In order for this to work, there would need to be legislation to enforce this separation, as there is cost associated with implementing things in this way.

And there's the problem.  There's no incentive for companies to give us privacy, or to give us control over our privacy, as it will loose them money.  There's no incentive for governments to give us control over privacy as they want to collect information on us too.  "The People" are unlikely to get thier shit together, as they are too easily distracted by the latest shiny product released by Google, or Yahoo or Facebook.   As a result I think we are doomed to a future where information is routinely collected on every aspect of our life.

This conversation has been going on for ages, and its good to see people were thinking about it decades ago.  Credit card companies have been constructing models of us consumers for years and years, based upon our purchasing history.  They then sell this information back to department stores and marketing companies.  A particularly good scenario surrounding how this could end up is played out as a side story in David Murasek's excellent novel Mind over Ship.  Its a sequel to Counting Heads, so you might like to read that first if you are interested.  Now that I think about it, there's also an interesting plot in there about the relative privacy of the rich/powerful vs the middle class.  In both novels the middle class is routinely scanned for information, and their personal AI's (called Mentars) are incapable of keeping the information gatherers out.  The rich have much better Mentars and as a result are able to navigate their way through life with relative anonymity (but of course the heroes undergo a lot more scrutiny because they are the focus of big events).  Perhaps that is our future...

New Version of Google Wave MindMap Gadget available

I've just updated my Google Wave Mind Map gadget to a new version.  New features include:

• It renders graphically now, so things are a little prettier
• It won't work on Firefox versions earlier than 3.0.  If you are running an earlier version I strongly suggest that you upgrade
• There is now the ability to edit properties, accessable via the properties menu button.
• There is a context menu that can be used to access the menu, accessible via the Right Mouse button.
• You can change the colour of nodes. There's also the ability to edit the background colour, but this is currently unused.
• You can change the outline of nodes:
• Underline
• Circle
• Cloud
• You can change the direction of layout of the root node. Where is places nodes is a bit random, and I do plan on fixing this
• You can specify whether nodes are collapsed upon intial view or not.
• Upload/Download now supports non-ASCII characters, such as Cyrillic

Its not perfect, and there are a few things that need to be fixed, but it is an improvement on the old version so I thought I'd release it. If thre are any problems, please do not hesitate to let me know. I can easily revert it back to the old version if its really broken, or issue quick patches.


I'd be delighted to hear your feedback.

Why don't more engineers follow the KISS principle?

I was having a drink with some colleagues last night, and the subject of the over-use of high-availability environments came up.   At too many customer sites we see requirements that the system must have 5 9s uptime (approximately 15 minutes a year downtime) when there is patently no reason for such a requirement.  As a result, we end up spending far more time, more hardware, and more software licenses on the solution than is required.  This not only hurts the project during development, but also during maintenance as a more complex solution requires more maintenance.  This problem isn't limited only to HA, but extends to all areas of design.  In my opinion, this tendency to over-complicate projects is more responsible for project overruns and failures than any other cause.  To make things even worse, organisations who decide to do things "just because" tend to under-invest in them which means that the result is half-arsed and doesn't work properly anyway.

In a quick survey that I conducted of the people around the table at the pub, all of us subscribe to the KISS principle, and I suspect that most engineers would agree with us.  So why do so many projects end up bloated and lumbering?  The knee jerk reaction that everyone other than myself is an idiot just doesn't hold water.  Chances are the next enterprise architect that I meet will be almost as smart as me (even if my ego tells me otherwise), so what is it?  Here's a couple of theories.

Firstly, keeping things simple means that you will be done quickly, but you may need to come back later to make changes or to add functionality.  This is normal and expected within an Agile development methodology, but more problematic in big institutions that take ages to approve budgets and have difficulty dealing with change.  In many organisations it is easier to ask for $1M in one go, rather than ask for 10 $100K budgets. This sucks, but it is the way things go.

Secondly, companies contain many individuals, each of which has their own view on what is important. The reporting guy thinks that all his reports are the most important thing.  The IT guy thinks that good data architecture is the be all and end all.  If one constructs a plan by consulting everybody in the organisation you will end up with gold plating and and a very very long build cycle.

In order for KISS to reign a project needs ruthless leadership.  They need to make sure that their staff understand the principles by which the project is being run, and the intended goal. Knowing what you want to achieve is very important.

P.S. Happy new year!