Easy Exchange Email Extraction

Okay, that should be forwarding, but I wanted some alliteration :)

Some time ago, I posted a very technical approach to forwarding email properly using Microsoft Exchange, which is useful if you work at an organisation that uses it. At the time, I was aware that the steps involved were to technical for some people, so I didn't really expect too many people to take it up.

Some colleagues at work expressed some interest in using themselves, but didn't want to go to the effort of setting up their own man-in-the-middle server to fix the emails. To combat this, and to make it easy enough for anyone to set up email forwarding for exchange, I updated my forwarder to work for any email address. I now present this work to the public, in the form of the Crimson Cactus Exchange Redirector. If you would like to perform exchange redirection easily, this might be of some use to you.

In fact, I set this up quite some time ago, but I've been too frightened to publicise it until now because I was concerned that it might become a source of spam or too much traffic on my web server. I've finally gotten up enough courage to try it out now, but please consider it a beta service. It may have to be turned of at short notice if it is abused.

Please give me your feedback if you find this service useful.

Bruce.

Google Apps allows for custom SMTP servers

I've been using my google account to store mail for a while now, including forwarded mail from client organisations, where this is allowed. The only problem that I've had so far is that when I'm sending mail as another identity (like a client one), it would always come up as "from me@brucecooper.net on behalf of me@the.right.domain" Its a bit annoying, but thats what Google had to do in order to be good email citizens and not get everything marked as spam. As it turned out, many Exchange servers automatically mark such email as spam anyway, so some people lost mail that I sent them.

Google recently introduced a new feature, whereby you can specify your own SMTP server to use when sending mail, and you can specify a different one for each identity. This is great, as you no longer get the on behalf of bullshit, and your mail doesn't get spammed any more. I was originally stumped by the fact that I couldn't get to the SMTP servers of the organisations that I wanted to send mail from, but then it dawned on me that I could use any SMTP server that would allow me to relay mail as other identities. I used my ADSL provider and all is good. Of course, if the organisations that I'm sending mail as had SPF rules set up to disallow this sort of stuff I'd still be in trouble, but so far that hasn't been a problem.

How to automatically forward email from Exchange without loosing headers

UPDATE: I've now created a service to make this much easier to forward email. If you are interested in this, please have a look at the service's site.

I've got a million email accounts. Every time I start work on a new client site, I get given yet another email account. Its a pain in the butt to manage all of these, so wherever possible I forward the mail onto my main gmail account where it can get filtered, stored and searched easily.

This works great for sites with unix based email, but more often than not my clients use Microsoft Exchange for their mail. You can set up a redirecting rule to forward the email (as long as the server has been set up to allow this), but when it does so it strips off the To: and CC: headers from the message when it sends it on. For example, if Bob Log had sent a message to Me@mycompany.com, SomebodyElse@mycompany.com, when it was forwarded on to my gmail account it would still appear to come from Bob Log but the To: would just be me@gmail.com, not the original recipients. This won't work!

I've toyed with a number of solutions to this problem, including writing a bot that uses EWS to poll the server, re-form the message and send it on, but that requires a bot to run on the client's lan and may not be able to forward on the message as getting access to Exchange to send the message can sometimes be difficult.

I now have a solution that works without requiring any additional software to be running on the client's network. The solution involves forwarding the message to an intermediate account as an attachment (which preserves the headers), filtering the message back out of the attachment at the intermediate account, then sending that message on to gmail in all its original glory. To do this, you require an intermediate email account that you can use purely for the purposes of filtering the mail, which is capable of piping incoming mail to a perl filter script. Generally, these aren't that hard to come by. I happen to have a getting started plan with Cove here in Melbourne which fits the bill nicely. Its free to run (although they do have a $2 setup fee) and its servers are in Australia which is a benefit for me. If you live elsewhere, there are probably other options that would do just as well.

To set it up, there are three steps,

step 1: create the filter script
Below is included a perl script which takes an email address as a parameter, and reads a MIME encoded email on STDIN. It will look for a part with a content-type of message/rfc822 which is an embedded message, and then it will stream that message out to sendmail with the supplied email address paramater as the final destination. This file should be uploaded onto your server somewhere where the mail filter can get hold of it.

I originally wrote a script that used the MIME::Parser perl module, but I've found that most hosting providers don't have that module installed, so it was easier to just do it from scratch. I'm not a perl programmer really, nor have I spent a lot of time on this script, so it definitely could be improved, but it works!

#!/usr/bin/perl

my $recipient = $ARGV[0];
my $boundary = '';
my $endMarker;
my $partType;
my $sendmail = "/usr/sbin/sendmail -oi $recipient";

# Reads a line from STDIN, and makes sure it isn't the EOF
sub fetchLine {
my $txt = <STDIN> or die "Reached end of file prematurely";
chomp $txt;
return $txt;
}

# reads a message part, looking for an rfc822 message.  If it finds one, it
# forwards it on to the recipient.  When it finds the part end, it returns 1
# if there are more parts, or 0 if it is the end of the message
sub parsePart {
my $isMessage = 0;
my $returnCode = -1;

# First, read the headers, looking for a content type
while ((my $text = &fetchLine()) ne '') {
if ($text eq 'Content-Type: message/rfc822') {
$isMessage = 1;
open(SENDMAIL, "|$sendmail") or die "Cannot open $sendmail: $!";
}
}
# Then read the body, streaming it out if it is a message
# End the loop when we find a boundary or the end marker
while ($returnCode == -1) {
$text = &fetchLine();
if ($text eq $boundary) {
$returnCode = 1; # Meaning we still have parts to parse
} elsif ($text eq $endMarker) {
$returnCode = 0; # Meaning we are finished parsing the message
} elsif ($isMessage) {
print SENDMAIL "$text\n";
}
}
if ($isMessage) {
close(SENDMAIL);
}
return $returnCode;
}

# First, Read Headers, looking for the multi-part content type and
# boundary separator
while ((my $text = &fetchLine()) ne '') {
if($text =~ m/^Content-Type: (.*)/i) {
$nextline = &fetchLine();
$nextline =~ m/\s+boundary="(.*)"/i or die "Could not get boundary after content type";
$boundary = "--$1";
$endMarker = "--$1--";
}
# We don't care about any other headers
}

# Check to see that we have the right mimetype and a boundary
die "No boundary found" if $boundary eq '';

# Skip until the first part separator
while ((my $text = &fetchLine()) ne $boundary) {
}

# Parse the message, looking for a part with a type of message/rfc822
while (&parsePart()) {
}

exit 0;

step 2: set up the filter in cpanel (or whatever else you use)
On your server, you now need to set up mail filtering so that any incoming mail from your work account that isn't a bounced message gets sent to your filter script. In cpanel, I did this by setting up an email filter for all mail, which looked something like this:



step 3: enable forwarding of mail in Exchange
Now that you've got your email forwarding filter set up, all that remains is to set up exchange to forward any incoming mail to your filter account. You do this by selecting Rules and Alerts



And then setting up a rule that looks like below:

Be careful with what you put in your rule definition, as some rules are "client only" which means they will only run when outlook is open. As an example, I tried to make it also mark the message as read when it forwards, but that is "client only" which means the rules won't run unless outlook is open :(

Once you've got that set up, you can test. Send a message to your work email and see if it makes it through. If anything goes wrong, it should bounce with a message telling you what went wrong. One thing I did notice though is that if I send the message from gmail itself, it tends to ignore the message when it gets forwarded through as it already has a copy (in sent), so I send test messages from an alternate account just to be safe.

Okay, so in summary, its possible to forward messages on from exchange, but it requires a man in the middle to extract the message contents, and its a bit fidlly. If you like this tip, let me know.