Although spam is defined as any piece of unsolicited email, it is more
commonly comprised of scams that flood millions of mailboxes and is rapidly
becoming the scourge of the Internet. Every known scheme in the
con man's book and snake oil salesman is now in electronic mail format
hoping to catch a few gullible victims out of thousands and in the process
causing mass collateral damage of our email infrastructure. Spam
traffic has now easily surpassed legitimate email and has rendered many
mailboxes virtually unusable. Even mobile devices that have to pay
by the byte for each message they receive are beginning to get spam. This not only threatens the growth and viability of mobile electronic
messaging, but email in general as a valuable communications tool.
Fundamental weaknesses in email:
The SMTP (Simple Mail Transfer Protocol) email protocol is fundamentally
flawed because it was never designed to be secure in the first place and
lacks any authentication of the source of an email. Simply put, SMTP is
based on the honor system, with no way to confirm the authenticity of
the sender let alone track the sender. What this means is that anyone
can send email as any assumed identity from anywhere in the world.
I can say I'm the CEO of your company or I can say I'm the Pope when I
send you an email and there is no way to confirm or deny it's legitimacy.
Although this may seem like a shock that we are standing on such a shoddy
foundation, this really isn't that difficult to understand because at
the time of it's invention, it was only used among the few academics and
researchers of the world and security was never an issue. A lesser
but still important issue is that SMTP servers use to be open relay, where
anyone can use and abuse or spam from that server as they please.
As a result, most email administrators now lock down their SMTP servers
to not behave in this promiscuous way and configure them to only relay
email from their own network or from users that have authenticated with
the server. These lock down efforts however have not reduced the
capacity of spammers to spam because there is nothing to prevent you from
putting up your own SMTP server. Just about any modern Windows,
Unix, or Linux operating system have their own built in SMTP server capabilities.
Even viruses as small as 10 kilobytes can contain their own SMTP server
to spread them selves. Because of this, spammers can send hundreds
of thousands of unsolicited mail in a few hour from abundant unsecured
wired or wireless networks with virtual impunity or traceability.
Laws prohibiting this are all but useless because Spammers are already
engaging in felonious activities in fraud and think nothing of breaking
another law that can't even be enforced. The fundamental problem
with email is that SMTP has no mechanism for authenticating the source
of the message, and therefore has no way to enforce proper behavior against
those who would abuse it even within our own borders let alone offshore.
Partial solutions and schemes:
As with any type of conflict, there are measures and counter measures. The problem with the current spam conflict is that we are fighting an
uphill battle because of the fundamental weaknesses in SMTP. Any solution
devised to fight under the current email standard is at most effective
when it is used by so few people that spammers don't care to target them. The minute any solution establishes a large enough user base, they become
targets and their defenses are easily overcome. This section will list
some of the most common anti-spam techniques and how they can all be circumvented.
Filtering (keyword matching or heuristic algorithms):
Keyword matching is one of the most common solutions out there and is
only somewhat effective. Heuristic statistical analysis algorithms are more sophisticated, but they still cannot be perfect. Just about
any type of email filtering system will have a certain percentage of false
positives and false negatives. Case in point, my own experience with
these have been false positive rates of only 0.01%, so you would loose 10
legitimate emails for every 100,000 pieces of spam that are collected during that
time. Although 10 out of 100,000 messages seem like a really good
score, you wouldn't be so happy if one of those 10 messages is yours or if
one was extremely important. Even if you accept those kind of
statistics, spammers still hold the ultimate ace in their sleeves with anti-parsing techniques such as bad spelling or padding a
word like sex as S.E.X with symbols between the characters. Any
filtering system you can buy can also be bought by the spammer to "test"
payload, and they will simply modify the message until it passes the filter. No automated algorithm stands a chance against these tactics and even if
they did, spammers can resort to sending spam in flash or handwritten
bitmaps which eliminate any chance of software analyzing the message. Filtering
will never work unless you have a trained professional human to read through
ever message, which is not possible without a bottomless pit of money.
If you look closely to the right, that is not html.
That is a bitmap which I received through in my MSN inbox which normally
filters a lot of the spam. The reason it came through is because
it can't be read by text filters. In order to filter something
like this, it would have to have an OCR (Optical Character Recognition)
engine built in. Anyone with any experience with OCR is well aware
of how inaccurate they can be if someone just accidentally uses a fancy
font. Now imagine if someone didn't want you to OCR their message.
This is why filters will never work no matter how
"intelligent" the filtering software purports to be.
This isn't your father's spam!
This is bitmap spam designed to defeat any anti-spam filtering system.
Blacklisting or Whitelisting: Blacklisting is where you block known spammers or the even IP addresses
of known spammers. Blocking known email addresses will only work
if you can verify authenticity of the sender. Since authenticity
of the sender is impossible with the current SMTP protocol, blacklisting
is useless and often results in collateral damage of legitimate messages.
This is exactly what the spammers intended to happen if you attempt to
block them in this manner. Blocking IP addresses has also started
becoming a popular practice, but it has a nasty habit of sometimes punishing
the innocent while the spammer flees to another IP address. You
have to keep in mind that with the scarcity of IPv4 addresses, many of
us poor souls have to share our IP addresses with other domains.
A good system should punish the offending SMTP server, not the IP
addresses it came through. Whitelisting is exactly the same as
Blacklisting only it contains a list of known good instead of known bad,
and like blacklisting it suffers the same fundamental shortcomings in lack
Copyright enforcement schemes: One of the most ridiculous schemes I've ever seen is where a company
offers you your own copyrighted poem to stamp at the end of each message.
The company promises to sue any person or business that uses your poem
to stamp their own messages. As a result, that poem could be used
as a form of a "digital signature" (I use this term loosely)
which could be used to put you in a whitelist. This scheme would
be perfect if spammers operated within the law in this country let alone
in some 3rd world country. The fact that someone can spam from anywhere
on the planet with virtual impunity and stealth means this scheme is absolutely
useless. The fact that truly strong digital signature technology
such as PGP or the standard x.509 digital signature have long existed
makes one wonder how such a company can get so much support behind it.
I have written extensively about this topic and you can read it
here. To quickly summarize, this is absolutely the dumbest thing
you can do right now. Such a list would be spammer's paradise.
Spammers spent significant resources to find lists of legitimate email
addresses, and now you are just going to hand them the biggest one of
all for free. Any do-not-spam list is effectively a please-spam-me
Go after the beneficiary legal tactics:
Recently on Charley Rose's PBS program, they had a panel of experts from
Industry, Press, and Government. One of the suggestions brought up
was something to the effect that; even though you cannot track down the
person that actually sent the spam, you can always track down the source of
the spam if you simply went after the product or service being advertised.
Well I'm no lawyer, but it would seem to me that people or organizations
are innocent until proven guilty. Given that it is so easy to send
spam with impunity and that there is no way to track down the personal who
sent the spam, how do you prove that the spammer was hired by said person
or business that allegedly benefited from the spam? A company could simply
pay a so called "marketing agency" to "advertise wink-wink" for them when it fact
they are really just a front for spammers. How do you prove the
agency is resorting to spam if you can never catch them in the act? The minute you start
bringing these cases to court and if I were the defendant of such a
case, I would simply plea that I never paid or hired anyone to spam for
me. Not only can you not prove that I paid or hired the spammer, but
I could prove to you that it was extremely easy for someone with a grudge
against me to frame me. Taking this proposal to it's logical
conclusion, I could conceivably sue any large
company for allegedly sending spam to me and others when in fact I was the
one who secretly spammed myself and others on behalf of that
company to facilitate a frivolous law suit. I don't see how such a
law could ever be enforceable or fair, because you could never prove a company
paid a spammer to spam for them nor can you prove their marketing agencies are
resorting to spam. Remember, you're dealing with
professional scam artists who already operate above the law. Under
the current unauthenticated email system, laws are useless.
The reason all of these methods are all doomed to failure because the
rules of the game give an infinite advantage to the spammer. You can
put titanium alloy reinforcement on a building only to watch it collapse
if the foundation is weak, and this is exactly what we face with the
current spam epidemic.
Defining the solution:
The solution must be universally compatible, and could adopt multiple
solutions as part of the standard.
The solution must first fix the SMTP protocol itself, primarily the
The next version of SMTP must be downwards compatible.
Any new SMTP protocol must be simple and cheap to implement.
Once such a protocol is devised, an intelligent and seamless transition
phase must be devised.
The initial pace of the transition phase must be in the control of
each end user.
A standardized public abuse statistic database must be devised.
Users must be able to set their own parameters for rejecting mail
before or after an email is ever transmitted.
Mail that is allowed through can be stamped with a standard header
to facilitate priority sorting.
Fixing the foundation:
The only way to level the playing field against spam is to upgrade the
SMTP protocol beyond the honor system and make spoofing nearly impossible.
For the remainder of this document, I will call the new
protocol as SMTP v2 and the existing SMTP protocol as SMTP v1.
Unlike some who are suggesting a new SMTP protocol all together which
could never be implemented unless behind the barrel of a gun, SMTP v2
should be downwards compatible to the existing protocol to facilitate a
seamless migration. To make SMTP v2 possible, a new set of security
extensions for the existing SMTP protocol would need to be developed.
Such a set of extensions to follow existing naming conventions could
simply be called SMTPSEC, as in IPSEC or DNSSEC, all of which would
use strong authentication and strong encryption. In searching the
news groups and google, I didn't find a single reference to the word
SMTPSEC used in this manner, so I am proposing the use of the word
SMTPSEC. SMTP plus SMTPSEC will be the new SMTP v2 protocol.
To make SMTP v2 spoof proof, we can consider the following candidates:
SMTP server digital certificates
End-user digital certificates
DNS Reverse lookup
Create new DNS field type to define valid SMTP relay servers
Require DNSSEC (authenticated DNS)
Digital certificates: Only one technology can truly fulfill those requirements, and that
is public key cryptography. ITU (International Telecommunication
Union) standard X.509 digital certificates are universally deployed for
any application that requires strong authentication. There are two
ways they can be used to strengthen the SMTP protocol. You either
require x.509 digital certificates for each and every end user, or you
require digital certificates on just the SMTP servers for SMTP to SMTP
server authentication. Since end users out number SMTP servers by
at least 100 to 1, the only feasible solution is to do is the latter.
You will never be able to convince each and every user to go through the
hassle of getting their own public digital certificates, and doing so also
makes it impossible for them to use web-based email from public consoles because
users don't carry their digital certificate everywhere they go.
This is exactly why you don't see client side certificate deployment for
E-Commerce sites and why the vast majority of SSL implementations only use server side
certificate deployment. Even ignoring the costs, management of all
those end user certificates is a massive undertaking because of the physical
verification process for all those users. Although just doing strong
SMTP server authentication only validates the identity of an SMTP server's
domain and not each and every one of it's end users, it does make it extremely
likely that the end user (sender) is in fact who they say they are if
their SMTP server enforces strict mail relay from only authenticated users
and protects it's user's credentials with SSL (Secure Socket Layer) based
email clients. Fortunately, the same digital certificate used to
prove the SMTP server's identity to other SMTP servers can also be used
to facilitate secure SSL encrypted communications between server-to-server and server-to-end user with SSL enabled POP, IMAP, and Web-based email clients. Not
only is this technology nothing new, but the concept of only using server
side certificates is widely deployed from Online shopping to Wireless
LAN security because of it's ease of implementation and relatively strong
This proposed standard doesn't exclude end users from getting their own
digital certificate for true end-to-end authentication and cryptography,
it simply doesn't require them for cost and deployment considerations.
Although end-to-end authentication is a laudable and perhaps ideal goal,
mandating it's universal implementation will only serve to hamper any kind
of solution for the foreseeable future. However, there is no reason
end user certificates can't be integrated into the new SMTPSEC standard as
an optional enhanced security level in addition to the mandated SMTP server certificates.
There is room enough in the proposed standard for both certificate based
approaches. The digital certificates would undeniably prove the 1024-bit public key and domain of the SMTP server, and for a
optionally higher level of authentication also include individual name,
business or organization name, business license, business address, and
other binding information. The certificate is then verified and
then digitally signed
by a publicly trusted CA (Certificate Authority) such as Verisign, Thawte,
or some other public CA. The only possible way you can fake an X.509
certificate is if you can steal the private key of the possessor of the
Certificate, and you can forget about stealing a Public CA's private
keys because they guard it with their lives. Key stealing is extremely impractical because even if you manage
to steal a private key, you can only use it a few times before that digital
certificate is revoked. As for brute forcing a 1024-bit digital
certificate, the best estimate for a super computer cluster to make a
world record attempt is not for another 30 years, and the certificate
is only good for 1 to 5 years anyways. Another important feature
of this solution is that you can use your SMTP server's private key to sign other 3rd
party SMTP server's digital certificates so that they can relay on behalf
of your SMTP server for a specified amount of time. This is an
important feature because SMTP relay is an important part of the email
system. Because of all these benefits, Digital certificates are the ideal candidate
for SMTPSEC and SMTP v2 as it is with just about any other "SEC"
Once you look at the billions of dollars used to combat spam with little
or no success, you can move beyond the $300/domain mail server
certificate. Not only do you end spoofing, but that same mail server
certificate can also be used to facilitate SSL enabled POP3, IMAP, and
Webmail. Now isn't that worth $300 for an entire domain?
Technical animation of the SMTPSEC process. Hit "Play" to
SMTP server IP validation:
IP validation is another solution proposed by some in the industry, but
is nowhere near as secure or flexible as the Certificate based approach
shown above. One of the current methods that can be employed is
DNS reverse lookup. This method basically compares the IP address
of the sending SMTP server against the publicly listed owner of the IP
address. Unfortunately, most people and businesses don't own their
own IP address blocks and instead "rent" their IP blocks from
their Internet Service Provider. Although most IP renters can control
their DNS forward lookup zones, few have the luxury of controlling their
public reverse lookups zones. This leaves DNS reverse lookup out of reach
for all but the ISPs and large corporations that own their own IP blocks,
so this automatically makes DNS reverse lookup a poor solution because
it lacks universal compatibility.
Anther proposed method is to upgrade the DNS specification
in the form of a new field type that defines the valid SMTP relay servers
of a domain, perhaps called an "RX" DNS field for "Relay
Exchange" where as the existing "MX" (Mail Exchange) field
is used to define the inbound mail server for a domain. Some will criticize this approach as spoofable because IP address headers
can be forged and while there is some merit to that concern, the reality is that you cannot easily spoof an IP address
where a connection oriented TCP session such as SMTP is involved.
IP spoofing is not as simple as just having raw sockets in your operating
system, you must also be able to hijack the Internet routing protocol
or the routers that control them. Sure you can just send off some
IP packets with forged IP source headers, but the instant the other side
responds to you, the response packet will go to the rightful owner of the IP
addresses and not the spoofed source making it extremely difficult to
have a back and forth dialog required in a connection oriented session. Nevertheless, there are
some rare instances where a blind timed attack can spoof an entire TCP
session so therefore, IP validation is no where near as secure as using
SMTPSEC. Even if the new proposed DNSSEC were employed, that would
only serve to guarantee that the IP lookup from that DNSSEC server for the
"RX" record is accurate. However, that doesn't help you if someone
can spoof that "accurate" IP address, and you're right back to requiring
SMTPSEC per domain to truly secure SMTP v2. This is exactly why
Digital Certificates on Web servers can be successfully used to secure
E-Commerce sites without the use of DNSSEC whereas the converse is not
true. That same logic also applies to SMTP. Of course, I'm not
down playing the need for DNSSEC since that is important for other
security reasons, just not for these.
Aside from it's weaker security, DNS validation also runs in to
problems because of it's requirement for every domain to have exclusive use of
it's own IP address. Often times a single IP address is shared by
multiple organizations and their SMTP servers. Although DNS validation
at least means you don't need to own your own IP block outright, it does
mean you have to at least not be sharing your IP address with anyone else
and this can be a problem for some. The biggest problem with DNS
validation is that it requires an upgrade to the DNS infrastructure in
addition to the SMTP infrastructure. Why require
an upgrade to DNS as well as SMTP when we're trying to solve the email
problem, especially since SMTPSEC is much more secure and
only require an upgrade to the SMTP protocol and not DNS? Therefore,
I believe IP validation should not be employed in SMTPSEC for SMTP v2, and
that Digital Certificates are the only way to go.
Client side control:
Once the foundation of email is secured by SMTPSEC under SMTP v2, we can begin the transition
phase away from SMTP v1. One of the biggest problems with current
anti-SPAM filtering technology aside from the fact that they don't work well is
that the damage is already partially done even if the filter works.
What I mean by that is that the bandwidth is already wasted before the
filter even gets a chance to do it's job. Under the SMTP v2
protocol, users could be given the chance by their ISP to opt out of SMTP
v1 outright by a simple user preferences database which the user is in
control of. If the users chooses to accept SMTP v1 (most will during
the initial transition phase), messages could be stamped with an SMTP
v1 or v2 header before it is given to the end user's email client.
That email client could have two inboxes, one for SMTP v1 unauthenticated
email and one for SMTP v2 authenticated email. The actual sender
is implicitly trusted as likely accurate if the SMTP server verifies it's
clients before sending email on behalf of them. Existing techniques
for filtering can still be applied to the legacy inbox, although with
less and less efficacy. This would basically set up a two tier
class system for email, verified and unverified. As SMTP v1 becomes
inundated by spam, users will be less and less inclined to drudge through
their massively polluted SMTP v1 inbox if it isn't already happening now.
If you want your email to be read first (or even read at all), you will
insist that your ISP supports SMTP v2 or you find a new ISP that does
so that your messages will end up in the SMTP
v2 inbox. No one will want to use second class email where
delivery is questionable. Once critical mass for SMTP v2 is achieved, the industry
can begin to negotiate a termination date for SMTP v1. Of course,
I am not claiming that this automatically means SMTP v2 it self will end
spam, it only makes it's senders accountable and traceable. Some
of those techniques that had very little meaning under SMTP v1 like new
laws, blacklists, and whitelists will all of a sudden have meaning under
SMTP v2. SMTP v2 only fixes the foundation, but now we can begin
to rebuild the house on this new solid foundation with meaningful laws,
filters, whitelists, and blacklists.
Statistical tracking and scoring:
There are currently many services that blacklist entire blocks of IP addresses
right now, but they are having a devastating effect on legitimate email
while not having much of an effect on spam. Unfortunately, the only
way current black lists to ever stop all spam is to block out 0.0.0.0/0,
which is technical jargon for the entire address range of the entire Internet
and is obviously ludicrous.
It is just plain silly to block out IP addresses you should only be blocking
out illegitimate outbound mail servers. What is really needed is more accuracy and granularity
with a numeric scoring system instead of just black or white reject or
accept criteria. The score
can be an effective criteria for sorting user's inboxes and the user
can even specify a cutoff for the minimum score. Once SMTP v2 is implemented,
legacy SMTP servers will be scored lowest, since their identity cannot
be confirmed. SMTP servers with certificates guaranteeing
domain name identity are the minimum standard for SMTP v2 and more specific
information such as individual name, business name, address, and/or license
# would boost the SMTP server to a much higher default score. Anyone
can buy a new domain name but it's a lot harder to acquire
new business licenses and physical addresses, and even extreme to get a new
The more information a digital certificate guarantees, the higher it's base rank and the higher it
is sorted in the user's inbox by default. If mail coming from a
certain domain is often abusive, points could be deducted from their base
score. The score will ultimately determine if the
message will be read first
or read at all by the email recipient. An accurate public database
of domains and companies that abuse email can be created as a deterrent to
sending unsolicited email and as a way to block those spammers out across
the board for anyone who uses that database. SMTP v2 will greatly
reduce collateral damage while also making spammers accountable because
for the first time they can be easily identified.
Kick starting the solution:
To start the process of fixing the email system, the Internet community
would need to ratify the new SMTPSEC extensions and the SMTP
v2 protocol, perhaps not exactly as how I proposed it but hopefully
meeting it's primary design goal of eliminating spoofing. Once such
a protocol is ratified, we could see a rapid acceptance of the new SMTP v2
protocol as soon as the major ISPs such as AOL, MSN/hotmail, Earthlink,
Netzero, and other large ISPs adopted it themselves. Compared to the
billions already spent to combat spam, my proposed
SMTPSEC extensions would be very cheap to implement because it only requires SMTP server side software upgrades along with the purchase of a
single SMTP domain certificate that can be used by all of the SMTP relay
servers of a single domain. All of the client technologies such as
SSL browsers and SSL enabled POP3/IMAP applications are all existing
technology. Although there are already efforts under way by the
major ISPs, any solution must be universal in order to fix email as a
whole. As David Berlind of
Jamspam.org said on the Charley Rose panel discussion on SPAM, we must
have a unified effort from the entire industry. Email belongs to
everyone, and it must be solved by everyone. Only when the majority
of people are using SMTP v2 authenticated email can we ever hope to see
the end of spam.
I am a freelance writer and the owner of this website
have been working on the frontlines in IT for over the last decade, and my purpose for writing this paper is to fix the email system
because I truly feel that I have something to contribute to the community
based on my experience and knowledge of network infrastructure.
I hope to
influence the future of email in a way to save it from impending doom. I hope as many people will read this
as possible and pass it on to as many people as possible. I intent
to submit this paper to any business or organization with a vested
interest in email infrastructure and hope that they will take to time to
consider my proposals. If any website or media organization wishes
to reprint or republish the contents of this paper in it's entirety, you
contact me and I can grant you permission to freely do so.
Questions are also welcome. Linking is always welcome, but do let me
know since I like the feedback.