27 May 2005
it's not just you. The volume of junk e-mail is spiraling out of control. Don't be ashamed if now you actually receive more spam than regular message
it's not just you. The volume of junk e-mail is spiraling out of control. Don't be ashamed if now you actually receive more spam than regular messages. Your e-mail program's built-in junk mail filter and third-party tools such as Cloudmark's SpamNet and McAfee's SpamKiller trap a lot of unsolicited garbage, but they're far from perfect.
These programs (and others like them) sometimes commit the spam filter's mortal sin--moving nonspam to a junk mail folder. The problem isn't that the mail is gone (it's not), it's that you must frequently comb through all of the messages identified as spam to be sure you're not deleting important personal mail. To me, this nullifies the spam software's main benefit--saving time. I'd rather have a little spam cluttering my in-box and know that the messages moved to my spam folder don't contain an urgent note from an editor offering me work.
What's a spam-beleaguered person to do? I say filter it yourself. All of the most popular e-mail programs let you create multiple filters that examine each incoming message, then flag or move those that meet your definition of spam. Because you control the filters, you can test how well they work, make modifications, control the order in which they run, and remove those that don't work the way you want. It sounds complicated, but it's not. Start simply by creating one filter, and then tweak it until it works just right. Once it does, add another one.
A good way to start is with a filter that centers on the most common criterion--the contents of the To: field. Much of the spam you receive is not addressed to you personally, which is an easy characteristic to check for. Here are the steps for creating such a filter in three popular e-mail programs.
Outlook 2002: Click File, New, Folder to create a folder to store spam. Name it Spam, Junk Mail, or the epithet of your choice, and click OK. Next, choose Tools, Rules Wizard, click the New button, and click Next. Uncheck from people or distribution list, check where my name is not in the To box, and click Next. Click the specified hyperlink in the 'rule description' field at the bottom of the wizard window, navigate to the spam folder you created, click OK, and then click Next to move on to the list of exceptions to your rule. Check except where my name is in the Cc box and, optionally, except if from people or distribution list and except if sent to people or distribution list. If you decide to use either of the latter two conditions, click their people or distribution list links in the rule description field to add addresses that you know aren't sources or destinations of spam, such as newsletter or mailing list subscriptions. (The addresses must be in your Outlook contacts list.) Click Next, give the rule a name if you want to, and then click Finish to create the rule.
Outlook Express 6: Create a spam destination folder ( File, New, Folder), name it Spam (for example), and click OK. Next, choose Tools, Message Rules, Mail, and click New. In the Conditions list box, scroll to and check Where the To or CC line contains people and, optionally, Where the From line contains people. Click the resulting contains people link in the Rule Description box, and fill in your e-mail addresses, mailing lists, newsletters, and other known nonspam sources or destinations as appropriate. After entering these addresses, click the Options button, select Message does not contain the people below, and then click OK twice. Finally, check Move it to the specified folder in the Actions list box, click the specified link in the Rule Description box, and click OK twice to save the rule.
Mozilla 1.x/Netscape 7.x: Create a spam folder ( File, New, Folder), give the folder a name, and then choose Tools, Message Filters. Click New, and enter a name for the filter; then select to or CC in the first drop-down list of criteria, and choose doesn't contain in the second list.
Type your e-mail address in the last field on the line. Click More to enter additional filter criteria (such as other e-mail accounts and known source or destination addresses of mailing lists and newsletters). Choose Move to folder, select the spam folder that you just created in the last drop-down lists, and then click OK twice to save the filter.
These programs (and others like them) sometimes commit the spam filter's mortal sin--moving nonspam to a junk mail folder. The problem isn't that the mail is gone (it's not), it's that you must frequently comb through all of the messages identified as spam to be sure you're not deleting important personal mail. To me, this nullifies the spam software's main benefit--saving time. I'd rather have a little spam cluttering my in-box and know that the messages moved to my spam folder don't contain an urgent note from an editor offering me work.
What's a spam-beleaguered person to do? I say filter it yourself. All of the most popular e-mail programs let you create multiple filters that examine each incoming message, then flag or move those that meet your definition of spam. Because you control the filters, you can test how well they work, make modifications, control the order in which they run, and remove those that don't work the way you want. It sounds complicated, but it's not. Start simply by creating one filter, and then tweak it until it works just right. Once it does, add another one.
A good way to start is with a filter that centers on the most common criterion--the contents of the To: field. Much of the spam you receive is not addressed to you personally, which is an easy characteristic to check for. Here are the steps for creating such a filter in three popular e-mail programs.
Outlook 2002: Click File, New, Folder to create a folder to store spam. Name it Spam, Junk Mail, or the epithet of your choice, and click OK. Next, choose Tools, Rules Wizard, click the New button, and click Next. Uncheck from people or distribution list, check where my name is not in the To box, and click Next. Click the specified hyperlink in the 'rule description' field at the bottom of the wizard window, navigate to the spam folder you created, click OK, and then click Next to move on to the list of exceptions to your rule. Check except where my name is in the Cc box and, optionally, except if from people or distribution list and except if sent to people or distribution list. If you decide to use either of the latter two conditions, click their people or distribution list links in the rule description field to add addresses that you know aren't sources or destinations of spam, such as newsletter or mailing list subscriptions. (The addresses must be in your Outlook contacts list.) Click Next, give the rule a name if you want to, and then click Finish to create the rule.
Outlook Express 6: Create a spam destination folder ( File, New, Folder), name it Spam (for example), and click OK. Next, choose Tools, Message Rules, Mail, and click New. In the Conditions list box, scroll to and check Where the To or CC line contains people and, optionally, Where the From line contains people. Click the resulting contains people link in the Rule Description box, and fill in your e-mail addresses, mailing lists, newsletters, and other known nonspam sources or destinations as appropriate. After entering these addresses, click the Options button, select Message does not contain the people below, and then click OK twice. Finally, check Move it to the specified folder in the Actions list box, click the specified link in the Rule Description box, and click OK twice to save the rule.
Mozilla 1.x/Netscape 7.x: Create a spam folder ( File, New, Folder), give the folder a name, and then choose Tools, Message Filters. Click New, and enter a name for the filter; then select to or CC in the first drop-down list of criteria, and choose doesn't contain in the second list.
Type your e-mail address in the last field on the line. Click More to enter additional filter criteria (such as other e-mail accounts and known source or destination addresses of mailing lists and newsletters). Choose Move to folder, select the spam folder that you just created in the last drop-down lists, and then click OK twice to save the filter.
Poll: Majority Want to Make Spam Illegal
A majority of users want the government to step in and make spamming illegal, according to a new Harris Interactive survey.
The Harris Poll shows that 80% of online users today find spam `very annoying'. Because of that frustration level, 74% say they favor making spamming illegal. And the poll shows that percentage holds independent of income level, gender and political affiliation.
``A look at these numbers and the rapidly growing anger at mass spamming, with the large majority in favor of banning it, suggests that -- if our elected politicians listen to their constituents -- spamming may go the way of mass faxing,'' says a Harris Interactive spokesman in a written statement. ``Unsolicited mass faxing was banned. Can mass spamming be far behind?''
The poll was conducted between November and December of last year, sampling 2,221 adult Internet users.
Certain kinds of spam, according to the survey, anger users more than others. More than 90% of users are angered by pornographic spam; while mortgage and loan-related spam annoys 79%; investments 68%, and real estate 61%.
But all is not bad. The survey shows that there have been some improvements.
The number of people annoyed by how long they have to wait for information to come up on their screen has fallen from 25% to 17% since a previous poll was done in March of 2000. And those who say they are `very annoyed' by the amount of time it takes to find the Web sites they need have fallen in number from 20% to 10%.
Holding steady though, are the percentage of users who are annoyed by the unreliable or inaccurate information found online -- that has stayed at 32% over the past nearly two years.
The Harris Poll shows that 80% of online users today find spam `very annoying'. Because of that frustration level, 74% say they favor making spamming illegal. And the poll shows that percentage holds independent of income level, gender and political affiliation.
``A look at these numbers and the rapidly growing anger at mass spamming, with the large majority in favor of banning it, suggests that -- if our elected politicians listen to their constituents -- spamming may go the way of mass faxing,'' says a Harris Interactive spokesman in a written statement. ``Unsolicited mass faxing was banned. Can mass spamming be far behind?''
The poll was conducted between November and December of last year, sampling 2,221 adult Internet users.
Certain kinds of spam, according to the survey, anger users more than others. More than 90% of users are angered by pornographic spam; while mortgage and loan-related spam annoys 79%; investments 68%, and real estate 61%.
But all is not bad. The survey shows that there have been some improvements.
The number of people annoyed by how long they have to wait for information to come up on their screen has fallen from 25% to 17% since a previous poll was done in March of 2000. And those who say they are `very annoyed' by the amount of time it takes to find the Web sites they need have fallen in number from 20% to 10%.
Holding steady though, are the percentage of users who are annoyed by the unreliable or inaccurate information found online -- that has stayed at 32% over the past nearly two years.
Spam's a Nuisance That Can Be Managed, Up to a Point
You can also try forwarding your spam to the authorities: Send a copy to the Federal Trade Commission at spam@uce.gov , or via the link at http://www.ftc.gov/spam . Forwarding another to the "abuse" address of the Internet provider that relayed the message ( abuse@aol.com , for example) is also smart. But to do that right, you'll need to use your mail program's "show full headers" option, which is often hidden.
You'd think that writing a program to delete spam would be easy, since even an Internet beginner can tell spam from real mail. But that hasn't happened-- yet another way in which the computer can't match the human brain.
Transcript
Personal Technology: Apple Unveils TigerPersonal technology columnist Rob Pegoraro was online on May 2 to answer reader questions about his review of Tiger, the latest version of Apple's Max OS X.
Your Internet provider's spam filtering will usually sweep the worst offenders out of sight, but some adopt an excessively strict policy that wrongly tags innocent e-mails as spam. Last winter, for example, Verizon's filtering suddenly began flushing away many legitimate e-mails sent from parts of Europe and Asia.
If you use your own mail program instead of a Web interface such as Hotmail or Yahoo, you can also run your own spam filters. The best learn from your use, watching what mail you label as spam and adjusting their screening to match. Mozilla Thunderbird ( http://www.mozilla.org/ ) and Apple's Mail, both free, include this type of filter, as does the $50 edition of Qualcomm's Eudora ( http://www.eudora.com/ ).
Microsoft's Outlook 2003, by contrast, has a non-learning spam filter, while its free Outlook Express includes no spam block at all. You can add a learning filter to either program with various add-ons; some, such as POPFile ( http://popfile.sourceforge.net/ ) and SpamPal ( http://www.spampal.org/ ), are free but may require tricky configuration; others, such as SpamBully ( http://www.spambully.com/ ) cost money.
A more stringent defense, "challenge-response" filtering, requires would-be correspondents to pass a simple test online that a bulk mailer can't or won't bother to complete -- usually, visiting a Web page and typing in letters shown in an image. Some Internet providers -- notably, EarthLink -- and such add-on software as ChoiceMail ( http://www.digiportal.com/ ) and SpamArrest ( http://www.spamarrest.com/ ) offer it.
But although these systems wave through mail from people in your address book, other legitimate senders must perform extra work. Perhaps as a result, challenge-response has not been widely adopted.
All of these techniques can only treat spam. A cure will have to be economic: When no money can be made from spam, nobody will send it. Filters, lawsuits and fines can raise the costs of sending junk e-mail, but there's still money to be made by defrauding the gullible.
So until everybody wises up, we're probably stuck with spam. That's irritating, but at least junk e-mail won't crash your computer or erase your data. If spam were the only risk of going online, the Internet would be a better place, if still thoroughly annoying at times
You'd think that writing a program to delete spam would be easy, since even an Internet beginner can tell spam from real mail. But that hasn't happened-- yet another way in which the computer can't match the human brain.
Transcript
Personal Technology: Apple Unveils TigerPersonal technology columnist Rob Pegoraro was online on May 2 to answer reader questions about his review of Tiger, the latest version of Apple's Max OS X.
Your Internet provider's spam filtering will usually sweep the worst offenders out of sight, but some adopt an excessively strict policy that wrongly tags innocent e-mails as spam. Last winter, for example, Verizon's filtering suddenly began flushing away many legitimate e-mails sent from parts of Europe and Asia.
If you use your own mail program instead of a Web interface such as Hotmail or Yahoo, you can also run your own spam filters. The best learn from your use, watching what mail you label as spam and adjusting their screening to match. Mozilla Thunderbird ( http://www.mozilla.org/ ) and Apple's Mail, both free, include this type of filter, as does the $50 edition of Qualcomm's Eudora ( http://www.eudora.com/ ).
Microsoft's Outlook 2003, by contrast, has a non-learning spam filter, while its free Outlook Express includes no spam block at all. You can add a learning filter to either program with various add-ons; some, such as POPFile ( http://popfile.sourceforge.net/ ) and SpamPal ( http://www.spampal.org/ ), are free but may require tricky configuration; others, such as SpamBully ( http://www.spambully.com/ ) cost money.
A more stringent defense, "challenge-response" filtering, requires would-be correspondents to pass a simple test online that a bulk mailer can't or won't bother to complete -- usually, visiting a Web page and typing in letters shown in an image. Some Internet providers -- notably, EarthLink -- and such add-on software as ChoiceMail ( http://www.digiportal.com/ ) and SpamArrest ( http://www.spamarrest.com/ ) offer it.
But although these systems wave through mail from people in your address book, other legitimate senders must perform extra work. Perhaps as a result, challenge-response has not been widely adopted.
All of these techniques can only treat spam. A cure will have to be economic: When no money can be made from spam, nobody will send it. Filters, lawsuits and fines can raise the costs of sending junk e-mail, but there's still money to be made by defrauding the gullible.
So until everybody wises up, we're probably stuck with spam. That's irritating, but at least junk e-mail won't crash your computer or erase your data. If spam were the only risk of going online, the Internet would be a better place, if still thoroughly annoying at times
Microsoft launches new spam offensive
Hotmail tools identify accounts used to send junk email
Microsoft has unveiled new tools for its Hotmail service, aiming to weed out the bulk of spam messages.
The software giant has released a preview version of Smart Network Data Services, which offers statistics about email being sent to Hotmail accounts.
The service shows the volume of email coming from single IP addresses, and whether Hotmail's filters have tagged it as spam.
Smart Network Data Services can help ISPs to identify accounts used to distribute spam emails, either deliberately or because they are infected by a virus and turned into a so-called zombie.
Microsoft also launched MSN Postmaster, a website where ISPs, email service providers and legitimate senders of bulk email can learn about spam issues.
Postmaster is intended to educate senders on how to prevent legitimate emails from getting caught in spam filters.
"MSN Postmaster and Smart Network Data Services represent a move towards broader, more comprehensive and transparent information sharing with ISPs and email senders to ensure that email continues to be an essential and valuable communications tool," said Kevin Doerr, product unit manager for MSN Hotmail at Microsoft.
"With over 200 million active email accounts worldwide, MSN Hotmail is in a unique position to collect and analyse email activity data.
"Working together, MSN Hotmail and service providers can make their customers happier and more satisfied with the services we all provide."
Postmaster and Smart Network Data Services are part of Microsoft's efforts to weed out spam. In June last year the company promised to filter out 98 to 99 per cent of all spam by June 2006.
The software vendor is involved in a series of projects, including Microsoft Smartscreen and Sender ID.
Sender ID tackles the problem of domain name spoofing, a technique used by spammers to hide the identity of the message sender. The technology checks whether the 'from' email address corresponds with the IP address from which the message originates.
Smartscreen is used to filter out phishing emails by rerouting them to the user's spam folder.
Microsoft has unveiled new tools for its Hotmail service, aiming to weed out the bulk of spam messages.
The software giant has released a preview version of Smart Network Data Services, which offers statistics about email being sent to Hotmail accounts.
The service shows the volume of email coming from single IP addresses, and whether Hotmail's filters have tagged it as spam.
Smart Network Data Services can help ISPs to identify accounts used to distribute spam emails, either deliberately or because they are infected by a virus and turned into a so-called zombie.
Microsoft also launched MSN Postmaster, a website where ISPs, email service providers and legitimate senders of bulk email can learn about spam issues.
Postmaster is intended to educate senders on how to prevent legitimate emails from getting caught in spam filters.
"MSN Postmaster and Smart Network Data Services represent a move towards broader, more comprehensive and transparent information sharing with ISPs and email senders to ensure that email continues to be an essential and valuable communications tool," said Kevin Doerr, product unit manager for MSN Hotmail at Microsoft.
"With over 200 million active email accounts worldwide, MSN Hotmail is in a unique position to collect and analyse email activity data.
"Working together, MSN Hotmail and service providers can make their customers happier and more satisfied with the services we all provide."
Postmaster and Smart Network Data Services are part of Microsoft's efforts to weed out spam. In June last year the company promised to filter out 98 to 99 per cent of all spam by June 2006.
The software vendor is involved in a series of projects, including Microsoft Smartscreen and Sender ID.
Sender ID tackles the problem of domain name spoofing, a technique used by spammers to hide the identity of the message sender. The technology checks whether the 'from' email address corresponds with the IP address from which the message originates.
Smartscreen is used to filter out phishing emails by rerouting them to the user's spam folder.
How Bayesian Spam Filtering Works
How Bayesian Spam Filtering Works
If a word, "Cartesian" for example, never appears in spam but often in your legitimate mail, the probability of "Cartesian" indicating spam is near zero. "Toner", on the other hand, appears exclusively, and often, in spam. "Toner" has a very high probability of being found in spam, not much below 1 (100%).
When a new message arrives, it is analyzed by the Bayesian spam filter, and the probability of the complete message being spam is calculated using the individual characteristics.
Let's say a message contains both "Cartesian" and "toner". From these words alone it's not yet clear whether we have spam or legit mail. But other characteristics will (most probably) indicate a probability that allows the filter to classify the message as either spam or good mail.
Bayesian Spam Filters Can Adapt Automatically
Now that we have a classification, the message can be used to train the filter further.
In this case, either the probability of "Cartesian" indicating good mail is lowered (if the message containing both "Cartesian" and "toner" is found to be spam), or the probability of "toner" indicating spam must be reconsidered.
Using this auto-adaptive technique, Bayesian filters can learn from both their own and the user's decisions (if she manually corrects a misjudgment by the filters). The adaptability of Bayesian filtering also makes sure they are most effective for the individual email user. While most people's spam may have similar characteristics, the legitimate mail is characteristically different for everybody.
How Can Spammers Get Past Bayesian Filters?
The characteristics of legitimate mail are just as important for the Bayesian spam filtering process as the spam is. If the filters are trained specifically for every user, spammers will have an even harder time working around everybody's (or even most people's) spam filters, and the filters can adapt to almost everything spammers try.
Spammers will only make it past well-trained Bayesian filters if they make their spam messages look perfectly like the ordinary email everybody may get. They could do that today, too.
Spammers do not usually send such ordinary emails, I presume, because they don't work. So chances are they won't be doing it when ordinary, boring emails are the only way to make it past the anti-spam filters.
If spammers do switch to mostly ordinary-looking emails, however, we will see a lot of spam in our Inboxes again, and email will may become as frustrating as it was in pre-Bayesian days (or even worse). It will also have ruined the market for most kinds of spam, though, and thus won't last for long.
One exception can be perceived for spammers to work their way through Bayesian filters even with their usual content. It's in the nature of Bayesian statistics that one word that very frequently appears in good mail can be so significant as to turn any message from looking like spam to being rated as ham by the filter.
If spammers find a way to determine your sure-fire good-mail words -- by using HTML return receipts to see which messages you opened, for example --, they can include one of them in a junk mail and reach you even through a well-trained Bayesian filter.
If a word, "Cartesian" for example, never appears in spam but often in your legitimate mail, the probability of "Cartesian" indicating spam is near zero. "Toner", on the other hand, appears exclusively, and often, in spam. "Toner" has a very high probability of being found in spam, not much below 1 (100%).
When a new message arrives, it is analyzed by the Bayesian spam filter, and the probability of the complete message being spam is calculated using the individual characteristics.
Let's say a message contains both "Cartesian" and "toner". From these words alone it's not yet clear whether we have spam or legit mail. But other characteristics will (most probably) indicate a probability that allows the filter to classify the message as either spam or good mail.
Bayesian Spam Filters Can Adapt Automatically
Now that we have a classification, the message can be used to train the filter further.
In this case, either the probability of "Cartesian" indicating good mail is lowered (if the message containing both "Cartesian" and "toner" is found to be spam), or the probability of "toner" indicating spam must be reconsidered.
Using this auto-adaptive technique, Bayesian filters can learn from both their own and the user's decisions (if she manually corrects a misjudgment by the filters). The adaptability of Bayesian filtering also makes sure they are most effective for the individual email user. While most people's spam may have similar characteristics, the legitimate mail is characteristically different for everybody.
How Can Spammers Get Past Bayesian Filters?
The characteristics of legitimate mail are just as important for the Bayesian spam filtering process as the spam is. If the filters are trained specifically for every user, spammers will have an even harder time working around everybody's (or even most people's) spam filters, and the filters can adapt to almost everything spammers try.
Spammers will only make it past well-trained Bayesian filters if they make their spam messages look perfectly like the ordinary email everybody may get. They could do that today, too.
Spammers do not usually send such ordinary emails, I presume, because they don't work. So chances are they won't be doing it when ordinary, boring emails are the only way to make it past the anti-spam filters.
If spammers do switch to mostly ordinary-looking emails, however, we will see a lot of spam in our Inboxes again, and email will may become as frustrating as it was in pre-Bayesian days (or even worse). It will also have ruined the market for most kinds of spam, though, and thus won't last for long.
One exception can be perceived for spammers to work their way through Bayesian filters even with their usual content. It's in the nature of Bayesian statistics that one word that very frequently appears in good mail can be so significant as to turn any message from looking like spam to being rated as ham by the filter.
If spammers find a way to determine your sure-fire good-mail words -- by using HTML return receipts to see which messages you opened, for example --, they can include one of them in a junk mail and reach you even through a well-trained Bayesian filter.
