I want to say - thank you for this!

uMVyJO

Yeah, I know how spamassassin works. A whole bunch of URLs
with an "organically" defined weight to them. Then as
spam changes you need new rules and new weighings. It is
redundant to my hand tuned rules, and slower since mine
are headers and first five lines of the body only.

Since yesterday, ifile has operated on 322 messages of
mine and only gotten two wrongly sorted (both were classified
as nonspam, one had a base64 encoded body, which is ignored
by ifile).

I think this is a your-mileage-may-vary thing.

SpamAssassin, at least, is quite transparent. You can see all the rules, drop any you don't
think are appropriate, add any new ones, and change any scores you think are inappropriate.

This is the case even when your ISP provides SpamAssassin. A file in your user directory
contains your overrides.

I don't use razor, spamassasion or spamcop. I just don't
trust leaving the rules to something outside my control.

My current technique is a header scoring system I wrote
myself and it works pretty well. But it requires tweaking
from time to time, so I've been looking around at other
methods.

Statistical analysis of content is starting to look quite
good. Here are some links:

A Plan for Spam (outlines the rational and evidence of
success for this):
http://www.paulgraham.com/paulgraham/spam.html

bogofilter (Eric Raymond's implementation of this)
http://www.tuxedo.org/~esr/bogofilter/

ifile (similar vein, but generalized to multiple categories)
http://www.ai.mit.edu/~jrennie/ifile/

I started tagging my email with ifile results today after
seeding it with my spam-box and non-spam mailboxes, and
it has properly tagged the 4 pieces of spam and 15 pieces
of non-spam I've gotten this afternoon.

Chris, yes and no.

In my experience, the #$%^& AOL user who didn't know how to unsubscribe reported me seven times, one for each email she received from my list. She started on Sunday evening, and I didn't get the reports until I got into the office Monday mid-day. By then, the damage had been done.

It's too easy to report non-spam using Spamcop, and their "Kill 'em all and let God sort 'em out" attitude is going to get them into legal trouble someday.

Edited by author 08-20-2002 09:21 PM
People need to read up on SpamCop - Felten's ISP gave out some misinformation. Unlike SPEWS, Spamcop isn't going to blacklist an entire netblock because of a single report.

From the spamcop blacklist FAQ:
- Reports regarding website and email-address spamvertisement ARE NOT counted at all.
- If a server has fewer than 3 spam reports against it and neither of these are newer than 6 hours, then it will be delisted, despite any otherwise-high ratio.

Felten's hypothesized train of events is completely wrong. What really happened? Check the blacklist for www.freedom-to-tinker.com:

http://spamcop.net/w3m?action=checkblock&ip=209.51.158.242

Huh. A real spammer sending mail from that server; they were listed on 7/31 and delisted on 8/7. Felton's "we're back" message was on 8/10. Seems pretty obvious - Felton's ISP screwed up and yanked his site because of a separate issue (they probably confused SpamCop with SPEWS, which really does use over-the-line vigilante tactics). Now they're trying to blame SpamCop for their mistake and, since Felten didn't double-check their story, the problem got far more publicity than it merited.

Dori's link referred to Zeldman's screed about spamcop. He needs desperately to buy a clue - yes, your abuse contact will get a report for each user report, just as if they sent it directly. That's the point - SpamCop aggregates reports and, among other things, provides a way to batch resolve the multiple reports which come in from spam recipients. All an ISP has to do is mark the problem as "resolved" after investigating - I've done this with mistakenly reported emails for IP space I control. Problem solved. SpamCop does monitor this so they can catch the ISPs which are lying about resolving problems - that procedure requires a good deal more than a single automatic report.

Edited by author 08-20-2002 06:33 PM
Spamcop is out of control with their "Guilty until proven innocent" attitude. One of the main problems is that even if you run a 100% clean mailing list, there's always going to be one bozo who doesn't know how to subscribe and thinks that complaining to Spamcop works, so why not use that method?

That's how I ran into trouble with them. Yes, I was able to prove that the person was properly subscribed, but (1) I shouldn't have to "show my papers" because some idiot didn't know what she was doing, and (2) even though I was found innocent, email of mine was blocked for a week. Spamcop is very simply an extremely easy way to run a denial of service attack on anyone, from an enemy to a competitor. And therefore, it's a lawsuit waiting to happen.

My experiences can be found here, here, and here.

Bottom line to me: Mr. Felten is foaming at the mouth. His problem is with his ISP, not with SpamCop. If his ISP had checked with him, the worst that would have happened is SpamCop customers would have emails from that list send to their "hold" box for a week.

Actually the worst is considerably worse than that. I think we're discussing the SpamCop DNS list - which, since it runs on IP addresses, means that all of the ISPs customers may have their outbound emails dropped when
they reach a SpamCop-DNS using system. This may be a different way of operating than the SpamCop that you are using.

I agree that the ISP is reaching too far - but keep in mind that the downside they are facing is that their outbound SMTP gateway is currently listed as a spam source, potentially causing undetectable dropping of mail every moment. The ISP wants to head off that problem FAST.

I think part of the problem goes to SpamCop's attitude toward email - that is, it's ok if this problem fixes itself in a week. This roughly implies that it is OK to silently drop inbound email for a week. A week? Excuse me?

The ISP may have been assuming that SpamCop did some due diligence on the spammer. SpamCop needs to make its policy
about single complaint blackholing much more clear.

(Disclosure: I'm a SpamCop customer.)

It seems Ed Felten is a taking out his frustration in the wrong place. He's making SpamCop out to be a Stalinist boogeyman, when the main problem is his ISP took SpamCop's email (which is generated automatically when a SpamCop user requests it) and performed no due diligence.

Yes, Mr. Felten is right, any nitwit can incorrectly complain about spam. They can use a tool like SpamCop to do it, or they can do it manually.

But if your ISP goes off half-cocked, don't blame SpamCop. SpamCop didn't shut the site down, his ISP did.

I don't think an ISP puts any more weight behind email sent from SpamCop's automated agent than they do a manual email.

As far as the SpamCop blacklist, it does age by itself, because the engineers at SpamCop realize people get confused and report legimate email lists fairly often. There's more information here.

SpamCop lists plenty of information for ISPs on what to do if they get reported. Though I'm sure they aren't exactly fast.

Bottom line to me: Mr. Felten is foaming at the mouth. His problem is with his ISP, not with SpamCop. If his ISP had checked with him, the worst that would have happened is SpamCop customers would have emails from that list send to their "hold" box for a week.

To me, this isn't a big problem.

kellan - the problem with SpamCop's resolution system is that it's targetted at ensuring people like you - actually running a legit list - can get a problem resolved.

If you check the requirements for resolving a "wasn't a spam" complaint, they expect you to show that you have a legitimate opt-in trail. But in Felten's case, he wasn't running a list, nor did he claim to run a list, nor did he even accidentally run a list. As a result, he doesn't have - nor should he be expected to have - the needed documentation.

SpamCops approach reminds me of "papers please?" "Hmm" "Your papers are not in order...". Or,
more to the point, guilty until proven innocent.

Interestingly, SpamCops own guidelines on the the DNS blacklist (what I assume we are discussing here) state
"This blocking list is somewhat experimental and should not be used in a production environment where legitimate email must be delivered." They further indicate that a "tag only"
mode is preferable. That suggests that a remote ISP may
be partly at fault for using SpamCop in a way that wasn't intended.

The other part of the problem here isn't really SpamCop -
it's Felten's ISP who implemented take down without actually
receiving a reasonable notice.

I use SpamAssassin, and find that there will always be some
legitimate email that gets tarred this way. Erik is on target when he points out that economics are the root of the problem - and we may end up paying by packet just to make the problem go away. or - better yet - how about paying by port 25 open attempts, with the first 500 free?

I can certainly understand (Note: Anyone who thinks "understand" = "condone" needs to go learn what those words mean.) why SpamCop doesn't belive anyone who says "That wasn't spam." *Every* spammer says that, and SpamCop is well aware of the first two rules of dealing with spammers.

1) Spammers lie.

2) See Rule #1.

However, they've cleary got a problem when one accusation shuts down access to multiple networks. Then again, see rule #1.

To be honest, I'm about that close to giving up, myself. My filters have gone from strong to massacre-level evil, and I frankly don't care if some real email gets caught -- frex, my "text/html: die" and "multipart/alternative: die" and "entirely base64: die." rules. I used to check the logs, I whitelist a few, and ignore a few more. Lately, I don't bother with that, as I've watched the log file that gets mailed to me every night continue to double in size about every four months.

I've fought the spam-wars for years. I'm beaten, bloody and bowed. Technical solutions don't work -- and don't solve the bandwidth clogging problem (one doesn't solve traffic problems by stopping the cars when they get off the highway.) Economic or Legal are the only answer, and given the current climate, I'm not sure they'll ever happen.

I think I've got about three, four months before I bag it entirely and either go to a preapproved, white-list only email system, or just give up on email entirely.

Edited by author 08-20-2002 10:47 AM
I don't understand what people are talking about with this "no appeal" policy. We run 400+ mailing lists at lists.indymedia.org, and therefore have our share of clueless people to deal with. The 2 times we've had problems with SpamCop have been when someone subscribed a third party list to recieve the emailng from one we were hosting.

It took an evening of emailing them, and probably helped that we loaded the email with all sorts of sysadmin jargon, but they solved the problem super quick, and since them have given us the benefit of the doubt.

The only bright spot in this picture is that our real justice system allows lawsuits to be filed against guys like SpamCop for libel and/or defamation.

Thats pretty sad (and very American), I can't believe you've got someone trying to stop spam, and you want to sue them out of existence.

I think I prefer the limited amount of spam that I receive to this sort of digilanteism.

The ACLU-Houston.org site got one of these recently--I'm not sure what came of it, but I didn't like it.

I'm fairly sure that a week or so of no one reporting you will see you fade from Spamcop's blacklist. I guess you go a little grey then finally turn white ;) -- either way, with Spamcop's list (specifically their own list, not their service that can include other lists) if you don't get reported for a (short) while then the block goes away all by iteself.