-
Notifications
You must be signed in to change notification settings - Fork 0
/
challenge-response.html
executable file
·410 lines (299 loc) · 15.6 KB
/
challenge-response.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
<html>
<head>
<meta http-equiv=Content-Type content="text/html; charset=windows-1252">
<meta name=Generator content="Microsoft Word 10 (filtered)">
<title>Really Simple Challenge/Response System</title>
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
/* List Definitions */
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
<link rel="stylesheet" href="http://beust.com/beust.css" type="text/css" />
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal align=center style='text-align:center'><span
style='font-size:16.0pt'>Why Challenge/Response Systems<br>
are the future of email<br>
and the biggest threat that<br>
spam has ever faced</span></p>
<p class=MsoNormal> </p>
<p align="right">
<font size="-1">
<em>Cedric Beust, July 9th 2004</em>
</font>
</p>
<p class=MsoNormal>I have resisted installing a Challenge/Response System (CRS)
for the longest time. The reason is that I didn’t want to put too much burden
on people who send me emails, and so far, Bayesian filtering had been doing a
pretty good job at protecting me from spam.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Things have changed. Bayesian filtering is still doing a
very good job, but the nature of spam is changing in subtle ways that make
filtering less and less adequate each day. For example, I have noticed the spam
I receive has more and more the following characteristics:</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>-<span
style='font:7.0pt "Times New Roman"'>
</span>They are very short (a couple of lines).</p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>-<span
style='font:7.0pt "Times New Roman"'>
</span>They are not always in English (usually not a problem for Bayesian
filtering as long as the message uses Unicode, since the filtering is based
purely in the frequencies and proximity of words).</p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>-<span
style='font:7.0pt "Times New Roman"'>
</span>They contain undisplayable characters (for example, Chinese or Russian,
and you don’t have that language pack installed on your machine).</p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>-<span
style='font:7.0pt "Times New Roman"'>
</span>Bayesian filters typically don’t add negative scores to emails carrying
a virus payload. You need to complement them with another type of filtering
tool, such as SpamAssassin or a virus removal software.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>After a while, I realized that I was spending too much time checking
out my junk email folder in the search of false positives (sign of a very bad
Bayesian filter) or receiving actual spam in my inbox (sign that either the
filtering level is too low, or that spam is getting harder to assess).</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>After this realization, I started reconsidering my view on
Challenge/Response Systems, analyzing its drawbacks and advantages, and I think
I have reached a decent compromise that should provide me with a close-to-optimal
protection against spam.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>This article describes my thoughts so far.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal><b><span style='font-size:14.0pt'>What makes spammers different</span></b></p>
<table class="float-right">
<tr>
<td>
<script type="text/javascript"><!--
google_ad_client = "pub-1467757024002850";
google_ad_width = 120;
google_ad_height = 600;
google_ad_format = "120x600_as";
google_ad_channel ="";
//--></script>
<script type="text/javascript"
src="http://pagead2.googlesyndication.com/pagead/show_ads.js">
</script>
</td>
</tr>
</table>
<p class=MsoNormal> </p>
<p class=MsoNormal>There is a fundamental difference between a spammer and
you. </p>
<p class=MsoNormal> </p>
<p class=MsoNormal align=center style='text-align:center'><b><i>100% of the
messages a spammer writes are sent to unknown individuals, <br>
whereas 99% of the emails you write are sent to people you know.</i></b></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Think about this carefully because this simple observation
is what provides us a deadly and final weapon against spam. </p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Unless you belong to a rare profession, I bet that most of
the emails you send every day go to people who are either in your address book
or in your inbox folder. Conversely, most of the emails that you receive come
from a well-identified person.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>How can we capitalize on this observation? By creating a
CRS on your email account that respects these constraints.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal><b><span style='font-size:14.0pt'>But Challenge/Response
Systems are a pain!</span></b></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Yes and no. There are two different aspects we need to
consider: </p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>1)<span
style='font:7.0pt "Times New Roman"'> </span>What
makes them a pain?</p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>2)<span
style='font:7.0pt "Times New Roman"'> </span>Who
are they a pain to?</p>
<p class=MsoNormal> </p>
<p class=MsoNormal><b><span style='font-size:14.0pt'>1) Who suffers from
Challenge/Response Systems?</span></b></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Answer: people who email you for the first time.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Based on the observation above, we know that these people
are very few in numbers.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>If you follow the guideline for an effective CRS listed
further down in this article, the CRS you install on your email account will be
absolutely transparent for 99% of the people you correspond with regularly. I
have installed such a system recently on my email account and I can guarantee
you that nobody around me (coworkers, friends, family, temporary email pals,
etc…) has even noticed.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>So let’s go back to this 1% stranger who is trying to email you.
He will receive a challenge in response to his email, and whether he will
decide to send the email anyway or drop the idea of emailing you altogether depends
on several factors:</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>-<span
style='font:7.0pt "Times New Roman"'>
</span>Is the email he is trying to send you very important?</p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>-<span
style='font:7.0pt "Times New Roman"'>
</span>Is the Challenge too complicated, unclear or too time-consuming?</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Obviously, we can’t do much about the first point since it’s
entirely dependent on what this correspondent is trying to tell you, but we can
address the second point by trying to create a CRS that people won’t mind
responding to.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal><b><span style='font-size:14.0pt'>2) Why are current
Challenge/Response Systems painful?</span></b></p>
<p class=MsoNormal><b> </b></p>
<p class=MsoNormal>Because they make a fundamental mistake: they assume that
spammers actually read the responses to the emails they send.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Most of the spammers use bogus or one-time-only email accounts.
If you think about it, it makes sense: they are going to be deluged by
mailer-daemon messages and angry people, so they are much better off ignoring
these responses altogether.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Here is another often overlooked fact: the senders of spam
are less and less the real originators of the spam.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>There are hundreds of “spam powerhouses” that make a
business of just sending bulk email. Whenever someone decides to resort to
spam to sell their merchandise, they are typically going to hire the services
of these bulk senders so they don’t have to worry about the technicalities (and
the legal implications) of sending spam. This is one more reason why responses
to the spam email account are never read.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Even if we assume that spammers do indeed read responses to
their spam (maybe to add the email address of the unfortunate responder in a
“validated email address lists”, another urban legend in my opinion), it purely
and simply doesn’t make any economic sense to process it. The spammer is much
better off letting his Web site handle orders or irate customers and focus on his
next batch of ten million emails rather than adding your email address to a
“golden list” of email addresses (which will most likely be protected by a spam
filter anyway, so they’re not even sure that sending you a different spam will
reach you).</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>The point I am getting at is this:</p>
<p class=MsoNormal> </p>
<p class=MsoNormal align=center style='text-align:center'><b><i>A Response
doesn’t have to ask the recipient to do something clever.</i></b></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>No need to add a keyword to the subject, to go to a web site
to confirm your identity or, even worse, to identify a distorted gif image so
you can prove you are not a robot.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>With that in mind, what would be the simplest action you
could ask from a legitimate sender? Simply responding to the Response email.
That’s right. A simple reply.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>If you respond to the email, you are validated. Period.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal><b><span style='font-size:14.0pt'>Creating the ultimate
Challenge/Response System</span></b></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>So here are my suggestions to create a very effective Challenge/Response
System:</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>1)<span
style='font:7.0pt "Times New Roman"'> </span>The
CRS must implement a white list. Anybody in this white list can send you email
directly without receiving a Challenge. I also recommend for the white list to
recognize patterns, so that you can add entire domains (such as your company,
e.g. <a href="mailto:*@bea.com">*@bea.com</a>). Optionally, you might want to
implement a black list containing email addresses that are automatically bounced
(by the way, the best way to do this is to simulate a Mailer-Daemon response).<br>
<br>
</p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>2)<span
style='font:7.0pt "Times New Roman"'> </span>Before
turning the system on, populate the white list with your entire address book
and the content of your Inbox and various other folders of interest (and more
importantly, all the mailing-lists you are subscribed to). This is very
important so that your current email activities go undisturbed. For all the
people you communicate with on a regular basis, your installing of the CRS will
go absolutely unnoticed. <br>
<br>
</p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>3)<span
style='font:7.0pt "Times New Roman"'> </span>When
an email that is not whitelisted arrives in your inbox, you bounce it back to
the sender, adding a note saying that this account is protected against spam.
And here comes the important part: <b>all they need to do is reply to that
email, and their original email will then be delivered</b>.<br>
<br>
</p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>4)<span
style='font:7.0pt "Times New Roman"'> </span>The
CRS must be able to deal with bounced emails gracefully. The simplest way to
do this is to add a specific header to any Challenge email. Then, any time you
receive an email that is either whitelisted or that contains the specific
header, you forward it to your inbox. This header should also be used to avoid
infinite loop between the CRS and the non-whitelisted sender.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal><b><span style='font-size:14.0pt'>Implementing the CRS</span></b></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>I implemented such a CRS with simple <i>procmail</i> rules
and I am running it as we speak, but it’s too early to disclose its
implementation since it most likely has quite a few bugs. There are less than
fifty lines of <i>procmail</i> rules and the implementation includes a couple
of external tools written in very simple shell to handle the white list.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Keep in mind that you can’t judge the effectiveness of a CRS
by the amount of spam you are receiving, since by definition, it will be zero.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>The only way you can know your system works is when you
receive email from an unknown sender. An even better way to assess the
effectiveness of your CRS is to check the logs regularly and:</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>-<span
style='font:7.0pt "Times New Roman"'>
</span>Identify when you received an email from a legitimate sender.</p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in'>-<span
style='font:7.0pt "Times New Roman"'>
</span>But this sender didn’t respond to the challenge.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>In such a case, I recommend contacting the sender directly
and ask them why they didn’t respond to the challenge in order to determine how
you could improve your system.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Of course, this kind of log-combing should be left to
implementers of the CRS (me) and not to final customers, which is exactly what
I intend to do in the coming weeks.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>I will post a follow-up when I have more data on this Really
Simple Challenge/Response System (RSCRS).</p>
<p class=MsoNormal> </p>
</div>
<hr width="100%">
<a href="http://beust.com/weblog/archives/000150.html">Leave a comment about this article</a>
<p>
<a href="http://beust.com/weblog">Back to my home page</a>
</body>
</html>