-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.SERVER
254 lines (197 loc) · 11.2 KB
/
README.SERVER
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
README FILE FOR DIABLO TRANSIT FEEDER
---
DIABLO is a news transit and reader system. This readme file describes
the transit side of things. You can operate the transit and reader
sides separately or together, or just operate one or the other depending
on your needs.
The transit side of Diablo is designed to transit news between one or
more incoming feeds and one or more outgoing feeds. The transit side
of Diablo is not designed to operate as a reader. However, it may be
used to back a reader's spool for readers that fetch articles by message-id
(which the reader side of Diablo does).
Since the transit portion of Diablo is not designed to support readers,
an active file is not usually used with it ('active off' in diablo.config).
If you want the transit portion of Diablo to act as the master article
number assignment point which downstream sites are slaved off of, you turn
ON feeder-side active file support in diablo.config and
run the reader side, dreaderd, with the -x option to slave it's article
numbering to the master's. When turned on, the active file only effects
how the Xref: header is generated by the transit side of Diablo. The
transit side of Diablo will still transit articles even if none of the
newsgroups are listed in the active file.
The feeder side of diablo does not process control messages, so if you
are mastering article numbers on the feeder, you either have to run the
reader on the same box in order to process control messages and keep the
active file up to date, or you must periodically synchronize the active
file from a remote source using dsyncgroups (taking care to use the proper
options so you do not overwrite the begin/end/other article numbering
parameters that Diablo uses to master XRef: lines).
The transit side of Diablo maintains a history file (the reader side does
not). This means that the transit side is able to take multiple feeds
in parallel without transiting duplicate articles. The reader side's
ability to handle duplicate articles is limited only to the case where
the -x option is used on dreaderd. If the option is not used dreaderd
CANNOT handle duplicate articles being fed to it. It is often beneficial
to run the feeder and reader on the same machine, with the feeder feeding
the reader internally, even if the feeder is not going to be used as
a (major) spool cache. It is more common to run the feeder and the
reader on the same machine if the feeder is given a large enough spool
to act as the backend cache for the reader. Or you may just want to
run a feeder-only machine with no reader elements on it at all.
The transit side of Diablo (the 'feeder') is strictly for news transit
and does not understand reader-related NNTP commands.
The Diablo spool, usually /news/spool/news, is maintained by the transit
system. You must size the spool and run dexpire as appropriate to your
needs. Diablo stores multiple articles per file in the spool in a
two-tiered directory structure. Article files & directories should
never be directly edited or removed or you risk corrupting the reference
data Diablo stores in the history file. You must use dexpire to free
space in the spool. Since Diablo does not write out article files singly
it tends to be much more efficient then INN without CNFS. It is
roughly on-par with INN + CNFS, though I personally believe it is better.
Typically, anyone taking a full feed these days must dedicate a
machine to it that is separate from the newsreader machine that your
users use to read news.
The DIABLO transit system is designed to replace the dedicated newsfeeds
machine and is designed to be a mostly hands-off affair once you get past
configuring and stabiliing it. See TUNING_NOTES for machine configuration
suggestions.
OS REQUIREMENTS
See TUNING_NOTES.
WHO SHOULD RUN DIABLO
If you need to run a USENET news feeding/transit system and/or a
USENET newsreading system, Diablo may be for you.
WHERE TO GET DIABLO
http://www.openusenet.org/diablo/
REPORTING BUGS
send the bug to: [email protected]
send non-bug stuff to: [email protected]
NEWSGROUPS: news.software.nntp
MAILING LISTS: http://www.plig.net/mailman/listinfo/diablo-users
USE OF REALTIME FEEDS AND FEED DELAYS
If you have several outgoing feeds, you should consider using the
realtime, queueskip and startdelay options in dnewsfeeds. All of your
local and internal feeds should be realtime. Cheap external paths
to the internet can also be realtime. To reduce the cost of running
outgoing feeds over your internet transit, you may wish to weight
the feeds according to cost. For example, our MAE-WEST connection
is a lot cheaper then our MCI T3, so I run outgoing feeds with
MAE-WEST destinations in realtime and run outgoing feeds which go
via MCI in batch mode with a 10 second delay. This way the articles
may actually propogate to the more expensive destinations via other
means prior to my actually attempting to send them direct.
Likewise, if you have T1 and frame customrs, it is usually cheaper
to supply them with a newsfeed yourself rather then force them to
go to someone over the internet. This way they are not eating your
transit bandwidth on newsfeeds. A realtime feed to those people is
best.
CATCHING UP AFTER BEING DOWN
The key item to monitor when catching up on incoming feeds after
being down for a while is the incoming article rate. Diablo will
generate a log line for every 1024 articles received that looks like
this:
Jun 24 11:03:59 news1 diablo[18153]: DIABLO uptime=7:46 arts=241.000K tested=0 bytes=1.842G fed=12.613M
You can calculate the article rate by looking at the delta activity
from two log lines that are around an hour apart from each other.
If the article rate is above 9 articles/sec, diablo is catching up
reasonably well.. as of today, a full feed is around 5 articles/sec.
With a moderate number of incoming feeds, diablo can do around 30
articles/sec. If you have a huge number of incoming feeds that are
all in catchup, in-kernel filesystem locking will begin to interfere
with the history file lookups and updates. Diablo will be able to
maintain a reasonable history file write transaction rate, but the
lookup rate will suffer.
This causes diablo to catch up on articles first without appreciably
reducing the backlog at remote sites due to slow check-responses.
Once it passes a certain threshold, however, and the load on the
history file turns to mostly-read rather then read/write, the
transaction rate will increase dramatically and diablo will generally
be able to cleanup the backlogs very quickly after that.
SPAMALIAS OPTION IN DNEWSFEEDS
Submitted by: uhclem @ nemesis.lonestar.org (Frank Durda IV)
Spamalias
---------
Related to an "alias" command, any originating server (not
transit servers) or username entry in the Path: header that
match a "spamalias" wildcard declaration will cause that
message to not be propagated to any neighbors. Diablo
behaves as though some "alias" command matched that Path:
header on all outbound feeds.
Intended for use on transit-only servers (since in this
implementation, "spamalias" checks are too late to prevent
the message from being viewable from a local reader), the
"spamalias" parameter acts globally and must appear in the
GLOBAL section.
For example,
groupdef GLOBAL
spamalias badpornsite
spamalias xxxnilla
spamalias *.unresponsive.net
spamalias abuseisokay*
spamalias annoyizer
end
These entries would block messages with these Path: headers:
Path: someplace!elsewhere!okaysite!badpornsite!not-for-mail
Path: someplace!elsewhere!okaysite!notspammershonest!annoyizer
Path: someplace!okaysite!xxxnilla!seemelive
Path: someplace!elsewhere!okaysite!west-coast.unresponsive.net!me
Path: someplace!elsewhere!okaysite!east-coast.unresponsive.net!them
Path: someplace!goodplace!elsewhere!abuseisokay
Path: someplace!goodplace!abuseisokay.com!nobody
However, these messages would not be blocked:
Path: someplace!xxxnilla!elsewhere!west.unresponsive.net!goodplace!xyz
Path: someplace!elsewhere!badpornsite!goodplace!gooduser
Path: someplace!elsewhere!abuseisokay.com!elsewhere!innocentuser
Only the last and next-to-last elements of the Path: header are checked
by the "spamalias" command.
All too frequently there are sites who originate spam or
abuse that need to be blocked (mainly to punish a lack of
response to abuse/spam problems), but you don't want to
lose articles that are just passing through that site and
didn't originate there, since your history database is now
poisoned, so you can't get the article via some other route
just to make sure it didn't really originate there. (If
you have limited redundant feeds, hoping an article will
show up via a less-tainted route may not be an option.)
Using "spamalias" provides reasonable blocking of specific
sites or spam/abuse signatures without blacking-out unintended
chunks of the USENET network.
When a "spamalias" match is made, an entry is logged to
news.debug, indicating if it was a position A (originating
site) or B (user) match. The entry in incoming.log for
such an article will show no sites to be fed.
Odd Things and Cautions about Spamalias:
If "spamalias" entries are used outside of the GLOBAL
section, it may either sly slow Diablo because of the amount
of redundant work it ends up doing and may still act globally
or act on some but not all feeds. The "spamalias" command
is not designed to be used to filter subsets of outbound
feeds.
There is a potential to defeat "spamalias" if the blocked
site allows Path: preloading, but in three years of use,
the incidents of any abuser bothering to work around such
filtering have been virtually non-existent, and the number
of big sites that still allow preloading and don't take
their own action against spam/abuse is getting quite small.
When writing "spamalias" expressions that are intended to
match usernames, be cautious since some "dummy" usernames
appear in posts originating from many sites, likely in
addition to whatever it is you are trying to block. For
example, never block "not-for-mail" or "news", since these
are commonly used in place of actual user names.
Since the "spamalias" command tests both originating site
and originating user fields, make sure an entry written
for one field isn't busily zapping non-abusive posts that
happen to match on the other field.
WHEREIS COMMAND
Diablo allows an optional NNTP command 'WHEREIS' which returns
the location of an article on the local spool (filename, offset
and size). This option is useful for a reader accessing the
article from an NFS mounted spool. The use of NFS in Diablo
is strongly discouraged, but some people are forced to use
diablo in an NFS-only environement.
200 news.example.com NNTP Service Ready
WHEREIS <[email protected]>
223 0 whereis <[email protected]> in \
/news/spool/news/D.00fe7ee6/B.036a offset 83825 length 1018
(line wrap added for this document)