DSPAM is a statistical spam filter. It is intended to be scalable, content-based spam filter for large multi-user systems. Documentation for dspam .
NOTE: This is the only component I’ve not found to NOT speak SSL over SMTP/LMTP. It does connect to PostgreSQL over SSL just fine.
Create/populate the database
First, we will need to create a database and then populate it. Creating is straight forward. Connect to Postgres and type in:
1CREATE ROLE dspam WITH LOGIN ENCRYPTED PASSWORD 'password';
2CREATE DATABASE dspam WITH OWNER = dspam;
Now its time to populate it. You can use the following to populate your dspam database:
1psql -U dspam -W dspam < /usr/local/share/examples/dspam/pgsql/pgsql_objects.sql
2psql -U dspam -W dspam < /usr/local/share/examples/dspam/pgsql/virtual_users.sql
You’ll get some warnings about permissions for analyzing. These can safely be ignored. We will setup a daily script that will perform database maintenance including analyization.
If you get errors like this (This doesn’t happen right away, but it may happen after things have been running for days/weeks/months):
1Dec 12 03:41:29 pgsql.cmhome postgres[92640]: [126-2] DETAIL: Key (uid, token)=(28, -7501006800599240569) already exists.
2Dec 12 03:41:29 pgsql.cmhome postgres[92640]: [126-3] STATEMENT: PREPARE dspam_update_plan (bigint) AS UPDATE dspam_token_data SET last_hit=CURRENT_DATE,innocent_hits=innocent_hits+1 WHERE uid=28 AND token=$1;PREPARE dspam_insert_plan (bigint,int,int) AS INSERT INTO dspam_token_data (uid,token,spam_hits,innocent_hits,last_hit) VALUES (28,$1,$2,$3,CURRENT_DATE);
You can add the following to postgres. It won’t fix dspam, but it will work around the problem by telling postgres to attempt to do an update, and if the update fails, redo the same thing as an insert. It also avoids the round-trip time of sending the error back to the mail server, and having dspam issue another sql statement.
1CREATE FUNCTION update_rec(TEXT,INTEGER,INTEGER) RETURNS VOID AS $$
2BEGIN
3UPDATE dspam.table SET f1=$2, f2=$3 WHERE k=$1;
4IF NOT FOUND THEN
5INSERT INTO dspam.table (k,f1,f2) VALUES ($1,$2,$3);
6END IF;
7END;
8$$ LANGUAGE plpgsql;
Configuring DSPAM
Now we can edit the dspam.conf (/usr/local/etc). Most of it is default. Of note, I’ve told it to store preferences in the database. I’ve also added a long list of headers to ignore (we don’t want somebody setting a header that causes us to not scan something, or that sends confusing messages to the user. Here is what mine looks like:
1## $Id: dspam.conf.in,v 1.103 2011/11/10 00:27:34 tomhendr Exp $
2## dspam.conf -- DSPAM configuration file
3##
4
5#
6# DSPAM Home: Specifies the base directory to be used for DSPAM storage
7#
8Home /var/db/dspam
9
10#
11# StorageDriver: Specifies the storage driver backend (library) to use.
12# You'll only need to set this if you are using dynamic storage driver plugins
13# from a binary distribution. The default build statically links the storage
14# driver (when only one is specified at configure time), overriding this
15# setting, which only comes into play if multiple storage drivers are specified
16# at configure time. When using dynamic linking, be sure to include the path
17# to the library if necessary, and some systems may use an extension other
18# than .so (e.g. OSX uses .dylib).
19#
20StorageDriver /usr/local/lib/dspam/libpgsql_drv.so
21
22# Trusted Delivery Agent: Specifies the local delivery agent DSPAM should call
23# when delivering mail as a trusted user. Use %u to specify the user DSPAM is
24# processing mail for. It is generally a good idea to allow the MTA to specify
25# the pass-through arguments at run-time, but they may also be specified here.
26#
27TrustedDeliveryAgent "/usr/local/sbin/sendmail" # FreeBSD
28
29# Untrusted Delivery Agent: Specifies the local delivery agent and arguments
30# DSPAM should use when delivering mail and running in untrusted user mode.
31# Because DSPAM will not allow pass-through arguments to be specified to
32# untrusted users, all arguments should be specified here. Use %u to specify
33# the user DSPAM is processing mail for. This configuration parameter is only
34# necessary if you plan on allowing untrusted processing.
35#
36#UntrustedDeliveryAgent "/usr/bin/procmail -d %u"
37#UntrustedDeliveryAgent "/usr/libexec/mail.local"
38
39#
40# SMTP or LMTP Delivery: Alternatively, you may wish to use SMTP or LMTP
41# delivery to deliver your message to the mail server instead of using a
42# delivery agent. You will need to configure with --enable-daemon to use host
43# delivery, however you do not need to operate in daemon mode. Specify an IP
44# address or UNIX path to a domain socket below as a host.
45#
46# If you would like to set up DeliveryHost's on a per-domain basis, use
47# the syntax: DeliveryHost.example.org 1.2.3.4
48#
49DeliveryHost 127.0.0.1
50DeliveryPort 24
51DeliveryIdent localhost
52DeliveryProto LMTP
53
54# Quarantine Agent: DSPAM's default behavior is to quarantine all mail it
55# thinks is spam. If you wish to override this behavior, you may specify
56# a quarantine agent which will be called with all messages DSPAM thinks is
57# spam. Use %u to specify the user DSPAM is processing mail for.
58#
59#QuarantineAgent "/usr/bin/procmail -d spam"
60
61# DSPAM can optionally process "plused users" (addresses in the user+detail
62# form) by truncating the username just before the "+", so all internal
63# processing occurs for "user", but delivery will be performed for
64# "user+detail". This is only useful if the LDA can handle "plused users"
65# (for example Cyrus IMAP) and when configured for LMTP delivery above
66#
67EnablePlusedDetail on
68
69# Character to use as seperator between user names and address extensions.
70# If you change this value then please adjust QuarantineMailbox to use the
71# new specified character. The default is '+'.
72#
73PlusedCharacter +
74
75# Turn this feature on if you want to force DSPAM to lowercase the "plused
76# users" username.
77#
78PlusedUserLowercase on
79
80# Quarantine Mailbox: DSPAM's LMTP code can send spam mail using LMTP to a
81# "plused" mailbox (such as user+quarantine) leaving quarantine processing
82# for retraining or deletion to be performed by the LDA and the mail client.
83# "plused" mailboxes are supported by Cyrus IMAP and possibly other LDAs. If
84# you don't set/change PlusedCharacter then the mailbox name must have the +
85# since the + is the default used character.
86#
87QuarantineMailbox +Junk
88
89# OnFail: What to do if local delivery or quarantine should fail. If set
90# to "unlearn", DSPAM will unlearn the message prior to exiting with an
91# un successful return code. The default option, "error" will not unlearn
92# the message but return the appropriate error code. The unlearn option
93# is useful on some systems where local delivery failures will cause the
94# message to be requeued for delivery, and could result in the message
95# being processed multiple times. During a very large failure, however,
96# this could cause a significant load increase.
97#
98OnFail unlearn
99
100# Trusted Users: Only the users specified below will be allowed to perform
101# administrative functions in DSPAM such as setting the active user and
102# accessing tools. All other users attempting to run DSPAM will be restricted;
103# their uids will be forced to match the active username and they will not be
104# able to specify delivery agent privileges or use tools.
105#
106Trust root
107Trust dspam
108Trust virtual
109Trust dovenull
110Trust mail
111Trust daemon
112Trust postfix
113
114# Debugging: Enables debugging for some or all users. IMPORTANT: DSPAM must
115# be compiled with debug support in order to use this option. DSPAM should
116# never be running in production with debug active unless you are
117# troubleshooting problems.
118#
119# DebugOpt: One or more of: process, classify, spam, fp, inoculation, corpus
120# process standard message processing
121# classify message classification using --classify
122# spam error correction of missed spam
123# fp error correction of false positives
124# inoculation message inoculations (source=inoculation)
125# corpus corpusfed messages (source=corpus)
126#
127#Debug *
128#DebugOpt process
129
130# Training Mode: The default training mode to use for all operations, when
131# one has not been specified on the commandline or in the user's preferences.
132# Acceptable values are:
133# toe Train on Error (Only)
134# teft Train Everything (Trains on every message)
135# tum Train Until Mature (Train only tokens without enough data)
136# notrain Do not train or store signatures (large ISP systems, post-train)
137#
138TrainingMode teft
139
140# TestConditionalTraining: By default, dspam will retrain certain errors
141# until the condition is no longer met. This usually accelerates learning.
142# Some people argue that this can increase the risk of errors, however.
143#
144TestConditionalTraining on
145
146# Features: Specify features to activate by default; can also be specified
147# on the commandline. See the documentation for a list of available features.
148# If _any_ features are specified on the commandline, these are ignored.
149#
150Feature noise
151Feature whitelist
152
153# Training Buffer: The training buffer waters down statistics during training.
154# It is designed to prevent false positives, but can also dramatically reduce
155# dspam's catch rate during initial training. This can be a number from 0
156# (no buffering) to 10 (maximum buffering). If you are paranoid about false
157# positives, you should probably enable this option.
158#
159#Feature tb=5
160
161# Algorithms: Specify the statistical algorithms to use, overriding any
162# defaults configured in the build. The options are:
163# naive Naive-Bayesian (All Tokens)
164# graham Graham-Bayesian ("A Plan for Spam")
165# burton Burton-Bayesian (SpamProbe)
166# robinson Robinson's Geometric Mean Test (Obsolete)
167# chi-square Fisher-Robinson's Chi-Square Algorithm
168#
169# You may have multiple algorithms active simultaneously, but it is strongly
170# recommended that you group Bayesian algorithms with other Bayesian
171# algorithms, and any use of Chi-Square remain exclusive.
172#
173# NOTE: For standard "CRM114" Markovian weighting, use 'naive', or consider
174# using 'burton' for slightly better accuracy
175#
176# Don't mess with this unless you know what you're doing
177#
178Algorithm graham burton
179
180# Tokenizer: Specify the tokenizer to use. The tokenizer is the piece
181# responsible for parsing the message into individual tokens. Depending on
182# how many resources you are willing to trade off vs. accuracy, you may
183# choose to use a less or more detailed tokenizer:
184# word uniGram (single word) tokenizer
185# Tokenizes message into single individual words/tokens
186# example: "free" and "viagra"
187# chain biGram (chained tokens) tokenizer (default)
188# Single words + chains adjacent tokens together
189# example: "free" and "viagra" and "free viagra"
190# sbph Sparse Binary Polynomial Hashing tokenizer
191# Creates sparse token patterns across sliding window of 5-tokens
192# example: "the quick * fox jumped" and "the * * fox jumped"
193# osb Orthogonal Sparse biGram tokenizer
194# Similar to SBPH, but only uses the biGrams
195# example: "the * * fox" and "the * * * jumped"
196#
197# In general the reccomendation is to use 'osb' for new installations.
198# The default value of 'chain' remains here as not to surprise anyone upgrading
199# that has not changed from the default value.
200#
201Tokenizer osb
202
203# PValue: Specify the technique used for calculating Probability Values,
204# overriding any defaults configured in the build. These options are:
205# bcr Bayesian Chain Rule (Graham's Technique - "A Plan for Spam")
206# robinson Robinson's Technique (used in Chi-Square)
207# markov Markovian Weighted Technique (for Markovian discrimination)
208#
209# Unlike the "Algorithms" property, you may only have one of these defined.
210# Use of the chi-square algorithm automatically changes this to robinson.
211#
212# Don't mess with this unless you know what you're doing.
213#
214PValue bcr
215
216#
217# WebStats: Enable this if you are using the CGI, which writes .stats files
218WebStats on
219
220#
221# ImprobabilityDrive: Calculate odds-ratios for ham/spam, and add to
222# X-DSPAM-Improbability headers
223#
224ImprobabilityDrive on
225
226#
227# Preferences: Specify any preferences to set by default, unless otherwise
228# overridden by the user (see next section) or a default.prefs file.
229# If user or default.prefs are found, the user's preferences will override any
230# defaults.
231#
232Preference "trainingMode= TOE" # { TOE | TUM | TEFT | NOTRAIN } -> default:teft
233Preference "spamAction=tag" # { quarantine | tag | deliver } -> default:quarantine
234Preference "spamSubject=" # { string } -> default:[SPAM]
235Preference "statisticalSedation=5" # { 0 - 10 } -> default:0
236Preference "enableBNR=on" # { on | off } -> default:off
237Preference "enableWhitelist=on" # { on | off } -> default:on
238Preference "signatureLocation=headers" # { message | headers } -> default:message
239Preference "tagSpam=on" # { on | off }
240Preference "tagNonspam=on" # { on | off }
241Preference "showFactors=on" # { on | off } -> default:off
242Preference "optIn=off" # { on | off }
243Preference "optOut=off" # { on | off }
244Preference "whitelistThreshold=10" # { Integer } -> default:10
245Preference "makeCorpus=off" # { on | off } -> default:off
246Preference "storeFragments=off" # { on | off } -> default:off
247Preference "localStore=" # { on | off } -> default:username
248Preference "processorBias=on" # { on | off } -> default:on
249Preference "fallbackDomain=off" # { on | off } -> default:off
250Preference "trainPristine=off" # { on | off } -> default:off
251Preference "optOutClamAV=off" # { on | off } -> default:off
252Preference "ignoreRBLLookups=off" # { on | off } -> default:off
253Preference "RBLInoculate=off" # { on | off } -> default:off
254Preference "notifications=off" # { on | off } -> default:off
255
256#
257# Overrides: Specifies the user preferences which may override configuration
258# and commandline defaults. Any other preferences supplied by an untrusted user
259# will be ignored.
260#
261AllowOverride enableBNR
262AllowOverride enableWhitelist
263AllowOverride fallbackDomain
264AllowOverride ignoreGroups
265AllowOverride ignoreRBLLookups
266AllowOverride localStore
267AllowOverride makeCorpus
268AllowOverride optIn
269AllowOverride optOut
270AllowOverride optOutClamAV
271AllowOverride processorBias
272AllowOverride RBLInoculate
273AllowOverride showFactors
274AllowOverride signatureLocation
275AllowOverride spamAction
276AllowOverride spamSubject
277AllowOverride statisticalSedation
278AllowOverride storeFragments
279AllowOverride tagNonspam
280AllowOverride tagSpam
281AllowOverride trainPristine
282AllowOverride trainingMode
283AllowOverride whitelistThreshold
284AllowOverride dailyQuarantineSummary
285AllowOverride notifications
286
287# --- PostgreSQL ---
288
289# For PgSQLServer you can Use a TCP/IP address or a socket. If your socket is
290# in /var/run/postgresql/.s.PGSQL.5432 specify just the path where the socket
291# resits (without .s.PGSQL.5432).
292
293PgSQLServer pgsql.cmhome
294PgSQLPort 5432
295PgSQLUser dspam
296PgSQLPass password
297PgSQLDb dspam
298
299# If you're running DSPAM in client/server (daemon) mode, uncomment the
300# setting below to override the default connection cache size (the number
301# of connections the server pools between all clients).
302#
303PgSQLConnectionCache 2
304
305# UIDInSignature: PgSQL supports the insertion of the user id into the DSPAM
306# signature. This allows you to create one single spam or fp alias
307# (pointing to some arbitrary user), and the uid in the signature will
308# switch to the correct user. Result: you need only one spam alias
309
310#PgSQLUIDInSignature on
311
312# If you're using vpopmail or some other type of virtual setup and wish to
313# change the table dspam uses to perform username/uid lookups, you can over-
314# ride it below
315
316PgSQLVirtualTable dspam_virtual_uids
317PgSQLVirtualUIDField uid
318PgSQLVirtualUsernameField username
319
320Notifications off
321
322# TxtDirectory: the directory that holds the templates for notification
323# messages (see Notifications) and tagging (see tagSpam/tagNonspam).
324#
325TxtDirectory /var/db/dspam/txt
326
327# Purge configuration: Set dspam_clean purge default options, if not otherwise
328# specified on the commandline
329#
330# Purge configuration for SQL-based installations using purge.sql
331#
332PurgeSignature off # Specified in purge.sql
333PurgeNeutral 90
334PurgeUnused off # Specified in purge.sql
335PurgeHapaxes off # Specified in purge.sql
336PurgeHits1S off # Specified in purge.sql
337PurgeHits1I off # Specified in purge.sql
338
339#
340# Local Mail Exchangers: Used for source address tracking, tells DSPAM which
341# mail exchangers are local and therefore should be ignored in the Received:
342# header when tracking the source of an email. Note: you should use the address
343# of the host as appears between brackets [ ] in the Received header.
344# By default DSPAM is considering the following IPs always as LocalMX:
345# 10.0.0.0/8 - Private IP addresses (RFC 1918)
346# 127.0.0.0/8 - Localhost Loopback Address (RFC 1700)
347# 169.254.0.0/16 - Zeroconf / APIPA (RFC 3330)
348# 172.16.0.0/12 - Private IP addresses (RFC 1918)
349# 192.168.0.0/16 - Private IP addresses (RFC 1918)
350#
351LocalMX 127.0.0.1 198.18.0.0/20
352
353#
354# Logging: Disabling logging for users will make usage graphs unavailable to
355# them. Disabling system logging will make admin graphs unavailable.
356#
357SystemLog on
358UserLog on
359
360#
361# TrainPristine: for systems where the original message remains server side
362# and can therefore be presented in pristine format for retraining. This option
363# will cause DSPAM to cease all writing of signatures and DSPAM headers to the
364# message, and deliver the message in as pristine format as possible. This mode
365# REQUIRES that the original message in its pristine format (as of delivery)
366# be presented for retraining, as in the case of webmail, imap, or other
367# applications where the message is actually kept server-side during reading,
368# and is preserved. DO NOT use this switch unless the original message can be
369# presented for retraining with the ORIGINAL HEADERS and NO MODIFICATIONS.
370#
371# NOTE: You can't use this setting with dspam_trian; if you're going to use it,
372# wait until after you train any corpora.
373#
374TrainPristine off
375
376#
377# Opt: in or out; determines DSPAM's default filtering behavior. If this value
378# is set to in, users must opt-in to filtering by dropping a .dspam file in
379# /var/dspam/opt-in/user.dspam (or if you have homedirs configured, a .dspam
380# folder in their home directory). The default is opt-out, which means all
381# users will be filtered unless a .nodspam file is dropped in
382# /var/dspam/opt-out/user.nodspam
383#
384Opt out
385
386#
387# TrackSources: specify which (if any) source addresses to track and report
388# them to syslog (mail.info). This is useful if you're running a firewall or
389# blacklist and would like to use this information. Spam reporting also drops
390# RABL blacklist files (see http://www.nuclearelephant.com/projects/rabl/).
391#
392TrackSources spam nonspam virus
393
394#
395# ParseToHeaders: In lieu of setting up individual aliases for each user,
396# DSPAM can be configured to automatically parse the To: address for spam and
397# false positive forwards. From there, it can be configured to either set the
398# DSPAM user based on the username specified in the header and/or change the
399# training class and source accordingly. The options below can be used to
400# customize most common types of header parsing behavior to avoid the need for
401# multiple aliases, or if using LMTP, aliases entirely..
402#
403# ParseToHeader: Parse the To: headers of an incoming message. This must be
404# set to 'on' to use either of the following features.
405#
406# ChangeModeOnParse: Automatically change the class (to spam or innocent)
407# depending on whether spam- or notspam- was specified, and change the source
408# to 'error'. This is convenient if you're not using aliases at all, but
409# are delivering via LMTP.
410#
411# ChangeUserOnParse: Automatically change the username to match that specified
412# in the To: header. For example, spam-bob@example.org will set the username
413# to bob, ignoring any --user passed in. This may not always be desirable if
414# you are using virtual email addresses as usernames. Options:
415# on or user take the portion before the @ sign only
416# full take everything after the initial {spam,notspam}-.
417#
418ParseToHeaders on
419ChangeModeOnParse off
420ChangeUserOnParse full
421
422#
423# Broken MTA Options: Some MTAs don't support the proper functionality
424# necessary. In these cases you can activate certain features in DSPAM to
425# compensate. 'returnCodes' causes DSPAM to return an exit code of 99 if
426# the message is spam, 0 if not, or a negative code if an error has occured.
427# Specifying 'case' causes DSPAM to force the input usernames to lowercase.
428# Specifying 'lineStripping' causes DSPAM to strip ^M's from messages passed
429# in.
430#
431#Broken returnCodes
432Broken case
433Broken lineStripping
434
435#
436# MaxMessageSize: You may specify a maximum message size for DSPAM to process.
437# If the message is larger than the maximum size, it will be delivered
438# without processing. Value is in bytes.
439#
440MaxMessageSize 5000000
441
442# --- ClamAV ---
443
444#
445# Virus Checking: If you are running clamd, DSPAM can perform stream-based
446# virus checking using TCP. Uncomment the values below to enable virus
447# checking.
448#
449# ClamAVResponse: reject (reject or drop the message with a permanent failure)
450# accept (accept the message and quietly drop the message)
451# spam (treat as spam and quarantine/tag/whatever)
452#
453ClamAVPort 3310
454ClamAVHost 127.0.0.1
455ClamAVResponse reject
456
457# --- CLIENT / SERVER ---
458
459#
460# Daemonized Server: If you are running DSPAM as a daemonized server using
461# --daemon, the following parameters will override the default. Use the
462# ServerPass option to set up accounts for each client machine. The DSPAM
463# server will process and deliver the message based on the parameters
464# specified. If you want the client machine to perform delivery, use
465# the --stdout option in conjunction with a local setup.
466#
467# ServerHost: Not enabling ServerHost will bind DSPAM server to all available
468# interfaces.
469#
470ServerHost 127.0.0.1
471ServerPort 2424
472ServerQueueSize 32
473ServerPID /var/run/dspam/dspam.pid
474
475#
476# ServerMode specifies the type of LMTP server to start. This can be one of:
477# dspam: DSPAM-proprietary DLMTP server, for communicating with dspamc
478# standard: Standard LMTP server, for communicating with Postfix or other MTA
479# auto: Speak both DLMTP and LMTP; auto-detect by ServerPass.IDENT
480#
481ServerMode auto
482
483# If supporting DLMTP (dspam) mode, dspam clients will require authentication
484# as they will be passing in parameters. The idents below will be used to
485# determine which clients will be speaking DLMTP, so if you will be using
486# both LMTP and DLMTP from the same host, be sure to use something other
487# than the server's hostname below (which will be sent by the MTA during a
488# standard LMTP LHLO).
489#
490#ServerPass.Relay1 "secret"
491ServerPass.client "password"
492
493# If supporting standard LMTP mode, server parameters will need to be specified
494# here, as they will not be passed in by the mail server. The ServerIdent
495# specifies the 250 response code ident sent back to connecting clients and
496# should be set to the hostname of your server, or an alias.
497#
498# NOTE: If you specify --user in ServerParameters, the RCPT TO will be
499# used only for delivery, and not set as the active user for processing.
500#
501ServerParameters "--deliver=innocent, spam -d %u"
502ServerIdent "mx.cryptomonkeys.com"
503
504# If you wish to use a local domain socket instead of a TCP socket, uncomment
505# the following. It is strongly recommended you use local domain sockets if
506# you are running the client and server on the same machine, as it eliminates
507# much of the bandwidth overhead.
508#
509#ServerDomainSocketPath "/var/run/dspam/dspam.sock"
510
511#
512# Client Mode: If you are running DSPAM in client/server mode, uncomment and
513# set these variables. A ClientHost beginning with a / will be treated as
514# a domain socket.
515#
516#ClientHost /var/run/dspam/dspam.sock
517#ClientIdent "secret@Relay1"
518#
519ClientHost 127.0.0.1
520ClientPort 2424
521ClientIdent "password@client"
522
523# ProcessorURLContext: By default, a URL context is generated for URLs, which
524# records their tokens as separate from words found in documents. To use
525# URL tokens in the same context as words, turn this feature off.
526#
527ProcessorURLContext on
528
529# ProcessorBias: Bias causes the filter to lean more toward 'innocent', and
530# usually greatly reduces false positives. It is the default behavior of
531# most Bayesian filters (including dspam).
532#
533# NOTE: You probably DONT want this if you're using Markovian Weighting, unless
534# you are paranoid about false positives.
535#
536ProcessorBias on
537
538# StripRcptDomain: Cut the domain (including the at sign) from recipients.
539# This is particularly useful if the recipient name is equal to real user
540# accounts as recipients with domains tend to cause permission issues with
541# dspam-web.
542#
543StripRcptDomain off
544
545# GroupConfig: The configuration file for groups. See the README file
546# for details on how to enable users to combine their training data to
547# get better results.
548GroupConfig /var/db/dspam/group
549
550
551IgnoreHeader Accept-Language
552IgnoreHeader Approved
553IgnoreHeader Archive
554IgnoreHeader Authentication-Results
555IgnoreHeader Cache-Post-Path
556IgnoreHeader Cancel-Key
557IgnoreHeader Cancel-Lock
558IgnoreHeader Complaints-To
559IgnoreHeader Content-Description
560IgnoreHeader Content-Disposition
561IgnoreHeader Content-ID
562IgnoreHeader Content-Language
563IgnoreHeader Content-Return
564IgnoreHeader Content-Transfer-Encoding
565IgnoreHeader Content-Type
566IgnoreHeader DKIM-Signature
567IgnoreHeader Date
568IgnoreHeader Disposition-Notification-To
569IgnoreHeader DomainKey-Signature
570IgnoreHeader Importance
571IgnoreHeader In-Reply-To
572IgnoreHeader Injection-Info
573IgnoreHeader Lines
574IgnoreHeader List-Archive
575IgnoreHeader List-Help
576IgnoreHeader List-Id
577IgnoreHeader List-Post
578IgnoreHeader List-Subscribe
579IgnoreHeader List-Unsubscribe
580IgnoreHeader Message-ID
581IgnoreHeader Message-Id
582IgnoreHeader NNTP-Posting-Date
583IgnoreHeader NNTP-Posting-Host
584IgnoreHeader Newsgroups
585IgnoreHeader OpenPGP
586IgnoreHeader Organization
587IgnoreHeader Originator
588IgnoreHeader PGP-ID
589IgnoreHeader Path
590IgnoreHeader Received
591IgnoreHeader Received-SPF
592IgnoreHeader References
593IgnoreHeader Reply-To
594IgnoreHeader Resent-Date
595IgnoreHeader Resent-From
596IgnoreHeader Resent-Message-ID
597IgnoreHeader Thread-Index
598IgnoreHeader Thread-Topic
599IgnoreHeader User-Agent
600IgnoreHeader X--MailScanner-SpamCheck
601IgnoreHeader X-AV-Scanned
602IgnoreHeader X-AVAS-Spam-Level
603IgnoreHeader X-AVAS-Spam-Score
604IgnoreHeader X-AVAS-Spam-Status
605IgnoreHeader X-AVAS-Spam-Symbols
606IgnoreHeader X-AVAS-Virus-Status
607IgnoreHeader X-AVK-Virus-Check
608IgnoreHeader X-Abuse
609IgnoreHeader X-Abuse-Contact
610IgnoreHeader X-Abuse-Info
611IgnoreHeader X-Abuse-Management
612IgnoreHeader X-Abuse-To
613IgnoreHeader X-Abuse-and-DMCA-Info
614IgnoreHeader X-Accept-Language
615IgnoreHeader X-Admission-MailScanner-SpamCheck
616IgnoreHeader X-Admission-MailScanner-SpamScore
617IgnoreHeader X-Amavis-Alert
618IgnoreHeader X-Amavis-Hold
619IgnoreHeader X-Amavis-Modified
620IgnoreHeader X-Amavis-OS-Fingerprint
621IgnoreHeader X-Amavis-PenPals
622IgnoreHeader X-Amavis-PolicyBank
623IgnoreHeader X-AntiVirus
624IgnoreHeader X-Antispam
625IgnoreHeader X-Antivirus
626IgnoreHeader X-Antivirus-Scanner
627IgnoreHeader X-Antivirus-Status
628IgnoreHeader X-Archive
629IgnoreHeader X-Assp-Spam-Prob
630IgnoreHeader X-Attention
631IgnoreHeader X-BTI-AntiSpam
632IgnoreHeader X-Barracuda
633IgnoreHeader X-Barracuda-Bayes
634IgnoreHeader X-Barracuda-Spam-Flag
635IgnoreHeader X-Barracuda-Spam-Report
636IgnoreHeader X-Barracuda-Spam-Score
637IgnoreHeader X-Barracuda-Spam-Status
638IgnoreHeader X-Barracuda-Virus-Scanned
639IgnoreHeader X-BeenThere
640IgnoreHeader X-Bogosity
641IgnoreHeader X-Brightmail-Tracker
642IgnoreHeader X-CRM114-CacheID
643IgnoreHeader X-CRM114-Status
644IgnoreHeader X-CRM114-Version
645IgnoreHeader X-CTASD-IP
646IgnoreHeader X-CTASD-RefID
647IgnoreHeader X-CTASD-Sender
648IgnoreHeader X-Cache
649IgnoreHeader X-ClamAntiVirus-Scanner
650IgnoreHeader X-Comment-To
651IgnoreHeader X-Comments
652IgnoreHeader X-Complaints
653IgnoreHeader X-Complaints-Info
654IgnoreHeader X-Complaints-To
655IgnoreHeader X-DKIM
656IgnoreHeader X-DMCA-Complaints-To
657IgnoreHeader X-DMCA-Notifications
658IgnoreHeader X-Despammed-Tracer
659IgnoreHeader X-ELTE-SpamCheck
660IgnoreHeader X-ELTE-SpamCheck-Details
661IgnoreHeader X-ELTE-SpamScore
662IgnoreHeader X-ELTE-SpamVersion
663IgnoreHeader X-ELTE-VirusStatus
664IgnoreHeader X-Enigmail-Supports
665IgnoreHeader X-Enigmail-Version
666IgnoreHeader X-Evolution-Source
667IgnoreHeader X-Extra-Info
668IgnoreHeader X-FSFE-MailScanner
669IgnoreHeader X-FSFE-MailScanner-From
670IgnoreHeader X-Face
671IgnoreHeader X-Fellowship-MailScanner
672IgnoreHeader X-Fellowship-MailScanner-From
673IgnoreHeader X-Forwarded
674IgnoreHeader X-GMX-Antispam
675IgnoreHeader X-GMX-Antivirus
676IgnoreHeader X-GPG-Fingerprint
677IgnoreHeader X-GPG-Key-ID
678IgnoreHeader X-GPS-DegDec
679IgnoreHeader X-GPS-MGRS
680IgnoreHeader X-GWSPAM
681IgnoreHeader X-Gateway
682IgnoreHeader X-Greylist
683IgnoreHeader X-HTMLM
684IgnoreHeader X-HTMLM-Info
685IgnoreHeader X-HTMLM-Score
686IgnoreHeader X-HTTP-Posting-Host
687IgnoreHeader X-HTTP-UserAgent
688IgnoreHeader X-HTTP-Via
689IgnoreHeader X-Headers-End
690IgnoreHeader X-ID
691IgnoreHeader X-IMAIL-SPAM-STATISTICS
692IgnoreHeader X-IMAIL-SPAM-URL-DBL
693IgnoreHeader X-IMAIL-SPAM-VALFROM
694IgnoreHeader X-IMAIL-SPAM-VALHELO
695IgnoreHeader X-IMAIL-SPAM-VALREVDNS
696IgnoreHeader X-Info
697IgnoreHeader X-IronPort-Anti-Spam-Filtered
698IgnoreHeader X-IronPort-Anti-Spam-Result
699IgnoreHeader X-KSV-Antispam
700IgnoreHeader X-Kaspersky-Antivirus
701IgnoreHeader X-MDAV-Processed
702IgnoreHeader X-MDRemoteIP
703IgnoreHeader X-MDaemon-Deliver-To
704IgnoreHeader X-MIE-MailScanner-SpamCheck
705IgnoreHeader X-MIMEOLE
706IgnoreHeader X-MIMETrack
707IgnoreHeader X-MMS-Spam-Filter-ID
708IgnoreHeader X-MS-Exchange-Forest-RulesExecuted
709IgnoreHeader X-MS-Exchange-Organization-Antispam-Report
710IgnoreHeader X-MS-Exchange-Organization-AuthAs
711IgnoreHeader X-MS-Exchange-Organization-AuthDomain
712IgnoreHeader X-MS-Exchange-Organization-AuthMechanism
713IgnoreHeader X-MS-Exchange-Organization-AuthSource
714IgnoreHeader X-MS-Exchange-Organization-Journal-Report
715IgnoreHeader X-MS-Exchange-Organization-Original-Scl
716IgnoreHeader X-MS-Exchange-Organization-Original-Sender
717IgnoreHeader X-MS-Exchange-Organization-OriginalArrivalTime
718IgnoreHeader X-MS-Exchange-Organization-OriginalSize
719IgnoreHeader X-MS-Exchange-Organization-PCL
720IgnoreHeader X-MS-Exchange-Organization-Quarantine
721IgnoreHeader X-MS-Exchange-Organization-SCL
722IgnoreHeader X-MS-Exchange-Organization-SenderIdResult
723IgnoreHeader X-MS-Has-Attach
724IgnoreHeader X-MS-TNEF-Correlator
725IgnoreHeader X-MSMail-Priority
726IgnoreHeader X-MailScanner
727IgnoreHeader X-MailScanner-Information
728IgnoreHeader X-MailScanner-SpamCheck
729IgnoreHeader X-Mailer
730IgnoreHeader X-Mailman-Version
731IgnoreHeader X-Mlf-Spam-Status
732IgnoreHeader X-NAI-Spam-Checker-Version
733IgnoreHeader X-NAI-Spam-Flag
734IgnoreHeader X-NAI-Spam-Level
735IgnoreHeader X-NAI-Spam-Report
736IgnoreHeader X-NAI-Spam-Route
737IgnoreHeader X-NAI-Spam-Rules
738IgnoreHeader X-NAI-Spam-Score
739IgnoreHeader X-NAI-Spam-Threshold
740IgnoreHeader X-NEWT-spamscore
741IgnoreHeader X-NNTP-Posting-Date
742IgnoreHeader X-NNTP-Posting-Host
743IgnoreHeader X-NetcoreISpam1-ECMScanner
744IgnoreHeader X-NetcoreISpam1-ECMScanner-From
745IgnoreHeader X-NetcoreISpam1-ECMScanner-Information
746IgnoreHeader X-NetcoreISpam1-ECMScanner-SpamCheck
747IgnoreHeader X-NetcoreISpam1-ECMScanner-SpamScore
748IgnoreHeader X-Newsreader
749IgnoreHeader X-Newsserver
750IgnoreHeader X-No-Archive
751IgnoreHeader X-No-Spam
752IgnoreHeader X-OSBF-Lua-Score
753IgnoreHeader X-OWM-SpamCheck
754IgnoreHeader X-OWM-VirusCheck
755IgnoreHeader X-Olypen-Virus
756IgnoreHeader X-Orig-Path
757IgnoreHeader X-OriginalArrivalTime
758IgnoreHeader X-Originating-IP
759IgnoreHeader X-PAA-AntiVirus
760IgnoreHeader X-PAA-AntiVirus-Message
761IgnoreHeader X-PGP-Fingerprint
762IgnoreHeader X-PGP-Hash
763IgnoreHeader X-PGP-ID
764IgnoreHeader X-PGP-Key
765IgnoreHeader X-PGP-Key-Fingerprint
766IgnoreHeader X-PGP-KeyID
767IgnoreHeader X-PGP-Sig
768IgnoreHeader X-PIRONET-NDH-MailScanner-SpamCheck
769IgnoreHeader X-PIRONET-NDH-MailScanner-SpamScore
770IgnoreHeader X-PMX
771IgnoreHeader X-PMX-Version
772IgnoreHeader X-PN-SPAMFiltered
773IgnoreHeader X-Posting-Agent
774IgnoreHeader X-Posting-ID
775IgnoreHeader X-Posting-IP
776IgnoreHeader X-Priority
777IgnoreHeader X-Proofpoint-Spam-Details
778IgnoreHeader X-Qmail-Scanner-1.25st
779IgnoreHeader X-Quarantine-ID
780IgnoreHeader X-RAV-AntiVirus
781IgnoreHeader X-RITmySpam
782IgnoreHeader X-RITmySpam-IP
783IgnoreHeader X-RITmySpam-Spam
784IgnoreHeader X-Rc-Spam
785IgnoreHeader X-Rc-Virus
786IgnoreHeader X-Received-Date
787IgnoreHeader X-RedHat-Spam-Score
788IgnoreHeader X-RedHat-Spam-Warning
789IgnoreHeader X-RegEx
790IgnoreHeader X-RegEx-Score
791IgnoreHeader X-Rocket-Spam
792IgnoreHeader X-SA-GROUP
793IgnoreHeader X-SA-RECEIPTSTATUS
794IgnoreHeader X-STA-NotSpam
795IgnoreHeader X-STA-Spam
796IgnoreHeader X-Scam-grey
797IgnoreHeader X-Scanned-By
798IgnoreHeader X-Sender
799IgnoreHeader X-SenderID
800IgnoreHeader X-Sohu-Antivirus
801IgnoreHeader X-Spam
802IgnoreHeader X-Spam-ASN
803IgnoreHeader X-Spam-Check
804IgnoreHeader X-Spam-Checked-By
805IgnoreHeader X-Spam-Checker
806IgnoreHeader X-Spam-Checker-Version
807IgnoreHeader X-Spam-Clean
808IgnoreHeader X-Spam-DCC
809IgnoreHeader X-Spam-Details
810IgnoreHeader X-Spam-Filter
811IgnoreHeader X-Spam-Filtered
812IgnoreHeader X-Spam-Flag
813IgnoreHeader X-Spam-Level
814IgnoreHeader X-Spam-OrigSender
815IgnoreHeader X-Spam-Pct
816IgnoreHeader X-Spam-Prev-Subject
817IgnoreHeader X-Spam-Processed
818IgnoreHeader X-Spam-Pyzor
819IgnoreHeader X-Spam-Rating
820IgnoreHeader X-Spam-Report
821IgnoreHeader X-Spam-Scanned
822IgnoreHeader X-Spam-Score
823IgnoreHeader X-Spam-Status
824IgnoreHeader X-Spam-Tagged
825IgnoreHeader X-Spam-Tests
826IgnoreHeader X-Spam-Tests-Failed
827IgnoreHeader X-Spam-Virus
828IgnoreHeader X-Spam-Warning
829IgnoreHeader X-Spam-detection-level
830IgnoreHeader X-SpamAssassin-Clean
831IgnoreHeader X-SpamAssassin-Warning
832IgnoreHeader X-SpamBouncer
833IgnoreHeader X-SpamCatcher-Score
834IgnoreHeader X-SpamCop-Checked
835IgnoreHeader X-SpamCop-Disposition
836IgnoreHeader X-SpamCop-Whitelisted
837IgnoreHeader X-SpamDetected
838IgnoreHeader X-SpamInfo
839IgnoreHeader X-SpamPal
840IgnoreHeader X-SpamPal-Timeout
841IgnoreHeader X-SpamReason
842IgnoreHeader X-SpamScore
843IgnoreHeader X-SpamTest-Categories
844IgnoreHeader X-SpamTest-Info
845IgnoreHeader X-SpamTest-Method
846IgnoreHeader X-SpamTest-Status
847IgnoreHeader X-SpamTest-Version
848IgnoreHeader X-Spamadvice
849IgnoreHeader X-Spamarrest-noauth
850IgnoreHeader X-Spamarrest-speedcode
851IgnoreHeader X-Spambayes-Classification
852IgnoreHeader X-Spamcount
853IgnoreHeader X-Spamsensitivity
854IgnoreHeader X-TERRACE-SPAMMARK
855IgnoreHeader X-TERRACE-SPAMRATE
856IgnoreHeader X-TM-AS-Category-Info
857IgnoreHeader X-TM-AS-MatchedID
858IgnoreHeader X-TM-AS-Product-Ver
859IgnoreHeader X-TM-AS-Result
860IgnoreHeader X-TMWD-Spam-Summary
861IgnoreHeader X-TNEFEvaluated
862IgnoreHeader X-Text-Classification
863IgnoreHeader X-Text-Classification-Data
864IgnoreHeader X-Trace
865IgnoreHeader X-UCD-Spam-Score
866IgnoreHeader X-User-Agent
867IgnoreHeader X-User-ID
868IgnoreHeader X-User-System
869IgnoreHeader X-Virus-Check
870IgnoreHeader X-Virus-Checked
871IgnoreHeader X-Virus-Checker-Version
872IgnoreHeader X-Virus-Scan
873IgnoreHeader X-Virus-Scanned
874IgnoreHeader X-Virus-Scanner
875IgnoreHeader X-Virus-Scanner-Result
876IgnoreHeader X-Virus-Status
877IgnoreHeader X-VirusChecked
878IgnoreHeader X-Virusscan
879IgnoreHeader X-WSS-ID
880IgnoreHeader X-WinProxy-AntiVirus
881IgnoreHeader X-WinProxy-AntiVirus-Message
882IgnoreHeader X-Yandex-Forward
883IgnoreHeader X-Yandex-Front
884IgnoreHeader X-Yandex-Spam
885IgnoreHeader X-Yandex-TimeMark
886IgnoreHeader X-cid
887IgnoreHeader X-iHateSpam-Checked
888IgnoreHeader X-iHateSpam-Quarantined
889IgnoreHeader X-policyd-weight
890IgnoreHeader X-purgate
891IgnoreHeader X-purgate-Ad
892IgnoreHeader X-purgate-ID
893IgnoreHeader X-sgxh1
894IgnoreHeader X-to-viruscore
895IgnoreHeader Xref
896IgnoreHeader acceptlanguage
897IgnoreHeader thread-index
898IgnoreHeader x-uscspam
899
900## EOF
Starting the service
Now we can start dspam(1) with:
1sudo service dspam start
You can check that its running with ps(1) (but don’t show all the PostgreSQL connections):
1[louisk@mx louisk 101 ]$ ps ax | grep dspam | grep -v postgres
2 742 v0- S 0:00.03 /usr/local/bin/dspam --daemon
3[louisk@mx louisk 102 ]$
You can also use sockstat(1) to show which sockets dspam is listening on:
1[louisk@mx louisk 102 ]$ sockstat -4l | grep dspam
2root dspam 742 6 tcp4 127.0.0.1:10024 *:*
3[louisk@mx louisk 103 ]$
Training
Periodically, you will want to clean up the dspam database. Helpfully, there is a purge.sql script in ‘/usr/local/share/examples/dspam/pgsql’. I put a wrapper around it and drove it from cron:
1@daily root /usr/local/bin/dspam-cleanup.sh
1cat /usr/local/bin/dspam-cleanup.sh
2#!/bin/sh
3#
4# run sql queries to clean old/unused tokens out of PostgreSQL
5/usr/local/bin/psql -U pgsql dspam < /usr/local/etc/dspam-purge.sql
6# clean old logs
7/usr/local/bin/dspam_logrotate -a 30 -d /var/db/dspam/data
Testing
If you need to download some spam (do you want to do this very often?), you can get some here .
1cat /path/to/mail_corpus_file | dspamc --client --user user@dom.tld --class=spam --source=corpus --deliver=summary
2X-DSPAM-Result: user@dom.tld; result="Spam"; class="Spam"; probability=1.0000; confidence=1.00; signature=N/A
If you ask dspam to classify this message again, it may still consider it “innocent”, but now it won’t be as confident. Compare the confidence number to the output from before training the message:
1cat /path/to/mail_corpus_file | dspam --user user@dom.tld --classify
2X-DSPAM-Result: user@dom.tld; result="Innocent"; class="Innocent"; probability=0.7127; confidence=0.75; signature=N/A
If you need to enable debugging (dspam must be built with debug option), you can do
1sudo sysrc dspam_debug=YES
Logs are in /var/log/dspam. Debug knobs are in dspam.conf.
Configuring Dovecot
Dovecot is a secure and scalable POP/IMAP server. It supports the relevant internet standards, offers storage via maildir, and is quite responsive under load. Dovecot’s job is to accept mail from Postfix, and cause it to be available when users connect via POP or IMAP. Documentation for Dovecot .
I’m only providing the output from ‘doveconf -n’. You can either put these directives into a dovecot.conf file, or you can put them into the more modern conf.d/*.conf files. The outcome is the same, and ‘doveconf -n’ output should also be the same. Should you wish to compare, you can put the bits into the conf.d/*.conf files, copy them all into a dovecot.conf file, start dovecot, and do a:
1diff -u `doveconf -n` dovecot.conf
If things are correct, you should get a prompt back and no differences listed.
Here is the output from ‘doveconf -n’:
1# 2.2.27 (c0f36b0): /usr/local/etc/dovecot/dovecot.conf
2# Pigeonhole version 0.4.16 (fed8554)
3# OS: FreeBSD 11.0-RELEASE-p2 amd64
4auth_socket_path = /var/run/dovecot/auth-userdb
5debug_log_path = /var/log/dovecot-debug.log
6first_valid_gid = 125
7first_valid_uid = 125
8hostname = mail.cryptomonkeys.org
9imap_client_workarounds = delay-newmail tb-extra-mailbox-sep
10last_valid_gid = 125
11last_valid_uid = 125
12lmtp_save_to_detail_mailbox = yes
13mail_gid = 125
14mail_home = /usr/local/virtual/%d/%n/
15mail_location = maildir:~/Maildir
16mail_privileged_group = postfix
17mail_uid = 125
18mailbox_list_index = yes
19managesieve_notify_capability = mailto
20managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart
21namespace inbox {
22 inbox = yes
23 location =
24 mailbox All {
25 auto = subscribe
26 special_use = \All
27 }
28 mailbox Archive {
29 auto = subscribe
30 special_use = \Archive
31 }
32 mailbox Drafts {
33 auto = subscribe
34 special_use = \Drafts
35 }
36 mailbox Flagged {
37 auto = subscribe
38 special_use = \Flagged
39 }
40 mailbox Junk {
41 auto = create
42 autoexpunge = 30 days
43 special_use = \Junk
44 }
45 mailbox Notes {
46 auto = subscribe
47 }
48 mailbox Sent {
49 auto = subscribe
50 special_use = \Sent
51 }
52 mailbox Trash {
53 auto = subscribe
54 special_use = \Trash
55 }
56 prefix =
57}
58passdb {
59 args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
60 driver = sql
61}
62plugin {
63 home = /usr/local/virtual/%d/%n/
64 mailbox_alias_new = Sent Messages
65 mailbox_alias_new2 = Sent Items
66 mailbox_alias_new3 = Deleted Items
67 mailbox_alias_old = Sent
68 mailbox_alias_old2 = Sent
69 mailbox_alias_old3 = Trash
70 mailbox_alias_old4 = Deleted Messages
71 sieve = /usr/local/virtual/%d/%n/dovecot.sieve
72 sieve_dir = /usr/local/virtual/%d/%n/
73 sieve_global_dir = /usr/local/virtual/sieve
74 sieve_global_path = /usr/local/virtual/sieve/globalfilter.sieve
75 stats_refresh = 30 secs
76 stats_track_cmds = yes
77}
78pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
79pop3_enable_last = yes
80postmaster_address = postmaster@cryptomonkeys.org
81protocols = imap pop3 lmtp sieve
82quota_full_tempfail = yes
83sendmail_path = /usr/local/sbin/sendmail
84service auth {
85 unix_listener /var/spool/postfix/private/auth {
86 group = postfix
87 mode = 0666
88 user = postfix
89 }
90 unix_listener auth-userdb {
91 group = postfix
92 mode = 0777
93 user = postfix
94 }
95}
96service imap-login {
97 client_limit = 10
98 inet_listener imap {
99 port = 0
100 }
101 inet_listener imaps {
102 port = 993
103 ssl = yes
104 }
105 process_min_avail = 4
106 service_count = 1
107 vsz_limit = 256 M
108}
109service lmtp {
110 inet_listener lmtp {
111 address = 127.0.0.1 ::1
112 port = 24
113 }
114 user = postfix
115}
116service managesieve-login {
117 inet_listener sieve {
118 port = 4190
119 }
120 process_min_avail = 0
121 service_count = 1
122 vsz_limit = 64 M
123}
124service managesieve {
125 process_limit = 1024
126}
127service pop3-login {
128 client_limit = 10
129 inet_listener pop3 {
130 port = 0
131 }
132 inet_listener pop3s {
133 port = 995
134 ssl = yes
135 }
136 process_limit = 128
137 process_min_avail = 2
138 service_count = 1
139 vsz_limit = 256 M
140}
141service stats {
142 fifo_listener stats-mail {
143 mode = 0600
144 user = postfix
145 }
146}
147ssl_ca = </usr/local/etc/ssl/cacert.pem
148ssl_cert = </usr/local/etc/ssl/server.crt
149ssl_cipher_list = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4
150ssl_dh_parameters_length = 2048
151ssl_key = # hidden, use -P to show it
152ssl_parameters_regenerate = 1 weeks
153ssl_prefer_server_ciphers = yes
154ssl_protocols = !SSLv2 !SSLv3
155userdb {
156 args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
157 driver = sql
158}
159protocol lmtp {
160 mail_plugins = " sieve"
161}
162protocol lda {
163 mail_plugins = " sieve"
164}
165protocol imap {
166 mail_max_userip_connections = 20
167 mail_plugins = " stats expire mail_log notify quota trash imap_quota"
168}
169protocol pop3 {
170 mail_max_userip_connections = 10
171}
Tightening things up a little
I’ve tightened security a little from the defaults. I’ve removed the SSLv2, and SSLv3 capabilities completely. I’ve Limited the ciphers to only those on the HIGH list. If you want to know more, check out the openssl docs on ciphers . Additionally, I added a dhparam with a length of 2048 bits. Dovecot generates this key the first time you start it after adding this directive, so you don’t have to worry about having a dhparam that is shared with anybody else. Probability is very low.
1# This file is commonly accessed via passdb {} or userdb {} section in
2# conf.d/auth-sql.conf.ext
3
4driver = pgsql
5connect = host=localhost dbname=postfix user=postfix password=password
6default_pass_scheme = MD5
7user_query = \
8 SELECT '/usr/local/virtual/'||maildir AS home, '*:bytes='||quota AS quota_rule \
9 FROM mailbox WHERE username = '%u' AND active = TRUE
10password_query = \
11 SELECT '/usr/local/virtual/'||maildir AS userdb_home, \
12 username AS user, password, '*:bytes='||quota AS userdb_quota_rule \
13 FROM mailbox WHERE username = '%u' AND active = TRUE
14iterate_query = SELECT username AS user FROM mailbox
Start up dovecot
With all this in place, we should be able to add the
1sudo sysrc dovecot_enable=YES
entry to /etc/rc.conf, and then do:
1sudo service dovecot start
We can check our success with ps(1):
1[root@mx etc 122 ]$ ps ax | grep dove
2 3641 - Is 0:00.03 /usr/local/sbin/dovecot -c /usr/local/etc/dovecot/dovecot.conf
3 3642 - I 0:00.02 dovecot/pop3-login
4 3643 - I 0:00.01 dovecot/imap-login
5 3644 - I 0:00.00 dovecot/anvil
6 3645 - I 0:00.01 dovecot/log
7 3647 - I 0:00.03 dovecot/pop3-login
8 3648 - I 0:00.01 dovecot/imap-login
9 3649 - I 0:00.09 dovecot/imap-login
10 3650 - I 0:00.00 dovecot/imap-login
11 3651 - I 0:00.08 dovecot/config
12 3653 - I 0:00.02 dovecot/auth
13 3657 - I 0:00.00 dovecot/ssl-params
14[root@mx etc 123 ]$
and with sockstat(1) (both IPv4 and IPv6, if you don’t want to check IPv6, omit the ‘6’):
1[root@mx dovecot 120 ]$ sockstat -46l | grep dove
2dovenull imap-login 2733 7 tcp4 *:993 *:*
3dovenull imap-login 2733 8 tcp6 *:993 *:*
4dovenull imap-login 2732 7 tcp4 *:993 *:*
5dovenull imap-login 2732 8 tcp6 *:993 *:*
6dovenull imap-login 2731 7 tcp4 *:993 *:*
7dovenull imap-login 2731 8 tcp6 *:993 *:*
8dovenull pop3-login 2729 7 tcp4 *:995 *:*
9dovenull pop3-login 2729 8 tcp6 *:995 *:*
10dovenull imap-login 2725 7 tcp4 *:993 *:*
11dovenull imap-login 2725 8 tcp6 *:993 *:*
12dovenull pop3-login 2724 7 tcp4 *:995 *:*
13dovenull pop3-login 2724 8 tcp6 *:995 *:*
14root dovecot 2723 15 tcp4 *:4190 *:*
15root dovecot 2723 16 tcp6 *:4190 *:*
16root dovecot 2723 26 tcp4 *:995 *:*
17root dovecot 2723 27 tcp6 *:995 *:*
18root dovecot 2723 40 tcp4 *:993 *:*
19root dovecot 2723 41 tcp6 *:993 *:*
20[root@mx dovecot 121 ]$
Configuring Postfix
Postfix is a popular and secure SMTP server. Its extensible, flexible, and light on resources. Many people like it because its config file isn’t written in M4. Documentation for Postfix here .
Originally I’d used v2 because it appeared that some of the patches necessary weren’t available for v3, but it turns out those patches are not necessary with this architecture, so yey!
Enable postfix (and disable sendmail) with sysrc.
1sudo sysrc postfix_enable=YES
2sudo sysrc sendmail_enable=NONE
Disable the sendmail periodic scripts. If /etc/periodic.conf doesn’t exist, create it and add the following contents:
1daily_clean_hoststat_enable="NO"
2daily_status_mail_rejects_enable="NO"
3daily_status_include_submit_mailq="NO"
4daily_submit_queuerun="NO"
Most of the config directives will go in the main.cf (/usr/local/etc/postfix). When you dump the config with postconf, those are the settings it looks at.
Here is the output from ‘postconf -n’ (it can be dumped straight into main.cf if you wish):
1alias_maps = hash:${config_directory}/aliases
2allow_mail_to_commands = alias
3body_checks = pcre:${config_directory}/body_checks
4broken_sasl_auth_clients = yes
5command_directory = /usr/local/sbin
6compatibility_level = 2
7config_directory = /usr/local/etc/postfix
8daemon_directory = /usr/local/libexec/postfix
9data_directory = /var/db/postfix
10delay_warning_time = 5d
11disable_vrfy_command = yes
12header_checks = pcre:${config_directory}/header_checks
13html_directory = /usr/local/share/doc/postfix
14inet_protocols = ipv4, ipv6
15lmtp_header_checks = pcre:${config_directory}/lmtp_header_checks
16mail_owner = postfix
17mailbox_size_limit = 536870912
18mailq_path = /usr/local/bin/mailq
19manpage_directory = /usr/local/man
20maximal_queue_lifetime = 8d
21message_size_limit = 33554432
22mydestination = $myorigin, localhost.$mydomain, localhost
23mydomain = cryptomonkeys.com
24myhostname = mail.cryptomonkeys.com
25mynetworks = 192.168.0.0/20, 127.0.0.0/8, [::1]/128, [fe80::]/10
26myorigin = $myhostname
27nested_header_checks =
28newaliases_path = /usr/local/bin/newaliases
29postscreen_access_list = permit_mynetworks, cidr:$config_directory/postscreen_access.cidr
30postscreen_bare_newline_action = ignore
31postscreen_bare_newline_enable = yes
32postscreen_bare_newline_ttl = 30d
33postscreen_blacklist_action = drop
34postscreen_cache_cleanup_interval = 12h
35postscreen_cache_map = btree:$data_directory/postscreen_cache
36postscreen_cache_retention_time = 7d
37postscreen_disable_vrfy_command = $disable_vrfy_command
38postscreen_dnsbl_action = enforce
39postscreen_dnsbl_reply_map = pcre:$config_directory/postscreen_dnsbl_reply_map.pcre
40postscreen_dnsbl_sites = zen.spamhaus.org*3 b.barracudacentral.org*2 bl.spameatingmonkey.net*2 bl.spamcop.net dnsbl.sorbs.net psbl.surriel.com bl.mailspike.net swl.spamhaus.org*-4 list.dnswl.org=127.[0..255].[0..255].0*-2 list.dnswl.org=127.[0..255].[0..255].1*-3 list.dnswl.org=127.[0..255].[0..255].[2..255]*-4
41postscreen_dnsbl_threshold = 3
42postscreen_dnsbl_ttl = 1h
43postscreen_dnsbl_whitelist_threshold = -1
44postscreen_greet_action = enforce
45postscreen_greet_banner = $smtpd_banner
46postscreen_greet_ttl = 1d
47postscreen_greet_wait = ${stress?2}${stress:6}s
48postscreen_helo_required = $smtpd_helo_required
49postscreen_non_smtp_command_action = drop
50postscreen_non_smtp_command_enable = yes
51postscreen_non_smtp_command_ttl = 30d
52postscreen_pipelining_enable = yes
53postscreen_whitelist_interfaces = static:all
54proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $virtual_mailbox_limit_maps
55proxy_write_maps = $smtp_sasl_auth_cache_name $lmtp_sasl_auth_cache_name $address_verify_map $postscreen_cache_map
56queue_directory = /var/spool/postfix
57readme_directory = /usr/local/share/doc/postfix
58recipient_delimiter = +
59relay_domains = proxy:pgsql:${config_directory}/pgsql_relay_domains_maps.cf
60sample_directory = /usr/local/etc/postfix
61sendmail_path = /usr/local/sbin/sendmail
62setgid_group = maildrop
63smtp_destination_concurrency_limit = 20
64smtp_header_checks = pcre:${config_directory}/smtp_header_checks
65smtp_helo_timeout = 30s
66smtp_sasl_auth_enable = no
67smtp_tls_loglevel = 1
68smtp_tls_mandatory_protocols = !SSLv2, !SSLv3
69smtp_tls_note_starttls_offer = yes
70smtp_tls_policy_maps = hash:${config_directory}/tls_policy
71smtp_tls_protocols = !SSLv2, !SSLv3
72smtp_tls_security_level = may
73smtp_use_tls = yes
74smtpd_banner = $myhostname ESMTP Sendmail 8.10 (Solaris 2.6)
75smtpd_client_connection_count_limit = 5
76smtpd_client_connection_rate_limit = 10
77smtpd_data_restrictions = reject_unauth_pipelining, reject_multi_recipient_bounce, permit
78smtpd_end_of_data_restrictions = check_policy_service inet:127.0.0.1:10031
79smtpd_error_sleep_time = 0
80smtpd_hard_error_limit = 10
81smtpd_helo_required = yes
82smtpd_recipient_limit = 50
83smtpd_recipient_restrictions = permit_dnswl_client list.dnswl.org=127.0.[0..255].[1..3], permit_sasl_authenticated, permit_mynetworks, permit_auth_destination, check_policy_service inet:127.0.0.1:10031, check_client_access hash:${config_directory}/access, check_recipient_access hash:${config_directory}/recipient_access, check_recipient_access hash:${config_directory}/sender_access, warn_if_reject reject_unknown_recipient_domain, warn_if_reject reject_non_fqdn_recipient, warn_if_reject reject_unknown_client, warn_if_reject reject_non_fqdn_sender, warn_if_reject reject_non_fqdn_hostname, reject_unverified_recipient, reject_invalid_hostname, reject_unknown_recipient_domain, reject_unlisted_recipient, reject_unverified_recipient, reject_unknown_hostname, reject_unauth_destination, permit
84smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
85smtpd_sasl_auth_enable = yes
86smtpd_sasl_authenticated_header = no
87smtpd_sasl_local_domain = $myhostname
88smtpd_sasl_path = private/auth
89smtpd_sasl_security_options = noanonymous
90smtpd_sasl_type = dovecot
91smtpd_sender_restrictions = permit_sasl_authenticated, permit_mynetworks, check_client_access hash:${config_directory}/access, pcre:/usr/local/etc/postfix/rejected_domains, warn_if_reject reject_non_fqdn_sender, reject_unknown_client, reject_unknown_sender_domain, permit
92smtpd_soft_error_limit = 5
93smtpd_timeout = 30s
94smtpd_tls_CAfile = /usr/local/etc/ssl/cacert.pem
95smtpd_tls_auth_only = yes
96smtpd_tls_cert_file = /usr/local/etc/ssl/server.crt
97smtpd_tls_dh1024_param_file = /usr/local/etc/ssl/dh2048.pem
98smtpd_tls_dh512_param_file = /usr/local/etc/ssl/dh512.pem
99smtpd_tls_eecdh_grade = strong
100smtpd_tls_key_file = /usr/local/etc/ssl/server.key
101smtpd_tls_loglevel = 1
102smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
103smtpd_tls_protocols = !SSLv2, !SSLv3
104smtpd_tls_received_header = yes
105smtpd_tls_security_level = may
106smtpd_tls_session_cache_timeout = 3600s
107smtpd_use_tls = yes
108tls_eecdh_strong_curve = prime256v1
109tls_eecdh_ultra_curve = secp384r1
110tls_random_source = dev:/dev/random
111transport_maps = hash:${config_directory}/transport
112unknown_local_recipient_reject_code = 550
113virtual_alias_maps = proxy:pgsql:${config_directory}/pgsql_virtual_alias_maps.cf
114virtual_gid_maps = static:125
115virtual_mailbox_base = /usr/local/virtual
116virtual_mailbox_domains = proxy:pgsql:${config_directory}/pgsql_virtual_domains_maps.cf
117virtual_mailbox_limit = 536870912
118virtual_mailbox_limit_maps = proxy:pgsql:${config_directory}/pgsql_virtual_mailbox_limit_maps.cf
119virtual_mailbox_maps = proxy:pgsql:${config_directory}/pgsql_virtual_mailbox_maps.cf
120virtual_minimum_uid = 125
121virtual_transport = lmtp:127.0.0.1:2424
122virtual_uid_maps = static:125
I’ve done a number of things, picked up over the years. A few things of note:
- Again, I’ve disabled SSLv2, and SSLv3 for both SMTP and SMTPD (handled separately).
- All the ‘proxy:pgsql’ statements are where postfix will do lookups against information in our PostgreSQL database (things like domains, email addresses, aliases, etc.
- Postscreen is setup to query a number of DNSBL (DNS Black List) and then weigh the answer accordingly. If there are too many hits, the connection is rejected. THis is nice because it keeps a single DNSBL from blocking a connection (sometimes a server is added to a list mistakenly, or because there was a problem, but its been fixed). if you’re not familiar with Postscreen, you can read about it here
The master.cf (also in /usr/local/etc/postfix) controls where and how postfix communicates with other programs. For example, if you want it to listen on a certain port, such as TCP/465, you need to define the ‘smtps’ entry you see below.
1# master.cf
2smtpd pass - - n - - smtpd
3
4submission inet n - n - - smtpd
5 -o smtpd_tls_security_level=encrypt
6 -o smtpd_enforce_tls=yes
7 -o smtpd_sasl_auth_enable=yes
8 -o smtpd_reject_unlisted_recipient=no
9 -o smtpd_helo_restrictions=
10 -o smtpd_sender_restrictions=
11 -o smtpd_client_restrictions=permit_sasl_authenticated,reject
12 -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject_non_fqdn_recipient,reject_unknown_recipient_domain,reject
13 -o milter_macro_daemon_name=ORIGINATING
14
15smtps inet n - n - - smtpd
16 -o smtpd_tls_wrappermode=yes
17 -o smtpd_sasl_auth_enable=yes
18 -o smtpd_client_restrictions=permit_sasl_authenticated,reject
19 -o milter_macro_daemon_name=ORIGINATING
20
21127.0.0.1:10026 inet n - n - - smtpd
22 -o content_filter=
23 -o receive_override_options=no_address_mappings,no_unknown_recipient_checks,no_header_body_checks
24 -o smtpd_helo_restrictions=
25 -o smtpd_client_restrictions=
26 -o smtpd_sender_restrictions=
27 -o smtpd_recipient_restrictions=permit_mynetworks,reject
28 -o mynetworks=127.0.0.0/8
29 -o smtpd_authorized_xforward_hosts=127.0.0.0/8
30
31smtp-amavis unix - - - - 2 smtp
32 -o smtp_data_done_timeout=1200
33 -o smtp_send_xforward_command=yes
34 -o disable_dns_lookups=yes
35 -o max_use=20
36
37127.0.0.1:10025 inet n - - - - smtpd
38 -o content_filter=
39 -o local_recipient_maps=
40 -o relay_recipient_maps=
41 -o smtpd_restriction_classes=
42 -o smtpd_delay_reject=no
43 -o smtpd_client_restrictions=permit_mynetworks,reject
44 -o smtpd_helo_restrictions=
45 -o smtpd_sender_restrictions=
46 -o smtpd_recipient_restrictions=permit_mynetworks,reject
47 -o smtpd_data_restrictions=reject_unauth_pipelining
48 -o smtpd_end_of_data_restrictions=
49 -o mynetworks=127.0.0.0/8
50 -o smtpd_error_sleep_time=0
51 -o smtpd_soft_error_limit=1001
52 -o smtpd_hard_error_limit=1000
53 -o smtpd_client_connection_count_limit=0
54 -o smtpd_client_connection_rate_limit=0
55 -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks
56
57pickup unix n - n 60 1 pickup
58 -o content_filter=
59 -o receive_override_options=no_header_body_checks
60cleanup unix n - n - 0 cleanup
61qmgr unix n - n 300 1 qmgr
62tlsmgr unix - - n 1000? 1 tlsmgr
63rewrite unix - - n - - trivial-rewrite
64bounce unix - - n - 0 bounce
65defer unix - - n - 0 bounce
66trace unix - - n - 0 bounce
67verify unix - - n - 1 verify
68flush unix n - n 1000? 0 flush
69proxymap unix - - n - - proxymap
70proxywrite unix - - n - 1 proxymap
71smtp unix - - n - - smtp
72relay unix - - n - - smtp
73showq unix n - n - - showq
74error unix - - n - - error
75retry unix - - n - - error
76discard unix - - n - - discard
77local unix - n n - - local
78virtual unix - n n - - virtual
79lmtp unix - - n - - lmtp
80anvil unix - - n - 1 anvil
81scache unix - - n - 1 scache
82smtp inet n - n - 1 postscreen
83dnsblog unix - - n - 0 dnsblog
84tlsproxy unix - - n - 0 tlsproxy
85# dspam retraining
86dspam-retrain unix - n n - - pipe
87 flags=Rhq
88 user=dspam
89 argv=/usr/local/bin/dspam --client --mode=teft --class=$nexthop --source=error --user ${sender}
Of note, I added the last couple lines so that mail could be forwarded to specific addresses and it would be passed directly to dspam for training.
Also, I’ve ensured that people who authenticate aren’t subjected to the DNSBL checks, because mobile users aren’t in control of the network they’re on and shouldn’t be subjected to its reputation (or lack there of).
I then added these entries to the transport file (hash with ‘postmap transport’). This allows anybody who is allowed to send mail through the system to submit (as an attachment) spam or ham to be reclassified by dspam.
1spam@spam.spam dspam-retrain:spam
2ham@ham.ham dspam-retrain:innocent
I want to strip out some headers as things pass through, including some of the authenticated user headers (don’t need to advertise that to the world), as well as the spam/virus scanner headers. It doesn’t matter if somebody else has scanned it or not, it all gets scanned locally.
1# We use the header_checks file to remove some headers that we find undesirable.
2# Return receipts and software versions are the most significant in this
3# situation.
4
5/^Received: from 127.0.0.1/ IGNORE
6/^X-Authenticated-Sender:/ IGNORE
7/^User-Agent:/ IGNORE
8/^X-Mailer:/ IGNORE
9/^X-MimeOLE:/ IGNORE
10/^X-MSMail-Priority:/ IGNORE
11/^X-Originating-IP:/ IGNORE
12/^X-Sanitizer:/ IGNORE
13/^X-Spam-Status:/ IGNORE
14/^X-Spam-Level:/ IGNORE
15/^X-Virus-Check:/ IGNORE
16/^X-Virus-Checked:/ IGNORE
17/^X-Virus-Scan:/ IGNORE
18/^X-Virus-Scanned:/ IGNORE
19/^X-Virus-Status:/ IGNORE
20/^(X-DSPAM-.*)/ IGNORE
These files control how Postfix looks up data in our PostgreSQL database. Each one contains the user, password, database, host, and query to be executed. These queries return a set of information that tells postfix how to behave when its accepting or sending messages (postfix doesn’t deliver messages with this setup, that’s left to dovecot.
1# pgsql_relay_domains_maps.cf
2user = postfix
3password = password
4hosts = localhost
5dbname = postfix
6query = SELECT domain FROM domain WHERE domain='%s' and backupmx = true
1# pgsql_virtual_alias_maps.cf
2user = postfix
3password = password
4hosts = localhost
5dbname = postfix
6query = SELECT goto FROM alias WHERE address='%s' AND active = true
1# pgsql_virtual_domains_maps.cf
2user = postfix
3password = password
4hosts = localhost
5dbname = postfix
6query = SELECT domain FROM domain WHERE domain='%s' and backupmx = false and active = true
1# pgsql_virtual_mailbox_limit_maps.cf
2user = postfix
3password = password
4hosts = localhost
5dbname = postfix
6query = SELECT quota FROM mailbox WHERE username='%s'
1# pgsql_virtual_mailbox_maps.cf
2user = postfix
3password = password
4hosts = localhost
5dbname = postfix
6query = SELECT maildir FROM mailbox WHERE username='%s' AND active = true
Starting up postfix
You should now be able to start postfix with:
1service postfix start
and check that its running with:
1[louisk@mail postfix 16 ]$ sockstat -46l | grep master
2root master 2985 17 tcp4 *:587 *:*
3root master 2985 18 tcp6 *:587 *:*
4root master 2985 21 tcp4 *:465 *:*
5root master 2985 22 tcp6 *:465 *:*
6root master 2985 25 tcp4 127.0.0.1:10026 *:*
7root master 2985 31 tcp4 127.0.0.1:10025 *:*
8root master 2985 105 tcp4 *:25 *:*
9root master 2985 106 tcp6 *:25 *:*
10[louisk@mail postfix 17 ]$
We see postfix listening on all the ports we want.
Comments