Project

General

Profile

Amazon SES for CiviCRM » History » Version 19

Joseph Lacey, 08/09/2016 12:57 PM

1 1 Jon Goldberg
h1. Amazon SES for CiviCRM
2 1 Jon Goldberg
3 2 Jon Goldberg
_Pre-requisites: DNS access_
4 2 Jon Goldberg
5 17 Jon Goldberg
* Have client sign up for Amazon AWS (at https://aws.amazon.com), which involves credit cards, identity verification, etc.
6 2 Jon Goldberg
* Ideally, have them click on "SES" at the AWS console, to do the phone verification.
7 2 Jon Goldberg
* Log onto AWS, select SES.  Note that you only have sandbox access at this time.
8 2 Jon Goldberg
* Click on "SMTP Settings" on the left, record the server info, generate SMTP credentials and record those too.
9 2 Jon Goldberg
* Click on "Verified Senders: Domains".  Click "Verify a new Domain".  Enter the client's domain name; also click "Generate DKIM settings".
10 2 Jon Goldberg
* Download these credentials, don't just copy/paste from the screen, because they tend to get cut off.
11 2 Jon Goldberg
* Update the DNS with the verification credentials.
12 5 Jon Goldberg
** *NOTE: This will take several hours (3-4?), even after DNS has propagated.*
13 3 Jon Goldberg
* Also add your own e-mail address the the verified sender - email address list.  This just requires receiving a verification email. You need this because while you're still in sandbox mode, you can only send to verified addresses.  Note that if your normal mail account has greylisting, you'll probably want to use an alternate account here, like GMail.
14 4 Jon Goldberg
15 4 Jon Goldberg
You now have two options for SMTP setup:
16 10 Jon Goldberg
17 10 Jon Goldberg
h3. SMTP direct from CiviCRM
18 10 Jon Goldberg
19 16 Joseph Lacey
* Set up the SMTP Outbound settings in CiviCRM to use the Amazon SMTP server.  
20 16 Joseph Lacey
** Without changing the core Net package, prefix the SMTP server with @ssl://@ and select a port of @465@.
21 16 Joseph Lacey
** If however the host doesn't allow mail to be sent over port 465, "patch the core file package/Net/SMTP.php":https://github.com/civicrm/civicrm-packages/pull/66/files and don't use the ssl:// prefix.  For more information, "see this post and the posted links":http://civicrm.stackexchange.com/questions/377/how-do-i-configure-smtp-with-starttls.
22 1 Jon Goldberg
* Click "Save and Test".
23 4 Jon Goldberg
* You may not see an error on the page - so check the ConfigAndLog log for error messages if necessary.
24 1 Jon Goldberg
25 10 Jon Goldberg
h3. SMTP relayed through Postfix (recommended)
26 4 Jon Goldberg
27 14 Jon Goldberg
See also here: http://docs.aws.amazon.com/ses/latest/DeveloperGuide/postfix.html
28 14 Jon Goldberg
29 10 Jon Goldberg
Postfix will give higher performance on large mailings, and also does a much better job of handling resends, greylisting, etc.  This should be implemented wherever possible.
30 1 Jon Goldberg
31 11 Jon Goldberg
32 1 Jon Goldberg
* Install Postfix.
33 11 Jon Goldberg
* Paste this at the bottom of @/etc/postfix/main.cf@:
34 12 Jon Goldberg
<pre>
35 11 Jon Goldberg
#jon@palantetech.coop SES setup
36 11 Jon Goldberg
relayhost = email-smtp.us-west-1.amazonaws.com:25
37 11 Jon Goldberg
smtp_sasl_auth_enable = yes
38 11 Jon Goldberg
smtp_sasl_security_options = noanonymous
39 11 Jon Goldberg
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
40 11 Jon Goldberg
smtp_use_tls = yes
41 11 Jon Goldberg
smtp_tls_security_level = encrypt
42 1 Jon Goldberg
smtp_tls_note_starttls_offer = yes
43 14 Jon Goldberg
#This last line isn't needed, I'm pretty sure
44 14 Jon Goldberg
#smtp_tls_CAfile = /etc/ssl/certs/ca-bundle.crt
45 11 Jon Goldberg
46 11 Jon Goldberg
#jon@palantetech.coop rate limiting to 4 messages/second for now
47 11 Jon Goldberg
smtp_destination_rate_delay = 1s
48 11 Jon Goldberg
smtp_destination_concurrency_limit = 4
49 12 Jon Goldberg
</pre>
50 12 Jon Goldberg
51 12 Jon Goldberg
* Create a file with credentials at @/etc/postfix/sasl_passwd@ in the format:
52 12 Jon Goldberg
<pre>
53 12 Jon Goldberg
email-smtp.us-east-1.amazonaws.com:25 username:password
54 12 Jon Goldberg
</pre>
55 13 Jon Goldberg
* Run @postmap /etc/postfix/sasl_passwd@.
56 15 Jon Goldberg
* Configure CiviCRM's SMTP settings to either a) point to localhost, or b) use mail().
57 10 Jon Goldberg
58 10 Jon Goldberg
h2. Bounce processing setup
59 10 Jon Goldberg
60 18 Jon Goldberg
There's code here to use a Amazon SNS to a bounce processing address instead of the typical bounce processing "here":https://etopian.com/blog/civicrm-bounce-processing/.  I have NOT tested this, but I don't think this provides an advantage over typical bounce processing.  An SNS endpoint in CiviCRM COULD provide higher performance than IMAP/POP-based bounce processing, I don't think this solution is it.
61 18 Jon Goldberg
62 18 Jon Goldberg
So instead, just see [[Set up local bounce processing]].
63 10 Jon Goldberg
64 10 Jon Goldberg
h2. Testing
65 10 Jon Goldberg
66 7 Jon Goldberg
* To test bounce processing, set up a test group made up of at least one verified address and a test contact with the e-mail address "bounce@simulator.amazonses.com".
67 1 Jon Goldberg
* Send a mailing to that group.  You can speed up testing by manually running the "Send Scheduled Mailings" and "Fetch Bounces" scheduled jobs.
68 1 Jon Goldberg
* Don't forget to request production access when you're done!
69 19 Joseph Lacey
70 19 Joseph Lacey
h3. Troubleshooting
71 19 Joseph Lacey
72 19 Joseph Lacey
In testing if CiviCRM reports that everything's been sent correctly, but then no emails are received, the problem could be with Postfix authenticating to SES.  /var/log/mail.log might contain entries like this.
73 19 Joseph Lacey
74 19 Joseph Lacey
<pre>
75 19 Joseph Lacey
Aug  9 11:37:48 XXXXXXXX postfix/smtpd[24665]: connect from localhost[127.0.0.1]
76 19 Joseph Lacey
Aug  9 11:37:48 XXXXXXXX postfix/smtpd[24665]: A30A4624F1: client=localhost[127.0.0.1]
77 19 Joseph Lacey
Aug  9 11:37:48 XXXXXXXX postfix/cleanup[24669]: A30A4624F1: message-id=<20160809163748.A30A4624F1@XXXXXXXX.example.org>
78 19 Joseph Lacey
Aug  9 11:37:48 XXXXXXXX postfix/qmgr[24664]: A30A4624F1: from=<XXXXXXXX@example.org>, size=453, nrcpt=1 (queue active)
79 19 Joseph Lacey
Aug  9 11:37:48 XXXXXXXX postfix/smtpd[24665]: disconnect from localhost[127.0.0.1]
80 19 Joseph Lacey
Aug  9 11:37:48 XXXXXXXX postfix/smtp[24670]: warning: SASL authentication failure: No worthy mechs found
81 19 Joseph Lacey
Aug  9 11:37:48 XXXXXXXX postfix/smtp[24670]: A30A4624F1: SASL authentication failed; cannot authenticate to server email-smtp.us-west-2.amazonaws.com[54.68.106.242]: no mechanism available
82 19 Joseph Lacey
Aug  9 11:37:48 XXXXXXXX postfix/smtp[24670]: warning: SASL authentication failure: No worthy mechs found
83 19 Joseph Lacey
Aug  9 11:37:48 XXXXXXXX postfix/smtp[24670]: A30A4624F1: SASL authentication failed; cannot authenticate to server email-smtp.us-west-2.amazonaws.com[52.35.58.187]: no mechanism available
84 19 Joseph Lacey
Aug  9 11:37:49 XXXXXXXX postfix/smtp[24670]: warning: SASL authentication failure: No worthy mechs found
85 19 Joseph Lacey
Aug  9 11:37:49 XXXXXXXX postfix/smtp[24670]: A30A4624F1: to=<XXXXXXXX@example.com>, relay=email-smtp.us-west-2.amazonaws.com[52.35.228.26]:25, delay=0.39, delays=0.04/0.01/0.34/0, dsn=4.7.0, status=deferred (SASL authentication failed; cannot authenticate to server email-smtp.us-west-2.amazonaws.com[52.35.228.26]: no mechanism available)
86 19 Joseph Lacey
</pre>
87 19 Joseph Lacey
88 19 Joseph Lacey
If that's the case, you might need to install some additional libraries, namely the @libsasl2-modules@ package in Debian/Ubuntu systems.
89 6 Jon Goldberg
90 10 Jon Goldberg
h3. SPECIAL NOTE FOR EC2 INSTANCES
91 10 Jon Goldberg
92 9 Jon Goldberg
EC2 severely throttles outgoing mail on ports 25/465/587.  You need to submit a request to Amazon to have this throttle lifted.  This is separate from any user-configurable firewalling!  You can do that (and set up PTR) here: https://portal.aws.amazon.com/gp/aws/html-forms-controller/contactus/ec2-email-limit-rdns-request
Go to top
Add picture from clipboard (Maximum size: 40 MB)