PT Commons

h1. Amazon SES for CiviCRM

_Pre-requisites: DNS access_

* Have client sign up for Amazon AWS (at https://aws.amazon.com), which involves credit cards, identity verification, etc.

* Ideally, have them click on "SES" at the AWS console, to do the phone verification.

* Log onto AWS, select SES.  Note that you only have sandbox access at this time.

* Click on "SMTP Settings" on the left, record the server info, generate SMTP credentials and record those too.

* Click on "Verified Senders: Domains".  Click "Verify a new Domain".  Enter the client's domain name; also click "Generate DKIM settings".

* Download these credentials, don't just copy/paste from the screen, because they tend to get cut off.

* Update the DNS with the verification credentials.

** *NOTE: This will take several hours (3-4?), even after DNS has propagated.*

* Also add your own e-mail address the the verified sender - email address list.  This just requires receiving a verification email. You need this because while you're still in sandbox mode, you can only send to verified addresses.  Note that if your normal mail account has greylisting, you'll probably want to use an alternate account here, like GMail.

You now have two options for SMTP setup:

h3. SMTP direct from CiviCRM

* Set up the SMTP Outbound settings in CiviCRM to use the Amazon SMTP server.  

** Without changing the core Net package, prefix the SMTP server with @ssl://@ and select a port of @465@.

** If however the host doesn't allow mail to be sent over port 465, "patch the core file package/Net/SMTP.php":https://github.com/civicrm/civicrm-packages/pull/66/files and don't use the ssl:// prefix.  For more information, "see this post and the posted links":http://civicrm.stackexchange.com/questions/377/how-do-i-configure-smtp-with-starttls.

* Click "Save and Test".

* You may not see an error on the page - so check the ConfigAndLog log for error messages if necessary.

h3. SMTP relayed through Postfix (recommended)

See also here: http://docs.aws.amazon.com/ses/latest/DeveloperGuide/postfix.html

Postfix will give higher performance on large mailings, and also does a much better job of handling resends, greylisting, etc.  This should be implemented wherever possible.

* Install Postfix.

* Paste this at the bottom of @/etc/postfix/main.cf@:

<pre>

#jon@palantetech.coop SES setup

relayhost = email-smtp.us-west-1.amazonaws.com:25

smtp_sasl_auth_enable = yes

smtp_sasl_security_options = noanonymous

smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd

smtp_use_tls = yes

smtp_tls_security_level = encrypt

smtp_tls_note_starttls_offer = yes

#This last line isn't needed, I'm pretty sure

#smtp_tls_CAfile = /etc/ssl/certs/ca-bundle.crt

#jon@palantetech.coop rate limiting to 4 messages/second for now

smtp_destination_rate_delay = 1s

smtp_destination_concurrency_limit = 4

</pre>

* Create a file with credentials at @/etc/postfix/sasl_passwd@ in the format:

<pre>

email-smtp.us-east-1.amazonaws.com:25 username:password

</pre>

* Run @postmap /etc/postfix/sasl_passwd@.

* Configure CiviCRM's SMTP settings to either a) point to localhost, or b) use mail().

h2. Bounce processing setup

There's code here to use a Amazon SNS to a bounce processing address instead of the typical bounce processing "here":https://etopian.com/blog/civicrm-bounce-processing/.  I have NOT tested this, but I don't think this provides an advantage over typical bounce processing.  An SNS endpoint in CiviCRM COULD provide higher performance than IMAP/POP-based bounce processing, I don't think this solution is it.

So instead, just see [[Set up local bounce processing]].

h2. Testing

* To test bounce processing, set up a test group made up of at least one verified address and a test contact with the e-mail address "bounce@simulator.amazonses.com".

* Send a mailing to that group.  You can speed up testing by manually running the "Send Scheduled Mailings" and "Fetch Bounces" scheduled jobs.

* Don't forget to request production access when you're done!

h3. Troubleshooting

In testing if CiviCRM reports that everything's been sent correctly, but then no emails are received, the problem could be with Postfix authenticating to SES.  /var/log/mail.log might contain entries like this.

<pre>

Aug  9 11:37:48 XXXXXXXX postfix/smtpd[24665]: connect from localhost[127.0.0.1]

Aug  9 11:37:48 XXXXXXXX postfix/smtpd[24665]: A30A4624F1: client=localhost[127.0.0.1]

Aug  9 11:37:48 XXXXXXXX postfix/cleanup[24669]: A30A4624F1: message-id=<20160809163748.A30A4624F1@XXXXXXXX.example.org>

Aug  9 11:37:48 XXXXXXXX postfix/qmgr[24664]: A30A4624F1: from=<XXXXXXXX@example.org>, size=453, nrcpt=1 (queue active)

Aug  9 11:37:48 XXXXXXXX postfix/smtpd[24665]: disconnect from localhost[127.0.0.1]

Aug  9 11:37:48 XXXXXXXX postfix/smtp[24670]: warning: SASL authentication failure: No worthy mechs found

Aug  9 11:37:48 XXXXXXXX postfix/smtp[24670]: A30A4624F1: SASL authentication failed; cannot authenticate to server email-smtp.us-west-2.amazonaws.com[54.68.106.242]: no mechanism available

Aug  9 11:37:48 XXXXXXXX postfix/smtp[24670]: warning: SASL authentication failure: No worthy mechs found

Aug  9 11:37:48 XXXXXXXX postfix/smtp[24670]: A30A4624F1: SASL authentication failed; cannot authenticate to server email-smtp.us-west-2.amazonaws.com[52.35.58.187]: no mechanism available

Aug  9 11:37:49 XXXXXXXX postfix/smtp[24670]: warning: SASL authentication failure: No worthy mechs found

Aug  9 11:37:49 XXXXXXXX postfix/smtp[24670]: A30A4624F1: to=<XXXXXXXX@example.com>, relay=email-smtp.us-west-2.amazonaws.com[52.35.228.26]:25, delay=0.39, delays=0.04/0.01/0.34/0, dsn=4.7.0, status=deferred (SASL authentication failed; cannot authenticate to server email-smtp.us-west-2.amazonaws.com[52.35.228.26]: no mechanism available)

</pre>

If that's the case, you might need to install some additional libraries, namely the @libsasl2-modules@ package in Debian/Ubuntu systems.

h3. SPECIAL NOTE FOR EC2 INSTANCES

EC2 severely throttles outgoing mail on ports 25/465/587.  You need to submit a request to Amazon to have this throttle lifted.  This is separate from any user-configurable firewalling!  You can do that (and set up PTR) here: https://portal.aws.amazon.com/gp/aws/html-forms-controller/contactus/ec2-email-limit-rdns-request