Status update

You can find the full incident report here: https://trust.engagingnetworks.net/

Update January 25th, 2024 (7:00pm GMT/ 2:00pm ET) –

We were very glad to report earlier this morning that you are now able to resume sending your email campaigns to your usual audiences. We are continuing to investigate what happened with Spamhaus and will be following up with as much information as we can in the coming days. 

One important precautionary measure we are taking to minimize the risk of a similar incident occurring in the future is requiring that all sender domains be authenticated with author-signed DKIM and Return Path. In the past, we have asked clients to do this; however after 2:30pm EST today you will be blocked from sending emails from any domains that have not been authenticated through our Domain Authentication tool. 

Additional Details: 

• If you are attempting to send a broadcast email using Email Classic or Marketing tools using a non-authenticated domain, you will see notification text on the sending tab for Marketing Tools and Email Classic (see images below) and the send/schedule button will be disabled. 
  Email classic
  Marketing tools
  Marketing automation email
  
• This is only relevant for sender domains. You can check which emails are configured as senders in Account Settings > Account Emails. 
• You must authenticate for both DKIM and Return Path. SPF is still recommended, but will not be checked as part of this authentication step.If you are attempting to send a Marketing Automation email using a non-authenticated domain, you will see notification text when attempting to save the email (see image below).

If you are unsure what to do to ensure you are in compliance with this new requirement, please visit this page on our Supportal for guidance. You can also reach out to our Support Team or your Account Success Manager who are standing by to help in any way that they can. 

Again, this update will go into effect as of 2:30pm EST / 7:30pm GMT / 11:30am PT, today, January 25th. Please take the necessary steps to ensure that your organization is in compliance. 

Update January 25th, 2024 (2:00pm GMT/ 9:00am ET) –

As of 9:00AM ET / 2:00PM GMT, Spamhaus has lifted all blocks of our sending IP addresses. This means that you can resume sending emails as you normally would.

You will need to ensure that you’ve authenticated your sending domains using the Domain Authentication tool under Account Settings, Email Settings, or Marketing Tools Setup.

We will have more information in the coming days and a full incident report to share. There will be a hotfix later today that will check that your sending domain is authenticated.

Thank you for your patience as we worked to get this resolved as quickly as possible.

Update January 25th, 2024 (9:50am GMT/ 4:50am ET) –

Spamhaus’s IP block is still in place and our recommendations remain the same. Please refer to the previous update and contact us with any questions. 

Please note that @gmail, @googlemail, @yahoo and @aol remain unblocked and can be put into a profile (Email Address contains @gmail~@googlemail~@yahoo~@aol) if you wish to only target those domains in a broadcast email.

Update January 24th, 2024 (7:10pm GMT/ 1:10pm ET) –

We are happy to report that e-activist.com was removed from Spamhaus’s listings around 4am GMT this morning; however, all of our IP addresses remain listed. This means that all email domains (with the exception of Google and Yahoo) are still bouncing at rates up to 100%.

We are actively reaching out to Spamhaus to get this resolved and will provide additional information and updates as soon as we are able.

How did this happen?

We are working to get the details from Spamhaus on what caused such a sudden reduction in our reputation yesterday. Spamhaus has restored our score to its previous level, which indicates that any issues on their end have been resolved, and they are working to remove our IP blocks.

Reports from Spamhaus also indicate that emails are being sent from Engaging Networks using unauthenticated domains. We will therefore be taking the necessary steps on our end to prevent this issue from recurring. Moving forward, we are going to require that all clients use author-signed DKIM for sending emails. This digital signature is a method of email authentication that, simply put, is a best practice everyone should follow – and therefore, we will start enforcing it. Additional details on this will be forthcoming.

Our Recommendations:

• Authenticate your domains: Please take a moment today to confirm that all of your sender domains are properly authenticated.  You can follow the instructions on our Supportal here, and if you have questions please feel free to reach out to our support team.

• Only target Google and Yahoo in Broadcast Emails: If you plan to continue to send email, we recommend setting up a simple profile that targets only Google and Yahoo email addresses. My colleague Noelle has created a short video that shows you how to do this.

• Pause your Marketing Automations: While current supporters may experience deliverability issues, it’s better not to enroll any new users until this issue is reconciled. When the automations are un-paused, they will pick up any users who qualify that have not yet been enrolled.

• Only Stop a Marketing Automation if you really have to. You might have to if you have an automation that has a significant number of users in the journey, and the chance / impact of those users not receiving emails or emails reaching SPAM is too significant. Re-enrollment is not possible. A new Marketing Automation would need to be created.

• Email to Target, Autoresponders, HUB confirmation emails: Keep your current campaigns live and consider adding a message about experiencing deliverability issues on the “post-action” page if you feel it’s important to be transparent that messages may not reach their intended targets. For New Campaigns, consider delaying launch until this issue is mitigated or until there is further guidance.

• ETTs targeting the US House of Representatives and Senate: All Representatives and most Senators use the Communicating with Congress API and are not impacted by this outage.

• Receipts: We recommend either directing supporters to your Supporter Hub to obtain their receipt or issuing them a replacement receipt via the Single Donation Gadget on their Supporter record. We recommend not issuing Annual Receipts until after this issue is resolved.

Update January 24th, 2024 (9:10am GMT/ 4:10am ET) –

We are in conversations with Spamhaus. Our recommendations remain as for the last update – all email jobs are currently processing in Engaging Networks but the Spamhaus block remains.

While Google and Yahoo email addresses continue to not be impacted by the Spamhaus block, most other email domains (Hotmail, Office365, Apple, Comcast, etc.) are bouncing at rates up to 100%.

Update January 23rd, 2024 (11:10pm GMT/ 6:10pm ET) –

All email jobs are currently processing in Engaging Networks. While Google and Yahoo email addresses continue to not be impacted by the Spamhaus block, most other email domains (Hotmail, Office365, Apple, Comcast, etc.) are bouncing at rates up to 100%.

Recommendations

• Cancel and Reschedule Broadcast Emails: If your campaigns can wait, please hold them for the next 24 hours. We hope to have more actionable information for clients tomorrow.
• Pause your Marketing Automations: While current supporters may experience deliverability issues, it’s better not to enroll any new users until this issue is reconciled. When the automations are un-paused, they will pick up any users who qualify that have not yet been enrolled.
  • Only Stop a Marketing Automation if you really have to. You might have to if you have an automation that has a significant number of users in the journey, and the chance / impact of those users not receiving emails or emails reaching SPAM is too significant. Re-enrollment is not possible. A new Marketing Automation would need to be created.
• Email to Target, Autoresponders, HUB confirmation emails: Keep your current campaigns live and consider adding a message about experiencing deliverability issues on the “post-action” page if you feel it’s important to be transparent that messages may not reach their intended targets. For New Campaigns, consider delaying launch until this issue is mitigated or until there is further guidance.
• ETTs targeting the US House of Representatives and Senate: All Representatives and most Senators use the Communicating with Congress API and are not impacted by this outage.
• Receipts: While some donors may not receive their receipt due to this issue, we are working on a resolution. Please continue to issue receipts.

Update January 23rd, 2024 (7:20pm GMT/ 2:20pm ET) –

We have paused all bulk email and marketing automation jobs, until 4:00pm ET / 1:00pm PT / 9:00pm GMT. Any emails still in the system at that time will process. If you do not want your emails to be sent, please delete the job from the job monitor.

Two pieces of updated information:

• The Spamhaus block is NOT impacting Gmail/Google and Yahoo email addresses.
• Any supporters whose email bounced as a result of the blockage will not be auto-suppressed in Engaging Networks. They have only soft-bounced, so you will still be able to email them once we are no longer blacklisted.

This short video shows you how to build a profile to only target supporters with those email domains if you wish to resume sending emails after 4:00pm ET / 1:00pm PT / 9:00pm GMT.

At this time, we would recommend pausing all marketing automations. Pausing an automation will stop any new supporters from being added. But, anyone in the automation will continue through the workflow. If you stop an automation, everyone in the workflow will be kicked out and you will not be able to easily add them back in.

Edit: We incorrectly stated that auto-responders and email-to-target emails were paused. This is not the case. Only broadcasts and automations were paused

January 23rd, 2024 (5:30pm GMT / 12:30pm ET) –

We received an alert from Spamhaus that they have listed e-activist.com, which is the primary hostname for our mail servers, to their domain-blocklist. This appears to have resulted in our mail server IPs being listed en masse as well. Spamhaus is the biggest blacklisting service out there.

We are working to resolve this issue as quickly as possible, but do not have an estimated time table currently.

Due to the nature of the listing, we would advise clients to discontinue sending emails through Engaging Networks until we provide an update on the Spamhaus listing.

Additionally, this is impacting our Support email address. Clients can email [email protected] to submit tickets or use https://www.engagingnetworks.support/contact-support/

This is only impacting our email servers. All other functions (fundraising pages, advocacy pages, etc.) continue to work as expected.

August 16, 2022 (11:42 BST / 06:42 EDT) – We have resolved the issue causing disruption to logging into accounts. You may need to clear your cache and log in again. Client web pages were not impacted during this time.

July 28, 2022 (07:16 BST / 02:16 EDT) – AWS instances have been restored, and all EN applications and pages are online. You may need to clear your cache and log in again. Any delayed jobs have begun to process.

July 28, 2022 (06:25 BST / 01:25 EDT) – AWS reported that the issue occurred due to a power outage and that power has now been restored. AWS expects recovery for all impacted instances within the next 2-3 hours. We will update when EN sites are restored.

July 28, 2022 (06:05 BST / 01:05 EDT) – Sites on the U.S. server are currently experiencing downtime due to AWS connectivity issue. This impacts access to our application and pages hosted on the U.S. instance. We are awaiting more updates from AWS. Please contact Support with any questions.

July 20, 2022 (03:11 BST / 10:11 EDT) – The Region-based Captcha issue has been resolved. The third-party vendor we use for location-based Captchas had experienced an issue which they have corrected. We are waiting on more details from them, but you can continue to effectively use region-based captchas on desired pages. Please contact Support if you have any questions.

July 19, 2022 (07:02 BST / 02:02 EDT) – Region-based Captcha is causing an issue on some client pages. We recommend removing region-based Captcha for the time being. We will send an update once we have a fix. 

June 21, 2022 (09:06 BST / 04:06 EDT) – Cloudflare have informed us that the issue is now resolved. Please contact Support if you are experiencing any ongoing problems.

June 21, 2022 (07:43 BST / 02:43 EDT) – We are aware of an incident with the providers of our web application firewall, Cloudflare, that’s impacting service from our application at various locations globally. The issue impacts the engagingnetworks.app domain that’s hosted in Cloudflare, but not our servers in AWS which are operating normally.
We are observing an interruption to the Engaging Networks application in locations where Cloudflare is still suffering connectivity problems.

You can check the Cloudflare status page for more information.

May 3, 2022 (10:40 ET / 15:40 GMT)  – The hotfix has been applied and initial tests suggest the issue is resolved. Any data pulled on the 30th April, 1st May, 2nd May should be manually pulled via the Bulk Export API. Please reach out to support if you require any assistance.

No live data will have been lost due to this issue. 

May 3, 2022 (10:00 ET / 15:00 GMT) – Our Export API continues to have an issue with the staged data. We pushed a hotfix last night that did not resolve the problem. We have identified a new issue and are working on a fix for that now. We should have more information in the next few hours.

Our advice continues to be to NOT export data from the Export API service. We will be sending instructions along with how to manually download an export file once we have fixed the problem.

May 1, 2022 (09:30 ET / 14:30 GMT) – It was brought to our attention that export data did not stage properly for exports on the morning of Sunday, May 1, 2022, for clients on our Canadian server. All non-US clients (Canada, UK, Australia, Europe) are on the Canada server, along with some clients based in the US.