Sedna Domain - Isolated Issue 15th August

Summary

An issue occurred with Sedna at approximately 00:23 BST on 15th August 2024. The cause of this incident was investigated immediately and has since been resolved. We are aware that multiple important aspects of our service were impacted for approximately three hours. 

Detailed Description

The incident report will be made available once the Engineering team has completed a thorough investigation. 

In summary what we know now is:

• At or around 00:23 BST on 15th August 2024 a key domain used by Sedna failed to renew its registration.
• Notifications for the renewal were sent to an email address that was not actively monitored.
• The domain was renewed as soon as the problem was identified, and DNS propagation began.
• Due to the nature of DNS interruptions, the after-effects of the temporarily disabled domain caused multiple failures within our platform, in an inconsistent manner. Failures sometimes included:
• Rejected login attempts
• Successful send/receive activity not observable in UI (search platform indexing failures)
• Rejected mail delivery via mail forwarding

Remediation and Prevention

• At 02:42 BST 15th of August 2024 the issue was resolved.
• There was no security or data breach in this incident and no Personally Identifiable Information (PII) was involved.
• We have mitigated this incident in the future by moving this legacy domain to our core AWS infrastructure for auto-renewal where all other domains are managed.

Emails undelivered to Sedna 

Some emails were not delivered to Sedna and onwards to your users during the affected timeframe. 

Instructions on how to trace the affected emails as well as forward them to Sedna can be found below: 

Message Trace

“Message Trace” can be used to find emails that weren’t successfully forwarded to Sedna for the duration of the issue. This tool shows the path of an email as it moves through the system and tells you whether it was delivered, failed, or delayed. 

• Using the Exchange Admin Center (EAC):
  • Go to the Exchange Admin Center (EAC).
  • Navigate to Mail Flow > Message Trace.
  • Enter the time period and search for the messages.
  • Check the status of the messages (Delivered, Failed, etc.)
• Using PowerShell: 
  • You can run this command to get detailed information on what happened to the emails during the time forwarding failed:
    • Get-MessageTrace -StartDate "YYYY-MM-DD" -EndDate "YYYY-MM-DD" -SenderAddress "user@domain.com"

Search for the Original Messages 

Where the forwarding failed, you can find the received emails in the shared mailbox and forward them manually again, or alternately they can be exported and provided to Sedna for import into the tenant:

• Use the Content Search tool in the Microsoft Purview Compliance Center to find specific emails in mailboxes.

Sedna Customer Support is here to help, including reloading these emails through a historical mail import if customers are unable to conduct the above actions. 

Please don’t hesitate to get in touch if we can assist on the matter via support@sedna.com. 

Your Understanding Is Appreciated 

We deeply understand the critical nature of the services that Sedna provides to your business. As such, we have implemented preventative measures to mitigate the risk of future interruptions. We will also conduct thorough retrospectives to learn from this issue.   

Thank you for your patience and understanding on this matter and we sincerely apologise for any inconvenience caused to your and your teams.