Maintenance at our data center
Scheduled Maintenance Report for Joomag, Inc.
Postmortem

Overview of the Incident: On October 24, 2023, during a scheduled network maintenance at our data center, an unforeseen issue occurred, leading to significant service disruptions on the Joomag platform. This postmortem report aims to outline the events, our immediate response, and the steps we are taking to prevent such incidents in the future.

Sequence of Events:

  • Planned Activity: As part of our routine maintenance, one server was scheduled to be taken offline and relocated to another rack.
  • Unexpected Downtime: Contrary to our plans and preparations, four additional servers were inadvertently taken offline.
  • Service Impact: This resulted in a downtime for Joomag, affecting various functionalities.

Impact Assessment:

  • Data Corruption: We encountered data corruption on one MySQL server and one ClickHouse server.
  • Recovery Measures: Both servers were successfully recovered from backups, ensuring no data loss.
  • Restoration Time: The total duration for restoring full service functionality was approximately 6 hours.

Investigation and Current Understanding:

  • We are actively investigating the root cause of the additional servers going offline.
  • Initial assessments indicate a potential procedural error or miscommunication, but further analysis is ongoing.

Corrective and Preventive Actions:

  • Immediate steps were taken to restore the affected services promptly.
  • We are revising our maintenance protocols to enhance safety and reliability.
  • We are working closely with our hosting provider to avoid such incidents in the future.

Commitment to Service Excellence: We apologize for the inconvenience caused by this incident and appreciate your patience and understanding. Our dedication to providing dependable and high-quality services is stronger than ever. We will keep you informed about our progress and findings as we continue our investigation.

Joomag, Inc.

Posted Oct 31, 2023 - 13:35 UTC

Completed
The scheduled maintenance has been completed.
Posted Oct 25, 2023 - 03:01 UTC
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Oct 24, 2023 - 22:02 UTC
Scheduled
Our data center will be undergoing scheduled network maintenance. This is a routine procedure aimed at enhancing the performance and reliability of our services.

Impact on Services
During the maintenance period, users may experience delays or temporary unavailability in actions such as creating publications from PDFs, sending email campaigns, and executing other background jobs. These interruptions are necessary to implement upgrades and improvements effectively.

Reader Experience
We anticipate that the reader experience will remain unaffected. All publications will be accessible, ensuring that readers can engage with content seamlessly.
Posted Oct 19, 2023 - 20:34 UTC
This scheduled maintenance affected: Joomag Publishing Cloud and Joomag Mailing Services.