It isn't until you don’t have something that you realise how ingrained it is in everything you do. This was our experience recently when we were, unfortunately, one of the “lucky” few customers to have been affected by the recent Atlassian outage.

We use Atlassian Cloud for Jira, Confluence, and Service Management in every aspect of our Managed Services. This includes Service Desk, CMDB, Reporting, and Knowledge Base. We also use Confluence for our technical documentation, including Release Run Sheets and Configuration. Most of our customers also have direct access to a Customer Portal for the Service Desk and Knowledge base.

What we didn’t know when the outage started was just how long it was going to be. In what ended up being a two-week outage, we turned to “old school” workarounds. Out came multiple Excel Workbooks, a plethora of emails, and a lot more phone calls. Releases were delayed, reporting was unavailable, and no access to the CMDB meant trusting we had already identified renewals coming up.

Like most organisations, when we subscribed to a cloud service, we signed up for 99.999% uptime. Never did we envisage a two-week outage. To mitigate risk, we have processes in place to manage should the need arise, and we do take our own backups of Jira and Confluence, but as demonstrated by Atlassian, restoring these is not a simple task and would result in data loss. This outage also leads us to review our current mitigation strategy to ensure that what we have is enough in the unlikely event that this might happen again.

What did we learn?

Cloud services fail—all of them. Most of the time, these failures are only a few minutes or hours. It is very unusual to have an outage lasting two weeks.

In this instance, the outage was caused by Human Error. This is almost impossible to prevent. This outage was caused when our Atlassian tenancy was hard deleted. Due to this action being a “hard” delete, there was no ability to simply “reactivate.” There was also no failure of the infrastructure it was hosted on, making it very complicated to restore. Atlassian was operating as expected. We no longer had a tenancy in it.

How do we protect ourselves if this were to happen again?

While we have manual processes as part of our BCP, there were a few things that we identified because of this outage that is critical to always have access to. We now have manual routine backup processes in place, outside of full system backups, to ensure we have access to this information in the unlikely event of another significant outage.

What was our biggest takeaway?

Plan for the worst-case scenario and hope that it never happens. But if it does happen, know you have done what you can to prevent disruption to your business.

About TEAM IM

TEAM IM is an experienced solution company that advises, develops, implements, supports, and manages enterprise grade process, information and content management systems. For more than twenty years, TEAM IM has acted as a trusted advisor to our clients through our offices in Australia, New Zealand, Europe and the United States. Our mission is to assist our client to get the most out of their investments in technology. Whether our clients are large government agencies or corporations, construction firms, accounting firms, heavy industry, or smaller organizations, we strive to deliver demonstrable business benefits and generate real return on investment for our clients.

Our products and services offer solutions to transform your business by automating and modernizing your operations. We work hand-in-hand with our clients to understand their goals and create and execute multi-year, continuous improvement plans. Our mission is to support and manage every solution we deliver, so we take care to design long term, future proof, maintainable solutions. We work with best-in-class technology partners that we have carefully selected to ensure we can execute our plan and achieve our clients continuous improvement goals.

Our products and solutions encompass Advisory Services, Implementation Services and Managed Support Services. We specialize in Business Process Automation and Optimization, Content Platforms and Content Services and we are also a leader in Mobile App/Field Services software development and Digital Workplaces. We have industry-specific solutions for the Construction and Accounting Services sectors, and cross-industry solutions for Accounts Payable, Contract Management, App Modernization, Field Services and File Sharing.

At TEAM IM we are passionate about delivering outstanding outcomes for you, our clients.