As many of you are aware, Dedoose services came crashing down Tuesday, May 6th at around 4:30 pm PST.
First, in short, after we complete the final stage of data restoration, the vast majority of Dedoose projects will not have been affected. Our deepest regret is for those projects which will have been directly impacted and we are prepared to work with any of you on a personal case-by-case basis to remedy and assist in any way we can.
This devastating system ‘collision’ of Tuesday night resulted from an series of events leading to the failure of Dedoose services running on the Microsoft Azure platform. To be clear, Dedoose services failed, not Azure. In short, work done on one aspect of Dedoose led to the failure of another, cascading to pull down all of Dedoose. The timing was particularly bad because it occurred in the midst of a full database encryption and backup. This backup process, in turn, corrupted our entire storage system. Our immediate work with Microsoft support did not result in any substantial recovery. Here’s where we are and where we’re going:
- The first restoration of Dedoose services took place Wednesday morning with Dedoose services restored, and the database up, accurate, and current as of the first week in March
- Initially we hesitated to bring the system back online at the March state but didn’t want to hang up any users whose data were not affected
- Any data, new codes and coding, etc. that any users add to your projects from this point forward will not be disrupted as we bring past back-ups on-board, so all are free to continue working in the system
- A second data recovery and restoration is taking place as I write. We have high confidence that data added to Dedoose up to mid-April will be recovered and restored. Stay tuned; we will broadcast on this shortly as progress is made on our blog and Facebook
- Again, most devastating, and despite our work with Microsoft, is that we are not optimistic we will be able to recover data added to the system for roughly the 2 – 3 week window preceding the failure on Tuesday May 6 at approximately 4:30 pm PST
- Once the second restoration is completed, we will all be better able to assess the state of each project. Where needed, and if at all possible, we are prepared to dig in and offer our assistance to help bring things fully up-to-date where at all possible
- We will post widely when the second restoration is completed and we’ll be ready to work from there to address any remaining issues
- We’ve already implemented certain protections on the Azure deployment to prevent such an event re-occurring
And, for the more technically curious, we’re going tour de force on protection by:
- Deploying a database mirror/slave in Azure
- Deploying a database mirror/slave into Amazon S3
- Keeping a mirror copy of the entire blob storage including all file data, backups, video data synchronized nightly to our private server in an encrypted volume
- Storing nightly database backups on the VHD, Azure Blob Storage, and Amazon S3 Storage
- Mirroring all Azure file data into an Amazon S3 bucket
- Carrying out a weekly restore exercise for the database backups to ensure integrity
- Carrying out monthly bare bones from nothing restores of the entire Dedoose Platform to ensure integrity
This is an unprecedented and completely unanticipated event in the history of Dedoose services which went on line in 2006. Regrettably, and as high as our technical guard was, we nonetheless did not recognize the danger and, as such, SCRC is fully responsible to every user, regardless of any issues with our vendors. As you can imagine, this is a painful and challenging moment for us and we are doing all we can to help restore and re-build user data and, certainly, confidence. We are here if you have any further questions and will be posting widely as we make progress. As you might imagine, we are overwhelmed at the moment with user inquiries and working as fast as we can to respond.
We thank you for your patience as we work to return Dedoose to an even stronger platform, and truly hope you’ll stay around so we can continue to serve your research needs and regain your full confidence.
President | CEO, SocioCultural Research Consultants, LLC (dedoose.com)