Dedoose’s Black Eye, Crash, and Recovery Efforts

As many of you are aware, Dedoose services came crashing down Tuesday, May 6th at around 4:30 pm PST.

First, in short, after we complete the final stage of data restoration, the vast majority of Dedoose projects will not have been affected.  Our deepest regret is for those projects which will have been directly impacted and we are prepared to work with any of you on a personal case-by-case basis to remedy and assist in any way we can.

This devastating system ‘collision’ of Tuesday night resulted from an series of events leading to the failure of Dedoose services running on the Microsoft Azure platform.  To be clear, Dedoose services failed, not Azure.  In short, work done on one aspect of Dedoose led to the failure of another, cascading to pull down all of Dedoose.  The timing was particularly bad because it occurred in the midst of a full database encryption and backup.  This backup process, in turn, corrupted our entire storage system. Our immediate work with Microsoft support did not result in any substantial recovery.  Here’s where we are and where we’re going:

  1. The first restoration of Dedoose services took place Wednesday morning with Dedoose services restored, and the database up, accurate, and current as of the first week in March
  2. Initially we hesitated to bring the system back online at the March state but didn’t want to hang up any users whose data were not affected
  3. Any data, new codes and coding, etc. that any users add to your projects  from this point forward will not be disrupted as we bring past back-ups on-board, so all are free to continue working in the system
  4. A second data recovery and restoration is taking place as I write.  We have high confidence that data added to Dedoose up to mid-April will be recovered and restored.  Stay tuned; we will broadcast on this shortly as progress is made on our blog and Facebook
  5. Again, most devastating, and despite our work with Microsoft, is that we are not optimistic we will be able to recover data added to the system for roughly the 2 – 3 week window preceding  the failure on Tuesday May 6 at approximately 4:30 pm PST
  6. Once the second restoration is completed, we will all be better able to assess the state of each project.  Where needed, and if at all possible, we are prepared to dig in and offer our assistance to help bring things fully up-to-date where at all possible
  7. We will post widely when the second restoration is completed and we’ll be ready to work from there to address any remaining issues
  8. We’ve already implemented certain protections on the Azure deployment to prevent such an event re-occurring

And, for the more technically curious, we’re going tour de force on protection by:

  1. Deploying a database mirror/slave in Azure
  2. Deploying a database mirror/slave into Amazon S3
  3. Keeping a mirror copy of the entire blob storage including all file data, backups, video data synchronized nightly to our private server in an encrypted volume
  4. Storing nightly database backups on the VHD, Azure Blob Storage, and Amazon S3 Storage
  5. Mirroring all Azure file data into an Amazon S3 bucket
  6. Carrying out a weekly restore exercise for the database backups to ensure integrity
  7. Carrying out monthly bare bones from nothing restores of the entire Dedoose Platform to ensure integrity

This is an unprecedented and completely unanticipated event in the history of Dedoose services which went on line in 2006.  Regrettably, and as high as our technical guard was, we nonetheless did not recognize the danger and, as such, SCRC is fully responsible to every user, regardless of any issues with our vendors.   As you can imagine, this is a painful and challenging moment for us and we are doing all we can to help restore and re-build user data and, certainly, confidence.  We are here if you have any further questions and will be posting widely as we make progress.  As you might imagine, we are overwhelmed at the moment with user inquiries and working as fast as we can to respond.

We thank you for your patience as we work to return Dedoose to an even stronger platform, and truly hope you’ll stay around so we can continue to serve your research needs and regain your full confidence.

Sincerely,

Eli Lieber

President | CEO, SocioCultural Research Consultants, LLC (dedoose.com)

2 thoughts on “Dedoose’s Black Eye, Crash, and Recovery Efforts

  1. Arnold Toutant

    I am still unable to get into my account to see if anything is there on my project. It tells me that my password or user name is wrong. Then, when I try to change the Password, I get the error message again. I keep being put into a runaround.

    Reply
    1. DedooseSCRC Post author

      If you created your account between March 3rd and April 11th, then please wait until the second restore is live. We will email you when that restore is up so you can go ahead and check.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>