Moving My Data To The Cloud: Stormy Weather
For years, I’ve maintained a single “main” computer. It was the computer that was the central authority of all of the personal data I’d accumulated over the years and from which it made me uncomfortable to be separated. Because I needed a single computer for everything, it had to work on my couch, on a plane, on a desk and everywhere else I ever needed to go. Also, it couldn’t have a giant monitor or multiple monitors, because it had to go everywhere. All of this was because I needed all of my data with me all of the time.
My process for moving to a new computer used to include a lot of manual copying of files from the old D hard drive (D is for Data) to my new hard drive, which was also carefully partitioned into C for Windows, Office, Visual Studio, etc. and D for a lifetime of books and articles, coding projects and utilities I’ve collected over the years, e.g. LinqPad, Reflector, WinMerge, etc. This is 30GB of stuff I wanted access to at all times. I was also backing up via Windows Home Server, keeping photos and music on the WHS box (another 30GB), then backing that up to the cloud via KeepVault. And finally, as I upgraded HDs to go bigger or go to solid state, I kept each old HD around as another redundant backup.
All of that gave me some confidence that I was actually keeping my data safe right up until my Windows Home Server crashed the system HD and I found out that the redundancy of WHS doesn’t quite work the way you’d like (this was before I installed KeepVault). This was a first generation HP Home Server box and when it went down, I took it apart so I could attach a monitor, keyboard and mouse to diagnose it, pulled the HDs out so I could read what files I could and ultimately had to drop it off in Redmond with the WHS team so I could get it up and running again.
There are some files I never got back.
KeepVault gave me back some of the confidence I’d had before WHS crashed, but they didn’t provide me a way to see what files they were backing up, so I didn’t have the transparency I wanted to be confident. Further, they don’t have clients on every kind of platform like Dropbox does.
Of course, simply sync’ing files isn’t enough – sync’ing my 10GB Outlook PST file every time I got a new email was not a good way to share 20 years of contacts, email and calendar items.
The trick is to sync each kind of data in the right way, be confident that it’s safe and have access to it across the various platforms I use: Windows, Windows Phone 7, iOS and possibly Android (you know, if I feel like walking on the wild side!). And since I’m currently under employed (my new gig doesn’t start till the new year), I figured I’d do it once and do it right. I almost got there.
Let’s start easy: files. Dropbox has made this a no-brainer. You install the software on any platform you care to use, drop everything you want into the folder and it just works, keeping files in sync on the cloud and across platforms, giving you adequate (although not great) status as it does so. Most platforms are supported natively, but even on platforms that aren’t, there are often alternative clients, e.g. I’m using Boxfiles for Windows Phone 7. When I gave up my Microsoft laptop, instead of doing the dance of the copy fairy to my new Mac Book Air, I installed Dropbox on both computers, dropped everything I want backed up and sync’d between computers into the Dropbox folder. 36 hours and 30GB later, all of it was copied into the cloud and onto my new laptop, at which point I reformatted my Microsoft laptop and handed it into my boss.
Further, as a replacement for WHS and KeepVault, I now keep all of the files that I was keeping just on my WHS server – photos and music primarily – into Dropbox.
This keeps me the confidence I need to know that my files are safe and backed up to the cloud, while making it very easy to keep it backed up locally by simply running Dropbox on more than one computer at my house. If at any time, I don’t want those files on any one computer, I tell Dropbox to stop sync’ing those folders, delete the local cache and I’m all done.
There are two tricks that I used to really make Dropbox sing for me. The first is to change my life: I no longer partition my HDs into C and D. The reason I’d always done that was so that I could repave my C with a fresh Windows, Office and VS install every six months w/o having to recopy all my data. Windows 7 makes this largely unnecessary anyway (bit rot is way down on Win7), but now it doesn’t matter – I can blow any computer away at will now, knowing that Dropbox has my back. In fact, Dropbox is my new D drive, but it’s better than that because it’s dynamic. The C drive is my one pool of space instead of having to guess ahead of time how to split the space between C and D.
The other thing I did was embrace my previous life: I wanted to keep D:\ at my fingertips as my logical “Data” drive. Luckily, Windows provides the “subst” command to do just that. Further, ntwind software provides the fabulous VSubst utility to do the mapping and keep it between reboots:
Now, I’ve got all the convenience of a dedicated “data” drive backed up to the cloud and sync’d between computers. Because I needed 60GB to start, I’m paying $200/year to Dropbox for their 100GB plan. This is more expensive than I’d like, but worth it to me for the data I’m storing.
There is a hitch in this story, however. Right now on Dropbox, data and metadata is available to Dropbox employees and therefore to anyone that hacks Dropbox (like the government). I don’t like that and for my very most sensitive data, I keep it off of Dropbox. When Dropbox employees themselves aren’t able to read Dropbox data or metadata, then I’ll move the sensitive data there, too.
I’m not actually very happy with how I’m storing music. I can play all my music on any PC, but I can only play it one song at a time on my WP7 because there’s no Dropbox music client. I could use the Amazon cloud drive that provides unlimited music storage for $20/year, but there’s no WP7 client for that, either. Or I could spend $100/year on Amazon and get my 100GB of storage, but their client isn’t as widely available as Dropbox. Ironically, Dropbox is using Amazon as their backend, so hopefully increased pressure in this space will drop Dropbox’s prices over time.
I’m not using Facebook or Flicr for my photos simply because I’m lazy. It’s very easy to copy a bunch of files into Dropbox and have the sync’ing just happen. I don’t want to futz with the Facebook and Flickr web interfaces for 15GB worth of photos. Right now, this is the digital equivalent of a shoebox full of 8x10s, but at least I’ve got it all if the house burns down.
For general, freeform notes, I moved away from Evernote when they took the search hotkey away on the Windows client (no Ctrl+F? really?) and went to OneNote. The web client sucks, but it’s better than nothing and the Windows and WP7 clients rock. I have a few notes pinned to my WP7 home screen that I use for groceries, tasks, etc., and I have all of my favorite recipes in there, too, along with my relatives’ wi-fi passwords that they don’t remember themselves, a recording of my son snoring, etc. It’s a fabulous way to keep track of random data across platforms.
On the task list side, I only sorta use OneNote for that. I also send myself emails and write little TODO.txt files every time I get a little bee in my bonnet. I’ve never found that the Exchange tasks sync well enough between platforms to invest in them. Maybe someday.
And speaking of Exchange, that’s a piece of software that Microsoft spoiled me on thoroughly. This is sync that works very well for contacts, emails and calendar items. IMAP does email folders, but server implementations are spotty. For years, I used Exchange for my personal contacts and calendar, only keeping my personal email separate in a giant PST file, pulling it down via POP3. This can sorta be made to work, but what I really wanted was hosted Exchange.
However, what I found cost between $5 and $11 a month per user. I’d probably have gone with Office 365 for sellsbrothers.com mail, even at $5/month except for two reasons. The first is that Microsoft requires you to move your entire DNS record to them, not just the MX record, which means there is all kinds of hassle getting sellsbrothers.com working again. They do this so that they can get all of the DNS records working easily for Lync, Sharepoint, etc., but I don’t want those things, so it’s just a PITA for me. If they change this, I’d probably move except for the other problem: I’m not the only user on sellsbrothers.com.
For years to be the big shot at family gatherings, I’ve been offering up permanent, free email addresses on my domain. That’s all well and good, but now to maintain my geek cred, I need to keep my mom, my step-mom, my brother, my sons, etc., in an email server that works and one that they don’t have to pay for. So, while I was willing to pay $5/month for hosted exchange for me, I wasn’t willing to pay it for my relatives, too!
One option I tried was asking securewebs.com (my rocking ISP!) to upgrade to SmarterMail 8.x, but that didn’t work. I even footed the one-time fee of $200 for the ActiveSync support for SmarterMail, but I couldn’t make that sync from Outlook on the desktop or the phone either.
Eventually I made an imperfect solution work: Hotmail. The nice thing about Hotmail is that it’s free for 25GB (yay webmail storage wars!) and it syncs contacts, mail and calendar items just like I want. Further, with some effort (vague error messages are not useful!), I was able to get Hotmail to pull in my personal email. And, after installing the Outlook Hotmail Connector (explicitly necessary because my Windows Live ID is not a @live.com or an @hotmail.com email address), I was able to sync almost everything, including the folders I copied from my giant PST file, via hotmail to both my desktop and phone Outlook. However, there are a few downsides:
- There is an intrinsic delay between when someone sends me an email and when it syncs to any device because Hotmail is polling via POP3. This polling is annoying and sometimes sends me directly to the web mail frontend where I can interact with my personal email directly.
- The Outlook Hotmail Connector sync’ing progress indication is terrible in that it seems to stack every time I press F9 (a bad habit from years of POP3 usage) and I can’t tell what it’s working or or when it will finish. Because of this, I’ve trimmed the set of email folders I sync to the ones I really use, using the PST file as an archive for days gone by.
- Hotmail does the right thing with the “Reply To”, but sometimes weird @hotmail addresses with random characters shows up in email threads, which breaks the fourth wall. That’s annoying.
- My RSS Folders don’t sync to my phone, which is a shame because I really loved having my Hacker News folder pinned to my WP7 home page letting me know where there were new items. None of the RSS readers on WP7 seem to work as well as a simple pinned email folder.
The good news is that this all works for free and my relatives continue to have working email. The bad news is that it doesn’t work nearly as well as the Exchange server I’m used to. Hopefully I will be able to revisit this in the future and get it working correctly.
I purchase all of my games via Steam now and install them as the mood strikes me. I love being able to reinstall Half-Life 2 or Portal on demand, then blow it away again when I need the hard drive space. Steam is the only viable app store for Windows right now, although I am looking forward to have the Microsoft app store in Windows 8.
I no longer maintain “backups” in the sense that I can slap in a new HD, boot from a USB stick and have my computer restored in 30 minutes or less (that never worked between WHS and Dell laptops anyway). I’ve had HD problems, of course, but they’re so rare that I no longer care about that scenario. Instead, what I do is keep all of the software that I normally install on a file server (the new job of my WHS box). If the file server goes down, then most of the software I install, i.e. Windows 7, Office and Visual Studio, is available for download via an MSDN Subscription. The rest is easily available from the internet (including Telerik tools and controls!) and I just install it as I need it.
Where Are We?
In order to free myself from any specific PC, I needed to pick a new centralized authority for my data: the cloud. The experience I was after for my PCs was the same one I already have on my phone – if I lose it, I can easily buy a new one, install the apps on demand and connect to the data I already had in Exchange, Hotmail, Skydrive, etc. Now that I’ve moved the rest of my world to Dropbox, I can treat my PCs and tablets like phones, i.e. easily replaceable. It’s not a perfect experience yet, but it’s leaps and bounds ahead of where it was even a few years ago.
Hardware and software comes and goes; data is forever.
Rant: It is just hard to believe how badly MS dropped the ball on the file sync story. YEARS ago they entered that space. At various times they had various products that did it in some way or another (Foldershare, Groove, Mesh). And all along they didn't throw weight behind it, and now others have surpassed them at large. Embarassing.
More constructive: I've got exactly the same email situation as you. Lots of family members on my main domain, so no way to get Office 365 for the whole domain. Here is how I ended up doing it, and that works very well. My main domain is hosted by Hotmail's custom domain thing, I guess the same thing you have. All my family members have hotmail mailboxes. I myself have one as well, but mine is upgraded to Hotmail Plus, which allows me to set an email forward to an external email address. I then have an Office 365 Small Business Exchange account. I only verified my main domain with Office 365, I did not change either the A nor MX record over to Office 365. If I remember correclty all I had to do to verify the domain was add some text record or something non-problematic to the DNS configuration. Once a domain is verified, you can have an Exchange account with an email address as the standard email address on that domain. When you send emails FROM that address, Exchange will do that. Of course, no email will be delivered to that address because all the DNS stuff points to hotmail. But then each Exchange account on Office 365 has another email address as a secondary address that is something like firstname.lastname@example.org. I then simply forward all email from my hotmail account to that email address. I have this setup working for many years already (previously used an Exchange account hosted by a different provider) and never had any problems with it, always been incredibly reliable.
Be warned about Outlook Connector for Hotmail. I tried that every couple of months, but that is the most crappy piece of software I've ever had. I ALWAYS ended up with either missing contacts (i.e. juste deleted) or all of a sudden all my contacts were duplicated or some other crap. Very, very unreliable piece of software, in my experience.
Wednesday, Dec 14, 2011, 12:18 PM
Wednesday, Dec 14, 2011, 4:36 PM
Gary Ewan Park:
I use Office 365, and I haven't transferred my whole domain over to Microsoft, that is still registered and configured through my web host. Instead, once registered, Office 365 tells you exactly what DNS records you need to create, and as long as you add these to your web host, everything such as Exchange, SharePoint and Lync, all just work.
Here is how I did it:
Wednesday, Dec 14, 2011, 11:03 PM
Great writeup - have you tried to use TrueCrypt and sync the encrypted drive with Dropbox (for your most sensitive data)?
Thursday, Dec 15, 2011, 12:23 AM
Thursday, Dec 15, 2011, 7:55 AM
Thursday, Dec 15, 2011, 7:56 AM
CloudBerry Lab team
Friday, Dec 16, 2011, 3:34 AM
Friday, Dec 16, 2011, 4:54 PM
Finally, when I want to keep sensitive data on dropbox I use TrueCrypt. I sync the portable app (so I can access it anywhere) and then I just create an encrypted "drive" on dropbox.
Thursday, Dec 22, 2011, 10:24 AM
Monday, Jan 23, 2012, 1:27 PM
Monday, Jan 23, 2012, 8:15 PM
Hope you can help me !! You are far more knowledgeable than I am so excuse using simple language. I run a small business from home. I installed dropbox on my laptop (Windows 7) and Ipad. I use Windows Office and all files from this suite get synchronised very well and without problems. I am also using Outlook 2003 so I have also moved .pst from Outlook 2003 into dropbox hoping to have it synchronised as well. I can worked OK on my laptop and everything is normal apart that I have noticed that the size of pst file gets smaller with the passage of time. It was approx 10.5 GB (yes it is that large) say 6 weeks ago and it is now 9.2 GB. IT kept dropping in size by say 0.2 GB per week. I noticed it as occasionally I also back up this file on an outside hard drive. I have not noticed what is missing from this fiole when it is getting smaller. I continue to work normally with my e-mails on Outlook 2003 and I did not spot any anomalies that would raise my alarm. This is however disturbing. I have read some bits on this blog about the need to get some extra software/solution while working with outlook within dropbox. Am I missing something or not understand mechanics. I consider working with .pst file from Outlook 2003 to be the same as working with files from Word or Excel. Do I not understand something ???
Monday, Feb 13, 2012, 1:03 PM
Monday, Feb 13, 2012, 3:07 PM