Thursday, March 05, 2009

Apparently, I've been a bad, bad boy....

Some of you might have heard me eluding to problems with atlasquest.org. What is atlasquest.org, you ask? It's a domain name I own but generally haven't had much use for. I bought it with the theory that I'd rather own it and not need it, than to want it and later find out I couldn't have it.

If you do go to the website atlasquest.org, you'll see that it is indeed a real, honest-to-goodness website. It's not very useful for anything, but it does serve one important use for me: backups. Each night, Atlas Quest dumps the database into files. Backups are extremely important for anything important because computers aren't perfect. Hard drives crash, things happen.

So I don't really want to store the backups ON the main Atlas Quest server. If something should happen to that server, not only would the website go down, but the backups for it would be lost as well! Definitely not a satisfactory solution.

So that's where atlasquest.org came into play. I picked a cheap web hosting provider. To make things work, they kind of like to have a domain name to work with. So I "sacrificed" atlasquest.org. That account is hosted on a completely different server, in a completely different state, by a completely different company. And each night, after Atlas Quest backups up the database for the day, it FTPs everything that needs to be backed up to atlasquest.org.

This includes the nightly backups of the database, along with photos that were uploaded during the day--including calendar submissions which get their own special place.

I'm now storing about 6 gigs of data at that location. Which isn't really a whole lot. When I signed up, I was given a disk quota of 10 gigs. I knew I'd outgrow the space eventually, but I figured I still had at least another year or two before I had to worry about it.

Until last week, when the company hosting my account sent me an e-mail saying that it appeared I was using the account to backup data, and accounts were not allowed to be used for that purpose. Actually, they said I could not back up data that was "personal and non-web-related." It came as rather a surprise--I don't remember seeing that clause in the terms of serice when I signed up a couple of years back.

I don't really consider the data "personal" and it's very much "web-related," so I tried talking them into letting it slide. But alas, no, they have standards or something and the terms of service must be complied with. I have to get rid of the data from their servers. I've been a bad boy, and they no longer want me around. =(

So, I'm in the process of setting up a new system for backing up data. Turns out, there's this interesting concept called "cloud computing," and it'll actually cut my backup expenses by about 80%. And there's no limit to the amount of data I can back up, so that won't be a concern. (They will charge more as I use more space, however, but the rates are quite reasonable.)

The new backup location doesn't require me to "use up" a domain name, so atlasquest.org will once again be free for me to use and abuse as I wish. I'll probably move it to the main AQ server and just have it redirect to atlasquest.com. At least until I find something better to do with the domain.....

You realize you folks have uploaded over 60,000 images to Atlas Quest? Some of them have been deleted over the years, of course, but there are a lot of images. Far more than I ever imagined I'd have to worry about less than five years after introducing Atlas Quest. Each image is stored three times--a thumbnail version, a "normal" version, and a "large" version. Most of the large versions I've been deleting from the main AQ server because of space concerns and because most of them are all that important, but the backups still had the original large versions. (For those clues that weren't properly marked as a "photo clue," that's how I was able to retrieve the large versions of the photo.) So with 60,000 images uploaded, times three for each of the three versions of it stored--that's 180,000 images.

I've at the library now (broadband connection--you don't deal with 180,000 images on a dial-up connection!)--and have been here for four hours. So far, I've retrieved 43,200 images. About 10,000 images per hour. At this rate, I'll need 18 hours here to get all of the images off of the backup site. But I'm still being productive. As I take data off the old backup site, I'm working on code to get Atlas Quest set up and working with the new backup location. FTP won't cut this time around. I'm learning how to get and set secure web pages and learning the API the new hosting provider has for storing and retieving files.

Which is all "back of the house" stuff that none of you will even notice. =) That's okay--I don't expect anyone to notice. In this case, if I do my job RIGHT, it means you WON'T notice. The one thing some observant members might notice is that atlasquest.org will no longer be an official stand-alone site like it is now. But it's not a site you'd normally visit anyhow.

But I do find the sheer number of photos I'm having to deal with somewhat astounding. It used to be I could copy them all to my personal computer in mere minutes. When I first started backing up images and the database, it filled perhaps 100M of space. Atlas Quest has come a long way since those humble orgins, and just another example of the tweaks and modifications necessary to keep AQ running smoothly despite the increase in size and usage. =)

8 comments:

1SG said...

Gee instead of just using dot org for a sandbox you could duplicate this site there as it it today.

Then leave all of it exactly as it is except continue to update the data files.

Now when folks don't like upgrades and improvements like bottom menu bar pull ups then can use the old faithful dot org site.

No I am not really serious

Grumpy Grinch said...

Our son Christopher has spent a large part of his software development career in developing backup software. I have taken the liberty of sending him this blog for his edification and amusement--and so that he can see that in a small way, our goofy hobby and his interests meld.

Anonymous said...

I was wondering if I could search for misc. photos that I have uploaded by my username? Maybe I could go find a few to delete , , ,

~resQlou

Anonymous said...

Thank God for all of us you love what you have and do through AQ!

Sugarpops

Carolina Wine Trail said...

Thanks for All you do behind the scenes, Ryan, to keep the site running smoothly. Really never thought of how much work you probably put into all that.

Anonymous said...

Thanks, Ryan -- you do so much for this group it's astounding. Wishing we could afford to send you to Hawaii for a week or something, when you get it all done. Even the beach in Washington State would be a refreshing change from the library. God bless you!

hansenclan

Anonymous said...

How many of those downloadable images are attributed to Virtual Letterboxes? I'm guessing about 75 to 80%.

Anonymous said...

Hansenclan,

Did you say done? AtlasQuest will never be done. It is organic, and technology never stops developing. This site will continue to provide Green Tortuga with endless work, as long as he is interested to continue doing so. Probably not what he wants to hear, but there you go.

Celtic Morphos