Major Announcement! And totally not click-bait!

So.... since the Last Big Update, you might be wondering what I've been up to. Mostly still working on speeding up slow queries. That Last Big Update largely fixed the worst of the issues, but that didn't mean there wasn't still room for improvement and I've continued to improve.

I wish I could play the Jaws music here. The major announcement
is ominous, but good in the long run! Photo provided by
Silver Eagle while snorkeling with whale sharks in the Maldives.
(I am SO totally jealous too!)

The new and improved code I wrote for Atlas Quest has been simply awesome. I'm very, very happy with how it's been performing, and once AQ seemed to be running well, I copied it over to LbNA where I fixed up some other slow-ish queries. Those weren't as problematic since not as many people use that website and the queries are a lot simpler anyhow, but since all my websites run on the same server, keeping things running fast on other databases will also help the AQ database run faster and better as well.

Then I copied the new query code over to Walking 4 Fun and repeated the process, improving the slower queries that I identified. The busiest page on W4F, the "My Walk" page, I'm happy to report, used to take--on average--about 1 to 2 seconds to run all of the queries needed to generate that page, and it's now down to about 35 ms. I just opened the page for myself and all of the queries--combined--took precisely 38 ms to run. Or, in other words, 0.038 seconds. Massive improvements!

And like I said before, that does help the AQ database run better as well. If you happened to run a search just as someone else was trying to look up their daily steps on Walking 4 Fun, your query might have run a second or two slower if the database was bogged down handling these other queries. So fixing these types of things are really good for all the databases on the system.

And then I've been hopping on and off each of the three major websites that use the database each day, looking for additional other queries that can be optimized. Some of the optimizations might not even seem like they're worth the effort. One query I improved a couple of days ago on LbNA now runs ten times faster! Which sounds impressive, until you realize that it originally took 0.20 seconds to run and now it takes 0.02 seconds to run. It is ten times faster, but it was already pretty fast to begin with.

But still, I've been making these kinds of "modest" improvements for weeks--months even, if you include the tweaks I made before the Last Big Update, and collectively, it starts to add up. A query that runs 1/10th of a second faster 10,000 times per day means the database doesn't spend 1000 seconds (about 16 minutes) every day running that query anymore.

And occasionally, I still stumbled onto an elephant of an improvement. Someone runs a search with weird parameters that take 20 seconds to run and I can tweak the query to make it run in less than a second. For that person, they'll see a dramatic improvement if they ever run that search again, and for everyone else, their queries won't be hung for 20 seconds while dealing with that other one. I'm not finding many of these kinds of improvements anymore, but it's always satisfying when I do. =)

But the database is still suffering from one issue that all the speed improvements in the world won't fix which is... a lack of memory. The database runs on a server with a mere 10GB of memory and the databases it uses are quite a bit larger than that. There's sometimes trouble getting all of the indexes to fit into internal memory--which helps queries run fast--and there's not really any way for me to optimize that. Basically, it sometimes needs to look up an index from the disk, which is slow. It's kind of like the difference between looking up a term in an index of a book or having the index memorized and being able to skip the process of actually looking it up because it's already in your head. 

And the next big improvement for the database is.... a bigger server! Specifically, a server with more RAM. So I started looking into upgrading the server. Originally I thought I'd just upgrade to a bigger VPS which is relatively painless and easy, but it turns out if I upgrade to a dedicated server, I'd actually get even more RAM, more disk space and at a lower cost than keeping it as part of a VPS!

So.... within the next week or so, I plan to move AQ (and LbNA, and W4F, and all of my other websites) to a fancy, new dedicated server.

Unfortunately, this will require a relatively significant period of downtime. The IP address of the websites will be changing, so there's also the whole issue of IP address propagation as well. Even if the new server is up and running fine, you might have trouble accessing it until the new IP address gets to your computer. For most people, this usually updates within a few hours, but some people in the past have had systems that took a week or longer before picking up the new IP address. It's not something I have a lot of control over either.

Will it be worth it, asks Wise Wanderer? I think so. *nodding* =)

The support team where I host the server reports the migration will take "24 to 48 hours"--that's the period of time that AQ (and LbNA) will be off limits to everyone. Once the migration is complete, the websites should be available within a few hours after that. Technically, the old servers will keep running during the migration, but any new letterboxes that are listed, messages posted, AQ mail sent, etc. won't be copied to the new server. It'll just appear like they were deleted.

I'm going to try seeing if I can just prevent people from logging in. If you can't log in, you can't really add any data to the database that won't get copied to the new server. This way, at least you can still run searches and look up publicly available boxes and message boards, even if you can't see the restricted ones that you would otherwise qualify for. I'm not actually sure how simple this will be since I never really designed AQ to keep running with the login options turned off. If it starts becoming too complicated, I might just choose to put up the "Under Construction" theme and make the entire website unavailable until the migration is complete.

But in any case, this will put both AQ and LbNA (and W4F for those of you who use it) out of commission for at least a day or two. Looking at the stats on AQ, mid-week (Tuesday through Thursday) is the least busy time for the website, so I'll probably try doing the update starting Tuesday of next week, and it should be available again by Thursday.

That's my tentative plan. It could change. When AQ and LbNA are down, I'll post updates to the AQ Facebook page at https://www.facebook.com/AtlasQuest so make sure you're subscribed there if you want updates.

Even after the transfer is complete, there will likely be times when I need to restart the database server. With more memory, I'll have to update the database configuration options and I'll watch to see how it performs for a day or so, make some tweaks, restart the server, and repeat. So there will likely be some hiccups for the first week after the update as I fine-tune the configuration settings. Once it's all done, however, I hope AQ (and LbNA) run better than ever! =)

The new dedicated server I'm eying, btw, has 16GB of RAM--which almost doubles the available amount of memory for the database to use. It also comes with a lot more disk space as well, so I plan to stop AQ from deleting the high-resolution photos from the photo albums. At least until disk space becomes more problematic again. Do not expect that the high-resolution photos will always be stored forever, though. This new server has about 3-4 times more disk space available and I don't really have 3-4 times more data to store, so I'll just allow the high-resolution photos to linger longer than before.

And thanks to all of you premium members (and LbNA donors) who pay for all these expenses! I actually started the premium memberships when AQ was on a shared server, fearful that I'd have to move to a dedicated server and I didn't have the money to afford one. Turned out, I never did upgrade to a dedicated server once I learned about the existence of VPSes (Virtual Private Servers), which basically acts like a dedicated server but is actually hosted on a server that shares resources with other websites and accounts. (And theoretically, one of those running a particularly slow query could cause other sites like AQ to be more sluggish as well, but I have no idea what the other websites are or how well they run.) The extra money premium memberships generated beyond the cost of the VPS wound up being my income since a VPS was far cheaper to run than a dedicated server.

I've upgraded the VPS a number of times over the years, increasing the memory and disk space along the way, so it's not nearly as cheap as when I first moved to the VPS, but it feels like this website has really come full circle now that it's more economical to run on a dedicated server than a VPS! At long last, the original reason I started the premium membership--to pay for a dedicated server--is finally coming true. =)

In completely unrelated news.... you'll still find eclipse glasses in the AQ Marketplace. If you want to watch the eclipse on October 14th, be sure to order them while there's still time! At no point anywhere in will it be safe to directly view the eclipse without eye protection and everyone in the contiguous United States can see it (weather permitting). And every time there's an eclipse, there are always stories about price-gouging by desperate people in the days leading up to the eclipse. Be prepared! (I won't price-gouge you if you wait until the last minute--I just won't sell them at all because I can't be certain that they would arrive to you in time.)

Happy trails! And sorry in advance if the downtime next week causes you any inconvenience!

Make sure you have your eclipse glasses before October 14th!