Database Updates

Photo provided by Baqash

As some of you might have noticed, I took AQ down for a bit today to do a massive update of the code. You won't see any changes--at least you shouldn't see any changes--because these were all "under the hood" updates. As I mentioned before, the database has been having trouble executing some of the queries efficiently, and the problem I've been having is that the piece of code I used for creating queries wasn't really designed for nitty-gritty details like telling the database precisely the order that tables should be joined or how best to execute certain queries. In the past, the database was generally pretty good at figuring out that stuff on its own without much help from me. Now.... it needs a little extra help, but the code hadn't been designed for that sort of thing.

So this update helps fix that problem. I completely replaced the code that put together queries, and it has a lot of options I can throw at it to suggest better ways of joining tables together. I can use subqueries, which I couldn't do before. I can change the type of join and force two specific tables to link together. I can throw parenthesis into the queries where needed. I can force the database to use certain indexes or ignore others as needed. (Well, technically, the old code could do that particular trick as well, but not as slickly or reliably as the updated code.)

All-in-all, it's a much more flexible piece of code that will allow me to fine-tune super complex queries like never before.

However, one thing it doesn't actually do is fix any slow queries. For a couple of reasons, really, but mainly because running a slow query from the live site doesn't necessarily show up when I run it on my development machine or vice-versa. The live site has traffic from lots of people at once which can cause locks and issues that won't show up on my development machine. Not to mention that hundreds of people using the live site will generate queries that cause problems that I could never hope to think of trying on my test machine.

So this update is just a stepping stone to faster, better queries. Now that it's running on the live site, I can monitor for problem queries and take the actual query and figure out where it's having trouble and figure out how I can modify it to work better--and the underlying code can be modified as needed to tweak the queries.

 Anyhow.... long story short.... progress is being made, but I'm not done yet. 

This, however, was a massive update that changed the very fundamental process of creating database queries, and it undoubtedly could have broken stuff. Please be patient--I'll try to fix any problems that crop up as quickly as possible. 

Thanks for being so patient!

Inaccessible logbooks

Photo provided by Thunderbird.

Yes, I know logbooks are still inaccessible. Yes, I'm still working on it. This issue with the slow queries has been an enormous headache for me. I lose sleep over it. I've been making progress, but it's been far slower than I had ever imagined. Basically, AQ broke the database. The size and complicity has overwhelmed the database.

But I keep getting emails and seeing posts about people thinking there's stuff in their logbook that they absolutely must get to and believe that there is no other way of doing it. For the most part, almost everything in your logbook can be accessed by doing an Advanced Search. From plants, to finds to unpublished plants, adoptions, and event signups.

Most of the logbook pages can be generated by searching for yourself as the planter, finder, attempter, carver, etc. Some pages, like donations, are a combination of those. (Search for boxes that you carved but are not the planter for.)

To see your unpublished boxes, just be sure to click the box for "Show unpublished boxes" under the "Other options" section. You don't even need to add planter information--AQ will only show you your own unpublished boxes anyhow. 

Admins can actually search everyone's unpublished boxes in this manner, however, so we have to include ourselves as the planter if we want to see our own unpublished boxes. But AQ knows that non-admins aren't supposed to see other people's non-published boxes and therefore will only show you the accessible ones--which are just your own. (Sometimes people report a problem with the location or are having trouble listing a box, so this gives admins a way to access the listing and see what's going on.)

If you need to see a list of your unlisted finds and/or attempts as well as your exchanges, you can find a link to those at the bottom of your profile. There a number of links there to download some of your data: finds, plants, trackers, exchanges, etc. They're in a CSV format so you should be able to import those files into a spreadsheet program and enjoy to your heart's content.

Anyhow, until logbooks are up and working again, be sure to check the Advanced Search page. It probably has a lot of options that you didn't even realize existed if you haven't used the page much before. It really is the ultimate search tool for letterboxes. *nodding* =)

And for the few things that aren't available from the Advanced Search page, there are still other places where the data can be accessed.

Anyhow... back to working on AQ again. Sorry again for the inconvenience! I'm really not asleep at the wheel, though! =)

AQ Completes Friendly Takeover of LbNA!

As most of you guessed, April 1st meant gags and jokes, and that hostile takeover of LbNA and blatant discrimination against LbNA members is... not true. In fact, it wasn't even the first time there was an alleged merger of AQ and LbNA. 

Unlike last time, however, this time around--the "merger" part is actually true. Kind of. The sites aren't merging--they'll continue to operate as two distinctly separate websites--but since Choi decided to retire from his day-to-day running of LbNA and nobody else had necessary skills and interest to take over, I've officially taken over that task.

It was last November or December or so when Choi first informed me that nobody had stepped up to take over, and it was then when he gave me the passwords for the server to download all the code and start going through it. There was a lot I needed to do before moving the site to the AQ server, and a few concerns I needed to deal with before I would move LbNA to share a server with AQ.

• The old server ran remarkably old versions of the database and scripting language. So old, in fact, that not even security updates were being provided anymore. So I needed to update LbNA to make sure it could run on the most modern versions of the software. In the case of the database in particular, I couldn't run two separate databases on the same server, and I definitely wasn't going to downgrade AQ's database just to be compatible to LbNA. Nope, LbNA's code had to be upgraded to work properly with the same software that AQ was running on.
• LbNA had a bad habit of generating those "too many connections" errors on the database. I needed to get to the bottom of that and make sure it got fixed. Not just because I wanted the site to run reliably, but also to insure it wouldn't take down AQ! (As well as other databases that run on the same server such as for Walking 4 Fun.) 
• I wanted a way to monitor for issues or problems. Over the years, I've developed a multitude of tools to help do this. The code will record errors, warnings and even particularly slow queries that might need my attention. I can easily check how long database queries take to run or the page takes to generate. And I wanted to include those same tools on LbNA.
  
• I wanted the site to be relatively quick and easy for me to update as needed. Maintainability is important. Every programmer has their own preferred styles and patterns that they use, and I wanted the code for LbNA to better fit my own mental model of how things should work. The more similar--internally, at least--the code worked like AQ, the easier and faster it would be for me to update later.
• And.... I just needed to understand how LbNA worked. I hadn't really used the site in years and there were numerous features and options that I didn't even realize existed much less how they all worked together. Doing a deep dive through the code would enlighten me.

Anyhow.... I spent a couple of months working on the site. The site turned out to be quite a bit bigger and more complicated than I had first imagined! A couple of problems I discovered I felt were so serious, I changed it on the live website immediately. Mainly a couple of exceptionally slow queries that could easily crash the database. (Or rather, cause a "too many connections" error because they took so long to process.) The website seemed to run a lot more reliably after that, but I know it still went down at least once after my tweaks. There were other slow queries that I didn't bother to fix because they seemed less critical and I didn't really want to spend a great deal of time fixing up a site that I was almost immediately going to replace with my own code anyhow.

Although I almost completely replaced the code on the backend of things, I tried to leave the front-facing user interface and functionality as unchanged as possible. There was some tweaking that happened. I re-created all the forms to fit the same patterns that I used on AQ--which also happens to be more "accessible" than the old layout used. (I use the term "accessible" in a technical sense--I'm referring to making websites accessible to those with disabilities.) Although I'm not aware of any blind people, for instance, using screen readers on LbNA, there's little reason not to support better access for them. (Accessibility doesn't just refer to blind people, though--it's a catchall term for all sorts of possible disabilities.) The site would still fail any serious accessibility testing that was done to it, but it's vastly improved over previous iterations.

Once everything seemed in order, I copied a test version of the site to the AQ server for a couple of months of testing.

Given the tens of thousands of lines of code I added, changed and deleted--I knew there would be a bunch of bugs and there were. I really hadn't had a chance to thoroughly test the code as much as I had wanted to, but Choi wanted to turn the lights off on the old server before it was due to renew for another year so I had somewhat of a deadline to get this done by. Technically, it wouldn't have been the end of the world if I hadn't finished before the deadline. At worst, LbNA would just be "offline" until it was ready to run again. But ideally, it would be better to keep the site running with no downtime, so that's what I was shooting for.

Testers were essential!

Special thanks to MichKathy and TrailMark for their testing. I gave them early access to the test site so they could update Clue Tracker to work with the updated site seemlessly, and they found a lot of the bugs that I really should have caught sooner!

By mid-February, I was largely ready to officially take the reigns of LbNA. Except.... I had plans to fly off to Africa to climb Kilimanjaro. It seemed like a bad idea to do a major move of the website then immediately fly off where I likely wouldn't be able to get online for two solid weeks. I should wait until after I got back.

And that's what I did. The day after returning to the United States, I immediately launched the Big Update--for Atlas Quest. That was also ready and waiting until I'd be around for an extended period of time to fix any bugs that certainly would pop up. Two days later, when the Big AQ Update seemed to be doing fairly well and seemed under control, I flipped the proverbial switch (specifically, the nameservers) for LbNA to point to the new server--the AQ server. (If you check, you'll find that LbNA and AQ now share the same IP address.) 

This was a massive update--even bigger than the "Big" AQ Update from two days earlier. But this one.... I had to do in secret because I wanted the "big announcement" to intersect with April Fools Day. So two days after a big update, I was doing another Big Update.

And the next several days was fixing bugs and issues and stuff, occasionally swinging back to the big AQ update to deal with something there. I'd often spend the morning fixing and tweaking the AQ update, then spend the afternoon and evening fixing and tweaking the LbNA update. It was a remarkably busy and exhausting week for me!

I also found myself extremely frustrated when I tried to use some administrative functions on my smartphone to check for bugs and delete spammer accounts on LbNA--which was never designed to work well with smartphones. I found myself so annoyed with that, I wound up spending most of a day updating LbNA to make it more mobile-friendly. Like with the accessibility thing, the update doesn't fix all the issues to make the website fully mobile-friendly, but it's a vast improvement over what was available before. Even Google thinks it's a huge improvement. I added letterboxing.org as a website to monitor for issues in their Google Search Console and on March 19th, Google reported just one page that was "mobile friendly"--zero percent of the pages after rounding off to the nearest two decimals. As of March 31st (the last day I currently have statistics for), Google reports almost 6000 mobile-friendly pages and just 92 still listed as "not usable." That not-usable number is going down, however, as Google re-checks previously-checked pages. I expect the number of "not usable" pages will continue to go down--although probably not down to zero. There are still usability issues to deal with--I only attacked the easier, low-hanging fruit in that regard. 

Then, once both websites were more-or-less running well and fixing problems wasn't occupying all my time, I started working on the April Fools announcement. Which meant a few gags and jokes for both websites. And... well, y'all saw how that went. I was working on that right up until I launched it less than an hour before midnight (Pacific time). 

Obviously, most of those announcements were jokes. The new crown and dagger icons will be retired, there will be no loyalty oaths to be taken, and pay-per-clue isn't on the table. The LbNA staff hasn't been fired--but there really wasn't any staff to begin with. I even left Choi as an official webmaster on LbNA so he can update passwords, delete spammers, etc. So he's still watching you. ;o)

So now that you're caught up to the present day, you might be asking yourself exactly what I plan to do with LbNA.

Crystal ball,
tell me the future of LbNA

And... for the most part, not much. Any big (perhaps controversial) ideas I come up with--I'd put on AQ. I don't really see any reason for me to recreate the wheel on LbNA every time I add a feature to AQ.

And, in fact, I think one of the core strengths of LbNA--where AQ can't even begin to compete against LbNA--is it's sheer simplicity. LbNA is perfect for people who just don't want a lot of bells and whistles. There aren't rabbit holes to confuse people like postals and LTCs, there aren't "trackers" or message boards to navigate. No marketplaces or online logbooks, no exchanges to track or dozens of icons to understand.

For many of you, you might consider the lack of those features as blasphemy, but I truly believe that letterboxing should be available to everyone--including those who prefer a more "old-tyme" letterboxing experience. After they develop their proverbial sea legs and dig into the hobby more, Atlas Quest might become more their style. Or perhaps they'll be perfectly happy to stick to the basics and stay away from the complexities of AQ--or maybe find a middle-ground between the two. A website like AQ can scare away the sorts of people who just want a simple, fun little hobby to participate in, and I'm grateful that there's a "safe space" for letterboxers who prefer the simple life.

So I have no ill-well toward LbNA or its members. Heck, that's where I "grew up" in my own letterboxing journey. Personally, I feel like I outgrew it, but bless all of you who haven't. =)

No blue diamonds for LbNA!

As a result, I have no plans to add features or functionality to LbNA. If anything, I'm more inclined to lean harder into its simplicity and remove features that I feel aren't essential or don't get used much. For instance, I removed the rating system already. It was interesting for me to look through the code and see how it worked, but it felt to me like it was added to "keep up with" Atlas Quest. And considering that LbNA doesn't get nearly as many visitors as AQ, I don't think it generated particularly good results anyhow. Ratings work best when a box can get a lot of ratings, but LbNA didn't really have the critical mass for that.

So... I decided to remove it. And now recording a find is one checkbox shorter and faster. There's one less "thumbs up" icon that people don't have to figure out or understand what its purpose is. Not mention there's the added benefit (for me) that there's less code to maintain and upgrade in the future.

At some point, I feel like I should really go through the entire website and figure out which features should be discontinued. Does the added complexity for the user make it worth the effort or distract from the more important details? I think back in my early days of letterboxing and what I wished LbNA could do: Run a simple search for boxes near my home that automatically removed all the boxes that I had planted or already found or attempted (and was certain were likely missing). That was all I really wanted the website to do. All the other search options--hike lengths, icons, etc. were fine, but I almost never used them and I certainly didn't need them. I kind of dream of taking LbNA back to its glory days where it might have been simple, but it was easier to understand and figure out how everything worked. I'd like for LbNA to distinguish itself from AQ--not try to copy it.

An obvious question many of you might have will be if I plan to merge the two websites--or at least merge the two databases that run it so one account can be used on both websites. Recording a find on one account will also record it on the other account automatically. It's tempting, but for now.... no. That would take a horrific amount of time and effort that I'm not ready to commit to, so they'll continue to stay as two completely independent websites. Maybe someday I might add some options to "link" accounts so recording a find or attempt on one site will automatically post it to the other, or recording a plant on one site will automatically post it to the other, etc. But even then, I still imagine it running as two completely separate, independent websites. Just two websites that happen to talk to each other occasionally and share information. I have no plans to rush out to get this sort of thing done immediately, however.

One feature I might implement sooner rather than later, however, is the ability to hide plants that are hosted on the other website. I actually wanted to create this on AQ since the day I started AQ--a way to not display LbNA boxes in the search results. Not as an "anti-LbNA" option, however, but as a way to make it easier for people to run searches that don't show the same plants on both websites. But I felt certain if I ever tried to implement such a search option, I'd be accused of being "anti-LbNA." Logically, I felt LbNA should have a similar "hide AQ boxes" options for the same reason. And then those who use both websites regularly to search for boxes can get all of the LbNA and AQ boxes with no overlap between the two sites. And it would be trivially easy to implement such a feature since the two websites don't even have to talk to each other to make it work. AQ already knows which clues are hosted on LbNA and LbNA already knows which clues are hosted on AQ.

Now that I'm running both websites, it seems ridiculous to make such an accusation. And since I'm running LbNA, I can easily make sure a "hide AQ-hosted clues" option is available on LbNA to balance things out. So this feature is one of the most-likely ones to come out of my running LbNA since it seems very useful and helpful--at least for those who use both websites. (Although if absolutely nobody likes the idea or wants it, I certainly wouldn't put any effort into it!)

The Tanzanian flag--just because
I have the license to use it and
I spent two weeks there recently. =)

You'll likely see me continuing to update some of the icons to use the same ones as AQ. There are three reasons for this: 

1. I already paid and purchased a license to legally use those icons, and the license allows me to use them on any and all websites I run so there's no problem with me using them on both sites. 
2. I'm not entirely sure of where the already-existing icons on LbNA came from or whether they're actually legal (or not) to use. It would really suck if someone came after me because one of their icons was being used without the proper rights. 
3. And finally, using the same icons on both sites can make it easier and faster for people to figure out how the "other" website works when they are already familiar with some of the icons.
   

Anyhow, that's how I imagine the future of LbNA, but I'd be curious to hear what all of you think or how you would re-imagine LbNA. But regardless of how much our opinions intersect (or not), I will do my best to run LbNA like it was my own creation--to care for it, keep it running, and do what I feel is best for LbNA--even if we might not always agree about precisely what that is.

But in broad, general strokes, I don't really plan to change much of anything. So far, my focus was mostly on the security, reliability and maintainability of LbNA.

Now that the two websites run on the same server, there will be rare times when both websites go down or become inaccessible if there's an issue with the server, so in that sense, reliability overall will go down. However, when I take down a website for a Big Update--those will be planned ahead of time and will never be happening at the same on both sites. I'd like to make sure at least one of the websites is always running at all times. If that's not the case, there's likely something out of my control going on. For the record, this doesn't necessarily mean there's something bad happening. Sometimes my hosting provider is doing updates on their own equipment and things may be temporarily unavailable--but the server is just fine. Sometimes I'll update software and it requires the entire server to be restarted--and that will make both websites temporarily inaccessible. I'll try to limit those times as much as possible but they are, overall, fairly rare.

So that's my own vision for LbNA. Feel free to contact me through AQ or LbNA if you wish to express your own thoughts on the topic. 

Magic 8-Ball, inquiring minds want to know:
Will LbNA continue to thrive for years to come?

"Without a doubt"

AQ Completes Hostile Takeover of LbNA!

Your AQ-loyalty will be
marked with a crown!

Yes, it's finally true. I've taken over LbNA. Some of you have noticed the subtle changes happening over on that other site. Suspicious icons that looked awfully similar to those used on AQ. Weird bugs suddenly showing up. There are, in any major coup, kinks to be worked out, and that's been keeping me pretty busy.

I've also been making some tweaks to AQ as a result of the takeover, which I've posted about in more detail there. The main changes you'll see are the new crown and dagger icons by member trailnames to mark AQ-loyalists (the crown) and LbNA loyalists (the dagger). More tweaks will be forthcoming, but that's the main one you'll see for now.

Suggestions and comments are welcome! And if you don't like any of my changes, I'd normally tell you to just use LbNA instead, but I guess that won't work anymore. So just start geocaching instead!

Happy trails!

LbNA-loyalists will be marked with a backstabbing knife

 

 

The Next Big Update Has Arrived!

T-Bone and Tic Tac of Happy Hyper Hikers

Nearly a year in the making, it's finally arrived! I'm not going to list every tiny thing that this update includes, but rather just a broad overview.

Before I start, however, just a reminder: If there's a page that looks a bit wonky, try clicking Ctrl-F5 (or equivalent) to refresh the page and see if that fixes it. There are changes to the CSS and if your browser is using a cached page, it might not look quite right.

The biggest part of the update are event listings. I've pretty much completely rewrote the entire event sub-section. I scoured the message boards and my archived mail looking back over a decade's worth of comments and suggestions dealing with event listings and incorporating as many of them into the update as possible.

Some of the most noticeable improvements with regards to events include:

• Attendees can now list multiple items for the potluck--so if you're bringing multiple items, you can list them all in the proper categories.
• Event admins can now request specific potluck items.
• Attendees who live near an event can list if they have space available to host out-of-area attendees, or those who rent a hotel room or campsite can request "roommates" to help split the costs.
• Timezones are better tracked, so virtual events should list the start and end times for your local time zones automatically. *knock on wood*
• Each event can now be posted to, so if you have a question or comment about the event, you can post it directly to the event page.
• Waitlists have been greatly improved. Once an event is full and people start adding themselves to the waitlist, event organizers can manually promote waitlisted members to a signup status (or demote them if necessary). In fact, you can create an event with just yourself as the only signup and a limit of 1, then everyone who signs up is automatically waitlisted. Which can be useful for events that cost a fee. As fees are paid, you can promote the waitlisted member into a signup. So it is possible to have more people signed up for an event than there is "space" for. Also, once a waitlist is created, people won't automatically be moved into the signup position if space becomes available. Event organizers decide who and when people are moved between the signup list and waitlist.
  
• You can signup (or waitlist) sub-accounts at the same time as yourself.
• There are now options to "watch" and "ignore" events.
• All types of boxes can now be listed as "boxes at an event" (not to be confused with "event boxes" which is a specific type of box). So event organizers, for instance, can specifically list each traditional box planted at the park--even those that were never planted specifically for the event. Personal travelers, hitchhikers, etc. Anything you bring to the event can be listed as a "box at the event" to make it easier for people to record the finds later.
• You can add events to your calendars. (Google and Apple calendar types, at least.)
  

I think that covers most of the changes regarding events. There's a lot of stuff that changed there, however, and this is certainly not a completely list. 

There's another feature somewhat related to events which involves the massive number of notifications people receive for large events. A hundred people finding a hundred boxes and leaving a comment on every one of them would generate ten thousand AQ mail messages. So event and box notifications now are collected for a period of time. Notifications of finds and attempts on boxes, for instance, now go into a waiting queue and everybody who found a specific box/series will have their reports merged into a single AQ mail message. So if you have 100 plants at an event and 100 people recording finding each of them, you'll no longer get 10,000 AQ mail messages--you'll only get 100 at most. (The number of boxes planted rather than the number of boxes planted multiplied by the number of finders.)

Similarly, the same thing happens with event notifications. Signups, dropoffs, new posts, potluck item changes, etc.... All of the notifications for a specific event will be merged periodically into a single message instead of dozens (or hundreds) of them.

The primary reason for this change is to make one's AQ mail more manageable, but it also helps ease the database load on AQ as well. The downside, however, is that you will no longer get immediate notifications whenever someone records a find or attempt or there's an update to an event. There's a time delay. The default delay is 12 hours, so if someone records a find on your box at 9:00am, AQ will wait until 9:00pm, check all of the pending notifications to you about that box that have occurred between 9:00am and 9:00pm and send them all as one AQ mail.

Premium members can shorten this frequency to as little as 5 minutes--still not immediate, but relatively close to it. If you have a lot of boxes or have planted for a large event with hundreds of people, you can increase the length to as long as a month, if I remember correctly. For those hosting large events, it might be worth lengthening the time to reduce the number of AQ mail messages you get--at least during the height of the event--then return it to whatever value you prefer a few days after the event has ended and most people have logged their finds.

You can adjust this setting in the Email & Notification Preferences settings, in the "Notification Timings" section. I suspect many of you aren't going to like this change (I figure it'll likely be the most controversial of the changes), but for those who host especially large events, you'll find it very convenient. If you have a question about finding a box and need an immediate answer, it'll be best to contact the owner directly rather than by recording a find/attempt since direct messages won't be delayed.

Now off to a completely different sub-system....

For those who use the weather widget, you'll find it too has been completely rewritten. The only reason for this update was that the API I used was set to expire at the end of this month and it was going to stop working. I thought about retiring it completely, but eventually decided to use the National Weather Service API (which is free and doesn't have ridiculously small quotas). The downside, however, is that it only works for the United States, but since most of those who use it live in the United States, hopefully that's not a problem. =) 

Especially with regards to the event and weather sub-systems, a Ctrl-F5 to refresh the CSS might help. Your browser will eventually pick up the new CSS on its own, but forcing a refresh like that will just speed things up.

All the other changes are relatively minor things. Those are the two sub-systems that I basically rewrote from scratch. Off the top of my head, I don't really remember what else I changed, but you're bound to find some minor tweaks here and there if you look closely.

If I said anything about a tweak or change being in the "Next Big Update", this was it. It's live now.

This is an absolutely massive update, and while I have run thousands of tests, there are still likely to be a few bugs that slipped through the cracks. Please be patient as I fix those.

Thanks again for putting up with me! If you have any questions, comments or suggestions, you know where to find me. =)

Burning Feet (and Sadie)