| written by Alex on Mar 12, 2019 06:08 |
 | |  | | ![]()  March 3rd, 2019: a random server hiccup
Hello.
Long time no see, I guess.
Well, we didn't lose much, there isn't much to see about what I did in these years. Unfortunately, I haven't been very active in terms of Anynowhere-related development. And even in general, not very active at all. It's a long story, so let's leave it for a possible later post.
Yesterday, Richard posted on my FB timeline to warn me that this site wasn't responding. Turned out to be just the WWW front-end, but the server was still up and running, in theory.
I opened an FTP session and yeah, that went just fine. Checked a certain lockfile to see if Postline (the forum engine) was stalled, and found it indeed looked so. So I tried forcefully unlocking it by emptying the file, but that didn't work: the site kept appearing unresponsive.
I went through SSH and logged into the console to run htop and see whether the Apache server was still running, and indeed, it was, along with the DBMS. Checked execution times for the various running processes and there were no anomalies, the server wasn't hung into an active loop of some sort, consuming CPU time. It was just idling, which was confirmed by the graphs on Linode's dashboard.
At that point, I felt a bit clueless and out of options, and went through the usual "er... let's try and reboot this thing and see what happens". In fact, it was running on autopilot, never having been re-booted since 212 days. The server's shutdown took over two minutes, though, which immediately felt cranky.
Once if finally rebooted (very quickly, in a matter of seconds, as opposed to the over-2-minute shutdown time), it was up again. Then I was reported that something was unfortunately missing. Indeed, it was. We lost a pack of forum posts. To be exact, 88 posts, ranging from message ID 136800 through 136887. All but 8 of the lost messages belonged into the Paranoia IC thread.
I'm sorry, but there are no back-ups of those messages that I know of. Their loss seems to indicate that the crash was due to the SQL DBMS. It never happened before through thirteen years of Postline activity, so I guess it's a very very rare occurrence, and it probably wasn't Postline's fault, hence not within my reach for a possible fix. I guess we'll just have to hope it doesn't occur again.
I can visually repair the damage by deleting pages and threads that appear empty now, because they refer to the lost messages, but I might not be able, unfortunately, to recover the lost messages themselves. I'll see if running a database repair, after a back-up, could turn out successful, but after a server shutdown, if those messages weren't yet committed to mass storage (and hence still in the server's RAM cache), they're lost for good.
I've also checked error logs and server-side activity logs in search for possible traces of an attack of some sort, but found nothing of relevance, so I'd exclude it was the work of an evil bot or unethical hacker. It was... just a gremlin, to say it in one word. Again, apologies, not that it was anyone's fault here: just an accident. Maybe one way to reduce the likelyhood of such an occurrence could be rebooting the server a bit more frequently than... once in a year, I guess. But it's just a wild guess.
Oh and sorry for being absent, I haven't much to say here, I'm rather fine at least physiologically. I'm depressed but that's been since 1991 so I got used to it. But it's something that keeps getting in the way, though. You know... | |  | |  |
|
|
