Better Backups: Pick a System

You have backups, right?

— SuperUser’s chat room motto


This started out as an intro to bup. Somewhere along the way it underwent a philosophical metamorphosis.

I’m certainly not the first person to say that backups are like insurance. They are a bit of a hassle to figure out which one will work best, you set it up and forget about it, and hopefully you won’t need it*.

Many moons ago, I had backups taken care of by a a simple shell script. Later, this got promoted to a python script which handled hourly, daily weekly and monthly rotation; and saved space by using hard links (cp -al ...). It even differentiated between local and remote backups. That was probably my backup zenith, at least when time and effort are factored in.

Really, the more sensible approach is rather than reinvent the wheel, use an existing tried-and-tested solution. So I moved to rdiff-backup and it was good; being simple it meant I could set up ‘fire-and-forget’ backups via cron. I was able to restore files from backups that I had set up and then forgotten about.

With the recent expansion of the fileserver ongoing, now’s a good time to take stock and re-evaluate options. I have created a Xen DomU dedicated to backups (called pandora, aptly) with it’s own dedicated logical volume. From here, I need to decide:

1) whether to keep going with rdiff-backup or switch to eg bup or borg
2) figure out if different machines could use different schedules or approaches (answer: probably); and if so, what those would be (answer: …)

I don’t want to spend too long on this — premature optimisation being the root of all evil — but the aim is to create a backup system which is:

  • robust
  • reliable
  • maintenance-minimal

: If you *do use your backups or insurance a lot, it’s probably a sign that something is going wrong somewhere

File creation time on ext4 (Linux)

tl;dr: since coreutils stat does not show file ‘birth’ time, use debugfs -R stat <inode> FS

I was curious as to when I wrote a particular time-saving script, so I figured I would look up the file creation time:

$ stat ~/scripts/goprofootage.sh
 File: /home/robert/scripts/goprofootage.sh
  Size: 1001            Blocks: 8          IO Block: 4096   regular file
Device: fe01h/65025d    Inode: 792618      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 1000/  robert)   Gid: ( 1000/  robert)
Access: 2018-01-07 08:23:04.816666962 +0000
Modify: 2017-05-13 19:09:30.760094062 +0100
Change: 2017-05-13 19:09:30.760094062 +0100                        
 Birth: -

Err, well. No birth date? ext4 does support file creation timestamps, so it’s just a simple matter of getting at them.

Enter debugfs, part of e2fsprogs (at least on this Arch install). We can stat an inode to get a creation time:


$ stat -c %i ~/scripts/goprofootage.sh
792618
# debugfs -R 'stat <792618>'  /dev/mapper/840ssd-home
debugfs 1.43.7 (16-Oct-2017)
Inode: 792618   Type: regular    Mode:  0755   Flags: 0x80000
Generation: 3863725318    Version: 0x00000000:00000001
User:  1000   Group:  1000   Project:     0   Size: 1001
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x59174bda:b53875b8 -- Sat May 13 19:09:30 2017
 atime: 0x5a51d8e8:c2b56548 -- Sun Jan  7 08:23:04 2018
 mtime: 0x59174bda:b53875b8 -- Sat May 13 19:09:30 2017
crtime: 0x58efaf27:2234f628 -- Thu Apr 13 18:02:31 2017
Size of extra inode fields: 32
EXTENTS:
(0):5096134

Or, if you’d rather combine the above into a one-liner (NB needs root):

 # debugfs -R "stat <$(stat -c %i ~/scripts/goprofootage.sh)>"  /dev/mapper/840ssd-home 2>/dev/null  | grep crtime | cut -d ' ' -f4-9
Thu Apr 13 18:02:31 2017

Quick beets import tip

Starting with beets to manage and organise your music library? Read the ‘getting started‘ guide? An additional quick tip:

Import your first few albums individually using

beet import -t $album_directory

The -t flag is for timid(ly).

Why? If you’re like me, you might not be in 100% agreement with how MusicBrainz represents the match metadata; and -t will ask for confirmation which you can either accept ([Enter] or A) or reject (U for ‘Use as-is’).

If the first few matches are fine, you can drop the flag; if not, you can figure out how to finesse it to import files to your liking via beets‘s excellent plugins.

[Fixed] No LED light on Logitech K750 Keyboard

K750 charging under a lamp

tl;dr: Use a combination of a bright light and the reset technique (off ? CAPSLOCK + some keys ? on)

Background

A few years ago while on holiday in sunny SC — a lovely place to visit, incidentally — I took the opportunity to purchase a Logitech K750 keyboard. The K750 was is a ‘solar’ keyboard- no changeable batteries, only solar cells to power it. I had been frustrated by other keyboards which needed battery changes timetabled in accordance with Sod’s Law.

They keyboard got passed on to my folks and all was well. However, since my dad passed away my mum has not used the computer as much, and so the lights in that room haven’t been on. When I went to use the computer there recently the keyboard was completely dead. Not even the red ‘sad face’ LED would light even if held close to a light source, which would normally elicit some response.

Leaving the keyboard in a decent amount of ambient light for a few days seemed to do very little to help.

Revivifying a K750

Enter Nut and his Tech, wherein they describe the reset procedure:

1. Turn off the keyboard.
2. While holding onto CAPS lock, keep pressing a few keys for the next 5 or more seconds.
3. Turn on the keyboard.

I did the above and held the K750 very close to a fairly bright bulb, and the LEDs sprang to life!

First the green happy face when still next to the light, then the red unhappy face when I took it out of full illumination.

So it works, for now. I was even able to type from the next room over where it sits charging beneath a mini anglepoise-type lamp.

The Sam Vimes Theory of Economic Injustice

The reason that the rich were so rich, Vimes reasoned, was because they managed to spend less money.

Take boots, for example. He earned thirty-eight dollars a month plus allowances. A really good pair of leather boots cost fifty dollars. But an affordable pair of boots, which were sort of OK for a season or two and then leaked like hell when the cardboard gave out, cost about ten dollars. Those were the kind of boots Vimes always bought, and wore until the soles were so thin that he could tell where he was in Ankh-Morpork on a foggy night by the feel of the cobbles.

But the thing was that good boots lasted for years and years. A man who could afford fifty dollars had a pair of boots that’d still be keeping his feet dry in ten years’ time, while the poor man who could only afford cheap boots would have spent a hundred dollars on boots in the same time and would still have wet feet.

This was the Captain Samuel Vimes ‘Boots’ theory of socioeconomic unfairness.

— Terry Pratchett, Men at Arms

(brought to mind by recent musings on cheap hardware)

On Cheap Hardware and Misbehaving Monitors

tl;dr: Cheap stuff can malfunction in unusual ways

I have a history of buying cheap hardware out of necessity. This has not changed in more than ten years. I wish it was different; but scrimping and saving and buying cheaper, off-brand is the only way I can afford to do things. That said, I am lucky be able to afford what I can afford, it is definitely more than some who are less fortunate.

I have a triple monitor setup, which some might call an unnecessary luxury; but with my eyesight being what it is more real estate means being better able to both fit a reasonable amount of things on the screens at a reasonable visibility.

Each of the three displays is 27″ and capable of a resolution of 2560×1440 at 60Hz. They are all also ‘cheap’ (relatively speaking) Korean knock-off imports- brands such as PCBANK, Crossover and DGM- the last of which I have at least come across before, in the form of the cheap TFT monitors referred to in the post linked at the start. At least they have a track record!

The third thing that these monitors three have in common is a tendency to malfunction; each in different ways. Taking them in the order of purchase, which dates from about five or six years ago to less than six months ago:

  1. PCBANK – The only one I bought new. Makes an audible noise when displaying white or mostly white (eg a a mostly text web page, such as Wikipedia). Also occasionally displays a distorted image for half a second before coming to its senses.
  2. DGM – intermittently blanked itself under Windows
  3. Crossover – If turned off or doing into standby, won’t turn back on for 5-20 minutes. Possibly a backlight issue. As a bonus, it was bought second hand and came with its integrated plastic monitor stand removed; the only way I could see to reattach it would be to separate the plastic casing, which I got halfway to doing before I figured I would purchase the monitor mount[s] I had been intending to for years

In addition, the transformer for the PCBANK or DGM monitor developed a fault which made my speakers screech like a banshee requiring me to take the highly-technical step of moving the transformer further away. Then it started buzzing itself, and needed replaced.

Those transformers are beasts, incidentally- 24V 5A or 120W.

These faults range from irritating to intolerable, but I reckon that if I had three monitors from reputable manufacturers at a commensurate price, I wouldn’t have the same issue with malfunctions.

Wishlist: An ncurses interface to Logitech Media Server

I’ve used Icecast2 + mpd for ages, but in the last couple of years I’ve been using Logitech Media Server (LMS) to get synchronised music playback. It’s mostly good, though quite memory-hungry and a bit picky about permissions. The control of playback via web interface is functional, but I find it clunky. Given there are CLI and json interfaces (even wrappers!) to LMS, it would be nice to have an interface in the style of mpd. Such a project is beyond me time-wise, alas…

Intermittent Monitor Blanking

tl;dr: Using a better-rated HDMI switcher improved the situation

For the past… maybe six months to a year one of the monitors I have has been ‘blanking’. That is, it will display properly most of the time, but intermittently go blank for around three seconds before the display reappearing. Strangely, the behaviour only occurred:

  • under Windows, not Linux
  • when using an HDMI switcher

For whatever reason, the issue got much worse in the last couple of weeks, where instead of happening once every five to sixty minutes (or sometimes not at all), it was happening back-to-back, several times per minute with an intolerable frequency. The situation was untenable.

I wondered about interference from other nearby cables, but adjusting them didn’t make any discernable difference. The monitor was running at 59Hz, but stepping it up to 60Hz made no difference either (and Windows 10 set it back to 59Hz). If anything, increasing the frequency may have made the problem worse.

So I figured the issue was the cheap HDMI switcher that I have. I don’t quite recall the circumstances of buying it (never a good sign), but it would have been cheap and therefore unlikely to be rated for 2560x1440x60Hz. After replacing the switcher with one with was rated for ? 2560x1440x60Hz, the issue seems to have mostly resolved- there was *one blank, but I think I can tolerate that.

For anyone else having a similar problem, allquixotic had the following to suggest for potential causes:

Periodic blanking could be: loose cable/connector; bad HDMI port; bad HDMI codec on the GPU/mobo; EMI interference on the cable (e.g., crosstalk by contact with another cable); or, yeah, insufficient cable quality for the data quantity being pushed… also could be pure software

you know, it could simply be Windows driving more data to the monitor; have you verified Windows isn’t overclocking the monitor beyond 60 Hz?

or it might be a GPU issue — Linux tends not to even use the GPU very often unless you’re decoding video, using the animation “whizzy” effects of a compositing window manager, or explicitly an OpenGL application (which in modern times could actually be a typical web browser, but not for everything)

*: I can’t quite explain why the problem only occurred under Windows, but I’ll wave my hands and say “driver differences” while nodding sagely.

Rescuing a Truncated Recording Using ffmpeg

tl;dr: Well, it’s cheating because it’s not ‘rescuing’ what isn’t recorded, just retrieving and appending

Last night I played a few rounds of some cooperative gameplay with some good friends, and as usual I recorded the session for posterity.

Hans Volter's on fire and about to die
“Oh man, oh god!”, Hans is about to croak.

However, when looking over the file created, neither VLC nor ffmpeg could determine a duration. I used ffmpeg to create a new file (ffmpeg -i old.mkv -c copy new.mkv), and when it got to about the 1 hour 50 minute mark in the stream copy, it printed the following error:

invalid return value 0 for stream protocol
[matroska,webm @ 0x55eb60b609c0] Read error
Invalid return value 0 for stream protocol

It’s possible that I ran out of space during recording, so it behoves me to make sure my disks are more clear — hopefully the process of migrating to a combined fileserver will help there! — before I start recording.

Fortunately, I also stream to Twitch and so I could create a ‘highlight’ of the latter part of gameplay- about 45 minutes’ worth. The quality isn’t as good (max bitrate 3500kbit), but it’s better than nothing. So that it can be seamlessly appended, I decided to upscale the size from 720p to 1080p using ffmpeg:

$ ffmpeg -i ~/downloads/211006613-69819912-48eb80ad-009f-42d5-83aa-f68dda7b7612.mp4 -vf scale=1920x1080:flags=lanczos
-c:a copy -crf 20 -preset slow ~/mounts/storage/video/2017-12-19\ 22-04-59b.mkv

frame=105361 fps= 24 q=28.0 size= 2402802kB time=00:29:16.20 bitrate=11208.1kbits/s speed=0.408x

(Conversion in progress! Using -vf scale=1920x1080:flags=lanczos to scale and -crf 20 to ensure quality stays reasonable)

After that, it should be a simple matter of using the concat demuxer, assuming the files are similar enough (both are pixel format yuvj420p)– I previously had issues when trying to create an auto-highlighter/clip creator that minimised re-encoding.

For posterity!

[Fixed] MySQL: Table is marked as crashed and last (automatic?) repair failed (+ WordPress)

tl;dr: run myisamchk on the problematic table

I’ve run into the following error in my Apache error.log recently:

Table 'database.tablename' is marked as crashed and last (automatic?) repair failed

Fortunately the fix is simple: run myisamchk on the table which is marked as crashed:


$ sudo su
# service mysql stop
# cd /var/lib/mysql/databasename
# myisamchk -r tablename
MyISAM-table 'tablename' is not fixed because of errors
Try fixing it by using the --safe-recover (-o), the --force (-f)
 option or by not using the --quick (-q) flag
# myisamchk -r -o -f tablename
Data records: 107435
Found block that points outside data file at 16166832
# service mysql start

I’ve run into these errors before due to running out of disk space on the (admittedly tiny) VPS I had.

I also had this problem with a WordPress database able, causing the often-seen and unhelpfully terse:

Error establishing a database connection

Interestingly, this wasn’t getting bounced to error.log, and I had to use the WordPress database repair screen to track down which one needed the fix (which was the same myisamchk).

All sorted now!