Rescuing a Truncated Recording Using ffmpeg

tl;dr: Well, it’s cheating because it’s not ‘rescuing’ what isn’t recorded, just retrieving and appending

Last night I played a few rounds of some cooperative gameplay with some good friends, and as usual I recorded the session for posterity.

Hans Volter's on fire and about to die
“Oh man, oh god!”, Hans is about to croak.

However, when looking over the file created, neither VLC nor ffmpeg could determine a duration. I used ffmpeg to create a new file (ffmpeg -i old.mkv -c copy new.mkv), and when it got to about the 1 hour 50 minute mark in the stream copy, it printed the following error:

invalid return value 0 for stream protocol
[matroska,webm @ 0x55eb60b609c0] Read error
Invalid return value 0 for stream protocol

It’s possible that I ran out of space during recording, so it behoves me to make sure my disks are more clear — hopefully the process of migrating to a combined fileserver will help there! — before I start recording.

Fortunately, I also stream to Twitch and so I could create a ‘highlight’ of the latter part of gameplay- about 45 minutes’ worth. The quality isn’t as good (max bitrate 3500kbit), but it’s better than nothing. So that it can be seamlessly appended, I decided to upscale the size from 720p to 1080p using ffmpeg:

$ ffmpeg -i ~/downloads/211006613-69819912-48eb80ad-009f-42d5-83aa-f68dda7b7612.mp4 -vf scale=1920x1080:flags=lanczos
-c:a copy -crf 20 -preset slow ~/mounts/storage/video/2017-12-19\ 22-04-59b.mkv

frame=105361 fps= 24 q=28.0 size= 2402802kB time=00:29:16.20 bitrate=11208.1kbits/s speed=0.408x

(Conversion in progress! Using -vf scale=1920x1080:flags=lanczos to scale and -crf 20 to ensure quality stays reasonable)

After that, it should be a simple matter of using the concat demuxer, assuming the files are similar enough (both are pixel format yuvj420p)– I previously had issues when trying to create an auto-highlighter/clip creator that minimised re-encoding.

For posterity!

[Fixed] MySQL: Table is marked as crashed and last (automatic?) repair failed (+ WordPress)

tl;dr: run myisamchk on the problematic table

I’ve run into the following error in my Apache error.log recently:

Table 'database.tablename' is marked as crashed and last (automatic?) repair failed

Fortunately the fix is simple: run myisamchk on the table which is marked as crashed:

$ sudo su
# service mysql stop
# cd /var/lib/mysql/databasename
# myisamchk -r tablename
MyISAM-table 'tablename' is not fixed because of errors
Try fixing it by using the --safe-recover (-o), the --force (-f)
 option or by not using the --quick (-q) flag
# myisamchk -r -o -f tablename
Data records: 107435
Found block that points outside data file at 16166832
# service mysql start

I’ve run into these errors before due to running out of disk space on the (admittedly tiny) VPS I had.

I also had this problem with a WordPress database able, causing the often-seen and unhelpfully terse:

Error establishing a database connection

Interestingly, this wasn’t getting bounced to error.log, and I had to use the WordPress database repair screen to track down which one needed the fix (which was the same myisamchk).

All sorted now!

Speeding up fdupes

tl;dr: use jdupes

I was merging some fileserver content, and realised I would inevitably end up with duplicates. “Aha”, I thought “time to use good old fdupes“. Well yes, except a few hours later, fdupes was still only at a few percent. Turns out running it on a collected merged mélange of files which are several terabytes in size is not a speedy process.

Enter jdupes, Jody Bruchon’s fork of fdupes. It’s reportedly many times faster than the original, but that’s only half the story. The key, as with things like Project Euler is to figure out the smart way of doing things– in this case smart way is to find duplicates on a subset of files. That might be between photo directories if you think you might have imported duplicates.

In my case, I care about disk space (still haven’t got that LTO drive), and so restricting the search to files over, say, 50 megabytes seemed reasonable. I could probably have gone higher. Even still, it finished in minutes, rather than interminable hours.

/jdupes -S -Z -Q -X size-:50M -r ~/storage/

NB: Jdoy Bruchon makes an excellent point below about the use of -Q. From the documentation:

-Q --quick skip byte-for-byte confirmation for quick matching
WARNING: -Q can result in data loss! Be very careful!

As I was going to manually review (± delete) the duplicates myself, potential collisions are not a huge issue. I would not recommend using it if data loss is a concern, or if using the automated removal option.

jdupes is in Arch AUR and some repos for Debian, but the source code is easy to compile in any case.

Compressing Teamspeak 3 Recordings Using sox

tl;dr: Loop through the files in bash, sox them to FLAC


I’ve been combining fileserver contents recently, and I came across a little archive of Teamspeak 3 recordings:

$ du -sh .
483G /home/robert/storage/media/ts_recordings/


I wrote a quick-and-dirty script to convert the files:


total=$(ls *.wav|wc)
ls *.wav | while read file; do
        sox -q ${file} ${file%.*}.flac
        if [ -e ${file%.*}.flac ]; then
                if ! [ -s {file%.*}.flac ]; then
                        rm ${file}
                        echo "${file%.*}.flac is zero-length!"
                echo "Failed on ${file}"

        if  ! ((n % 10 )); then
                echo "${n} of ${total}"

The script checks that the FLACs replacing the WAVs exist and are not zero-length before removing the original.

This was fine, but after finishing, I was still left with a bunch of uncompressed files in RF64 format, which unfortunately errored.

It turns out sox 14.4.2 added RF64 read support, so I grabbed that on my Arch machine, and converted the few remaining files (substituting wav ? rf64 twice in the script above.

The final result?

$ du -sh .
64G /home/robert/storage/raid6/media/ts_recordings/

400 gigs less space and still lossless? Ahh, much better.

[Solved] “Logical volume is used by another device”

tl;dr: use dmsetup remove before trying lvremove

Note: Volume group and logical volume names have been substituted here. I’m not entirely sure it’s necessary, but better safe than sorry. If following this, please use the names of your volume group[s] and logical volume[s]

I am in the process of combining fileserver information, and so I have been touching parts of the system not usually looked at in the normal case of day-to-day operations. For some reason, on one of my logical volumes I had created a partition table and added a partition. Of course, that worked normally so there was no reason to be aware of this — clearly I had blanked the fact that I did it at all not long after doing so — until recently.

The Problem

Logical volume vg/lv-old is used by another device.

After copying the data over to a new logical volume, I wanted to remove the now-unnecessary original logical volume that contained the partition. Easy, right?

# lvremove -v /dev/vg/lv-old
    DEGRADED MODE. Incomplete RAID LVs will be processed.
    Using logical volume(s) on command line
  Logical volume vg/lv-old is used by another device.

Okay, what’s using it? cat /proc/mounts reports that it isn’t mounted. lsof and fuser return nothing. Maybe retrying the command will work*… nope.

There are a bunch of posts around this, mostly saying “make sure it is umounted first”, or “try using -f with lvremove“. And the old favourite: “a reboot fixed it”.

Find Out device-mapper’s Mapping

Well, the culprit in this case seemed to be device-mapper creating a mapping which counted as ‘in-use’. Check for the mapping via:

# dmsetup info -c | grep old
vg-lv--old       253   9 L--w    1    2      1 LVM-6O3jLvI6ZR3fg6ZpMgTlkqAudvgkfphCyPcP8AwpU2H57VjVBNmFBpL

Find Out Mapped Device

Then use that to find out what is holding it:

$ ls -la /sys/dev/block/253\:9/holders

drwxr-xr-x 2 root root 0 Dec 12 01:07 .
drwxr-xr-x 8 root root 0 Dec 12 01:07 ..
lrwxrwxrwx 1 root root 0 Dec 12 01:07 dm-18 -> ../../dm-18

Remove Device (via `dmsetup remove`)

Then do a dmsetup remove on that device-mapper device:

# dmsetup remove /dev/dm-18

Retry `lvremove`

And you’re good to go with lvremove:

# lvremove -v /dev/vgraid6/lv-old
    DEGRADED MODE. Incomplete RAID LVs will be processed.
    Using logical volume(s) on command line
Do you really want to remove active logical volume lv-old? [y/n]: y
    Archiving volume group "vg" metadata (seqno 35).
    Removing vg-lv--old (253:9)
    Releasing logical volume "lv-old"
    Creating volume group backup "/etc/lvm/backup/vg" (seqno 36).
  Logical volume "lv-old" successfully removed

Bish bash bosh!


*: I’m not sure of the thought process behind “just try it again”.

I’m reminded of a short bit of Darrell Hammond’s stand up (paraphrased):

“You know that message you get when you dial the wrong number that tells you to ‘check you have the right number and dial again’? Well, women will check the number and try again. Men will try the same number, but this time we’ll push the buttons a ******** harder…”

[Solved] “Filesystem is already n blocks long. Nothing to do!”

tl;dr: if you’re sure you did everything right, use lsblk or parted (etc) to see if a partition table is present on your logical volume.

So I am in the process of merging the content of two fileservers, and had the need to extend a logical volume to accommodate some additional data. No problem- that’s one of the benefits of using LVM!

Except after resizing, I ran into a problem:

$ lvextend +150G /dev/vg/lvinquestion
$ resize2fs /dev/vg/lvinquestion
> The filesystem is already 268435200 (4k) blocks long. Nothing to do!

Wait, what? Aside from the fact I could have combined the comments by including the --resizefs option to lvextend, why was resize2fs complaining that there was “Nothing to do!”?

Fortunately SE Arquade user ToxicFrog had the answer:

@bertieb parted reports the partition size, not the filesystem size
If it’s a partitioned LV you need to resize the partition after expanding the VL

Ah, whoops! I’m not sure why I partitioned the LV (it only had one partition) but I must have done so.

lsblk confirmed the partition:

sdh                                8:112  0   2.7T  0 disk
??sdh1                             8:113  0   2.7T  0 part
  ??md1                            9:1    0  10.9T  0 raid6
    ??vg-lv                      253:9    0   1.2T  0 lvm
    ? ??vg-lv1                   253:18   0   1.1T  0 part

So, then what? Well, I used dd to copy the filesystem to a new logical volume, then extended that, and finally removed the original:

# dd if=/dev/dm-18 bs=1M | pv -s 1T |  dd of=/dev/vg/lv-new bs=1M
# lvextend --resizefs -L 1.15T /dev/vg/lv-new
# lvremove /dev/vg/lv
# lvrename vg lv-new lv

(pv was included to give a nice progress indicator, rather than faffing around with SIGUSR1)

And that was that. There was a slight problem with removing the original logical volume, but more on that later…

Browsing MySQL Backups

tl;dr: Seems the quickest way of doing this was to fire up a VM, install mysql-server and mysql-client and browse that way.

I have backups of things. This is important, because as the old adage goes: running without backups is data loss waiting to happen. I’m not sure if that’s the adage, but it’s something resembling what I say to people. I’m a real hit at parties.

I wanted to check the backups of the database powering this blog, as there was a post that could swear I remembered referring to (iterating over files in bash) but couldn’t find. I had a gzipped dump of the MySQL database, and wanted to check that.

zgrep bash mysql.sql.gz | less was my first thought, but that gave me a huge amount of irrelevant stuff.

A few iterations later and I was at zgrep bash mysql.sql.gz | grep -i iterate | grep -i files | grep -v comments and none the wiser. I had hoped there was some tool to perform arbitrary queries on dump files, rather than going through a proper database server, but that’s basically sqlite and to my limited searches, didn’t seem to exist for MySQL.

What I ended up doing was firing up a VM, installing mysql-server and mysql-client and dumping the dump into that server via zcat:

zcat mysql.sql.gz | mysql -u 'root' -p database

And then querying the database: select post_title, post_date from wp_posts where post_title like '%bash%' followed by select post_content from wp_posts where post_title like '%terate%';

And the post is back!

Wanted: Boolean Filters For Gumtree

or them to apply their own standards for listings

Gumtree – owned by eBay – is much more popular in the UK than Craigslist, for some reason. Whatever that reason is, it’s definitely not due to the advanced search options they offer. As far as I can tell, they have no support for simple boolean operators.

So instead of the (say) Ford Docus you’re looking for, you might see a “Vauxhall Astra 2006 Like Ford Fiesta Focus Golf Polo”, which is definitely not what is wanted…

*: I mean, I tried -, NOT, (NOT)

Timesaver: import and combine GoPro Footage with FFmpeg

I’ve been taking my GoPro to Sunday Morning Football (as it is known) for a while now, so I figured I’d automate the process of importing the footage (moving it from microSD) and combining it into one file (GoPro splits recordings by default).

So I have the following script:


DATE="$(date +%Y-%m-%d)"

if [ -e $GOPRO ]; then
        echo "Copying..."
        rsync -aP --info=progress2 --remove-source-files --include='*.MP4' $GOPRO/DCIM/100GOPRO/ $VIDEO_DEST/
        echo "Joining..."
        cd $VIDEO_DEST
        #cd $GOPRO/DCIM/100GOPRO/
        for file in `ls *.MP4`; do echo "file '$file'" >> stitch.txt; done
        #RECORD_DATE="$(ffprobe -v quiet `ls *.MP4 | head -n1` -show_entries stream=index,codec_type:stream_tags=creation_time:format_tags=creation_time | grep creation_time | head -n1| cut -d '=' -f 2| cut -d ' ' -f1)"
        # new format:
        RECORD_DATE="$(ffprobe -v quiet `ls *.MP4 | head -n1` -show_entries stream=index,codec_type:stream_tags=creation_time:format_tags=creation_time | grep creation_time | head -n1| cut -d '=' -f 2| cut -d ' ' -f1| cut -d 'T' -f1)"
        #echo "$RECORD_DATE"
        ffmpeg -y -f concat -i stitch.txt -c copy $RECORD_DATE.mp4
        echo "GoPro microSD not mounted?"


  • the microSD is already mounted before running (under /tmp/gopro) – I had considered automating this, but I figured running a script in response to insertion of removable media was a bad idea; I could add the mkdir and mount commands here, but since the latter requires root privileges I’d rather not and it is quickly recalled from bash history in any case
    • the $VIDEO_BASE directory is mounted and created (this is pretty stable)
    • the GoPro won’t number directories higher than 100GOPRO (eg 101GOPRO)- it possibly would if dealing with eg timelapse, but I am not covering that case
    • the GoPro will set creation time correctly; so far it has reset to the default date a few times, probably related to battery
    • I want to keep the source files around after creation (the script could remove them)

Given the above the script may seem a bit fragile – and it is definitely tightly coupled to my assumptions – but it’s done the job for a few weeks at least, and the commands it was based on have been pretty stable since I started recording football last year.

Wanted: One LTO-4/5/6/7 Drive!

I am something of a digital hoarder. I have files dating back to one of the earliest computers that anyone in my family owned. I think I even still have diskettes for an older word processor, the name of which escapes me at the moment. As such, I have slightly more than average storage requirements.

At present I handle these requirements via a Linux fileserver, using 3TB drives RAID6’d via mdadm. On top of that I use LVM to serve up some volumes for Xen, but that’s not strictly relevant to storage.

Looking at the capacities of LTO makes me quite covetous. LTO tapes are small, capacious and reliable– with a few tapes, I could archive a fair amount of data. I could also move the tapes outside my house- and lo, offline offsite backups!

Sadly, drives are expensive, unless you’re stepping back relatively small capacity* LTO-2 drives.

At present, given the cost of drives, some back-of-the-envelope calculations show that for any reasonable** dataset, simply buying hard drives (at time of writing, 3TB is cheapest per GP) is the most cost-effective means of archiving. Given that is where the focus of development is, I don’t think this is likely to change soon.

I’ll just have to wait for a going-out-of-business auction, and hope the liquidators overlook the value of the backup system…