Services, Servers, DomUs and Containers, Oh My!

What a tangled web we weave…

I was confused. Staring at a console window, wondering how I’d installed a program†. The system package manager knew nothing about it, pip pretended like it never heard of it, and I hadn’t downloaded and compiled the source.

I’ve never really had what anyone would call a sane management approach to my servers and services. The closest I’ve got is using Xen as a hypervisor and trying to separate DomUs (VM guests) by service type, backed by LVM-on-mdraid storage*. That sounds alright in theory, but in practice most services have tended to congregate on the largest guest, turning that DomU into a virtualised general purpose server. A server that mixes and matches the system package manager, other package managers like pip, SteamCMD and manual installs.

Pug fugly, in other words.

Pug-fugly, as coined in Pgymoelian

* the storage also sounds good in theory but the implementation has led me to dub the machine the ‘Frankenserver’ (more on that another time)

This is fairly typical. You need to do X. Not only that, but you need to do it RIGHT NOW. Software ABC does that. So you download it from whatever source seems most convenient and up-to-date, glance at the ‘Quickstart guide’ (while saying thank goodness for those!), do a bit of minimal configuration and then you’re up and running, doing X.

But what’s the big deal? The service[s] work, after all.

The issue is a general one: it takes time to figure out the setup before you can usefully interact with it.

This doesn’t just apply to DevOps; but to coding, writing, maintenance and repair, DIY, cooking, house management.

Or more simply: Fail to plan, plan to fail.

I found that it was taking me time to get my head around:

  •  what I was dealing with
  • how it had been set up
  • why it wasn’t working
  • how to fix it
  • how to update it

I’d do those things, sort whatever, get the service working again, then six months later I’d have to figure it all out again.

“What a way to run a railroad…”

Clearly, there must be a better way.

In fact there are several better ways, depending on what you want to do. DevOps is huge business, and scales into the multinational megacorp range. But the home user can benefit too. There are clear benefits in using a well-organised system for pretty much anything, and managing servers, services and other applications is no exception. Used well, maintainability, security, reliability are all enhanced.

But how does one get started? There are plenty to choose from. Some folks I know love Docker, others opt for LXD on LXC (those options are not exclusive). There are also the configuration management tools, like Puppet, Ansible, Chef (etc).

Well, I briefly used Docker in the past, and now have it on one of the DomU guests, hosting a few services I used to run elsewhere. This seems like reason enough to dip my toes deeper in the waters and move more services to containers, or at least to automated processes.

It’s rarely glamorous, but writing good documentation can make a huge difference to the person that follows you. Even when that person is you.

As an aside, the other key ingredient other than having good systems in place is to have good documentation.

For example, before writing this up I wondered about installing a new spam plugin. I used to use Spam Karma 2, but that’s been unmaintained for a long while. But which one? Well, seems I’ve used Akismet and Anti Spam Bee in the past, but why did I stop using them? I have a vague recollection of the former re-moderating old comments and declaring them spam, and the latter not working in some way, but what?

Good documentation, make it your non-New Year’s Resolution.

So the take-home message here is that ad-hoc setup pop up and stick around for longer than they should; don’t do that, have a good system instead and document what you’re doing and why.

Because it’s good to have a goal, my aim is to get low-hanging fruit services moved over to Docker in the first instance (heh) to learn more about the tech. Fore there I can decide what I can move to containers, and maybe even see if LXC would fit my needs anywhere. I’d also like to see if I can apply this to the wee tools I write myself to help automate my workflows- rather than running them manually, perhaps I can develop them as services. And while doing all of this documenting what I am doing and why.

Then maybe one day I won’t have to ask “I have a (python-based) program installed that doesn’t seem to have been installed either by apt or pip, and obviously I can’t remember… is there any way to figure out how I actually installed it? :D”.


beets, for music library organisation/tagging/management

Featured image by steve gibson on Flickr

Softly, Softly

Sometimes when confronted by a setback, people lose their cool.

It happens. You have spent hours, perhaps days going over something in your head. You do research, can’t quite get to the answer you want to you post on a QA site, thinking “these guys really know what they’re talking about, I’ll get an answer real quick”.

But the process goes wrong. Instead of seeing your post and congratulating your for your witty and well-chosen phrasing, they take issue with terminology, ask questions that seem obvious, or irrelevant, or both! Then the question is put on hold until such time as your can ‘improve’ it.

Being told to improve isn’t the end of the world, but it’s a definite poke in the ego. The tribe has circled, they won’t let you in. As a result, some choose to express their frustrations in a non-constructive manner. This is unfortunate, because it evokes less consideration and feelings of altruism, not more.

In the example above, I’ve tried to pull things back from that brink:

Hi Raven, welcome to Super User. It seems you are a bit frustrated by the QA process here. Questions are sometimes put “on hold” so that they can be more easily answered- this has the benefit of making it more likely for you to get an answer. While it may be clear what you mean and intend, some of our experienced members find the specifics of what you are asking unclear.

However, I’m afraid you cannot ‘require’ that we take a post from being “on hold”, the best option for this to happen is to clarify or add detail as requested. Furthermore, your comments come across as quite aggressive; that’s hopefully not what you intended (since that would not be okay); you should consider removing those before a moderator does.

Futhermore, as your edit does not contain information pertinent to answering your question, I am going to ‘rollback’ to your original version. You are welcome to make further edits to add information to make answering the question easier.

(typos mine)

There’s two things that need to be done, as I see it: 1) let the person know what their approach won’t work and is not acceptable; 2) try to get things back on track.

The first, if done right, is doable; in fact it’s very satisfying when it happens. You can go from someone insulting you and being aggressive, to having a laugh with them (in the best case!). That said, in this case getting the tone back to being civil would be enough. Being nice when someone expects hostility can be incredibly disarming. It has to be sincere though; phoney friendliness backfires.

The second is important form a QA point of view. Interactions are hazardous and sometimes go wrong and the wrong things are said sometimes but those are all a distraction. Ideally, a question would be improved so that it can be answered (some cannot). A reasonable response from the community side is more likely to engender a helpful clarification or vital detail. It’s not easy to keep that up though, especially seeing the same thing day after day.

Post-scriptum: in this case the user characterised the on-hold review process and comments as ‘nefarious’ and declared that what they had written was ‘ALREADY 100% CLARIFIED’, amongst other things. It doesn’t always work.

Metal Gear Solid V: The Phantom Pain Mini-Review

I intended to include this in my post about reaching 100%, but that turned into a story about sneaking into a base armed with trousers.

So here’s the mini-review I was going to include with that.

The good: Controls and gameplay are (for the most part) deliciously smooth, and this strongly complements the organic gameplay possible- attacking a military fortress as a super-soldier with high tech gear plays as fluidly as ambushing an outpost wearing nothing but a pair of trousers. I ended up enjoying the story, but that’s going to be variable for others.

The bad: It’s only 2/3 complete, thanks Konami! Why do I have to stare at the Boss in a chopper every time I want to do something from the ACC? The grind for 100% was tedious, though optional. The animations and controls, while smooth were on occasion a bit slow- I kept getting into AA emplacements rather than fultoning them; and during certain fights, dodging some attacks failed because snake wanted to lie down or/or crawl instead of dive-dodge.

The ugly: At this point I’m not sure if Hideo Kojima is trying to illuminate/draw attention to sexism/misogyny etc, but uh, there were some highly questionable bits.

100% in MGSV:TPP

I finally got there! I have mixed feelings, as I no longer have an excuse to load the game up and mess around; but on the other hand I am happy to be done with some of the slog.

Even towards the end, after a hundred hours the game could still surprise me. I’ll give you a quick example: I was attempting a Subsistence mission (where you start with nothing but a pair of trousers and your wits). Some careful sneaking at an outpost netted me an AK and a bit of ammo, and so armed I went for the main objective- a small base that served as a radio relay station. I dispatched their forward patrol non-lethally and made my way through a pass that lead to a ledge directly above the base.

I do like the non-lethal approach, which has been consistently rewarded by other titles in the Metal Gear series. My normal approach would be to methodically and silently tranquilise the various patrols and guards without arousing suspicion; but with my only ranged weapon being unsilenced this wasn’t an option. Could I lure the guards somewhere secluded and incapacitate them? Perhaps! I made a bit of a movement on the ledge and the guard decided he would run the several hundred meters around the hill and through the pass to get to me. A quick bit of CQC and one down! Okay it took a few minutes but the theory was solid. Another guard was similarly lured.

There my plan stalled. No matter what wild, strange dance I did on that ledge, I couldn’t attract anyone’s attention. It being night probably didn’t help. There was nothing to do — aside from firing my gun, but that would have attracted a little too much attention — but drop down into the base proper. I was able to knock out another guard when I got the harsh musical sting and slowmo (which will forever belong to Max Payne in my head) meaning someone else had seen me. What’s more, they were too far away to silence before the brief window that lets you prevent a full-blown alert closed. Whoopsy.

I was determined to roll with the punches. Even if the punches were in the form of many automatic rifle rounds to the not-armoured-trousers.

So, there was a full-blown alert! I ducked behind the small building housing the objective (some radio comms equipment), peeking out to exchange fire. I soaked up a bit of damage — trousers are not as good armour as you might think — and was seriously considering reloading from checkpoint. But by this point in the game I had already achieved the S rank for the mission (which generally depends on either speed or being super-stealthy or both) so I was determined to roll with the punches. Even if the punches were in the form of many automatic rifle rounds to the not-armoured-trousers.

Combat was faring less well than I hoped. The guards were well-armed and well-armoured. Ducking around one end of the building was rewarded with a warning of a missile lock on me (!), and the other had a shotgun/machine gun duo who were surprisingly effective. I was low on health, low on ammo and lacking a weapon that dealt sufficient damage to those who wished to do me harm. Then I was thrown a bone.

Warning: Sandstorm approaching

A very windy, sandy bone; but a bone nonetheless. Sandstorms limit visibility to about 2 feet, so this was an opportunity to scarper. Which I did, in the direction of a nearby mortar.

In the nigh-on 100 hours of gameplay, I don’t think I’d used a mortar before. They just didn’t mesh terribly well with the whole stealthy-and-non-lethal approach I so heavily favoured. As I could still see marked enemies in the sandstorm, I was able to take out most of the guards that had previously been gunning me with great effect. And some of the guards I hadn’t spotted before and couldn’t see through the sand. And some of the radio transmitters. And the anti-air radar.

Mortars are great!

After that finishing off the last few non-mortally-wounded (or is that non-mortarily-wounded?) soldiers and completing the objective was a relative breeze.

MGSV:TPP had many of these moments, when things didn’t work out the way they were planned; but the outcome was even better than expected.

tl;dr: good game, would recommend.

Closing in (95%)

Nearly there!

It’s getting annoyingly grindy now! I spend most of the time in the chopper, either:

  • deploying to an area to put down capture cages then immediately leaving
  • returning to the medical platform on mother base to hand over photographs

The latter is especially irritating as there’s no indication that you need to do it, and you can’t give all ten photographs at once! So chopper in, run to room, cutscene into room, hand over photo, run back to chopper, leave; repeat.

krusty_groan.wav

I have a sheet of paper that I’m crossing off the things as I do them. It’s slightly illegible due to my broken fingers, but usable.

On “Back up, Back Down”

The gods of irony got together with the gods of gaming after my recent gripe about having to do and redo things in MGSV:TPP:

Some missions have mutually-exclusive objectives – I’m looking at you, Backup, Back Down – so may require more

Well, I played through the “Extreme” version of Back Up, Back Down to do the additional objectives, and the team searching for the prisoner got there, stood around him and then… very kindly didn’t execute him.

So I ran up, stunned them all with a non-lethal assault rifle and fultoned him out! All optional objectives complete.

Seeking 100% in MGS V: The Phantom Pain

I’m getting close to 100% completion of Metal Gear Solid V: The Phantom Pain. I’ve completed the story (more on that in a follow up retrospective when I’m done) and achieved S-rank in all of the missions, which is easier than it seems at first glance. So I’m chasing the other things that need done:

  • achieve S-rank on every main mission

    wait, I said this already, weren’t you paying attention?

  • complete 157 side ops

    most of the time spent on this is taken on getting to the side-ops location; highly repetitive

  • capture / save a specimen of every animal

    Can you say ‘fetch quest’? I thought you could!

  • complete all important combat deployments

    click a button on the menu and wait? sure

  • collect all blueprints

    should be achieved if doing all the other stuff anyway

  • collect all key items

    similarly, most should be gained in the course of things, except the first aid kit (off the top of my head)

  • collect all 10 Paz photographs

    all but one gained via side ops

  • complete all mission tasks

    last but not least! practically speaking, this means missions need ‘around 3’ plays: first time its new, second time for s-rank, and third for the remaining objectives. Some missions have mutually-exclusive objectives – I’m looking at you, Backup, Back Down – so may require more

The final one is probably the biggest time sink; though the side ops come close. a rough guess, I reckon I’ve done all the mission objectives in at least 25% to 40% of missions, maybe more. Some of the objectives take a while, particularly the ones which involve following a target and listening to their conversations.

The Good

Playing without care for rank or speed generally means more fun! The optional objectives reward things that are a little more out of the way to do (like capturing patrolling armoured vehicles, or recovering a blueprint) but are in themselves rewarding.

But the most time-consuming ones, ‘listen to a series of conversations’ I’ve found the most interesting as they reveal more about the plot, so are cool in retrospect. Having completed the story, its a bit like rereading a book and going “oh, so that’s what they were foreshadowing!”.

The Bad

It’s a grind. Redoing things you’ve already done, watching the Boss (in the guise of whichever character) fly into the AO over and over again gets repetitive quickly. I’ve seen Hideo Kojima’s name in the credits more times than I count, though that’s in every mission a few times so is pretty repetitive in itself.

There’s also the nagging feeling that chasing a meaningless number in a game is a huge waste of life, but I try to push that to the back of my mind. I’m having fun!

The Ugly

There are occasional bugs; one mission (Lingua Franca) is prone it it. I’ve also had people/outposts/bases get spooked and never leave the heightened security state, necessitating a restart.

Am I still enjoying playing the game, even though I’ve completed the story? Yes, but only just.

[solved] js52: /usr/lib/libmozjs-52.so.0 exists in filesystem

Living on the edge in Arch Linux land is a fun activity everyone should try (at least once). However, a full system package upgrade caused the following today:

# pacman -Syyu
(...)
error: failed to commit transaction (conflicting files) 
js52: /usr/lib/libmozjs-52.so.0 exists in filesystem

I’m not the only one to have the issue. Seems the official way of getting past this is to rename the file, at least per this bug report.

Update: There’s an Arch news post that adds a modicum more information:

Due to the SONAME of /usr/lib/libmozjs-52.so not matching its file name, ldconfig created an untracked file /usr/lib/libmozjs-52.so.0. This is now fixed and both files are present in the package.

To pass the upgrade, remove /usr/lib/libmozjs-52.so.0 prior to upgrading.

I think this is the first time I’ve needed to do a manual intervention for a package upgrade for the time I’ve been running Arch; so all in all not bad.

Quick Hacks: A Script to Extract a Single Image/Frame From Video

Long ago, I posted the simple way to get a frame of a video using ffmpeg. I’ve been using that technique for a long time.

It can be a bit unwieldy for iteratively finding a specific frame, as when using a terminal you have to move the cursor to the time specification. So I wrote a very small wrapper script to put the time part at or towards the end:


#!/bin/bash
# f.sh - single frame

USAGE="f.sh infile timecode [outfile]"

if [ "$#" == "0" ]; then
        echo "$USAGE"
        exit 1
fi

if [ -e "$1" ]; then
        video="$1"
else
        echo "file not found: $1"
        exit 1
fi

if [ ! -z "$2" ]; then
        time="$2"
else
        echo "Need timecode!"
        exit 1
fi

# if we have a filename write to that, else imagemagick display

if [ ! -z "$3" ]; then
        echo "ffmpeg -i \"$video\" -ss $time  -vframes 1 -f image2 \"$3\""
        ffmpeg -loglevel quiet -hide_banner -ss $time -i "$video" -vframes 1 -f image2 "$3"
else
        echo "ffmpeg -i \"$video\" -ss $3  -vframes 1 -f image2 - | display"
        ffmpeg -hide_banner -loglevel quiet -ss $time  -i "$video" -vframes 1 -f image2 - | display
fi

Most of that is usage explanation, but broadly it has two modes:

  • display an image (f.sh video time)
  • write an image (f.sh video time image)

It’s more convenient to use it, hit ? and amend the time than to move the cursor into the depth of an ffmpeg command.

Better Backups: Decide What You’re Going To Back Up

tl;dr: Picking what you are going to back up helps (i) keep the backup space usage minimal (ii) helps to inform choice of backup program

Following on from picking a backup system in the backups series, now that you’ve picked a system, what exactly should you back up?

You could make the argument that really, what you’re going to back up is part of your requirements gathering. Frequently-changing data (eg documents) is different from a snapshot of a Windows installation is different from an archive of the family photos.

My my case, I want to back up my home directory, which is a mix of things:

  • documents of all sorts
  • code (some mine, some open source tools)
  • application configuration data
  • browser history etc
  • miscellaneous downloads

It totals less than 20Gb, most of which is split between downloads, browser and code (around 3:1:1, according to ncdu). Some things like documents, code and browser data will change semi-frequently and old versions are useful; others will stay relatively static and version history is not so important (like downloads).

Some downloads were for a one-off specific purpose and removed. It would be possible to pare down further by removing some downloads and some code — wine is the largest directory in ~/code/, and I don’t remember the last time I used it — but it’s not enough that I feel it’s a priority to do.

Is there anything in this set of data that doesn’t need kept? Frequently-changing-but-low-utility files like browser cache would be worth excluding as they will cause the (incremental) backups to grow in size. Incidentally, cache was the next largest item in the ratio above!

Some of the files will change relatively frequently, and I’d like to keep the history of them. I have decided that I want to keep my entire home directory, minus browser cache. This help to inform me what things I need my backup program to do, and what to do with it when I decide.