SaltyVagrant

Why I won’t work for Google

2021-05-01T00:00:00+01:00

TL;DR Building a YouTube business is not something I will ever do. Here’s why.

I do have a YouTube channel but it is simply a method of giving away ‘free’ content and making people aware of my own web site. Where Google permits I monetize the ‘free’ content on this channel (Google takes my work and add it’s own adverts regardless, so I make bank where possible).

The problem, as so many have realised in the past, is that any YouTube creator is subject to the indifferent caprice of Google so the idea of using YouTube as a platform to develop a business is risible. So long as you understand this the deal is a good one. Google provide the platform to distribute video content without cost to the creator, the creator reaches a potentially larger audience than they might otherwise.

If, however, you believe that using YouTube to develop a business is a good idea I think you’re in for a bad time. (You might get lucky, but the odds are against you.)

Google cares not a jot about creators. YouTube is a machine for getting adverts in front of eyeballs. Period. This means Google will prioritize any action that preserves or enhances this function. This is why DMCA take downs, copyright strikes, and the like are automated with virtually no way for creators to win (regardless of the merits of their case). You can literally see your business vanish in seconds and unless you are fortunate enough to have access to the right back channels you are unlikely to ever see it return.

Furthermore, Google’s algorithm optimises for revenue generation so if your content is not feeding this algorithm your content will languish in relative obscurity. This algorithm is responsible for the ‘race to the bottom’ we see on platforms such as YouTube. Sure, there are many excellent channels, but the one’s that attract the most attention from Google are the ones that the algorithm sees as potentially revenue generating. Even large YouTubers have noted the fragility of their position in this regard.

This is not a Google problem. Blaming Google for this is like blaming a Bear for shitting in the woods. It’s just what they naturally do.

If you focus on building a ‘YouTube business’ then you are working for Google.

It’s easy!

2021-04-09T00:00:00+01:00

TL;DR A few thoughts on learning to program and the ‘ease’ of prgramming.

Inspired by this article.

People who say (and mean) ‘programming is easy’ are generally those in the middle of their career (or their learning curve). To beginners programming is hard because it is new and unfamiliar, they have a sparse pre-existing knowledge structure to hook into. The beginner needs to develop both basic knowledge and the instinct for applying that knowledge. This is hard.

For master programmers programming is hard because they take on hard problems and hard problems tend to require novel thinking and more advanced programming skills.

The journeyman programmer, the programmer who has learned and internalised the basic skills and can apply them consistently begins to believe that programming is easy. Time will disabuse the good journeyman of this conceit. Many programmers do not progress beyond this ‘programming is easy’ naivety. They know ‘enough’ and that is sufficient to earn a living, and that’s fine. They will spend their career essentially ‘solving’ the same problem over and over. A few though will forever be looking to challenge themselves with new problems requiring improved skills. These few will eventually become master programmers and realise that no matter how much they know there is always more to learn. No matter how easy some elements of programming become for them there are always more elements that remain difficult (until mastered).

My own journey began over 40 years ago, in many respects I am still a journeyman programmer. I learned to program in 1980. I have earned a living programming since 1987. I am still learning. For me some elements of programming are now ‘easy’ but many remain hard (if they weren’t I would have no incentive to continue, what’s the point in doing something easy?)

The well intentioned lie, “programming is easy”

I think (hope) that most people saying ‘programming is easy” are trying to encourage people to try their hand at programming. This is a laudable intention but I think it often has the opposite effect, especially on beginners.

If you’re learning something completely new, say (picking something entirely at random) juggling, and someone even slightly proficient tells you ‘it’s easy’ the impact is not encouragement it is demoralising. If it’s easy and I can’t get even this basic skill right I must be incapable of juggling. The truth is that almost any new skill is hard, very hard, when you start learning it. The more you practice the easier it becomes. What’s more once you have developed knowledge of the principles involved in a new skill you generally find extending your knowledge easier than learning the initial skill. The difference between juggling three balls and four balls is less than acquiring the skill to juggle those first three.

But there comes a point where things become hard again. Sticking with my juggling example, three to four balls is easy enough, four to five is more challenging, five to six is hard, and six to seven is much harder still. Beyond seven and you are into world record territory. Then there is the challenge of juggling objects of different size and weight. You may have mastered juggling three balls of equal size, then someone hands you a ping pong ball, an orange, and a pen. Juggling three things just took on a whole new level of difficulty.

Programming (or any other skill) has the same sort of challenges as you progress. First everything is new and it seems impossibly difficult. Sure, some people take to it easier than others. For some it just ‘clicks’ earlier. I was fortunate. I enjoyed the challenge of making a computer do things. I enjoyed taking my early efforts and figuring out how to make them better. Was it easy? Nope. At the start it was frustrating and hard but over time it became easier to see the solution. It became easier to understand how I had messed up.

Then I discovered assembly code. A whole new way of thinking, a whole new set of limitations. Suddenly what was easy became hard. A few assembly programs later I was confident enough to declare, ‘assembly is easy’. (I especially liked declaring this to school friends who had no idea what assembly code was. Yeah, I was a bit of a jerk when I was 13.)

Then a school friend showed me something they had written and… I felt dumb as a brick again. He’d written some assembler (actually he wrote a program allowing the ZX80 to display what was then ‘high resolution’ images—long story for another day). Suffice to say his code was elegant. His solution to something I did not even recognise was a problem until I saw his code was just so far advanced from my own efforts that for a couple of days I despaired. Then, once I’d figured out his code, I felt empowered. New knowledge, new abilities, and consequently a whole new set of problems and solutions became available. Sure, programming had become hard again but in a good way.

In the intervening decades I’ve experienced this roller-coaster many times, enough to realise that ‘it’s easy’ is seldom accurate. At best one can say, ‘this is easy for me now’. Any time I find myself thinking this I realise I’m not challenging myself enough. If you find yourself declaring ‘programming is easy’ I suggest you try something more challenging. I’ll wager most of you learned Java, C, JavaScript, Python, or PHP. If so, try Lisp or Haskell. Then tell me programming is easy on your first day. Still too easy, try Prolog. Still too easy? Okay, take any language and write a simple web server. Wait a second! No frameworks. No libraries. No cut-and-paste from the internet. I mean write one from scratch.

Yes, learning correct syntax for a language is ‘easy’. Learning a second functional language once you know one functional language is ‘easy’. But switching to a declarative language when you’ve only ever known functional languages is hard. Considering even ‘simple’ tasks (knocking up a web server in Python is only a dozen lines when you have access to Flask, but that’s not really programming it’s mechanics).

It’s easy to be bad. Another trait I see in people who declare ‘programming is easy’ is that their code sucks (and often is nothing more than cut-and-paste with a bit of glue code around the edges). I’ve seen people claim programming is nothing more than being able to use StackOverflow and Google. These people are either joking or idiots. Being able to find resources is an important skill, but if misused it leads to dependence and shitty code.

It’s easy to do the easy stuff. Another feature of the ‘programming is easy’ crowd is that they seldom tackle difficult problems. Any monkey can knock together a simple web server in Python. You really don’t need much more than a rudimentary level of knowledge or skill. Heck, you can just cut and paste the code for a Python web server from any of thousands of tutorials. But tackling a novel problem, or even simply scaling up the simple web server you just made, is a harder proposition. Programming is more than writing code, it’s solving problems.

Tutorials make it worse. By this I mean you can get a false sense of your achievement from a tutorial. It is trivial to follow along with a tutorial and think ‘that was easy, so programming is easy’. Then you tackle your first real-world problem and… programming become hard. Don’t get we wrong, tutorials have their place. I’m just saying they can give a false sense of achievement.

You are not exceptional. (There may be one of you reading this that is, in some limited scope, exceptional but for the rest of us we’re not exceptional.) This is not a bad thing! It is a simple fact of statistics that by definition most people are about average. Some are a little above average some a little below. Ironically those most below average tend to think themselves exceptional— the Dunning-Kruger effect. What is more, even if you are exceptional in one respect you are average in many, many others. Live with it. Embrace it.

Two guys walking in the woods happen upon a bear. The bear starts to chase them and one suddenly stops, opens his backpack and starts changing his boot to his track shoes. “Are you mad!” his friend shouts, “you’ll never outrun the bear even in track shoes”. “Oh, I know,” says the man, “but I only need to outrun you.” It takes a lot of pressure off if you remember to be successful you only need to be slightly better than the next guy.

The humble-brag, “programming is easy”

There are some people who feel the need to inflate their own ego by putting other down. The statement ‘programming is easy’ is sometimes to put others down. ‘How can you not get this? Programming is so easy!’ This a malicious form of the humble-brag, statements that, on their face, are modest but are actually intended as a brag.

Programming is hard

Okay, the mechanics of programming are generally not hard once you grasp the basics. What is hard is the problem solving required in programming. Learning how to translate from the problem, to a potential solution, to writing code to execute that solution, is difficult but gets easier as you build a store of previous solutions and get better at breaking problems down into manageable chunks susceptible to solution.

Programming is accessible

There is nothing preventing anyone with access to the internet learning to program. Particularly when you remember ‘on the internet no one knows you’re a dog’, in other words no one knows your age, gender, skin colour, sexual orientation, previous education, current occupation, or anything else generally associated with exclusion.

You will need access to a computer, access to the internet, and your own motivation. That’s it.

But don’t I need an expensive computer? No. Sure, there are many situations where having a whiz-bang computer will be beneficial but when you’re starting out just about any computer will be fine. (I learned to program on an 8-bit computer with 1k of memory—and no internet! So I am sure you’ll be fine with any modern computer. Get yourself on e-bay and buy a cheap second hand laptop.) Can’t afford a computer? Go to your local library, many allow free access to the internet and you can learn programming online without the need to install anything at all.

What I’m saying is, ‘no matter your current resources, if you are sufficiently motivated you can find a way to learn programming.’ It may be difficult at first, but it will get easier (and then harder as you tackle more complex problems).

Outline of a CI/CD Pipeline for Video Processing

2021-04-06T00:00:00+01:00

TL;DR In this article we are going to discuss the outline for a Continuous Integration/Continuous Deployment (CI/CD)¹ pipeline to process videos. Specifically, we are going to discuss a pipeline to take a main video, an edit list, some inserts (ads, promos, etc.), some title cards (opening, closing, transitions, etc.), and some metadata (like the description, title, author, tags, categories, etc.) and use these to assemble a number of final versions to be uploaded to various target services (YouTube, Vimeo, lbry.tv, etc.).

The basic flowchart for our pipeline looks something like this.

Figure 1:Video Process Pipeline Basic Flowchart

We need to take care not to create a pipeline that requires massive storage. A typical CI/CD pipeline starts with a change in something like a git repository, perhaps the source code for our project. This implies that the resource holds all the source for the pipeline. In our video workflow this is probably not practicable as the resources are copied into the local fly worker container. Copying a growing set of videos would swiftly result in gigabytes of files being copied through our pipeline; not cool.

To avoid the potential growth of data we will key everything from our metadata, using this as control to trigger a pipeline run and the content of the metadata will drive tasks in our pipeline to selectively upload other materials from remote ‘pools’.

We will still be importing some resources in their entirety:

metadata: This may end up being several files, or even several resources. Things like Edit Decision List (EDL)² used to determine cut points for inserts, metadata for deployment and announcements, data describing which videos are to be edited and which videos are to be inserted in promo or ad slots, that sort of thing.
templates: This resource provides things like the LaTeX template used to generate title cards.

On the output side. Do we want to keep all the generated video locally? Not really. So long as all the source material exists we do not really need to keep the final product (it is after all being uploaded to remote services), but we do want to have some may of verifying remote copies (i.e. those that have been deployed so that we can determine if they need to be redeployed).

The Development Environment

We’ll start with a Python development environment I anticipate several tools we need will be better as Python scripts than Bash scripts.

Obtain the base Python VM³.

1git clone https://gitlab.com/python-utils2/pythonvm.git videoprocessor

You may need to change the Vagrantfile to map the concourse port to something other than 9080. If you already have something mapped to that port vagrant will complain when starting the VM.

Move into the new VM (this first run will take several minutes as the system is updated and various Python tools are installed).

1cd videoprocessor 
2vagrant up

Once your VM is up and running, ssh into it.

1vagrant ssh

From this point on we will work almost exclusively in this VM, so I will highlight only operations not run in the VM/

Taking a look around

You will find that this VM has a few things already installed.

Concourse: This is the ‘doer of things’ (their description) that we will use to manage and run our build pipeline.
Docker: Used to run the Concourse CI system on this development VM. We will also use this to create containers to use in the Concourse pipeline.
Python: We will use this to create the more complicated scripts in our pipeline.
Poetry: A tool for Python package management.

A few commands help confirm everything is installed.

1docker --version 
2docker ps 
3fly --version 
4python --version 
5poetry --version

And you should see output something like the following (I’ve allowed the version to ‘float’, so you’re versions will likely be different to those shown below)..

docker

1Docker version 20.10.3, build 48d30b5 
2CONTAINER ID   IMAGE                 COMMAND                  CREATED       STATUS       PORTS                    NAMES 
3fcf478d39d95   concourse/concourse   "dumb-init /usr/loca…"   2 hours ago   Up 2 hours   0.0.0.0:8080->8080/tcp   concourse_ci_concourse_1 
479df83906661   postgres              "docker-entrypoint.s…"   2 hours ago   Up 2 hours   5432/tcp                 concourse_ci_concourse-db_1

fly –version

16.7.5

python –version

1Python 3.8.2

poetry –version

1Poetry version 1.1.4

The First Build Task: Making title cards

This is fairly straightforward; take a LaTeX template, feed it a title and logo and have it produce a PNG titlecard. Our inputs are therefore:

LaTeX template
Logo
Title string

Our outputs are:

A title card PNG

The tools required to transform our inputs to our outputs are:

LaTeX processor
ImageMagic, to convert the LaTeX output (a PDF) to a PNG

We could create a specialised Docker Image for this, but I will reuse my general purpose document generator image saltyvagrant/latex-docker (this is itself a custom made image, but is not specialised in the sense that it contains almost the entire LaTeX Live installation making it a very large image, but I reuse it for many Concourse tasks).

I have a logo already available (I prefer to preserve all images in SVG format until final conversion to a target format). The LaTeX template it fairly simple.

titlecard.tex

1\documentclass[oneside]{standalone} 
2 
3\usepackage{saltyvagrant} 
4 
5\usepackage{graphicx} 
6\usepackage{xcolor} 
7\usepackage{tgadventor} 
8\renewcommand*\familydefault{\sfdefault} 
9\usepackage[T1]{fontenc} 
10 
11\SetWatermarkScale{ 0.5 } 
12 
13\begin{document} 
14  \begin{minipage}{572bp} 
15    \includegraphics[height=720bp]{logo_and_mascot/salty-logo-avatar-clipped.png} 
16  \end{minipage} 
17  \begin{minipage}[c][720bp]{705bp} 
18    \fontsize{100}{130}\selectfont 
19    \color{svblue} 
20    \begin{center} 
21      \getenv{TITLE} 
22    \end{center} 
23  \end{minipage} 
24\end{document}

The title itself is provided from a metadata file and inserted into the LaTeX process using the TITLE environment variable. We will discuss the LaTeX process in more detail in a later article in this series.

Where to start a build

One decision you need to make when designing build systems is ‘where does the build start?’ By this I mean what do you consider the raw material for your build. With straightforward software builds the most obvious raw material is your source code. Less obvious raw materials are things like internationalisation strings, embedded images, and the build itself (e.g. the Makefile). In the real world it is seldom this obvious.

Even in a relatively straightforward build you have complications to consider. If you’re building for multiple target platforms then it is likely that your Makefile will be generated using configuration specific to the target platform. This either moves your raw materials back one level (an additional configuration and Makefile generation step) or you manually create several Makefiles and thereby shift some of the burden from the build process to whomever makes the builds (not a good option, but an option).

In the case of video processor some of the raw materials are also apparently obvious; the main video and audio, the metadata (title, description, etc.). Other, less obvious, sources include the build system itself. The various scripts and tools used to perform the build. These scripts may well also be making decisions about what and how to perform the build.

Beyond these ‘obvious’ points we may decide to create the title videos manually, so these now become sources rather than the metadata and templates etc. proposed above. Where you start your build will often be a matter of judgement, experience, project requirements, and technical limitations. A simple example from our current build; why not start from raw video and audio footage and have the build cut together the main video using an EDL also provided as source? This is entirely possible but also impractical for our purpose. Editing the main video together is simpler using an Non-Linear Editor (NLE)⁴, this video processing pipeline is performing the tedious tasks that can be easily automated whereas pulling raw video and audio together into a coherent video is (at least for me) much simpler using manual tools. (That said, if I were to be truly bloody-minded I could have the NLE generate the EDL and then have the build effectively repeat the main video cut, but that seems redundant to my workflow.)

Musings on metadata

We now encounter our first issue. Generally we want changes to the ‘make title’ task sources to trigger a build (we can trigger tasks in several ways, but source changes are the most common). Our source are; the metadata, the logo, and the template. Here’s the issue, if the template changes do we really want to trigger the task? If so, should it rebuild all title cards using the new template? (Bearing in mind that creating new title cards implies that any videos that used those title cards should also be regenerated.) What about if the logo changes? Same issue as the template. What about the metadata? The only piece of metadata we want to trigger a new title card is the title, any other changes can be ignored for now.

I think that generally I do not want template or logo changes to trigger complete rebuilds (maybe later I’ll look at manually triggering these wholesale changes, following a rebranding for example, but day-to-day it’s safe to assume changes to template or logo should effect only new title cards).

That leaves us with metadata changes. Title cards come in two types; those specific to a video (for example the lead title card) and more general ‘inserts’ (like ‘…and now a word from our sponsor’) that may be reused in several productions. Should we therefore maintain metadata as one large ‘database’, or ‘one data file per production’, or ‘one data file per title card’?

We could make a case for any of these solutions. I’m going with one per production but allowing a ’production’ to be one or more stand alone title cards. This seems like a reasonable compromise to me as it keeps everything more or less in one place for most videos but allows some flexibility to generate stand alone items too. The corollary to this being that each production can refer to other productions (so video A can use a title card produced by title B). I may live to regret this decision, but this is an iterative process and we need somewhere to start.

Why not go for one big lump of metadata? Mainly I think it would quickly become unwieldy and difficult to edit by hand (no doubt leading to yet another utility to maintain just the metadata and that roads leads to using databases, all of which seems like a bit of overkill at the moment). Also, the design of this whole pipeline means I may well not want to keep all metadata feeding in to the start of the pipeline (more on this in a while).

The new task

Our new task has one more input, the script to perform the build. It is possible, when your task’s image is specialised enough, to continue with the simple ‘one command’ but this task needs something more complex. Specifically, we need the task to scan metadata and make decisions about which title cards to produce.

Title Videos

We have a choice here. We can either make short stand alone videos from our title cards or we can use some ffmpeg magic later on to insert the title cards as short static section.

If we make short videos now it may reduce time rebuilding later (for example reusing the titles in another project) but this also implies storing the title videos somewhere so they can be reused. This seems unnecessary to shave a few seconds of downstream activities.

If we insert the titles ‘on-the-fly’ we make the assembly of the final video more complex (and consequently potentially more error prone) and more demanding on our downstream activity (the video must be reconstructed each time).

I think that, for now, we will create them on-the-fly and only create them separately if this proves troublesome.

Assembling the Final Video(s)

We may be assembling several version of the final video. Different ads, sponsors, or promotions being placed into, or omitted from, the main video according to the target platform. All this to be driven by our metadata.

Deploying the Final Video(s)

Our metadata can provide common information for each remote source (such as authentication details). It can also present specific information for each target, for example descriptions and ‘flags’ (those pop-over links available on platforms like YouTube).

Announcing New Videos

Once the videos are deployed we can announce their availability. As with deployment our metadata can provide authentication details for the various announcement channels along with details of what each announcement should contain. One of the main things in this section is to ensure we do not announce multiple times if something goes wrong.

Final Thoughts

In this article I’ve started thinking about the CI/CD process for making and publishing videos. Many of the details remain to be established but the broad outline seems like a good start.

¹Automated process for building, testing, and deploying assets.

²A list of ‘edits’; cut, insert, delete, etc. Usually presented in a standard text format such as CMX 3600

³If you want to be sure you’re using the setup in this tutorial then git checkout -b v0.1.0 once you’re in the workspace.

⁴Tool/technique for editing video that allows the user to choose any point in the project to work on.

Being explicit

2021-02-28T00:00:00+00:00

TL;DR There are many opportunities for developers to rely on implicit defaults. In this article I make the general proposition that it is better to be explicit rather than rely on defaults.

I am as guilty as anyone of relying on defaults. That said, it’s a bad habit.

The main problem is that defaults change from time to time. As the creator of a service (API, library, etc.) it makes sense to provide default values. They make the service easier to consume and simplifies writing the service. As the consumer of a service defaults can certainly appear to make using the service simpler but at the cost of potential breakages when the defaults change.

Let’s look at a trivial (and horribly contrived) example to make the point.

Suppose I have created a library to draw a circle.

1def drawCircle(radius=10): 
2  pass

My interface provides a default radius (perhaps the most common size for some graphic I want to create). This is convenient and users of my library use this default, calling the routine as drawCircle().

The issue is that drawCircle is now ‘locked-in’ to this default. If I decide to change the default (perhaps my application need cicles of diameter 20 now and the quickest way to make the change in my application is the change the default). The problem is doing so will wreak havoc in applications that used the default and continue to need circles of radius 10.

As a real-world example of the changing defaults issue, consider the cookie SameSite attribute. Originally cookies were broadly exchanged but problems emerged with cookies being misappropriated so the SameSite setting was added. Initially the default was None (to match behviour before SameSite became available) but in Chrome 80 this was to change to Lax. The problem with this change being that it would break many existing resource links (e.g. <img>) while letting links (e.g. <a>) continue to work. This change was pushed out (due to the disruption the change caused) to Chrome 85).

Host Setup for SaltyVagrant

2021-02-27T00:00:00+00:00

TL;DR Many SaltyVagrant articles and books use a common set of tools. This article outlines the tools used.

You will need to install three tools to your computer:

virtualbox—to run virtual machines.
vagrant—to supervise the creation and setup of virtual machines.
git—for version control (and more immediately to get the files needed to setup the virtual machines).

The installation instructions for these tools tend to vary over time so it’s best if you visit the websites, download the tool and follow the installation instructions relevant to your computer’s operating system.

The Horrors of Third-party Authentication

2021-02-26T00:00:00+00:00

TL;DR It is becoming increasingly common for sites to offer third-party authentication (using, for example, your Google account to log in to xyz.com). This is all very convenient but has some significant downside.

As I write this I am in the process of adding third-party authentication to SaltyVagrant.com, hold the cries of ‘hypocrite’ though as my issues are more to do with careless users than sites enabling them.

Privacy

Using third-party identity providers just puts even more personal information in the hands or these companies. Every site you use your social account to log in with immediately gains access to information about how often you visit the site, from where you access the site, and a potential wealth of other information. It is, fortunately, not likely that they will have access to the content the service for which they provide identity, but these other data are still very valuable and allow the likes of Google and Facebook to assemble a profile of information about your internet use.

Security

While there is no direct concern about the robustness of most third-party identity providers (beyond the usual problems of users choosing poor passwords) there remains the concern that using one identity provider across multiple services is equivalent to using one password. If the social account you use to log in to many sites with becomes compromised then instantly all the sites you used this social account on are also compromised.

Availability

As a site using third-party authenticators you have other issue. Principal among these are availability. If the third-party become unavailable you lose customers. This applies generally (the provider becomes unavailable to all users) and specifically (one user is denies access to the third-party—an increasingly likely situation). In either case you lose business entirely because of the third-party’s behaviour.

Support

Then there is the support service problem. If a user has difficuly logging on, is this a problem with your service of with the identity service? It is difficult to explain to users that you cannot help them because it is Google, Facebook, or whomever, denying them login access, not you.

Additional Resources

[LU]: A Levi and M Ufuk Çağlayani. The Problem of Trusted Third Party in Authentication and Digital Signature Protocols. url: http://people.sabanciuniv.edu/levi/iscis12.pdf (visited on 02/26/2021).

Amateur vs. Professional: Optimisation

2021-02-22T00:00:00+00:00

TL;DR What distinguishes an amateur from a professional? Let’s consider optimisation. (Obviously much of this is opinion.)

Optimisation is a topic often brought up online, and often incorrectly.

People will obsess about trivia, such as how to shave a few keystrokes from their ViM use or how to shave milliseconds from their code run-time, or avoid spawning sub-processes in the Bash scripts. Don’t get me wrong, there is a time and place for being concerned about each of these, but it is not something the professional worries about most of the time. The majority of these debates are amateurs trying the flex, showing their ‘amazing’ skills. In truth, the harder people try to impress like this the less professionals care.

A professional’s mantra is, ’get it working’—this is, after all, what we’re paid to do.

Let’s consider a professional approach to tool use. Yes, you can save time and effort by using your tools efficiently. It is worth learning efficient keystrokes on ViM but only if you repeat the operation those keystrokes achieve thousands of times. Spending time to learn (and by ‘learn’ I mean commit to muscle memory so it becomes automatic) a keystroke pattern takes time and effort. The time and effort you commit to this learning had better be less than the time or effort saved by the repeat of those saved keystrokes.

Here’s the thing. If you do something often enough your muscle memory will make it efficient over time. If you learn some shorter keystroke for an operation you repeat hundreds of times a day then you may slow a little at first but will quickly save time as your muscle memory improves. No need to spend time actively trying to learn it, you’re going to just by doing it. And if you don’t repeat it often in the course of your working day, then most likely you didn’t need to save those keystrokes in the first place¹.

When writing code the professional writes first for clarity and to make the code work (yeah, obvious I know). Only once the code is working will they worry about optimising. Arguing over whether a for loop is faster than a while loop in compiler xyz is something best left to amateurs (or compiler designers). Here’s the thing, we mere mortals can only know which parts of our code need to be optimised once we can measure its performance. The ‘premature optimisation’ of code is a leading cause of shitty over-complex code. Some amateur thinks they know that accessing hash tables is microseconds slower than array accesses in some programming language, so they torture their code to use array even though the problem is better suited to hash tables. This takes them hours longer to develop and debug, the code is a tangled mess but, ’hey! it runs 3 milliseconds faster than the hash version’. Great, but this code is invoked once every minute in real use scenarios. Was all that effort worth it? Is your dog shit code easy to maintain?

Contrast with the professional. Take the problem. Solve the problem in as clear a way as possible. Run the code. Does it work? Yes. Is it fast enough for the situation? Yes. Is it resource efficient enough? Yes. Job done, move on. Result? Clear, maintainable code that satisfies the problem and was quickly developed. It’s easier to debug and maintain because it is clearly written.

Suppose one of the conditions is not satisfied? If the clearly written code does not run fast enough? No problem. Now the professional starts looking at optimisation. First, the professional does not assume they know where the problem lies, they measure the scale of the problem. Use performance and coverage tools to figure out precisely where the code is being held up. Only then do they start changing things around. It may turn out that the hash/array access times are an insignificant factor. Let’s say we have a loop over some data that’s slowing our code. Can I do a simple test to avoid that loop all together in some circumstances? Have I got a loop within a loop (a very common costly process), can I unpack these loops or short-circuit the loops? If so, does this clear my performance bottleneck? If yes, then that’s the problem solved. If no, then maybe, just maybe, it’s time to use array rather than hash².

At this point the amateur may be feeling very smug because the professional has had to finally admit that ’yes, arrays are an optimization in this case’, but in truth the professional approach will produce better code overall and situations requiring this sort of fine optimisation are the exception rather than the rule. In over 30 years of working IT I can count on the fingers of one hand the times that I’ve had to perform low-level optimisations. Most optimisations have been less obvious but much grosser (like realising that maintaining a larger session pool, or unpacking nested loops, will solve a problem, rather than fiddling with shaving microseconds of my code³).

Does this mean we should avoid learning more efficient ways of working? No, of course not. Just don’t waste time fussing the detail too early.

¹With regards to ViM, learning the general pattern command→movement is definitely useful, as is learning the how to move about ViM. Trust me, you’ll be doing this a lot so forcing yourself to, for example, skip to last line on screen with L (rather then repeatedly mashing j) is going to stick real quick once you’ve overcome the short-term slowdown.

²When making this sort of change leave a comment detailing why you did it. This is one of the circumstances in which extended comments are invaluable.

³Again, don’t get the idea you should simply throw resources at a problem! That sort of thinking results in bloated shitty systems too. It’s about knowing where best to invest your resources—especially you time.

Fences and Ambulances

2021-01-29T00:00:00+00:00

TL;DR Prevention is better than cure, but we need both available when managing complex IT systems.

When you live near a cliff edge it is common-sense to build a fence along the edge to prevent people falling over that cliff. That said, it is also worthwhile having ambulances available to take those who do fall to hospital.

Fences are always a compromise; too low and they may as well not exist, too high and the block the view. Also, some people are insistent on climbing over fences and fences inevitably fall into disrepair over time.

What am I blathering on about fences for?

Consider your IT system, it is likely you have some proactive protections in place. Limiting access to production systems, using encryption, setting up firewalls, and so forth. These are your fences.

I would also wager that you have system failures too. When your system fails you have process and procedure for recovering the system back to an operational state. These reactive recovery process is your ambulance fleet.

Removing a Locked Virtualbox VM

2021-01-18T00:00:00+00:00

TL;DR Occasionally you will get a Virtualbox VM into a locked state and this will prevent it being deleted. Attempts to delete or unregister the VM result in the error Stderr: VBoxManage: error: The machine ’centos64_c6402_1454036461345_59755’ is already locked for a session (or being unlocked). This article describes how to resolve this problem.

The Virtualbox virtual machine (VM) you are trying to remove or unregister is ‘locked’ producing an error like the folowing.

1Stderr: VBoxManage: error: The machine 'centos64_c6402_1454036461345_59755' is already locked for a session (or being unlocked) 
2VBoxManage: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component MachineWrap, interface IMachine, callee nsISupports 
3VBoxManage: error: Context: "LockMachine(a->session, LockType_Write)" at line 493 of file VBoxManageModifyVM.cpp

The simplest way I know the ‘unlock’ this sort of VM is using an ‘emergencystop.

1VBoxManage startvm <vm-uuid> --type emergencystop

Where <vm-uuid> is found using VMoxManage list vms (the UUID being the long collection of characters between the {} at the end of each line).

1VBoxManage list vms

1"Whonix-Gateway" {d89a93c5-d0f3-48ef-9e5c-bf0df89931f1} 
2"Whonix-Workstation" {05647ae6-792c-4caf-be5e-1508c6d012cf} 
3"Buscador-1.2(3-10-18)" {c44145ec-5a71-44b2-a6cb-c590ab92d273}

Once the VM is unlocked you can remove it as usual.

1VBoxManage unregistervm --delete <vm-uuid>

Or simply unregister it if you have deleted the VM files already.

1VBoxManage unregistervm <vm-uuid>

sudo users

2020-12-10T00:00:00+00:00

TL;DR Protecting your system from accidental or malicious change is made simpler by allowing users only limited access. This especially means limiting users who have root or ‘superuser’ access as these users have effectively unlimited power. When it is necessary for a user to have this access it is best that they have a normal (unprivileged) account for day-to-day activities with the ability to temporarily promote themselves to ‘superuser’ to run commands that require this elevated access.

First, some basics of Linux access control.

Linux accounts all have two basic identities that the operating system uses to control access to facilities like which files the account holder can access and what programs the user can execute; the user id (uid) and the group id (gid). You can easily see the uid and gid for an account using the id command.

1id vagrant

This will show the vagrant account’s current uid and gid along with a list of ‘groups’ to which that account belongs.

1uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),109(netdev),111(bluetooth),997(docker)

Issuing the command id without specifing a username will return these details for the current account.

What does this all mean?

Access to files is specified on three axes; user, group, and other. For each axis we can specify read access, write access, and execute access. All of this information can be seen using the ls command.

1ls -l

1total 64 
2drwxr-xr-x  7 vagrant vagrant 4096 Nov 18 15:33 . 
3drwxr-xr-x 37 vagrant vagrant 4096 Dec  3 09:01 .. 
4-rw-r--r--  1 vagrant vagrant  419 Nov 18 09:23 404.html 
5-rw-r--r--  1 vagrant vagrant  412 Nov 18 09:23 about.markdown 
6drwxr-xr-x  3 vagrant vagrant 4096 Nov 18 09:23 assets 
7-rw-r--r--  1 vagrant vagrant  672 Nov 18 09:23 books.markdown 
8-rw-r--r--  1 vagrant vagrant 2352 Nov 18 09:23 _config.yml 
9-rw-r--r--  1 vagrant vagrant 1131 Nov 18 09:23 Gemfile 
10-rw-r--r--  1 vagrant vagrant 1869 Nov 18 09:23 Gemfile.lock 
11drwxr-xr-x  8 vagrant vagrant 4096 Nov 18 15:33 .git 
12-rw-r--r--  1 vagrant vagrant   56 Oct 10 13:36 .gitignore 
13drwxr-xr-x  2 vagrant vagrant 4096 Nov 18 09:23 _includes 
14-rw-r--r--  1 vagrant vagrant  178 Nov 18 09:23 index.markdown 
15drwxr-xr-x  3 vagrant vagrant 4096 Oct 10 13:41 .jekyll-cache 
16-rw-r--r--  1 vagrant vagrant  629 Nov 18 09:23 saltyvagrant.markdown 
17drwxr-xr-x  4 vagrant vagrant 4096 Nov 18 09:23 _sass§

Looking at line 3 we see that the 404.html file has its access ‘mode’ set to -rw-r–r–. We read the first character as a special indicator (the first, in this example -, meaning ‘not set’). The remaining nine indicators are read in groups of three. Each group contains; r read, w write, x execute, or - for ‘not set’. The three groups represent owner, group, and other.

Reading the 404.html file mode (-rw-r–r–) we see; special is not set (-), user (vagrant as indicated by the entry in the third column of output) can read and write but not execute (rw-), Any account in the group (vagrant as indicated by the entry in the fourth column of output) can read this file but can neither write not execute it (r–). The final three entries show that any account on the machine can read but neither write nor execute it (r–).

Looking at line 5 we see that assets is a directory, indicated by the d special indicator (the first character on the line). User vagrant can read, write, and execute the assets directory. What does ‘execute’ mean on a directory? Any account with execute access x can list the directory content (in this example any account can list the assets directory content as indicated by the x in the final position).

Under normal circumstances only the account vagrant would be able to add a file to the assets directry or make changes (write) to the 404.html file. Superusers however are above these restrictions and are able to do aything they please to these files.

Who are superusers?

The special root account is always a superuser. Generally it is good practice though to prevent direct login access to the root account. Instead we provide non-privileged accounts with a way to run privileged commands but only when some special action is taken. This need to take some special action to run commands as a superuser will (hopefully) reduce the potential for mistakes.

The special action required is called sudo, short for superuser do. This is a special command, normally used as a prefix to whatever command we want to run with superuser privilege.

Why is it bad practice to allow direct login access to root but instead use sudo? Four reasons;

Using sudo requires an explicit action to perform privileged actions (actions that may have significant effects on your system).
All sudo actions are logged, making it simpler to find who actually performed each privileged action.
sudo actions can be controlled such that the account may run only certain commands with superuser rights.
The right to run sudo commands can be granted and revoked very simply.

Logged in to the non-privileged account vagrant we can look at ls -l /etc/hosts, the local hosts file.

1-rw-r--r-- 1 root root 254 Oct 14 13:39 /etc/hosts

This tells us that only the root user has write access to this file. Suppose the vagrant user needs to edit this file.

First we need to tell the operating system that vagrant is allowed to run elevated commands using sudo. This is very simple, the vagrant account must be added to the sudo group. Typically, any account that is a member of this sudo group is permitted to run the sudo command, and by extension is allowed to run commands as a superuser. I say ‘typically’ because the sudo facility allows much more subtle configuration of what each account or group may do. We could, for example, create a group reboot that would allow member of the group to issue a command to reboot the server but have access to no other privileged operation. Similarly, we could change the sudo group to allow limited access, but as I say, it is more common to leave sudo as all powerful and create other more specialised groups as required.

As it happens our vagrant account is already a member of the sudo group (we can see this in the output of the id vagrant command shown above).

If the vagrant account was not already a member of this group we could add it using the command usermod -g -G sudo vagrant. As you might expect this command requires superuser privilege (otherwise any account would be able to add itself to the sudo group, making a mockery of our security). So the command we want is sudo usermod -g -G sudo vagrant run by an account that is already a member of the sudo group (alternatively we could use the root account without using sudo but as we said above, direct login access to the root account is to be avoided). The obvious point here being that when we initially set up our system to use sudo we must have at least one account as a member of the sudo group.

1sudo vi /etc/hosts

Prefixing the edit comand with sudo allows the account to edit the file regardless of the normal access rights on the file.