Whats in the AEC data feed, and how to use it

With an election coming up, it’s probably a good time to post some notes about using Australian election data feeds. This is mainly aimed at any interested programmers who have a use for this sort of data.

I’ve left out any example code here, as what you write will be specific to how you use the data.

Overview

The AEC publishes results on election night, by simply posting zipped files on an FTP server every few minutes. The detailed documentation is at aec.gov.au.

I have experimented with loading the data into a database during a recent by-election with some success. The data model that I assembled for this was:

2016-06-data-model-aec-feed

You will be able to locate many electoral concepts in this data model, as candidates with a particular affiliation receive votes (contest eachother) at different polling places for a a seat in one of the houses. This maps to the more extensive XML format to extract the fields that I was most interested in mapping.

Getting the data

The FTP server is mediafeed.aec.gov.au, and it accepts anonymous login on election night.

Here you will find a numbered folder for the night’s election. Within this, there is a lot of repetition, so you can ignore most of the files straight away. The sub-folders that I suggest loading are Detailed/LightProgress and Detailed/Preload. For example, in a past election, the file layout was:

18126/
18126/Detailed
18126/Detailed/LightProgress
18126/Detailed/LightProgress/aec-mediafeed-Detailed-LightProgress-18126-20150919194400.zip
18126/Detailed/LightProgress/aec-mediafeed-Detailed-LightProgress-18126-20150919194527.zip
18126/Detailed/LightProgress/aec-mediafeed-Detailed-LightProgress-18126-20150919194702.zip
18126/Detailed/LightProgress/aec-mediafeed-Detailed-LightProgress-18126-20150911140244.zip
18126/Detailed/LightProgress/aec-mediafeed-Detailed-LightProgress-18126-20150919195500.zip
18126/Detailed/LightProgress/...
18126/Detailed/Preload
18126/Detailed/Preload/aec-mediafeed-Detailed-Preload-18126-20150911140243.zip

Loading the data

Start with the Preload data. This zip file is available first, and contains the candidate names, polling places, event details, and an initial (zeroed-out) results feed. Using a custom script and a schema like the one I’ve posted above, you will be able to import this into your database.

Next, the LightProgress feed updates will start being added every few minutes from 7pm. These Zip files contain a small piece of XML, each one superseding older files.

I’ve cut down one of these files as an example:

<?xml version="1.0" encoding="utf-8"?>
<MediaFeed Id="e9334806-990e-4773-8b00-7d74fa58b6af" Created="2015-09-19T19:55:00" SchemaVersion="3" EmlVersion="5" xmlns="http://www.aec.gov.au/xml/schema/mediafeed" xmlns:eml="urn:oasis:names:tc:evs:schema:eml" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:xal="urn:oasis:names:tc:ciq:xsdschema:xAL:2.0" xmlns:xnl="urn:oasis:names:tc:ciq:xsdschema:xNL:2.0" xmlns:ts="urn:oasis:names:tc:evs:schema:eml:ts" xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:schemaLocation="http://www.aec.gov.au/xml/schema/mediafeed ../Schema/AEC/aec-mediafeed-results-v3-0.xsd">
  <ManagingAuthority>
    <eml:AuthorityIdentifier Id="AEC">Australian Electoral Commission</eml:AuthorityIdentifier>
  </ManagingAuthority>
  <MessageLanguage>en</MessageLanguage>
  <MessageGenerator>
    <Name>Virtual Tally Room</Name>
    <Environment>PROD</Environment>
    <Site>CDC</Site>
    <Server>DB02</Server>
    <Platform>x64</Platform>
    <Version>9.2.0.28493</Version>
  </MessageGenerator>
  <Cycle Created="2015-09-19T19:54:47">37f177b4-ab20-4073-be85-0edefa5c8e96</Cycle>
  <Results Phase="ElectionNight" Verbosity="LightProgress" Granularity="Detailed">
    <eml:EventIdentifier Id="18126">
      <eml:EventName>Canning By-election</eml:EventName>
    </eml:EventIdentifier>
    <Election>
      <eml:ElectionIdentifier Id="H">
        <eml:ElectionName>House of Representatives By-election for the division of Canning</eml:ElectionName>
        <eml:ElectionCategory>ByElection</eml:ElectionCategory>
      </eml:ElectionIdentifier>
      <House>
        <Contests>
          <Contest Projected="true">
            <eml:ContestIdentifier Id="236">
              <eml:ContestName>Canning</eml:ContestName>
            </eml:ContestIdentifier>
            <Enrolment>112809</Enrolment>
            <FirstPreferences PollingPlacesReturned="0" PollingPlacesExpected="53">
              <Candidate>
                <eml:CandidateIdentifier Id="25424" />
                <BallotPosition>1</BallotPosition>
                <Elected>false</Elected>
                <Votes MatchedHistoric="0">0</Votes>
                <VotesByType>
                  <Votes Type="Ordinary">0</Votes>
                  <Votes Type="Absent">0</Votes>
                  <Votes Type="Provisional">0</Votes>
                  <Votes Type="PrePoll">0</Votes>
                  <Votes Type="Postal">0</Votes>
                </VotesByType>
              </Candidate>
              <!-- (other candidates, ghost candidates) -->
              <Formal>
                <Votes MatchedHistoric="0">0</Votes>
                <VotesByType>
                  <Votes Type="Ordinary">0</Votes>
                  <Votes Type="Absent">0</Votes>
                  <Votes Type="Provisional">0</Votes>
                  <Votes Type="PrePoll">0</Votes>
                  <Votes Type="Postal">0</Votes>
                </VotesByType>
              </Formal>
              <Informal>
                <Votes MatchedHistoric="0">0</Votes>
                <VotesByType>
                  <Votes Type="Ordinary">0</Votes>
                  <Votes Type="Absent">0</Votes>
                  <Votes Type="Provisional">0</Votes>
                  <Votes Type="PrePoll">0</Votes>
                  <Votes Type="Postal">0</Votes>
                </VotesByType>
              </Informal>
              <Total>
                <Votes MatchedHistoric="0">0</Votes>
                <VotesByType>
                  <Votes Type="Ordinary">0</Votes>
                  <Votes Type="Absent">0</Votes>
                  <Votes Type="Provisional">0</Votes>
                  <Votes Type="PrePoll">0</Votes>
                  <Votes Type="Postal">0</Votes>
                </VotesByType>
              </Total>
            </FirstPreferences>
            <TwoCandidatePreferred Restricted="true" PollingPlacesReturned="0" PollingPlacesExpected="53" />
            <TwoPartyPreferred>
              <Coalition>
                <CoalitionIdentifier Id="2" />
                <Votes>0</Votes>
              </Coalition>
              <Coalition>
                <CoalitionIdentifier Id="1" />
                <Votes>0</Votes>
              </Coalition>
            </TwoPartyPreferred>
            <PollingPlaces>
              <PollingPlace>
                <PollingPlaceIdentifier Id="7467" />
                <FirstPreferences />
                <TwoCandidatePreferred Restricted="true" />
              </PollingPlace>
              <!-- (other polling places, ghost candidates) -->
            </PollingPlaces>
          </Contest>
        </Contests>
      </House>
    </Election>
  </Results>
</MediaFeed>

Spatial data

You can also download electorate boundaries as spatial data here to back any visualisations of the results.

The locations of polling places are already encoded in the Preload data file.

Good luck!

How to set up Docker containers in Travis CI

This post outlines one way that I’ve found to spin up an application in Docker for testing on Travis CI. I use this style of testing for a web-based word puzzle generator, such that every code change is deployed to a test environment and automatically tested in a web browser before it is merged. This setup will be most applicable to web application developers who use GitHub.

I got the idea for writing this from a few lines in the docker-compose documentation, which suggested that Docker is an easy way to perform automated testing over a running application:

$ docker-compose up -d
$ ./run_tests
$ docker-compose down

This snippet was missing some setup and an example app, but these three lines do all the heavy lifting.

The Docker setup

In order to focus on the Docker setup, I made a server which simply responds to TCP requests on port 5000 with the text “Hello World”.

This file is called server.sh, and sits in a directory called foo_server:

#!/bin/sh
while true; do
  # Send 'Hello World' to anybody who connects on port 5000
  echo "Hello World" | nc -l 5000
done

Alongside it, I added a Dockerfile to instruct docker to execute this tiny ‘application’ in a container, after installing the dependencies. This machine is built from the Docker-official Debian image:

FROM debian
ADD . /usr/share/test-server
WORKDIR /usr/share/test-server
RUN apt-get update && apt-get install --assume-yes netcat-openbsd
CMD ./server.sh

Lastly, a .dockerignore file is used to avoid loading the Dockerfile to the container:

# Ignore docker files
Dockerfile
.dockerignore

In the directory above, a simple test script, test.sh is used to see that the server is returning the expected output:

#!/bin/sh
set -e
expected="Hello World"
actual=`nc -v localhost 5000`
echo "Expecting: $expected"
echo "Server says: $actual"
if [ "$expected" != "$actual" ]; then
  echo "Test failed"
  exit 1
else
  echo "Test passed"
  exit 0
fi

Alongside the test file, a file called docker-compose.yml instructs Docker to create a container out of the foo_server example, and forward port 127.0.0.1:5000 to it.

version: '2'
services:
  foo:
    build: foo_server
    ports:
     - "5000:5000"
    container_name: foo_1

To try it out for yourself, you need a relatively recent version of Docker and docker-compose. The versions provided in Debian were not new enough to execute the examples, but the Docker project provides repos containing newer builds for Debian & Ubuntu. For my distro, the install was:

curl -sSL "https://get.docker.com/gpg" | sudo -E apt-key add -
echo "deb https://apt.dockerproject.org/repo debian-stretch main" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install docker-engine
sudo pip install docker-compose

The versions this got me were docker 1.11.2, and docker-compose 1.7.1. Straight after the install, I could deploy & test an example locally:

$ docker-compose up --build -d
$ ./test.sh
$ docker-compose down

The CI setup

I’ll assume that if you’re reading this, you are familiar with the basics of Travis CI. The large block of code below is the .travis.yml file to set up the test machine, then execute the tests against a container.

---
# Use Ubuntu 'trusty' distribution
sudo: required
dist: trusty

install:
  # Update docker-engine using Ubuntu 'trusty' apt repo
  - >
    curl -sSL "https://get.docker.com/gpg" |
     sudo -E apt-key add -
  - >
    echo "deb https://apt.dockerproject.org/repo ubuntu-trusty main" |
     sudo tee -a /etc/apt/sources.list
  - sudo apt-get update
  - >
    sudo apt-get -o Dpkg::Options::="--force-confdef" \
     -o Dpkg::Options::="--force-confold" --assume-yes install docker-engine
  - docker version
  # Update docker-compose via pip
  - sudo pip install docker-compose
  - docker-compose version

before_script:
  - docker-compose up --build -d

script:
  - ./test.sh

after_script:
  - docker-compose down
...

Note: This uses Travis CI’s trusty distribution, which at the time of writing is the newest stable build platform available on Travis CI. This shipped an outdated version of Docker, which had to be installed over. Because the existing Docker was configured, I had to override a debconf prompt, which is why the apt addon syntax was not used to set up dependencies.

Result

The build result for each commit is displayed in Travis CI:

2016-06-ci-result-1

Under this, the output of the passing test script is shown, showing what has been set up:

2016-06-ci-result-2

Using this setup in practice

Moving this from a demo setup to a real setup would be fairly simple:

  1. Replace the installation with a real software stack
  2. Replace the server run with a command to serve the application (such as a Apache HTTP, Tomcat or Node)
  3. Replace the tests with real tests (such as Cucumber or Selenium).

The example in the pre-amble installs a LAMP stack and tests it with Selenium in its CI build.

If your application is a bit larger, your only extra complexity will come from running multiple containers with docker-compose.

Get the code

All of these scripts in a working CI example are available on GitHub:
mike42/minimal-docker-ci

Print larger or smaller text on a thermal receipt printer

If you print to a thermal receipt printer which support the ESC/POS protocol, then you can format the receipts to make larger or smaller text. If this is your first time reading about ESC/POS, have a read of What is ESC/POS, and how do I use it?. Some of these text size examples are borrowed from there, while some are new.

Smaller text

Receipt fonts example
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Fonts (many printers do not have a 'Font C') */
$fonts = array(
	Printer:::FONT_A,
	Printer:::FONT_B,
	Printer:::FONT_C);
for($i = 0; $i < count($fonts); $i++) {
	$printer -> setFont($fonts[$i]);
	$printer -> text("The quick brown fox jumps over the lazy dog\n");
}
$printer -> setFont(); // Reset
$printer -> cut();
$printer -> close();
00000000  1b 40 1b 4d 00 54 68 65  20 71 75 69 63 6b 20 62  |.@.M.The quick b|
00000010  72 6f 77 6e 20 66 6f 78  20 6a 75 6d 70 73 20 6f  |rown fox jumps o|
00000020  76 65 72 20 74 68 65 20  6c 61 7a 79 20 64 6f 67  |ver the lazy dog|
00000030  0a 1b 4d 01 54 68 65 20  71 75 69 63 6b 20 62 72  |..M.The quick br|
00000040  6f 77 6e 20 66 6f 78 20  6a 75 6d 70 73 20 6f 76  |own fox jumps ov|
00000050  65 72 20 74 68 65 20 6c  61 7a 79 20 64 6f 67 0a  |er the lazy dog.|
00000060  1b 4d 02 54 68 65 20 71  75 69 63 6b 20 62 72 6f  |.M.The quick bro|
00000070  77 6e 20 66 6f 78 20 6a  75 6d 70 73 20 6f 76 65  |wn fox jumps ove|
00000080  72 20 74 68 65 20 6c 61  7a 79 20 64 6f 67 0a 1b  |r the lazy dog..|
00000090  4d 00 1d 56 41 03                                 |M..VA.|
00000096

Change height & width

2016-06-textsize-00
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Text of various (in-proportion) sizes */
title($printer, "Change height & width\n");
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize($i, $i);
    $printer -> text($i);
}
$printer -> text("\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}
00000000  1b 40 1b 21 08 0a 43 68  61 6e 67 65 20 68 65 69  |.@.!..Change hei|
00000010  67 68 74 20 26 20 77 69  64 74 68 0a 1b 21 00 1d  |ght & width..!..|
00000020  21 00 31 1d 21 11 32 1d  21 22 33 1d 21 33 34 1d  |!.1.!.2.!"3.!34.|
00000030  21 44 35 1d 21 55 36 1d  21 66 37 1d 21 77 38 0a  |!D5.!U6.!f7.!w8.|
00000040  1d 56 41 03                                       |.VA.|
00000044

Change width only (height=4)

2016-06-textsize-01
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Width changing only */
title($printer, "Change width only (height=4):\n");
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize($i, 4);
    $printer -> text($i);
}
$printer -> text("\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}
00000000  1b 40 1b 21 08 0a 43 68  61 6e 67 65 20 77 69 64  |.@.!..Change wid|
00000010  74 68 20 6f 6e 6c 79 20  28 68 65 69 67 68 74 3d  |th only (height=|
00000020  34 29 3a 0a 1b 21 00 1d  21 03 31 1d 21 13 32 1d  |4):..!..!.1.!.2.|
00000030  21 23 33 1d 21 33 34 1d  21 43 35 1d 21 53 36 1d  |!#3.!34.!C5.!S6.|
00000040  21 63 37 1d 21 73 38 0a  1d 56 41 03              |!c7.!s8..VA.|
0000004c

Change height only (width=4)

2016-06-textsize-02
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Height changing only */
title($printer, "Change height only (width=4):\n");
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize(4, $i);
    $printer -> text($i);
}
$printer -> text("\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}
00000000  1b 40 1b 21 08 0a 43 68  61 6e 67 65 20 68 65 69  |.@.!..Change hei|
00000010  67 68 74 20 6f 6e 6c 79  20 28 77 69 64 74 68 3d  |ght only (width=|
00000020  34 29 3a 0a 1b 21 00 1d  21 30 31 1d 21 31 32 1d  |4):..!..!01.!12.|
00000030  21 32 33 1d 21 33 34 1d  21 34 35 1d 21 35 36 1d  |!23.!34.!45.!56.|
00000040  21 36 37 1d 21 37 38 0a  1d 56 41 03              |!67.!78..VA.|
0000004c

Very narrow font

2016-06-textsize-03
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Very narrow text */
title($printer, "Very narrow text:\n");
$printer -> setTextSize(1, 8);
$printer -> text("The quick brown fox jumps over the lazy dog.\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}
00000000  1b 40 1b 21 08 0a 56 65  72 79 20 6e 61 72 72 6f  |.@.!..Very narro|
00000010  77 20 74 65 78 74 3a 0a  1b 21 00 1d 21 07 54 68  |w text:..!..!.Th|
00000020  65 20 71 75 69 63 6b 20  62 72 6f 77 6e 20 66 6f  |e quick brown fo|
00000030  78 20 6a 75 6d 70 73 20  6f 76 65 72 20 74 68 65  |x jumps over the|
00000040  20 6c 61 7a 79 20 64 6f  67 2e 0a 1d 56 41 03     | lazy dog...VA.|
0000004f

Very tall font

2016-06-textsize-04
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Very flat text */
title($printer, "Very wide text:\n");
$printer -> setTextSize(4, 1);
$printer -> text("Hello world!\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}
00000000  1b 40 1b 21 08 0a 56 65  72 79 20 77 69 64 65 20  |.@.!..Very wide |
00000010  74 65 78 74 3a 0a 1b 21  00 1d 21 30 48 65 6c 6c  |text:..!..!0Hell|
00000020  6f 20 77 6f 72 6c 64 21  0a 1d 56 41 03           |o world!..VA.|
0000002d

Largest possible font

2016-06-textsize-05
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Very large text */
title($printer, "Largest possible text:\n");
$printer -> setTextSize(8, 8);
$printer -> text("Hello\nworld!\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}
00000000  1b 40 1b 21 08 0a 4c 61  72 67 65 73 74 20 70 6f  |.@.!..Largest po|
00000010  73 73 69 62 6c 65 20 74  65 78 74 3a 0a 1b 21 00  |ssible text:..!.|
00000020  1d 21 77 48 65 6c 6c 6f  0a 77 6f 72 6c 64 21 0a  |.!wHello.world!.|
00000030  1d 56 41 03                                       |.VA.|
00000034

Double-height and doule-width

There are also commands which specifically double the width and height. These overlap with the commands covered here, but you can find them in the escpos-php documentation.

Full example

These examples were authored for escpos-php, a PHP printer driver for thermal receipt printers. The full file ships as an example with the driver, and outputs a block of ESC/POS code which can be sent a printer to give this output:

Receipt

f4218f44-395a-11e5-8079-25472c5ad340

PHP code

<?php
/**
 * This print-out shows how large the available font sizes are. It is included
 * separately due to the amount of text it prints.
 */
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;

$connector = new FilePrintConnector("php://stdout");
$printer = new Printer($connector);

/* Initialize */
$printer -> initialize();

/* Text of various (in-proportion) sizes */
title($printer, "Change height & width\n");
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize($i, $i);
    $printer -> text($i);
}
$printer -> text("\n");

/* Width changing only */
title($printer, "Change width only (height=4):\n");
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize($i, 4);
    $printer -> text($i);
}
$printer -> text("\n");

/* Height changing only */
title($printer, "Change height only (width=4):\n");
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize(4, $i);
    $printer -> text($i);
}
$printer -> text("\n");

/* Very narrow text */
title($printer, "Very narrow text:\n");
$printer -> setTextSize(1, 8);
$printer -> text("The quick brown fox jumps over the lazy dog.\n");

/* Very flat text */
title($printer, "Very wide text:\n");
$printer -> setTextSize(4, 1);
$printer -> text("Hello world!\n");

/* Very large text */
title($printer, "Largest possible text:\n");
$printer -> setTextSize(8, 8);
$printer -> text("Hello\nworld!\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}

Other formatting?

If text size is not what you’re after, then you can find similar examples in other posts I’ve tagged escpos.

How to compile a C++11 app on Travis CI

I have recently been adding Travis CI builds to code that I host on GitHub, so that I don’t need to host my own build infrastructure.

To users, this just means that there is a green badge at the top of the README, but not much else:

passing.svg

To build a simple C++ project, I added in this .travis.yml file:

langauge: cpp
sudo: false

addons:
  apt:
    packages:
      - libusb-1.0-0-dev

script:
  - make

Unfortunately, on this infrastructure, the default build tools are currently ancient, and installed on Ubuntu Precise (12.04):

$ make
g++ src/missile.cpp examples/basic-sync/basic-sync.cpp -o bin/basic-sync -lpthread -lusb-1.0 -std=c++11 -Wall
cc1plus: error: unrecognized command line option ‘-std=c++11’
cc1plus: error: unrecognized command line option ‘-std=c++11’

Option 1: Update the toolchain

There is some structures you can use to install an extra repository and some named packages, instead of using apt-get directly.

langauge: cpp
sudo: false

addons:
  apt:
    sources:
    - ubuntu-toolchain-r-test
    packages:
    - gcc-4.8
    - g++-4.8
    - libusb-1.0-0-dev

script:
  - make

Because the old version was still installed, I had to refer to the exact version in the Makefile, as in:

g++-4.8 src/missile.cpp examples/basic-sync/basic-sync.cpp -o bin/basic-sync -lpthread -lusb-1.0 -std=c++11 -Wall

Option 2: Update the platform

You can also change to a more recent Ubuntu distribution. Presumably Ubuntu Precise is only the default because existing builds use it.

If you need to build C++11 apps on Travis CI, then builds will work under Ubuntu Trusty (14.04), which happens to be the newest distribution currently available:

langauge: cpp
sudo: required
dist: trusty

addons:
  apt:
    packages:
      - libusb-1.0-0-dev

script:
  - make

How to edit emulator flags in Android Studio

I’ve recently updated to the new IntelliJ-based Android studio 2.1.

I ran into some issues attempting to launch a “hello world” project in the emulator, which I’m writing up here for the benefit of others.

For context, I run Debian GNU/Linux Jessie on an AMD64 box, with Radeon graphics card, using the free drivers.

The OpenGL error

2016-05-22-android-1

When attempting to launch a simple project with the emulator, the emulator died with the following message:

Cannot launch AVD in emulator.
Output:
libGL error: unable to load driver: radeonsi_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: radeonsi
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
X Error of failed request:  GLXBadContext
  Major opcode of failed request:  155 (GLX)
  Minor opcode of failed request:  6 (X_GLXIsDirect)
  Serial number of failed request:  49
  Current serial number in output stream:  48
libGL error: unable to load driver: radeonsi_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: radeonsi
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
X Error of failed request:  GLXBadContext
  Major opcode of failed request:  155 (GLX)
  Minor opcode of failed request:  6 (X_GLXIsDirect)
  Serial number of failed request:  49
  Current serial number in output stream:  48
libGL error: unable to load driver: radeonsi_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: radeonsi
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  155 (GLX)
  Minor opcode of failed request:  24 (X_GLXCreateNewContext)
  Value in failed request:  0x0
  Serial number of failed request:  33
emulator: WARNING: VM heap size set below hardware specified minimum of 228MB
emulator: WARNING: Setting VM heap size to 384MB
  Current serial number in output stream:  34
QObject::~QObject: Timers cannot be stopped from another thread

The console logs that the command being executed is:

/home/mike/Android/Sdk/tools/emulator -netdelay none -netspeed full -avd Nexus_5X_API_23

Fixing on the command-line

Quick list of things that didn’t work:

  • Installing more libraries via apt-get
    apt-get install libstdc++6 xserver-xorg-video-radeon
  • Pre-loading libraries
    LD_PRELOAD='/usr/lib64/libstdc++.so.6' /home/mike/Android/Sdk/tools/emulator -netdelay none -netspeed full -avd Nexus_5X_API_23
  • Attempting to understand LibGL’s debug output
    LIBGL_DEBUG=verbose /home/mike/Android/Sdk/tools/emulator -netdelay none -netspeed full -avd Nexus_5X_API_23

But the solution turned out to be this very specific flag:

$ ./emulator -help | grep libstdc++
    -use-system-libs               Use system libstdc++ instead of bundled one

So launching the emulator from the CLI worked for me, using this command:

/home/mike/Android/Sdk/tools/emulator -netdelay none -netspeed full -avd Nexus_5X_API_23

Fixing in the IDE

So this was fantastic progress, but without being able to launch this emulator from Android Studio, the development environment wasn’t really integrated yet.

Apparently there used to be a feature for adding command-line flags to the emulator, but this is now gone.

So, adapting a post here, I jumped in and replaced the emulator binary into a wrapper to inject some flags.

 $ cd ~/Android/Sdk/tools/
$ mv emulator emulator.0
$ touch emulator
$ chmod +x emulator

I then opened up the emulator in a text editor and punched in this:

#!/bin/sh
set -ex
$0.0 $@ -use-system-libs

Result

Success!

2016-05-22-android-2

Quick guide: Running stock Debian on the Raspberry Pi 2

At the time of writing, the ‘Raspbian’ port of Debian is often used on the Raspberry Pi. It was created to match the CPU architecture, for better performance. These reasons don’t apply to the newer Raspberry Pi 2, so if you’re a Debian desktop or server user, you can do away with the fork and just run Debian Jessie armhf.

The info from Debian is: https://wiki.debian.org/RaspberryPi2

A bit more background about why this only applies to the Raspberry Pi 2-

  • The Raspberry Pi 1 uses ARMv6 chipset with hard floats
    • The Debian armhf port requires ARMv7
    • The Debian armel port doesn’t use hard floats, so is unnecessarily slow on the Pi.
    • So Raspbian was created for the Raspberry Pi 1’s ARMv6 w/ hard-floats, and gets the most juice out of the CPU on the Raspberry Pi 1.
  • The Raspberry Pi 2 uses ARMv7 with hard floats, so Debian armhf port is fine.

Install the image

Image is linked to from this page:

I will assume that your machine has an SD card slot. To find the device name, list out disks and look for one of the correct size, which appears when you plug in the card:

df

Download a copy of the image, extract it out, and dd the file on to the card:

wget -c https://images.collabora.co.uk/rpi2/jessie-rpi2-20150705.img.gz
gunzip jessie-rpi2-20150705.img.gz 
sudo dd if=jessie-rpi2-20150705.img of=/dev/sdX bs=4M
sudo sync
umount /media/$USER/*

Plug in the Raspberry pi, and then log in. If you are using SSH, then arp-scan is a good tool to pick up devices on the network:

sudo apt-get install arp-scan
sudo arp-scan -l
ssh root@x.y.z.w

Configure pi- Things like screen resolution and HDMI go here:

cd /boot/firmware/
nano config.txt

Perform a software upgrade:

nano /etc/apt/sources.list
apt-get update
apt-get dist-upgrade

Start fixing security defaults. Remember that this is not a clean install, so start by setting your own passwords:

passwd

Check that there are no other accounts with passwords set:

cat /etc/shadow

Regenerate all SSH Server keys (commands from here):

ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa -b 521
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa

Lastly, generate some locales:

sudo locale-gen en_US en_US.UTF-8 en_GB en_GB.UTF-8

How to resize a Windows VM image with virt-resize

I recently had a Windows 7 Virtual Machine stored on an undersized qcow2 file. This post steps through the simplest way that I know to produce a new, bigger disk and expand the filesystem onto it.

Empty out the empty space

Because the guest VM is stored on a QCOW2 file, we can recover un-used space on disk by zeroing it out now. Download the sdelete utility from Microsoft and run it on the system.

sdelete -z

One this is done, power off the guest.

Assuming the host is linux, you need the qemu-utls and libguestfs-tools packages to follow these steps. On Debian-

apt-get install libguestfs-tools qemu-utls

Move the VM image to a new filename and inspect it.

mv windows.img windows.img.bak

The file command indicates that the disk is about 30GB expanded.

$ file windows.img.bak 
windows.img.bak: QEMU QCOW Image (v3), 32212254720 bytes

The qemu-img command shows that the disk is 83% full:

$ qemu-img check windows.img.bak 
No errors were found on the image.
411337/491520 = 83.69% allocated, 5.66% fragmented, 0.00% compressed clusters
Image end offset: 26961969152

Check out your FS names, note that /dev/sda2 is the disk we want to up-size in this case

$ virt-filesystems -a windows.img -l
Name       Type        VFS   Label            Size         Parent
/dev/sda1  filesystem  ntfs  System Reserved  104857600    -
/dev/sda2  filesystem  ntfs  -                32105299968  -

Make a new, bigger disk image

Create a new disk of the desired size. In my case, 50G is sufficient:

$ qemu-img create -f qcow2 windows.img 50G
Formatting 'windows.img', fmt=qcow2 size=53687091200 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

The file command shows that this new empty disk image is larger than the old image.

$ file windows.img
windows.img: QEMU QCOW Image (v3), 53687091200 bytes

Copy the old disk to the new one. The --expand option names a partition which will be grown to fill the extra space.

virt-resize --expand /dev/sda2 windows.img.bak windows.img

The virt-resize command shows a progress bar while it works, and zero-blocks will be reclaimed as a result of the output format:

2016-04-disk01

The final line of output suggests holding on to your backup until you’ve checked it, which is wise:

Resize operation completed with no errors. Before deleting the old disk,
carefully check that the resized disk boots and works correctly.

Check that the new disk is valid and contains partitions at the expected size:

$ virt-filesystems -a windows.img -l
Name       Type        VFS   Label            Size         Parent
/dev/sda1  filesystem  ntfs  System Reserved  104857600    -
/dev/sda2  filesystem  ntfs  -                53579939840  -

Boot up the guest

When the machine boots up, you may get a disk check prompt. Because the console I was using triggered the ‘Press any key to cancel’ prompt, I had to reboot and leave the console disconnected in order for the check to start.

2016-04-disk02

2016-04-disk03

After booting, the C:\ drive should display at its new size:

2016-04-disk04

Convert a PC to a HTPC with Debian and Kodi

I recently converted an old workstation to run as a home-theatre PC (HTPC). I’ve noted down the setup here for others who are making an installation like this. Some steps depend on using a radeon chipset, and will need to be adjusted for your computer.

Hardware

First up, Desktop ‘towers’ are not a good form-factor for sitting in TV cabinets. If your PC is this sort of size, then source a small form-factor case and power supply, and load the computer’s components into it:

2016-02-htpc

I also used a Logitech k400r keyboard and mouse for wireless input.

Install Debian and apps

Write the latest copy of Debian Stable to a CD or flash drive (this is version 8.3 at time of writing), and install it on the computer. Check “Debian Desktop environment” / GNOME during setup.

After installation, open a terminal, and type “su” to get root privileges.

su

Edit the software sources to include ‘contrib’ and ‘non-free’, as well as ‘jessie-backports’.

nano /etc/apt/sources.list
deb http://ftp.us.debian.org/debian/ jessie main contrib non-free
deb-src http://ftp.us.debian.org/debian/ jessie main contrib non-free

deb http://security.debian.org/ jessie/updates main contrib non-free
deb-src http://security.debian.org/ jessie/updates main contrib non-free

# jessie-updates, previously known as 'volatile'
deb http://ftp.us.debian.org/debian/ jessie-updates main contrib non-free
deb-src http://ftp.us.debian.org/debian/ jessie-updates main contrib non-free

# jessie-backports
deb http://ftp.us.debian.org/debian/ jessie-backports main contrib non-free
deb-src http://ftp.us.debian.org/debian/ jessie-backports main contrib non-free

Update sources and install Kodi:

apt-get install --install-suggests kodi

Also install the firmware packages that you may need.

apt-get install firmware-linux-free firmware-amd-graphics

Tweaks

Sudo

sudo allows you to run commands as root from your regular user account. Install the package and add yourself to the sudo group:

apt-get install sudo
usermod -a -G sudo mike

To apply the change, log out and back in again. The rest of this guide assumes you are logged in as yourself, and will use sudo where necessary.

Auto-start Kodi

Open the tweak tool, and locate the list of startup programs.

gnome-tweak-tool

Add Kodi to the list, log out, log in, and Kodi will launch automatically.

Auto-login

For a PC attached to a TV, user permissions are not so importnat, so set the user to log in automatically.

sudo nano /etc/gdm3/daemon.conf

Un-comment this block and enter your username:


# Enabling automatic login
#  AutomaticLoginEnable = true
#  AutomaticLogin = user1

Plymouth start-up screen

Install plymouth and configure grub to change the Debian boot sequence (a menu with timeout, followed by lots of text) into a graphical splash screen. This takes a bit of configuration.

sudo apt-get install plymouth

Set it up according to these instructions:

sudo nano /etc/initramfs-tools/modules

Set drm correctly for your chipset:

# KMS
drm
radeon modeset=1

Configure grub:

sudo nano /etc/default/grub
...
GRUB_TIMEOUT=0
...
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
...
GRUB_GFXMODE=1920x1080
...

Update grub, set the theme in Plymouth:

sudo update-grub2
sudo /usr/sbin/plymouth-set-default-theme --list
sudo /usr/sbin/plymouth-set-default-theme joy

Run update-initramfs to apply the changes

sudo update-initramfs -u

Samba

Samba will let you share folders over your network. A basic folder with guest read/write is simple to set up:

sudo apt-get install nautilus-share samba libpam-smbpass winbind
sudo usermod -a -G sambashare mike

Log out, and back in to apply the group change, and then share the Public folder over the network by right-clicking on it and opening the “Sharing Options”:

2016-02-samba-share

Gnome will warn that the folder as shared if you open it:

2016-02-samba-shared

Test the setup by typing smb://localhost into the address bar:

2016-02-samba-test

Overscan correction

In my case, I was able to set the TV to treat the input as a “PC” input. If that doesn’t work for you, then use xrandr in a login script:

Find the name of your input:

xrandr --query

Set underscan (get the horizontal and vertical values by trial and error):

xrandr --output HDMI-0 --set underscan on
xrandr --output HDMI-0 --set "underscan hborder" 32 --set "underscan vborder" 16

Kodi plugins

Add these as needed. The Australian catchup TV plugins repository from GitHub worked well.

Kodi RSS

The RSS feed shows Kodi updates by default, and is part of your user profile.

2016-02-feed-file

Edit the configuration file, and adjust the paths to your news sources of choice.

2016-02-feed

Boot speed

Readahead is the tool of choice for boot speed optimisation. Install it, and reboot.

sudo apt-get install readahead
sudo touch  /.readahead_collect
sudo reboot

Desktop Apps

If you quit Kodi, you are dropped back to the GNOME desktop. These apps are simply to improve the desktop user experience.

Google Chrome

Download the .deb file for Chrome from Google, install with dpkg, and then clean up dependencies:

dpkg -i google-chrome-stable_current_amd64.deb 
apt-get -f install

Firefox

Download and extract the Firefox for Linux tarball from Mozilla.

Move it to /usr/share, and change the owner to match other applications there.

mv firefox /usr/share/
cd /usr/share/
ls -Ahl
chown root:root firefox
chown -R root:root firefox

Find the main menu editor, and add Firefox to the menu.

2016-02-menu

Name
Firefox Web Browser
Command
/usr/share/firefox/firefox-bin
Icon
/usr/share/firefox/browser/icons/mozicon128.png

2016-02-firefox-icon

2016-02-firefox-menu

Test the new icon by searching:

2016-02-ff

Auto-clear browser profiles

Because you don’t need a password to log in to the user account, you can add this as a bit of insurance so that your box wont remember any passwords or sessions.

crontab -e

This job removes the Firefox and Chrome user profiles each boot.

@reboot rm --preserve-root -Rf --~/.config/google-chrome ~/.cache/google-chrome ~/.mozilla/firefox ~/.cache/mozilla/firefox

VLC

For file format support, best to have another media player:

sudo apt-get install vlc

Result

You should now have a PC which boots into Kodi for media and TV, and lets you quit into a desktop to browse the web or run regular desktop apps.

2016-02-kodi

2016-02-gnome

On the 1GB RAM/ dual core workstation, the shortened the boot to around 45 seconds from BIOS handing over control, to Kodi being ready.

How to regenerate SSH client keys

Recently, a SSH client bug was discovered that could let the server read client private keys in some situations.

If you’re affected, then follow these steps. SSH regulars will be familiar with most of the commands used.

If you use keys

If you do use keys, you should regenerate them. Start by backing up your old key. Assuming it’s at the default location, just use-

$ mv ~/.ssh/id_rsa ~/.ssh/id_rsa.old
$ mv ~/.ssh/id_rsa.pub ~/.ssh/id_rsa.pub.old

Next, create or edit ~/.ssh/config, and add the following line. This disables the roaming feature, which was part of the reason:

UseRoaming no

Generate new keys-


$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/example/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/example/.ssh/id_rsa.
Your public key has been saved in /home/example/.ssh/id_rsa.pub.

Now copy the new key over, using the old keys. You would normally use ssh-copy-id, but it adds keys rather than replacing them, which is why I’ll do it manually-

scp -i ~/.ssh/id_rsa.old user@example.com:/home/user/id_rsa.new

SSH in with the old key:

ssh -i ~/.ssh/id_rsa.old

Replace all current authorized keys with the newly generated one:

mv -f id_rsa.new ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Logout, log back in with the new key to make sure it works, and you’re done.

If you don’t use keys

If you don’t use keys for login, now could be a good time to start. Public-key authentication provides much stronger protection from brute force attacks, and also prevents the need for the server to be sent your password each time you log in.

Create or edit ~/.ssh/config as above, and make sure it contains this line-

UseRoaming no

Generate a keypair and add your public key to the server-

ssh-keygen
ssh-copy-id user@example.com

Test it out-

ssh user@example.com

At his point, you should disable password-based authentication in OpenSSH server. If you don’t control the server, then just get a good random password for your account to prevent it being brute-orced:

openssl rand -base64 32
cat /dev/urandom | head -c32 | base64

Set it on your account:

passwd