Posts on Mike's Software Blog

Farewell, wordpress!

Mon, 29 Dec 2025 00:00:00 +0000

I’ve recently re-built this blog using the static site generator, Hugo. This post is a few notes about the motivations and challenges behind that.

Old theme

New theme

New theme (dark mode)

Why Wordpress had to go

I’ve maintained some sort of blog since 2007. I’ve mainly hosted it on Wordpress, but also switched to a custom blog engine for a few years there.

Editing experience

One reason I considered switching away from Wordpress was that it was becoming tedious to use it for the type of writing I do.

I usually post technical notes which I’ve written in markdown (typical example), and support for that has changed over the years. I’ve most recently been using Jetpack’s markdown support, but previously used a standalone Wordpress plugin for markdown support, and also had plenty of posts marked up in HTML directly.

I needed to re-do the theme because it was starting to show its age, but at this point, editing old content would randomly break things.

The AI elephant in the room

It was really the rise of AI crawlers that killed Wordpress for me. This website is a lean self-hosted setup with a few small apps - not just a blog - and the huge volume of crawler traffic was causing the mariadb database behind Wordpress to run out of memory.

The crawler behaviour I observed was positively unhinged, aside from showing no consideration for resource usage. For now the main change I am making will be to serve this blog entirely from static files, since it will improve resilience to traffic spikes in general.

I didn’t record specific data on the web crawler traffic in my case, but this HackerNews thread has some sysadmins who are experiencing similar issues.

Making the switch

I chose Hugo because, among widely-used static side generators, it has the strong differentiator of being runnable as a single binary file. It was not a perfect fit though, and I needed to customise a few things.

Preserving URLs

I am bringing across a back catalogue of posts, and I want to continue to serve pages from the same URLs as before.

Hugo adds a trailing slash on the end of URLs, which does not match my existing scheme, and this behaviour is frustratingly difficult to override.

I took inspiration from one of the many threads about this on the Hugo forums.

“Fortunately I did come up with a solution in the end by overriding rel and relref to trim the trailing slash for all URLs except for the ones I wanted to keep the / on.” - nickjanetakis on the Hugo forums

I decided to implement this tip by forking the theme I’m using, which in my case is CaiJimmy/hugo-theme-stack, and editing any part that produces links. There’s a bit more to it, but it is mostly a lot of changes like this:

diff --git a/themes/hugo-theme-stack/layouts/partials/article-list/tile.html b/themes/hugo-theme-stack/layouts/partials/article-list/tile.html
index be5744b..c8219fd 100644
--- a/themes/hugo-theme-stack/layouts/partials/article-list/tile.html
+++ b/themes/hugo-theme-stack/layouts/partials/article-list/tile.html
@@ -1,6 +1,6 @@
 {{ $image := partialCached "helper/image" (dict "Context" .context "Type" .Type) .context.RelPermalink .Type }}
 <article class="{{ if $image.exists }}has-image{{ end }}">
-    <a href="{{ .context.RelPermalink }}">
+    <a href="{{ .context.RelPermalink | strings.TrimSuffix "/" }}">
         
         {{ if $image.exists }}
             <div class="article-image">

I also found default templates in the Hugo source code for rendering RSS feeds and sitemaps, copied them into the theme, and made similar changes.

In my case I’m running Apache, and this is the first part of the .htaccess for my /blog/ directory, which configures it to serve pages without the trailing slash among other fixes.

# Send to Hugo 404 page
ErrorDocument 404 /blog/404.html

# Do not add slashes to dir names
DirectorySlash Off

# Use mod_rewrite to provide other redirects
RewriteEngine On
RewriteBase /blog/

# Redirect trailing-slash URLs to non-trailing-slash URL
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^(.*)/$ $1 [R=301,L]

# Serve blog content when given dir name w/o trailing slash
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^(.*)$ %{REQUEST_URI}/index.html [L]

# Redirect old AMP URL's back to where canonical page would be (note: we do not check here if the page actually exists)
RewriteCond %{REQUEST_URI} ^(.*)/amp$
RewriteRule ^(.*)/amp$ $1 [R=301]

Exporting the content

I needed to port approximately 200 blog posts to from a live Wordpress to Hugo.

I saved every blog post to disk using Wordpress’s built-in export, which produces an extended RSS file containing HTML markup and metadata.

I combined this with the image site map, and with the help of some Python, was able to produce a directory for each post containing front matter, text and images.

I then also made a Wordpress database dump, imported on my desktop, and dumped the wp_posts table to JSON. This gave me access to the markdown representation of the article where it exists, via the post_content_filtered column in this table.

The 80/20 rule

Most of the blog content was intelligible at this point, but I still needed to do a time-consuming walk through of each blog post to check and correct the markup, so that it would properly use the theme’s features.

Just some of the examples:

Some blog posts place content in tabs, such as What is ESC/POS, and how do I use it?. I borrowed the tab implementation from the Geekdoc theme.
Images and code blocks had been consistently included using HTML <code> blocks before. I could mostly fix these with find/replace patterns.
Many blog posts were fully or partly written in HTML and needed to be converted to Markdown. Again this was mostly find/replace.
Some blog posts used raw blobs of HTML which cannot be represented in markdown, such as Some scripts to make word puzzles. I added a shortcode for raw HTML, and spent some time updating these pages to work with light/dark theming.

Remaining issues

Dark mode and transparency

The new styling is an improvement, but there are graphics with transparent backgrounds in my existing content which are difficult to see in dark mode, or when opened up in the image gallery view.

An example would be the circuit diagram in the screen captures below.

I’m rendering SVG files over a white background as a work-around - SVG files are not supported in the gallery plugin, but even this breaks in Firefox reader mode for example.

I plan to test a few possible solutions as I write new pages, and go back and fix posts where time allows.

Side note: Long ago a reader commented that my schematics were unreadable, and I’m only now realising that they could have been using AMP, a feed reader, an email client, or number of other alternative ways to access the blog which could have had this issue.

Comments

My blog allowed interactivity in the form of comments, but I made the difficult decision to remove this. I’ve saved old comments just in case, but don’t plan to re-publish them at this stage.

Static sites don’t directly support comments, but can embed a commenting widget from an external service.

I checked through a few of the available options, and couldn’t find any which made sense for this site when considering the trade-offs between user privacy, cost and moderation burden.

Subscribers

Another loss from the Wordpress ecosystem was the ability for users to subscribe to new posts, so that they receive a notification when they are published.

I exported my list of subscribers, so that I can send a one-off email to suggest migration if I ever set up an alternative system for receiving notifications.

Analytics

Wordpress also provided me with basic analytics. I didn’t use these extensively, but it was good to know which pages were popular.

This is the most likely area where I’ll deploy something self-hosted.

Search

My theme ships with a basic client-side search function. This is good enough, but I have some ideas for how to improve/replace this if I ever have the time to work on it.

Wrap-up

Hugo had a more challenging out-of-the-box experience than I expected, with multiple editions, no reference theme, and a need to fork & edit the theme to achieve any customisation.

I still think it was the right move for this site, and Hugo’s rough edges are worth it for the benefit of having a self-contained tool to build the site. Comparable static site generators seem to all require a working Node, Ruby or Python setup, plus their associated package managers, which is too many moving parts for publishing the occasional blog post.

I checked Google’s pagespeed score on one of my blog posts before and after this change. Even though higher resolution images are now being used, it went up from 79 to 91.

Switching between VFIO-enabled virtual machines

Thu, 25 Apr 2024 00:00:00 +0000

For the past few months, I’ve been using a setup for software development where I pass the mouse, keyboard and graphics card to a virtual machine via Linux VFIO. The setup is described in detail in its own blog post.

In short though, the idea is to provision a separate virtual machine for each project, so that I can install whatever tools I need to solve the problem at hand, with no real consequences if I break my install.

The problem

I have multiple virtual machines which are configured to use the same hardware, so I can only run one at a time. Before making the changes outlined in this blog post, the method I used to switch to a different VM was unwieldy: I needed to shut down the current VM, switch monitor inputs to the host graphics, start a different VM, then switch monitor inputs back the the VM graphics.

I intend to do all of my work in VM’s, so my basic goal was to make a ‘switch VM’ mechanism available from within a VM, to remove any need to interact with the host operating system directly.

Creating an API

My plan for this was to run an agent on the host, which will accept a request to switch VM’s. It would then shut down the current VM, and start up a different one instead.

This was the first time I tried using the libvirt Python bindings before, and I was able to quickly make a script which toggled between two VM’s, which are nested within the VM I’m using for development, because of course they are.

I then extended this to use an orderly shutdown, and to work with an arbitrary number of VM’s. CirrOS was not completing its boot in this environment, so I switched the test VM’s to Debian, and also passed through a USB device to each of them so that I would get an error if I ever tried to boot both VM’s simultaneously.

Last, I exposed this over HTTP as a simple API.

For the first iteration, I installed this on the host, and placed some scripts in a folder which is shared between all of the VM’s. The scripts use curl commands to call the API. To trigger a VM switch (or a shutdown of the host), I could go to a folder, and run one of these scripts.

Web UI

With that proof-of-concept out of the way, I made a web interface, so that I can use this instead of some bash scripts in a folder. This is a very simple Angular app.

GNOME extension

For the most part, I’m working on Debian using the GNOME desktop environment, which gives me some options for integrating this further into my desktop.

I first considered making a search provider, but settled instead for making a simple standalone extension to display an indicator instead, based on the example app shown running here.

As if nested virtualization was not mind-bending enough, I started testing this using a GNOME session within a GNOME session.

The extension is written in JavaScript, though the JavaScript API for GNOME extensions was very unfamiliar to me. There some obvious quality issues in the code, but it’s not being published to a repository, and it fires off the correct HTTP requests, so I’ll call it a success.

Deploying a GNOME extension manually is as simple as placing some files my home directory.

Result and next steps

I’ve bookmarked the path to my VM switcher web app in each of my VM’s.

And I’ve also installed the GNOME extension on the VM that runs on boot.

The idea of this setup is to have one virtual machine per project, so a good next step would be to make it easier to kick off the install process.

I’ve put the code for this project online at mike42/vfio-vm-switcher on GitHub.

Going all-in on GPU passthrough for software development

Mon, 11 Dec 2023 00:00:00 +0000

I recently spent some time improving my software development workflow at home, since my previous setup was starting to limit me. I settled on a configuration which uses GPU passthrough with the KVM hypervisor, running Debian as both a host and a guest.

This post aims to show some of the benefits and drawbacks of this more complex setup for my use case, as well as a specific combination of hardware and software which can be made to work.

Background and previous setup

I was using Debian testing as a host system, and provisioning project-specific virtual machines to work in via virt-manager. I use two monitors, and typically had an IDE (in the VM) open on one monitor, and a web browser (on the host) open on the other.

The setup is simple and effective:

The VMs ran in the QEMU user session - previously blogged about here.
The path /home/mike/workspace on the host was shared with every VM via a 9p fileshare.
Each desktop was set to 1920 x 1080 resolution, with auto-resize disabled. I kept the VM’s at this lower resolution to get acceptable graphical performance on 4K monitors when I set this up.

This allowed me to keep the host operating system uncluttered, which I value for both maintainability and security. Modern software development involves running a lot of code, such as dependencies pulled in via a language-specific package manager, random binaries from GitHub, or tools installed via a sketchy curl ... | sudo sh command. Better to keep that in a VM.

The main drawbacks are with the interface into the VM.

Due to scaling, I had imperfect image quality. I also had a small but noticeable input lag when working in my IDE.
Development involving 3D graphics was impractical due to the performance difference. I reported a trivial bug in Blender earlier this year but didn’t have an environment suitable for more extensive development on that codebase.
There was an audio delay when testing within the VM. I created a NES demo last year and this delay was less than ideal when it came to adding sound.

I was at one point triple-booting Debian, Ubuntu 22.04 and Windows, so that I could also run some software which wouldn’t work well in this setup.

Hardware

GPU passthrough on consumer hardware is highly dependent on motherboard and BIOS support. The most critical components in my setup are:

Motherboard: ASUS TUF X470-PLUS GAMING
CPU: AMD Ryzen 7 5800X
Monitors: 2 x LG 27UD58
Graphics card: AMD Radeon RX 6950 XT
- Installed at PCIEX16_1, directly connected to the CPU
- Connected to DisplayPort inputs on the two monitors

I added one component specifically for this setup:

Second graphics card: AMD Radeon RX 6400
- Installed at PCIEX16_2, connected to chipset
- Connected to HDMI inputs on the two monitors (one directly, one via an adapter)

The second graphics card will be used by the host. The RX 6400 is a low-cost, low-power, low-profile card which supports UEFI, Vulkan and the modern amdgpu driver.

In the slot I’ve installed it, it’s limited to PCIE 2.0 x4, and 1920 x 1080 is the highest resolution I can run at 60 Hz on these monitor inputs. I only need to use this to set up the VM’s, or as a recovery interface, so I’m not expecting this to be a major issue.

Inspecting IOMMU groups

On a fresh install of Debian 12, I started walking through the PCI passthrough via OVMF guide on the Arch Wiki, adapting it for my specific setup as necessary.

I verified that I was using UEFI boot, had SVM and IOMMU enabled, and I also enabled resizable BAR. I did not set any IOMMU-related Linux boot options, and also could not find a firmware setting to select which graphics card to use during boot.

Using the script on the Arch Wiki, I then printed out the IOMMU groups on my hardware, which are as follows:

IOMMU Group 0:
	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
	00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2:
	00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3:
	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4:
	00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5:
	00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6:
	00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7:
	00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
	00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9:
	00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 10:
	00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 11:
	00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 12:
	00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
	00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13:
	00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 [1022:1440]
	00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 [1022:1441]
	00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 [1022:1442]
	00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 [1022:1443]
	00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 [1022:1444]
	00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 [1022:1445]
	00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 [1022:1446]
	00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 [1022:1447]
IOMMU Group 14:
	01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 15:
	02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43d0] (rev 01)
	02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
	02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
	03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
	08:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c7)
	09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
	0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 24 [Radeon RX 6400/6500 XT/6500M] [1002:743f] (rev c7)
	0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
	0b:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Blue SN550 NVMe SSD [15b7:5009] (rev 01)
IOMMU Group 16:
	0c:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c0)
IOMMU Group 17:
	0d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 18:
	0e:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6950 XT] [1002:73a5] (rev c0)
IOMMU Group 19:
	0e:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
IOMMU Group 20:
	0f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 21:
	10:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 22:
	10:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 23:
	10:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 24:
	10:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]

Passthrough for an IOMMU group is all-or nothing. For example, all of the chipset-connected devices are grouped together, and I can’t pass any of them through to a VM unless I pass them all through.

I’m mainly interested in a graphics card, an audio device, and a USB controller, and helpfully I have one of each isolated in their own groups, presumably because they are connected to PCIe lanes which go directly to the CPU.

Graphics

I created a virtual machine, also Debian 12, in the QEMU system session. The only important setting for now is the chipset and firmware, and in my case I selected Q35, and the OVMF UEFI firmware. GPU passthrough will not work if a virtual machine is booting with legacy BIOS.

I use the WebGL Aquarium as a simple test for whether 3D acceleration is working. It runs much faster on the host system (this is using the RX 6400). The copy in the VM runs at just 3 frames per second, using SPICE display and virtio virtual GPU at this stage.

The next step was to isolate the GPU from the host. The relevant line from lspci -nn is:

0e:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6950 XT] [1002:73a5] (rev c0)

The vendor and device ID for this card is shown at the end of the line, 1002:73a5.

This needs to be added to the vfio-pci.ids boot option, which in my case involves updating /etc/default/grub.

GRUB_CMDLINE_LINUX="vfio-pci.ids=1002:73a5"

This is then applied by running update-grub2, and rebooting. It’s apparently possible to accomplish this without a reboot, but I’m following the easy path for now.

I verified that it worked by checking that the vfio-pci kernel module is in use for this device.

$ lspci -v
0e:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6950 XT] (rev c0) (prog-if 00 [VGA controller])
...
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

Before booting up the VM, I added the card as a “Host PCI Device”. It was then visible in lspci output within the VM, but was not being used for video output yet.

To encourage the VM to output to the physical graphics card, I switched the virtualised video card model from “virtio” to “None”.

Booting up the VM after this change, the SPICE display no longer produces output, but USB redirection still works. I passed through the keyboard, then the mouse.

I then switched monitor inputs to the VM, now equipped with a graphics card, and the WebGL aquarium test ran at 30 FPS.

Now that I was switching between near-identical Debian systems, I started using different desktop backgrounds to stay oriented.

Audio output

I needed to make sure that audio output worked reliably in the virtual machine.

Sound was working out of the box through the emulated AC97 device, but I checked an online Audio Video Sync Test, and confirmed that there was a significant delay, somewhere in in the 200-300ms range. This is not good enough, so I deleted the emulated device and decided to try some other options.

I first tried passing through the audio device associated with the graphics card, identified as 1002:ab28.

0e:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]

I took the ID, and added it to the option in /etc/default/grub. Now that there are multiple devices, they are separated by commas.

GRUB_CMDLINE_LINUX="vfio-pci.ids=1002:73a5,1002:ab28"

As before, I ran update-grub2, rebooted, added the device to the VM through virt-manager, booted up the VM, and tried to use the device.

With that change, I could connect headphones to the monitor and the audio was no longer delayed. This environment would now be viable for developing apps with sound, or following along with a video tutorial for example.

Audio input

Next I attempted to pass through the on-board audio controller to see if I could get both audio input and output. Discussions online suggest that this doesn’t always work, but there is no harm in trying.

I’ll skip through the exact process this time, but I again identified the device, isolated it from the host, and passed it through to a VM.

10:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]

Output worked immediately, but the settings app did not show any microphone level, and attempts to capture would immediately stop. I did some basic reading and troubleshooting, but didn’t have a good idea of what was happening.

What worked for me was blindly upgrading my way out of the problem by switching to the Debian testing rolling release in the guest VM.

I was then able to see the input level in settings, and capture the audio with Audacity.

Audio input is not critical for me, but does allow me to move more types of work into virtual machines without switching to a USB sound card.

USB

My computer has two USB controllers. One is available for passthrough, while the other is in the same IOMMU group as all other chipset-attached devices.

02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43d0] (rev 01)
10:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

The controller I passed through is the “Matisse USB 3.0 Host Controller”, device ID 1022:149c, which I initially expected would have plenty of USB 3 ports attached to it.

Reading the manual for my motherboard more carefully, I discovered that this controller is only responsible for 2 USB-A ports and 1 USB-C port, or 20% of the USB ports on the system.

I use a lot of USB peripherals for hardware development. More physical USB ports would be ideal, but I can work around this by using a USB hub.

I’ll also continue to use USB passthrough via libvirt to get the keyboard and mouse into the VM. Instead of manually passing these through each time I start the VM, I added each device in the configuration as a “Physical USB Device”.

This automatically connects the device when the VM boots, and returns it to the host when the VM shuts down, without the SPICE console needing to be open. If I need to control the host and guest at the same time, I can connect a different mouse and keyboard temporarily - I’ve got plenty of USB ports on the host after all.

If this ever becomes a major issue, it should also be possible to switch this setup around, and pass through the IOMMU group containing all chipset-attached PCIe devices instead of the devices I’ve chosen. This would provide extensive I/O and expansion options to the VM, at the cost of things like on-board networking, SATA ports, and an NVMe slot on the host. The 3 USB ports on the “Matisse USB 3.0 Host Controller”, if left to the host, would be just enough for a mouse, keyboard, and USB-C Ethernet adapter.

On my previous setup, I used a 9p fileshare to map a directory on the host to the same path on every VM, allowing an easy way to exchange files.

The path was /home/mike/workspace - a carry-over from when I used Eclipse. In practice it has been slower to work on a fileshare, so I’ll switch to developing in a local directory.

I’ll still set up a fileshare, but with two changes:

I’ll map a share to a more generic /home/mike/Shared, and start to back up anything that lands there.
I’ll use virtiofs instead of virtio-9p. This claims to be more performant, and it’s apparently possible to get this working on Windows as well.

This is added as a hardware device in virt-manager on the host.

In the guest, I added the following line to /etc/fstab.

shared /home/mike/Shared/ virtiofs

To activate this change, I would previously have created the mount-point and run mount -a. I recently learned that systemd creates mount-points automatically on modern systems, so I instead ran the correct incantation to trigger this process, which is:

$ sudo systemctl daemon-reload
$ sudo systemctl restart local-fs.target

Power management

In my current setup, sleep/wake causes instability. I initially disabled idle suspend on the host in GNOME:

While testing a VM which I had configured to start on boot, the system decided to go to sleep while I was using it. In hindsight this makes complete sense: as far as the host could tell, it was sitting on the login page with no mouse or keyboard input, and had been configured to go to sleep after an idle time-out (the setting within GNOME only applies after login).

Dec 05 16:24:43 mike-kvm systemd-logind[706]: The system will suspend now!

To avoid this, I additionally disabled relevant systemd units (source of this command).

$ sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
Created symlink /etc/systemd/system/sleep.target → /dev/null.
Created symlink /etc/systemd/system/suspend.target → /dev/null.
Created symlink /etc/systemd/system/hibernate.target → /dev/null.
Created symlink /etc/systemd/system/hybrid-sleep.target → /dev/null.

I’ve also configured each VM to simply blank the screen when idle, since allowing a guest to suspend causes its own problems.

Recap of unexpected issues

Despite my efforts to plan around the limitations of my hardware, I did hit four unanticipated problems, mostly highlighted above.

I assumed that there would be a setting in my BIOS to change which graphics card to use during boot, but I couldn’t find one.
The audio devices associated with both of my graphics cards had the same vendor and device ID. The way I have it configured, no audio output is available on the host as a result.
Microphone input didn’t work on the HD Audio Controller until I upgraded the guest operating system. The usual workaround for this is apparently to use a USB sound card.
I misunderstood the limitations of my USB setup, so I have fewer USB ports directly available in the VM than I had hoped for.

Wrap-up and future ideas

I really like the idea of switching into an environment which only has the tools I need for the task at hand. Compared with my previous setup for developing in a VM, GPU passthrough (and sound controller passthrough, and USB controller passthrough) is a huge improvement.

I’ve manually provisioned 3 VM’s, including a general-purpose development VM which has a mix of basic tools so that I can get started.

Since I can only have one VM using the graphics card at a time, this setup works similarly to having multiple operating systems installed as a multi-boot setup. It does however have far better separation between the systems - they can’t read and write each others’ disks for example.

The next steps for me will be to streamline the process of switching VM’s and shutting down the system. Both of these currently require me to manually switch monitor inputs to interact with the host, which I would prefer to avoid.

Building a 1U quiet NAS

Sun, 22 Oct 2023 00:00:00 +0000

I recently built a compact, quiet rackmount NAS for home. I haven’t seen any builds quite like it online, so I’m writing a bit about how it came together.

The problem

My old backup was a mirrored pair of 2 TB hard drives in an old desktop computer, with a portable 2 TB hard drive as an off-site copy. The disks are now 9 years old, and 2 TB is small enough that I need to ration the space. I also recently repurposed most of the components in that system, so I no longer have an up-to-date backup.

I want to solve this properly, and hopefully build a replacement setup which I can install in my network cabinet, to run 24/7.

But if I’m going to do that, the must-haves are:

It fits in a 1U rack-mount form-factor with maximum 25cm depth
It’s quiet
It has front USB for making an offline copy of the backups
It has 2 SATA disks - I don’t want to be running a NAS off USB-SATA adapters, for example
It has wired ethernet - my network is 1 Gigabit

Nice-to-haves would be:

Fast front USB
Hot-swap disks or more disks
Faster network

I’ve worked with servers, and I’ve built small-form-factor computers, so how hard can it be to build a small-form-factor server?

Parts list

I spent a lot of time sketching out possible builds in LibreOffice Draw. Every combination of parts had some compromise, which is a familiar theme from small-form-factor PC building.

I ended up deciding on parts which follow normal PC standards, hopefully giving me a good chance of keeping it working for many years.

Case: Case Athena Power RM1UC138
Power supply: HDPLEX GAN 250W
Motherboard: Topton N6005 Mini ITX
RAM: Crucial 16GB (2 x 8GB) DDR4 3200 SO-DIMM
Boot disk: Samsung 970 EVO Plus 1TB M.2 SSD
Storage disks: 2 x Samsung 870 QVO 8TiB 2.5" SSD
Fans: 4 x Noctua A4x20 PWM
Drive cage: Icy Dock MB608SP-B - 6 x 2.5" SATA bays
Set of 6 x 50cm SATA cables

There are some compatibility issues in the above part list, and it took a little bit of problem-solving to get everything working together.

Closer look at the Athena Power RM1UC138

Short-depth 1U computer cases are nearly impossible to find in Australia. I ordered the Athena Power RM1UC138 from the United States, which is an OEM case with some flexibility built in.

On the front it has a 5.25" bay, which I plan to use to add a drive cage. The front USB is only USB 2.0, which would normally be a disadvantage, but in this instance is a good match for the motherboard I’m using. The power button looks like a toggle switch, but is actually a momentary switch.

The side shows that the rack ears can be put on the front or back, allowing the case to be mounted in either direction.

On the back there is a wire mesh panel for cutting out a custom I/O shield, which is handy since I have an off-brand motherboard.

On the inside, it’s configured for 2 x 3.5" hard drives by default. It also includes 2 x 2-pin 12v fans. They are loud like you would find in most managed network switches, but are not jet-engine loud like most servers. The fan controller simply distributes 12v, and has no speed control.

This is not a very common case, and I read everything I could find online about it. In order to help the next person who is searching for it, here are two more random pieces of information which I could not confirm until I had the case in-hand:

Stand-offs on the case are all 4mm tall and non-removable
Screw spacing of USB 2.0 front panel connector is approx 30.5mm (from centre of each screw). Stacked USB 3 headers that are 30mm spacing are available online and could be made to work.

Closer look at the Topton N6005 Mini ITX

I chose to use a Topton motherboard with a built-in Intel N6005 CPU for this build, since the alternatives were either too tall, use a socketed CPU, use an old CPU, would require add-in cards to get multiple SATA ports, or were not sold in Australia. All of these would make it far more difficult to complete the small, quiet build which I was aiming for.

From the few threads online about this board, I gathered that it is fussy about RAM compatibility, so I booted it up at the first opportunity with an SSD containing Pop!_OS to check that it worked. It’s not my use case, but this motherboard would definitely be viable as a lightweight desktop.

The specific memory I used was a 16 GB kit with the model number CT2K8G4SFRA32A. According to Intel’s product documentation, the N6005 only supports 16 GB maximum, and while I could find claims that higher-capacity memory does work, I couldn’t find anybody who posted actual part numbers.

I was happy to find that the built-in cooler is inaudible at idle loads. This was a bit of a risk: the cooler doesn’t have standard dimensions, so I couldn’t have easily replaced it with an alternative if it was noisy. Based on other people’s experiences with this board, I re-pasted the cooler with Noctua NT-H1 thermal paste, to hopefully help keep temperatures down at higher loads so that the fan will not need to spin up as much. I also avoided using the M.2 slot which receives the most hot air from the cooler.

Topton also sells an N5105 variant which appears to be more popular (more info here), as well as an alternative layout which has a PCIe slot instead of a second M.2 slot.

Custom power supply adapter plate

The case is designed for a Flex ATX power supply, which is not what I’m using. I’ve instead opted to use a passively-cooled HDPLEX GAN 250W power supply, which ships with both an IEC C6 and IEC C14 cable.

I needed to choose one of these cables, and figure out how to securely mount it to the case.

I designed an adapter plate in FreeCAD around the included IEC C6 cable, since it had threaded holes, and screws were included.

I ordered it from a prototype supplier in laser cut 1mm steel, painted in matte black.

This is the first time I’ve used FreeCAD to design a part, and parametric CAD certainly has a learning curve. For this build, it was well worth it, since the result is better (and safer) than anything I could have improvised.

At the time of writing this post, HDPLEX sells plates for mounting their IEC C14 cables in cases accepting SFX and ATX power supplies, but none for cases which accept Flex ATX power supplies.

Custom fan controller

Cooling this build quietly was always going to be a challenge. The case shipped with 2 x 12 V fans, and had a simple splitter which ran them at max speed, which was just too loud.

I designed a replacement fan controller in KiCad, which allows me to upgrade to high-quality 4-pin fans with PWM speed control, and to set the speed using a potentiometer. I wrote about prototyping this in a separate blog post.

This photo shows the custom controller alongside the original one it replaces.

As you can probably guess, I’ve designed this to use the same mounting location, at the front of the case. My power supply has no Molex connectors, so I’m using a SATA-Molex power adapter.

The main drawback to this simple design is that once I close up the case, I can no longer adjust the fan speed.

Final assembly

Before continuing any further, I took apart the case completely and deleted three standoffs with a belt sander, to leave a flat area for power supply installation later.

Once I got the case back together, the motherboard went in first. I raised it by 1mm using plastic washers, hoping to line it up better with the I/O shield included with the motherboard.

The I/O opening for 1U servers is narrower than standard PC builds, so I needed to cut the I/O shield, which I unfortunately did not do correctly.

Since that did not work, I carefully marked and cut out the wire mesh I/O shield included with the case instead. I still left the motherboard raised up on 1mm washers, though this is not necessary anymore. You need to use slightly longer screws if you try this.

After that I installed the four case fans, plus the fan controller. I’m using a front-to-back airflow, with 2 x 40mm fans mounted at the front, and 2 x 40mm fans at the back. I added a Y splitter to the back fans, which did not have long enough cables to reach the fan controller.

The next component I installed was the drive cage. It’s worth mentioning at this point that the drive cage also has a fan header, which is the same as a 3-pin header that you would find on a PC motherboard. It supplies a different voltage for each of the speed settings. Medium is approximately 7.5 volts and is relatively quiet with the included “Good quality DC fan” fan, and high speed is 12 volts. I set mine to off but left the fan installed.

To install items into the 5.25" bay in this case, you attach a bracket, then fasten it from above. The bracket allows the depth to be adjusted as well.

I also installed disks in the drive cage at this point, and numbers on the front. Disk 1 is connected to SATA0 on the motherboard, disk 2 is connected to SATA1, and so on.

Next was the power supply. I installed the custom plate for the power connector, and also installed the mounting plate on the bottom of the PSU so that it would have a flat surface. After confirming that it would fit, I cleaned both surfaces with alcohol, and applied double-sided tape.

I then followed a rehearsed path to drop the PSU into place. There is no opportunity to adjust it once it sticks.

At this point I connected everything up and booted up the system to start checking for problems, since it’s easier to troubleshoot in this state. Two modifications I made here were to disconnect the bright red HDD LED, and to introduce a SATA power Y splitter, because the power supply SATA cables were stretched to the limit.

It took a lot of work (and cable ties) to arrange the cables flat so that the case could close. In defence of cable ties, they do make maintenance more difficult, but that’s a worthwhile trade-off for keeping cables clear of airflow paths, fan blades, and the guillotine-like action of the top cover sliding shut.

Completed build

After closing the case, the build is, 434mm × 254mm x 44mm, or 4.8 litres, excluding rack ears.

This is how it appears from the front.

And this is how it appears from the back.

Software

I’m starting with Proxmox, with OpenMediaVault deployed as a virtual machine. I haven’t used either of these before, but both are Debian-based and provide convenient web front-ends to the tools I would otherwise be configuring on the command-line.

I’m passing through the disks as block devices. Running the NAS like this should make it possible to provision extra workloads which need their own SATA disks in future, or to switch from OpenMediaVault to stock Debian if necessary, all without connecting a monitor.

Within OpenMediaVault, I’ve configured Linux software RAID, with an ext4 filesystem, shared via Samba, and can access that file share over the network.

I’ve enabled some basic power management features such as C-states. The system idles in the range of 12-14 watts measured from the wall, and goes up to 20 watts when moving files around.

I don’t need a lot of disk capacity, so I’ve been able to preserve a useful property of my old setup, where every disk in the system has a full copy of the data, in a format which can be understood by a normal Linux system. This it makes single-disk recovery possible using any surviving disk from the system on practically any computer, and that disk can be from either an offline copy or one of the disks in the RAID mirror.

I haven’t tested the process of making an offline copy of the backup volume, but that will be up next.

Wrap-up

This is possibly the most effort I’ve ever put into a PC build. The only unexpected issue I encountered is how heavy it is, and wont be rack-mounting it until I get some generic rails.

The computer uses a strange mix of parts, but meets my requirements well. I hope that by writing this up, I’ll be providing some useful notes to anybody attempting to build something similar.

This project also gave me a chance to practice my entry-level CAD skills to build something which I’ll actually be using. I find a lot of utility in paper prototyping, and printed each design in 1:1 scale to check the physical dimensions before ordering anything.

For the circuit board, I used a print-out to check each part footprint, as well as the hole locations for fitting it in the case.

As with many of the projects which I blog about, I’ve put the design files up on GitHub. The fan controller is at mike42/fan-controller-athena-power, while the Flex ATX adapter plate is at mike42/flexatx-adapter-hdplex.

Controlling computer fans with a microcontroller

Thu, 21 Sep 2023 00:00:00 +0000

I’m currently working on building a small computer, and want to add some 4-pin computer fans, running quietly at a fixed speed.

This blog post is just a quick set of notes from prototyping, since it covers a few topics which I haven’t written about before.

The problem

The speed of 4-pin computer fans can be controlled by varying the duty cycle on a 25 KHz PWM signal. This signal normally comes from a 4-pin case fan header on the motherboard, which will not be available in this situation. Rather than run the fans at 100%, I’m going to try to generate my own PWM signal.

Two main considerations led me to choose to use a microcontroller for this:

I’ll need some way to adjust the PWM duty cycle after building the circuit, because I don’t know which value will give the best trade-off between airflow and noise yet.
Fans need a higher PWM duty cycle to start than they do to run. If I want to run the fans at a very low speed, then I’ll need them to ramp up on start-up, before slowing to the configured value.

It’s worth noting that I’m a complete beginner with microcontrollers. I’ve run some example programs on the Raspberry Pi Pico, but that’s it.

First attempt

I already had MicroPython running on my Raspberry Pi Pico, so that was my starting point.

For a development environment, I installed the MicroPython plugin for PyCharm, since I use JetBrains IDE’s already for programming. Most guides for beginners suggest using Thonny.

There are introductory examples on GitHub at raspberrypi/pico-micropython-examples which showed me everything I needed to know. I was able to combine an ADC example and a PWM example within a few minutes.

import time
from machine import Pin, PWM, ADC, Timer

level_input = ADC(0)
pwm_output = PWM(Pin(27))
timer = Timer()

# 25 KHz
pwm_output.freq(25000)


def update_pwm(timer):
    """ Update PWM duty cycle based on ADC input """
    duty = level_input.read_u16()
    pwm_output.duty_u16(duty)


# Start with 50% duty cycle for 2 seconds (to start fan)
pwm_output.duty_u16(32768)
time.sleep(2)

# Update from ADC input after that
timer.init(mode=Timer.PERIODIC, period=100, callback=update_pwm)

On my oscilloscope, I could confirm that the PWM signal had a 25 KHz frequency, and that the code was adjusting the duty cycle as expected. When the analog input (above) is set to a high value, the PWM signal has a high duty cycle.

When set to a low value, the PWM signal has a low duty cycle.

But can it control fans?

I wired up a PC fan to 12 V power, and also sent it this PWM signal, but the speed didn’t change. This was expected, since I knew that I would most likely need to convert the 3.3 V signal up to 5 V.

I ran it through a 74LS04 hex inverter with VCC at 5 V, which did the trick. I could then adjust a potentiometer, and the fan would speed up or slow down.

I captured the breadboard setup in Fritzing. Just note that there are floating inputs on the 74LS04 chip (not generally a good idea) and that the part is incorrectly labelled 74HC04, when the actual part I used was a 74LS04.

This shows a working setup, but it’s got more components than I would like. I decided to implement it a second time, on a simpler micro-controller which can work at 5 V directly.

Porting to the ATtiny85

For a second attempt, I tried using an ATtiny85. This uses the AVR architecture (of Arduino fame), which I’ve never looked at before.

This chip is small, widely available, and can run at 5 V. I can also program it using the TL-866II+ rather than investing into the ecosystem with Arduino development boards or programmers.

I found GPL-licensed code by Marcelo Aquino for controlling 4-wire fans.

After a few false starts trying to compile manually, I followed this guide to get the ATtiny85 ‘board’ definition loaded into the Arduino IDE.

From there I was able to build and get an intel hex file, using the “Export compiled binary” feature.

Writing fuses

The code assumes an 8 MHz clock. The Attiny85 ships from the factory with a “divide by 8” fuse active. This needs to be turned off, otherwise the clock will be 1 MHz. This involves setting some magic values, separate to the program code.

I found these values using a fuse calculator.

The factory default for this chip is:

lfuse 0x62, hfuse 0xdf, efuse 0xff.

To disable the divide-by-8 clock but leave all other values default, this needs to be:

lfuse 0xe2, hfuse 0xdf, efuse 0xff.

I am using the minipro open source tool to program the chip, via a TL-866II+ programmer. First, to get the fuses:

$ minipro -p ATTINY85@DIP8 -r fuses.txt -c config
Found TL866II+ 04.2.126 (0x27e)
Warning: Firmware is out of date.
  Expected  04.2.132 (0x284)
  Found     04.2.126 (0x27e)
Chip ID: 0x1E930B  OK
Reading config... 0.00Sec  OK

This returns the values expected in a text file.

lfuse = 0x62
hfuse = 0xdf
efuse = 0x00
lock = 0xff

I then set lfuse = 0xe2, and wrote the values back with this command.

$ minipro -p ATTINY85@DIP8 -w fuses.txt -c config
Found TL866II+ 04.2.126 (0x27e)
Warning: Firmware is out of date.
  Expected  04.2.132 (0x284)
  Found     04.2.126 (0x27e)
Chip ID: 0x1E930B  OK
Writing fuses... 0.01Sec  OK
Writing lock bits... 0.01Sec  OK

Writing code

Now the microcontroller is ready to accept the exported binary containing the program.

$ minipro -s -p ATTINY85@DIP8 -w sketch_pwm.ino.tiny8.hex -f ihex
Found TL866II+ 04.2.126 (0x27e)
Warning: Firmware is out of date.
  Expected  04.2.132 (0x284)
  Found     04.2.126 (0x27e)
Chip ID: 0x1E930B  OK
Found Intel hex file.
Erasing... 0.01Sec OK
Writing Code...  1.09Sec  OK
Reading Code...  0.45Sec  OK
Verification OK

With the chip fully programmed, I wired it up on a breadboard with 5 V power.

Checking the output again, the signal was the correct amplitude this time, but the frequency does move around a bit. This is likely because I’m using the internal RC timing on the chip, which is not very accurate. My understanding is that anything near 25 KHz will work fine.

Updates

I made only one change to Marcelo’s code, which is to spin the fan at 50% for a few seconds before using the potentiometer-set value. This is to avoid any issues where the fans fail to start because I’ve set them to run at a very low PWM value.

/*
 *                         ATtiny85
 *                      -------u-------
 *  RST - A0 - (D 5) --| 1 PB5   VCC 8 |-- +5V
 *                     |               |
 *        A3 - (D 3) --| 2 PB3   PB2 7 |-- (D 2) - A1  --> 10K Potentiometer
 *                     |               | 
 *        A2 - (D 4) --| 3 PB4   PB1 6 |-- (D 1) - PWM --> Fan Blue wire
 *                     |               |      
 *              Gnd ---| 4 GND   PB0 5 |-- (D 0) - PWM --> Disabled
 *                     -----------------
 */
 
// normal delay() won't work anymore because we are changing Timer1 behavior
// Adds delay_ms and delay_us functions
#include <util/delay.h>    // Adds delay_ms and delay_us functions

// Clock at 8mHz
#define F_CPU 8000000  // This is used by delay.h library

const int PWMPin = 1;  // Only works with Pin 1(PB1)
const int PotPin = A1;

void setup()
{
  pinMode(PWMPin, OUTPUT);
  // Phase Correct PWM Mode, no Prescaler
  // PWM on Pin 1(PB1), Pin 0(PB0) disabled
  // 8Mhz / 160 / 2 = 25Khz
  TCCR0A = _BV(COM0B1) | _BV(WGM00);
  TCCR0B = _BV(WGM02) | _BV(CS00); 
  // Set TOP and initialize duty cycle to 50%
  OCR0A = 160;  // TOP - DO NOT CHANGE, SETS PWM PULSE RATE
  OCR0B = 80; // duty cycle for Pin 1(PB1)
  // initial bring-up: leave at 50% for 4 seconds
  _delay_ms(4000);
}

void loop()
{
  int in, out;
  // follow potentiometer-set speed from there
  in = analogRead(PotPin);
  out = map(in, 0, 1023, 0, 160);
  OCR0B = out;
  _delay_ms(200);
}

The wiring of the breadboard is shown below. The capacitor is 0.1 µF for decoupling.

This is far more compact than the Raspberry Pi Pico prototype. I could also miniaturise it further by simply using surface-mount versions of the same components, where using an RP2040 microcontroller from the Pico directly on a custom board would incur some design effort.

Next steps & lessons learned

Although this project is simple, I had to learn quite a few things to prototype it successfully. Using a generic chip programmer like the TL-866II+ appears to be uncommon in the AVR world, and most online guides instead suggest repurposing an Arduino development board or using an Arduino ICSP programmer to program the Attiny85. I was glad to confirm that I could use my existing hardware instead of buying ecosystem-specific items which I would have no other use for. I find the development experience to be far better with the Raspberry Pi Pico, and that’s what I would be choosing for a more complex project.

I also captured the breadboard wiring in Fritzing for this blog post. The diagrams are clearer than a photo of a breadboard, but I’m not confident that they communicate information about the circuit as well as alternative approaches. For future posts, I’ll return to using KiCad EDA for schematic capture, unless there is some reason to highlight the physical layout of a breadboard prototype.

As a next step, I’ll be building a simple break-out PCB for a specific computer case to power the fans and supply a PWM signal, based on the ATtiny85 prototype shown here.

Converting my 65C816 computer project to 3.3 V

Thu, 12 Jan 2023 00:00:00 +0000

I recently spent some time re-building my 65C816 computer project to run at 3.3 volts, instead of 5 volts which it used previously. This blog post covers some of the things I needed to take into consideration for the change.

It involved making use of options which I added when I designed this test board, as well as re-visiting all of the mistakes in that design.

Re-cap: Why 3.3 V is useful

One of the goals of this project is to build a modern computer which uses a 65C816 CPU, using only in-production parts.

A lot of interesting retro chips run at 5 V, but it’s much easier to find modern components which run at 3.3 V. I can make use of these options without adding level-shifting if I can switch important buses and control signals to 3.3 V.

ROM

The ROM chip is the most visible change, because the chip has a different footprint. When assembling this board for 5 V use, I used an AT28C256 Parallel EEPROM, in a PDIP-28 ZIF socket. I couldn’t find 3.3 V drop-in replacement for this, so I added a footprint for a SST39LF010 flash chip, which has a similar-enough interface.

This was the first time I’ve soldered a surface-mount PLCC socket. I attempted to solder this with hot air, which was unsuccessful, so I instead cut the center of the socket out so that I could use a soldering iron. I then added a small square of 1000 GSM card (approx 1mm thick) as a spacer under the chip.

I also added a new make target to build the system ROM for this chip, which involves padding the file to a larger size, and invoking minipro with a different option so that it could write the file. I’m using a TL866II+ for programming, and adapter boards are available for programming PLCC-packaged chips.

$ make flashrom_sst39lf010
cp -f rom.bin rom_sst39lf010.bin
truncate -s 131072 rom_sst39lf010.bin
minipro -p "SST39LF010@PLCC32" -w rom_sst39lf010.bin
Found TL866II+ 04.2.126 (0x27e)
Warning: Firmware is newer than expected.
  Expected  04.2.123 (0x27b)
  Found     04.2.126 (0x27e)
Chip ID OK: 0xBFD5
Erasing... 0.40Sec OK
Writing Code...  8.38Sec  OK
Reading Code...  1.18Sec  OK
Verification OK

UART

The UART chip is an NXP SC16C752B, which is 3.3 V or 5 V compatible. I rescued one of these from an adapter board (previous experiment). This was my first time de-soldering a component with hot air gun. Hopefully it still works!

My FTDI-based USB/UART adapter has a configuration jumper which I adjusted to 3.3 V.

Clock, reset and address decode

I again used a MIC2775 supervisory IC for power-on reset, though the exact part was different. I previously used a MIC2275-46YM5 (4.6 V threshold), where the re-build used a MIC2275-31YM5 (3.1 V threshold).

I needed a 1.8432 MHz oscillator for the UART. The 3.3 V surface-mount part I substituted in had a different footprint, which I had prepared for.

I also prepared for this change by using an ATF22LV10 PLD for address decoding, which is both 3.3 V and 5 V compatible. The ATF22V10 which I used for earlier experiments works at 5 V only.

SD card

I installed the DEV-13743 SD card module more securely this time by soldering it in place and adding some double-sided tape as a spacer. This module is compatible with 3.3 V or 5 V.

The level-shifting and voltage regulation on this module is now superfluous, so I could probably simplify things by adding an SD card slot and some resistors directly in a later revision.

Modifications

There are three errors in the board which I know about (all described in my earlier blog post). This is my second time fixing them, so I tried to make sure the modifications were neat and reliable this time.

Firstly, the flip-flop used as a clock divider is wired up incorrectly, so I cut a trace and run a wire under the board.

Second, the reset button pin assignments are incorrect. Simply rotating the button worked on the previous board, but it wasn’t fitted securely. This time I cut one of the legs and ran a short wire under the board, and the modification is barely noticeable.

Lastly, the address bus is wired incorrectly into the chip which selects I/O devices. I previously worked around this in software, but this time I cut two traces and ran two wires (the orange wires in the picture). I made an equivalent modification to the 5 V board, so that I could update the software and test it.

The wire I was using for these mods is far too thick and inflexible. I’ve added some 30 AWG silicon-insulated wire to my inventory for the next time I need to do this type of work, which should be more suitable.

Wrap-up

I transferred components from the old board, and attempted to boot it every time I made change. The power-on self test routine built in to the ROM showed that each new chip was being detected, and I soon had a working 65C816 test board again, now running at 3.3 V. I’m using my tiny linear power supply module to power it.

I can now interface to a variety of chips which only run at 3.3 V. This opens up some interesting possibilities for adding peripherals, which I hope to explore in future blog posts.

It will be more difficult to remove and re-program the ROM chip going forward though. This is hopefully not a problem, since can now run code from an SD card or serial connection as well (part 1, part 2).

Porting the Amiga bouncing ball demo to the NES

Wed, 23 Nov 2022 00:00:00 +0000

Earlier this year, I ported the Amiga bouncing ball demo to the Nintendo Entertainment System. This is a video capture from a NES emulator.

I completed this back in January, but I’m only publishing this blog post now, because I was considering entering it into a demo competition.

How it works

If you are familiar with NES development, then you will probably notice that there is nothing ground-breaking going on here. The ball is made up of 64 8x8 sprites, bouncing around the screen over a static background.

There are two different versions of the ball in sprite memory, and with some palette swapping, this can be stretched to 4 frames of animation, just enough to make the ball appear to spin.

There is enough sprite memory to extend this to 8 frames of animation without any banking tricks, but I would need to start again from scratch to achieve that.

Pre-rendered 3D

I drew the ball in Blender, with twice the number of segments required for the final image. This is a UV sphere with 8 rings, and 32 segments.

I coloured each face with one of 4 colours, then rendered it with the Workbench renderer, which I had configured to use Flat lighting, no anti-aliasing, and the texture colour.

I also tilted the ball.

This gave me a crisp, high-resolution ball with solid colours.

To test the idea, I processed this down to a 64x64 image on a transparent background. I need to substitute colours, so there is no antialiasing here.

I then wrote up a Python script to swap out colours to make the 4-frame pattern. Note that frames 3 and 4 are the same as frames 1 and 2, but with white and red swapped.

I ran this through ImageMagick to convert it into a GIF preview.

convert -delay 3.5 -loop 0 *.png animation.gif

The result appeared workable, so I went about making the same animation in a NES rom.

NES implmentation

On the NES, it’s not possible to create the spinning ball effect with palette changes only, because it would require four colours plus transparency. To overcome this, I split the image into two frames, each stored as two colours plus transparency, which is possible.

To start the code, I checked out a fresh copy of Brad Smith’s NES example project, then deleted things that I didn’t need.

When working with sprites on the NES, it is typical to store object attributes in RAM, then perform a DMA operation once per frame to copy this over to the Picture Processing Unit (PPU). For this project, I wanted to try setting up two different copies of the object attribute memory in RAM - one for each of the two rotations.

This should allow me to set any of the four frames by choosing between two possible colour palettes, and two possible DMA source addresses. This worked, and the first milestone was a spinning ball.

Switching between two DMA sources did not save much effort in the end, because I still needed quite a lot of code to set the X/Y positions of 64 sprites each frame. I set up some assembler macros to help with this.

Physics and sound

I have made one NES game before, and I was not happy with the physics. For this project, I wanted to do a bit better.

For the X position, I use a fixed speed, and simple collision detection with left/right boundaries. The animation frame is calculated from the X position, so the ball changes rotation depending on which direction it is moving, just like the Amiga demo.

The Y location of the ball is a simple loop, and follows the absolute value of a sine wave. I pre-computed this with some Python.

My last project also had no sound, so I read up on the NES APU, and added some noise when the ball changes direction.

Development setup

Since my last NES project, I have improved my development setup quite a bit. As always, I am using the ca65 assembler on Linux.

Previously, I was using a text editor. I have since moved to a custom 6502 assembly plugin on PyCharm, which allows me to quickly jump to definitions/usages. I developed this for my other 65C02 and 65C816 projects, which you can read about on this blog.

When I click run, I’ve set up the project to assemble a NES ROM, then launch with fceux, which has debugging features.

The debug features were previously only available on Windows builds of fceux, so when I made this demo, I was running the emulator via WINE. This has been added to the Linux builds as of v2.5.0, so I’ll most likely switch to that for my next project.

Wrap-up

I learn something new every time I write code for the NES, and it’s a lot of fun to make simple demos for these old systems. It’s also refreshing to create something standalone which I can explain through screenshots, instead of pages of assembly code.

For those who do want to read pages of assembly code, though, this project is up on GitHub at mike42/nes-ball-demo.

Let’s make a bootloader – Part 2

Thu, 22 Sep 2022 00:00:00 +0000

In my previous blog post, I wrote about loading and executing a 512 byte program from an SD card for my 65C816 computer project.

For part 2, I’ll look at what it takes to turn this into a bootloader, which can load a larger program from the SD card. I want the bootloader to control where to read data from, and how much data to load, which will allow me to change the program structure/size without needing to update the code on ROM.

Getting more data

The code in my previous post used software interrupts to print some text to the console, and my first challenge was to implement a similar routine for loading additional data from external storage.

The interface from the bootloader is quite simple: it sets a few registers to specify the source (a block number), destination memory address, and number of blocks to read, then triggers a software interrupt. The current implementation can read up to 64 KiB of data from the SD card each time it is called.

ROM_READ_DISK := $03
; other stuff ...
    ldx #$0000                      ; lower 2 bytes of destination address
    lda #$01                        ; block number
    ldy #$10                        ; number of blocks to read - 8 KiB
    cop ROM_READ_DISK               ; read kernel to RAM

Higher memory addresses

To load data in to higher banks (memory addresses above $ffff, I set the data bank register.

I needed to update a lot of my code to use absolute long (24-bit) addressing for I/O access, where it was previously using absolute (16-bit) addressing, for example:

sta $0c00

In my assembler, ca65, I already use the a: prefix to specify 16-bit addressing. I learned here that I can use the f: prefix for 24-bit addressing.

sta f:$0c00

Without doing this, the assembler chooses the smallest possible address size. This is fine in a 65C02 system, but I find it less helpful on the 65C816, where the meaning of a 16-bit address depends on the data bank register, and the meaning of an 8-bit address depends on the direct page register.

New bootloader

With the ROM code sorted out, I went on to write the new bootloader.

This new code sets the data bank address, then uses a software interrupt to load 8 KiB of data to address $010000

; boot.s: 512-byte bootloader. This utilizes services defined in ROM.
ROM_PRINT_STRING := $02
ROM_READ_DISK := $03

.segment "CODE"
    .a16
    .i16
    jmp code_start

code_start:
    ; load kernel
    ldx #loading_kernel             ; Print loading message (kernel)
    cop ROM_PRINT_STRING
    phb                             ; Save current data bank register
    .a8                             ; Set data bank register to 1
    php
    sep #%00100000
    lda #$01
    pha
    plb
    plp
    .a16
    ldx #$0000                      ; lower 2 bytes of destination address
    lda #$01                        ; block number
    ldy #$10                        ; number of blocks to read - 8 KiB
    cop ROM_READ_DISK               ; read kernel to RAM
    plb                             ; Restore previous data bank register

    ldx #boot                       ; Print boot message
    cop ROM_PRINT_STRING
    jml $010000

loading_kernel: .asciiz "Loading kernel\r\n"
boot: .asciiz "Booting\r\n"

.segment "SIGNATURE"
    wdm $42                         ; Ensure x86 systems don't recognise this as bootable.

The linker configuration for the bootloader is unchanged from part 1.

A placeholder kernel

The bootloader needed some code to load. I don’t have any real operating system to load, so I created the “Hello World” of kernels to work with in the meantime.

This is the simplest possible code I can come up with to test the bootloader. This assembles to 3 machine-language instructions, which occupy just 6 bytes. It is also is position-independent, and will work from any memory bank.

; kernel_tmp.s: A temporary placeholder kernel to test boot process.

ROM_PRINT_CHAR   := $00

.segment "CODE"
    .a16
    .i16
    lda #'z'
    cop ROM_PRINT_CHAR
    stp

The linker configuration for this, kernel_tmp.cfg, creates an 8 KiB binary.

MEMORY {
    ZP:     start = $00,    size = $0100, type = rw, file = "";
    RAM:    start = $0200,  size = $7e00, type = rw, file = "";
    PRG:    start = $e000,  size = $2000, type = ro, file = %O, fill = yes, fillval = $00;
}

SEGMENTS {
    ZEROPAGE: load = ZP,  type = zp;
    BSS:      load = RAM, type = bss;
    CODE:     load = PRG, type = ro,  start = $e000;
}

The commands I used to assemble and link the bootloader are:

ca65 --feature string_escapes --cpu 65816 --debug-info boot.s
ld65 -o boot.bin -C boot.cfg boot.o

The commands I used to assemble and link the placeholder kernel are:

ca65 --feature string_escapes --cpu 65816 --debug-info kernel_tmp.s
ld65 -o kernel_tmp.bin -C kernel_tmp.cfg kernel_tmp.o

I assembled the final disk image by concatenating these files together.

cat bootloader/boot.bin kernel_tmp/kernel_tmp.bin > disk.img

I used the GNOME disk utility to write the image to an SD card, which helpfully reports that I’m not using all of the available space.

Result

It took me many attempts to get the ROM code right, but this did work. This screen capture shows the test kernel source code on the left, which is being executed at the end of the boot-up process.

If I wanted to load the kernel from within a proper filesystem on the SD card (eg. FAT32), then I would need to update the starting block in the bootloader (hard-coded to 1 at the moment).

The limitations of this mechanism are that the kernel needs to be stored contiguously, be sector-aligned, and located within the first 64 MiB of the SD card.

Speed increase

This was the first time that I wrote code for this system and needed to wait for it to execute. The CPU was clocked at just 921.6 KHz. My SPI/SD card code was also quite generic, and was optimised for ease of debugging rather than execution speed.

I improved this from two angles. My 65C816 test board allows me to use a different clock speed for the CPU and UART chip, so I sped the CPU up to 4 MHz by dropping in an 8 MHz oscillator (it is halved to produce a two-phase clock). As I sped this up, I also needed to add more retries to the SD card initialisation code, since it does not respond immediately on power-up.

I also spent a lot of time hand-optimising the assembly language routine to receive a 512-byte block of data from the SD card. There is room to speed this up further, but it’s fast enough to not be a problem for now.

I had hoped to load an in-memory filesystem (ramdisk) alongside the test kernel, but I’ve deferred this until I can compress it, since reading from SD card is still quite slow.

A debugging detour

Writing bare-metal assembly with no debugger is very rewarding when it works, and very frustrating when there is something I’m not understanding correctly.

I ran into one debugging challenge here which is obvious in hindsight, but I almost couldn’t solve at the time.

This code is (I assure you) a bug-free way to print one hex character, assuming the code is loaded in bank 0. I was using this to hexdump the code I was loading into memory.

.a8                             ; Accumulator is 8-bit
.i16                            ; Index registers are 16-bit
lda #0                          ; A is 0 for example
and #$0f                        ; Take low-nibble only
tax                             ; Transfer to X
lda f:hex_chars, X              ; Load hex char for X
jsr uart_print_char             ; Print hex char for X

hex_chars: .byte '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'

The problem I had was that if the data bank register was 0, this would print ASCII character ‘0’ as expected. But if the data bank register was 1, it would print some binary character.

Nothing in this code should be affected by the data bank register, so I went through a methodical debugging process to try to list and check all of my assumptions. At one point, I even checked that the assembler was producing the correct opcode for this lda addressing mode (there are red herrings on the mailing lists about this).

I was able to narrow down the problem by writing different variations of the code which should all do the same thing, but used different opcodes to get there. This quickly revealed that it was the tax instruction which did not work as I thought, after finding that I could get the code working if I avoided it:

.a8                             ; Accumulator is 8-bit
.i16                            ; Index registers are 16-bit
ldx #0                          ; X is 0 for example
lda f:hex_chars, X              ; Load hex char for X
jsr uart_print_char             ; Print hex char for X

hex_chars: .byte '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'

My first faulty assumption was that if I set the accumulator to 0, then transferring the accumulator value to X would set the X register to 0 as well.

.a8                             ; Accumulator is 8-bit
.i16                            ; Index registers are 16-bit
lda #0                          ; Accumulator is 0
tax                             ; X is 0?

On this architecture, there are two bits of CPU status register which specify the size of the accumulator and index registers. When the accumulator is 8-bits, only the lower 8-bits of the 16-bit value are set by lda. I hadn’t realised that when the index registers are set to 16-bit, the tax instruction transfers all 16 bits of the accumulator to the X register (regardless of the accumulator size), which was causing surprising results.

Far away from the bug-free code I was focusing on, I had used a lazy method to set the data bank register, which involved setting the accumulator to $0101.

My second faulty assumption was that the data bank register was involved at all - in fact lda #$0101 would have been enough to break the later code.

.a16
lda #$0101                      ; Set data bank register to 1
pha
plb
plb

To fix this, when switching to an 8-bit accumulator, I now zero out the high 8-bits.

.a8
lda #0                          ; zero out the B accumulator
xba

Alternative boot method

I also added an option to boot from serial, based on my implementation of an XMODEM receive function for the 6502 processor.

This is a bit like PXE boot for retro computers. From a program such as minicom, you tell the ROM to boot from serial, then upload the kernel via an XMODEM send. It loads up to 64KiB to bank 1 (the same location as the bootloader on the SD card would place it), then executes it.

This is an important option for testing, since it’s a fast way to get some code running while I don’t have an operating system, and writing to an SD card from a modern computer is not a fast process. It may also be an important fallback if I need to troubleshoot SD card routines.

Wrap-up

Previously, I needed to use a ROM programmer to load code onto this computer. I can now use an SD card or serial connection, and have a stable interface for the bootstrapping code to access some minimal services provided by the ROM.

It is also the first time I’m running code outside of bank 0, breaking through the 64 KiB limit of the classic 6502. There is an entire megabyte of RAM on this test board.

Of course, this computer still does nothing useful, but at least it now controlled by an SD card rather than flashing a ROM. On the hardware side of the project, this will help me to convert the design from 5 V to 3.3 V. I’ll need to convert the ROM to a PLCC-packaged flash chip for that, which is not something I’ll want to be programming frequently.

As far as software goes, my plan is to work on some more interesting demo programs, so that I can start to build a library of open source 65C816 code to work with. The hardware design, software and emulator for this computer can be found on GitHub, at mike42/65816-computer.

Let's make a bootloader - Part 1

Thu, 01 Sep 2022 00:00:00 +0000

I’ve been working on a homebrew computer based on the 16-bit 65C816 CPU. All of my test programs so far have run from an EEPROM chip, which I need to remove and re-program each time I need to make a change.

Plenty of retro systems ran all of their programs from ROM, but I only want to use it for bootstrapping this computer. I’ve got 8 KiB of space for ROM-based programs in the memory map, which should be plenty to check the hardware and load software from disk.

In this two-part blog post, I’ll take a look at handing over control from the ROM to a program loaded from an SD card.

Technical background

I did some quick reading about the process for bootstrapping a PC during the 16-bit era. My homebrew computer is a completely different architecture to an IBM-compatible PC, but I’m planning to follow a few of the conventions from this ecosystem, since I’ll have some similar challenges.

The best resources on this topic are aimed at bootloader developers, such as this wikibooks page.

For a disk to be considered bootable, the first 512-byte sector needs to end with hex 0xAA55 (little-endian) . This is 01010101 10101010 in binary (a great test pattern!). My system is not x68-based, so I’ll store a different value there.

If a disk is bootable, then the BIOS will transfer the 512 byte boot sector to $7C00 and jump to it. The only assumption that the BIOS seems to make about the bootloader structure is that it starts with executable code. I’ll do the same on my system.

It’s worth noting that the first sector may also contain data structures for a filesystem or partitioning scheme, and it’s up to the bootloader code to work around that. For now, my SD card will contain only a bootloader, which does simplify things a bit.

Most bootloaders will then make a series of BIOS calls via software interrupts, which enables them produce text output or load additional data from disk. This is where I’ll have the biggest challenge, since my ROM has no stable interface for a bootloader to call.

Re-visiting SD card handling

My first task was to load the bootloader itself from SD card, storing the first 512 bytes from the disk to RAM, at address $7C00 onwards. This should be straightforward, since I have working 6502 assembly routines for reading from an SD card, and I’ve added a port for an SD card module to my 65C816 test board.

I came up with a routine which prints the contents of the boot sector, then prompts for whether to execute it. My ROM code is not checking the signature at this stage, and is not aware that the boot sector in this screen capture contains x86 machine code within a FAT32 boot sector, but this is a good start.

It took quite a few revisions to get this working, since my old 65C02 code for reading from SD produced strange output on this system. On my 65C816 test board, it showed almost the right values, but it was jumbled up, and mixed with SPI fill bytes ($FF). The below screen capture shows a diff between the expected and actual output of the ROM.

After a long process to rule out other programming and hardware errors, I finally noticed that I was writing the data starting from address $0104, which was never going to work. The default stack pointer on this CPU is $01ff and grows down, so writing 512 bytes to $0104 would always corrupt the stack after a few hundred bytes.

At this stage I was using the assembler to statically allocate a 512 byte space for IO. It appeared in code like this:

.segment "BSS"
io_block_id:              .res 4
io_buffer:                .res 512

The error was in the linker configuration, which I updated to start assigning RAM addresses from $0200 onwards.

 MEMORY {
     ZP:     start = $00,    size = $0100, type = rw, file = "";
-    RAM:    start = $0100,  size = $7e00, type = rw, file = "";
+    RAM:    start = $0200,  size = $7d00, type = rw, file = "";
     PRG:    start = $e000,  size = $2000, type = ro, file = %O, fill = yes, fillval = $00;
 }

The full SD card handling code is too long to post in this blog, but now allows any 512-byte segment from the first 32 MB of the SD card (identified by a 16-bit segment ID) to be loaded into an arbitrary memory address.

Making an API

My next challenge was to define an API for the bootloader to call into the ROM to perform I/O.

I considered using a jump table, but decided to use the cop instruction instead. This triggers a software interrupt on the 65C816, and has parallels to how the int instruction is used to trigger BIOS routines on x86 systems.

I defined a quick API for four basic routines, passing data via registers.

print char
read char
print string
load data from storage

The caller would need to set some registers, then call cop from assembly language. Any return data would also be passed via registers.

The cop instruction takes a one-byte operand, which in this case specifies the ID of the function to call.

cop $00

To prove that the interface would work, I implemented just the routine for printing strings.

; interrupt.s: Handling of software interrupts, the interface into the ROM for
; software (eg. bootloaders)
;
; Usage: Set registers and use 'cop' to trigger software interrupt.
; Eg:
;   ldx #'a'
;   cop ROM_PRINT_CHAR
; CPU should be in native mode with all registers 16-bit.

.import uart_printz, uart_print_char
.export cop_handler
.export ROM_PRINT_CHAR, ROM_READ_CHAR, ROM_PRINT_STRING, ROM_READ_DISK

; Routines available in ROM via software interrupts.
; Print one ASCII char.
;   A is char to print
ROM_PRINT_CHAR   := $00

; Read one ASCII char.
;   Returns typed character in A register
ROM_READ_CHAR    := $01

; Print a null-terminated ASCII string.
;   X is address of string, use data bank register for addresses outside bank 0.
ROM_PRINT_STRING := $02

; Read data from disk to RAM in 512 byte blocks.
;   X is address to write to, use data bank register for addresses outside bank 0.
;   A is low 2 bytes of block number
;   Y is number of blocks to read
ROM_READ_DISK    := $03

.segment "CODE"
; table of routines
cop_routines:
.word rom_print_char_handler
.word rom_read_char_hanlder
.word rom_print_string_handler
.word rom_read_disk_handler

cop_handler:
    .a16                            ; use 16-bit accumulator and index registers
    .i16
    rep #%00110000
    ; Save caller context to stack
    pha                             ; Push A, X, Y
    phx
    phy
    phb                             ; Push data bank, direct register
    phd
    ; Set up stack frame for COP handler
    tsc                             ; WIP set direct register to equal stack pointer
    sec
    sbc #cop_handler_local_vars_size
    tcs
    phd
    tcd
caller_k := 15
caller_ret := 13
caller_p := 12
caller_a := 10
caller_x := 8
caller_y := 6
caller_b := 5
caller_d := 3
cop_call_addr := 0
    ; set up 24 bit pointer to COP instruction
    ldx <frame_base+caller_ret
    dex
    dex
    stx <frame_base+cop_call_addr
    .a8                             ; Use 8-bit accumulator
    sep #%00100000
    lda <frame_base+caller_k
    sta <frame_base+cop_call_addr+2
    .a16                            ; Revert to 16-bit accumulator
    rep #%00100000

    ; load COP instruction which triggered this interrupt to figure out routine to run
    lda [<frame_base+cop_call_addr]
    xba                             ; interested only in second byte
    and #%00000011                  ; mask down to final two bits (there are only 4 valid functions at the moment)
    asl                             ; multiply by 2 to index into table of routines
    tax
    jsr (cop_routines, X)

    ; Remove stack frame for COP handler
    pld
    tsc
    clc
    adc #cop_handler_local_vars_size
    tcs

    ; Restore caller context from stack, reverse order
    pld                             ; Pull direct register, data bank
    plb
    ply                             ; Pull Y, X, A
    plx
    pla
    rti

cop_handler_local_vars_size := 3
frame_base := 1

rom_print_char_handler:
    ldx #aa
    jsr uart_printz
    rts

rom_read_char_hanlder:
    ldx #bb
    jsr uart_printz
    rts

rom_print_string_handler:
    ; Print string from X register
    ldx <frame_base+caller_x
    jsr uart_printz
    rts

rom_read_disk_handler:
    ldx #cc
    jsr uart_printz
    rts

aa: .asciiz "Not implemented A\r\n"
bb: .asciiz "Not implemented B\r\n"
cc: .asciiz "Not implemented C\r\n"

This snippet is quite dense, and uses several features which are new to the 65C816, not just the cop instruction.

I’m relocating the direct page to use as a stack frame, which is an idea I got from reading the output of the WDC 65C816 C compiler. Pointers are much easier to work with on the direct page.

This is the first snippet I’ve shared which uses a 24-bit pointer, via “direct page, indirect long” addressing. The pointer is used to load the instruction which triggered the interrupt, so that the code can figure out which function to call.

lda [<frame_base+cop_call_addr]

This snippet is also the first time I’ve used the jump to subroutine instruction (jsr) with the “absolute indirect, indexed with X” address mode. On the 65C02, I could only use this addressing mode on the jmp instruction. The only example of that on this blog is also an interrupt handling example.

jsr (cop_routines, X)

The “Hello World” of bootloaders

My next goal was to load a small program from disk, and show that it can call routines from the ROM. For now it is just a program on the boot sector on an SD card, which demonstrates that the new software interrupt API works.

This assembly file boot.s prints out two strings, so that I can be sure that the ROM is returning control back to the bootloader after a software interrupt completes.

ROM_PRINT_STRING := $02

.segment "CODE"
    .a16
    .i16
    ldx #test_string_1
    cop ROM_PRINT_STRING
    ldx #test_string_2
    cop ROM_PRINT_STRING
    stp

test_string_1: .asciiz "Test 1\r\n"
test_string_2: .asciiz "Test 2\r\n"

.segment "SIGNATURE"
    wdm $42                         ; Ensure x86 systems don't recognise this as bootable.

The linker configuration which goes with this is boot.cfg:

MEMORY {
    ZP:     start = $00,    size = $0100, type = rw, file = "";
    RAM:    start = $7e00,  size = $0200, type = rw, file = "";
    PRG:    start = $7c00,  size = $0200, type = rw, file = %O, fill = yes, fillval = $00;
}

SEGMENTS {
    ZEROPAGE:   load = ZP,  type = zp;
    BSS:        load = RAM, type = bss;
    CODE:       load = PRG, type = rw,  start = $7c00;
    SIGNATURE:  load = PRG, type = rw,  start = $7dfe;
}

The commands to assemble and link this are:

ca65 --feature string_escapes --cpu 65816 boot.s
ld65 -o boot.bin -C boot.cfg boot.o

This produces a 512 byte file, which I wrote to SD card.

This is the first time this computer is running code from RAM, which is an important milestone for this project.

Editor improvements

I needed to do some work on my 6502 assembly plugin for IntelliJ during this process, since it didn’t understand the square brackets used for the long address modes.

    lda [<frame_base+cop_call_addr]

While I was fixing this, I also implemented an auto-format feature. This saves me the manual effort of lining up all the comments in a column, as is typical in assembly code.

Lastly, I added support for jumping to unnamed labels, which are a ca65-specific feature.

Next steps

In the second half of this blog post, I’ll get the bootloader to load a larger program from the SD card. I’m hoping to allow the bootloader to control how much code to load, and where to load it from.

Building a simple power supply module

Thu, 18 Aug 2022 00:00:00 +0000

I recently put together a tiny module to power my electronics projects. This is a 3cm x 4cm circuit board, and can be assembled to deliver either a fixed 3.3V or 5V output.

Background: Breadboard power supplies

I start a lot of my projects on a breadboard, and previously used a breadboard power supply. The voltage regulators on these are not particularly resilient, and I’ve damaged two of them by wiring something up incorrectly and drawing too much current.

They do not exactly fail-safe, and now pass through the input voltage to whatever I’m working on. This is 12 V in my case, which is enough to fry most of the components I work with.

A bit of searching shows that this is a common issue. The first power supply which did this was a Bud Industries BBP-32701, which uses an LM1117 regulator.

The second one was a DFRobot DFR0140, which uses the AMS1117 regulator.

For the past year or so, I’ve used an L7805 regulator connected to a breadboard instead. This is not particularly efficient, but it works well, and I’ve caused enough short circuits to know that the over-current protection works.

Converting to a PCB

My only goal was to take the exact parts I’ve already got, and make them into a more compact module for permanent use.

I used KiCad to capture the schematic and design the board. The minimum order is generally 5 circuit boards, so I planned to assemble some with an L7805 (5 V), and others with a LD33V (3.3 V) regulator. These have different pin-outs, so I added a series of solder jumpers so that I can use the same board for either regulator.

Layout for this PCB was straightforward. It’s a two-layer board, and I added a ground pour on the bottom layer, and the output voltage on most of the top layer.

I also decided to switch off thermal relief on the copper pours, to see if a direct connection to a ground plane really makes soldering more difficult (it does! lesson learned).

I tried to make this board quite dense, but didn’t go completely overboard: It’s useful to have holes to install standoffs, plus a small area to write on, so that I can distinguish the 5 V and 3.3 V modules later.

Wrap-up

I’m still a little surprised that custom PCB’s are so accessible for hobby use. The unit price of these boards was just 1.36 AUD before shipping, which is cost-competitive with blank prototyping board.

It’s easy to get assembled DC-DC converters, so I don’t suggest building any of these modules yourself. However, as with many of the things I blog about, I’ve uploaded the project files to GitHub under an open source license. The KiCad project, Gerber files and parts list for this project can be found at mike42/simple-power-supply.

Let's implement power-on self test (POST)

Thu, 16 Jun 2022 00:00:00 +0000

I recently implemented simple Power-on self test (POST) routine for my 65C816 test board, so that it can stop and indicate a hardware failure before attempting to run any normal code.

This was an interesting adventure for two main reasons:

This code needs to work with no RAM present in the computer.
I wanted to try re-purposing the emulation status (E) output on the 65C816 CPU to blink an LED.

Background

Even modern computers will stop and provide a blinking light or series of beeps if you don’t install RAM or a video card. This is implemented in the BIOS or UEFI.

I got the idea to use the E or MX CPU outputs for this purpose from this thread on the 6502.org forums. This method would allow me to blink a light with just a CPU, clock input, and ROM working.

My main goal is to perform a quick test that each device is present, so that start-up fails in a predictable way if I’ve connected something incorrectly. This is much simpler than the POST routine from a real BIOS, because I’m not doing device detection, and I’m not testing every byte of memory.

Boot-up process

On my test board, I’ve connected an LED directly to the emulation status (E) output on the 65C816 CPU. The CPU starts in emulation mode (E is high). However I have noticed that on power-up, the value of E appears to be random until /RES goes high. If I were wiring this up again, I would also prevent the LED from lighting up while the CPU is in reset:

The first thing the CPU does is read an address from ROM, called the reset vector, which tells it where start executing code.

In my case, the first two instructions set the CPU to native mode, which are clc (clear carry) and xce (exchange carry with emulation).

.segment "CODE"
reset:
    .a8
    .i8
    clc                            ; switch to native mode
    xce
    jmp post_check_loram

By default accumulator and index registers are 8-bit, the .a8 and .i8 directives simply tell the assembler ca65 that this is the case.

Next, the code will jmp to the start of the POST process.

Checking low RAM

The first part of the POST procedure checks if the lower part of RAM is available, by writing values to two address and checking that the same values can be read back.

Note that a:$00 causes the assembler to interpret $00 as an absolute address. This will otherwise be interpreted as direct-page address, which is not what’s intended here.

post_check_loram:
    ldx #%01010101                 ; Power-on self test (POST) - do we have low RAM?
    ldy #%10101010
    stx a:$00                      ; store known values at two addresses
    sty a:$01
    ldx a:$00                      ; read back the values - unlikely to be correct if RAM not present
    ldy a:$01
    cpx #%01010101
    bne post_fail_loram
    cpy #%10101010
    bne post_fail_loram
    jmp post_check_hiram

If this fails, then the boot process stops, and the emulation LED blinks in a distinctive pattern (two blinks) forever.

post_fail_loram:                   ; blink emulation mode output with two pulses
    pause 8
    blink
    blink
    jmp post_fail_loram            ; repeat indefinitely

Macros: pause and blink

It’s a mini-challenge to write code to blink an LED in a distinctive pattern without assuming that RAM works. This means no stack operations (eg. jsr and rts instructions), and that I need to store anything I need in 3 bytes: the A, X, Y registers. A triple-nested loop is the best I can come up with.

I wrote a pause macro, which runs a time-wasting loop for the requested duration - approximately a multiple of 100ms at this clock speed. Every time this macro is used, the len value is substituted in, and this code is included in the source file. This example also uses unnamed labels, which is a ca65 feature for writing messy code.

.macro pause len                   ; time-wasting loop as macro
    lda #len
:
    ldx #64
:
    ldy #255
:
    dey
    cpy #0
    bne :-
    dex
    cpx #0
    bne :--
    dec
    cmp #0
    bne :---
.endmacro

The second macro I wrote is blink, which briefly lights up the LED attached to the E output by toggling emulation mode. I’m using the pause macro from both native mode and emulation mode in this snippet, so I can only treat A, X and Y as 8-bit registers.

.macro blink
    sec                            ; switch to emulation mode
    xce
    pause 1
    clc                            ; switch to native mode
    xce
    pause 2
    sec                            ; switch to emulation mode
.endmacro

Checking high RAM

There is also a second RAM chip, and this process is repeated with some differences. For one, I can now use the stack, which is how I set the data bank byte in this snippet.

Here a:$01 is important, because with direct page addressing, $01 means $000001 at this point in the code, where I want to test that I can write to the memory address $080001.

post_check_hiram:
    ldx #%10101010                 ; Power-on self test (POST) - do we have high RAM?
    ldy #%01010101
    lda #$08                       ; data bank to high ram
    pha
    plb
    stx a:$00                      ; store known values at two addresses
    sty a:$01
    ldx a:$00                      ; read back the values - unlikely to be correct if RAM not present
    ldy a:$01
    cpx #%10101010
    bne post_fail_hiram
    cpy #%01010101
    bne post_fail_hiram
    lda #$00                       ; reset data bank to boot-up value
    pha
    plb
    jmp post_check_via

The failure here is similar, but the LED will blink 3 times instead of 2.

post_fail_hiram:                   ; blink emulation mode output with three pulses. we could use RAM here?
    pause 8
    blink
    blink
    blink
    jmp post_fail_hiram            ; repeat indefinitely

To make sure that I was writing to different chips, I installed the RAM chips one at a time, first observing the expected failures, and then observing that the code continued past this point with the chip installed.

I also checked with an oscilloscope that both RAM chips are now being accessed during start-up. Now that I’ve got some confidence that the computer now requires both chips to start, I can skip a few debugging steps if I’ve got code that isn’t working later.

Checking the Versatile Interface Adapter (VIA)

The third chip I wanted to add to the POST process is the 65C22 VIA. I kept this check simple, because one read to check for a start-up default is sufficient to test for device presence.

VIA_IER = $c00e
post_check_via:                    ; Power-on self test (POST) - do we have a 65C22 Versatile Interface Adapter (VIA)?
    lda a:VIA_IER
    cmp #%10000000                 ; start-up state, interrupts enabled overall (IFR7) but all interrupt sources (IFR0-6) disabled.
    bne post_fail_via
    jmp post_ok

This stops and blinks 4 times if it fails. I recorded the GIF at the top of this blog post by removing the component which generates a chip-select for the VIA, which causes this code to trigger on boot.

post_fail_via:
    pause 8
    blink
    blink
    blink
    blink
    jmp post_fail_via

Beep for good measure

At the end of the POST process, I put in some code to generate a short beep.

This uses the fact that the 65C22 can toggle the PB7 output each time a certain number of clock-cycles pass. I’ve connected a piezo buzzer to that output, which I’m using as a PC speaker. The 65C22 is serving the role of a programmable interrupt timer from the PC world.

VIA_DDRB = $c002
VIA_T1C_L = $c004
VIA_T1C_H = $c005
VIA_ACR = $c00b
BEEP_FREQ_DIVIDER = 461            ; 1KHz, formula is CPU clock / (desired frequency * 2), or 921600 / (1000 * 2) ~= 461
post_ok:                           ; all good, emit celebratory beep, approx 1KHz for 1/10th second
    ; Start beep
    lda #%10000000                 ; VIA PIN PB7 only
    sta VIA_DDRB
    lda #%11000000                 ; set ACR. first two bits = 11 is continuous square wave output on PB7
    sta VIA_ACR
    lda #<BEEP_FREQ_DIVIDER        ; set T1 low-order counter
    sta VIA_T1C_L
    lda #>BEEP_FREQ_DIVIDER        ; set T1 high-order counter
    sta VIA_T1C_H
    ; wait approx 0.1 seconds
    pause 1
    ; Stop beep
    lda #%11000000                 ; set ACR. returns to a one-shot mode
    sta VIA_ACR
    stz VIA_T1C_L                  ; zero the counters
    stz VIA_T1C_H
    ; POST is now done
    jmp post_done

The post_done label points to the start of the old ROM code, which is currently just a “hello world” program.

Next steps

I’m now able to lock in some of my assumptions about should be available in software, so that I can write more complex programs without second-guessing the hardware.

Once the boot ROM is interacting with more hardware, I may add additional checks. I will probably need to split this into different sections, and make use of jsr/rts once RAM has been tested, because the macros are currently generating a huge amount of machine code. I have 8KiB of ROM in the memory map for this computer, and the code on this page takes up around 1.1KiB.

Building a 65C816 test board

Thu, 02 Jun 2022 00:00:00 +0000

In the last few months, I’ve been learning about the 65C816 processor, and trying to build a working computer which uses it. My latest breadboard-based prototype was not reliable, and I decided to convert it to a PCB to hopefully eliminate the problem, or to at least identify it.

Quick goals

I was aiming to make a debug-friendly 4-layer PCB, the size of two standard breadboards. This will be my first time designing a 4-layer board, and also my first time using KiCad 6 to create a PCB.

I didn’t have a working prototype, so I built in test points for connecting an oscilloscope or logic analyser to troubleshoot. In case of errors, I made it possible to leave some components unpopulated, and instead drive signals externally.

I also left some extra footprints which I might use for future improvements, or can fall back to if some of my ideas don’t work out.

The clock

Some components require the inverse of the CPU clock, and there was previously a small delay from inverting the signal. I don’t think this is a problem on its own, but I introduced a D-type flip-flop to create a proper two-phase clock.

This does halve the CPU clock speed, so I added jumpers to allow an alternative clock source to be selected for the CPU, where previously the UART and CPU clocks needed to be the same. Note that there is an error in the wiring here, which I only discovered later.

The oscillator I’m using on the breadboard prototype is a DIP-8 package. A larger variety of oscillators are available as a 5x7mm QFN package, where the four corner pins have the same function as the DIP-8 version. I added an alternative footprint, which fits without taking up any extra board space. I expect that I’ll only use this if I attempt to run the board at 3.3V later.

Some 6-pin QFN oscillators provide an inverted clock output on one of the pins, and it would be an idea to use one of those in a future design to reduce the component count.

New reset controller

Several components on the board require a reset input. These are mostly active-low inputs, but one component has an active-high reset. I discovered the tiny MIC2775 supervisory IC, which provides both.

I haven’t used this part before, so in case of problems, I also made it possible to remove this part and use a DS1813, which generates an active-low reset on my prototype.

These parts work by sensing voltage, and drop-in alternatives exist for either part if I move to 3.3V.

ROM

I am currently using a parallel EEPROM to store code for my prototype, and will add this to the PCB for this test board. As with the the clock and reset controller, I’m working with 5V components today, but considering what I would need to change to run the whole board at 3.3V.

The few EEPROMs which are available at 3.3V use a different package, and are quite slow (200ns or worse). Parallel flash chips seem to be the most promising alternative, since they have a similar interface, although I haven’t confirmed that I can program them yet. I added a footprint for the SST39LF010, which also has a 5V variant with the same pin-out.

Other footprints

I also added footprints so that I could connect the SD card module which I used on my 6502 computer, a piezo buzzer, and some electrolytic capacitors if needed.

Design lessons learned

Although my first attempt at making a PCB was a success, I learned a few things which I’m using here.

For that previous project, I found that machined-pin sockets are not very tolerant of used chips with bent legs, so I’ve used cheaper stamped pin sockets for this test board. I’ve also left spacing at the ends of each socketed IC, so that they can be pulled out more easily.

I made an effort to keep unrelated traces away from points which I need to solder, because I had some problems with this on my last board.

I also avoided creating one large expansion header, and instead exposed signals in places which are easy to route. For example, the I/O device select signals are exposed on a pin header right beside the chip which generates them.

PCB layout

The standard process is to make a schematic, then place components, then route traces.

For this test board, I instead chose the physical dimensions of the board first, and added components incrementally (particularly test points, expansion headers and alternative footprints) until I was making good use of the available space. KiCad has an item in the “Tools” menu to “Update PCB from Schematic”, which I used extensively.

Just to see how it would look, I also added the letters “65C816” to the back copper layer, which is visible in the top-right here.

I put a lot of time into labeling different parts of the board to help with debug/assembly, and used the 3D view to check that it wasn’t too crowded.

Before sending the files to a manufacturer, I printed it at 1:1 scale for a reality check.

Among other things, this confirmed that the ROM sockets had enough clearance from other components.

It also confirmed that either DIP-8 or QFN-packaged oscillators would fit.

Assembly

I ordered the boards from a manufacturer which I’ve used before. They are large-ish, lead-free 4-layer boards, but I’m not optimising for cost. I’ve left a lot of options in the board, so I’m hoping to make use of several of these PCBs in different configurations, depending on which direction this project goes.

I assembled the board incrementally, starting with the power LED. Most of the passive components on this board are 0603 (imperial) size surface-mount parts, and I’ve used footprints with long pads for hand soldering.

The first problem I found was with the reset button - I had switched to using a footprint with the correct pin spacing, but had assigned the pins incorrectly, so I needed to cut/bend some legs and install it rotated 90 degrees.

I next found a problem with the clock, where I had wired up the flip-flop incorrectly in the schematic - the /Q output should go to D. I could recover from this by cutting a trace and running a short wire. Of course the board is mirrored when flipped upside-down, so I cut the wrong trace first and needed to repair that too.

I added enough components to get the CPU to run NOP instructions from ROM, then built up to a running some test programs which I’ve blogged about here. The final mistake I discovered was a mix-up with some of the lines used for selecting I/O devices, which means that devices are not mapped to their intended addresses. I can work around this in software.

This is already an improvement over the breadboard prototype, because I can quickly swap the ROM chip without accidentally disconnecting anything.

The board looked fairly complete at this point, and the only major component missing was the UART chip. This was the part which did not work reliably in my prototype, so I was prepared to do some debugging here. Note that I’ve got orange test points all over the board to connect important signals to an oscilloscope, with a few black test points for GND connections. All of the 74-series chips are in the 74AC logic family, and I sourced 74HC versions as well in case I needed to switch any to see the difference.

However, I was able to run more or less the same test program I used before, and it now works reliably. This is captured through a Cypress FX2-based logic analyser using Sigrok.

It’s great that this works, but I don’t know for sure why this was so unreliable on my previous prototype. Several possible causes were eliminated through this process, since I used a new UART chip and freshly programmed address decode PLD, and eliminated a possible timing issue. On a PCB, I’m also able to make better electrical connections, and add a ground plane, which is an immediate advantage over breadboards.

The design as it stands

Now that I am back to having a working prototype, I’ll take this chance to post some updated schematics.

Just a note of caution: this is snapshot of a work-in-progress learning project. I’m absolutely aware that there are errors in here, and that the layout is quite messy. Still, I hope that this is useful to anybody else who is attempting to use this relatively obscure CPU.

The design no longer fits on one page, so I’ve split it into 3 sheets.

CPU

The CPU sheet contains the circuitry for de-multiplexing the data bus and bank address byte. All power, reset, and clock components are in here as well, along with pin headers for the address & data bus, and all those test points.

There are a lot of “just in case” components as well, such as pull-up resistors on the data bus, which I have not fitted.

Memory

The memory section is quite straightforward. I’ve got a PLD generating chip-selects for ROM, RAM0, or RAM1, with some extra components to add some flexibility.

I/O

In case I/O is selected, there are three possible choices: the 65C22 VIA, or one of the two UART interfaces. Most of the other components on this sheet are optional footprints or external ports. Note that the clock going into the UART is mis-labelled on this sheet.

Next steps

I’m going to spend some time using this board as a development platform for low-level software which targets the 65C816 CPU. I’ll most likely also use the emulator which I put together a few weeks ago to speed up development, since I’ve been able to confirm that the code I’m writing works on real hardware.

The basic functionality is now a lot more stable than what I had before, so this test board will allow me to prototype some different hardware options once I’ve got some simple text-based programs up and running.

The hardware design, software and emulator for this computer can be found in the GitHub repository for this project. I’m updating the repository as I make progress with this project, and the version used for this blog post is here.

65C816 computer - second prototype

Sat, 28 May 2022 00:00:00 +0000

Back in February, I blogged about my 65816 computer prototype on breadboards. I recently spent some time re-building this in an attempt to add some improvements.

The new prototype didn’t work particularly well, so I’ve left out quite a lot of detail from this update.

Power delivery

I soldered stranded wire to pin-headers to deliver a reliable ground and +5V power connection to each breadboard. This corrected some problems from my previous prototype, which used a long chain of unreliable connections.

Adding more RAM

I added a second 512 KiB RAM chip, and extended the address decoding scheme to provide a chip-select signal for it. The computer now uses the planned 20-bit address bus, though the wiring around these chips became very dense.

Adding a UART chip

I previously tested an NXP SC16C752 UART, and added it to this computer to provide text I/O. This is a small surface-mount chip, which I am connecting via a break-out board.

My previous prototype only supported one I/O device, so I also extended the address decoding scheme with a 74AC138 decoder.

Problems

I could get some simple test programs running, but text I/O did not work reliably at all. With my limited debugging tools (just a logic analyser), I was only able to conclude that data being written by the CPU was different to what was being received by the UART.

This seemed like it could be a timing or signal integrity issue, but I couldn’t be certain from looking at digital output with a low sample-rate. This image shows some control signals from when I was troubleshooting this.

Next steps

After hitting a dead-end, I made a list of possible causes, and decided to re-build this prototype on a debug-friendly PCB, and to source an oscilloscope for troubleshooting.

Building an emulator for my 65C816 computer

Thu, 07 Apr 2022 00:00:00 +0000

I’ve been working on adding text-based input and output to my 65C816 computer prototype, but it’s not yet working reliably.

To unblock software development, I decided to put together an emulator, so that I can test my code while the hardware is out of action.

Choosing a base project

There were three main options I considered for running 65C816 programs on a modern computer,

Write a new emulator from scratch
Extract the useful parts of an open source Super Nintendo or Apple IIgs emulator and adapt them
Find a 65C816 CPU emulator, and extend it.

I found a good candidate for the third option in Lib65816 by Francesco Rigoni, which is a C++ library for emulating a 65C816 CPU. The code is GPLv3 licensed, and I could see where I would need to make changes to match the design of my computer.

Process

After reading the code, I could see some places where I would need to modify the library, so I copied it in to a new folder, together with its logging dependency and sample program, and refactored it into one project.

I started by extending RAM up to 16 banks (1 megabyte), and mapping in a ROM device which serves bytes from a file. I also updated the handling of interrupt vectors, so that the CPU would retrieve them from the system bus.

Adding 65C22 support

My first test program was an example of preemptive multi-tasking from my previous blog post. At this point it ran, but did not switch between tasks, since that requires extra hardware support.

I implemented a minimal support for the 65C22 VIA used in my computer design, just enough to log changes to the outputs, and to fire interrupts from a timer.

The Lib65816 library does not support NMI interrupts, which are used in this test program. I added edge detection, and connected the interrupt output of the 65C22 to the CPU NMI input in code.

This screen capture shows the test program during switching between two tasks, with a different VIA port being accessed before and after the switch.

This gave me some confidence that the CPU emulation was good enough to develop on, so I moved on to the next test program.

Adding serial support

The second test program is also from an earlier blog post, and tests printing and reading characters from an NXP SC16C752 UART.

I added a very minimal emulation of the UART chip I’m using, suppressed all of the logging, and converted output to use the curses library. This last step enabled character-by-character I/O, instead of the default line-buffered I/O.

This screen capture shows the program running under emulation. It simply prints “Hello world”, then echoes back the next 3 characters that the user types.

Wrap-up

I didn’t plan to write an emulator for my custom computer, but I am currently debugging some tricky hardware problems, and this detour will allow me to continue to make progress in other areas while I try to find a solution.

I plan to keep the emulator up to date with hardware changes, since it will make it possible to demonstrate the system without any hardware access. It’s also easier to debug issues now that I can choose two different execution environments.

The hardware design, software and emulator for this computer are online at mike42/65816-computer. I would like to again acknowledge Francesco Rigoni’s work on Lib65816. I’m very grateful that he chose to release this under the GPL, since it allows me to re-use the code for my project.

Let's implement preemptive multitasking

Thu, 10 Mar 2022 00:00:00 +0000

I am currently working on building a computer based on the 65C816 processor. This processor has a 16-bit stack pointer, which should make it far more capable than the earlier 65C02, at least in theory.

I wanted to check my understanding of how the stack works on this processor, so I tried to build the simplest possible implementation of preemptive multitasking in 65C816 assembly.

What I’m working with

I am working with a breadboard computer. It has no operating system, and no ability to load programs or start/stop processes.

To get code running, I am assembling it on a modern computer, then writing the resulting binary to an EEPROM chip.

To see the result of the code, I am using a logic analyser to watch some output pins on the 65C22 VIA. The 65C22 chip also has a timer, so I connected its interrupt output to the NMI input of the CPU for this test.

Test programs

I wrote two simple programs which infinitely loop. These programs use ca65 assembler syntax.

The 65C22 has two 8-bit ports, so I made each test program write a recognisable pattern to its own output port, so that it’s easy to see which one is running at any given time, and check that it is not disrupted by the task switching.

The first program writes to Port A of the 65C22, and alternates the value between 0000 0000, and 0000 0011.

.segment "CODE"
; Task 1
task_1_main:
  .a8                             ; use 8-bit accumulator and index registers
  .i8
  sep #%00110000
@repeat_1:
  lda #%00000000                  ; alternate between two values
  sta PORTA
  lda #%00000011
  sta PORTA
  jmp @repeat_1                   ; repeat forever

The second program writes to Port B of the 65C22. This uses a ror instruction to produce a pattern across all digits of the output port.

.segment "CODE"
; Task 2
task_2_main:
  .a8                             ; use 8-bit accumulator and index registers
  .i8
  sep #%00110000
  clc
  lda #%01010101                  ; grab a start value
@repeat_2:
  ror                             ; rotate right
  sta PORTB
  jmp @repeat_2                   ; repeat forever

Context switching

I’m implementing round-robin scheduling between two processes.

My basic plan was to use a regular interrupt routine to save context, switch the stack pointer to the other process, restore context for the next process, then return from the interrupt.

.segment "CODE"
nmi:
  .a16
  .i16
  rep #%00110000
  ; Save task context to stack
  pha                             ; Push A, X, Y
  phx
  phy
  phb                             ; Push data bank, direct register
  phd

  ; swap stack pointer so that we will restore the other task
  tsc                             ; transfer current stack pointer to memory
  sta temp
  lda next_task_sp                ; load next stack pointer from memory
  tcs
  lda temp                        ; previous task is up next
  sta next_task_sp

  ; Clear interrut
  .a8
  .i8
  sep #%00110000                  ; save X only (assumes it is 8 bits..)
  ldx T1C_L                       ; Clear the interrupt, side-effect of reading

  .a16
  .i16
  rep #%00110000
  ; Restore process context from stack, reverse order
  pld                             ; Pull direct register, data bank
  plb
  ply                             ; Pull Y, X, A
  plx
  pla
  rti

Setup process

The above switch works while the two processes are up and running, that takes some setting up. Firstly, the two variables need to be referenced somewhere.

.segment "BSS"
next_task_sp: .res 2              ; Stack pointer of whichever task is not currently running
temp: .res 2

The boot-up routine then sets everything up. It’s broken up a bit here. First is the CPU initialisation, since the 65C816 will boot to emulation mode.

.segment "CODE"
reset:
  clc                             ; switch to native mode
  xce

The second step is to start a task in the background. This involves pushing appropriate values to the stack, so that when we context switch, “Task 2” will start executing from the top.

  ; Save context as if we are at task_2_main, so we can switch to it later.
  .a16                            ; use 16-bit accumulator and index registers
  .i16
  rep #%00110000
  lda #$3000                      ; set up stack, direct page at $3000
  tcs
  tcd
  ; emulate what is pushed to the stack before NMI is called: program bank, program counter, processor status register
  phk                             ; program bank register, same as current code, will be 0 here.
  pea task_2_main                 ; 16 bit program counter for the start of this task
  php                             ; processor status register
  ; match what we push to the stack in the nmi routine
  lda #0
  tax
  tay
  pha                             ; Push A, X, Y
  phx
  phy
  phb                             ; Push data bank, direct register
  phd
  tsc                             ; save stack pointer to next_task_sp
  sta next_task_sp

  lda #$2000                      ; set up stack, direct page at $2000 for task_1_main
  tcs
  tcd

The nex step is to set up a timer, so that the interrupt routine will fire regularly. This causes interrupts to occur ~28 times per second.

  ; Set up the interrupt timer
  .a8                             ; use 8-bit accumulator and index registers
  .i8
  sep #%00110000
  lda #%11111111                  ; set all VIA pins to output
  sta DDRA
  sta DDRB
  ; set up timer 1
  lda #%01000000                  ; set ACR. first two bits = 01 is continuous interrupts for T1
  sta ACR
  lda #%11000000                  ; enable VIA interrupt for T1
  sta IER
  ; set up a timer at ~65535 clock pulses.
  lda #$ff                        ; set T1 low-order counter
  sta T1C_L
  lda #$ff                        ; set T1 high-order counter
  sta T1C_H

Finally, “Task 1” can be started in the foreground.

  ; start running task 1
  jmp task_1_main

Linker configuration and boilerplate

This is the first time I’m blogging about code written in 65C816 native mode, so for the sake of completeness, I’ll also include the updated linker configuration. The most important change (compared with this blog post) to that 32 bytes are now set aside for interrupt vectors.

MEMORY {
    ZP:     start = $00,    size = $0100, type = rw, file = "";
    RAM:    start = $0100,  size = $7e00, type = rw, file = "";
    PRG:    start = $e000,  size = $2000, type = ro, file = %O, fill = yes, fillval = $00;
}

SEGMENTS {
    ZEROPAGE: load = ZP,  type = zp;
    BSS:      load = RAM, type = bss;
    CODE:     load = PRG, type = ro,  start = $e000;
    VECTORS:  load = PRG, type = ro,  start = $ffe0;
}

Note that this linker configuration is not quite complete, but it is good enough to get code running. The zero page is not relevant for the code I’ll be writing, so I might remove that at some point.

I also have definitions for the 65C22 I/O registers, which correspond to the location that it is mapped into RAM on my prototype computer.

PORTB = $c000
PORTA = $c001
DDRB = $c002
DDRA = $c003
T1C_L = $c004
T1C_H = $c005
ACR = $c00b
IFR = $c00b
IER = $c00e

The final piece of the puzzle then, are definitions for all those interrupt vectors.

.segment "CODE"
irq:
  rti

unused_interrupt:                 ; Probably make this into a crash.
  rti

.segment "VECTORS"
; native mode interrupt vectors
.word unused_interrupt            ; Reserved
.word unused_interrupt            ; Reserved
.word unused_interrupt            ; COP
.word unused_interrupt            ; BRK
.word unused_interrupt            ; Abort
.word nmi                         ; NMI
.word unused_interrupt            ; Reserved
.word irq                         ; IRQ

; emulation mode interrupt vectors
.word unused_interrupt            ; Reserved
.word unused_interrupt            ; Reserved
.word unused_interrupt            ; COP
.word unused_interrupt            ; Reserved
.word unused_interrupt            ; Abort
.word unused_interrupt            ; NMI
.word reset                       ; Reset
.word unused_interrupt            ; IRQ/BRK

The process for building this assembly code into a usable ROM are:

ca65 --cpu 65816 main.s
ld65 -o rom.bin -C system.cfg main.o

This outputs an 8KiB file. I need to pad this with zeroes up to 32KiB, which is the size of the ROM chip I am burning it to.

truncate -s 32768 rom.bin

Lastly, I write this to the ROM.

minipro -p AT28C256 -w rom.bin

Result

I connected a logic analyser to two wires from each output port, the NMI interrupt line, and the reset signal, and opened up sigrok.

This clearly shows the CPU switching between the two processes each time an interrupt is triggered.

Zooming in on the task cut-over, I can also see that the patterns on each port are different, as expected.

Of course this did not work the first time, and I spent quite a bit of time de-constructing and re-constructing the code to isolate bugs.

As result, I shipped some improvements to my 6502 assembly plugin for JetBrains IDE’s, which I have blogged about previously. The plugin will now provide suggestions for mnemonics. There is a project setting for 6502 vs 65C02 vs 65C816 mode, and it will only suggest mnemonics which are available for the CPU.

It also provides a weak warning for binary or hex numbers which are not 8, 16 or 24 bits, which would have saved me some time.

Lastly, it has optional checking for undefined/unused values. This helps catch problems in the editor, and allows me to identify unused code and variables.

Wrap-up

I already knew how multitasking works on a high-level, but it was quite interesting to implement it myself. I have been using this as a test program for my 65C816 computer, since it exercises RAM, ROM, I/O, and uses interrupts.

I don’t think I will be able to use this exact method for more complex programs, because it will break down a bit with multiple interrupt sources. After writing this code, I also found that it’s not common for modern systems to save register values to the stack when context-switching, and that a process control block is used instead.

The next step for my 65C816 computer will be the addition of an NXP UART, which I tested in an earlier blog post.

Interfacing an NXP UART to an 8-bit computer

Thu, 24 Feb 2022 00:00:00 +0000

I recently tested the SC16C752 UART from NXP with my 6502 computer, and I was able to use it to receive and send characters. This blog post shows the simple test program and circuit that I used.

Some background

I built a 65C02-based computer last year, which uses a serial connection for text I/O. I used the WDC 65C51N to provide a UART interface, and I found it to be quite limited.

I am now building a 16-bit computer, and I want to avoid the 65C51N, for two main reasons:

It runs at 5v, and I’m planning to convert the design to 3.3v later on. The other 65-series chips I’m using support a wide voltage range.
It can only store one byte at a time. I am planning to run at a higher baud rate, and the system will have multiple interrupt sources. It would take very careful programming to avoid losing bytes with a 65C51N in this setup.

I spent some time investigating alternative UART IC’s, and found some discussion on 6502.org about replacing the 6551 with an NXP industrial (SC26 or SC28 series) UART. At the time of writing, many of the hobbyist-friendly PLCC versions of these chips are listed as end of life, though I’ve since found pin and software-compatible versions of this made by MaxLinear (eg. the XR88C92).

For my uses though, I chose the NXP SC16C752. It has two channels, 64-byte receive and transmit buffers, and supports a wide voltage range. I don’t mind that it’s a different series to these standard chips, because I plan to write software from scratch. The only downside is the packaging, which is LQFP48 with 0.5mm pin pitch. I needed to solder this tiny chip to an adapter board in order to use it.

The test circuit

I wired this up to my 6502 computer project as shown in the diagram below. IO3 is a spare address decoding output, which maps the chip to address $8c00, while the rest of the signals connect to the 6502 CPU directly.

I used a 74AC00 and 74AC04 to generate the active-high reset and separate read/write signals (as suggested on the same thread linked before). I plan to use a PLD to generate a qualified write signals on my 65C816 computer build.

Note that there are separate chip-selects for each UART, and I’ve only connected one here. I have also not connected the interrupt output for this test, and have some unused inputs which are left floating.

Developing a test program

I started writing some code, and connected a logic analyser to the UART output. The early tests did not work well, and I now know that this is because I had not enabled the FIFOs, causing characters to be lost.

This screen capture shows an attempt to send the entire alphabet to the UART, but only the letters “A”, “B” and “M” appearing at the output.

I came up with this test program, which does 8-N-1 communication at 115,200 bits per second.

;
; Test program for the SC16C752B dual UART from NXP
;
.import sys_exit

; Assuming /CSA is connected to IO3
UART_BASE = $8c00

; UART registers
; Not included: FIFO ready register, Xon1 Xon2 words.
UART_RHR = UART_BASE        ; Receiver Holding Register (RHR) - R
UART_THR = UART_BASE        ; Transmit Holding Register (THR) - W
UART_IER = UART_BASE + 1    ; Interrupt Enable Register (IER) - R/W
UART_IIR = UART_BASE + 2    ; Interrupt Identification Register (IIR) - R
UART_FCR = UART_BASE + 2    ; FIFO Control Register (FCR) - W
UART_LCR = UART_BASE + 3    ; Line Control Register (LCR) - R/W
UART_MCR = UART_BASE + 4    ; Modem Control Register (MCR) - R/W
UART_LSR = UART_BASE + 5    ; Line Status Register (LSR) - R
UART_MSR = UART_BASE + 6    ; Modem Status Register (MSR) - R
UART_SPR = UART_BASE + 7    ; Scratchpad Register (SPR) - R/W
; Different meaning when LCR is logic 1
UART_DLL = UART_BASE        ; Divisor latch LSB - R/W
UART_DLM = UART_BASE + 1    ; Divisor latch MSB - R/W
; Different meaning when LCR is %1011 1111 ($bh).
UART_EFR = UART_BASE + 2    ; Enhanced Feature Register (EFR) - R/W
UART_TCR = UART_BASE + 6    ; Transmission Control Register (TCR) - R/W
UART_TLR = UART_BASE + 7    ; Trigger Level Register (TLR) - R/W

.segment "CODE"
main:
  lda #$80                  ; Enable divisor latches
  sta UART_LCR
  lda #1                    ; Set divisior to 1 - on a 1.8432 MHZ XTAL1, this gets 115200bps.
  sta UART_DLL
  lda #0
  sta UART_DLM
  lda #%00010111            ; Sets up 8-n-1
  sta UART_LCR
  lda #%00001111            ; Enable FIFO, set DMA mode 1
  sta UART_FCR
  jsr test_print
  jsr test_recv
  lda #0
  jmp sys_exit

test_print:                 ; Send test string to UART
  ldx #0
@test_char:
  lda test_string, X
  beq @test_done
  sta UART_THR
  inx
  jmp @test_char
@test_done:
  rts

test_string: .asciiz "Hello, world!"

test_recv:                  ; Receive three characters and echo them back
  jsr uart_recv_char
  sta UART_THR
  jsr uart_recv_char
  sta UART_THR
  jsr uart_recv_char
  sta UART_THR
  rts

uart_recv_char:             ; Receive next char to A register
  lda UART_LSR
  and #%00000001
  cmp #%00000001
  bne uart_recv_char
  lda UART_RHR
  rts

The program sends the text “Hello, world!”, then waits for input. The next three characters it receives are sent back to the user, and then the program terminates. The only thing in this test program which is specific to my 6502 computer is sys_exit, which a routine to transfer control back to the operating system.

It was very rewarding to see the fast, error-free text output on the logic analyser for the first time. This is a great improvement from the 65C51N.

I was also surprised at how short the delay is to echo back a character. It will be interesting to see how much the latency increases once there is some more complex software handling the input.

Wrap up and next steps

I plan to use the NXP SC16C752 for my 65C816 computer project, which will be controlled via text-based input and output.

I’m testing it on its own first, so that I can refer to a known-good setup if it doesn’t work. This is enough to get started, though there are some features which I wasn’t able to test yet, since I can’t use custom interrupt service routines on this setup.

65C816 computer - initial prototype

Thu, 10 Feb 2022 00:00:00 +0000

I am currently working on building a 65C816-based computer from scratch. I have some ideas for how I want to do this, but my first task is to put together a working prototype. I’ll be extending this in different ways to work towards the goals outlined in my last blog post.

This is about the simplest computer I can come up with which uses the 65C816 processor, but there is still quite a lot to go though.

Parts

All of the integrated circuits on this prototype are through-hole packages, suitable for use on a breadboard.

WDC W65C816S CPU
WDC W65C22S VIA I/O chip.
Alliance AS6C4008-55PCN 512k x 8 55ns SRAM
Atmel AT28C256 32k x 8 EEPROM
Atmel ATF22LV10 PLD for address decoding
74AC04, 74AC245, 74AC573 for handling the multiplexed data bus and bank address bits
1.8432 MHz oscillator
DS1813-5+ supervisory circuit

I also used the following equipment to program the chips, and view the result of the test program:

TL-866II+ EEPROM programmer
8-channel Cypress FX2-based logic analyser

I’m working on Linux, and also use the following software tools:

ca65 - for assembling code for the CPU
minipro - for programming the EEPROM and PLD.
galette - for assembling the logic definitions for the PLD.
sigrok - for working with the logic analyser

Bank address latching

The 65C816 has a 24-bit address bus, and the top 8 bits are multiplexed onto the data bus. This is not a big problem, but it does mean that I need some extra glue logic compared to a plain 6502 system. A diagram in the 65C816 data sheet shows a suggestion for how to do this, and I’m starting with that implementation.

I used a 74AC04 to invert the clock, 74AC245 bus transceiver for the data bus, and a 74AC573 latch for the bank address (upper 8 bits of the address bus). This first page also shows the DS1813-5+ supervisory circuit on the reset line, which resets all of the components on power-up.

I’m aware that I should be generating a two-phase clock (there is some discussion about that here), though I can’t see any problems arising from the the delay on the inverted clock on this particular circuit.

Memory map and address decoding

The memory map for this computer is very simple. All of this will change, but I did need to start with something to get some code running.

Only the lower 19 bits of the address space are properly decoded, and mapped to one of three chips.

Address	Maps to
010000 - 07FFFF	RAM - 448 KiB
00E000 - 00FFFF	ROM - 8 KiB
00C000 - 00DFFF	I/O - 8 KiB
000000 - 00BFFF	RAM - 48 KiB

I used a programmable logic device (PLD) to implement this. It’s easy enough to do this with discrete gates, but a PLD will make it much easier to keep chip count and propagation delays under control as I add more features. I’ve written about programming PLD’s in previous blog posts (here, here and here), and I plan to use them to implement several features.

The particular part I am programming here is an ATF22LV10. These are still produced the time of writing, and operate at either 3.3 volts or 5 volts, which is useful for this project. They can be also programmed with the common TL-866II+ chip programmer, which I also use for programming EEPROMs.

The definitions for decoding this are below. I called this file address_decode.pld.

GAL22V10
AddrDecode

Clock  RW     NC     NC     A19    A18    A17    A16    A15    A14    A13    GND
NC     /RAMCS /ROMCS /IOCS  /RD    /WR    NC     NC     NC     NC     NC     VCC

; ROM is the top 8 KiB of bank 0.
; prefix is 0000 111xxxxx xxxxxxxx
ROMCS = /A19*/A18*/A17*/A16*A15*A14*A13

; IO addresses are mapped below ROM in bank 0.
; prefix is 0000 110xxxxx xxxxxxxx
IOCS = /A19*/A18*/A17*/A16*A15*A14*/A13

; RAM is selected for any address does *not* match 0000 11xxxxxx xxxxxxxx
RAMCS = A19 + A18 + A17 + A16 + /A15 + /A14

; Qualify writes with clock
RD = RW
WR = Clock*/RW

DESCRIPTION

Address decode logic for 65C816 computer.

ROM is top 8K of bank 0, I/O is 8k below that, everything else is RAM.

I assembled this with galette:

galette address_decode.pld

This generates a .jed file, which can be used to program the chip. The ATF22LV10 is not on the compatibility list for my programmer, but it works just fine if I identify it as an ATF22V10CQZ.

minipro -p ATF22V10CQZ -w address_decode.jed

Firmware and testing

I came up with a simple test program which would require working ROM, RAM, and I/O. The CPU boots to 6502 emulation mode, and this test program does not access memory outside bank 0.

I created a linker configuration file which mostly reflects the system’s memory map, and called it system.cfg.

MEMORY {
    ZP:     start = $00,    size = $0100, type = rw, file = &quot;&quot;;
    RAM:    start = $0100,  size = $7e00, type = rw, file = &quot;&quot;;
    PRG:    start = $e000,  size = $2000, type = ro, file = %O, fill = yes, fillval = $00;
}

SEGMENTS {
    ZEROPAGE: load = ZP,  type = zp;
    BSS:      load = RAM, type = bss;
    CODE:     load = PRG, type = ro,  start = $e000;
    VECTORS:  load = PRG, type = ro,  start = $fffa;
}

I then wrote this test program, main.s. The program runs from ROM, and writes a hard-coded value to one of the output ports. It then stores a 0 in RAM, then increments it and stores that to an output port in an infinite loop, which will show a counting pattern if RAM can be read and written.

PORTB = $c000
DDRB = $c002

SOME_ADDRESS_IN_RAM = $beef

.segment &quot;CODE&quot;

; Write 42 to VIA port B
reset:
  lda #%11111111 ; Set all pins to output
  sta DDRB

  lda #42
  sta PORTB

; Start at 0, and increment a value in RAM
  lda #00
  sta SOME_ADDRESS_IN_RAM

loop:
  inc SOME_ADDRESS_IN_RAM
  lda SOME_ADDRESS_IN_RAM
  sta PORTB			; Output the value
  jmp loop

nmi:
  rti

irq:
  rti

.segment &quot;VECTORS&quot;
.word nmi
.word reset
.word irq

I assembled this with ca65, then linked it with ld65. As far as the assembler knows, it is generating code for a 6502 CPU, not a 65C816. I am not using any advanced features, so this should be fine for this test.

ca65 main.s
ld65 -o rom.bin -C system.cfg main.o

This outputs an 8 KiB binary file, which is the amount of ROM space mapped in the computer’s memory. The chip is actually 32 KiB though, so I filled the rest with zeroes.

truncate -s 32768 rom.bin

I then wrote the program to the EEPROM using minipro.

minipro -p AT28C256 -w rom.bin

I ran this program, with a logic analyser connected to port B of the 65C22 VIA. The capture below shows the initial hard-coded number, followed by the counting pattern, which indicates that everything is working as expected.

Wrap-up

I’m starting simple here, so that I have a working base system to modify. Everything seems to be working fine on breadboards, so I’ll proceed with that for now. I may still need to re-build everything with shorter wires, since this is getting quite difficult to work with.

It is also worth noting that most of the components here should run at 3.3 volts, which is where this project is eventually headed. Only the clock, reset IC and EEPROM would need to be substituted at the moment.

This blog post is only a snapshot of the project. You can check the current status on GitHub here.

A first look at the 65C816 processor

Thu, 27 Jan 2022 00:00:00 +0000

The 65C816 is an interesting processor from the 1980’s. I recently wired one up on a breadboard, and I’ll be blogging here about my attempts to build something useful with it.

How I got here

I spent a couple of months last year designing and building a computer based on the 65C02, an old 8 bit processor. I wanted to create a hardware platform for running 6502 assembly programs. This was successful, and I learned a lot about how computers work while building and programming it.

This has got me interested in the 65C816, which is a later extension of the same architecture. The only well-known systems which used it were the Super Nintendo and Apple IIGS. From a programmer’s point of view, this chip seems far more capable than its pre-decessor, though developer tools are a bit scarce.

New project goals

My plan is to build a modern computer which uses the 65C816 CPU, so that I can really learn how it works.

Over a few revisions, I am aiming to build up to a system with a serial connection for simple text I/O, a modest clock speed, and 1 megabyte of static RAM.

I’ve researched some existing designs, and settled on two simple constraints to give this project its own character.

Use only in-production parts.
Don’t use an FPGA or microcontroller to bootstrap the system.

This will ensure that I am learning what it takes to build a computer around this processor, and not just offloading the tricky parts to more powerful device with better tooling. I also hope this increases the accessibility of the design, since anybody could build it without needing to obtain obscure retro parts.

I am not avoiding programmable logic entirely though. Where classic designs often used custom chips, I will be using ATF22V10 PLD’s. I will also leave the door open adding video output via a microcontroller-based terminal emulator in the future.

Lastly, I’m selecting parts with a view to converting everything from 5 volts to 3.3 volts part-way through the project, which will open up many possibilities.

Other than the Apple IIGS and Super Nintendo, there are three hobbyist designs which I’m using for inspiration:

BCS Technology’s proof of concept W65C816S computers.
Gordon Henderson’s Ruby ‘816 project.
Adrien Kohlbecker’s 65C816 project.

A quick note about open source

It’s never been a stated goal of this blog, but wherever possible, I use open source tools, and produce open source software. I expect that to be difficult for this project. The 65C816 not widely used, and most hobbyist code for the CPU is licensed as source-available freeware, with restrictions on commercial use.

I would like to be able to release my code under an OSI-approved open source license, and have the option of incorporating copyleft code later on. Unfortunately for me, it seems I will need to write a lot of low-level code from scratch to get this kind of licensing certainty. I will be working primarily in assembly language, though C code would be very useful, so I will certainly be spending some time investigating compiler options along the way.

A smoke test

This is the first iteration of the design, which will live for now on a series of solderless breadboards.

This is simply a 65C816 processor, being fed a hard-coded NOP (no-operation) command. When it runs, the lights show the address bus counting upwards, as the CPU program counter increases.

The RDY, ABORTB, and BE inputs are connected to +5v through resistors, while RESB, NMIB, IRQB are connected to +5v directly. The NOP opcode is fed to the CPU over the data bus, by making the pattern 11101010 (0xEA) with 1k resistors. The schematic for this is below.

The clock is an LM555 timer, fed through a 74AC04 inverter, because the rise and fall times on a 555 are not fast enough. The clock speed is 6.9Hz (that is not a typo), since R1 is 10 kΩ, and R2 is 100 kΩ, and the capacitor is 1 µF.

Next steps

This is going to be a long project, which I will develop incrementally. The next step will be adding ROM, RAM, glue logic and a 65C22 I/O chip.

I am also testing alternatives to the 65C51N UART chip, which I found quite limited when I used it for my last project.

Rendering my 6502 computer project in Blender

Thu, 13 Jan 2022 00:00:00 +0000

For the past couple of months, I’ve been working on building a 6502-based computer from scratch.

In a previous blog post, I wrote about designing a 3D-printed enclosure for this project in Blender.

During this build, I needed to wait for some parts, so I had some time to see what I could do with the 3D models of the case and circuit board. I ended up animating the assembly process, and learned a couple of tricks along the way.

Splitting components and PCB

I already had a 3D model of the circuit board and components. This was initially exported from KiCad, in STEP format. I then used FreeCAD to convert it from STEP to STL, which I imported into Blender for designing the case. I also repeated this export with some different footprints, since I needed sockets and microchips.

It is a challenge to make this look good, because the circuit board and all components come in as one object, and have lost all of the detail from the KiCad 3D view.

I spent a lot of time splitting objects, then selecting faces so that I could assign different materials. I’m using a free BlenderKit account for materials, and using procedural materials where possible.

I also modeled the DC barrel jack and slide switches, which were the first of many components which were missing from the original KiCad export. Still, something was missing.

Adding missing details

I thought that the PCB in the previous image was too bare. The real PCB uses plated through-holes, and almost every hole in a real PCB has a shiny solder pad around it.

To fix this, I exported the front solder-mask layer from KiCad as an SVG, which looks something like this:

I imported this SVG into Blender, scaled it to fit the PCB, extruded it, then went through a series of boolean modifiers. This resulted in a new object which contained every part of the PCB which was not covered in solder mask. I assigned this a shiny metal material.

After this was applied, I subtracted this from the original PCB model, giving the parts of the PCB which are covered in solder mask.

Showing both of these at the same time, I have a PCB with solder pads and plated through-holes.

Importing the silkscreen

This still was not right. On the real PCB, everything is labelled.

I exported the front silkscreen layer from KiCad as an SVG. This layer looks something like this:

I then used InkScape to change this to white text on a transparent background, before finally exporting it as a PNG. I mixed this into the procedural texture which I was using for the PCB, using the transparency as the mix factor. As a final touch, I also exported the front copper layer, and used this as the displacement input for the final material. The nodes are shown in the screen capture below.

This allows me to render a PCB which looks a lot like the real thing.

UV unwrapping everything

In the earlier render, all of the chips had clear surfaces, while real chips are covered with text and logos.

I created small transparent/brown PNG images for each chip, using Inkscape, then UV unwrapped the chips. I mixed the logo into the texture, much the same as I did for the PCB.

I also UV unwrapped each resistor, in order to paint on the correct colour codes. This was perhaps the most egregious waste of time in this whole thing.

Adding final parts

There were plenty more parts which I couldn’t export from KiCad, and needed to model from scratch. I used the bolt factory add-on to create screws, nuts, and hex standoffs (both a screw and a nut, combined).

I also added power and reset buttons (which are partly threaded), plus the speaker. I decided not to add wires, since it would be beyond my skill at the moment.

When everything is displayed at once, a lot of this detail is hidden behind other objects.

Animating

With that last render, I decided to see if I could make an animation of all of the parts coming together. I’ve never animated anything in Blender, so I had a lot to learn.

I added a tracking constraint to the camera, which is panning around an invisible object on the table, and animated everything using key-framed movement. Parts are raised out of view with slightly randomised rotation/location before falling into place.

Rendering

I exported the animation from Blender as 360 separate 1920 x 1080 images, to be played back at 30 frames per second. I would normally use ffmpeg to convert an image sequence into a video, but I wanted to try adding an audio track.

I imported the rendered animation into Kdenlive, which can process image sequences (the button is “Add clip” / “Import as sequence”), fetched a track from the Free Music Archive (Scanglobe - Current. CC BY-NC), and rendered the project.

Wrap-up

I wanted to get a textured, 3D model of this entire project so that I could easily illustrate documentation and project pages. I definitely achieved this, though working with the exported circuit board and components in Blender was tedious.

The version of Blender used here is 2.93, and the files were exported from KiCad 5.1.10.

Assembling my 6502 computer

Thu, 30 Dec 2021 00:00:00 +0000

I recently finished assembling my 6502 computer, so that the whole project is housed in a 3D-printed case.

This is something that should be easy. I already blogged about making the case, and making the circuit board. All that is left is to put the board in the case, right? Well thanks to some mistakes on my part, things are not quite so simple.

More parts

I needed quite a few parts to finish this project, including a power button, reset button, hex standoffs, short screws, and rubber feet. Since this is the first time I’m hitting this stage of a project, I also needed to buy a Du Pont connector kit, stranded wire, heat-shrink tube, and glue.

I have been powering the project from a USB/serial adapter, so I also bought a new one to install permanently. A full parts list is available in the GitHub repository for this project.

I also needed to widen the hole for the reset button due to a mix-up. The specific part I used requires a 7mm hole, while I had designed the case for a button with a 6mm hole.

Making connections

The first step was to modify the board so that it would be compatible with the new UART, which involved de-soldering the female pin headers which I installed previously, and replacing them with male pin headers. I used only a soldering iron, solder wick, and flux for de-soldering, and it was not a fun process. The board survived, so I’ll call it a success.

Next, I soldered wires to the underside of the board for the power button, reset button, and speaker to connect. Each of these is a pair of wires, soldered at one end, and a 2-pin female Du Pont connector on the other.

I attached corresponding connectors on the buttons and speaker. The power button also has an in-built LED, and there are plenty of places to power this from the board’s expansion headers. I added a 2-pin female Du Pont connector, plus a resistor, neatly hidden by heat-shrink.

I checked everything with a multimeter before powering it on. The UART pins were connected correctly to the 65C51N chip and ground. No continuity between the board’s power and the external power unless the power button is pressed, no continuity between the reset line and ground unless the reset button is pressed, and so on.

Troubleshooting

When I powered on the computer, nothing came up on the terminal, which means that I broke something. I tested some voltages, and found the computer was stuck in reset.

The reset signal on the 6502 is active-low, and I’ve added a supervisory IC. Any time the reset signal is low, or the power drops below a minimum voltage, this IC holds it low for 150ms before returning it to +5v. I was confident that the reset buttons and power voltage were fine, so I suspected that one one of the other chips could be causing the problem.

Everything is socketed, so I removed the 65C51N and tested again. This chip is closest to the de-soldering work I was doing earlier, so I thought I may have damaged it.

There was no change, so I started removing other chips, and found that after removing the CPU, the computer was no longer getting stuck in reset.

Following that lead, I found that one of the data bus lines, D4, was connected to the reset line. This would cause activity on the data bus to reset the computer, which would cause this problem.

A quick look at the board in KiCad showed that at one of the points I had attached the reset button, D4 is 0.5mm away from the reset line.

I de-soldered that wire and found a tiny break in the solder mask on D4. This was easy enough to fix, I simply attached the wire for the external reset button elsewhere.

I also bent the CPU pins when re-installing, but after fixing that, I was back in business.

SD card changes

I have been writing some test programs which use a microSD card via one of the 65C22 I/O ports, and wanted to fit this in the case as well.

I swapped the Adafruit ADA254 microSD break-out for the SparkFun DEV-13743. It is slightly smaller, includes pull-up resistors, and still provides level-shifting.

I made a small adapter by attaching a female pin header to a Du Pont housing with superglue. I couldn’t get the “card detect” feature to work, so I wired it up differently to my previous blog post, and updated the code to match. Details of the new wiring is on GitHub.

I also added some masking tape when I installed this to prevent any short circuits. The metal on the outside of the microSD socket is connected to ground, and it could touch some IC pins.

Quick lessons

This is only a learning project for me, so I am trying to take some lessons away from each part of it.

The high-level lessons would be to plan for external connections before building the PCB, and to have all parts on hand before designing the case. That would have saved a lot of re-work, though adapting a design without starting from scratch is a valuable skill in itself.

If I were starting from scratch, I would add pin headers to the board for connecting the power and reset buttons instead of including buttons on the board, and remove the (now unused) DC barrel jack as a power input. I would also make the board larger, and the case taller, so that everything is less cramped. It takes a lot of force to remove chips from machined pin sockets, and the board is too compact to get a screwdriver under the chips.

SD card support is also a bit of a hack, and could be built-in to the board with some effort.

Final results

I now have a fully self-contained 8 bit computer, and it looks more like a finished project than an in-progress experiment. It boots to BASIC, or with the flip of a switch, launches a ROM which allows me to load machine code assembled on my PC.

With the lid removed, I can access the 40-pin expansion header, and one spare I/O port.

The hardware side of this project is now done, and I have a platform for testing any 6502 assembly code on real hardware.

Adding an SD card reader to my 6502 computer

Thu, 16 Dec 2021 00:00:00 +0000

I recently attempted to add SD card support to my home-built 6502 computer.

This was successful, but more challenging than I expected. This blog post is just a few things that I learned along the way.

Background

SD cards are a popular way to add storage to electronics projects. In my case, I will be using the SD card in SPI mode. My home-built computer has an expansion port with some GPIO pins, which should be sufficient for this.

My basic plan was to:

write some test data to an SD card from a modern computer
interface the SD card to my home-built 6502 computer
write a 6502 assembly program to print out the test data

Since this is a learning project, I started from scratch, and avoided reading any existing implementations of SPI or SD cards before jumping in.

Test data

I started this project by generating a small test file, repeating each character in the alphabet 512 times.

echo abcdefghijklmnopqrstuvwxyz | fold -w1 | while read line; do yes $line | head -n512 ; done | tr -d "\n" > disk.img

It takes a one-line shell command to write this image to an SD card on all good operating systems, or a few clicks in the disk imaging utility.

dd if=disk.img of=/dev/my_sd_card

My laptop has an SD card slot, but I also have Windows on it at the moment, which turned this into a far larger problem than I could have imagined.

Driver support for this particular SD card reader was dropped in Windows 10, there are no built-in tools for imaging disks on Windows, and I had to try multiple tools before I could find one which would write a raw disk image (one which does not contain a valid filesystem).

In retrospect, this was the first of many lessons about not having the right tools for this project.

Physical interface

I connected an SD card module to my 6502 computer, via a breadboard. I used a module which could level-shift, because my computer runs at 5V, where SD cards usually run at 3.3V. This particular module is an Adafruit ADA254.

I also added pull-up resistors to all of the data lines, after noticing that there were several sitations where they would float. The wiring I used is:

The full schematic for my 6502 computer is on this blog post, while the schematic for the SD card module is online here.

Software: Part 1

With some test data and a hardware interface, I got to the third step, which was to read data from the SD card. I was completely unprepared for how tricky it is to do this from scratch, without the right experience or debugging tools.

As a quick re-cap, my development environment is still very limited: I am writing 6502 assembly on a modern computer, and I have the ability to send the assembled program to my home-built 6502 computer over a serial connection.

I started by using lecture notes from the University at Buffalo’s EE-379 course, available online here. This describes SPI, SD card command structure, and the sequence to initialise a card.

The two protocols I needed to implement are SPI, which transfers the bits, plus the SD card commands which run over that. I had to write quite a lot of code with no feedback, because it takes several commands before SD cards are ready to work with in SPI mode, and the exact details vary depending on the generation of card (SDHC vs SDXC, etc).

After days of debugging, I was unable to get a response from the SD card. With few other options, I started filling my code with the assembly-language equivalent of print statements, and found that every byte from the SD card was the default FF. This means ’no data’ in this context.

Print statements also slowed the program down to a crawl, which allowed me to use a pizeo buzzer to listen to each line. This clicks each time a signal changes between 0 and 1, so I was able to check that each line had the expected activity. I found that the SD card was sending responses, but that the computer was not receiving them, because they were not connected to eachother on the breadboard.

Software: Part 2

After connecting the card correctly, I could see responses for the first time. The format here is byte sent / byte received, with line breaks between commands.

#
SD card detected
40/7f 00/ff 00/ff 00/ff 00/ff 95/ff ff/ff ff/01 ff/ff
48/7f 00/ff 00/ff 01/ff aa/ff 0f/ff ff/ff ff/09 ff/ff ff/ff ff/ff ff/ff
7a/7f 00/ff 00/ff 00/ff 00/ff 75/ff ff/ff ff/01 ff/00 ff/ff ff/80 ff/00 ff/ff ff/ff
#

The correct initialisation sequence depends on the type of SD card, and I spent a lot of time going through each possibility, looking for one one which worked. When this failed, I found this summary online, which includes more information about decoding the responses: Secure Digital (SD) Card Spec and Info.

As it turned out, I was not sending the correct CRC for CMD8, and a CRC error is indicated by the 09 (binary 0000 1001) response to the second command above. Each bit in the response has a different meaning, and I had mistaken this for indicating an unsupported command, which is expected for some card types.

It’s a good thing that I had this problem, because searching for the CRC algorithm turned up this collection of AVR C tutorials, which includes a 4-part series on implementing SD card support.

I noticed that all of the command examples were wrapped with code like this:

    // assert chip select
    SPI_transfer(0xFF);
    CS_ENABLE();
    SPI_transfer(0xFF);
    // ... code to send a command...
    // deassert chip select
    SPI_transfer(0xFF);
    CS_DISABLE();
    SPI_transfer(0xFF);

Later on, I had a problem where responses were sometimes mis-aligned by one bit. When I also tried sending empty bytes between commands, while the SD card is not selected, these problems disappeared entirely.

Software: Part 3

With the initialisation code complete, I could finally request data from the SD card.

Data on an SD card is arranged in 512-byte blocks by default. I wrote a routine to read one block of data from the SD card into RAM, plus a routine to print sections of RAM.

The result was a program which prints the first kilobyte of the SD card, which is the letter a 512 times, followed by the letter b 512 times.

This screen capture shows the output of a program matching a hexdump of the test file, which is exactly what I was aiming for.

The full code is too long to include in a blog post, but can be found in the repository for this project, mike42/6502-computer on GitHub.

Some lessons

While I was able to read data from an SD card, this took a long time, and there were several points where I nearly hit a dead-end.

I couldn’t see what was happening at a hardware level, and I couldn’t compare my setup with one that worked, because I had never done anything similar before. Spot-the-difference is a powerful debugging tool, and by jumping straight into 6502 assembly, while avoiding existing 6502 implementations, I definitely made things difficult for myself.

With that in mind, there are several ways I could have made this easier:

Better equipment. A logic analyser, an SD card writer, and spare SD cards would have been useful for testing.
Building somewhere less-constrained first. I could have implemented this first on a Raspberry Pi, then ported the result to 6502 assembly. It would then be possible to verify my hardware setup by running existing code, while still getting the learning experience from writing my implementation from scratch. Note that this would have required a different SD card break-out board.
Building something simpler first. For example, Ben Eater released a a YouTube video around the time I was working on this, where he reads from an SPI temperature sensor on a 6502 computer. It would be much easier to get started if I had used any other SPI device on a 6502 first.

The next step in this project is to load programs from an SD card, which would require me to implement a filesystem. This is significantly more complex than what I’ve done here, and I know not to approach it the same way.

More equipment

I picked up a logic analyser, USB SD card reader, alternative SD card module, and a spare SD card.

The logic analyser is compatible with sigrok, which is open source. This is the first time I’ve used this software, and it can decode SD card commands running over SPI, among other things.

The USB SD card reader will allow me to write disk images directly from my development environment, instead of copying them over to a laptop. I do most of my development from a virtual machine, and USB peripherals are easy to pass though.

The alternative SD card module is from SparkFun. Compared with the module I was using in this blog post, it has a smaller footprint, and includes the pull-up resistors. I’m hoping I can use this to fit an SD card into the 3D-printed enclosure for this project.

Implementing the XMODEM protocol for file transfer

Thu, 02 Dec 2021 00:00:00 +0000

I recently built a 6502-based computer from scratch, and I’m using it as a platform for testing 6502 assembly code. The software on this computer is currently very limited, and re-programming an EEPROM chip to test changes is a slow process.

This blog post covers changes which I made to implement the XMODEM protocol, allowing me to upload programs over a serial connection.

The protocol itself

I chose XMODEM because it’s the simplest protocol which is compatible with modern terminal software. All I need right now is the ability to place some data (machine code) into RAM, so that I can execute it.

An important note that is that modern file transfer protocols would normally run over TCP/IP, while XMODEM runs over a serial connection. This works for me, since it’s the only interface I’ve got.

For what it’s worth, XMODEM is also era-appropriate technology for this retro computer, since it was first implemented in 1977. The 6502 processor came out in 1975.

Implementation

Implementing XMODEM was the easy part of this process. There are existing 6502 assembly implementations which I could have used, but I decided to work from the description on the Wikipedia article instead. This is partly to make sure I understand it, and partly to keep the software licensing of my ROM as clear as possible.

I have the beginnings of a shell for running commands from ROM, so I implemented two new commands:

rx, to receive a file over XMODEM and load it into memory.
run to jump to the program

To start the transfer, I found it useful to write a method for reading a character with a time-out. I already have routines for reading and writing characters from serial.

; Like acia_recv_char, but terminates after a short time if nothing is received
shell_rx_receive_with_timeout:
  ldy #$ff
@y_loop:
  ldx #$ff
@x_loop:
  lda ACIA_STATUS              ; check ACIA status in inner loop
  and #$08                     ; mask rx buffer status flag
  bne @rx_got_char
  dex
  cpx #0
  bne @x_loop
  dey
  cpy #0
  bne @y_loop
  clc                          ; no byte received in time
  rts
@rx_got_char:
  lda ACIA_RX                  ; get byte from ACIA data port
  sec                          ; set carry bit
  rts

The entire receive process is around 60 lines of assembly code. The process involves ASCII NAK character repeatedly until a SOH character is received, which indicates the start of a 128 byte packet. All padding around each packet is discarded, and the payload is written to memory. This process is repeated until the end of transmission (EOT) is received.

; Built-in command: rx
; recives a file over XMODEM protocol, and loads it into memory
USER_PROGRAM_START = $0400   ; Address for start of user programs
USER_PROGRAM_WRITE_PTR = $00 ; ZP address for writing user program

shell_rx_main:
  ; Set pointers
  lda #<USER_PROGRAM_START   ; Low byte first
  sta USER_PROGRAM_WRITE_PTR
  lda #>USER_PROGRAM_START   ; High byte next
  sta USER_PROGRAM_WRITE_PTR + 1
  ; Delay so that we can set up file send
  lda #1                     ; wait ~1 second
  jsr shell_rx_sleep_seconds
  ; NAK, ACK once
@shell_block_nak:
  lda #$15                   ; NAK gets started
  jsr acia_print_char
  lda SPEAKER                ; Click each time we send a NAK or ACK
  jsr shell_rx_receive_with_timeout  ; Check in loop w/ timeout
  bcc @shell_block_nak       ; Not received yet
  cmp #$01                   ; If we do have char, should be SOH
  bne @shell_rx_fail         ; Terminate transfer if we don't get SOH
@shell_rx_block:
  ; Receive one block
  jsr acia_recv_char         ; Block number
  jsr acia_recv_char         ; Inverse block number
  ldy #0                     ; Start at char 0
@shell_rx_char:
  jsr acia_recv_char
  sta (USER_PROGRAM_WRITE_PTR), Y
  iny
  cpy #128
  bne @shell_rx_char
  jsr acia_recv_char         ; Checksum - TODO verify this and jump to shell_block_nak to repeat if not mathing
  lda #$06                   ; ACK the packet
  jsr acia_print_char
  lda SPEAKER                ; Click each time we send a NAK or ACK
  jsr acia_recv_char
  cmp #$04                   ; EOT char, no more blocks
  beq @shell_rx_done
  cmp #$01                   ; SOH char, next block on the way
  bne @shell_block_nak       ; Anything else fail transfer
  lda USER_PROGRAM_WRITE_PTR ; This next part moves write pointer along by 128 bytes
  cmp #$00
  beq @block_half_advance
  lda #$00                   ; If low byte != 0, set to 0 and inc high byte
  sta USER_PROGRAM_WRITE_PTR
  inc USER_PROGRAM_WRITE_PTR + 1
  jmp @shell_rx_block
@block_half_advance:         ; If low byte = 0, set it to 128
  lda #$80
  sta USER_PROGRAM_WRITE_PTR
  jmp @shell_rx_block
@shell_rx_done:
  lda #$6                    ; ACK the EOT as well.
  jsr acia_print_char
  lda SPEAKER                ; Click each time we send a NAK or ACK
  lda #1                     ; wait a moment (printing does not work otherwise..)
  jsr shell_rx_sleep_seconds
  jsr shell_rx_print_user_program
  jsr shell_newline
  lda #0
  jmp sys_exit
@shell_rx_fail:
  lda #1
  jmp sys_exit

This code is the first time that I’ve used the “Zero Page Indirect Indexed with Y” address mode. It uses a pointer to the program’s location, which is incremented 128 each time a packet is accepted.

The index in the current packet is stored in the Y register, allowing each incoming byte to be written to memory with one operation, which is called in a tight loop.

sta (USER_PROGRAM_WRITE_PTR), Y

The other routines mentioned in this source listing are:

shell_rx_sleep_seconds - sleeps for the number of seconds in the A register.
shell_rx_print_user_program - prints the first 256 bytes of the program for verification.

The checksum is entirely ignored. Everything here is running on my desk, so I’m not too worried about transmission errors.

After a few iterations, I was able to upload placeholder text to RAM.

The screen capture shows 256 bytes of memory (2 packets in XMODEM). The ASCII SUB character (1a in hex) fills out unused bytes at the end, since everything sent over XMODEM needs to be a multiple of 128 bytes.

Assembling some programs

I thought I was almost done at this point, but it turns out that writing a “Hello World” test program to run from RAM is easier said than done.

I started by creating a new ld65 linker configuration which links code to run from address $0400 onwards, which is where programs are loaded into RAM. This system does not have a loader for re-locatable code, and absolute addresses are assigned by the linker.

This is the configuration which I used for these tests. It has some flaws, but it’s good enough to place the CODE segment in the right place.

# Test configuration for user programs
MEMORY {
    ZP:     start = $00,    size = $0100, type = rw, file = "";
    RAM:    start = $0100,  size = $0300, type = rw, file = "";
    MAIN:   start = $0400,  size = $1000, type = rw, file = %O;
    ROM:    start = $C000,  size = $4000, type = ro, file = "";
}

SEGMENTS {
    ZEROPAGE: load = ZP,  type = zp;
    BSS:      load = RAM, type = bss;
    CODE:     load = MAIN, type = ro;
    VECTORS:  load = ROM, type = ro,  start = $FFFA, optional=yes;
}

If I wanted my test program to terminate, then I needed to find the address of sys_exit in ROM. This routine hands control back to the shell after a command completes.

To get this, I re-assembled the ROM, but added the flag --debug-info to the ca65 assembler, and --dbgfile boot.dbg to the ld65 linker. This produces a text file, which contains the final address of each symbol.

sym id=45,name="sys_exit",addrsize=absolute,scope=0,def=28,ref=298+192+426+390+157,val=0xC11E,seg=0,type=lab

The simplest test program I could come up with was:

; location in ROM
sys_exit = $c11e

.segment "CODE"
main:
  jmp sys_exit

I was able to assemble this program to 3 bytes of binary and run it. It does absolutely nothing, and simply returns to the shell.

To extend this into a “Hello World” program, I needed to look up the location of two more routines in the ROM.

; locations of some functions in ROM
acia_print_char = $c020
shell_newline   = $c113
sys_exit        = $c11e

.segment "CODE"
main:
  ldx #0
@test_char:
  lda test_string, X
  beq @test_done
  jsr acia_print_char
  inx
  jmp @test_char
@test_done:
  jsr shell_newline
  lda #0
  jmp sys_exit

test_string: .asciiz "Test program"

This animation shows the process of uploading the binary and executing it via minicom.

There is nothing special about the invocation for minicom here, it’s just a bit-rate and device name, which in my case is:

minicom -b 19200 -D /dev/ttyUSB0

Automating the boring stuff

This approach is very fragile, because the code is full of hard-coded memory addresses.

Any time I change the ROM, the addresses for these routines could change, and this program will no longer work.

On any real architecture, applications call into the OS via system calls, or maybe a jump table for 8-bit systems. I don’t need a stable binary interface, and there is no userspace/kernel distinction, so I decided not to go down that path.

Instead, I wrote a Python script to generate an assembly source file, using the same debug symbols which I was manually reading. This runs after each ROM build.

The format is:

...
.export VIA_T2C_H                        = $8009
.export VIA_T2C_L                        = $8008
.export acia_delay                       := $c02b
.export acia_print_char                  := $c014
...

When this file is assembled, it doesn’t generate any code, but it does define all of the constants and labels from the ROM. When applications link against this, they can .import anything which would be available to ROM-based code.

The start of the above script is re-written as:

.import acia_print_char, shell_newline, sys_exit

This means that I have source compatibility, just not binary compatibility.

Bringing everything together

With my initial goal achieved, I was still not happy with the development experience. I started to dig into the murky world of minicom scripting to see if I could automate the process of uploading/running programs, and found that it is indeed possible.

The scripting capabilities of minicom do not appear to be widely used, and I needed to read through the source code to find ways to work around the problems I encountered.

For example, the “send” command was sending characters too quickly for the target machine. There are hard-coded delays between commands in the interpreter, but you can’t use multiple “send” commands to slow things down, because it is also hard-coded to append a line ending each time.

The fix was to use !<, which sends the output of a command verbatim. This script uses echo -n command to send each character individually. This is the exact same procedure as the animation in the previous section. First, the script types “rx”, then sends the file, then types “run”.

# confirm we have a shell prompt
send ""
sleep 0.1
# put into receive mode
expect "# "
sleep 0.1
!< echo -n 'r'
!< echo -n 'x'
send ""
sleep 0.1
# send the program and wait for prompt
! sx -q $TEST_PROGRAM 2> /dev/null
expect "# "
# run program
sleep 0.1
!< echo -n 'r'
!< echo -n 'u'
!< echo -n 'n'
send ""
sleep 0.1
# Terminate minicom when program completes (bit crazy..)
expect "# "
sleep 0.1
! killall -9 minicom

It’s also worth talking about the end of this script. When minicom scripts end, the user is returned to an interactive session. I wanted the minicom process to terminate without clearing the screen, so I run killall -9 minicom as the last line of the script.

This is not the safest thing to include in a script, but it’s certainly effective, and I am quite certain that there is no built-in way to achieve this with the current version of minicom. Realistic alternatives include patching minicom, switching to different terminal software.

The minicom invocation includes the name of the minicom script to execute, and the script will use the TEST_PROGRAM environment variable to name the binary file to run.

TEST_PROGRAM=test.bin minicom -b 19200 -S minicom_autorun.txt -D /dev/ttyUSB0

When running this command, the program executes on the 6502 computer seamlessly, though minicom leaves the terminal in a broken state when it is killed, and bash prints the “Killed” text to the screen.

To work around this, I wrapped the minicom invocation in a shell script which builds the program, suppresses all errors, and always returns 0. This avoids constant reporting that killall -9 is a problem, but it does hide any other problems.

#!/bin/bash
set -eu -o pipefail
make $1.bin
(TEST_PROGRAM=$1.bin minicom -b 19200 -S minicom_autorun.txt -D /dev/ttyUSB0 || true) 2> /dev/null

This script is invoked as:

./assemble_and_run.sh test

A few months back, I wrote an IntelliJ plugin for 6502 assembly, and I’m writing all of my code in PyCharm. The last step was to get this running from my IDE.

I added a build configuration which invokes the assemble_and_run.sh script. The script is completely generic, and I can use it to run other programs by changing the argument.

I can now write a program, click run, and watch it execute on real hardware, all without leaving my IDE. It took a lot of effort, but this is a huge improvement from where I started.

Wrap-up

A fast compile/test/run cycle is a huge gain for developer productivity on any system. I can now get fast, accurate feedback on whether my 6502 assembly programs work on real hardware. The first program I’m writing involves interfacing with an SD card, which will be a lot more approachable with this improvement.

I was surprised at how tricky it was to assemble and link a small program to run in RAM instead of ROM, and I can see that my linker configuration will need to be improved as I write programs which use more data. My basic plan is to test features by uploading them over XMODEM, then add them to the ROM when they are working.

The full source code for this project, which includes hardware and software, can be found on GitHub at mike42/6502-computer.

Designing a 3D printed enclosure for my KiCad project in Blender

Thu, 18 Nov 2021 00:00:00 +0000

I recently built a small 6502-based computer on a custom PCB, which I designed in KiCad. This blog post is about the process of building a 3D-printed case to house this project, using Blender.

This is my first ever 3D print, so I had a few things to learn.

Quick background

My effort on this project has been focused on making a design which works, then transferring that design to a PCB. Since this is my first electronics project, I’m working sequentially, and have completely ignored the need for a case until now.

I could have saved some time if I positioned the ports and mounting holes to match pre-made electronics enclosures, but I’ll take that as a lesson for my next project.

The requirements for the enclosure are simple:

Ability to mount the PCB inside the case
Cut-out for power input
Cut-outs for cable routing to expansion ports
Cut-outs for power and reset buttons

My PCB design files are in KiCad, and I decided to design the case in Blender, since I already know how to use it. Blender is not a CAD tool, but it is excellent for general-purpose 3D work, and will do just fine for this task.

Getting scale

My first challenge was to get the PCB into Blender at the correct scale, so that I could build a model around it. This is mostly just juggling file formats, but also note that Blender works in metres by default. I’ve set it to millimetres, with a scale of 0.001. There are guides on the web about how to set this.

I tried two different methods.

Method 1: Image import

In KiCad, I added two “dimensions” to show the distance between the centre of the mounting holes, then plotted the front copper layer as an SVG.

I used Inkscape to convert from SVG to PNG. I then used GIMP to add cross-hairs to the centre of the mounting holes.

In Blender, I added a single vertex at the origin, then extruded it on the X axis to match the measured dimension. I then added a reference image (the PNG file exported above), and scaled/moved the reference image until the cross-hairs lined up with the vertex.

Method 2: Model import

KiCad can export its 3D model to a STEP file. I imported this file into FreeCAD, then exported it to STL.

Lastly, I imported the STL into Blender.

The scale is correct with both of these methods, because I can align the image and model.

The second method produces much larger .blend files, but has less room to make undetected errors, so I’ll be using it in the future.

Building a box

I modeled out the case as a lid and base with 3mm walls, and used boolean modifiers to cut out spaces for buttons and cables.

One challenge was rounding the corners without the two pieces intersecting. I built the box as a square, then beveled it in wire-frame view. The spaces are larger than necessary, and I will try using modifiers slot the pieces together with a fixed tolerance if I need to do this again.

The result was two pieces which sit together but do not attach, with rounded corners and no sharp overhangs, for ease of manufacturing.

Manufacturing and test

I exported the STL files, and sent them to a local manufacturer, since I don’t have my own 3D printer. The print is black PLA, printed with fused deposition modeling, with an 0.2mm layer height and 30% infill.

When I received the parts, I first checked that the two pieces fitted together, then placed a blank PCB over the base to check that the holes lined up. PLA shrinks as it cools, but this aligned perfectly, which is either good luck or correct calibration.

The computer’s board will be installed on standoffs, which are secured by a nut on the other side. I like the idea of using heat-set inserts instead, but I don’t want to try too many new things at once, so I’ve skipped the idea for now.

Next steps

At the time of writing, I’m still waiting for some parts to assemble this computer with a power/reset button, which will be the end of the hardware side of this project.

I’m quite happy with how this turned out as my first ever 3D print. If I need to make enclosures for future projects, I’ll take another look at open source parametric CAD tools such as OpenSCAD and FreeCAD, which are more targeted to this kind of work.

The STL files for this case are available on GitHub at mike42/6502-computer, critique is welcome.

Building a hardware interrupt controller

Thu, 04 Nov 2021 00:00:00 +0000

I’ve recently been adding simple hardware devices to my home-built 6502 computer, and ran into a problem.

The 6502 has two active-low interrupt inputs (IRQ and NMI), both of which are used in my design. If I add any devices which can trigger their own interrupts, I will need a way to combine multiple interrupt signals into one.

Choosing an approach

I researched some retro computer designs to see how this problem was handled, and found some very simple approaches.

Most commonly, multiple I/O chips would be connected to the 6502’s IRQ line, which is level-triggered and active-low. As long as all connected chips had an open-drain IRQ output, they could be connected in a so-called wired-or configuration:

In software, each interrupt source would be checked in priority order.

I can’t use something so straightforward to add hardware to my design, for two main reasons:

I would need to use logic gates to combine the inputs instead. The I/O chip which is connected to the CPU’s IRQ input at the moment is a WDC 65C22S, which does not have an open-drain IRQ output.
It’s not going to be practical to check every I/O device from an interrupt service routine. I’m planning to add devices which are accessed over SPI, and will take many clock cycles to return their status.

Over in the PC world, programmable interrupt controllers such as the Intel 8259 were used. Among other features, these chips combine multiple interrupt sources into one, and can report the interrupt source on the data bus.

Rather than use out-of-production retro parts, I decided to program an ATF22V10 programmable logic device with these functions. A PLD is cheaper, and I’ve just figured out how to program them, so I might as well put those skills to use.

Creating the interrupt controller

I am using galette to program these PLD’s, and went through 5 revisions of the PLD source file before landing on something which could complete these functions.

combine multiple interrupt lines into one.
allow the CPU to quickly identify the highest-priority interrupt to service.

For this section, I’ll briefly describe each part of the PLD definition is doing.

The file starts with the name of the target device, and a name for the chip.

GAL22V10
IrqController

Next pin definitions are listed. I’m using the ATF22V10, which has 24 pins. The first row is pins 1-12, while the second row is pins 13-24. Numbers go left-to-right in both rows, unlike a physical microchip!

Clock    /IRQ0 /IRQ1 IRQ2  /IRQ3 /IRQ4 IRQ5  /IRQ6 /IRQ7 /IRQ8 /IRQ9  GND
/CS      D7    D6    D5    D4    D3    D2    D1    D0    /IRQ  /WE    VCC

For combining the interrupts, I simply use a big OR function.

IRQ = IRQ0 + IRQ1 + IRQ2 + IRQ3 + IRQ4 + IRQ5 + IRQ6 + IRQ7 + IRQ8 + IRQ9

This reads “IRQ is active if IRQ0 is active or IRQ1 is active or IRQ2 is active, etc.”. All logic is the positive case, so “IRQ0” means “IRQ0 is active”. Whether “active” means 1 or 0 at the input depends on those pin definitions. The active-low inputs are inverted before being fed to this expression, and the result will be inverted if the output is active-low. This confused me at first, because I thought that “/IRQ0” is just a pin name.

The expressions for the data bus come next.

These are all ‘registered’ outputs (indicated with the .R suffix). This runs the output through a D-type flip-flop, so that it will only change on clock transitions, rather than asynchronously. This is to avoid garbage output if an interrupt triggers while we are reading from the controller.
D1..D4 is a priority encoder, encoding the number 0 to 10 to identify the lowest-numbered interrupt source, or 11 if there is no interrupt.
D0 is always 0. This multiples the output by 2, which makes things easier for the software.

D0.R = GND
D1.R = IRQ1*/IRQ0 + IRQ3*/IRQ2*/IRQ1*/IRQ0 + IRQ5*/IRQ4*/IRQ3*/IRQ2*/IRQ1*/IRQ0 + IRQ7*/IRQ6*/IRQ5*/IRQ4*/IRQ3*/IRQ2*/IRQ1*/IRQ0 + IRQ9*/IRQ8*/IRQ7*/IRQ6*/IRQ5*/IRQ4*/IRQ3*/IRQ2*/IRQ1*/IRQ0
D2.R = IRQ2*/IRQ1*/IRQ0 + IRQ3*/IRQ2*/IRQ1*/IRQ0 + IRQ6*/IRQ5*/IRQ4*/IRQ3*/IRQ2*/IRQ1*/IRQ0 + IRQ7*/IRQ6*/IRQ5*/IRQ4*/IRQ3*/IRQ2*/IRQ1*/IRQ0 + /IRQ9*/IRQ8*/IRQ7*/IRQ6*/IRQ5*/IRQ4*/IRQ3*/IRQ2*/IRQ1*/IRQ0
D3.R = IRQ4*/IRQ3*/IRQ2*/IRQ1*/IRQ0 + IRQ5*/IRQ4*/IRQ3*/IRQ2*/IRQ1*/IRQ0 + IRQ6*/IRQ5*/IRQ4*/IRQ3*/IRQ2*/IRQ1*/IRQ0 + IRQ7*/IRQ6*/IRQ5*/IRQ4*/IRQ3*/IRQ2*/IRQ1*/IRQ0
D4.R = /IRQ7*/IRQ6*/IRQ5*/IRQ4*/IRQ3*/IRQ2*/IRQ1*/IRQ0
D5.R = GND
D6.R = GND
D7.R = GND

For the 22V10, not all pins support the same number of product terms. As I built up to 10 interrupt sources, I needed to re-arrange the pins a few times.

Lastly, I needed to tri-state the output when the chip was not being read. This is a second definition for each pin (.E suffix). I used both a chip select (/CS) and write enable (/WE) input. The interrupt controller cannot be written to, this just allows me to avoid bus contention if somebody is attempting to.

D0.E = CS*/WE
D1.E = CS*/WE
D2.E = CS*/WE
D3.E = CS*/WE
D4.E = CS*/WE
D5.E = CS*/WE
D6.E = CS*/WE
D7.E = CS*/WE

Test circuit and software

After testing everything on breadboard, I connected the new interrupt controller to my 6502 computer. All of the required signals are exposed via pin headers on my 6502 computer, more information about that can be found on previous blog posts.

This is how it looks. It’s not the neatest, but at least I don’t have the whole computer on breadboards anymore.

I then started writing some 6502 assembly code to test this out. I’ve been working on a shell which runs named commands, so I added a command called irqtest. This code references other parts of the ROM, which can be found in the GitHub repository for this project.

This code sets up a timer to trigger an interrupt on the 65C22 VIA, then uses the wai instruction to wait for interrupts. When the interrupt service routine completes, the code resumes. I’ve assigned two addresses (DEBUG_LAST_INTERRUPT_INDEX and DEBUG_INTERRUPT_COUNT) so that we can check that the correct interrupt routines are running.

; Set up a timer to trigger an IRQ
IRQ_CONTROLLER = $8C00

; Some values to help us debug
DEBUG_LAST_INTERRUPT_INDEX = $00
DEBUG_INTERRUPT_COUNT = $01

shell_irqtest_main:
  lda #$ff              ; set interrupt index to dummy value (so we can see if it's not being overridden)
  sta DEBUG_LAST_INTERRUPT_INDEX
  lda #$00              ; reset interrupt counter
  sta $01
  ; setup for via
  lda #%00000000        ; set ACR. first two bits = 00 is one-shot for T1
  sta VIA_ACR
  lda #%11000000        ; enable VIA interrupt for T1
  sta VIA_IER
  sei                   ; enable IRQ at CPU - normally off in this code
  ; set up a timer at ~65535 clock pulses.
  lda #$ff              ; set T1 low-order counter
  sta VIA_T1C_L
  lda #$ff              ; set T1 high-order counter
  sta VIA_T1C_H
  wai                   ; wait for interrupt
  ; reset for via
  cli                   ; disable IRQ at CPU - normally off in this code
  lda #%01000000        ; disable VIA interrupt for T1
  ; Print out which interrupt was used, should be 02 if irq1_isr ran
  lda DEBUG_LAST_INTERRUPT_INDEX
  jsr hex_print_byte
  jsr shell_newline
  ; print number of times interrupt ran, should be 01 if it only ran once
  lda DEBUG_INTERRUPT_COUNT
  jsr hex_print_byte
  jsr shell_newline
  lda #0
  jmp sys_exit

I set my interrupt service routine read from the interrupt controller, then jump to the correct routine.

irq:
  phx                       ; push x for later
  inc DEBUG_INTERRUPT_COUNT ; count how many times this runs..
  ldx IRQ_CONTROLLER        ; read interrupt controller to find highest-priority interrupt to service
  jmp (isr_jump_table, X)   ; jump to matching service routine

irq_return:
  plx                       ; restore x
  rti

The interrupt controller can return 11 possible values, so I made an 11-entry table, with the address for each interrupt handler (2 bytes each). I have connected the VIA to IRQ1, so I set the second entry to a subroutine called irq1_isr.

isr_jump_table:              ; 10 possible interrupt sources
.word nop_isr
.word irq1_isr
.word nop_isr
.word nop_isr
.word nop_isr
.word nop_isr
.word nop_isr
.word nop_isr
.word nop_isr
.word nop_isr
.word nop_isr               ; 11th option for when no source is triggering the interrupt (phantom interrupt)

Most of the interrupt routines map to nothing at all. If this were a real OS, an interrupt from an unknown source would need to be a fatal error, since it can’t be individually masked/ignored on this hardware.

nop_isr:                         ; interrupt routine for anything else
  stx DEBUG_LAST_INTERRUPT_INDEX ; store interrupt index for debugging
  jmp irq_return

When IRQ1 is triggered though, this routine will clear the interrupt on the VIA. If we fail to do this, then the CPU will become stuck, processing interrupts forever.

irq1_isr:                        ; interrupt routine for VIA
  stx DEBUG_LAST_INTERRUPT_INDEX ; store interrupt index for debugging
  ldx VIA_T1C_L                  ; clear IFR bit 6 on VIA (side-effect of reading T1 low-order counter)
  jmp irq_return

Once I fixed some bugs, I was able to get the expected output, which is 02 (the index we are using to jump into isr_jump_table), followed by 01 (the number of times the interrupt service routine runs).

While I was researching this, I also found that online 6502 assembly guides often suggest clearing the interrupt flag on I/O chips near the start of the interrupt service routine, apparently to avoid running the routine twice. From this test, I know that the interrupt service routine is only running once, so I am happily disregarding that advice. Thinking about it, I can only see this applying to open-drain IRQ outputs.

Note about KiCad symbols

Since my last blog post about PLD’s, I’ve started to create separate symbols in KiCad for each programmed chip. You can see one of these in the schematic above.

I’m producing these automatically. Galette has a .pin file as one of its outputs. The file for this chip is as follows:

 Pin # | Name     | Pin Type
-----------------------------
   1   | Clock    | Clock/Input
   2   | /IRQ0    | Input
   3   | /IRQ1    | Input
   4   | IRQ2     | Input
   5   | /IRQ3    | Input
   6   | /IRQ4    | Input
   7   | IRQ5     | Input
   8   | /IRQ6    | Input
   9   | /IRQ7    | Input
  10   | /IRQ8    | Input
  11   | /IRQ9    | Input
  12   | GND      | GND
  13   | /CS      | Input
  14   | D7       | Output
  15   | D6       | Output
  16   | D5       | Output
  17   | D4       | Output
  18   | D3       | Output
  19   | D2       | Output
  20   | D1       | Output
  21   | D0       | Output
  22   | /IRQ     | Output
  23   | /WE      | Input
  24   | VCC      | VCC

To produce schematic symbols, I wrote a small Python script to convert this text format to a CSV file, suitable for KiPart.

IRQ_Controller

Pin,Type,Name
1,input,Clock
2,input,~IRQ0
3,input,~IRQ1
4,input,IRQ2
5,input,~IRQ3
6,input,~IRQ4
7,input,IRQ5
8,input,~IRQ6
9,input,~IRQ7
10,input,~IRQ8
11,input,~IRQ9
12,power_in,GND
13,input,~CS
14,output,D7
15,output,D6
16,output,D5
17,output,D4
18,output,D3
19,output,D2
20,output,D1
21,output,D0
22,output,~IRQ
23,input,~WE
24,power_in,VCC

KiPart ships with profiles for FPGA chips, but the generic input worked fine for the ATF22V10 PLD.

./KiPart/venv/bin/kipart -r generic --overwrite irq_controller.csv

The output is a .lib file, which I can include in any KiCad project.

Wrap-up

This approach works, and introduces far less overhead compared with checking each device in the interrupt service routine. In particular, it will allow me to test interrupts from slow-to-access SPI devices.

I will be disconnecting this this for now, but may include it in an expansion board or updated computer design if I ever make one!

Programming PLD's with open source software

Thu, 21 Oct 2021 00:00:00 +0000

A few weeks ago, I blogged about my setup for programming PLD’s from Linux, which are the simpler ancestors of modern FPGA’s.

The device I’m using is the ATF22V10, and programming it in involved running decades-old proprietary software called WinCUPL under WINE. I recently found an opportinity to test-run an open-source alternative, galette, and this blog post is a few notes about how it went.

Re-visiting my assumptions

The Atmel ATF22V10 is pin-compatible with the Lattice GAL22V10, which was discontinued over 10 years ago. There is a lot more information and software available for Lattice GAL’s, which I assume had the mind-share back when these devices were relevant.

There are some slight differences between the Atmel and Lattice devices. Some programming hardware works with one but not the other, and I had assumed that the fuse map (JED file) would be incompatible as well. This turned out to be incorrect.

A few statements online made me look at this again “You can also write Lattice GAL maps to the Atmel ATF16V8 and 22V10 parts” (Andrew B on retrobrewcomputers.org), and “The most suitable replacement for the GAL22V10 from Atmel is the ATF22V10C/CQ/CQZ. They are pin-to-pin and JEDEC fusemap compatible.” (GAL22V10 replacement on the Microchip knowledgebase).

The open source tool galette can create JED files for the Lattice GAL22V10, so I decided to try to write its output to an ATF22V10 to see for myself.

Detour: Installing a Rust toolchain

Galette is written in Rust, and available only as source code at the time of writing. The instructions for setting up Rust are online here.

apt install curl
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Then to compile any Rust project is the same process:

$ cargo build

Finding an example JED file

Next I tried running the tests, which did not pass

$ ./run_tests.sh

I found that this was simply the assembler version being different, plus a trailing number (maybe a checksum?).

diff -ru baseline/vcc.jed test_tmp/vcc.jed
--- baseline/vcc.jed    2021-08-19 20:16:20.526847059 +1000
+++ test_tmp/vcc.jed    2021-08-19 20:26:00.367537520 +1000
@@ -1,5 +1,5 @@

-GAL-Assembler:  Galette 0.2.0
+GAL-Assembler:  Galette 0.3.0
 Device:         GAL16V8

 *F0
@@ -25,4 +25,4 @@
 *L2193 0
 *C3c8f
 *
-924f
+9250

I then took one of the test outputs, and tried writing it to a device. As detailed in my earlier blog post, I’m using minipro, with a TL866II+ programmer, which could only program these devices after a firmware update.

$ minipro -p ATF22V10CQZ -w baseline/GAL22V10_combinatorial.jed
Found TL866II+ 04.2.126 (0x27e)
Warning: Firmware is newer than expected.
  Expected  04.2.123 (0x27b)
  Found     04.2.126 (0x27e)

VPP=12V
Warning! JED file doesn't match the selected device!

Declared fuse checksum: 0x87E2 Calculated: 0x87E2 ... OK
Declared file checksum: 0x12CF Calculated: 0x12D0 ... Mismatch!
JED file parsed OK

Use -P to skip write protect

Erasing... 0.33Sec OK
Writing jedec file...  5.04Sec  OK
Reading device...  0.41Sec  OK
Writing lock bit... 0.36Sec OK

With failed tests, checksum errors, firmware version mismatch and device ID mismatch, I was not expecting this to work.

Checking the source for this file, one of the lines is a simple AND expression.

O0 = I0 * I1

I wired up two inputs and one output (pins 2, 3 and 14), to confirm that it behaved as expected.

I used a multimeter to check that it was drawing a reasonably low current (about 7 mA). In previous experiments, these chips had made a lot of heat when programmed incorrectly.

Everything worked, so this open-source replacement is looking good so far.

Update: I repeated the simple test of combinatorial logic in an ATF22LV10C running at 3.3v. At the time of writing, this ’low voltage’ variant is not listed as a supported chip in minipro, but can be programmed using the exact same procedure as an ATF22V10, and then operated at either 5V or 3.3V. The command I am using to program this chip is:

minipro -p ATF22V10CQZ -w baseline/GAL22V10_combinatorial.jed

Interfacing with a 6502 computer

I have some ideas for using programmable logic to extend my 6502-based computer, so I wanted to test tri-state output.

The previous test only used combinatorial output, where each pin is set high or low according to a logic function. For the ATF22V10, it should also possible to program an “output enable” function for each pin, in order to use tri-state logic. This would allow me to connect it to the 8-bit data bus on my home-built computer.

The best documentation for this was:

To test this, I programmed an ATF22V10 to output “42” (0010 1010) to the data bus when an output enable pin is low. In the source file, 0 is GND, and 1 is VCC.

Tri-state outputs are suffixed with .T, while outputs suffixed with .E are the “output enable” functions for that pin (the pin is high-impedance when false). I left one output in the combinatorial mode (always enabled), to verify that this is being controlled separately for each pin.

GAL22V10
Test1

NC    NC    I1    OE    NC    NC    NC    NC    NC    NC    NC   GND
NC    NC    O1    D7    D6    D5    D4    D3    D2    D1    D0   VCC

O1 = I1

D0.T = GND
D1.T = VCC
D2.T = GND
D3.T = VCC
D4.T = GND
D5.T = VCC
D6.T = GND
D7.T = GND

D0.E = /OE
D1.E = /OE
D2.E = /OE
D3.E = /OE
D4.E = /OE
D5.E = /OE
D6.E = /OE
D7.E = /OE

DESCRIPTION

Output 42 on the data bus when output is enabled.

I used my local build of Galette to generate a JED file, then wrote the JED file to the chip using the same process as before.

$ ./galette/target/debug/galette test1.pld
$ minipro -p ATF22V10CQZ -w test1.jed

I then built this test circuit, which connects one I/O select line from my computer to the PLD, plus the 8-bit data bus. The combinatorial output is connected to an LED for testing.

The IO2 input maps to address $8800 in my computer’s address decoding scheme. I have other blog posts about mapping new hardware devices into memory if you’re interested to know how that works.

This computer boots to BASIC, and I was able to print “42” by reading this memory address via the PEEK command.

This also confirms that D0 is the least-significant bit in the data bus, which I was not sure of before.

Next steps

This has been an interesting discovery for me, and I’m planning to use Galette instead of WinCUPL for my electronics projects. PLD’s are relatively obscure, and it’s great to have an open-source tool for working with them. The only thing I will miss from WinCUPL it its test/simulation feature. I now need to program a chip and test it in a circuit, which is laborious, and I’m quickly using up the limited number of write cycles that these devices are rated for.

I created an “ATF22V10” symbol in KiCad to draw the schematics for this blog post, which is not particularly clear, since the pin names do not correspond with the ones defined in the .pld file. The next time I’m including one of these in a schematic, I’ll try creating a separate symbol for each programmed chip.

I’m planning to use an ATF22V10 PLD to build an interrupt controller for my computer, look out for a future blog post about that.

Re-creating the world's worst sound card

Thu, 07 Oct 2021 00:00:00 +0000

I recently built a computer from scratch, and wanted to add some basic sound output. I am not attempting to produce any 8-bit music, but I would like to be able to write programs use beeps and clicks to provide feedback, instead of only text.

I did a bit of digging into the subject, and found quite possibly the world’s worst sound card, in the Apple II. This is my attempt to re-draw the relevant part of the schematic, which I found in a 1983 publication called The Apple II Circuit Description.

The linked book does a good job of explaining the details, but the important part for this blog post is simply that Z3 is an address decoding output. It is normally high, but is low for half a cycle each time that memory address $C030 is accessed. This is fed to a flip-flop, which will toggle each time this happens, driving a speaker.

This should be easy for me to implement, since I have spare address decoding outputs on my computer.

First attempt

I decided to use a piezo buzzer, which can be driven directly from an IC pin, simplifying things greatly.

I first connected the buzzer directly to one of the spare address decoding outputs of my computer.

This is so simple that it is almost not worth diagramming, but here goes:

The input is IO2, which works out to address $8800 in my computer’s memory map. I wrote a bit about in an earlier blog post.

From BASIC (yes, this computer runs BASIC), I can read from this address and produce a click.

A=PEEK($8800)

Second attempt

To produce a beep, I wanted a square wave with a 50% duty cycle. The Apple II used a 74LS74 dual D flip flop IC for this, which I don’t have in my inventory. Instead, I used a 74AC163 4-bit counter.

The circuit on the breadboard is wired up like this:

I was then able to produce a beeping sound by reading a memory address in a loop, just as you could on the Apple II.

10 A=PEEK($8800)
20 GOTO 10
RUN

Mission accomplished.

Alternatives and next steps

I’ve already started adding clicks and beeps to assembly code while testing, and my immediate plan is to add a separate startup beep for each of the two boot ROMs. If I ever do a second revision of this computer board, I’ll most likely add a speaker as an on-board peripheral, using a 74LS74 instead of the counter.

There are two other methods which considered for adding beeps to this computer. The first was to utilise its 65C22 chip, which can be set to output a square wave on its PB7 pin. I have other plans for that output, which is the reason I haven’t used it for sound.

The second idea I looked into was sampled audio, which would require me to latch 8-bit audio samples a few thousand times per second, and then use some resistors to build a digital-to-analog converter. There are a few ways that I could achieve this with parts which I have on-hand, such as a SN74AC573 latch, or the 65C22.

6502 computer - from breadboard to PCB

Thu, 23 Sep 2021 00:00:00 +0000

I’ve recently been working on building a 6502-based computer on breadboards as a learning project. After making a few revisions, and porting some software to run on it, I was confident that my computer worked well enough to connect up permanently on a circuit board.

This blog post about the process that I went through to convert my working breadboard prototype to a PCB. As somebody who is not trained in electronics, this involved some fresh challenges for me.

Schematic capture

As a first step, I needed to draw up a proper schematic in a CAD tool. I’ve already been learning to use KiCad’s Eeschema for schematic capture, so I went ahead and drew up the whole thing.

I decided to put all of the components on one crowded page, just because I haven’t learned to manage multiple pages yet.

Mostly, this was just replicating what I’d already built, but I also needed to:

create a symbol for the DS-1813, which I could not find in the libraries I’m using
decide on a pinout for expansion ports
add a jumper block, so that the two interrupt lines (NMI and IRQ) can be disconnected from the on-board chips

I avoided changing the scope of the board to include important features (storage, audio), and instead added a header with all of the important buses and signals. My goal is to have a stable base system to work with, and I can always do a second board once I’ve figured out how these should work.

I did not use KiCad’s inbuilt BOM plugins, but instead manually wrote a parts list for future reference. The only components which I needed to order were sockets and pin headers, everything else was being lifted from the prototype.

Footprint assignment

Once the Eeschema electrical rules check was passing, I moved on to selecting footprints for everything. I spent quite a bit of time checking the datasheets for my planned components against the KiCad library to choose footprints that would fit.

I could not find anything which matched the footprint of the mini SPDT switches that I’m using, so I made my own.

I had not yet chosen a ZIF socket for the EEPROM, so I selected a footprint for a standard socket instead. This was a risky move, because I had to guess how much space to leave when laying out the board.

PCB design

I had already selected a board manufacturer, so I set up my track widths and via sizes according to their capabilities.

Next, I imported the netlist and placed the components. I attempted to fit everything in 10cm x 10cm, 2-layer board to save on manufacturing costs, but it was very crowded. I was worried about having enough space for traces, fitting an (unmeasured) ZIF socket, and adding some mounting holes, so I spaced it out to fit on a 10cm x 11.6cm board instead.

I also added footprints for M3-sized mounting holes, and edge cuts with rounded corners.

It took me 4 attempts to successfully route all of the traces. The result breaks every PCB layout best practice which I had read about, but I was happy just to have everything connected by that point.

In hindsight, I could have made this task easier by splitting up the expansion header, and building an accurate footprint for a ZIF socket. I also could have placed the decoupling capacitors closer to the IC power pins, and avoided interrupting the ground plane.

The silkscreen setup was more time-consuming than expected. I wanted to make the board as self-documenting as possible, so that I would not need to open the design files on a computer when writing code and adding hardware peripherals. I labelled every switch, button, light and IC, as well as every expansion header pin (74 of them!). I also added some simple assembly hints such as resistor values, and +/- signs to indicate the polarity of the lone electrolytic capacitor.

The 3D render view in KiCad shows how the finished board would look.

Components did not display on the board at first, but there is a menu option to download the 3D models. After setting this up, and setting the solder mask colour to black, I was able to render the board properly. This looks very similar to the assembled board, with just a few parts missing.

Manufacturing

I only recently discovered that low-volume PCB manufacturing is accessible to hobbyists. Ordering the boards was really easy. I exported gerber and drill files according to their instructions, loaded them into a zip file, and uploaded them to a web portal.

I am glad that an online gerber viewer was available, because it allowed me to spot an error which I had not noticed in KiCad, where I had two pins labelled PA1.

The default settings are reasonable, but I chose black solder mask, a lead-free surface finish, and chose the option to exclude the manufacturer’s order number from the board.

The gerber viewer also has a tab which shows checks against their manufacturing rules.

I placed the order, and eagerly checked each day as the boards progressed through the manufacturing steps. From placing the order to getting the boards on my desk, the whole process took 7 days.

Assembly & test

Once all of the sockets arrived in the mail, I started assembling.

The standard advice I’ve read for through-hole assembly is to solder low-profile components first. Instead, I added enough components to light up the power LED.

I know that the power LED dims if anything is drawing too much current (eg. a short circuit). By getting this working first, I will know I’ve made a mistake (and can switch off the board quickly) if it doesn’t light up later.

I don’t think I’ve ever soldered more than a few pins or wires at a time, and this board had 323 pins to solder.

After a about 2 hours, I had a fully-populated board. It’s smaller and looks better than the breadboards, but it was time to move all the chips across to see if it actually worked.

Much to my surprise, the computer booted up to EhBASIC on the first attempt.

Wrap-up

This has been quite a fun project. It has proven to me that open source tools are perfectly sufficient for this type of hardware development. I’m also very happy to have reached a point where can call this project “done”, since I have a permanent home-built computer.

My plans for learning about low-level computing are not done yet, of course. I’ve kept everything at a low 1.8 MHz, so that I can continue to prototype hardware peripherals on a breadboard. I’ve also left a switch to select which half of ROM to boot from, so that I don’t brick my computer every time I make an error in an assembly language program.

I’ve uploaded the full parts list, design files, and firmware to GitHub at mike42/6502-computer.

Porting BASIC to my 6502 computer

Thu, 09 Sep 2021 00:00:00 +0000

I’ve recently been building my own 6502-based computer. After a lot of work on the hardware side, I decided that it’s time to move past “Hello World” programs, and port a BASIC interpreter.

Background

This computer is architecturally similar to a 1980’s home computer, where BASIC was a widely-used interpreted language. Porting an existing interpreter will be a fast way to get a command-prompt, and it will also add the ability to load arbitrary software, by typing out a program.

I heard about an interpreter called EhBASIC (“Enhanced BASIC”) from on this YouTube video by Chris Bird. It’s source-available (free for non-commercial use only), and seems to be well-regarded by 6502 enthusiasts.

EhBASIC was written by the late Lee Davidson. I based my port on the version v2.22, hosted on Klaus Dormann’s GitHub. I used Hans Otten’s mirror of Lee’s website as a reference, plus the EhBASIC section of the 6502.org forum.

Porting process

EhBASIC was a breeze to port.

I started with the ‘patched’ code from Klaus2m5/6502_EhBASIC_V2.22, which I placed in a local git repository, so that I could track the changes.

I first changed some of the syntax so that the code would assemble with ca65, since that is the assembler I’m using for everything else. There are other ca65 ports around, though they do not include the same patches.

The --feature labels_without_colons setting in ca65 was useful here, since the original source does not include colons after label names. The output was 10.5KiB of machine code, which will fit in the 16KiB of ROM space which I have available in my home-built computer.

I also confirmed that the code did not depend on using undocumented 6502 CPU opcodes, which would have been incompatible with my newer-generation 65C02 CPU.

Next, I dropped in the correct routines to read and write characters. I already had something similar from an earlier blog post. I needed to make some small changes to use the carry bit (SEC / CLC opcodes) to indicate whether a character had been received, and to set up the 6551 ACIA on startup. I programmed the code to an EEPROM, and it worked the first time.

All of my changes against the original code, including the Makefile and ca65 config file are in this commit. The memory addresses line up with the decoding scheme described here.

Wrap-up

BASIC is an interesting language, if only for historical reasons. Classic BASIC feels very clunky by modern standards, but the user experience is not that different to the modern Python REPL. Or at least, it is more similar than you might expect given how far computers have advanced.

I’m running EhBASIC on an 8-bit CPU, where the main alternative is plain 6502 assembly language. BASIC has allowed me to hit the ground running, and gives me access to floating point maths and user-loadable programs, on a computer that does not have an operating system or removable storage.

My home-built computer has a switch for selecting between two ROM’s. My plan is to write my own code in assembly language, but keep this port of EhBASIC in the secondary ROM, so that the computer can always boot to something that works.

Upgrades and improvements to my 6502 computer

Thu, 26 Aug 2021 00:00:00 +0000

For the past few weeks, I’ve been building a 6502-based retro computer, based originally on a tutorial and design by Ben Eater. My first major change was to add a serial port, so that I can write programs which accept text as input.

This blog post is a list of things which I’ve changed since, to try to make a computer which is a bit more suited to my intended use. My aim at the moment is to lock in a simple base system, so that I can use it as permanent platform for 6502 assembly and hardware experiments.

Re-visiting address decoding

The computer has a 32 KiB RAM chip, and a 32 KiB ROM chip. Due to the way that the glue logic was set up, only 16 KiB of the RAM can be used, but the full ROM space is available.

I wanted to flip this around, since I’m planning to run most programs from RAM, with only a loader or interpreter in the ROM. The idea would be to add a switch to select between the upper and lower half of ROM, map the full RAM in, and leave a large space available for I/O and other experiments.

I planned out a coarse, 2-chip address decoder to achieve this.

The resulting memory map is:

Address	Maps to
C000-FFFF	ROM - 16 KiB
A000-BFFF	Not decoded - 8 KiB
8000-9FFF	I/O - 8 KiB
0000-7FFF	RAM

This is partly based on the info in Garth Wilson’s address decoding primer, and also Daryl Richtor’s SBC-2 computer.

I re-wired every chip select, modified my test program to use the new memory map, and also added a switch to select between the two halves of the ROM.

Implementing power-on reset

In Ben Eater’s 6502 computer, the reset line is connected to +5v through a resistor, and grounded when a momentary switch is pressed. At power-on, I needed to press the reset button before the computer would do anything useful.

I removed the pull-up resistor from the reset line, and added a DS1813-5+ supervisory IC instead. This holds the reset line low for 150ms at power-on, or when reset is pressed. This suggestion was from the 6502.org forums.

There was no KiCad symbol for the DS1813-5 in the default library, so I needed to create one.

Interrupt lines

I found that I had made an error when connecting the interrupt lines.

The 6502 has two interrupt inputs: IRQ and NMI. On my 6502 computer, the interrupt output of the 65C22S is connected to the IRQ input of the CPU, while the interrupt output of the 65C51S is connected to the NMI input of the CPU.

The 65C51N uses an open-drain IRQ output, so this configuration left the CPU’s NMI input floating most of the time, which caused many spurious interrupts. This was easily fixed by connecting the NMI line to +5v through a 3.3k resistor.

I also implemented some tips from Garth Wilson’s 6502 primer, and connected RDY and BE to +5v through 3.3k resistors, where they had previously been connected to +5v directly.

Power

Up until this point, I had been powering my computer with a breadboard power supply. I’m hoping to build something permanent, so I designed a circuit to power it through a DC barrel jack.

This takes a DC voltage, and steps it down to 5v through a voltage regulator. The other components are a power switch, plus a diode so that I can avoid damaging anything if I connect a plug with the wrong polarity.

This computer does not draw much current, so it’s not necessary to add a heat-sink on the voltage regulator.

Getting my wires crossed

My next change was to remove the LCD screen. This had been useful for debugging, but I would prefer to free up breadboard space and I/O connections for other parts of this project.

I did this in four steps, testing after each one. First I moved the LCD off to its own breadboard, then updated the computer’s test program to stop using the LCD, ran the computer without the LCD connected, and finally re-located the UART chip into the newly-vacant space.

After the last step, the computer started producing the wrong text (eg. “H” became “D”). I had mixed up two bits in the data bus, and by some luck the UART setup code was not affected. The orange and red wires on the left of this photo are the ones which were swapped.

I was able narrow down the source of the problem quickly, since I was breaking everything down into small steps, which were easy to individually verify.

Next steps

For all this effort, not much has changed. I still just have an 8-bit computer which says “Hello” when it starts. The next part of this project will involve porting across some non-trivial software.

On the hardware side, I’ve made a lot of progress learning how to use KiCad. As long as everything works, my plan is to migrate this design from breadboard to a PCB in the near future.

A first look at programmable logic

Thu, 12 Aug 2021 00:00:00 +0000

I’ve recently been learning a bit about how computers work on a low-level by building a 6502-based retro computer from scratch. I’ve noticed that plenty of retro computer designs use simple programmable logic devices, including GAL’s, PAL’s and PLA’s.

I decided to take a closer look at these, since they may be a useful part of my toolkit. The particular part which grabbed my attention was the ATF22V10 programmable logic device (PLD).

This ticks a few important boxes for my current projects:

Available in DIP packaging, so it is suitable for use on a breadboard.
Operates at 5 volts.
Currently in production.
Has a relatively high pin count (slightly more than the ATF16V8).
Can be programmed using a TL866II+ programmer, which I already use for EEPROM programming.

These devices have been around for many decades, and I found mixed information online about the software and hardware requirements for programming these chips. A lot more has been written about the (discontinued) Lattice 22V10, which has similar capabilities, but a different programming process.

Setting up a programming environment

The logic functions for programming the ATF22V10 are specified using a human-readable hardware description language, which needs to be compiled into a device-specific “fuse map” for programming the device.

The only real choice for the ATF22V10 seems to be an obscure language called CUPL, which can be processed by Atmel/Microchip’s proprietary compiler, WinCUPL. This is distributed as a freeware Windows binary, downloadable from this page on the Microchip website.

I do all of my other software development on Linux, so I installed WinCUPL in a WINE environment. On Ubuntu 20.04, WINE is installed with the following command:

sudo apt-get install wine

For later parts of the process, I found that I also needed winetricks and cabextract:

sudo apt-get install cabextract
wget https://raw.githubusercontent.com/Winetricks/winetricks/master/src/winetricks
chmod +x winetricks

The install was simple:

wine awincupl.exe

I was then launching the program with the following command.

$ wine  ~/.wine/drive_c/Wincupl/WinCupl/Wincupl.exe

On first start-up, it prompted for registration info. This is a carry-over from before this tool was freeware, and the download page lists the correct key as 60008009.

I got persistent errors about missing DLL modules.

I initially thought that this was a library registration or 32/64-bit compatibility issue, though I eventually found that I needed to install mfc40 via winetrics:

$ ./winetricks mfc40

After this installer ran, I was able to launch WinCUPL. The editor was using a a proportional font, though the dialog indicated that it was attempting to use Courier New, which is monospace.

This is fixed by installing the Microsoft fonts via the ttf-mscorefonts-installer package.

sudo apt-get --reinstall install ttf-mscorefonts-installer

At this point I had a working editor. If it was not clear yet that this is ancient technology, this screen capture shows the I/O decoding example which is bundled with WinCUPL, which was apparently written in 1984.

Writing a test program

WinCUPL ships with a folder of example programs, and some documentation which includes a reference manual and tutorial. Most introductory information online seems to be from old electrical or computer engineering courses.

I read through:

The closest thing to a “Hello World” program for one of these chips is the GATES.PLD example, which shows how to apply some basic logic operations. It was written for the ATF16V8, so I looked up my own device to find its mnemonic, g22v10:

After swapping around the pins, I ended up with this program for testing the ATF22V10:

Name     MyTest;
PartNo   00;
Date     6/07/2021;
Revision 01;
Designer Mike;
Company  None;
Assembly None;
Location None;
Device   g22v10;

Pin 1 = a;
Pin 2 = b;

Pin 14 = inva;
Pin 15 = invb;
Pin 16 = and;
Pin 17 = nand;
Pin 18 = or;
Pin 19 = nor;
Pin 20 = xor;
Pin 21 = xnor;

inva = !a;
invb = !b;
and = a & b;
nand = !(a & b);
or = a # b;
nor = !(a # b);
xor = a $ b;
xnor = !(a $ b);

To get a JED file from this, I navigated to compile options, and disabled the “Simulate” and “Absolute” options. This skips the simulation step, which otherwise blocks compilation.

I did not find the simulator very intuitive, but I did return to it later after watching this recent tutorial. The devices can only be reprogrammed 100 times, so for anything complex, I’ll definitely be checking outputs in the simulator rather than relying on trial-and-error.

Programming the chip

I use TL866II+ programmer with the minipro open source programming software for other chips, and I found mixed information online about whether this would work for the ATF22V10.

In my case, it worked, but only after a few attempts. My first mistake was to read the chip instead of writing it.

$ minipro -p ATF22V10CQZ -r MyTest.jed
Found TL866II+ 04.2.86 (0x256)
Warning: Firmware is out of date.
  Expected  04.2.123 (0x27b)
  Found     04.2.86 (0x256)
Reading device...  0.54Sec  OK

When I plugged this into my test circuit (shown later), the blank chip warmed up, and produced erratic output.

After re-generating the JED file, I programmed the chip again, but got some warnings and verification errors. The chip still produced heat and erratic output in my test circuit.

$ minipro -p ATF22V10CQZ -w MyTest.jed
Found TL866II+ 04.2.86 (0x256)
Warning: Firmware is out of date.
  Expected  04.2.123 (0x27b)
  Found     04.2.86 (0x256)

VPP=12V
Warning! JED file doesn't match the selected device!

Declared fuse checksum: 0x7005 Calculated: 0x7005 ... OK
Declared file checksum: 0x6DDD Calculated: 0x6DDD ... OK
JED file parsed OK

Use -P to skip write protect

Erasing... 0.82Sec OK
Writing jedec file...  4.97Sec  OK
Reading device...  0.54Sec  OK
Writing lock bit... 0.26Sec OK
Verification failed at address 0x0006: File=0x00, Device=0x01

Reading back the device produced a completely different JED file (different length, different contents).

Detour: Firmware update

I started digging into the warnings, and decided to attempt a firmware update. The best instructions I could find for this were from this GitLab issue.

The short version is that once you have a firmware file from the hardware vendor, they can be applied by minipro.

$ minipro -F updateII.dat
Found TL866II+ 04.2.86 (0x256)
Warning: Firmware is out of date.
  Expected  04.2.123 (0x27b)
  Found     04.2.86 (0x256)
updateII.dat contains firmware version 4.2.126 (newer)

Do you want to continue with firmware update? y/n:y
Switching to bootloader... failed!

I got this error, probably because the device was restarted. On my setup, I am passing the USB device through to a VM, and needed to press a button to redirect it again.

$ minipro -F updateII.dat 
Found TL866II+ in bootloader mode!
updateII.dat contains firmware version 4.2.126 (newer)

Do you want to continue with firmware update? y/n:y
Erasing... OK
Reflashing... 100%
Resetting device... OK
Reflash... OK

Programming the chip again

With the updated firmware, I made a third attempt to write the chip, using the same JED file as before.

$ minipro -p ATF22V10CQZ -w MyTest.jed
Found TL866II+ 04.2.126 (0x27e)
Warning: Firmware is newer than expected.
  Expected  04.2.123 (0x27b)
  Found     04.2.126 (0x27e)

VPP=12V
Warning! JED file doesn't match the selected device!

Declared fuse checksum: 0x7005 Calculated: 0x7005 ... OK
Declared file checksum: 0x6DDD Calculated: 0x6DDD ... OK
JED file parsed OK

Use -P to skip write protect

Erasing... 0.33Sec OK
Writing jedec file...  5.04Sec  OK
Reading device...  0.41Sec  OK
Writing lock bit... 0.35Sec OK
Verification failed at address 0x16C6: File=0x01, Device=0x00

Reading the device returned a JED file with all 0’s, and there are still some warnings, but the circuit appeared to work correctly.

Alternatives

The firmware update seemed to fix any problems for me, but I have read plenty of accounts of tricky issues when attempting to program the ATF22V10 with the TL866II+ programmer.

An alternative programmer might be the Wellon VP-290 (mentioned here). On the software side, using the ATF22V10 really seems to require WinCUPL, though it does have a command-line.

There is plenty of software designed for the discontinued Lattice 22V10 GAL, though my best guess is that these are not compatible with the Atmel chips I’m using here. These include GALasm, which has a license which prohibits commercial use (enough for me to steer clear), plus an open source re-implementation called galette.

Wrap-up

It’s been interesting to take a look at the lowest-possible level of the software stack, even just to produce a “Hello World” program.

I am not using PLD’s in my current project, but it’s good to know that I could program one if I needed to. The main use for me would be as an alternative to discrete logic chips, so that I can keep parts count, layout space, and propagation delays under control. For prototyping, it will also be helpful to be able to replicate old circuits which use similar components, or to simulate logic chips which aren’t in my inventory.

One thing that I learned is that a fuse is not the same as a gate. The device I used has thousands of fuses (programmable connections) which are capable of configuring the inputs to a few hundred gates. If the temperature of my first two programming attempts is anything to go by, it seems to be perfectly possible to configure the fuses to produce a short circuit.

Adding a serial port to my 6502 computer

Thu, 22 Jul 2021 00:00:00 +0000

In my last blog post, I wrote about the 8-bit computer which I’ve been building, using an existing design by Ben Eater. The I/O capabilities of the original design are rather limited, so one of the first enhancements I’m making is to add a serial port.

Hardware

The main chip I am adding is the WDC 65C51N ACIA, which is a modern version of the MOS 6551. Versions of this chip have been on the market for around 40 years, and lots of classic computer designs use some version of it for serial output. I bought this one new, and the date code indicates that it was manufactured 11 years ago, so it’s a fair guess that they are not selling as fast as they used to.

I also needed a 1.8432 MHz oscillator, which I used to clock both the computer and the UART.

Lastly, I used a USB/UART module to interface with a modern computer. This module hosts a FT232RL chip, and the pins on this one are DTR, RX, TX, VCC, CTS and GND.

Address decoding

The 65C02 CPU in my computer uses memory-mapped I/O, so I needed to fit this new I/O chip into the memory map before I could start using it.

The original design uses a single-chip solution for address decoding, where the select lines for the ROM, RAM, and a 65C22 VIA chip are connected via 3 NAND gates.

This leaves an unused space between address 4000 and address 5FFF.

Address	Maps to
8000-FFFF	ROM
6000-7FFF	I/O - 65C22 VIA
4000-5FFF	Not decoded
0000-3FFF	RAM

The 65C51 ACIA has two chip select inputs: one active-high and one active-low, much the same as the 65C22 VIA. All I needed to do was invert A13, and there was an unused NAND gate in the existing design which I could use for it.

This places the 65C51 in the unused address space. It’s not exactly efficient to assign 8KB of address space to a device which needs 4 bytes, but it does work.

Address	Maps to
8000-FFFF	ROM
6000-7FFF	I/O - 65C22 VIA.
4000-5FFF	UART - 65C51 ACIA
0000-3FFF	RAM

While editing this blog post, I also re-read Garth Wilson’s address decoding guide for the 6502, which shows some alternative schemes for achieving this.

Wiring

I’m using the following wiring between the 65C51 and the USB/UART module.

Note: The the two clock inputs are connected to the 1.8432 MHz oscillator, which is not shown correctly here.

Software

This particular revision of the 6551 has some hardware bugs, though they are well-documented and can be worked around in software. Most of the excellent example code online is aimed at older (less buggy) revisions.

After four attempts, I was able to write an assembly-language program which could produce some output. The information on this 6502.org thread, and this 6502.org comment were the most accurate for my hardware setup.

I’m using this code to set up the ACIA for 8-N-1 communication at 19,200 bytes per second, with no interrupts.

ACIA_RX = $4000
ACIA_TX = $4000
ACIA_STATUS = $4001
ACIA_COMMAND = $4002
ACIA_CONTROL = $4003

reset:
	; ... other stuff
	; ACIA setup
	lda #$00
	sta ACIA_STATUS
	lda #$0b
	sta ACIA_COMMAND
	lda #$1f
	sta ACIA_CONTROL
	; ... other stuff

To send, I needed to add a delay between bytes, since the hardware bug prevents the transmit bit in the status register from operating correctly. I found some code with nested loops, but it only worked after increasing the delay far beyond what should have been necessary. An alternative work-around is to generate a timed interrupt from the 65C22 VIA, which I’m hoping to try later.

; print A register to ACIA
; Based on http://forum.6502.org/viewtopic.php?f=4&t=2543&start=30#p29795
print_char_acia:
  pha
  lda ACIA_STATUS
  pla
  sta ACIA_TX
  jsr delay_6551
  rts

delay_6551:
	phy
	phx
delay_loop:
  ldy #6 ; inflated from numbers in original code.
minidly:
  ldx #$68
delay_1:
  dex
  bne delay_1
  dey
  bne minidly
  plx
  ply
delay_done:
  rts

I am using this routine to receive characters. It will block until the next character is received, and I will most likely need to replace this with something interrupt-driven once I start to add more complex programs.

; hang until we have a character, return it via A register.
recv_char_acia:
  lda ACIA_STATUS
  and #$08
  beq recv_char_acia
  lda ACIA_RX
  rts

I ended up with a program which prints “Hello” to both the LCD and serial port when the computer resets, then accepts text input. Any characters received over serial are then printed back to the terminal, and also to the LCD.

Mistakes were made

Since this is a learning project, I’m keeping a log of mistakes that I’m making along the way. Today’s lesson is to check everything, because it’s very difficult to debug multiple problems at once. When I ran my first test program, there were four faults.

I interfaced the USB/UART module to the 65C51 by matching up pin names, which does not work - RX on one side of the serial connection should go to TX on the other.
I incorrectly calculated the memory map, so the test program was writing to address 8000 while the 65C51 was mapped to address 4000.
I based my code on examples which do not work on this chip revision, because of the hardware bug noted above.
I also miscalculated the baud rate, so even if I didn’t have the other faults, my settings for minicom would not have worked.

Wrap-up

It’s very straightforward to modify Ben Eater’s 6502 computer design by adding a 65C51 ACIA. This upgrade will allow me to write (or port) software which uses text I/O. I’m planning a few more changes to this design before I port anything too serious though.

This is also the first time I’ve included (pieces of) schematics in a blog post. I’m drawing these with KiCad, and using a slightly modified version of Nicholas Parks Young’s 6502 KiCad library, which has saved me a bit of time.

Building a 6502 computer

Thu, 08 Jul 2021 00:00:00 +0000

I’ve been using 6502 assembly for some hobby projects recently, but only testing in an emulator. It’s about time to target some real hardware, so for the past few weeks I’ve been following Ben Eater’s 6502 computer tutorial.

I am a complete beginner when it comes to electronics, so I spent a bit of time making useless circuits to toggle LED’s, then jumped right in to building the simplest possible circuit exercise a 65c02 CPU, known as a NOP generator. I used a 555 timer and an inverter to get a 3Hz clock, which is quite a bit slower than my desktop PC.

I then extended the circuit to run NOP instructions from a ROM. I generated a ROM filled with the 6502 NOP instruction by printing a character to a file, then concatenating the file to itself times to fill the ROM.

$ printf '\xEA' > rom-original.bin
$ dd if=rom-original.bin of=rom-original.bin bs=1 seek=1 count=32768

I am using an open-source tool called minipro with a TL866II+ programmer to burn the ROM.

$ minipro -p AT28C256 -w rom-original.bin
Found TL866II+ 04.2.86 (0x256)
Warning: Firmware is out of date.
  Expected  04.2.123 (0x27b)
  Found     04.2.86 (0x256)
Erasing... 0.02Sec OK
Protect off...OK
Writing Code...  7.44Sec  OK
Reading Code...  0.77Sec  OK
Verification OK
Protect on...OK

Next, I tried to extend the circuit to blink some LED’s based on a programmed sequence. When I ran the program, the 65c22 I/O chip warmed up, and my row of LED lights did not blink. It turns out that I had mixed up the meaning of VCC and VSS, and applied a reverse voltage to the chip. I found a post from somebody else who had made the same mistake, and corrected it before the chip was damaged.

This program worked well initially, but the computer would sometimes crash when running at this slow speed, so I started running it with a 1MHz or 1.8MHz oscillator instead. I now know that this is because I had plugged the 65c02 and 65c22 clock inputs into the 555 timer output, when I should have been running it through the inverter first. The rising-edge and falling-edge times of the 555 timer are apparently not fast enough to clock these chips reliably.

The next step was to add an LCD and some RAM. My first attempt did not work, and it took me a few hours of troubleshooting to rule out any hardware problems. In the end it was a simple programming error, where I had used a jmp instruction instead of jsr in my test program.

So it’s not much, but it works! Based on some of the problems that I had while building this, it was definitely a good idea to start with a known-good design on a breadboard.

I’ve got a few ideas (and components) for extending this computer already, and I’m hoping to learn a thing or two about hardware and software along the way.

IntelliJ plugin for 6502 assembly language

Thu, 27 May 2021 00:00:00 +0000

I’ve been writing some 6502 assembly code in the past year, and have found the developer experience for this language to be lacking some modern conveniences.

In my last blog post, I described the development tools that I used to implement a simple NES game on Linux. I used text editor to write the code, and it couldn’t do much more than syntax highlighting.

I write most of my other code in IntelliJ, so I decided to take a look at what would be required to write a plugin for 6502 assembly support. I managed to together something that mostly works, which I published to the JetBrains Marketplace last weekend.

You can find it by searching “6502 Assembly” in most JetBrains IDE’s.

Features

Firstly, it supports syntax highlighting. I limited the scope of the plugin to ca65 assembler syntax only, since it’s the assembler that I know best.

You can navigate to any label with Ctrl+Click. This will not yet work for other types of symbols, such as constants, macros, and imports.

If the plugin sees a jump or branch statement, and can figure out where the jump goes, then it will show a gutter icon which navigates to the target.

You can find the usages of a label.

In ca65, you can use nested scopes. The plugin shows code folding buttons to collapse these scopes.

You can navigate to a symbol by name using the “Go To Symbol” action.

The plugin allows for commenting and un-commenting blocks of code, though there is no formatter, so indentation sometimes still has to be fixed manually.

It also allows you to rename a label and its usages, which is a great time-saver.

Limitations

One limitation is that the plugin does not fully parse expressions, so it will not detect errors from mis-matched brackets.

Secondly, the plugin does not understand the project structure. This means that if you re-use the same label name in different places in your project, the “Rename” and “Find Usages” function will match all of them at once, because it is not smart enough to follow imports and apply scope rules.

Future improvements

A lot of things can be done with an IDE plugin, but I’m planning to use this initial implementation for a while before attempting to add any big-ticket features.

A reasonable goal might be to have “Hello World” project skeletons for common 6502 systems, and to support launching with some common emulators. I would like to be able to set breakpoints and debug 6502 programs in an IDE, but external debugging interfaces are not commonly available for retro emulators, so this is probably not a reasonable goal.

The code is on GitHub under an MIT license. If you are using this plugin and would like to help improve it, then pull requests are welcome.

Building my first NES game: A retrospective

Thu, 22 Apr 2021 00:00:00 +0000

Last year, I spent a fair amount of time learning how to make games for the Nintendo Entertainment System. The homebrew scene for this system is very much alive, and I was quite proud that I was able to make a simple, working video game which runs on a NES emulator.

So, I present to you: 8-bit Table Tennis.

This blog post just a few notes about the tools and resources that I used for NES development on Linux, plus a few things that I learned along the way.

Development tools

The CPU in the NES is a derivative of the once-ubiquitous MOS 6502, and games for it are mostly written in 6502 assembly. I already do a fair amount of programming, but quickly found that my usual editors, compilers and debuggers were useless for this platform.

The main resources I used were:

The Nesdev Wiki for information about the platform.
The 6502.org tutorials list.
This example project by Brad Smith, which I used as a base.
This instruction set reference - many similar references are available online.

To build my code, I used the cc65 toolchain, which includes a cross-assembler. I did most of my testing in the Nestopia emulator, and most of my programming in Gedit. All three of these are available in the official Debian repositories.

Gedit setup

Out of the box, Gedit can’t syntax highlight 6502 assembly. I found a language spec file for it on the 6502.org forums, and edited it slightly before using it.

At the time of writing, new language specs can be installed on Debian like this:

sudo cp asm6502.lang /usr/share/gtksourceview-4/language-specs/asm6502.lang && sudo chmod 0644 /usr/share/gtksourceview-4/language-specs/asm6502.lang

It is possible to use plugins to add a console and git support to gedit, but this is still a long way from a full IDE.

Geany

Geany is a programming text editor, and is also available in the official Debian repositories.

I wasn’t able to get it to recognise 6502 assembly, but it does have good x86 assembly (NASM) support, which is similar enough for it to allow navigation through the source code using labels.

Geany can also set up projects with build scripts (such as a Makefile), which could allow for faster testing.

FCEUX

FCEUX is a NES emulator which has an in-built debugger, and it runs well under WINE. I was also able to build and run it natively on Linux, but the debugger is only present in the Windows build.

I also briefly experimented with using the FCEUX Lua interface to pause/resume the emulator over a TCP socket for debugging, since that interface is available in the Linux build. I could only pause the emulator between frames, so I decided to abandon this.

Graphics

I created the graphics in the GNU Image Manipulation Program, and manually broke the title page down into tiles.

I also wrote a custom program to convert a 4-colour PNG file into the native NES CHR graphics format. I run this conversion as part of the build process, so that the graphics files can be stored in a modern format.

Things I learned

Programming for the NES is quite simple once the project is up and running, but even simple tasks can be a real hassle. I spent more time than I expected on mundane tasks such as collision detection, and certainly did a poor job of implementing game physics. The 6502 CPU has no built-in way to multiply, divide, or perform floating point operations. The version used for the NES additionally lacks any way to perform a binary-to-decimal conversion, which would have been very useful for displaying the player scores! In hindsight, I would have been able to improve the physics by pre-computing some lookup tables.

On the other hand, I expected it to be difficult to work within 2KiB of RAM, but due to the simplicity of the game, I only used around 25% of it, and had plenty of CHR ROM space leftover as well.

It took me three weekends to make this project, without having ever written a line of 6502 assembly before. I think that development would be a lot faster if I were able to set breakpoints from my code editor, so I will definitely be looking at other debugging emulators before attempting my next 6502 assembly project.

Conclusion

8-bit Table Tennis is available as an iNES ROM on GitHub.

I have only been running it with an emulator, so if any readers of this blog own a flash cartridge for the NES (such as an Everdrive), then please let me know if it works on the real hardware!

Building a tiny Linux gaming PC

Sun, 31 Jan 2021 00:00:00 +0000

I recently put together a small form factor PC to use with my TV, with the aim of building a Linux-based, console-like gaming setup. This mostly went to plan, and is certainly a big upgrade from my previous hardware, which was based on a retired desktop computer.

I am writing a bit about it today, partly because it’s been an interesting project, but also to show a working setup for anybody who is attempting something similar.

Quick reference

The build is based around these components (PC part picker list here):

In Win Chopin case
AMD Ryzen 5 3400G CPU
Gigabyte B450 I AORUS PRO Motherboard
Noctua NH-L9a-AM4 Cooler
Corsair Vengeance LPX 16 GB (2x8GB) DDR4-3200 RAM
Samsung 970 Evo 1 TB

Peripherals:

Sony DualShock 4 controller - Works over Bluetooth and can be used as a touchpad.
8BitDo SF30 Pro controller - Also works over Bluetooth, and can emulate an Xbox controller.
Logitech K400R wireless keyboard/mouse

Hardware setup

This is the smallest PC I’ve built with desktop parts, and there is not a lot of spare space in the case. I first put everything together on my desk confirm that it would POST, then disassembled the case to make some modifications.

Three screws hold in the power supply, two torx screws hold the aluminium shell to the chassis, and two Phillips-head screws and some tabs hold the plastic front panel cover to the chassis.

The motherboard almost fills the case, and a power supply has to fit in there as well.

My plan was to route the power cables around the back of the case rather than leaving them in the main cavity. This involved cutting out a square near one of the drive trays for all of the power cables to exit, and another for the ATX power connector to connect to the motherboard. I also removed a metal tab from the power supply, since I had cut out the metal that it was supposed to be screwed to.

There was an existing hole which I could use for the CPU power connector, which was already the correct size.

I then swapped the power supply fan for a Noctua A4x10 PWM so that I could control the fan speed from software. I do not recommend this, since it is unnecessary, and opening a power supply is an electrocution hazard.

Next, I got all of the cables into place, since there is no space to do this after the motherboard is installed

Once the power cables were in the right place, I added the motherboard, then connected the front panel I/O, feeding all of the excess cables to the space behind the case. This is a very tight fit, and I accidentally bent the case and squashed some cables before finally getting the plastic front panel to attach.

After finally lining everything up, I checked that the case closed, and re-attached the shell.

Lastly, I was able to re-fit one of the two drive trays, in case that is ever needed in future.

BIOS setup

This build uses integrated graphics, so the system memory is also being used as video memory. This means that RAM speed is more important than usual.

I enabled XMP to run the RAM at its rated 3200 MHz speed. This is higher than the highest officially supported speed of 2933 MHz for this CPU, but for me that was not an issue.

I set the built-in motherboard LED’s to blue, to match the default colour on the Sony DualShock 4 controller. This also meant that I would not need to get any RGB software working on Linux.

Some online sources suggest disabling the AMD Cool & Quiet function to get extra gaming performance, but I consider this advice to be out-dated, unless you’re overclocking. Instead, I am leaving this setting enabled, then automatically setting the CPU to ‘performance’ mode while gaming via software.

Once I got everything working, I disabled everything that I wasn’t using, and enabled Ultra Fast Boot. This makes the system start faster, but also removes the ability to re-configure the BIOS, because the keyboard will not work. The settings can be reset via a jumper on the motherboard if I need to get back into it.

Software setup

I installed Ubuntu 20.04 LTS. I would suggest sticking with the Long Term Support version of Ubuntu if you are using it for gaming, since Steam is available in the package manager. Bluetooth, Wi-Fi, sound and 3D-accelerated graphics all worked out of the box.

Configuring Ubuntu

These settings make Ubuntu act less like a desktop, and more like a Home Theater PC.

First I enabled automatic login, then cleared the password for the login keyring. This avoids needing to use a keyboard on startup, or getting prompted for a password when web browsers attempt to access the keyring.

Under power settings, I set “Blank Screen” to “Never”, and Automatic Suspend to “Off”.

Under screen lock settings, I set “Blank Screen Delay” to “Never”, and disabled “Automatic Screen Lock”.

Next, I set the theme to dark mode, the display output to 1920x1080 60Hz, and the audio output device to HDMI. The display is 4K, but compatibility will be challenging enough without adding scaling issues in there, so I’m using a lower resolution for now.

Lastly, I set up a tool called psensor to start on boot, so that I could check temperatures.

Installing applications

Linux gaming has advanced a fair bit recently, and since it is a large topic, I won’t try to cover in too much detail here. I’m connecting this computer up to a TV, so this type of setup is only suitable for games with controller support. Still, this includes:

Native Linux games.
Windows games running via a compatibility layer (WINE/DXVK or Proton).
Games for other systems played through an emulator.

An extensive catalogue is available through the Linux version of Steam, which I installed via the Ubuntu package manager. I could have set Steam to launch on startup and call it a day, but there are two other launchers which I installed alongside it:

Lutris (from the lutris-team/lutris PPA).
Retroarch (from the libretro/testing PPA)

Next, I installed WINE (32 bit and 64 bit), winetricks, and gamemode from the Ubuntu package manager, then Proton GE from GitHub. Proton GE is a widely-used Proton fork, which integrates fixes from upstream WINE. This gave me several different run-times for Windows applications, and simply switching between them or adding some environment variables has been sufficient to work around every compatibility issue that I’ve encountered so-far.

Lastly, I installed Kodi from the team-xbmc/ppa PPA.

Game testing

For this section, I’m listing out a few games from my library that I’ve tested. As a quick reminder, this is all running on integrated graphics, on Linux.

I was mainly checking that I could get a (subjectively) playable frame-rate, and that I could get two controllers to work in local multiplayer where available.

Steam

GRIP combat racing runs via Proton. It is a fast-paced racing game, has local split-screen multiplayer, and runs very smoothly at 720p.

Untitled Goose Game also runs via Proton. It has recently added a 2-player mode, and runs at 1080p with no hiccups.

The Tomb Raider reboot is available native for Linux. With low settings, it runs fine at 1080p, with the benchmark indicating a 99 FPS average.

Shadow of the Tomb Raider, also available native for Linux, is a real challenge for this integrated GPU. I am running it at 1080p, lowest settings, though the benchmark comes back with just a 43 FPS average.

Lutris

Lutris has an Epic Games store installer, which runs the store-front under WINE. I decided to try it out, though it is easily the most buggy software mentioned in this blog post.

The only game I tested from there was Rocket League. Native Linux support was recently dropped from this title on Steam, but the Windows version from Epic runs just fine on integrated graphics, and has local multiplayer. I run it at 1080p.

I also tested the original Crysis by manually setting up a WINE prefix and adding it to Lutris. I run this at 720p, medium settings. After a few loops, the benchmark indicates that this averages 112 FPS, so I could probably increase the quality or resolution.

I also tested Crysis 2 via WINE, which had some concerns about my graphics card. I also run it at 720p, using the ‘Gamer’ profile. The benchmark indicates that this averages 60 FPS.

Super Tux Kart is an open source racing game, which can be installed via Lutris. This is a native Linux build, and runs just fine at 1080p. It has controller support and local multiplayer.

Wrap-up

This blog post touches on quite a few topics, so I’ve left out a lot of the details. If you’ve read this far, though, then it’s time to talk about down-sides. Some things did not work as planned.

Reading/controlling fan speeds did not work from Linux on this Gigabyte motherboard. This has worked so consistently for me on other hardware that I did not even think to check for compatibility here. This means I can only set fan speeds in the BIOS.
The power supply is noisier than I expected under load. This is due to coil whine, not fan noise.
When using the 8BitDo SF30 controller over Bluetooth, RetroArch would pause for 10 or 15 seconds at a time. I tracked this down to the fact that I am connecting it as an Xbox controller, and RetroArch was attempting (and failing) to check its battery level. Connecting the controller after a core has launched avoids the problem.
I had hoped to run everything under Wayland, but the Epic Games store had terrible graphical glitches. Everything else on this page (Steam, Lutris, Retroarch) worked on Wayland, and this is apparently due to the way that this app uses OpenGL.

This setup works very well for me, and I am glad to be able to show that it’s possible to do some basic gaming without Microsoft Windows or a dedicated GPU. Still, there are a lot of trade-offs that come from this form-factor and platform, and that’s not for everybody.

How to auto-scale the display in GNOME Boxes

Thu, 22 Aug 2019 00:00:00 +0000

I recently installed a virtual machine in GNOME Boxes, and the display was stuck at 1024x768.

The type of display used here is called SPICE, and it includes a channel for auto-scaling. The guest simply needs the agent to be installed.

In this case, I’m running a Debian guest, which means that I must have forgotten to install the spice-vdagent package.

# apt-get install spice-vdagent
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  spice-vdagent
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 47.6 kB of archives.
After this operation, 174 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 spice-vdagent amd64 0.18.0-1 [47.6 kB]
Fetched 47.6 kB in 0s (99.9 kB/s)  
Selecting previously unselected package spice-vdagent.
(Reading database ... 132855 files and directories currently installed.)
Preparing to unpack .../spice-vdagent_0.18.0-1_amd64.deb ...
Unpacking spice-vdagent (0.18.0-1) ...
Setting up spice-vdagent (0.18.0-1) ...
Created symlink /etc/systemd/system/sockets.target.wants/spice-vdagentd.socket → /lib/systemd/system/spice-vdagentd.socket.
[spice-vdagentd.conf:2] Line references path below legacy directory /var/run/, updating /var/run/spice-vdagentd → /run/spice-vdagentd; please update the tmpfiles.d/ drop-in file accordingly.
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for systemd (241-5) ...

The easiest way to ensure that everything is running correctly is to reboot the guest, since the agent will start on boot, and this also forces a new log-in, as well as a new connection to the display.

# reboot

Result

Assuming that you are otherwise on the default settings, the display in the guest VM will now automatically adjust as you resize the window.

If the “Share Clipboard” setting is enabled for the virtual machine, then spice-vdagent will also enable you to copy & paste text between the host and guest.

How to use the qemu-bridge-helper on Debian 10

Thu, 08 Aug 2019 00:00:00 +0000

If you use the libvirt virtualisation libraries, then you will be familiar with the “user session”. This feature lets you provision virtual machines to run under a regular, unprivileged user account.

The user session is used by GNOME Boxes, and can also be managed from Virtual Machine Manager.

The main downside to this setup is that a regular user can only access a very limited range of networking options. The last time that I mentioned this in a blog post, a reader pointed out that you can actually use qemu-bridge-helper to provide bridged networking to unprivileged virtual machines.

Today I finally tried this out, and it worked really well. With a bit of configuration, you can extend proper networking to this type of VM.

The host

I’m running a graphical Debian 10 desktop, with a few basic virtualisation packages.

gnome-boxes for creating VM’s as a local user. This depends on libvirt-daemon, which is enough to host VM’s on the system.
virt-manager for a more advanced graphical interface.

The tool that I’m writing about today is qemu-bridge-helper, which is in the qemu-system-common package.

After installation, you will also need to ensure that libvirtd is running.

$ systemctl enable libvirtd.service
$ systemctl start libvirtd.service

Set up a bridge

Libvirt ships with a basic network bridge configuration, you just need to enable it.

Command-line method

Start the default network bridge, and configure it to run on startup.

$ sudo virsh net-autostart --network default
$ sudo virsh net-start --network default

Once this is set up, you should see the bridge virbr0, reporting the IP range 192.168.122.1/24.

$ ip addr show virbr0
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

Graphical method

First, open up Virtual Machine Manager, and authenticate. Right click on QEMU/KVM, and select Details.

Under Virtual Networks → default → Autostart, check On Boot, then click Apply.

Setting up qemu-bridge-helper

Create the file /etc/qemu/bridge.conf with the content:

allow virbr0

Restrict the permissions of this file to make sure it can’t be edited by regular users.

# chown root:root /etc/qemu/bridge.conf
# chmod 0640 /etc/qemu/bridge.conf

Add setuid to the qemu-bridge-helper binary.

# chmod u+s /usr/lib/qemu/qemu-bridge-helper

If you do not correctly set this last step, then you will receive the following error when you attempt to connect a VM to the bridge:

Error starting domain: internal error: /usr/lib/qemu/qemu-bridge-helper --use-vnet --br=virbr0 --fd=28: failed to communicate with bridge helper: Transport endpoint is not connected
stderr=failed to create tun device: Operation not permitted

Setting up the VM

Create a virtual machine, either though GNOME Boxes or Virtual Machine Manager. I am using a CentOS VM as an example here, but the guest platform is not particularly important.

Using Virtual Machine Manage, change the network card to the “shared network” virbr0.

The graphical configuration above is equivalent to the following libvirt domain XML, as below.

<interface type='bridge'>
  <mac address='52:54:00:08:5a:7c'/>
  <source bridge='virbr0'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

Result

After restarting the network interface in the guest, I was able to ping the the guest from the host and vice versa.

This is a significant improvement from “user-mode” networking, which does not facilitate host-to-guest and guest-to-guest communication.

The default virbr0 bridge uses an internal subnet, so the guest here is still inaccessible from the wider LAN. If this doesn’t match your setup, then you can use the same technique to connect unprivileged virtual machines to another bridge of your choice.

Benchmarking PHP code with PhpBench

Thu, 04 Jul 2019 00:00:00 +0000

This blog post is all about measuring the speed of PHP code through micro-benchmarks.

Why micro-benchmarks

I think of micro-benchmarks as a complement to tests. A well-written test would give a developer a definite pass or fail result, while well-written micro-benchmark will give the developer an indication of the duration of an operation.

To put it another way, tests can help you find out whether the code is behaving correctly, while micro-benchmarks can help you to track whether your changes are making the code faster or slower.

Make sure that your code is doing real processing work before you spend time on this: Algorithms such as sorting, compressing, or parsing are good candidates. My guess is that most of the code in a regular PHP application would not benefit significantly from having a suite of benchmarks, because web apps tend to be I/O bound (eg database, network and disk access).

Introducing PhpBench

I have previously written a standalone test script every time I’ve needed to measure the speed of something in PHP. This works up to a point, but it gets quite hard to maintain as the number of benchmarks increases.

PhpBench is one of the available tools for running micro-benchmarks from PHP, and there are a few reasons why I think it’s worth a try:

it’s actively maintained
it installs with composer
it runs in a similar way to PHPUnit, so the benchmarks are fully specified in PHP code and run with a PHP tool.
it implements familiar concepts that you would find in benchmarking tools from other languages (eg. JMH from the Java world).

Installation

Start out with a blank composer project.

$ composer init

Add some auto-loading settings to composer.yml that PHP knows to find ExampleApp classes in the src folder.

{
    "name": "mike42/php-benchmark-examples",
    "description": "Example PHP benchmark project",
    "type": "project",
    "license": "MIT",
    "minimum-stability": "dev",
    "require": {},
    "require-dev": {},
    "autoload": {
        "psr-4": {
            "ExampleApp\\": "src"
        }
    }
}

At this point, install the phpbench dev version.

$ composer require --dev phpbench/phpbench:@dev
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 22 installs, 0 updates, 0 removals
  - Installing symfony/process (4.4.x-dev 1a42849): Cloning 1a42849a7f from cache
  - Installing symfony/options-resolver (4.4.x-dev 94cbb72): Cloning 94cbb72bb9 from cache
  - Installing symfony/finder (4.4.x-dev 1b5ec12): Cloning 1b5ec12340 from cache
  - Installing symfony/polyfill-ctype (dev-master 82ebae0): Cloning 82ebae0220 from cache
  - Installing symfony/filesystem (4.4.x-dev 5914824): Cloning 59148241f7 from cache
  - Installing psr/log (dev-master c4421fc): Cloning c4421fcac1 from cache
  - Installing symfony/debug (4.4.x-dev 8278839): Cloning 8278839457 from cache
  - Installing symfony/service-contracts (dev-master 0c81a04): Cloning 0c81a04f68 from cache
  - Installing symfony/polyfill-php73 (dev-master d1fb4ab): Cloning d1fb4abcc0 from cache
  - Installing symfony/polyfill-mbstring (dev-master fe5e94c): Cloning fe5e94c604 from cache
  - Installing symfony/console (4.4.x-dev e2fe100): Cloning e2fe1002fd from cache
  - Installing lstrojny/functional-php (1.9.0): Loading from cache
  - Installing beberlei/assert (v3.x-dev ce139b6): Cloning ce139b6bf8 from cache
  - Installing seld/jsonlint (1.7.1): Loading from cache
  - Installing psr/container (dev-master 014d250): Cloning 014d250dae from cache
  - Installing phpbench/container (1.2): Loading from cache
  - Installing webmozart/assert (1.4.0): Loading from cache
  - Installing webmozart/path-util (dev-master 95a8f7a): Cloning 95a8f7ad15 from cache
  - Installing phpbench/dom (0.2.0): Loading from cache
  - Installing doctrine/lexer (dev-master ee614dd): Cloning ee614dd93a from cache
  - Installing doctrine/annotations (1.7.x-dev 3f35255): Cloning 3f35255290 from cache
  - Installing phpbench/phpbench (dev-master dccc67d): Cloning dccc67dd52 from cache
symfony/service-contracts suggests installing symfony/service-implementation
symfony/console suggests installing symfony/event-dispatcher
symfony/console suggests installing symfony/lock
Writing lock file
Generating autoload files

Writing a benchmark

Add some code to src/ExampleThing.php to measure.

<?php

namespace ExampleApp;

class ExampleThing {
  /**
   * Inefficiently multiply numbers together
   */
  public function multiply(int $x, int $y) : int {
    $ret = 0;
    for($i = 0; $i < $x; $i++) {
      for($j = 0; $j < $y; $j++) {
        $ret++;
      }
    }
    return $ret;
  }
}

Write a benchmark for the code in benchmarks/ExampleThingBenchmark.php

<?php

use ExampleApp\ExampleThing;

/**
 * @BeforeMethods({"init"})
 * @Revs(1000)
 * @Iterations(5)
 */
class ExampleThingBenchmark {

    private static $exampleThing;

    public function init()
    {
        self::$exampleThing = new ExampleThing();
    }

    /**
     * @Subject
     */
    public function doMultiply()
    {
        self::$exampleThing -> multiply(100, 100);
    }
}

Before you run anything, you will also need a configuration file. Add this minimal configuration to phpbench.json.dist.

{
    "php_disable_ini": true,
    "bootstrap": "vendor/autoload.php",
    "path": "benchmark",
    "php_config": {
        "extension": [ "json.so" ]
    },
    "time_unit": "milliseconds"
}

Running the benchmark

The most basic way to run all benchmarks at once is phpbench run, which looks like this:

$ php vendor/bin/phpbench run
PhpBench @git_tag@. Running benchmarks.
Using configuration file: /home/mike/workspace/blog/phpbench/php-benchmarks/phpbench.json.dist

\ExampleThingBenchmark

    doMultiply..............................I4 [μ Mo]/r: 0.232 0.232 (ms) [μSD μRSD]/r: 0.000ms 0.10%

1 subjects, 5 iterations, 1,000 revs, 0 rejects, 0 failures, 0 warnings
(best [mean mode] worst) = 0.232 [0.232 0.232] 0.233 (ms)
⅀T: 1.162ms μSD/r 0.000ms μRSD/r: 0.100%

As you might expect, there are options to run specific benchmarks, or to present the results in different ways. However, there were some PhpBench-specific things which you will need to navigate to use it successfully.

Firstly, there are two different output settings to consider:

report options allow you to decide which data to print in a table.
output options allow you to decide how to format it for output

I also found it useful to add --progress=none to suppress the text displayed by default, since you will otherwise get broken HTML output.

Lastly, be aware that confusingly, the default report is not active by default. Specify it to get a table listing out each iteration.

$ php vendor/bin/phpbench run --progress=none --report=default
suite: 13415431dc0db41c12decee0fa83c26d1db0f678, date: 2019-05-31, stime: 22:01:34
+-----------------------+------------+-----+------+------+----------+-----------+--------------+----------------+
| benchmark             | subject    | set | revs | iter | mem_peak | time_rev  | comp_z_value | comp_deviation |
+-----------------------+------------+-----+------+------+----------+-----------+--------------+----------------+
| ExampleThingBenchmark | doMultiply | 0   | 1000 | 0    | 915,736b | 236.716μs | +1.68σ       | +1.08%         |
| ExampleThingBenchmark | doMultiply | 0   | 1000 | 1    | 915,736b | 231.999μs | -1.45σ       | -0.93%         |
| ExampleThingBenchmark | doMultiply | 0   | 1000 | 2    | 915,736b | 233.960μs | -0.15σ       | -0.1%          |
| ExampleThingBenchmark | doMultiply | 0   | 1000 | 3    | 915,736b | 233.878μs | -0.2σ        | -0.13%         |
| ExampleThingBenchmark | doMultiply | 0   | 1000 | 4    | 915,736b | 234.361μs | +0.12σ       | +0.08%         |
+-----------------------+------------+-----+------+------+----------+-----------+--------------+----------------+

The type of report that I found most useful is aggregate, since it provides a summary of each iteration.

$ php vendor/bin/phpbench run --progress=none --report=aggregate
suite: 1341543c02ebee4031d165665c2724c379bf9c98, date: 2019-05-31, stime: 22:01:18
+-----------------------+------------+-----+------+-----+----------+-----------+-----------+-----------+-----------+---------+--------+-------+
| benchmark             | subject    | set | revs | its | mem_peak | best      | mean      | mode      | worst     | stdev   | rstdev | diff  |
+-----------------------+------------+-----+------+-----+----------+-----------+-----------+-----------+-----------+---------+--------+-------+
| ExampleThingBenchmark | doMultiply | 0   | 1000 | 5   | 915,736b | 231.907μs | 232.614μs | 232.089μs | 233.538μs | 0.713μs | 0.31%  | 1.00x |
+-----------------------+------------+-----+------+-----+----------+-----------+-----------+-----------+-----------+---------+--------+-------+

Both of the example above use console output, which is the default.

HTML reports

To capture phpbench output from a CI environment, you can generate a HTML report as well. Add an output section to phpbench.json.dist.

{
    ...
    "outputs": {
         "html_file": {
             "extends": "html",
             "file": "benchmarks.html",
             "title": "Example benchmark report"
         }
    }
}

This new html_file output can be used with --output=html_file

$ php vendor/bin/phpbench run \
    --progress=none --report=aggregate \
    --output=console --output=html_file

From Jenkins

This is a Jenkinsfile that I use to run benchmarks. You need to install the “HTML Publisher” and “AnsiColor” Jenkins plugins.

pipeline {
  agent any

  stages {
    stage('Install') {
      steps {
        ansiColor('xterm') {
          sh 'composer install'
        }
      }
    }
    stage('Run benchmarks') {
      steps {
        ansiColor('xterm') {
          sh 'php vendor/bin/phpbench run --progress=none --report aggregate --output=console --output=html_file --ansi'
          publishHTML([
            allowMissing: false,
            alwaysLinkToLastBuild: true,
            keepAll: true,
            reportDir: '',
            reportFiles: 'benchmarks.html',
            reportName:
            'phpbench report',
            reportTitles: ''
          ])       
        }
      }
    }
  }
}

Each build is then benchmarked, and the results are available as a link on the left.

The console output of the build shows the results.

To actually view the HTML report you will need to apply this tweak.

Next steps

I’ve started to add phpbench micro-benchmarks to some PHP projects that I work on, and I’m finding it a lot more useful than standalone test scripts.

The next step would be to integrate micro-benchmarks better into the development process. For example, I can already review code coverage improvements/regressions of a change on GitHub using Coveralls, which posts a comment to each pull request. This saves you from merging changes which don’t meet your project’s standards.

As far as I can tell, you would need to write custom code for more expressive phpbench reports along these lines, such as:

the history of a benchmark over time, or
the most-changed benchmarks against a baseline

phpbench does provide the building-blocks though, since you can set some options to archive the results of each run to a folder.

How to integrate Gitea and Jenkins

Thu, 30 May 2019 00:00:00 +0000

Gitea is a web app for hosting Git repositories. It’s open source, and very simple to get running. With some extra setup, it can also trigger Jenkins builds, and display the Jenkins build status of each commit once it has been built.

Because the documentation for the Jenkins plugin is very minimalist, I decided to write about it for future reference.

About this setup

I installed Jenkins and Gitea on the same Debian 9 server on the LAN. They communicate only over HTTP, so they could just as easily be installed separately.

To make the configuration clear, I’ve used jenkins.example.com in URLs which refer to Jenkins, and gitea.example.com for the Gitea.

Gitea installation

This command will install and start the linux-amd64 version of Gitea as the user “git”.

useradd git -r --create-home && \
  mkdir /opt/gitea && chown -R git: /opt/gitea && \
  wget -O /opt/gitea/gitea https://dl.gitea.io/gitea/1.7.0/gitea-1.7.0-linux-amd64 && \
  chmod +x /opt/gitea/gitea && \
  sudo -u git bash -c "cd /opt/gitea && ./gitea web"

Shut it down, and configure some paths at /opt/gitea/custom/conf/app.ini. These will depend on your environment.

SSH_DOMAIN       = gitea.example.com
DOMAIN           = gitea.example.com
HTTP_PORT        = 3000
ROOT_URL         = http://gitea.example.com:3000/

Start it back up as a systemd service at this point, by creating /etc/systemd/system/gitea.service with this content:

[Unit]
Description=gitea
After=network.target

[Service]
ExecStart=/opt/gitea/gitea web
WorkingDirectory=/opt/gitea
User=git
Type=simple

[Install]
WantedBy=multi-user.target

Once this is saved, start the service.

systemctl daemon-reload
systemctl start gitea

Optionally, also configure it to start on boot.

systemctl enable gitea

Jenkins installation

I installed Jenkins from the official Debian repo at jenkins-ci.org, and clicked through the initial install.

Add plugin

Open up Manage Jenkins → Manage Plugins. Navigate to Available and check the Gitea plugin.

Next, install the plugin and restart Jenkins.

The configuration for the plugin is located under Manage Jenkins → Configure System.

At this point you will want to tell Jenkins where to find your Gitea server.

I don’t suggest choosing Manage hooks, because it uses the same account to manage hooks across all repos, which would violate the principle of least privilege.

Set up a project in Gitea

In Gitea, create a project, then a repository under that.

Log out, log back in as yourself, and add Jenkins as a collaborator to the repo, with Write access.

This is the only permission you need for public repositories. If you plan to lock down your Gitea organization later, then you will also need to give this Jenkins account Read access at the organization level.

Set up a project in Jenkins

Add a new Gitea Organization Jenkins job.

Enter the name of the organization, and the account to log in with.

Add the details for the new account, and make sure it’s selected.

The other options don’t need to be changed at this stage.

When you press ‘save’, Jenkins will immediately attempt to find any repositories in the Gitea organization, and kick off any builds. Unless everything is correct, this is unlikely to work the first time, so pay attention to error logs.

These three places will show what’s happening:

Scan Gitea Organization Log, which lists repositories in the organization.
Scan Multibranch Pipeline Log for each repository, which shows the discovery of branches.
Console output for each build, which will show errors if the build status could not be submitted.

Problems which I’ve found here include:

The URL of Gitea in the Jenkins configuration must match the URL to Gitea in its own configuration.
The Jenkins user account must have permission to list repositories, clone, and update statuses.
Empty repositories, and repositories without a ‘Jenkinsfile’ are ignored.

For that last step, here is an empty Jenkinsfile that you can put in your repository to test this integration:

pipeline {
    agent any

    stages {
        stage('Do nothing') {
            steps {
                sh '/bin/true'
            }
        }
    }
}

Once this is sorted out, you can expect to see your repository in Jenkins.

Every branch with a Jenkinsfile will appear.

And each time a commit is mentioned in Gitea, it will display a small icon to indicate the build status.

Set up a web hook in Gitea

At this point, builds need to be manually triggered. To trigger them each time the repository changes, we need to get a notification out to Jenkins.

Under the repository settings, click Webhooks → Add webhook → Gitea.

The correct values to use are:

URL: http://[ your jenkins server ]/gitea-webhook/post
POST Content Type: application/json

Once you press Add Webhook, the path will appear with a small grey dot, indicating that it hasn’t been run before.

If you click edit, then the Test Delivery button can be used to check that it’s working.

The icon indicates the status. If things aren’t working correctly, then click the delivery UUID to expand the full request information, which should help with debugging.

Final result

With Jenkins and Gitea, you have a simple self-hosted a continuous integration environment.

In Gitea, you can store, update and review your code. Any build and test steps in a Jenkinsfile will be run automatically each time the repository changes.

The detailed output for each build is visible in Jenkins, where you can track build results with a variety of plugins.

Handling I/O errors in PHP

Thu, 07 Mar 2019 00:00:00 +0000

This blog post is all about how to handle errors from the PHP file_get_contents function, and others which work like it.

The file_get_contents function will read the contents of a file into a string. For example:

<?php
$text = file_get_contents("hello.txt");
echo $text;

You can try this out on the command-line like so:

$ echo "hello" > hello.txt
$ php test.php
hello

This function is widely used, but I’ve observed that error handling around it is often not quite right. I’ve fixed a few bugs involving incorrect I/O error handling recently, so here are my thoughts on how it should be done.

How file_get_contents fails

For legacy reasons, this function does not throw an exception when something goes wrong. Instead, it will both log a warning, and return false.

<?php
$filename = "not-a-real-file.txt";
$text = file_get_contents($filename);
echo $text;

Which looks like this when you run it:

$ php test.php
PHP Warning:  file_get_contents(not-a-real-file.txt): failed to open stream: No such file or directory in test.php on line 3
PHP Stack trace:
PHP   1. {main}() test.php:0
PHP   2. file_get_contents() test.php:3

Warnings are not very useful on their own, because the code will continue on without the correct data.

Error handling in four steps

If anything goes wrong when you are reading a file, your code should be throwing some type of Exception which describes the problem. This allows developers to put a try {} catch {} around it, and avoids nasty surprises where invalid data is used later.

Step 1: Detect that the file was not read

Any call to file_get_contents should be immediately followed by a check for that false return value. This is how you know that there is a problem.

<?php
$filename = "not-a-real-file.txt";
$text = file_get_contents($filename);
if($text === false) {
  throw new Exception("File was not loaded");
}
echo $text;

This now gives both a warning and an uncaught exception:

$ php test.php
PHP Warning:  file_get_contents(not-a-real-file.txt): failed to open stream: No such file or directory in test.php on line 3
PHP Stack trace:
PHP   1. {main}() test.php:0
PHP   2. file_get_contents() test.php:3
PHP Fatal error:  Uncaught Exception: File was not loaded in test.php:5
Stack trace:
#0 {main}
  thrown in test.php on line 5

Step 2: Suppress the warning

Warnings are usually harmless, but there are several good reasons to suppress them:

It ensures that you are not depending on a global error handler (or the absence of one) for correct behaviour.
The warning might appear in the middle of the output, depending on php.ini.
Warnings can produce a lot of noise in the logs

Use @ to silence any warnings from a function call.

<?php
$filename = "not-a-real-file.txt";
$text = @file_get_contents($filename);
if($text === false) {
  throw new Exception("File was not loaded");
}
echo $text;

The output is now only the uncaught Exception:

$ php test.php
PHP Fatal error:  Uncaught Exception: File was not loaded in test.php:5
Stack trace:
#0 {main}
  thrown in test.php on line 5

Step 3: Get the reason for the failure

Unfortunately, we lost the “No such file or directory” message, which is pretty important information, which should go in the Exception. This information is retrieved from the old-style error_get_last method.

This function might just return empty data, so you should check that everything is set and non-empty before you try to use it.

<?php
$filename = "not-a-real-file.txt";
error_clear_last();
$text = @file_get_contents($filename);
if($text === false) {
  $e = error_get_last();
  $error = (isset($e) && isset($e['message']) && $e['message'] != "") ?
      $e['message'] : "Check that the file exists and can be read.";
  throw new Exception("File '$filename' was not loaded. $error");
}
echo $text;

This now embeds the failure reason directly in the message.

$ php test.php
PHP Fatal error:  Uncaught Exception: File 'not-a-real-file.txt' was not loaded. file_get_contents(not-a-real-file.txt): failed to open stream: No such file or directory in test.php:9
Stack trace:
#0 {main}
  thrown in test.php on line 9

Step 4: Add a fallback

The last time I introduced error_clear_last()/get_last_error() into a code-base, I learned that HHVM does not have these functions.

Call to undefined function error_clear_last()

The fix for this is to write some wrapper code, to verify that each function exists.

<?php
$filename = 'not-a-real-file';
clearLastError();
$text = @file_get_contents($filename);
if ($text === false) {
    $error = getLastErrorOrDefault("Check that the file exists and can be read.");
    throw new \Exception("Could not retrieve image data from '$filename'. $error");
}
echo $text;

/**
 * Call error_clear_last() if it exists. This is dependent on which PHP runtime is used.
 */
function clearLastError()
{
    if (function_exists('error_clear_last')) {
        error_clear_last();
    }
}

/**
 * Retrieve the message from error_get_last() if possible. This is very useful for debugging, but it will not
 * always exist or return anything useful.
 */
function getLastErrorOrDefault(string $default)
{
    if (function_exists('error_clear_last')) {
        $e = error_get_last();
        if (isset($e) && isset($e['message']) && $e['message'] != "") {
            return $e['message'];
        }
    }
    return $default;
}

This does the same thing as before, but without breaking other PHP runtimes.

$ php test.php
PHP Fatal error:  Uncaught Exception: Could not retrieve image data from 'not-a-real-file'. file_get_contents(not-a-real-file): failed to open stream: No such file or directory in test.php:7
Stack trace:
#0 {main}
  thrown in test.php on line 7

Since HHVM is dropping support for PHP, I expect that this last step will soon become unnecessary.

How not to handle errors

Some applications put a series of checks before each I/O operation, and then simply perform the operation with no error handling. An example of this would be:

<?php
$filename = "not-a-real-file.txt";
// Check everything!
if(!file_exists($filename)) {
  throw new Exception("$filename does not exist");
}
if(!is_file($filename)) {
  throw new Exception("$filename is not a file");
}
if(!is_readable($filename)) {
  throw new Exception("$filename cannot be read");
}
// Assume that nothing can possibly go wrong..
$text = @file_get_contents($filename);
echo $text;

You could probably make a reasonable-sounding argument that pre-emptive checks are a good idea, but I consider them to be misguided:

If you skip any actual error handling, then your code is going to fail in more surprising ways when you encounter an I/O problem that could not be detected.
If you do perform correct error handling as well, then the extra checks add nothing other than more branches to test.

Lastly, beware of false positives. For example, the above snippet will reject HTTP URL’s, which are perfectly valid for file_get_contents.

Conclusion

Most PHP code now uses try/catch/finally blocks to handle problems, but the ecosystem really values backwards compatibility, so existing functions are rarely changed.

The style of error reporting used in these I/O functions is by now a legacy quirk, and should be wrapped to consistently throw a useful Exception.

The top 100 Ansible modules

Wed, 30 Jan 2019 00:00:00 +0000

Ansible is an automation system which is widely used for deployments and configuration. It contains a dizzying array of modules for interfacing with things like files, services, package managers, and various pieces of software and hardware.

Background

There are some modules which I would expect every Ansible user to be familiar with, while others are barely used at all. I could have a guess about which modules I use most, but what about the wider community?

I couldn’t find any existing data, so I set out to determine which Ansible modules were most widely-used. The best source for this is Ansible Galaxy, which is a directory of around 18,000 Ansible roles. Using some scripts, I sifted through the 18,259 publicly accessible roles from Ansible Galaxy, found 71,610 files to look through, and tallied up which modules were being used by each of the 317,757 tasks in those files.

The list

These are the top 100 modules as of January 2019. I used the same method to build such a list in January 2018, so you can also see the change in popularity of some modules over the last 12 months.

Rank	Module	Uses	Rank +/-
1	file	26,224	+1
2	include	25,267	-1
3	template	24,062	-
4	command	23,952	-
5	service	19,436	-
6	shell	19,401	-
7	set_fact	17,079	+1
8	apt	16,006	+1
9	lineinfile	12,432	-2
10	copy	8,794	+1
11	yum	8,733	-1
12	assert	7,504	+7
13	include_tasks	6,686	+15
14	stat	5,922	-2
15	package	5,422	-1
16	get_url	5,323	-3
17	debug	5,147	-2
18	import_tasks	5,016	+20
19	include_vars	4,251	-2
20	apt_repository	3,789	-4
21	user	3,233	-3
22	fail	2,812	+2
23	unarchive	2,734	-2
24	apt_key	2,665	-4
25	pip	2,474	-
26	systemd	2,389	+8
27	action	2,114	-4
28	git	2,043	-1
29	uri	1,680	+7
30	group	1,665	-1
31	sysctl	1,331	-5
32	raw	1,323	-2
33	mysql_user	1,266	-
34	meta	1,204	+3
35	replace	1,125	-3
36	ini_file	1,125	-5
37	find	1,030	-15
38	local_action	986	-3
39	mysql_db	959	-
40	cron	925	-
41	wait_for	898	-
42	rpm_key	771	-
43	include_role	742	+15
44	yum_repository	723	+2
45	mount	669	-2
46	blockinfile	619	+3
47	firewalld	579	+1
48	ufw	563	-1
49	authorized_key	557	-4
50	docker_container	488	+2
51	dnf	455	+5
52	seboolean	417	-8
53	homebrew	409	-2
54	fetch	378	+1
55	npm	367	-5
56	osx_defaults	363	+8
57	postgresql_user	350	-4
58	pkgng	345	-4
59	pause	340	+4
60	script	314	+5
61	setup	290	+7
62	postgresql_db	289	-2
63	mysql_replication	286	-1
64	win_regedit	281	+6
65	pacman	257	-4
66	debconf	256	+1
67	slurp	255	+10
68	gem	253	-11
69	iptables	240	+13
70	apache2_module	231	-4
71	synchronize	213	-2
72	docker	213	-13
73	alternatives	210	-
74	selinux	202	-
75	oc_obj	199	+98
76	make	194	-1
77	win_shell	191	+13
78	modprobe	187	+11
79	hostname	181	-7
80	zypper	174	-4
81	xml	160	-
82	supervisorctl	160	-11
83	win_file	149	+15
84	homebrew_cask	149	-6
85	add_host	144	-2
86	rabbitmq_user	129	-7
87	pamd	124	+81
88	win_command	116	+13
89	assemble	116	-9
90	htpasswd	115	-6
91	apk	112	-5
92	openbsd_pkg	111	-7
93	win_get_url	109	+14
94	win_chocolatey	109	-
95	docker_image	109	+4
96	tempfile	106	+25
97	locale_gen	105	-5
98	easy_install	97	-10
99	django_manage	97	-3
100	composer	96	-3

Out of over 2,500 modules mentioned in Ansible Galaxy, it turns out that only a few dozen are widely used. Following this, there is a long tail of infrequently-used and custom modules.

How many modules do I need to know?

You can write 80% of the roles in Ansible Galaxy using only the 74 most popular Ansible modules.

For writing an individual Ansible role, you don’t need anywhere near this many. The median number of modules used by an role in Ansible Galaxy is just 6.

There are a few reasons that you won’t need to use many popular modules all at the same time:

Different technologies: eg. mysql_db and postgresql_db set up different databases, and would be invoked by different roles if you need both.
Obsolete modules: eg. docker has been split up, and new code will use docker_container instead.
Different levels of abstraction: eg. cross-platform developers will use the package module, while developers who know their platform may directly use the apt, dnf and yum modules. You are unlikely to see these mixed.
Different styles: include_tasks vs. import_tasks.

Some notes

Here are some things to keep in mind when reading the table.

I skipped any role that was not hosted in a public GitHub repository, or could not be parsed. A php-yaml bug also kept a few files out of the 2019 tally.
Each task was counted only once, so local_action and action are obscuring which modules were eventually executed.
The OpenShift oc_obj module on this list is not bundled with Ansible (the module itself is distributed through Galaxy).
I did not attempt to exclude abandoned or incomplete packages. Eg. The docker module on this list has been removed from the latest version, but remains popular in existing Ansible Galaxy roles.

Thermal Sans Mono: A new bitmap font for receipt printers

Thu, 01 Nov 2018 00:00:00 +0000

Today I’m writing about a new project that I’ve started to create a font, called “Thermal Sans Mono”. I’m hoping to make this into a set of specialized free bitmap fonts for use on thermal receipt printers and printer emulators.

Here is what it looks like so far:

In this blog post I’ll talk a bit about why I needed a new font, how it’s derived from GNU Unifont, and what I’m planning to do with it.

Background

Firstly, I am solving a technical problem, not a design problem.

Printing non-ASCII characters on receipts can get a bit complex, especially if you have a lot of different printers to support. Each thermal receipt printer has a different set of available “code pages” of glyphs, which makes localisation a real adventure.

Using the escpos-printer-db project, we can currently create software which accepts UTF-8 text, then switches between the available code pages to render text for a specific printer. I’m now looking at how we can encode characters that are not in any available code page.

For example, in this code from the python-escpos library, the ? substitution character still has to be printed for some characters:

encoding = self.encoder.find_suitable_encoding(text[0])
if not encoding:
    self._handle_character_failed(text[0])

Rather than send a ? character, I think we could retrieve a glyph from a suitable font and send it as a bitmap instead. I previously wrote about a technique to cache glyphs from GNU Unifont in printer memory, but these glyphs don’t have the correct size or weight to be displayed alongside the glyphs that the printer has rendered itself.

Of course we can’t do this yet, because there is no suitable bitmap font.

How it’s made

The closest font to what we need is GNU Unifont, which is thankfully freely licensed. A Unifont glyph is typically 8x16 with a thin line and a lot of whitespace. I need to print in 12x24, with thick lines and no whitespace.

To control as many variables as possible, I decided to pre-render a transformed version of GNU Unifont. This involved tracing the original glyph and re-drawing it with new parameters.

The first step is to use a purpose-built algorithm to trace the lines from the glyph:

The points are then re-drawn onto a new canvas with a thicker outline, and a lot less whitespace.

When this canvas is scaled back to 12x24, we have our glyph.

This glyph is now the correct size and weight for use on most thermal receipt printers.

Roadmap

Currently, I have only processed the ASCII characters. As a GNU Unifont derivative, we should (with some effort) be able to include many more glyphs in the future.

As a starting goal, I am aiming to make the font complete enough to include as a fallback font in the escpos-php printing library.

Download

I’ve made my initial work on “Thermal Sans Mono” available on GitHub at mike42/thermal-sans-mono. You can find it as a bitmap font in PCF and BDF format under the “Releases” section of the GitHub project.

As always, bugs, comments and suggestions are appreciated.

Three ways to archive a website

Thu, 13 Sep 2018 00:00:00 +0000

I recently needed to archive a small website before decommissioning it. There are a few distinct reasons you might want an archive of a website:

To archive the information in case you need it later.
To archive the look and feel so that you can see how it has progressed.
To archive the digital artifacts so that you can host them elsewhere as a mirror.

Each of these produces files in a different format, which are useful over different time-periods. In this post, I’ll write a bit about all three, since it’s easiest to archive a website while it is still online.

Saving webpage content to PDF

To write an individual page to a PDF, you can use wkhtmltopdf. On Debian/Ubuntu, this can be installed with:

sudo apt-get install wkhtmltopdf

The only extra setting I use for this is the “Javascript delay”, since some parts of the page will be loaded after the main content.

mkdir -p pdf/
wkhtmltopdf --javascript-delay 1000 https://example.com/ pdf/index.pdf

This produces a PDF file, which you can copy/paste text from, or print.

You then simply repeat this for every page which you want to archive.

Saving webpage content to an image

If you are more interested in how the website looked, rather than what it contained, then you can use the same package to write it to an image. I use the jpg format here, because the file sizes are reasonable at higher resolution. I also zoom the page 200% to get higher quality, and selected sizes which are typical of desktop, tablet and mobile screen sizes.

mkdir -p jpg/desktop jpg/mobile jpg/tablet
wkhtmltoimage --zoom 2.0 --javascript-delay 1000 --width 4380 https://example.com/ jpg/desktop/index.jpg
wkhtmltoimage --zoom 2.0 --javascript-delay 1000 --width 2048 https://example.com/ jpg/tablet/index.jpg
wkhtmltoimage --zoom 2.0 --javascript-delay 1000 --width 960 https://example.com/ jpg/mobile/index.jpg

This gives you three images for the page. This example page is quite short, but a larger page produces a very tall image.

The mobile and tablet versions are narrower.

While the desktop version is full-width.

As above, this needs to be repeated for each page which you want to archive.

Mirroring the entire site as HTML

A full mirror of the site is a good short-term archive. Some websites have a lot of embedded external content like maps and external social media widgets, which I would expect to gradually stop working over time as these services change. Still, you might still be able to browse the website on your local computer in 10 or 20 years time, depending on how browsers change.

wget is the go-to tool for mirroring sites, but it has a lot of options!

mkdir -p html/
cd html
wget \
  --trust-server-names \
  -e robots=off \
  --mirror \
  --convert-links \
  --adjust-extension \
  --page-requisites \
  --no-parent \
  https://example.com

There are quite a few options here, I’ll briefly explain why I used each one:

Option	Purpose
`--trust-server-names`	Allow the correct filename to be used when a redirect is used.
`-e robots=off`	Disable rate limiting. This is OK to use if you own the site and are sure that mirroring it will not cause capacity issues.
`--mirror`	Short-hand for some options to recursively download the site.
`--convert-links`	Change links on the target site to local ones.
`--adjust-extension`	If you get a page called “foo”, save it as “foo.html”.
`--page-requisites`	Also download CSS and Javascript files referenced on the page
`--no-parent`	Only download sub-pages from the starting page. This is useful if you want to fetch only part of the domain.

The result can be opened locally in a web browser:

These options worked well for me on a Wordpress site.

Putting it all together

The site I was mirroring was quite small, so I manually assembled a list of pages to mirror, gave each a name, and wrote them in a text file called urls.txt in this format:

https://site.example/ index
https://site.example/foo foo
https://site.example/bar bar

I then ran this script to mirror each URL as an image and PDF, before mirroring the entire site locally in HTML.

#!/bin/bash
set -exu -o pipefail

mkdir -p jpg/desktop jpg/mobile jpg/tablet html/ pdf
while read page_url page_name; do
  echo "## $page_url ($page_name)"
  # JPEG archive
  wkhtmltoimage --zoom 2.0 --javascript-delay 1000 --width 4380 $page_url jpg/desktop/$page_name.jpg
  wkhtmltoimage --zoom 2.0 --javascript-delay 1000 --width 2048 $page_url jpg/tablet/$page_name.jpg
  wkhtmltoimage --zoom 2.0 --javascript-delay 1000 --width 960 $page_url jpg/mobile/$page_name.jpg
  # Printable archive
  wkhtmltopdf --javascript-delay 1000 $page_url pdf/$page_name.pdf
done < urls.txt

# Browsable archive
MAIN_PAGE=$(head -n1 urls.txt | cut -d' ' -f1)
mkdir -p html/
(cd html && \
  wget --trust-server-names -e robots=off --mirror --convert-links --adjust-extension --page-requisites --no-parent $MAIN_PAGE)

The example.com domain only has an index page, so after running the script against it, it downloads this set of files:

├── archive.sh
├── html
│   └── example.com
│       └── index.html
├── jpg
│   ├── desktop
│   │   └── index.jpg
│   ├── mobile
│   │   └── index.jpg
│   └── tablet
│       └── index.jpg
├── pdf
│   └── index.pdf
└── urls.txt

Happy archiving!

Monitoring network throughput with Prometheus

Thu, 23 Aug 2018 00:00:00 +0000

Today I’m writing a bit about a Prometheus deployment that I made last year on a Raspberry Pi, to get better data about congestion on my uplink to the Internet.

The problem

You have probably run an Internet speed test before, like this:

A speed test will tell you how slow your computer’s connection is, but it can’t narrow down whether it’s because of other LAN devices, the line quality, or congestion at the provider.

You can start to assemble this information from the router, which has counters for each network interface.

This table is from a Sagemcom F@ST 3864, which is a consumer-grade router. It has no SNMP interface, so the only way to get these metrics is to query /statsifc.html and /info.html from the LAN.

Getting the data

I can derive throughput metrics for the uplink if I scrape these metrics every few seconds and load them into a time-series database. To do this, I wrote a small adapter (called an “exporter” in Prometheus lingo), which exposed the metrics in a more structured way.

The result was a web page on the Raspberry Pi, which returns interface data like this:

# HELP lan_network_receive_bytes Received bytes for network interface
# TYPE lan_network_receive_bytes gauge
lan_network_receive_bytes{device="eth0"} 0.0
lan_network_receive_bytes{device="eth1"} 0.0
lan_network_receive_bytes{device="eth2"} 0.0
lan_network_receive_bytes{device="eth3"} 0.0
lan_network_receive_bytes{device="wl0"} 737476060.0
# HELP lan_network_send_bytes Sent bytes for network interface
# TYPE lan_network_send_bytes gauge
lan_network_send_bytes{device="eth0"} 363957004.0
lan_network_send_bytes{device="eth1"} 0.0
lan_network_send_bytes{device="eth2"} 0.0
lan_network_send_bytes{device="eth3"} 0.0
lan_network_send_bytes{device="wl0"} 2147483647.0
# HELP lan_network_receive_packets Received packets for network interface
# TYPE lan_network_receive_packets gauge
lan_network_receive_packets{device="eth0",disposition="transfer"} 1766250.0
lan_network_receive_packets{device="eth0",disposition="error"} 0.0
lan_network_receive_packets{device="eth0",disposition="drop"} 0.0
lan_network_receive_packets{device="eth1",disposition="transfer"} 0.0
lan_network_receive_packets{device="eth1",disposition="error"} 0.0
lan_network_receive_packets{device="eth1",disposition="drop"} 0.0
lan_network_receive_packets{device="eth2",disposition="transfer"} 0.0
lan_network_receive_packets{device="eth2",disposition="error"} 0.0
lan_network_receive_packets{device="eth2",disposition="drop"} 0.0
lan_network_receive_packets{device="eth3",disposition="transfer"} 0.0
lan_network_receive_packets{device="eth3",disposition="error"} 0.0
lan_network_receive_packets{device="eth3",disposition="drop"} 0.0
lan_network_receive_packets{device="wl0",disposition="transfer"} 6622351.0
lan_network_receive_packets{device="wl0",disposition="error"} 0.0
lan_network_receive_packets{device="wl0",disposition="drop"} 0.0
# HELP lan_network_send_packets Sent packets for network interface
# TYPE lan_network_send_packets gauge
lan_network_send_packets{device="eth0",disposition="transfer"} 3148577.0
lan_network_send_packets{device="eth0",disposition="error"} 0.0
lan_network_send_packets{device="eth0",disposition="drop"} 0.0
lan_network_send_packets{device="eth1",disposition="transfer"} 0.0
lan_network_send_packets{device="eth1",disposition="error"} 0.0
lan_network_send_packets{device="eth1",disposition="drop"} 0.0
lan_network_send_packets{device="eth2",disposition="transfer"} 0.0
lan_network_send_packets{device="eth2",disposition="error"} 0.0
lan_network_send_packets{device="eth2",disposition="drop"} 0.0
lan_network_send_packets{device="eth3",disposition="transfer"} 0.0
lan_network_send_packets{device="eth3",disposition="error"} 0.0
lan_network_send_packets{device="eth3",disposition="drop"} 0.0
lan_network_send_packets{device="wl0",disposition="transfer"} 8803737.0
lan_network_send_packets{device="wl0",disposition="error"} 0.0
lan_network_send_packets{device="wl0",disposition="drop"} 0.0
# HELP wan_network_receive_bytes Received bytes for network interface
# TYPE wan_network_receive_bytes gauge
wan_network_receive_bytes{device="ppp2.1"} 3013958333.0
wan_network_receive_bytes{device="ptm0.1"} 0.0
wan_network_receive_bytes{device="eth4.3"} 0.0
wan_network_receive_bytes{device="ppp1.1"} 0.0
wan_network_receive_bytes{device="ppp3.2"} 0.0
# HELP wan_network_send_bytes Sent bytes for network interface
# TYPE wan_network_send_bytes gauge
wan_network_send_bytes{device="ppp2.1"} 717118493.0
wan_network_send_bytes{device="ptm0.1"} 0.0
wan_network_send_bytes{device="eth4.3"} 0.0
wan_network_send_bytes{device="ppp1.1"} 0.0
wan_network_send_bytes{device="ppp3.2"} 0.0
# HELP wan_network_receive_packets Received packets for network interface
# TYPE wan_network_receive_packets gauge
wan_network_receive_packets{device="ppp2.1",disposition="transfer"} 11525693.0
wan_network_receive_packets{device="ppp2.1",disposition="error"} 0.0
wan_network_receive_packets{device="ppp2.1",disposition="drop"} 0.0
wan_network_receive_packets{device="ptm0.1",disposition="transfer"} 0.0
wan_network_receive_packets{device="ptm0.1",disposition="error"} 0.0
wan_network_receive_packets{device="ptm0.1",disposition="drop"} 0.0
wan_network_receive_packets{device="eth4.3",disposition="transfer"} 0.0
wan_network_receive_packets{device="eth4.3",disposition="error"} 0.0
wan_network_receive_packets{device="eth4.3",disposition="drop"} 0.0
wan_network_receive_packets{device="ppp1.1",disposition="transfer"} 0.0
wan_network_receive_packets{device="ppp1.1",disposition="error"} 0.0
wan_network_receive_packets{device="ppp1.1",disposition="drop"} 0.0
wan_network_receive_packets{device="ppp3.2",disposition="transfer"} 0.0
wan_network_receive_packets{device="ppp3.2",disposition="error"} 0.0
wan_network_receive_packets{device="ppp3.2",disposition="drop"} 0.0
# HELP wan_network_send_packets Sent packets for network interface
# TYPE wan_network_send_packets gauge
wan_network_send_packets{device="ppp2.1",disposition="transfer"} 7728904.0
wan_network_send_packets{device="ppp2.1",disposition="error"} 0.0
wan_network_send_packets{device="ppp2.1",disposition="drop"} 0.0
wan_network_send_packets{device="ptm0.1",disposition="transfer"} 0.0
wan_network_send_packets{device="ptm0.1",disposition="error"} 0.0
wan_network_send_packets{device="ptm0.1",disposition="drop"} 0.0
wan_network_send_packets{device="eth4.3",disposition="transfer"} 0.0
wan_network_send_packets{device="eth4.3",disposition="error"} 0.0
wan_network_send_packets{device="eth4.3",disposition="drop"} 0.0
wan_network_send_packets{device="ppp1.1",disposition="transfer"} 0.0
wan_network_send_packets{device="ppp1.1",disposition="error"} 0.0
wan_network_send_packets{device="ppp1.1",disposition="drop"} 0.0
wan_network_send_packets{device="ppp3.2",disposition="transfer"} 0.0
wan_network_send_packets{device="ppp3.2",disposition="error"} 0.0
wan_network_send_packets{device="ppp3.2",disposition="drop"} 0.0
# HELP adsl_attainable_rate_down_kbps ADSL Attainable Rate down (Kbps)
# TYPE adsl_attainable_rate_down_kbps gauge
adsl_attainable_rate_down_kbps 19708.0
# HELP adsl_attainable_rate_up_kbps ADSL Attainable Rate up (Kbps)
# TYPE adsl_attainable_rate_up_kbps gauge
adsl_attainable_rate_up_kbps 1087.0
# HELP adsl_rate_down_kbps ADSL Rate down (Kbps)
# TYPE adsl_rate_down_kbps gauge
adsl_rate_down_kbps 18175.0
# HELP adsl_rate_up_kbps ADSL Rate up (Kbps)
# TYPE adsl_rate_up_kbps gauge
adsl_rate_up_kbps 1087.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 34197504.0
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 22441984.0
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1497148890.92
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 3254.92
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 7.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024.0

I then deployed Prometheus to the same Raspberry Pi, and configured it to read these metrics every few seconds by editing prometheus.yml

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: net
    static_configs:
    - targets: ["localhost:8000"]

Making some queries

Prometheus has a query language, which I find similar to spreadsheet formulas. You can enter a query directly into the web interface to get a graph or data table.

I settled on these queries to get the data I needed. They show me the maximum attainable line rate, actual sync rate, and current throughput over the WAN interface.

Download rate

Throughput:

rate(wan_network_receive_bytes{device="ppp2.1"}[10s])*8/1024/1024

ADSL attainable:

adsl_attainable_rate_down_kbps/1024

ADSL sync:

adsl_rate_down_kbps/1024

Upload rate

Usage:

rate(wan_network_send_bytes{device="ppp2.1"}[10s])*8/1024/1024

ADSL attainable:

adsl_attainable_rate_up_kbps/1024

ADSL sync:

adsl_rate_up_kbps/1024

Onto a dashboard

I then deployed the last component in this setup, Grafana, to the Raspberry Pi. This tool lets you save your queries on a dashboard.

I made two plots, one for uploads, and one for downloads:

By saturating the link with traffic (such as when running a speed test), it was now possible to compare the actual network speed with the ADSL sync speed.

In my case, the best attainable network speed changed depending on the time of day, while the ADSL sync speed was constant. That’s a simple case of congestion.

Conclusion

I’ve deployed a few tiny Prometheus setups like this, because of how simple it is to work with new sources of metrics. It’s designed for much larger setups than an individual router, so it’s a worthwhile tool to be familiar with. Data is always a good reality-check for your assumptions, of course.

This setup had the level of security that you would expect of a Raspberry Pi project (none), and crashed after 4 days because I did didn’t configure Grafana for a RAM-limited environment. It was still a useful learning exercise, so I uploaded the python and Ansible code to GitHub at mike42/network-speed-graphs.

How to export a Maildir email inbox as a CSV file

Thu, 12 Jul 2018 00:00:00 +0000

I’ve often thought it would be useful to have an “Export as CSV” button on my inbox, and recently had an excuse to implement one.

I needed to parse the Subject of each email coming from an automated process, but the inbox was in the Maildir format, and I needed something more useful for data interchange.

So I wrote a utility, mail2csv, to export the Maildir as CSV file, which many existing tools can work with. It produces one row per email, and one column per email header:

The basic usage is:

$ mail2csv example/
Date,Subject,From
"Wed, 16 May 2018 20:05:16 +0000",An email,Bob <bob@example.com>
"Wed, 16 May 2018 20:07:52 +0000",Also an email,Alice <alice@example.com>

You can select any header with the --headers field, which accepts globs:

$ mail2csv example/ --headers Message-ID Date Subject
Message-ID,Date,Subject
<89125@example.com>,"Wed, 16 May 2018 20:05:16 +0000",An email
<180c6@example.com>,"Wed, 16 May 2018 20:07:52 +0000",Also an email

If you’re not sure which headers your emails have, then you can make a CSV with all the headers, and just read the first line:

$ mail2csv  example/ --headers '*' | head -n1

You can find the Python code for this tool at mike42/mail2csv on GitHub.

How to make HHVM 3.21 identify itself as PHP 7

Thu, 05 Jul 2018 00:00:00 +0000

HHVM is an alternative runtime for PHP, which I try to maintain compatibility with.

I recently upgraded a project to PHPUnit 6, and the tests started failing on hhvm-3.21 with this error:

This version of PHPUnit is supported on PHP 7.0 and PHP 7.1.
You are using PHP 5.6.99-hhvm (/usr/bin/hhvm).

Although HHVM supports many PHP 7 features, it is identifying itself as PHP 5.6. The official documentation includes a setting for enabling additional PHP 7 features, which also sets the reported version.

This project was building on Travis CI, so adding this one-liner to .travis.yml sets this flag in the configuration, and the issue goes away:

before_script:
  - bash -c 'if [[ $TRAVIS_PHP_VERSION == hhvm* ]]; then echo "hhvm.php7.all = 1" | sudo tee -a /etc/hhvm/php.ini; fi'

I’m writing about this for other developers who hit the same issue. The current releases of HHVM don’t seem to have this issue, so hopefully this work-around is temporary.

How to check PHP code style in a continuous integration environment

Thu, 28 Jun 2018 00:00:00 +0000

I’m going to write a bit today about PHP_CodeSniffer, which is a tool for detecting PHP style violations.

Background

Convention is good. It’s like following the principle of least surprise in software design, but for your developer audience.

PHP has two code standards that I see in the wild:

If you are writing new PHP code, or refactoring code that follows no discernible standard, you should use the PSR-2 code style.

I use PHP_CodeSniffer on a few projects to stop the build early if the code is not up to standard. It has the ability to fix the formatting problems it finds, so it is not a difficult test to clear.

Add PHP_CodeSniffer to your project

I’m assuming that you are already using composer to manage your PHP dependencies.

Ensure you are specifying your minimum PHP version in your composer.json:

"require": {
    "php": ">=7.0.0"
}

Next, add squizlabs/php_codesniffer as a development dependency of your project:

composer require --dev squizlabs/php_codesniffer

Check your code

I use this command to check the src folder. It can be run locally or on a CI box:

php vendor/bin/phpcs --standard=psr2 src/ -n

This will return nonzero if there are problems with the code style, which should stop the build on most systems.

The -n flag avoids less severe “warnings”. One of these is the “Line exceeds 120 characters”, which can be difficult to completely eliminate.

Fixing the style

Running the phpcbf command with the same arguments will correct simple formatting issues:

php vendor/bin/phpcs --standard=psr2 src/ -n

Travis CI example

A full .travis.yml which uses this would look like this (adapted from here):

language: php

php:
  - 7.0
  - 7.1
  - 7.2

install:
  - composer install

script:
  - php vendor/bin/phpcs --standard=psr2 test/ -n

If you are running a PHP project without any sort of build, then this is a good place to start: Aside from style, this checks that dependencies can be installed on a supported range of PHP versions, and will also break if there are files with syntax errors.

Make better CLI progress bars with Unicode block characters

Thu, 14 Jun 2018 00:00:00 +0000

As a programmer, you might add a progress bar so that the user has feedback while they wait for a slow task.

If you are writing a console (CLI) application, then you need to make your progress bars from text. A good command-line progress bar should update in small increments, like this example:

This uses Unicode Block Elements to give the progress bar a higher resolution.

Code	Character
U+2588	█
U+2589	▉
U+258A	▊
U+258B	▋
U+258C	▌
U+258D	▍
U+258E	▎
U+258F	▏

A lot of applications use plain ASCII in their progress bars. The progress bar in wget, for example, uses ===> characters only, like this:

Progress bars made from ASCII characters like = and # signs are very common, most likely because of the historical portability issues around non-ASCII text. Nowadays, UTF-8 support is ubiquitous, and it’s pointless to adhere to such limitations.

Example: Better progress bars in python

The animation at the top of this blog post is a simple python script.

import sys
import time

from progress_bar import ProgressBar

"""
Example usage of ProgressBar class
"""
print("Doing work\n")

with ProgressBar(sys.stdout) as progress:
    for i in range(0,800):
        progress.update(i / 800)
        time.sleep(0.05)
print("\nDone.\n");

The script progress_bars.py, written for Python 3, contains a class that allows the progress bar to be created and drawn on different types of terminals.

import abc
import math
import shutil
import sys
import time

from typing import TextIO

"""
Produce progress bar with ANSI code output.
"""
class ProgressBar(object):
    def __init__(self, target: TextIO = sys.stdout):
        self._target = target
        self._text_only = not self._target.isatty()
        self._update_width()

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        if(exc_type is None):
            # Set to 100% for neatness, if no exception is thrown
            self.update(1.0)
        if not self._text_only:
            # ANSI-output should be rounded off with a newline
            self._target.write('\n')
        self._target.flush()
        pass

    def _update_width(self):
        self._width, _ = shutil.get_terminal_size((80, 20))

    def update(self, progress : float):
        # Update width in case of resize
        self._update_width()
        # Progress bar itself
        if self._width < 12:
            # No label in excessively small terminal
            percent_str = ''
            progress_bar_str = ProgressBar.progress_bar_str(progress, self._width - 2)
        elif self._width < 40:
            # No padding at smaller size
            percent_str = "{:6.2f} %".format(progress * 100)
            progress_bar_str = ProgressBar.progress_bar_str(progress, self._width - 11) + ' '
        else:
            # Standard progress bar with padding and label
            percent_str = "{:6.2f} %".format(progress * 100) + "  "
            progress_bar_str = " " * 5 + ProgressBar.progress_bar_str(progress, self._width - 21)
        # Write output
        if self._text_only:
            self._target.write(progress_bar_str + percent_str + '\n')
            self._target.flush()
        else:
            self._target.write('\033[G' + progress_bar_str + percent_str)
            self._target.flush()

    @staticmethod
    def progress_bar_str(progress : float, width : int):
        # 0 <= progress <= 1
        progress = min(1, max(0, progress))
        whole_width = math.floor(progress * width)
        remainder_width = (progress * width) % 1
        part_width = math.floor(remainder_width * 8)
        part_char = [" ", "▏", "▎", "▍", "▌", "▋", "▊", "▉"][part_width]
        if (width - whole_width - 1) < 0:
          part_char = ""
        line = "[" + "█" * whole_width + part_char + " " * (width - whole_width - 1) + "]"
        return line

Aside from the use of box drawing characters, this script includes a few other things which a good progress bar should implement:

Resize the progress bar when you resize the terminal
Simplify the progress bar on very small terminals
Don’t print ANSI terminal codes if the script is not connected to a terminal
Round off to 100% once the use of the progress bar completes without error

After writing this, I discovered that the progress pypi package can also use these box characters, so I haven’t packaged this code up. I haven’t used progress yet, but if you want better progress bars, then you might like to evaluate it for your own application.

How to create effective PHP project documentation with Read the Docs

Thu, 07 Jun 2018 00:00:00 +0000

Documentation is one of the ways that software projects can communicate information to their users. Effective documentation is high-quality, meaning that it’s complete, accurate, and up-to-date. At least for open source libraries, it also means that you can find it with a search engine. For many small PHP projects, the reality is very far removed from the ideal.

Read the Docs (readthedocs.io) makes it easy to host an up-to-date copy of your project’s documentation online. There are around 2,000 PHP projects which host their documentation on the site, which makes PHP the third most popular programming language for projects on the site.

This post covers the process that is used to automatically publish the documentation for the gfx-php PHP graphics library, which is one of those 2000 projects. You should consider using this setup as a template if you your project is small enough that it does not have its own infrastructure.

Basic concept

Typically, people are using Read the Docs with a tool called Sphinx. If you are writing in Python, it’s also possible to use the autodoc Sphinx plugin to add API documentation, based on docstrings in the code.

PHP programmers are already spoiled for choice if they want to produce HTML documentation from their code. These tools all have huge PHP user bases:

These will each output their own HTML, which is only useful if you want to self-host the documentation. I wanted a tool that was more like “autodoc for PHP”, so that I can have my API docs magically appear in Sphinx output that is hosted on Read the Docs.

Doxygen is the most useful tool for this purpose, because it has a stable XML output format and good-enough PHP support. I decided to write a tool which to take the Doxygen XML info and generate rest for Sphinx:

This introduces some extra tools, which looks complex at first. The stack is geared towards running within the Read the Docs build environment, so most developers can treat it as a black box after the initial setup:

This setup is entirely hosted with free cloud services, so you don’t need to run any applications on your own hardware.

Tools to install on local workstation

First, we will set up each of these tools locally, so that we know everything is working before we upload it.

Doxygen
Sphinx
doxyphp2sphinx

Doxygen

Doxygen can read PHP files to extract class names, documentation and method signatures. Linux and Mac install this from most package managers (apt-get, dnf or brew) under the name doxygen, while Windows users need to chase down binaries.

In your repo, make a sub-folder called docs/, and create a Doxyfile with some defaults:

mkdir docs/
doxygen -g Doxyfile

You need to edit the configuration to make it suitable or generating XML output for your PHP project. The version of Doxygen used here is 1.8.13, where you only need to change a few values to set Doxygen to:

Recursively search PHP files in your project’s source folder
Generate XML and HTML only
Log warnings to a file

For a typical project, these settings are:

PROJECT_NAME           = "Example Project"
INPUT                  = ../src
WARN_LOGFILE           = warnings.log
RECURSIVE              = YES
USE_MDFILE_AS_MAINPAGE = ../README.md
GENERATE_LATEX         = NO
GENERATE_XML           = YES

Once you set these in Doxyfile, you can run Doxygen to generate HTML and XML output.

$ doxygen

Doxygen will pick up most method signatures automatically, and you can add to them via docblocks, which work along the same lines as docstrings in Python. Read Doxygen: Documenting the Code to learn the syntax if you have not used a documentation generator in a curly-bracket language before.

The Doxygen HTML will never be published, but you might need to read it to see how well Doxygen understands your code.

The XML output is much more useful for our purposes. It contains the same information, and we will read it to generate pages of documentation for Sphinx to render.

Sphinx

Sphinx is the tool that we will use to render the final HTML output. If you haven’t used it before, then see the official Getting Started page.

We are using Sphinx because it can be executed by online services like Read the Docs. It uses the reStructuredText format, which is a whole lot more complex than Markdown, but supports cross-references. I’ll only describe these steps briefly, because there are existing how-to guides on making Sphinx work for manually-written PHP documentation elsewhere on the Internet, such as:

Still in the docs folder with your Doxyfile, create and render an empty Sphinx project.

pip install sphinx
sphinx-quickstart --quiet --project example_project --author example_bob
make html

The generated HTML will initially appear like this:

We need to customize this in a way that adds PHP support. The quickest way is to drop this text into requirements.txt:

Sphinx==1.7.4
sphinx-rtd-theme==0.3.0
sphinxcontrib-phpdomain==0.4.1
doxyphp2sphinx>=1.0.1

Then update config.py to set extensions:

extensions = [
  "sphinxcontrib.phpdomain"
]

Then also set the theme.

html_theme = 'sphinx_rtd_theme'

Lastly, add this to the end of the file:

# PHP Syntax
from sphinx.highlighting import lexers
from pygments.lexers.web import PhpLexer
lexers["php"] = PhpLexer(startinline=True, linenos=1)
lexers["php-annotations"] = PhpLexer(startinline=True, linenos=1)

# Set domain
primary_domain = "php"

Next drop this contents in _templates/breadcrumbs.html (explanation)

{%- extends "sphinx_rtd_theme/breadcrumbs.html" %}

{% block breadcrumbs_aside %}
{% endblock %}

Then finally re-install dependencies and re-build:

pip install -r requirements.txt
make html

The HTML output under _build will now appear as:

This setup gives us three things:

The documentation looks the same as Read the Docs.
We can use PHP snippets and class documentation.
There are no ‘Edit’ links, which is important because some of the files will be generated in the next steps.

doxyphp2sphinx

The doxyphp2sphinx tool will generate .rst files from the Doxygen XML files. This was installed from your requirements.txt in the last step, but you can also install it standalone via pip:

pip install doxyphp2sphinx

The only thing you need to specify is the name of the namespace that you are documenting, using :: as a separator.

doxyphp2sphinx FooCorp::Example

This command will read the xml/ subdirectory, and will create api.rst. It will fill the api/ directory with documentation for each class in the \FooCorp\Example namespace.

To verify that this has worked, check your class structure:

$ tree ../src
../src
├── Dooverwhacky.php
└── Widget.php

You should have documentation for each of these:

$ tree xml/ -P 'class*'
xml/
├── classFooCorp_1_1Example_1_1Dooverwhacky.xml
└── classFooCorp_1_1Example_1_1Widget.xml

And if you have the correct namespace name, you will have .rst files for each as well:

$ tree api
api
├── dooverwhacky.rst
└── widget.rst

Now, add a reference to api.rst somewhere in index.rst:

.. toctree::
   :maxdepth: 2
   :caption: API Documentation

   Classes <api.rst>

And re-compile:

make html

You can now navigate to your classes in the HTML documentation.

The quality of the generated class documentation can be improved by adding docstrings. An example of the generated documentation for a method is:

Writing documentation

As you add pages to your documentation, you can include PHP snippets and reference the usage of your classes.

.. code-block:: php
   
   <?php
   echo "Hello World"

Lorem ipsum dolor sit :class:`Dooverwhacky`, foo bar baz :meth:`Widget::getFeatureCount`.

This will create syntax highlighting for your examples and inline links to the generated API docs.

Beyond this, you will need to learn some reStructuredText. I found this reference to be useful.

Local docs build

A full build has these dependencies:

apt-get install doxygen make python-pip
pip install -r docs/requirements.txt

And these steps:

cd docs/
doxygen 
doxyphp2sphinx FooCorp::Example
make html

Cloud tools

Next, we will take this local build, and run it on a cloud setup instead, using these services.

GitHub
Read the docs

GitHub

I will assume that you know how to use Git and GitHub, and that you are creating code that is intended for re-use.

Add these files to your .gitignore:

docs/_build/
docs/warnings.log
docs/xml/
docs/html/

Upload the remainder of your repository to GitHub. The gfx-php project is an example of a working project with all of the correct files included.

To execute the initial two build steps execute on Read the Docs, add this to the end of docs/conf.py. Don’t forget to update the namespace to match the command you were running locally.

# Regenerate API docs via doxygen + doxyphp2sphinx
import subprocess, os
read_the_docs_build = os.environ.get('READTHEDOCS', None) == 'True'
if read_the_docs_build:
    subprocess.call(['doxygen', 'Doxyfile'])
    subprocess.call(['doxyphp2sphinx', 'FooCorp::Example'])

Read the Docs

After all this setup, Read the Docs should be able to build the project. Create the project on Read the Docs by using the Import from GitHub option. There are only two settings which need to be set:

The requirements file location must be docs/requirements.txt:

And the programming language should be PHP.

After this, you can go ahead and run a build.

As a last step, you will want to ensure that you have a Webhook set up on GitHub to trigger the builds automatically in future.

Conclusion

It is emerging as a best practice for small libraries to host their documentation with Read the Docs, but this is not yet common in the PHP world. Larger PHP projects tend to self-host documentation on the project website, but smaller projects will often have no hosted API documentation.

Once you write your docs, publishing them should be easy! Hopefully this provides a good example of what’s possible.

Acknowledgements

Credit where credit is due: The Breathe project fills this niche for C++ developers using Doxygen, and has been around for some time. Breathe is in the early stages of adding PHP support, but is not yet ready at the time of writing.

How to get a notification when your site appears on HackerNews

Thu, 31 May 2018 00:00:00 +0000

A few weeks ago, somebody submitted this article about GNU parallel to HackerNews, and I got a small wave of new visitors trawling my blog.

I don’t actively monitor referrers to this site, so I was oblivious to this until a few days afterward. Aware of the Slashdot Effect, I thought I should set up some free tools to remind me to log in and check the site’s health if it happens again:

Hacker News RSS feeds

Hacker news does not publish its own RSS feeds, so I had to use a third-party service. I found a URL that would feed me the feed to the latest articles off this site, by searching the “url” attribute:

https://hnrss.org/newest?q=mike42.me&search_attrs=url

This URL gives an RSS feed, as you might expect:

IFTTT

To save me installing and checking a local reader, I set up IFTTT to send me an email when new articles are published to this feed.

The “RSS feed to email” applet is perfect for this kind of consumer-grade automation.

I set it up with the URL, and well, nothing interesting happens. Only new articles are emailed, so this is expected.

Example email

Since I also use this IFTTT applet to get notifications for other RSS feeds, I do know that it works. Within an hour or two of a new article appearing in the feed, the applet gives you an email from RSS Feed via IFTTT <action@ifttt.com>:

It’s not exactly a real-time notification, but I know when my posts are being linked from a specific high-traffic site, which is a good start.

For any site bigger than a personal blog, you might be interested in handling extra traffic rather than just be vaguely aware of it, but I’ll save that for a future post.

Quick statistics about documentation hosted on Read the Docs

Thu, 24 May 2018 00:00:00 +0000

Many open source projects host their online documentation with Read The Docs. Since I’m currently developing a tool for Read The Docs users, I recently took a look at the types of projects that are hosted there.

The data in this post is derived from the list of projects available in the public Read The Docs API, and has its limitations. In particular, spam, inactive, or abandoned projects are not filtered.

Totals

This data is current as of May 20, 2018. At that time, there were 90,129 total projects on readthedocs.io.

Starting out, I had assumed that the majority of projects had all four of these attributes:

Hosted in Git
Programmed in Python
Documented in English
Documentation generated with Sphinx

As it turned out, this particular combination is only used on 35.8% (32,218) of projects, so let’s take a look at how each of these vary.

Programming languages

The two main conclusions I drew by looking at the programming languages are:

Python is the largest developer community that is using the Read the Docs platform
A lot of projects are hosting documentation that is not tagged with a programming language

#	%	Projects	Programming language
1	39.92%	35978	Not code (“just words”)
2	39.27%	35391	Python
3	9.27%	8354	(No language listed)
4	2.83%	2551	Javascript
5	2.29%	2060	PHP
6	1.15%	1040	C++
7	1.13%	1018	Java
8	0.85%	769	Other
9	0.77%	690	C#
10	0.71%	640	C
11	0.42%	380	CSS
12	0.34%	304	Go
13	0.31%	283	Ruby
14	0.31%	277	Julia
15	0.11%	96	Perl
16	0.06%	55	Objective C
17	0.06%	54	R
18	0.04%	36	TypeScript
19	0.03%	31	Scala
20	0.03%	30	Haskell
21	0.03%	29	Lua
22	0.02%	22	Swift
23	0.02%	15	CoffeeScript
24	0.02%	14	Visual Basic
25	0.01%	12	Groovy

Documentation languages

You might have guessed that English dominates in software documentation, but here is the data:

#	%	Projects	Language
1	93.1%	83888	English (en)
2	2.4%	2178	Indonesian (id)
3	1.4%	1287	Chinese (zh, zh_TW, zh_CN)
4	0.5%	456	Russian (ru)
5	0.5%	425	Spanish (es)
6	0.5%	412	French (fr)
7	0.3%	254	Portuguese (pt)
8	0.3%	233	German (de)
9	0.2%	209	Italian (it)
10	0.2%	208	Japanese (jp)
-	0.6%	579	All other languages

In total, documentation on Read the Docs has been published in 54 languages. The “All other languages” item here represents 44 other languages.

Documentation build tool

The majority are using the Sphinx documentation generator. The table here counts sphinx, sphinx_htmldir and sphinx_singlehtml together.

#	%	Projects	Language
1	92.5%	83364	Sphinx
2	6.9%	6229	MkDocs
3	0.6%	536	Auto-select

Version control

Git is the version control system used for the vast majority of projects.

#	%	Projects	Version control
1	96.5%	86988	Git
2	2.1%	1879	Mercurial
3	0.9%	816	SVN
4	0.5%	446	Bazaar

Recovering text from a receipt with escpos-tools

Thu, 17 May 2018 00:00:00 +0000

I have written previously about how to generate receipts for printers which understand ESC/POS. Today, I thought I would write about the opposite process.

Unlike PostScript, the ESC/POS binary language is not commonly understood by software. I wrote a few utilities last year to help change that, called escpos-tools.

In this post, I’ll step through an example ESC/POS binary file that an escpos-tools user sent to me, and show how we can turn it back into a usable format. The tools we are using are:

You might need this sort of process if you need to email a copy of your receipts, or to archive them for audit use.

Printing the file

Binary print files are generated from drivers. I can feed this one back to my printer like this:

cat receipt.bin > /dev/usb/lp0

My Epson TM-T20 receipt printer understands ESC/POS, and prints this out:

Installing escpos-tools

escpos-tools is not packaged yet, so you need git and composer (from the PHP eco-system) to use it.

$ git clone https://github.com/receipt-print-hq/escpos-tools
$ cd escpos-tools
$ composer install

Inspecting the file

There is text in the file, so the first thing you should try to do is esc2text. In this case, which works like this:

$ php esc2text.php receipt.bin

In this case, I got no output, so I switch to -v to show the commands being found.

$ php esc2text.php receipt.bin  -v
[DEBUG] SetRelativeVerticalPrintPositionCmd 
[DEBUG] GraphicsDataCmd 
[DEBUG] GraphicsDataCmd 
[DEBUG] SetRelativeVerticalPrintPositionCmd 
...

This indicates that there is no text being sent to the receipt, only images. We know from the print-out that the images contain text, so we need a few more utilities.

Recovering images from the receipt

To extract the images, use escimages. It runs like this:

$ php escimages.php --file receipt.bin
[ Image 1: 576x56 ]
[ Image 2: 576x56 ]
[ Image 3: 576x56 ]
[ Image 4: 576x56 ]
[ Image 5: 576x56 ]
[ Image 6: 576x56 ]
[ Image 7: 576x56 ]
[ Image 8: 576x52 ]

This gave me 8 narrow images:

Using ImageMagick’s convert command, these can be combined into one image like this:

convert -append receipt-*.png -density 70 -units PixelsPerInch receipt.png

The result is now the same as what the printer produces, but in crisp digital form:

Recovering text from the receipt

Lastly, tesseract is an open source OCR engine which can recover text from images. This image is a lossless copy of what we sent to the printer, which is an “easy” input for OCR.

$ tesseract receipt.png -
Estimating resolution as 279
Test Receipt for USB Printer 1

Mar 17, 2018
10:12 PM

 

Ticket: 01

 

Item $0,00

Total $0.00

This quality of output is fairly accurate for an untrained OCR engine.

Conclusion

The escpos-tools family of utilities gives some visibility into the contents of ESC/POS binary files.

If you have a use case which requires working with this type of file, then I would encourage you to consider contributing code or example files to the project, so that the utilities can be improved over time.

The project is hosted on GitHub at receipt-print-hq/escpos-tools.

First impressions of the Rust programming language

Thu, 10 May 2018 00:00:00 +0000

I needed to write a small command-line utility recently, and thought that it would be a good chance to finally try out the Rust programming language.

I was hoping to learn the basics and implement this program in a few hours, but I found that it wasn’t trivial to just pick up Rust and start writing programs immediately. The compiler is quite strict about how you use variables, so progress is slow if you don’t really know what you are doing. Still, Rust allows you to produce programs with some valuable properties, so I’m planning to set aside some time to learn it properly.

In the meantime, these are a few of the first impressions that I had of Rust, as somebody completely new to this particular software ecosystem.

Initial installation is easy

I needed to install the rustc compiler and cargo dependency manager. I found that the command packages for Debian testing were up-to-date, so I installed directly from apt.

sudo apt-get install rustc cargo

From there, it took all of one minute to create a file called hello.rs, drop the “Hello world” program, and see it run:

fn main() {
  println!("Hello");
}

The rustc compiler has a simple-enough invocation:

$ rustc hello.rs

And the resulting program ran as expected.

$ ./hello 
Hello

There were no surprises here. It seemed to work just like C so-far.

Output files are large

The output binary file was much larger than I expected, at 246K.

$ ls -Ahl hello
-rwxr-xr-x 1 mike mike 246K Apr 13 21:55 hello

For comparison, I wrote out a similar program in C to see how large it was.

#include <stdio.h>

int main() {
  printf("Hello\n");
  return 0;
}

It compiles and runs with the below commands, was 8K.

$ gcc hello.c -o hello
$ ./hello 
Hello
$ ls -Ahl hello
-rwxr-xr-x 1 mike mike 8.3K Apr 13 21:57 hello

The FAQ explained a few specific reasons why this was the case.

Compile is slow

Compilation took around a quarter of a second. This does not seem like much, but it certainty slower than I expected for such trivial code.

$ time rustc hello.rs
real	0m0.298s

Compared with the C program:

$ time gcc hello.c -o hello
real	0m0.044s

Again, the FAQ points out that zero-cost abstractions incur a compile-time penalty.

Since compilation is a notoriously time-consuming activity with existing languages, I’m hoping that the compile times for Rust are acceptable for small projects.

Legendary compile errors

I’ve read about rustc being very strict, but printing good errors. So, I deleted a ! character and tried to re-compile.

fn main() {
  println("Hello");
}

This code produced an error as expected.

$ rustc hello.rs 
error[E0423]: expected function, found macro `println`
 --> hello.rs:2:3
  |
2 |   println("Hello");
  |   ^^^^^^^ did you mean `println!(...)`?

error: aborting due to previous error

The error message suggested the correct code to use, which is better than what you get from most compilers. I’m pretty happy with that.

Editor support is good

Next, I needed a better Rust editor. Eclipse support for Rust was present, but had a patchy maintenance status.

So I tried adding the Rust plugin for IntelliJ IDEA community edition, which is still a fully open source stack. This was point-and-click. The basic steps follow-

Open IntelliJ and click Configure:

Then Plugins:

Search for “rust”, and click Search in Repositories:

Then click Rust → Install, and wait:

Once it installs, restart the IDE.

The New Project dialog now shows Rust as an option.

I needed the rust-src package to fill out these options:

$ sudo apt-get install rust-src
$ apt-file list rust-src
/usr/src/rustc-1.24.1/src/ ..

In the IntelliJ Rust environment, most editor features worked, but the debugger was not supported.

So, IDE support is mostly there, but it’s not easy to start hacking as something like Java.

Dependency management

I noticed that Cargo produces .lock files which contain checksums and exact versions of dependencies.

As a composer and yarn user, this was familiar.

“Clippy”

I couldn’t use the clippy linter to check my code, because it does not work on the stable Rust releases. The ’nightly’ Rust releases are not available in the Debian archive.

So, for installing rustc, I should have used rustup, not apt-get. Lesson learned!

Documentation

I followed the “Guessing game tutorial” from the official Rust book in the IDE, which taught basic I/O, variable scope and control structures in Rust.

The go-to source reference for Rust is ‘The Rust Book’, which is also collaboratively built and freely licensed.

The Rust Programming Language (Second Edition)

There is also an immense amount of high-quality discussion taking place between Rust users, which I found to be surprisingly accessible as a novice.

ctjhoa/rust-learning on github.com
Questions tagged ‘rust’ on stackoverflow.com

Having never used the language before, I didn’t have to ask any questions to get these basics working, which suggests that the available resources are pretty good.

How to print the characters in an ESC/POS printer code page

Thu, 03 May 2018 00:00:00 +0000

I’ve been working on software that interacts with ESC/POS receipt printers for some time, and a constant source of trouble is the archaic character encoding scheme used on these printers.

Most commonly, non-ASCII characters are accessed by swapping the extended range to a different 128-character code page. The main open source drivers (escpos-php and python-escpos) are both capable of auto-selecting an encoding, but they need a good database of known encodings to power this feature for each individual printer.

Today, I’ll share a small utility that can print out the contents of a code page, like this:

A printer’s documentation vaguely labeled this encoding as “[1] Katakana”. By printing it out, I can see that if I map single-byte half-width Katakana from Code Page 932, it will appear correctly in this code page. That’s type of information you need when you’re asked about it on an issue tracker!

Usage

You will generally find a list of code pages with a corresponding number for each one (0-255) in an ESC/POS printer’s documentation.

This command-line tool then takes a list of code pages to inspect, and will output raw binary that generates a table like the one above when sent to the printer:

php escpos-caracter-table.php NUMBER ...

So to print the code pages 1, 2 and 3 to a binary file, the command would be:

php escpos-character-tables.php 1 2 3 > code-page-1.bin

Next, you need to know how to do raw printing. Raw USB printing on Linux typically works like this:

cat code-page-1.bin > /dev/usb/lp0

For other platforms, it will be different! You will need to do a bit of research on raw printing for your platform if you haven’t tried it before.

The code: escpos-character-tables.php

<?php
/**
 * This standalone script can be used to print the contents of a code page
 * for troubleshooting.
 *
 * Usage: php escpos-caracter-table.php NUMBER ...
 *
 * Code pages are numbered 0-255.
 *
 * The ESC/POS binary will be send to stdout, and should be redirected to a
 * file or printer:
 *
 *   php escpos-caracter-table.php 20 > /dev/usb/lp0
 *
 * @author Michael Billington < michael.billington@gmail.com >
 * @license MIT
 */

/* Sanity check */
if(php_sapi_name() !== "cli") {
    die("This is a command-line script, invoke via php.exen");
}
if(count($argv) < 2) {
    die("At least one code page number must be specifiedn");
}
array_shift($argv);
foreach($argv as $codePage) {
    if(!is_numeric($codePage) || $codePage < 0 || $codePage > 255) {
        die("Code pages must be numbered 0-255");
    }
}

/* Reset */
$str = "\x1b@";
foreach($argv as $codePage) {
    /* Print header, switch code page */
    $str .= "\x1bt" . chr($codePage);
    $str .= "\x1bE\x01Code page $codePage\x1bE\x00\n";
    $str .= "\x1bE\x01  0123456789ABCDEF0123456789ABCDEF\x1bE\x00\n";
    $chars = str_repeat(' ', 128);
    for ($i = 0; $i < 128; $i++) {
        $chars[$i] = chr($i + 128);
    }
    for ($y = 0; $y < 4; $y++) {
        $row = "" . " ";
        $rowHeader = "\x1bE\x01" . strtoupper(dechex($y + 8)) . "\x1bE\x00";
        $row = substr($chars, $y * 32, 32);
        $str .= "$rowHeader $row\n";
    }
}

/* Cut */
$str .= "\x1dV\x41\x03";

/* Output to STDOUT */
file_put_contents("php://stdout", $str);

Optimization: How I made my PHP code run 100 times faster

Thu, 26 Apr 2018 00:00:00 +0000

I’ve had a PHP wikitext parser as a dependency of some of my projects since 2012. It has always been a minor performance bottleneck, and I recently decided to do something about it.

I prepared an update to the code over the course of a month, and achieved a speedup of 142 times over the original implementation.

A lot of the information that I could find online about PHP performance was very outdated, so I decided to write a bit about what I’ve learned. This post walks through the process I used, and the things which were were slowing down my code.

This is a long read – I’ve included examples which show slow code, and what I replaced them with. If you’re a serious PHP programmer, then read on!

Lesson 1: Know when to optimize

Conventional wisdom seems to dictate that making your code faster is a waste of developer time.

I think it makes you a better programmer to occasionally optimize something in a language that you normally work with. Having a well-calibrated intuition about how your code will run is part of becoming proficient in a language, and you will tend to create fewer performance problems if you’ve got that intuition.

But you do need to be selective about it. This code has survived in the wild for over five years, and I think I will still be using it in another five. This code is also a good candidate because it does not access external resources, so there is only one component to examine.

Lesson 2: Write tests

In the spirit of MakeItWorkMakeItRightMakeItFast, I started by checking my test suite so that I could refactor the code with confidence.

In my case, I haven’t got good unit tests, but I have input files that I can feed through the parser to compare with known-good HTML output, which serves the same purpose:

php example.php > out.txt
diff good.txt out.txt

I ran this after every change to the code, so that I could be sure that the changes were not affecting the output.

Lesson 3: Profile your code & Question your assumptions

Code profiling allows you see how each part of your program is contributing to its overall run-time. This helps you to target your optimization efforts.

The two main debuggers for PHP are Zend and Xdebug, which can both profile your code. I have xdebug installed, which is the free debugger, and I use the Eclipse IDE, which is the free IDE. Unfortunately, the built-in profiling tool in Eclipse seems to only support the Zend debugger, so I have to profile my scripts on the command-line.

The best sources of information for this are:

On Debian or Ubuntu, xdebug is installed via apt-get:

sudo apt-get install php-cli php-xdebug

On Fedora, the package is called php-pecl-xdebug, and is installed as:

sudo dnf install php-pecl-xdebug

Next, I executed a slow-running example script with profiling enabled:

php -dxdebug.profiler_enable=1 -dxdebug.profiler_output_dir=. example.php

This produces a profile file, which you can use any valgrind-compatible tools to inspect. I used kcachegrind

sudo apt-get install kcachegrind

And for fedora:

sudo dnf install kcachegrind

You can locate and open the profile on the command-line like so:

ls
kcachegrind cachegrind.out.13385

Before profiling, I had guessed that the best way to speed up the code would be to reduce the amount of string concatenation. I have lots of tight loops which append characters one-at-a-time:

$buffer .= "$c"

Xdebug showed me that my guess was wrong, and I would have wasted a lot of time if I tried to remove string concatenation.

Instead, it was clear that I was

Calculating the same thing hundreds of times over.
Doing it inefficiently.

Lesson 4: Avoid multibyte string functions

I had used functions from the mbstring extension (mb_strlen, mb_substr) to replace strlen and substr throughout my code. This is the simplest way to add UTF-8 support when iterating strings, is commonly suggested, and is a bad idea.

What developers do

If you have an ASCII string in PHP and want to iterate over each byte, the idiomatic way to do it is with a for loop which indexes into the string, something like this:

<?php
// Make a test string
$testString = str_repeat('a', 60000);
// Loop through test string
$len = strlen($testString);
for($i = 0; $i < $len; $i++) {
  $c = substr($testString, $i, 1);
  // Do work on $c
  // ...
}

I’ve used substr here so that I can show that it has the same usage as mb_substr, which generally operates on UTF-8 characters. The idiomatic PHP for iterating over a multibyte string one character at a time would be:

<?php
// Make a test string
$testString = str_repeat('a', 60000);
// Loop through test string
$len = mb_strlen($testString);
for($i = 0; $i < $len; $i++) {
  $c = mb_substr($testString, $i, 1);
  // Do work on $c
  // ...
}

Since mb_substr needs to parse UTF-8 from the start of the string each time it is called, the second snippet runs in polynomial time, where the snippet that calls substr in a loop is linear.

With a few kilobytes of input, this makes mb_substr unacceptably slow.

Averaging over 10 runs, the mb_substr snippet takes 4.23 seconds, while the snippet using substr takes 0.03 seconds.

What developers can do instead

Split your strings into bytes or characters before you iterate, and write methods which operate on arrays rather than strings.

You can use str_split to iterate over bytes:

<?php
// Make a test string
$testString = str_repeat('a', 60000);
// Loop through test string
$testArray = str_split($testString);
$len = count($testArray);
for($i = 0; $i < $len; $i++) {
  $c = $testArray[$i];
  // Do work on $c
  // ...
}

And for Unicode strings, use preg_split. I learned about this trick from StackOverflow, but it might not be the fastest way. Please leave a comment if you have an alternative!

<?php
// Make a test string
$testString = str_repeat('a', 60000);
// Loop through test string
$testArray = preg_split('//u', $testString, -1, PREG_SPLIT_NO_EMPTY);
$len = count($testArray);
for($i = 0; $i < $len; $i++) {
  $c = $testArray[$i];
  // Do work on $c
  // ...
}

By converting the string to an array, you can pay the penalty of decoding the UTF-8 up-front. This is a few milliseconds at the start of the script, rather than a few milliseconds each time you need to read a character.

After discovering this faster alternative to mb_substr, I systematically removed every mb_substr and mb_strlen from the code I was working on.

Lesson 5: Optimize for the most common case

Around 50% of the remaining runtime was spent in a method which expanded templates.

To parse wikitext, you first need to expand templates, which involves detecting tags like {{ template-name | foo=bar }} and <noinclude></noinclude>.

My 40 kilobyte test file had fewer than 100 instances of { and <, | and =, so I added a short-circuit to the code to skip most of the processing, for most of the characters.

<?php
self::$preprocessorChars = [
    '<' => true,
    '=' => true,
    '|' => true,
    '{' => true
];

// ...
for ($i = 0; $i < $len; $i++) {
    $c = $textChars[$i];
    if (!isset(self::$preprocessorChars[$c])) {
        /* Fast exit for characters that do not start a tag. */
        $parsed .= $c;
        continue;
    }
   // ... Slower processing 
}

The slower processing is now avoided 99.75% of the time.

Checking for the presence of a key in a map is very fast. To illustrate, here are two examples which each branch on { and <, | and =.

This one uses a map to check each character:

<?php
// Make a test string
$testString = str_repeat('a', 600000);
$chars = [
    '<' => true,
    '=' => true,
    '|' => true,
    '{' => true
];
// Loop through test string
$testArray = preg_split('//u', $testString, -1, PREG_SPLIT_NO_EMPTY);
$len = count($testArray);
$parsed = "";
for($i = 0; $i < $len; $i++) {
  $c = $testArray[$i];
  if(!isset($chars[$c])) {
    $parsed .= $c;
    continue;
  }
  // Never executed
}

While one uses no map, and has four !== checks instead:

<?php
// Make a test string
$testString = str_repeat('a', 600000);
// Loop through test string
$testArray = preg_split('//u', $testString, -1, PREG_SPLIT_NO_EMPTY);
$len = count($testArray);
$parsed = "";
for($i = 0; $i < $len; $i++) {
  $c = $testArray[$i];
  if($c !== "<" && $c !== "=" && $c !== "|" && $c !== "{") {
    $parsed .= $c;
    continue;
  }
  // Never executed
}

Even though the run time of each script includes the generation of a 600kB test string, the difference is still visible:

Averaging over 10 runs, the code took 0.29 seconds when using a map, while it took 0.37 seconds to run the example with used four !== statements.

I was a little surprised by this result, but I’ll let the data speak for itself rather than try to explain why this is the case.

The next item to appear in the profiler was the copious use of array_slice. My code uses recursive descent, and was constantly slicing up the input to pass around information. The array slicing had replaced earlier string slicing, which was even slower.

I refactored the code to pass around the entire string with indices rather than actually cutting it up.

As a contrived example, these scripts each use a (very unnecessary) recursive-descent parser to take words from the dictionary and transform them like this:

example --> (example!)

The first example slices up the input array at each recursion step:

<?php
function handleWord(string $word) {
  return "($word!)\n";
}

/**
 * Parse a word up to the next newline.
 */
function parseWord(array $textChars) {
  $parsed = "";
  $len = count($textChars);
  for($i = 0; $i < $len; $i++) {
    $c = $textChars[$i];
    if($c === "\n") {
      // Word is finished because we hit a newline
      $start = $i + 1; // Past the newline
      $remainderChars = array_slice($textChars, $start , $len - $start);
      return array('parsed' => handleWord($parsed), 'remainderChars' => $remainderChars);
    }
    $parsed .= $c;
  }
  // Word is finished because we hit the end of the input
  return array('parsed' => handleWord($parsed), 'remainderChars' => []);
}

/**
 * Accept newline-delimited dictionary
 */
function parseDictionary(array $textChars) {
  $parsed = "";
  $len = count($textChars);
  for($i = 0; $i < $len; $i++) {
    $c = $textChars[$i];
    if($c === "\n") {
      // Not a word...
      continue;
    }
    // This is part of a word
    $start = $i;
    $remainderChars = array_slice($textChars, $start, $len - $start);
    $result = parseWord($remainderChars);
    $textChars = $result['remainderChars'];
    $len = count($textChars);
    $i = -1;
    $parsed .= $result['parsed'];
  }
  return array('parsed' => $parsed, 'remainderChars' => []);
}

// Load file, split into characters, parse, print result
$testString = file_get_contents("words");
$testArray = preg_split('//u', $testString, -1, PREG_SPLIT_NO_EMPTY);
$ret = parseDictionary($testArray);
file_put_contents("words2", $ret['parsed']);

While the second one always takes an index into the input array:

<?php
function handleWord(string $word) {
  return "($word!)\n";
}

/**
 * Parse a word up to the next newline.
 */
function parseWord(array $textChars, int $idxFrom = 0) {
  $parsed = "";
  $len = count($textChars);
  for($i = $idxFrom; $i < $len; $i++) {
    $c = $textChars[$i];
    if($c === "\n") {
      // Word is finished because we hit a newline
      $start = $i + 1; // Past the newline
      return array('parsed' => handleWord($parsed), 'remainderIdx' => $start);
    }
    $parsed .= $c;
  }
  // Word is finished because we hit the end of the input
  return array('parsed' => handleWord($parsed), $i);
}

/**
 * Accept newline-delimited dictionary
 */
function parseDictionary(array $textChars, int $idxFrom = 0) {
  $parsed = "";
  $len = count($textChars);
  for($i = $idxFrom; $i < $len; $i++) {
    $c = $textChars[$i];
    if($c === "\n") {
      // Not a word...
      continue;
    }
    // This is part of a word
    $start = $i;
    $result = parseWord($textChars, $start);
    $i = $result['remainderIdx'] - 1;
    $parsed .= $result['parsed'];
  }
  return array('parsed' => $parsed, 'remainderChars' => []);
}

// Load file, split into characters, parse, print result
$testString = file_get_contents("words");
$testArray = preg_split('//u', $testString, -1, PREG_SPLIT_NO_EMPTY);
$ret = parseDictionary($testArray);
file_put_contents("words2", $ret['parsed']);

The run-time difference between these examples is again very pronounced:

Averaging over 10 runs, the code snippet which extracts sub-arrays took 3.04 seconds, while the code that passes around the entire array ran in 0.0302 seconds.

It’s amazing that such an obvious inefficiency in my code had been hiding behind larger problems before.

Lesson 7: Use scalar type hinting

Scalar type hinting looks like this:

function foo(int bar, string $baz) {
...
}

This is the secret PHP performance feature that they don’t tell you about. It does not actually change the speed of your code, but does ensure that it can’t be run on the slower PHP releases before 7.0.

PHP 7.0 was released in 2015, and it’s been established that it is twice as fast as PHP 5.6 in many cases.

I think it’s reasonable to have a dependency on a supported version of your platform, and performance improvements like this are a good reason to update your minimum supported version of PHP.

By introducing scalar type hints, you can ensure that your software does not appear to “work” in a degraded state.

Simplifying the compatibility landscape will also make performance a more tractable problem.

Lesson 8: Ignore advice that isn’t backed up by (relevant) data

While I was updating this code, I found a lot of out-dated claims about PHP performance, which did not hold true for the versions that I support.

To call out a few myths that still seem to be alive:

Style of quotes impacts performance.
Use of by-val is slower than by-ref for variable passing.
String concatenation is bad for performance.

I attempted to implement each of these, and wasted a lot of time. Thankfully, I was measuring the execution time and using version control, so it was easy for me to identify and discard changes which had a negligible or negative performance impact.

If somebody makes a claim about something being fast or slow in PHP, best assume that it doesn’t apply to you, unless you see some example code with timings on a recent PHP version.

Conclusion

If you’ve read this far, then I hope you’ve seen that modern PHP is not an intrinsically slow language. A significant speed-up is probably achievable on any real-world code-base with such endemic performance issues.

To repeat the graphic from the introduction: the test file could be parsed in 20.65 seconds on the old code, and 0.145 seconds on the improved code (averaging over 10 runs, as before).

At this point I declared my efforts “good enough” and moved on. The job is done: although another pass could speed it up further, this code is no longer slow enough to justify the effort.

Since I’ve closed the book on PHP 5, I would be interested in knowing whether there is a faster way to parse UTF-8 with the new IntlChar functions, but I’ll have to save that for the next project.

Now that you’ve seen some very inefficient ways to write PHP, I also hope that you will be able to avoid introducing similar problems in your own projects!

Using a receipt printer with the Amazon Echo

Thu, 19 Apr 2018 00:00:00 +0000

Today, I’m sharing this write-up by fellow developer Chris, who used the escpos-php thermal printing library as part of a setup which printed shopping lists via voice commands, using the Alexa Voice Service API to send the lists back to a Raspberry Pi.

“Naturally, the easiest solution […] is to print it in thermal paper… Now, combine this with a voice interface, such as ALEXA, and you made yourself a voice controlled list printer.”

Amazon ALEXA Lists Exercise – Put’em on Paper

I found out about this one through a blog comment, and it’s a recommended read for anybody who is interested in how all of these parts integrate.

When I first uploaded this printing library four years ago, the Amazon Echo did not exist yet, and I was solving a very specific problem. For an old technology, it’s interesting to see that new applications for thermal printers are still appearing, and I’m certainly glad to see my software popping up in cool projects like this.

Raster to vector conversion tips

Thu, 12 Apr 2018 00:00:00 +0000

I have recently been working with some low-resolution bitmap fonts for a few projects, which needed to be re-sized for different uses.

I’ll share here a few tricks that I use to get the detail out of each letter as a vector, so that it can be rendered at a higher resolution.

Example

A good example might be this picture of a hieroglyph from the WikiHiero MediaWiki extension, which is 28 pixels wide:

Scaled up, it looks like this:

So small images become very pixelated when you resize them. The good news is that even from a small image, there is quite a lot of detail which we can use. If we’re smart about it, the glyph can be rendered like this:

You can still see some artifacts because of the low resolution of the input, but it’s clearly an improvement.

You will need

ImageMagick for raster operations.
potrace to trace the image.
Inkscape to produce a high-quality raster.

Steps

Prepare

The tracing program will convert the image to pure black & white as its first step. These transformations make sure that the detail is preserved for tracing.

Padding by 10px on every side to reduce distortion around the edges.
Scaling by 10x with interpolation
Convert transparency to white

convert hiero_A1.png -bordercolor white -border 10x10 \
    -resize 1000% -flatten hiero_A1.pnm

The 29x38 grey+alpha input becomes a blurry 490 x 580 greymap surrounded by whitespace.

This preparation is important, because a large blurry graymap will retain a lot more detail than the original image when a threshold is applied to convert it pure black and white:

Trace

The potrace program will threshold and trace the input image. Here, we will produce an SVG so that we can make it transparent.

The k value affects the threshold operation. It can be increased for a bolder, darker glyph, or reduced for a finer one.

potrace hiero_A1.pnm -k 0.30 --svg

This gives you an SVG with padding:

If you only want a vector, then you can stop here. The next steps will reproduce a smaller PNG with transparency and the correct padding.

I couldn’t find a reliable way to programmatically crop the image back to its original padding as an SVG, but in my case I needed to convert it back to a bitmap anyway, so I cropped it later.

Render

Use Inkscape to convert the SVG back to a large PNG. The output size here is twice as large as the file we traced, just to leave plenty of pixels to work with.

inkscape -z -e hiero_A1_big.png hiero_A1.svg -w 980

Like the SVG, there is still a lot of empty space surrounding the image here. The image is now a PNG with a transparent background, and unlike the file we traced, the edges of the curves are now anti-aliased.

Crop

Everything is 20x its original size, to get the image, we need to drop 200px of padding from the left and top, then read 580x760 pixels (20 times the 29x38 start).

convert hiero_A1_big.png -crop 580x760+200+200 hiero_A1_cropped.png

This produces a 580x760 image in the same aspect ratio as the original input file.

Scale down

In my case, I only needed to double the resolution of the input file, so I scaled this file down from there.

hiero_A1_cropped.png -resize 58x76 hiero_A1_outp1.png

Success!

As a script

These steps are a breakdown of a script that I wrote for doubling the size of a PNG image so that it would look better on newer high-DPI displays.

Usage:

./tracepng.sh foo.png

Where tracepng.sh is:

#!/bin/bash
# A script to upscale small bitmaps in PNG format.
# Pad, upscale, trace, render, crop then downscale.
if [ $# != 1 ]; then
  echo "Usage $0 input.png"
  exit
fi
set -exu
# Names of all the files we will produce
INP_FILE=$1
SVG_FILE="${INP_FILE%.*}.svg"
PNM_FILE="${INP_FILE%.*}.pnm"
LARGE_FILE="${INP_FILE%.*}_big.png"
LARGE_FILE_CROPPED="${INP_FILE%.*}_cropped.png"
OUTP_FILE="${INP_FILE%.*}_outp1.png"
COMPARISON_FILE="${INP_FILE%.*}_outp2.png"

# Width originally
# https://stackoverflow.com/questions/4670013/fast-way-to-get-image-dimensions-not-filesize
IMG_WIDTH=$(identify -format "%w" "$INP_FILE")
IMG_HEIGHT=$(identify -format "%h" "$INP_FILE")
TARGET_WIDTH=$((IMG_WIDTH * 2))
TARGET_HEIGHT=$((IMG_HEIGHT * 2))

# Make huge raster w/ border (whitespace is your friend for black/white interpolation and tracing), then convert to SVG
convert ${INP_FILE} -bordercolor white -border 10x10 -resize 1000% -flatten ${PNM_FILE}

# https://en.wikipedia.org/wiki/Potrace
potrace ${PNM_FILE} -k 0.30 --svg > ${SVG_FILE}

# Target width for intermediate file
EXPANDED_WIDTH=$(((IMG_WIDTH + 20) * 20))
EXPANDED_HEIGHT=$(((IMG_HEIGHT + 20) * 20))
INNER_WIDTH=$((IMG_WIDTH * 20))
INNER_HEIGHT=$((IMG_HEIGHT * 20))
# https://stackoverflow.com/questions/9853325/how-to-convert-a-svg-to-a-png-with-image-magick
inkscape -z -e ${LARGE_FILE} ${SVG_FILE} -w ${EXPANDED_WIDTH}

# Cut new edges off
# http://www.imagemagick.org/Usage/crop/
convert ${LARGE_FILE} -crop ${INNER_WIDTH}x${INNER_HEIGHT}+200+200 ${LARGE_FILE_CROPPED}
convert ${LARGE_FILE_CROPPED} -resize ${TARGET_WIDTH}x${TARGET_HEIGHT} ${OUTP_FILE}

Acknowledgment

The images here are from WikiHiero, and can be remixed under the GNU General Public License 2.0.

Git Introductory Material - Tips for Users, Admins and Git Champions

Thu, 05 Apr 2018 00:00:00 +0000

This is a collection of tips that I have found useful for introducing software developers to the Git version control system.

Bookmarks for reference

The “feature branch” workflow as documented by Atlassian.
Flight manual.
Git simulator.

For users

When you say that you use Git, most developers will assume that you are familiar with the Git CLI. Regardless of whether you plan to use the CLI long-term, when you do a web search to find out something about Git, you should be prepared to read answers that apply to the Git CLI.

Since you will be asking a lot of questions in the first few weeks, you will have less work to do if you use the same interface as everybody else. Rest assured that there are plenty of graphical tools and IDE plugins that you can try out later.

Takeaway: Use the Git CLI while you are learning

If you have more than one person working with the code, then you will want to share your repository between them.

You need product that meets your budget, workflow, security and licensing needs: As of 2018, you really only need to consider GitHub, Bitbucket, Phabricator or GitLab to make a thorough comparison.

Takeaway: Use a collaborative environment that supports ‘pull requests’

Repository admins

Aim to make your setup both beginner-friendly, and beginner-proof.

Git is not very opinionated, and has features which will support many advanced workflows. I would suggest that you save these features until you have a critical mass of experienced users who are comfortable troubleshooting them. The feature-branch workflow is beginner-friendly.

Ensure that your whole team has mastered the basics before you try to get creative.

Takeaway: Just use the feature branch workflow initially

There are a few switches that you should set in your system to prevent mistakes from disrupting users who do the right thing:

disable forced-push on the master branch
require a pull request and one approval to modify the master branch

The name of these settings will depend on the software you are hosting on, but for example, on GitHub, this is known as a “protected branch”:

These settings prevent people from re-writing history on the main branch, which is bad Git etiquette. Usually people are trying to re-write history with good intentions, such as trying to scrub credentials, huge files, or other extras that were accidentally included.

Good software process aside, a lightweight code review process will make unintended changes more visible, so that you can give developers a chance to fix their mistakes without propagating them to other developers.

Takeaway: Don’t allow direct access to main

Git champions

There are a few setup steps that users may not discover on their own, which avoid common frustrations.

A classic beginner complaint about Git is that it makes you enter passwords all the time. Usually what is really happening here is that the user is working with a repository which was cloned over HTTPS, and has not set up SSH keys.

Using SSH keys involves generating a key and uploading the public portion of the key to your hosting software. Subsequently, you will never be prompted for credentials.

Takeaway: Show users how to clone with SSH

Immediately after you make your first commit, Git will point out that the author name has not been set, and suggest that you should amend the commit. I think that the experience is far better when this is configured proactively:

git config --global user.name "Example Bob"
git config --global user.email "bob@example.com"

The first time that this user finds a merge conflict, they will appreciate having a graphical, point-and-click merge like meld.

Follow OS-specific instructions, then run:

git config --global difftool meld

Takeaway: Step new users through the user and difftool settings.

Conclusion

The dominance of Git as a software revision control tool means that it’s worth investing some time to learn it properly.

If you get one thing out of this post, though, it’s that you shouldn’t attempt to be creative about your use of Git until you know what you are doing.

I’ve tried to avoid mentioning intermediate or advanced features in this post. As you become more comfortable with the basics, you can use the flight rules and simulator at the top of the post to discover some of the things git can do, and the situations where you might want to use each feature.

How to communicate with USB and networked devices from in-browser Javascript

Thu, 29 Mar 2018 00:00:00 +0000

I recently combined a few tools on Linux to create a local Websocket listener, which could forward raw data to a USB printer, so that it could be accessed using Javascript in a web browser.

Why would you want this? I have point of sale applications (POS) in mind, which need to send raw data to a printer. For these applications, the browser and operating system print systems are not appropriate, since they prompt, spool, and badly render pages by converting them to low-fidelity raster images.

Web interfaces are becoming common for point-of-sale applications. The web page could be served from somewhere outside your local network, which is why we need to get the client-side Javascript involved.

The tools

To run on the client computer:

And to generate the print data on the webserver:

escpos-php

We will use these tools to provide some plumbing, so that we can retrieve the print data, and send it off to the printer from client-side Javascript.

Client computer

The client computer was a Linux desktop system. Both of the tools we need are available in the Debian repositories:

sudo apt-get install websockify socat

Listen for websocket connections on port 5555 and pass them to localhost:7000:

websockify 5555 localhost:7000

Listen for TCP connections on localhost port 7000 and pass them to the USB device (more advanced version of this previous post):

socat -u TCP-LISTEN:7000,fork,reuseaddr,bind=127.0.0.1 OPEN:/dev/usb/lp0

Web page

I made a self-contained web-page to provide a button which requested a print file from the network and passed it to the local websocket.

This is slightly modified from a similar example that I used for a previous project.

<html>
<head>
    <meta charset="UTF-8">
    <title>Web-based raw printing example</title>
</head>
<body>
<h1>Web-based raw printing example</h1>

<p>This snippet forwards raw data to a local websocket.</p>

<form>
  <input type="button" onclick="directPrintBytes(printSocket, [0x1b, 0x40, 0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x77, 0x6f, 0x72, 0x6c, 0x64, 0x0a, 0x1d, 0x56, 0x41, 0x03]);" value="Print test string"/>
  <input type="button" onclick="directPrintFile(printSocket, 'receipt-with-logo.bin');" value="Load and print 'receipt-with-logo'" />
</form>

<script type="text/javascript">
/**
 * Retrieve binary data via XMLHttpRequest and print it.
 */
function directPrintFile(socket, path) {
  // Get binary data
  var req = new XMLHttpRequest();
  req.open("GET", path, true);
  req.responseType = "arraybuffer";
  console.log("directPrintFile(): Making request for binary file");
  req.onload = function (oEvent) {
    console.log("directPrintFile(): Response received");
    var arrayBuffer = req.response; // Note: not req.responseText
    if (arrayBuffer) {
      var result = directPrint(socket, arrayBuffer);
      if(!result) {
        alert('Failed, check the console for more info.');
      }
    }
  };
  req.send(null);
}

/**
 * Extract binary data from a byte array print it.
 */
function directPrintBytes(socket, bytes) {
  var result = directPrint(socket, new Uint8Array(bytes).buffer);
  if(!result) {
    alert('Failed, check the console for more info.');
  }
}

/**
 * Send ArrayBuffer of binary data.
 */
function directPrint(socket, printData) {
  // Type check
  if (!(printData instanceof ArrayBuffer)) {
    console.log("directPrint(): Argument type must be ArrayBuffer.")
    return false;
  }
  if(printSocket.readyState !== printSocket.OPEN) {
    console.log("directPrint(): Socket is not open!");
    return false;
  }
  // Serialise, send.
  console.log("Sending " + printData.byteLength + " bytes of print data.");
  printSocket.send(printData);
  return true;
}

/**
 * Connect to print server on startup.
 */
var printSocket = new WebSocket("ws://localhost:5555", ["binary"]);
printSocket.binaryType = 'arraybuffer';
printSocket.onopen = function (event) {
  console.log("Socket is connected.");
}
printSocket.onerror = function(event) {
  console.log('Socket error', event);
};
printSocket.onclose = function(event) {
  console.log('Socket is closed');
}
</script>
</body>
</html>

Webserver

On an Apache HTTP webserver, I uploaded the above webpage, and a file with some raw print data, called receipt-with-logo.bin. This file was generated with escpos-php and is available in the repository:

receipt-with-logo.bin

For reference, the test file receipt-with-logo.bin looks like this when printed:

Test

I opened up the web page on the client computer with socat, websockify and an Epson TM-T20II connected. After clicking the “Print” button, the file was sent to my printer. Success!

Because I wasn’t closing the websocket connection, only one browser window could access the printer at a time. Still, it’s a good demo of the basic idea.

To take this from an example to something you might deploy, you would basically just need to keep socat and websockify running in the background as a service (eg. via systemd), close the socket when it’s not being used, and integrate it into a real app.

Different printers, different forwarding

The socat tool can connect to USB, Serial, or Ethernet printers fairly easily.

USB

Forward TCP connections from port 7000 to the receipt printer at /dev/usb/lp0:

socat TCP4-LISTEN:7000,fork /dev/usb/lp0

You can also access the device files directly under /sys/bus/usb/devices/

Serial

Forward TCP connections from port 7000 to the receipt printer at /dev/usb/ttyS0:

socat TCP4-LISTEN:7000,fork /dev/usb/ttyS0

Network

Forward TCP connections from port 7000 to the receipt printer at 10.1.2.3:9100:

socat -u TCP-LISTEN:7000,fork,reuseaddr,bind=127.0.0.1 TCP4-CONNECT:10.1.2.3:9100

You can forward websocket connections directly to an Ethernet printer with websockify:

socat -u TCP-LISTEN:7000,fork,reuseaddr,bind=127.0.0.1 localhost:7000

Other types of printer

If you have another type of printer, such as one accessible only via smbclient or lpr, then you will need to write a helper script.

Direct printing is faster, so I don’t use this method. Check the socat EXEC documentation or man socat if you want to try this.

Future

I’ve had a lot of questions on the escpos-php bug tracker from users who are attempting to print from cloud-hosted apps, which is why I tried this setup.

The browser is a moving target. I have previously written receipt-print-hq/chrome-raw-print, a dedicated app for forwarding WebSocket connections to USB, but that will stop working in a few months when Chrome apps are discontinued. Some time later, WebUSB should become available to make this type of printer available in the browser, which should be invaluable for connecting to accessories in point-of-sale setups.

The available tools for generating ESC/POS (receipt printer) binary from the browser are a long way off reaching feature parity with the likes of escpos-php and python-escpos. If you are looking for a side-project, then this a good choice.

Lastly, the socat -u flag makes this all unidirectional, but many types of devices (not just printers) can respond to commands. I couldn’t get the end-to-end path to work without this flag, so don’t expect to be able to read from the printer without doing some extra work.

Useful links

Some links that I found while setting this up-

Get the code

The code which I used to build this demo is available in the mike42/escpos-snippets repository on GitHub.

Automating LXC container creation with Ansible

Thu, 22 Mar 2018 00:00:00 +0000

LXC is a Linux container technology that I use for both development and production setups hosted on Debian.

This type of container acts a lot like a lightweight virtual machine, and can be administered with standard linux tools. When configured over SSH, you should be able to use the same scripts against either an LXC container or VM without noticing the difference.

This setup will provision “privileged” containers behind a NAT, which is a setup that is most useful for a developer workstation. A setup in a server rack would be more likely to use “unprivileged” containers on a host bridge, which is slightly more complex to set up. The good news is that the guest container will behave very similarly once it’s provisioned, so developers shouldn’t need to adapt their code to those details either.

Manual setup of an LXC container

You need to know how to do something manually before you can automate it.

The best reference guide for this is the current Debian documentation. This is a shorter version of those instructions, with only the parts we need.

Packages

Everything you need for LXC is in the lxc Debian package:

$ sudo apt-get install lxc
...
The following additional packages will be installed:
  bridge-utils debootstrap liblxc1 libpam-cgfs lxcfs python3-lxc uidmap
Suggested packages:
  btrfs-progs lvm2
The following NEW packages will be installed:
  bridge-utils debootstrap liblxc1 libpam-cgfs lxc lxcfs python3-lxc uidmap
0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,367 kB of archives.
After this operation, 3,762 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
...

Network

Enable the LXC bridge, and start it up:

echo 'USE_LXC_BRIDGE="true"' | sudo tee -a /etc/default/lxc-net
sudo systemctl start lxc-net

This gives you an internal network for your containers to connect to. From there, they can connect out to the Internet, or communicate with each-other:

...
3: lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 scope global lxcbr0
       valid_lft forever preferred_lft forever

Defaults

Instruct LXC to attach a NIC to this network each time you make a containers:

$ sudo vi /etc/lxc/default.conf

Replace that file with:

lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx

You can then create a ’test1’ box, using the Debian image online. Note the output here indicates that the container has no SSH server or root password.

$ sudo lxc-create --name test1 --template=download -- --dist=debian --release=stretch --arch=amd64
Setting up the GPG keyring
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created a Debian container (release=stretch, arch=amd64, variant=default)

To enable sshd, run: apt-get install openssh-server

For security reason, container images ship without user accounts
and without a root password.

Use lxc-attach or chroot directly into the rootfs to set a root password
or create user accounts.

The container is created in a stopped state. Start it up now:

$ sudo lxc-start --name test1

It now appears with an automatically assigned IP.

$ sudo lxc-ls --fancy
NAME  STATE   AUTOSTART GROUPS IPV4       IPV6 
test1 RUNNING 0         -      10.0.3.250 -

Start by getting your SSH public key ready. You can locate at ~/.ssh/id_rsa.pub. You can use ssh-keygen to create this if it doesn’t exist.

To SSH in, you need to install an SSH server, and get this public key into the /root/authorized_keys file in the container.

$ sudo lxc-attach --name test1
root@test1:/# apt-get update
root@test1:/# apt-get -y install openssh-server
root@test1:/# mkdir -p ~/.ssh
root@test1:/# echo "ssh-rsa (public key) user@host" >> ~/.ssh/authorized_keys

Type exit or press Ctrl+D to quit, and try to log in from your regular account over SSH:

$ ssh root@10.0.3.250
The authenticity of host '10.0.3.250 (10.0.3.250)' can't be established.
ECDSA key fingerprint is SHA256:EWH1zUW4BEZUzfkrFL1K+24gTzpd8q8JRVc5grKaZfg.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.3.250' (ECDSA) to the list of known hosts.
Linux test1 4.14.0-3-amd64 #1 SMP Debian 4.14.13-1 (2018-01-14) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@test1:~#

And you’re in. You may be surprised how minimal the LXC images are by default, but the full power of Debian is available from apt-get.

This container is not configured to start on boot. For that, you would add this line to /var/lib/lxc/test1/config:

lxc.start.auto = 1

Teardown

To stop the test1 container and then delete it permanently, run:

sudo lxc-stop --name test1
sudo lxc-destroy --name test1

Automated setup of LXC containers with Ansible

Now that the basic steps have been done manually, I’ll show you how to Ansible to create a set of LXC containers. If you haven’t used it before, Ansible is an automation tool for managing computers. At its heart, it just logs into machines and runs things. These scripts are an approximate automation of the steps above, so that you can create 10 or 100 containers at once if you need to.

I use this method on a small project that I maintain on GitHub called ansible-live, which bootstraps a containerized training environment for Ansible.

Host setup

You need a few packages and config files on the host. In addition to the lxc package, we need lxc-dev and the lxc-python2 python package to manage the containers from Ansible:

- hosts: localhost
  connection: local
  become: true
  vars:
  - interface: lxcbr0

  tasks:
  - name: apt lxc packages are installed on host
    apt: name={{ item }}
    with_items:
    - lxc
    - lxc-dev
    - python-pip

  - copy:
      dest: /etc/default/lxc-net
      content: |
        USE_LXC_BRIDGE="true"

  - copy:
      dest: /etc/lxc/default.conf
      content: |
        lxc.network.type = veth
        lxc.network.link = {{ interface }}
        lxc.network.flags = up
        lxc.network.hwaddr = 00:16:3e:xx:xx:xx

  - service:
      name: lxc-net
      state: started

  - name: pip lxc packages are installed on host
    pip:
      name: "{{ item }}"
    with_items:
    - lxc-python2
    run_once: true

This can be executed with this command:

ansible-playbook setup.yml --ask-become-pass --diff

Container creation

Add a file called inventory to specify the containers to use. These are two IP addresses in the range of the LXC network.

deb1 ansible_host=10.0.3.100
deb2 ansible_host=10.0.3.101

For local work, I find it easier to set an IP address with Ansible and use the /etc/hosts file, which is why IP addresses are included here. Without it, you need to wait for each container to boot, then detect its IP address before you can log in.

Add this to setup.yml

- hosts: all
  connection: local
  become: true
  vars:
  - interface: lxcbr0
  tasks:
  - name: Load in local SSH key path
    set_fact:
      my_ssh_key: "{{ lookup('env','HOME') }}/.ssh/id_rsa.pub"

  - name: interface device exists
    command: ip addr show {{ interface }}
    changed_when: false
    run_once: true

  - name: Local user has an SSH key
    command: stat {{ my_ssh_key }}
    changed_when: false
    run_once: true

  - name: containers exist and have local SSH key
    delegate_to: localhost
    lxc_container:
      name: "{{ inventory_hostname }}"
      container_log: true
      template: debian
      state: started
      template_options: --release stretch
      container_config:
        - "lxc.network.type = veth"
        - "lxc.network.flags = up"
        - "lxc.network.link = {{ interface }}"
        - "lxc.network.ipv4 = {{ ansible_host }}/24"
        - "lxc.network.ipv4.gateway = auto"
      container_command: |
        if [ ! -d ~/.ssh ]; then
          mkdir ~/.ssh
          echo "{{ lookup('file', my_ssh_key) }}" | tee -a ~/.ssh/authorized_keys
          sed -i 's/dhcp/manual/' /etc/network/interfaces && systemctl restart network
        fi

In the next block of setup.yml, use keyscan to get the SSH keys of each machine as it becomes available.

- hosts: all
  connection: local
  become: false
  serial: 1
  tasks:
  - wait_for: host={{ ansible_host }} port=22

  - name: container key is up-to-date locally
    shell: ssh-keygen -R {{ ansible_host }}; (ssh-keyscan {{ ansible_host }} >> ~/.ssh/known_hosts)

Lastly, jump in via SSH and install python. This is required for any follow-up configuration that uses Ansible.

- hosts: all
  gather_facts: no
  vars:
  - ansible_user: root
  tasks:
  - name: install python on target machines
    raw: which python || (apt-get -y update && apt-get install -y python)

Next, you can use this one-liner to create the two containers listed in the inventory.

ansible-playbook setup.yml --ask-become-pass --diff

Scaling to hundreds of containers

Now that you have created two containers, it is easy enough to see how you would make 20 containers by making a bigger inventory:

$ for i in {1..20}; do echo deb$(printf "%03d" $i).example.com ansible_host=10.0.3.$((i+1)); done | tee inventory
deb001.example.com ansible_host=10.0.3.2
deb002.example.com ansible_host=10.0.3.3
deb003.example.com ansible_host=10.0.3.4
...

You can then run the playbook again:

ansible-playbook -i inventory setup.yml --ask-become-pass

This produces 20 machines after a few minutes.

The tasks running during this setup were mostly rync processes (copying the container contents), plus the network waiting to retrieve python many times. If you need to optimise to frequent container spin-ups, LXC supports storage back-ends that have copy-on-write, and you can either cache package downloads with a local webserver, or build some packages into the template.

Running these 20 containers plus a Debian desktop, I found that my computer was using just 2.9GB of RAM, so I figured I would test 200 empty containers at once.

for i in {1..200}; do echo deb$(printf "%03d" $i).example.com ansible_host=10.0.3.$((i+1)); done > inventory
ansible -i inventory setup.yml

It took truly a very long time to add Python to each install, but the result is what I would expect:

$ sudo lxc-ls --fancy
NAME               STATE   AUTOSTART GROUPS IPV4       IPV6 
deb001.example.com RUNNING 0         -      10.0.3.2   -    
deb002.example.com RUNNING 0         -      10.0.3.3   -    
deb003.example.com RUNNING 0         -      10.0.3.4   -    
...
deb198.example.com RUNNING 0         -      10.0.3.199 -    
deb199.example.com RUNNING 0         -      10.0.3.200 -    
deb200.example.com RUNNING 0         -      10.0.3.201 -

The base resource usage of an idle container is absolutely tiny, around 13 megabytes - the system moved from 2.9GB to 5.4GB of RAM used when I added 180 containers. Containers clearly have a lower overhead than VM’s, since no RAM has been reserved here.

Software updates

The containers are updated just like regular VM’s-

apt-get update
apt-get dist-upgrade

Backups

In this setup, the container’s content is stored under /var/lib/lxc/. As long as the container is stopped, you can safely use tar or rsync to make a full copy:

sudo tar -czf deb001.20180209.tar.gz /var/lib/lxc/deb001.example.com/
rsync -avz /var/lib/lxc/deb001.example.com/ remote-computer@example.com:/backups/deb001.example.com/

Full-machine snapshots are also available on the Ceph or LVM back-ends, if you use those.

Teardown

The same Ansible module can be used to delete all of these machines in a few seconds.

- hosts: all
  connection: local
  become: true
  tasks:
  - name: Containers do not exist
    delegate_to: localhost
    lxc_container:
      name: "{{ inventory_hostname }}"
      state: absent

ansible-playbook -i inventory teardown.yml --ask-become-pass

Conclusion

Hopefully this post has given you some insight into one way that Linux containers can be used. I have found LXC to be a great technology to work with for standalone setups, and regularly use the same scripts to configure either an LXC container or a VM’s depending on the target environment.

The low resource usage also means that I can run fairly complex setups on a laptop, where the overhead of large VM’s would be prohibitive.

I don’t think that LXC is directly comparable to full container ecosystems like Docker, since they are geared towards different use cases. These are both useful tools to know, and have complementary strengths.

Using custom fonts to add new glyphs to an Epson printer

Thu, 15 Mar 2018 00:00:00 +0000

I develop a printer driver for ESC/POS receipt printers, and regularly get feature requests for encoding text in the Chinese, Japanese and Korean languages (“CJK”).

I have recently been looking for a way to add support for these on receipt printers that have no native ability to render them, and thought I would write a bit about some progress so far.

I previously wrote a bit about printing individual bitmaps for each character, where here I am aiming to print entire scripts.

Background

Programmers usually deal with text in UTF-8, but receipt printers don’t. Instead, they still use a series of legacy code pages to represent non-ASCII text. Mapping arbitrary text to something understood by these printers is a huge challenge.

The escpos-php driver will automatically map a lot of western scripts to these code pages. However, if you attempt to send an example string like “日本語” to escpos-php currently, the driver will substitute it with “???”, since it doesn’t know how to convert them to ESC/POS.

On some printers, there are native commands to print Japanese, but for a driver project, we need something with broad compatibility. So, I decided to try to get this working on an Epson TM-T20 variant which has no CJK fonts.

I started by making a new standalone test script, which converts text input into ESC/POS using a cut-down version of the escpos-php printer driver.

$text = file_get_contents("php://stdin");
$connector = new FilePrintConnector("php://stdout");
$printer = new Printer($connector);
$printer -> text($text);
$printer -> cut();
$printer -> close();

I then modified this to print arbitrary UTF-8 text with a local bitmap font. These next sections go through some of the things I had to write to get it all working.

Character representation

I decided to start with the GNU Unifont project, because it ships fixed-width binary fonts in a text format that can be parsed without a font library, is freely licensed, and has excellent coverage.

So the first issue to solve was to do with font sizes:

Unifont contains characters that are 8 or 16 pixels wide, that cover the entire Unicode Basic Multilingual Plane (BMP), at 16 characters tall.
ESC/POS supports a fixed 12x24 or a smaller 9x17 font.
ESC/POS fonts are submitted in a 24 pixel tall format regardless of print area.

Since the characters would be surrounded by too much whitespace in the “Font A” (12x24) representation, I settled on printing in “Font B” (9x17), leaving a one-pixel space underneath, and to the right of each character. These pictures show how the glpyhs (grey) are laid out in the available print area (unused print area in white), in the available memory (unused memory in red).

Note that wider characters have a two-pixel dead-zone on the right. The non-printable 7 pixels at the bottom of the images are ignored by the printer.

The format on the printer for each character stores bits in a column-major format, while most raster formats are row-major, so I wrote a quick converter to rotate the bits. The converter code is not very concise, so I’ll just share a screen capture here to give you an idea of what it looks like. The full code is linked at the end of this post.

Lastly, the output size on paper was tiny, so I set the printer to double the size, which results in text that is around 50% larger than the default output.

Storage of fonts

There is only space for 95 single-width characters in an ESC/POS font, but the scripts are much larger than this.

I treated the font as a queue in this implementation. During the print-out, new characters are added to the font as necessary, and the font is re-written from the front as space runs out. This is also known as a FIFO cache eviction policy.

Input

I converted the string input to an array of Unicode code points to avoid canonicalisaton issues.

$chrArray = preg_split('//u', $text, -1, PREG_SPLIT_NO_EMPTY);
$codePoints = array_map("IntlChar::ord", $chrArray);
foreach($codePoints as $char) {
  $this -> writeChar($char);
}

The IntlChar class is provided by an extension which is very useful but not widespread, which limits the portability of this code.

Result

I used the list of languages from the sidebar of a Wikipedia article as a test string, since it contains short strings in a large number of scripts.

cat test.txt | php unifont-example.php > /dev/usb/lp0

The output contains a large number of correctly rendered scripts, including the CJK output, which was not previously possible on my printer.

Success!

Advantages

Previously, I have tried generating small images from system fonts to send text to the printer. This is quite costly in terms of processing and data transfer, and the printer is unable to format or wrap the text for you.

Storing glyphs in the custom font area involves transferring less raster data, and allows most text formatting commands to be used.

Limits

These characters are a different size to the native printer fonts, so we can’t mix them on the same line. This means that we can’t use this code to implement an automatic fallback in escpos-php. However, it may appear in a future version as an alternative “PrintBuffer”, which can be explicitly enabled by developers who are not interested in using the native fonts.

The esc2html utility is not able to emulate custom fonts, so the output cannot currently be rendered without an Epson printer.

Also, we simply printed a stream of characters, which is not really how text works. To implement Unicode, we need to be able to join and compose characters, and respect bi-directional text. Unicode text layout is not trivial at all.

Get the code

The full script is available in the mike42/escpos-snippets repo on GitHub, where I store prototypes of new functionality that is not yet ready for prime-time.

escpos-php 2.0 released

Tue, 13 Mar 2018 00:00:00 +0000

There is a new release of the open source receipt printing library escpos-php available from today. For composer users, it is available as mike42/escpos-php

This is the first release to drop PHP 5.3 support, which is good news for many.

The v2.0 release notes detail the changes and lists 10 additional printers tested out by the user base.

Please direct any bug reports to the issue tracker on GitHub. The escpos-php tag on this blog has some tips and examples.

New Wordpress theme (2018 edition)

Thu, 08 Mar 2018 00:00:00 +0000

This week I replaced the previous WordPress theme on this blog with a new one.

Old theme

New theme

I used Bootstrap to place widgets in my blog content, and Prism.js to do syntax highlighting on code snippets.

This is a heavily modified version of the default twentyseventeen theme. I chose this as a base because of its good use of white-space for typesetting. The updated bootstrap-based layout is also a big improvement for mobile users, who now make up the majority of web traffic:

How to print custom currency symbols on a receipt printer

Thu, 01 Mar 2018 00:00:00 +0000

Most receipt printers have a font that contains a ‘$’ sign, and many have settings to print ‘£’ ‘¥’ and ‘€’. However, I don’t know of any that can display the Bitcoin ‘₿’ or Indian Rupee ‘₹’ symbols yet.

I recently answered a question about displaying inline images on receipts from PHP, and I think this would be the best way to output newer currency codes at the moment.

Based on that answer, I used an Epson TM-T20, which understands the ESC/POS page description language, and extended the escpos-php library to list prices on a receipt in Bitcoin.

Option 1: Use an inline image

Start with a 16x24 picture of your custom character. I traced the Font Awesome fa-btc icon:

I then extended escpos-php to issue the ESC * command without breaking the line, and injected this picture like so:

<?php
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\EscposImage;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;

class CustomPrinter extends Mike42\Escpos\Printer {

    // Print image inline. If it is a multi-line image, then only the first line is printed!
    public function inlineImage(EscposImage $img, $size = Printer::IMG_DEFAULT)
    {
        $highDensityVertical = ! (($size & self::IMG_DOUBLE_HEIGHT) == Printer::IMG_DOUBLE_HEIGHT);
        $highDensityHorizontal = ! (($size & self::IMG_DOUBLE_WIDTH) == Printer::IMG_DOUBLE_WIDTH);
        // Header and density code (0, 1, 32, 33) re-used for every line
        $densityCode = ($highDensityHorizontal ? 1 : 0) + ($highDensityVertical ? 32 : 0);
        $colFormatData = $img -> toColumnFormat($highDensityVertical);
        $header = Printer::dataHeader(array($img -> getWidth()), true);
        foreach ($colFormatData as $line) {
            // Print each line, double density etc for printing are set here also
            $this -> connector -> write(self::ESC . "*" . chr($densityCode) . $header . $line);
            break;
        }
    }
}

/* Fill in your own connector here */
$connector = new FilePrintConnector("php://stdout");
$printer = new CustomPrinter($connector);

$btc = EscposImage::load("btc.png");

$printer -> text("Item    ");
$printer -> inlineImage($btc);
$printer -> text("2.50\n");

$printer -> cut();
$printer -> close();

The result was:

If you just have a few custom character to print, then this clearly works. Unfortunately, you can’t use text formatting commands alongside this method.

Option 2: Use a custom character set

This is a more complex method, where we will instruct the printer to use a BTC symbol in place of a dollar sign, through a custom character.

You must use a 12x24 image for this method, which is the standard size of the receipt printer font for most Epson models:

The example code now changes to include the ESC % and ESC % commands:

<?php
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\EscposImage;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;

class MyCoolPrinter extends Mike42\Escpos\Printer {

    public function setUserDefinedCharacter(EscposImage $img, $char)
    {
        $verticalBytes = 3;
        $colFormatData = $img -> toColumnFormat(true);
        foreach ($colFormatData as $line) {
            // Print each line, double density etc for printing are set here also
            $this -> connector -> write(self::ESC . "&" . chr($verticalBytes) . $char . $char . chr($img -> getWidth()) . $line);
            break;
        }
    }

    public function selectUserDefinedCharacterSet($on = true)
    {
        self::validateBoolean($on, __FUNCTION__);
        $this -> connector -> write(self::ESC . "%". ($on ? chr(1) : chr(0)));
    }
}

/* Fill in your own connector here */
$connector = new FilePrintConnector("php://stdout");
$printer = new MyCoolPrinter($connector);

// Replace '$' with a 24x12 image.
$char = EscposImage::load("btc2.png");
$printer -> setUserDefinedCharacter($char, "$");
$printer -> selectUserDefinedCharacterSet(true);

// Print some stuff normally
$printer -> text("Item    $2.50\n");

$printer -> cut();
$printer -> close();

Which produces this output:

The difference here is that you can also apply text formatting commands to these characters, such as double-height or emphasis:

// ... Add to the end of the previous example

// Print some stuff normally
$printer -> text("Item    $2.50\n");

// Go taller!
$printer -> selectPrintMode(Printer::MODE_DOUBLE_HEIGHT);
$printer -> text("Item    $2.50\n");

// Emphasis too?
$printer -> selectPrintMode(Printer::MODE_DOUBLE_HEIGHT | Printer::MODE_EMPHASIZED);
$printer -> text("Item    $2.50\n");

// Adjust height and width
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize($i, $i);
    $printer -> text("$");
}
$printer -> text("\n");

$printer -> cut();
$printer -> close();

Both of these methods let us encode arbitrary characters that we couldn’t otherwise be able to print. All we need is bitmaps for each character.

There is no plan to include bitmap fonts with escpos-php at the moment, but this basic mechanism might be used to improve Unicode support for a lot of receipt printers in the future.

Update 2018-03-11: I added the custom character set option to this post.

How to use HiDPI displays on Debian 9

Thu, 22 Feb 2018 00:00:00 +0000

I recently added a 4K monitor to my Debian box, and had to set a few things to make it display things at a good size. These high-density monitors that are becoming common on laptops and desktops are known as “HiDPI” displays.

Currently I get the best results with:

Window scaling factor of 2
Font scaling 0.90 to make text slightly smaller

Note that “window scaling” is not “upscaling” (stretching an image). In this version of Gnome, it means “single/double/triple DPI”. The implementations are in the process of changing: Soon you should be able to set any scaling factor.

This post assumes a Gnome version around 3.26, which is what you would get as a default if you installed Debian 9 today.

Apply to one user only

Under Settings → Devices → Displays set the Scale to 200%.

Under Tweaks → Fonts, set the Scaling Factor to 0.90.

Next, add these variables to ~/bashrc to apply similar scaling to QT apps.

QT_AUTO_SCREEN_SCALE_FACTOR=0
QT_SCALE_FACTOR=2

Log out and back in to ensure that the settings have applied everywhere.

Apply to all users

If you have a shared system (eg. domain accounts), or want to style the login box as well, then you can set the same settings as below.

These steps are based on answers to the Ask Ubuntu question: Adjust text scaling factor for all users<.

nano /usr/share/glib-2.0/schemas/org.gnome.desktop.interface.gschema.xml

Set the text-scaling-factor to 0.9, and the scaling-factor to 2.

<key name="text-scaling-factor" type="d">
  <range min="0.5" max="3.0"/>
  <default>0.9</default>
  <summary>Text scaling factor</summary>
  <description>
    Factor used to enlarge or reduce text display, without changing font si$
  </description>
</key>
<key name="scaling-factor" type="u">
  <default>2</default>
  <summary>Window scaling factor</summary>
  <description>
    Integer factor used to scale windows by. For use on high-dpi screens.
    0 means pick automatically based on monitor.
  </description>
</key>

Re-compile the schemas:

glib-compile-schemas /usr/share/glib-2.0/schemas

Next drop some similar environment variables for QT apps in /etc/profile.d/hidpi.sh to apply it to all users:

export QT_AUTO_SCREEN_SCALE_FACTOR=0
export QT_SCALE_FACTOR=2

After this, reboot. If the setting has applied, then the gdm3 login box will be scaled as well.

How to print Spanish characters on a thermal receipt printer

Thu, 15 Feb 2018 00:00:00 +0000

Today we’re going to send some Spanish text to an Epson TM-T20 thermal receipt printer. This post is intended for people who know how to send ASCII to a similar printer, but can’t get the diacritics from Spanish to displayu properly, characters like á, é, í, ó, ñ, ú or ü.

Many receipt printers understand a language called ESC/POS, which uses legacy code pages for non-ASCII text.

Spanish is very easy to encode manually, because it can be represented fully using just Code Page 437, which happens to be the default code page on most receipt printers.

Generating CP437-encoded text

To convert UTF-8 to CP437, I will use the iconv utility on the Linux command-line. API’s for encoding conversion are available for your favourite programming language:

echo "áéíóñúü" | iconv --from-code=UTF-8 --to-code=CP437

Or, as a more complete example, let’s use a pangram from here:

“Benjamín pidió una bebida de kiwi y fresa; Noé, sin vergüenza, la más exquisita champaña del menú”

echo "Benjamín pidió una bebida de kiwi y fresa; \
Noé, sin vergüenza, la más exquisita champaña del menú" | \
iconv --from-code=UTF-8 --to-code=CP437

Printing the text

Next, let’s send some text to the printer. I have a USB printer on Linux, and this is how I would send it data:

echo "Hello World" > /dev/usb/lp0

Putting it all together.

echo "Benjamín pidió una bebida de kiwi y fresa; \
Noé, sin vergüenza, la más exquisita champaña del menú" | \
iconv --from-code=UTF-8 --to-code=CP437 > /dev/usb/lp0

This now displays the diacritics correctly when printed.

Libraries for PHP and Python users

I’ve helped out writing the internationalization features for two printing libraries (escpos-php and python-escpos, so that programmers can work with text in UTF-8, and let the library translate the text to something that your printer understand.

The same example above, if you use the PHP library, would look like this:

<?php
require __DIR__ . '/vendor/autoload.php';
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
use Mike42\Escpos\Printer;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);
$printer -> text("Benjamín pidió una bebida de kiwi y fresa; Noé, sin vergüenza, la más exquisita champaña del menú\n");
$printer -> cut();
$printer -> close();

And in Python, it would look like this:

from escpos import *
my_printer = printer.File("/dev/usb/lp0")
my_printer.text("Benjamín pidió una bebida de kiwi y fresa; Noé, sin vergüenza, la más exquisita champaña del menú\n")
my_printer.cut()

Most of the time you need more than text: Formatting, adding barcodes, or cutting the paper will all need ESC/POS commands, and these libraries will help with that too.

Take a look at the project links for more detail.

How to install PHP Composer as a regular user

Thu, 08 Feb 2018 00:00:00 +0000

Composer is an essential utility for PHP programmers, and allows you to manage dependencies.

Dependencies

You can use your regular account to install composer, use it, and even update it. You do need to have a few packages installed first though:

sudo apt-get install git curl php-cli

Or on Fedora:

sudo dnf install git curl php-cli

Local install

Next, fetch the installer and deploy composer to your home directory

curl https://getcomposer.org/installer > composer-setup.php
mkdir -p ~/.local/bin
php composer-setup.php --install-dir=$HOME/.local/bin --filename=composer
rm composer-setup.php

Last, add ~/.local/bin to your $PATH:

echo 'PATH=$PATH:~/.local/bin' >> ~/.bashrc
source  ~/.bashrc
echo $PATH

You can now run composer:

$ composer --help
Usage:
  help [options] [--] [<command_name>]
...
$ composer self-update
You are already using composer version 1.5.6 (stable channel).

Make Composer available for all users

Just run this line if you decide that all users should have access to your copy of Composer:

sudo mv ~/.local/bin/composer /usr/local/bin/composer

If you look up a how to install Composer, you will find a tempting one-liner that uses curl to fetch a script from the Composer website, then executes it as root. I don’t think it’s good practice to install software like that, so I would encourage you to just run ‘sudo mv’ at the end.

How to boot Debian in 4 seconds

Thu, 01 Feb 2018 00:00:00 +0000

This blog post is a throwback to “Booting Debian in 14 seconds” from debian-administration.org, where the author went through some fairly advanced steps to get his low-spec Debian laptop to boot quickly. Debian was version 4.0 at the time, and I recall it taking around 40 seconds to boot on a default desktop install.

In a rare exception to Wirth’s law, waiting for a computer to boot is no longer “a thing”. A default desktop install of Debian includes systemd, and uses a multicore CPU and SSD quite efficiently. Also, sleep/wake works more reliably than it used to, so boot speed is not as important as it used to be.

On a modern desktop PC, booting Debian 9 (default desktop install) takes me 14 seconds with no extra configuration, so that’s our new low water mark.

Mainly to illustrate how far open source operating systems have come, I’m going to step through a boot process speed-up, the way it looks in 2018.

Summary

Out

You will read about some of these older tricks if you search for Linux Boot speed, and they are all quite irrelevant in 2018, in my humble opinion-

Swapping the /bin/sh shell to dash - it is already the default, but also init scripts are no longer used.
Using readahead - gains are marginal unless you have a HDD.
“noatime” setting on mounts - “relatime” is a default mount option since Linux 2.6.

In

New things that you won’t find in pre-systemd guides:

systemd-analyze to instrument the boot
systemctl to exclude processes from boot

Still relevant

bootchart is still useful for drawing pretty graphs
Configure GRUB & UEFI not to prompt for input
Don’t enable services you don’t need

Process

Remove bootloader delay

Between UEFI and the OS, you will get the bootloader, which will wait for 5 seconds by default to see if you want to select a different item. Start by switching the grub timeout from 5 seconds to 0.

sudo nano /etc/default/grub

Set GRUB_TIMEOUT=0

Apply the grub configuration change:

sudo update-grub2

Look at systemd

Use the tool systemd-analyze to draw a picture:

systemd-analyze plot > plot.svg

In my case, it was clear that 9 seconds of the boot was an optional “waiting for network” step.

So, (thank you askubuntu), I disabled that service and rebooted:

$ sudo systemctl disable NetworkManager-wait-online.service
Removed /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service.

I then ran systemd-analyze again, and found that the boot was still taking 4.4 seconds, so, more analysis was in order:

The systemd-timesyncd service was holding things up.

This service runs early in the boot process, reads an old time from a file, and tries to update time over the network. Since I have a working RTC, this is all unnecessary for me, so I removed it and replaced it with chronyd, which is happy to operate in the background.

sudo systemctl disable systemd-timesyncd.service
sudo apt-get install chronyd
sudo systemctl enable chrony

After another reboot:

systemd-analyze plot > plot4.svg

There we go, down to 4.096 seconds with a few minutes of effort. I think that’s acceptable.

The systemd developers are quite certain that you can boot in under 2 seconds, but I wasn’t willing to customise my system to that extent.

How to use parallel to speed up your work

Thu, 25 Jan 2018 00:00:00 +0000

GNU Parallel is a tool to execute multiple commands at once. In its basic usage, you would list your commands in a file, so that it can execute them, several at a time.

It gives the most benefit on processes that don’t fully utilise your CPU. Almost every laptop, desktop and single board computer now has multiple CPU cores available, so you are probably missing out if you frequently perform batch operations without it.

Installation

On Debian or Ubuntu:

sudo apt-get install parallel
parallel --cite

On Fedora the package name is the same:

sudo dnf install parallel
parallel --cite

Example 1: Convert loops to pipes

Using the ImageMagick tool to convert a folder of GIF images to PNG format can be done in a loop:

for i in *.gif; do convert $i -scale 200% ${i%.*}.png; done

Or, you could print each command in a loop then pass them to parallel.

for i in *.gif; do echo convert $i -scale 200% ${i%.*}.png; done | parallel

The second command is many times faster on a multicore computer.

Example 2: Replace xargs with parallel

This command executes a single “pngcrush” command on each PNG file in a directory, one at a time.

find . -type f -name '*.png' -print0  | xargs -0 -n1 -r pngcrush -q -ow -brute

To convert this to use parallel, you would use the following command-line:

find . -type f -name '*.png' | parallel "pngcrush -q -ow -brute {}"

Don’t use xargs in parallel mode

Expert command line users will also know about xargs -P, which seems to do the same thing at a glance.

xargs is good at making really long command-lines, and not so good at executing multiple commands at once. It will mix the output of the commands, and requires you to specify the number of jobs to run.

Parallel is designed to do lots of things at once, and it does it well. It will choose some good defaults for the number of processes to execute, and adds an insane collection of features that you need for large batches. To name just a few:

Control spawning of new jobs based on things like available memory, system load, or an absolute number of jobs to keep running
Distribute jobs to remote computers
Show progress
Control of when to terminate the jobs

How to generate star fields

Thu, 18 Jan 2018 00:00:00 +0000

I recently needed a texture for the skybox in a 3D space game.

I used this ImageMagick one-liner to generate a dotted canvas with some grey and white pixels. It displays well if the texture will be stretched.

convert -size 1600x900 xc: +noise Random -channel R -threshold 0.5% \
        -negate -channel RG -separate +channel \
        -compose multiply -composite stars.png

Alternative style

With a simple modification, the bright stars are made bigger, and the dull ones are made smaller, with only black and white used. This works well if the texture will be shrunk for display.

convert -size 800x450 xc: +noise Random -channel R -threshold 0.5% \
        -negate -channel RG -separate +channel \
        -compose multiply -composite -resize 200% \
        -threshold 10% stars-rounded.png

Full process

Although these are great one-liners, the actual process is a bit hard to follow without some smaller steps.

Here, I’ll walk through how to generate a 150x90 star field in several steps. I’ve scaled each of these pictures to 200% of their original size and converted them to PNG for display on the web. I’ve used BMP in the commands only because it saves some plumbing around colour spaces.

Start with a blank canvas:

convert -size 150x90 xc: stars-01.bmp

Add random RGB noise:

convert stars-01.bmp +noise Random stars-02.bmp

In the red channel of the image, apply a black/white threshold: If the red channel is greater than 0.5%, it is set to the maximum, otherwise it is set to the minimum.

Mostly the red channel is now 100%, with random dots of 0%. The blue and green channels are still completely random:

convert stars-02.bmp -channel R -threshold 0.5% stars-03.bmp

Negate the image, so that the red channel is mostly 0%, with dots of 100%:

convert stars-03.bmp -negate stars-04.bmp

Extract the red channel and green channel.

convert stars-04.bmp -channel RG -separate +channel stars-05.bmp

The red channel will be dots of white:

The green channel will be random:

Multiply the channels together, so that the white dots become grey dots, each with a random brightness:

convert stars-05-0.bmp stars-05-1.bmp -compose multiply -composite stars-06.bmp

For the alternative style, scale the image up, then apply a new threshold. Brighter dots will appear as larger patches of white, while dull stars will be smaller or invisible:

convert stars-06.bmp -resize 200% -threshold 10% stars-06-rounded.bmp

On running deployments via GitHub

Wed, 10 Jan 2018 00:00:00 +0000

TL;DR; - GitHub can be pretty unreliable, depend on it at your peril.

GitHub was down for about 20 minutes today. I happened to be logged in, so I’ll share a few screen captures.

What a GitHub outage looks like

The status page and twitter showed no activity for the first nine minutes of the outage, but were then updated with erroneous information.

Meanwhile, the website started displaying unicorns.

If you use GitHub regularly, you might know that this is not a rare event. I don’t have any data on how often GitHub is actually broken, but based on that status page, I’m not sure that they know either.

Why I don’t deploy via GitHub

Quick story - around May 2013, a website that I maintained started rendering incorrectly because of a bug in my code: I had made some MySQL fields TEXT type, which have a limited size, and part of the application had exceeded the limit, resulting in truncated pages.

This app is written in PHP, and my deployment workflow at the time involved pushing up a change to GitHub, then triggering a git pull on the server, which ran an update script to bring everything up to speed.

In this case, I was on mobile internet, so I diagnosed the problem and prepared a hotfix on my laptop. When I tried to push it to GitHub, it was offline. I ended up logging in and running a few ALTER TABLE statements over SSH, which is a long shot from the robust deployment pipeline I had planned to use.

There are some emerging SaaS products that offer to deploy directly from GitHub. For example, I use Travis CI extensively for open source, and you can hook it up with deployment keys.

These product seems really promising, but I imagine that an error message containing rainbow unicorns would not be very funny if you needed to fix something in a hurry. This particular app is still deployed with a git pull, but I’ve started to avoid mixing deployment with version control, so that I can run a build and deploy anywhere in case of emergency.

How to use a Radeon graphics card on Debian 9

Thu, 04 Jan 2018 00:00:00 +0000

I have previously blogged about Radeon graphics cards on different Debian installs.

ATI has now released a new free driver which works brilliantly on Debian. In the past, Debian users had to choose between using the community-provided free software driver, or the proprietary one. Generally the proprietary driver was more feature-rich, but the free driver worked more reliably across upgrades. So now, you can safely ignore old guides and start using it.

This is a quick guide about how to do that.

Upgrade

Make sure you are on Debian 9 (Stretch) or newer.

These steps apply to a fresh install.

Identify

You should use lspci to confirm that you have an ATI card.

$ lspci | grep Radeon
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X]

Install firmware

You need to install a package called firmware-linux-free to get the driver working at all. If you want decent graphics performance, you will need firmware-linux-nonfree as well, which involves adding non-free sources.

nano /etc/apt/sources.list

Add the words “contrib non-free” to the end of your mirror:

deb http://.../debian/debian/ stretch main contrib non-free

Add the packages:

apt-get update
apt-get install firmware-linux-free firmware-linux-nonfree

And reboot:

reboot

What, that’s it?

Well, yes, for a fresh install that’s it. If your install is old, you might also have to remove old drivers or install the xserver-xorg-video-amdgpu and xserver-xorg-video-ati packages (in my case, these were already installed).

The Debian Wiki AtiHowto contains some more detailed information, most of which is not relevant for a simple desktop setup.

How to assemble a Linux software RAID array on a different computer

Thu, 28 Dec 2017 00:00:00 +0000

With Linux software RAID, if you ever toast your computer, you can retrieve the disks and open up the array on a different computer.

They appear as “Linux Software RAID Member” in the disk utility.

Simply install mdadm, and scan for arrays:

$ sudo apt-get install mdadm
$ sudo mdadm --assemble --scan
mdadm: /dev/md/0 has been started with 2 drives.

The array will then appear as a new disk, which can be formatted, mounted, or cloned via the usual tools.

Crowd-sourced POS printer compatibility site is online

Sun, 26 Nov 2017 00:00:00 +0000

Point-of-sale (POS) printers are all different, so the implementation of the more advanced ESC/POS features varies considerably between vendors and models.

Some months ago I extracted compatibility information out of the escpos-php library, so that it could be used by the python-escpos team as well. We’re still in the early stages, but both of these projects now have similar compatibility features.

You can the new shared database on GitHub as receipt-print-hq/escpos-printer-db, and I’m beginning to add it to some printing projects that need it.

Today I’m blogging a few screen captures from the new web viewer for this database that has come out today, hosted here (source code here).

Character encodings

Vendors

Printer profiles

Tech used

I’ve used materialize.css with backbone.js and jQuery. This is the first time I’ve attempted using materialize - I would normally have used bootstrap.css for this sort of task.

The back-end data is generated with Python, but the actual site is served as a single-page application, with static JSON files hosting the data.

escpos-php 1.6 released

Sun, 05 Nov 2017 00:00:00 +0000

Another update to the open source receipt printing library escpos-php has been released today. For composer users, it is available as mike42/escpos-php

This is expected to be the final release in the 1.x series. Newer versions will drop support for some end-of-life PHP versions.

The v1.6 release notes detail the changes and lists 13 additional printers tested out by the user base.

Please direct any bug reports to the issue tracker on GitHub. The escpos-php tag on this blog has some tips and examples.

I’ll also call out some related projects that I’ve been involved with, which you should consider contributing to if you are working with thermal receipt printers:

receipt-print-hq/escpos-printer-db - Crowd-sourced database of printer features. Add your printer for better support in open source drivers!
receipt-print-hq/escpos-tools - Tools to work with ESC/POS binary, including text or image extraction, HTML conversion.
receipt-print-hq/chrome-raw-print - A browser plugin to access local printers from a web-page.

mdcheckr: Practical testing for Markdown files

Sun, 25 Jun 2017 00:00:00 +0000

If you are a continuous integration enthusiast, you might be annoyed by preventable errors in project documentation.

I recently found a broken link in one project’s documentation, and a syntax error in an example on another project. I quickly wrote up a simple tool to detect this type of issue, solving the problem for me.

$ mdcheckr README.md
Checking README.md ..
- Code block starting line 21 (language: bash) [ OK ]
- Link https://travis-ci.org/receipt-print-hq/escpos-tools [ OK ]
- Link https://getcomposer.org/ [ OK ]
- Link doc/esc2text.md [ OK ]
- Link doc/esc2html.md [ OK ]
- Link doc/escimages.md [ OK ]
- Link CONTRIBUTING.md [ OK ]
- Link LICENSE.md [ OK ]
- Image https://travis-ci.org/receipt-print-hq/escpos-tools.svg?branch=master [ OK ]

I disregarded some existing “linter” tools. I don’t believe that it is necessarily productive for small projects to enforce a style-based quality gate for Markdown, but you may be able to configure these tools to be less pedantic. My goal here was to simply churn out a script which could reject objectively broken Markdown files.

I’ve posted this project to GitHub as mdcheckr. It is available as a package for Debian, Ubuntu, and RHEL/CentOS. Travis CI users may be interested to check out this pull request, where mdcheckr is added to the project’s build.

How to access a raw network printer as a file on Linux

Mon, 12 Jun 2017 00:00:00 +0000

I got this interesting question on my blog post ‘Setting up an Epson receipt printer’:

Wondering if you had an idea on how to map a linux device to the netcat command so that I could “convert” the printer to be a local one? -Marco

I have previously written about the opposite use case: How to use a Raspberry Pi as a print server, where I used netcat to pass data to a local USB printer.

The setup

In this setup, I will assume that you have a working Ethernet printer, which accepts raw data on port 9100.

The aim is to make this appear as a file, so that you can print to it as if it were a local USB printer:

Before you begin

Make sure you have netcat. It’s not used for the real setup, but you will need it for testing. There are a few versions of this tool around, the one used here is from the Debian repositories:

apt-get install netcat

Verify that your printer accepts text on port 9100 via netcat:

echo "Hello world" | nc -q 1 192.168.x.y 9100

Set up a fake printer on localhost, and leave it running:

nc -klp 9100

Now, test that your fake printer shows output when you send it:

echo "Hello world" | nc -q 1 127.0.0.1 9100

Why I’m not using ___

/dev/tcp

If you search for how to redirect data to a TCP socket, a common suggestion is:

echo "Hello world" > /dev/tcp/127.0.0.1/9100

This uses a bash built-in, so unless your use case involves printing from bash, read on.

CUPS

CUPS does not expose the printer as a file, which is the aim here. It also takes a few seconds to print, so we get much faster results if we remove CUPS from the loop and speak to the printer directly.

Use socat to move data

Socat is capable of making a FIFO file (‘pipe’), and writing this out over the network.

apt-get install socat

Redirect /tmp/my-printer to localhost:9100:

socat PIPE:/tmp/my-printer TCP:localhost:9100

Next, test that you can see the printer file and write to it. The text sent here should appear on your local netcat “fake printer”:

echo "Hello world" > /tmp/my-printer

This file is a “pipe”, so it behaves very similarly to a character device.

If the connection is dropped, socat will exit, and the file will be deleted.

There are some big problems though:

If your printer is offline, and you try to print, then data will be left in regular file at “/tmp/my-printer”, which breaks this setup
This is not a self-restarting service, nor does it start on boot
Only the user who runs the command can print

Setting this up as a service

The usblp driver allows anybody from the lp group to print, so we will try to do something similar, and get a group-writable pipe:

$ ls /dev/usb/lp0  -Ahl
crw-rw---- 1 root lp 180, 0 Jun 12 12:51 /dev/usb/lp0

We don’t need to run socat as root, so make a new user called fileprint who is in the lp group.

useradd --groups lp fileprint

Add yourself to the lp group as well:

usermod -a -G lp mike

Next, write this systemd service file to /etc/systemd/system/fileprint.service.

[Unit]
Description=fileprint
After=network.target

[Service]
User=fileprint
Group=lp
ExecStartPre=/usr/bin/mkfifo -m '0664' /var/run/fileprint/printer
ExecStart=/usr/bin/socat PIPE:/var/run/fileprint/printer TCP:127.0.0.1:9100
Restart=always
RuntimeDirectory=fileprint

What’s going on here? This service file:

Runs socat as user fileprint and group lp
Manages a runtime directory at /var/run/fileprint
Creates a pipe that is group-writeable at /var/run/fileprint/printer
Forwards traffic to 127.0.0.1:9100
Re-starts automatically

Load and start the service:

systemctl daemon-reload
systemctl start fileprint
systemctl status fileprint

Test out printing to your local netcat “fake printer”:

echo "Test" > /var/run/fileprint/printer

Note that if you are not in the lp group (check with groups command), you should expect a permission error here - just the same as usblp.

Next, replace 127.0.0.1 with your real printer IP, reload systemd, and restart the service:

nano /etc/systemd/system/fileprint.service
systemctl daemon-reload
systemctl restart fileprint
systemctl status fileprint

Test it again, and your actual printer should print a line of text this time:

echo "Test" > /var/run/fileprint/printer

Finally, enable on boot:

systemctl enable fileprint

When the printer is unplugged, the /var/run/fileprint directory will vanish, so that you can see the missing printer as a “File Not Found” error — just the same as usblp.

Drawbacks

This will hold a connection open at all times, which (depending on your printer) may prevent other computers from using it.

The 5890 printer does not support barcodes

Mon, 12 Jun 2017 00:00:00 +0000

I haven’t been able to print barcodes on the 5890 model thermal receipt printer, and it looks like other developers can’t either.

The same printer is sold under a few names. As far as I can tell (mainly from bug reports to mike42/escpos-php), these are all the same printer, and could have this issue:

AGPtEK SC35-5890F
EC Line 5890
Gainscha GP-5890
HSPOS HS-589C
POS-5890
WinBond 5890
Zjiang ZJ-5890 or POS-5890

In my case, I have a Zjiang POS-5890C, which list the barcode command GS k in its documentation, but does not actually respond to the command.

Separately, a developer has contacted their vendor and confirmed that this is an error in the documentation.

A few bookmarks for reference:

How to test for barcode support in USB printers

Get yourself a Linux computer or VM, set up the printer with usblp, and run this command:

echo -e '\x1d\x6b\x04000\x00' > /dev/usb/lp0\

This command prints a CODE39 barcode containing ‘000’ on printers which support barcodes, and prints garbage on printers which do not.

How to add a proper Android Studio launcher on Linux

Mon, 24 Apr 2017 00:00:00 +0000

If you install Android Studio by extracting a Zip file on Linux, it will come without any launch icon. This post assumes that have extracted the install archive to /usr/local and run any recent version of Linux which has Gnome:

unzip android-studio-ide-145.3537/39-linux.zip -d /usr/local

Simply write the following text to ~/.local/share/applications/android-studio.desktop (change paths if your install location differs).

[Desktop Entry]
Comment=Android Studio
Terminal=false
Name=Android Studio
Exec=/usr/local/android-studio/bin/studio.sh
Type=Application
Icon=/usr/local/android-studio/bin/studio.png

Once you save the file, it will appear in your local applications like this:

This can also be added graphically though the Alacarte menu editor if you have it installed.

libgdx 3D particle effects in HTML

Mon, 24 Apr 2017 00:00:00 +0000

It is not immediately obvious in libgdx why the 3D particle effects don’t work in a HTML target. I’m sharing this snippet for future readers.

In short, the “reflection cache” that is created from the Java code does not include everything required, since many of the classes are only referenced at runtime, when the particle definition is loaded.

These class names are visible in the saved files from the 3D effects editor:

$ cat engine.p | fold -w 80
{unique:{billboardBatch:{class:com.badlogic.gdx.graphics.g3d.particles.ResourceD
ata$SaveData,data:{cfg:{class:com.badlogic.gdx.graphics.g3d.particles.batches.Bi
llboardParticleBatch$Config,mode:Screen}},indices:[0]}},data:[],assets:[{filenam
e:"pre_particle.png",type:com.badlogic.gdx.graphics.Texture}],resource:{class:co
....

At runtime, this message is displayed on the web page:

GwtApplication: exception: com.badlogic.gdx.utils.GdxRuntimeException: Could not submit AsyncTask: Error reading file: (filename)
com.badlogic.gdx.utils.GdxRuntimeException: Could not submit AsyncTask: Error reading file: (filename)
Could not submit AsyncTask: Error reading file: (filename)
Error reading file: (filename)
Couldn't find Type for class 'com.badlogic.gdx.graphics.g3d.particles.batches.BillboardParticleBatch$Config'

The reflection is documented on the libgdx Wiki here, and notes that-

*.gwt.xml files store this data
dependencies defined in the Java are loaded automatically
inner classes are also loaded automatically, no need to add them separately

With this in mind, I added the following two lines to the root element in a file called GdxDefinition.gwt.xml-

<extend-configuration-property name="gdx.reflect.include" value="com.badlogic.gdx.graphics.g3d.particles.batches.BillboardParticleBatch" />
<extend-configuration-property name="gdx.reflect.include" value="com.badlogic.gdx.graphics.g3d.particles.ParticleShader" />

This did the trick, and the 3D particle feature does indeed work in the libgdx HTML target.

escpos-php 1.5 released

Fri, 10 Feb 2017 00:00:00 +0000

I’ve released another update to the open source receipt printing library mike42/escpos-php today.

Device support has been continuing to improve. See the v1.5 release notes for all the details.

Please direct any bug reports to the issue tracker on GitHub. The escpos-php tag on this blog has some posts about the driver for PHP programmers who are looking to use it.

escpos-php 1.4 released

Mon, 24 Oct 2016 00:00:00 +0000

The next minor release of the open source receipt printing library for escpos-php was published today.

See the release notes for all the details.

Aside from the official documentation, if you are a PHP programmer with a receipt printer, you can learn how to use escpos-php by reading the escpos-php tag on this blog.

Please direct any questions or suggestions to the issue tracker on GitHub.

OpenWrt setup on Netgear WNR2200

Sun, 23 Oct 2016 00:00:00 +0000

I recently wanted to connect some devices for a temporary setup, where a wireless LTE modem would provide Internet access. Unfortunately, one of the devices was not close enough to pick up the signal with its USB WiFi dongle.

Because the modem does not have a LAN port, the usual “run a cable” solution was out. There’s a few other options, from range extenders, to getting better modem, or just upgrading to a “real” USB WiFi dongle. Before purchasing new hardware, I decided to try re-purposing an old Netgear WNR2200 as a wireless client and 4 port switch.

In this setup, the LTE modem does the heavy lifting, with all of the wireless clients using it for LAN and Internet access. In the next room, the Netgear router is placed close enough to pick up the signal, and an Ethernet cable runs to the PC, beyond the reach of WiFi.

Deciding to re-flash

Replacing firmware is worth investigating when the hardware is capable, but you aren’t given the option to configure it the way you want.

The Netgear WNR2200 is a low-end wireless router, and the vendor firmware does not support joining a Wi-Fi network as a client.

It also pays to update your research. OpenWrt added support for this router a few days after I bought it, but I hadn't looked it up again.

Uploading firmware

My main resource was this page on the OpenWRT Wiki. Firmware is organised by wireless chipset, then by router model.

The file I used to update my router was named openwrt-15.05.1-ar71xx-generic-wnr2200-squashfs-factory.img.

This is simply uploaded on the Adminisration → Firmware Upgrade screen:

First impressions

The first thing I noticed was that I lost Wi-Fi, and that the page I had bookmarked for logging in was no longer valid!

This makes sense, of course. The configuration will not be carried across from the vendor firmware, and a different web administration tool is being used.

The Linux userspace is very rich compared with vendor firmware. It has things like dmesg, SSH, ifconfig, ping, and even a networked package manager.

Configuration checklist

I performed all configuration through the web in this setup. The “LuCi” interface allows setting the Wi-Fi chip into “Client” mode, and then searching and joining a network. Once this was done, I assigned it as the “WAN” interface, so that it occupied a single IP address on the Wi-Fi network, and providing a NAT and wired, four port switch.

There are more advanced, bridged setups that are possible. You should investigate this if you want one network, so that things like printer auto-discovery and internal SSH work consistently. I was only interested in sharing the Internet connection, which is why the setup was so simple.

What didn’t work

USB, but I didn’t spend long on this either. I was considering using USB to connect the modem to the Netgear router. The Wiki suggests that this is now possible, but after installing some packages for “USB tethering” and rebooting, I had no luck. Typing lsusb, only the “root hub” was listed, and the device was not getting any power.

This was necessary for the setup, so I just abandoned it. The vendor firmware couldn’t use the USB port for networking either, so no real loss.

libvirt: Migrate a VM from qemu:///session to qemu:///system

Thu, 29 Sep 2016 00:00:00 +0000

In recent versions of the libvirt virtualisation libraries, you to create and manage virtual machines as a regular user, using the qemu:///session connection.

This is great, but the networking is quite limited. I found that machines defined in Gnome Boxes could not speak to each-other, and that libvirt commands for networking were unavailable.

For this reason, I’ve written this quick guide for booting up an existing same VM image under the qemu:///system instance, which is faster than re-installing the machine. Unlike most sorts of migrations, this leaves the disk image at the same location on the same host machine.

There’s many different ways to do VM’s in Linux. This setup will be useful only if you use libvirt/kvm using qcow2 images on Debian. As always, consider doing a backup before trying new things.

Configuration files

First, find your virtual machine in virsh, and dump its configuration to a text file in your home directory, as a regular user.

$ virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     foo-machine                    shut off
$ virsh dumpxml foo-machine > foo-machine.xml

Now remove the VM definition from your user:

$ virsh undefine foo-machine
Domain foo-machine has been undefined

Import the definitions into virsh as the root user:

$ sudo virsh define foo-machine.xml 
Domain foo-machin defined from foo-machine.xml

Attempt to start the new VM definition. Depending on where the disk image is, expect an error.

$ sudo virsh start foo-machine

Disk images

The disk image needs to be accessible to the libvirt-qemu user. There’s two basic ways to achieve this: Re-permission the directories above it, or move it.

I chose to just re-permission it, since it’s not an issue to have world-readable directories on this particular box:

$ cat foo-machine.xml | grep source
      <source file='/home/example/.local/share/gnome-boxes/images/foo-machine'/>;

This one-liner outputs the commands to run to make a directory work/navigable:

$ dir=`pwd`; while [ "$dir" != "/" ]; do echo "chmod o+x,g+x \"$dir\""; dir=`dirname $dir`; done
chmod o+x,g+x "/home/example/.local/share/gnome-boxes/images"
chmod o+x,g+x "/home/example/.local/share/gnome-boxes"
chmod o+x,g+x "/home/example/.local/share"
chmod o+x,g+x "/home/example/.local"
chmod o+x,g+x "/home/example"
chmod o+x,g+x "/home"

And the user account needs to be able to write as well:

$ sudo chown libvirt-qemu /home/example/.local/share/gnome-boxes/images/foo-machine

Once you have the permissions right, the VM should start, using the same command as before:

$ sudo virsh start foo-machine

More importantly, you can now hook up virt-manager and view your machine on qemu:///system, allowing you to configure the VM with any network settings you need.

How to print PDF417 codes with escpos-php

Thu, 08 Sep 2016 00:00:00 +0000

This post is a reference for printing PDF417 2-dimensional codes to a receipt printer, using the escpos-php.

I’ve got an older post about printing QR codes with escpos-php, which follows the same format and has some more background and links if you haven’t printed receipts from PHP before.

Straight from the documentation, the syntax for the command that I’m demonstrating is:

pdf417Code($content, $width, $heightMultiplier, $dataColumnCount, $ec, $options)

Print a two-dimensional data code using the PDF417 standard.

string $content: Text or numbers to store in the code

number $width: Width of a module (pixel) in the printed code. Default is 3 dots.

number $heightMultiplier: Multiplier for height of a module. Default is 3 times the width.

number $dataColumnCount: Number of data columns to use. 0 (default) is to auto-calculate. Smaller numbers will result in a narrower code, making larger pixel sizes possible. Larger numbers require smaller pixel sizes.

real $ec: Error correction ratio, from 0.01 to 4.00. Default is 0.10 (10%).

number $options: Standard code Printer::PDF417_STANDARD with start/end bars, or truncated code Printer::PDF417_TRUNCATED with start bars only.

These PDF417 snippets above appear in the examples of escpos-php.

Simple example

A basic code that just says ’testing 123’, and a demonstration of a narrower code that has been aligned:

Example

PHP

// Most simple example
title($printer, "PDF417 code demo\n");
$testStr = "Testing 123";
$printer -> pdf417Code($testStr);
$printer -> text("Most simple example\n");
$printer -> feed();

// Demo that alignment is the same as text
$printer -> setJustification(Printer::JUSTIFY_CENTER);
$printer -> pdf417Code($testStr, 3, 3, 2);
$printer -> text("Same content, narrow and centred\n");
$printer -> setJustification();
$printer -> feed();

Error correction

This implementation accepts an error correction ratio as a percentage. The minimum is 1%, the highest is 400%, expressed as a decimal (0.01 - 4.00).

Higher error correction settings create lager codes that are more resilient to scanning errors due to damage.

Example

PHP

// Demo of error correction
title($printer, "Error correction\n");
$testStr = "Testing 123";
$ec = array(0.1, 0.5, 1.0, 2.0, 4.0);
foreach ($ec as $level) {
    $printer -> pdf417Code($testStr, 3, 3, 0, $level);
    $printer -> text("Error correction ratio $level\n");
    $printer -> feed();
}

Pixel size (width)

The same example string, with some different module widths. Note that the blocks in the code scale in bot directions when the width is changed.

Larger print is easier for a scanner to read, but takes up more paper.

Example

PHP

// Change size
title($printer, "Pixel size\n");
$testStr = "Testing 123";
$sizes = array(
    2 => "(minimum)",
    3 => "(default)",
    4 => "",
    8 => "(maximum)");
foreach ($sizes as $size => $label) {
    $printer -> pdf417Code($testStr, $size);
    $printer -> text("Module width $size dots $label\n");
    $printer -> feed();
}

Height multiplier

The height of the modules in the code can also be changed, stretching it do a different degree. PDF417 that are too vertically squashy are more prone to scanning errors.

Example

PHP

// Change height
title($printer, "Height multiplier\n");
$testStr = "Testing 123";
$sizes = array(
    2 => "(minimum)",
    3 => "(default)",
    4 => "",
    8 => "(maximum)");
foreach ($sizes as $size => $label) {
    $printer -> pdf417Code($testStr, 3, $size);
    $printer -> text("Height multiplier $size $label\n");
    $printer -> feed();
}

Data column count

The of data columns to print in the code can be customised to produce a narrower code. But beware, if you request a code that’s too big for the paper, nothing will be printed!

Example

PHP

// Change data column count
title($printer, "Data column count\n");
$testStr = "Testing 123";
$columnCounts = array(
    0 => "(auto, default)",
    1 => "",
    2 => "",
    3 => "",
    4 => "",
    5 => "",
    30 => "(maximum, doesnt fit!)");
foreach ($columnCounts as $columnCount => $label) {
    $printer -> pdf417Code($testStr, 3, 3, $columnCount);
    $printer -> text("Column count $columnCount $label\n");
    $printer -> feed();
}

Truncated code option

Use this setting to select the alternative, ’truncated’ code format.

Example

PHP

// Change options
title($printer, "Options\n");
$testStr = "Testing 123";
$models = array(
    Printer::PDF417_STANDARD => "Standard",
    Printer::PDF417_TRUNCATED => "Truncated");
foreach ($models as $model => $name) {
    $printer -> pdf417Code($testStr, 3, 3, 0, 0.10, $model);
    $printer -> text("$name\n");
    $printer -> feed();
}

Notes

To run the snippets, you need to initialise the printer, and define a title() function to print headings, like so:

<?php
/* Demonstration of available options on the qrCode() command */
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

// ....

// Cut & close
$printer -> cut();
$printer -> close();

function title(Escpos $printer, $str) {
    $printer -> selectPrintMode(Printer::MODE_DOUBLE_HEIGHT | Printer::MODE_DOUBLE_WIDTH);
    $printer -> text($str);
    $printer -> selectPrintMode();
}

In the QR code post, I posted a fallback which used software rendering. As I don’t have a PHP-based PDF417 code library, you will need a printer which supports them to be blue to use these examples.

What's in the AEC data feed, and how to use it

Thu, 30 Jun 2016 00:00:00 +0000

With an election coming up, it’s probably a good time to post some notes about using Australian election data feeds. This is mainly aimed at any interested programmers who have a use for this sort of data.

I’ve left out any example code here, as what you write will be specific to how you use the data.

Overview

The AEC publishes results on election night, by simply posting zipped files on an FTP server every few minutes. The detailed documentation is at aec.gov.au.

I have experimented with loading the data into a database during a recent by-election with some success. The data model that I assembled for this was:

You will be able to locate many electoral concepts in this data model, as candidates with a particular affiliation receive votes (contest eachother) at different polling places for a seat in one of the houses. This maps to the more extensive XML format to extract the fields that I was most interested in mapping.

Getting the data

The FTP server is mediafeed.aec.gov.au, and it accepts anonymous login on election night.

Here you will find a numbered folder for the night’s election. Within this, there is a lot of repetition, so you can ignore most of the files straight away. The sub-folders that I suggest loading are Detailed/LightProgress and Detailed/Preload. For example, in a past election, the file layout was:

18126/
18126/Detailed
18126/Detailed/LightProgress
18126/Detailed/LightProgress/aec-mediafeed-Detailed-LightProgress-18126-20150919194400.zip
18126/Detailed/LightProgress/aec-mediafeed-Detailed-LightProgress-18126-20150919194527.zip
18126/Detailed/LightProgress/aec-mediafeed-Detailed-LightProgress-18126-20150919194702.zip
18126/Detailed/LightProgress/aec-mediafeed-Detailed-LightProgress-18126-20150911140244.zip
18126/Detailed/LightProgress/aec-mediafeed-Detailed-LightProgress-18126-20150919195500.zip
18126/Detailed/LightProgress/...
18126/Detailed/Preload
18126/Detailed/Preload/aec-mediafeed-Detailed-Preload-18126-20150911140243.zip

Loading the data

Start with the Preload data. This zip file is available first, and contains the candidate names, polling places, event details, and an initial (zeroed-out) results feed. Using a custom script and a schema like the one I’ve posted above, you will be able to import this into your database.

Next, the LightProgress feed updates will start being added every few minutes from 7pm. These Zip files contain a small piece of XML, each one superseding older files.

I’ve cut down one of these files as an example:

<?xml version="1.0" encoding="utf-8"?>
<MediaFeed Id="e9334806-990e-4773-8b00-7d74fa58b6af" Created="2015-09-19T19:55:00" SchemaVersion="3" EmlVersion="5" xmlns="http://www.aec.gov.au/xml/schema/mediafeed" xmlns:eml="urn:oasis:names:tc:evs:schema:eml" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:xal="urn:oasis:names:tc:ciq:xsdschema:xAL:2.0" xmlns:xnl="urn:oasis:names:tc:ciq:xsdschema:xNL:2.0" xmlns:ts="urn:oasis:names:tc:evs:schema:eml:ts" xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:schemaLocation="http://www.aec.gov.au/xml/schema/mediafeed ../Schema/AEC/aec-mediafeed-results-v3-0.xsd">
  <ManagingAuthority>
    <eml:AuthorityIdentifier Id="AEC">Australian Electoral Commission</eml:AuthorityIdentifier>
  </ManagingAuthority>
  <MessageLanguage>en</MessageLanguage>
  <MessageGenerator>
    <Name>Virtual Tally Room</Name>
    <Environment>PROD</Environment>
    <Site>CDC</Site>
    <Server>DB02</Server>
    <Platform>x64</Platform>
    <Version>9.2.0.28493</Version>
  </MessageGenerator>
  <Cycle Created="2015-09-19T19:54:47">37f177b4-ab20-4073-be85-0edefa5c8e96</Cycle>
  <Results Phase="ElectionNight" Verbosity="LightProgress" Granularity="Detailed">
    <eml:EventIdentifier Id="18126">
      <eml:EventName>Canning By-election</eml:EventName>
    </eml:EventIdentifier>
    <Election>
      <eml:ElectionIdentifier Id="H">
        <eml:ElectionName>House of Representatives By-election for the division of Canning</eml:ElectionName>
        <eml:ElectionCategory>ByElection</eml:ElectionCategory>
      </eml:ElectionIdentifier>
      <House>
        <Contests>
          <Contest Projected="true">
            <eml:ContestIdentifier Id="236">
              <eml:ContestName>Canning</eml:ContestName>
            </eml:ContestIdentifier>
            <Enrolment>112809</Enrolment>
            <FirstPreferences PollingPlacesReturned="0" PollingPlacesExpected="53">
              <Candidate>
                <eml:CandidateIdentifier Id="25424" />
                <BallotPosition>1</BallotPosition>
                <Elected>false</Elected>
                <Votes MatchedHistoric="0">0</Votes>
                <VotesByType>
                  <Votes Type="Ordinary">0</Votes>
                  <Votes Type="Absent">0</Votes>
                  <Votes Type="Provisional">0</Votes>
                  <Votes Type="PrePoll">0</Votes>
                  <Votes Type="Postal">0</Votes>
                </VotesByType>
              </Candidate>
              <!-- (other candidates, ghost candidates) -->
              <Formal>
                <Votes MatchedHistoric="0">0</Votes>
                <VotesByType>
                  <Votes Type="Ordinary">0</Votes>
                  <Votes Type="Absent">0</Votes>
                  <Votes Type="Provisional">0</Votes>
                  <Votes Type="PrePoll">0</Votes>
                  <Votes Type="Postal">0</Votes>
                </VotesByType>
              </Formal>
              <Informal>
                <Votes MatchedHistoric="0">0</Votes>
                <VotesByType>
                  <Votes Type="Ordinary">0</Votes>
                  <Votes Type="Absent">0</Votes>
                  <Votes Type="Provisional">0</Votes>
                  <Votes Type="PrePoll">0</Votes>
                  <Votes Type="Postal">0</Votes>
                </VotesByType>
              </Informal>
              <Total>
                <Votes MatchedHistoric="0">0</Votes>
                <VotesByType>
                  <Votes Type="Ordinary">0</Votes>
                  <Votes Type="Absent">0</Votes>
                  <Votes Type="Provisional">0</Votes>
                  <Votes Type="PrePoll">0</Votes>
                  <Votes Type="Postal">0</Votes>
                </VotesByType>
              </Total>
            </FirstPreferences>
            <TwoCandidatePreferred Restricted="true" PollingPlacesReturned="0" PollingPlacesExpected="53" />
            <TwoPartyPreferred>
              <Coalition>
                <CoalitionIdentifier Id="2" />
                <Votes>0</Votes>
              </Coalition>
              <Coalition>
                <CoalitionIdentifier Id="1" />
                <Votes>0</Votes>
              </Coalition>
            </TwoPartyPreferred>
            <PollingPlaces>
              <PollingPlace>
                <PollingPlaceIdentifier Id="7467" />
                <FirstPreferences />
                <TwoCandidatePreferred Restricted="true" />
              </PollingPlace>
              <!-- (other polling places, ghost candidates) -->
            </PollingPlaces>
          </Contest>
        </Contests>
      </House>
    </Election>
  </Results>
</MediaFeed>

Spatial data

You can also download electorate boundaries as spatial data here to back any visualisations of the results.

The locations of polling places are already encoded in the Preload data file.

How to set up Docker containers in Travis CI

Thu, 23 Jun 2016 00:00:00 +0000

This post outlines a method for using Docker for testing on Travis CI. It may be useful to you if you are a web application developer who uses GitHub.

I use this setup in my web-based word puzzle generator, so that every change is spun up and tested with a web browser before it is merged.

I got the idea for writing this from a few lines in the docker-compose documentation, which suggested that Docker is an easy way to perform automated testing over a running application:

$ docker-compose up -d
$ ./run_tests
$ docker-compose down

This snippet was missing some setup and an example app, but these three lines do all the heavy lifting.

The Docker setup

In order to focus on the Docker setup, I made a server which simply responds to TCP requests on port 5000 with the text "Hello World".

This file is called server.sh, and sits in a directory called foo_server:

#!/bin/sh
while true; do
  # Send 'Hello World' to anybody who connects on port 5000
  echo "Hello World" | nc -l 5000
done

Alongside it, I added a Dockerfile to instruct docker to execute this tiny ‘application’ in a container, after installing the dependencies. This machine is built from the Docker-official Debian image:

FROM debian
ADD . /usr/share/test-server
WORKDIR /usr/share/test-server
RUN apt-get update && apt-get install --assume-yes netcat-openbsd
CMD ./server.sh

Lastly, a .dockerignore file is used to avoid loading the Dockerfile to the container:

# Ignore docker files
Dockerfile
.dockerignore

In the directory above, a simple test script, test.sh is used to see that the server is returning the expected output:

#!/bin/sh
set -e
expected="Hello World"
actual=`nc -v localhost 5000`
echo "Expecting: $expected"
echo "Server says: $actual"
if [ "$expected" != "$actual" ]; then
  echo "Test failed"
  exit 1
else
  echo "Test passed"
  exit 0
fi

Alongside the test file, a file called docker-compose.yml instructs Docker to create a container out of the foo_server example, and forward port 127.0.0.1:5000 to it.

version: '2'
services:
  foo:
    build: foo_server
    ports:
     - "5000:5000"
    container_name: foo_1

To try it out for yourself, you need a relatively recent version of Docker and docker-compose. The versions provided in Debian were not new enough to execute the examples, but the Docker project provides repos containing newer builds for Debian & Ubuntu. For my distro, the install was:

curl -sSL "https://get.docker.com/gpg" | sudo -E apt-key add -
echo "deb https://apt.dockerproject.org/repo debian-stretch main" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install docker-engine
sudo pip install docker-compose

The versions this got me were docker 1.11.2, and docker-compose 1.7.1. Straight after the install, I could deploy & test an example locally:

$ docker-compose up --build -d
$ ./test.sh
$ docker-compose down

The CI setup

I’ll assume that if you’re reading this, you are familiar with the basics of Travis CI. The large block of code below is the .travis.yml file to set up the test machine, then execute the tests against a container.

---
# Use Ubuntu 'trusty' distribution
sudo: required
dist: trusty

install:
  # Update docker-engine using Ubuntu 'trusty' apt repo
  - >
    curl -sSL "https://get.docker.com/gpg" |
     sudo -E apt-key add -
  - >
    echo "deb https://apt.dockerproject.org/repo ubuntu-trusty main" |
     sudo tee -a /etc/apt/sources.list
  - sudo apt-get update
  - >
    sudo apt-get -o Dpkg::Options::="--force-confdef" \
     -o Dpkg::Options::="--force-confold" --assume-yes install docker-engine
  - docker version
  # Update docker-compose via pip
  - sudo pip install docker-compose
  - docker-compose version

before_script:
  - docker-compose up --build -d

script:
  - ./test.sh

after_script:
  - docker-compose down
...

Note: This uses Travis CI’s trusty distribution, which at the time of writing is the newest stable build platform available on Travis CI. This shipped an outdated version of Docker, which had to be installed over. Because the existing Docker was configured, I had to override a debconf prompt, which is why the apt addon syntax was not used to set up dependencies.

Result

The build result for each commit is displayed in Travis CI:

Under this, the output of the passing test script is shown, showing what has been set up:

Using this setup in practice

Moving this from a demo setup to a real setup would be fairly simple:

Replace the installation with a real software stack
Replace the server run with a command to serve the application (such as a Apache HTTP, Tomcat or Node)
Replace the tests with real tests (such as Cucumber or Selenium).

The example in the pre-amble installs a LAMP stack and tests it with Selenium in its CI build.

If your application is a bit larger, your only extra complexity will come from running multiple containers with docker-compose.

Get the code

All of these scripts in a working CI example are available on GitHub: mike42/minimal-docker-ci

Print larger or smaller text on a thermal receipt printer

Thu, 16 Jun 2016 00:00:00 +0000

If you print to a thermal receipt printer which support the ESC/POS protocol, then you can format the receipts to make larger or smaller text.

If this is your first time reading about ESC/POS, have a read of What is ESC/POS, and how do I use it?. Some of these text size examples are borrowed from there, while some are new for this blog post.

Smaller text

Example

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Fonts (many printers do not have a 'Font C') */
$fonts = array(
	Printer:::FONT_A,
	Printer:::FONT_B,
	Printer:::FONT_C);
for($i = 0; $i < count($fonts); $i++) {
	$printer -> setFont($fonts[$i]);
	$printer -> text("The quick brown fox jumps over the lazy dog\n");
}
$printer -> setFont(); // Reset
$printer -> cut();
$printer -> close();

Hexdump

00000000  1b 40 1b 4d 00 54 68 65  20 71 75 69 63 6b 20 62  |.@.M.The quick b|
00000010  72 6f 77 6e 20 66 6f 78  20 6a 75 6d 70 73 20 6f  |rown fox jumps o|
00000020  76 65 72 20 74 68 65 20  6c 61 7a 79 20 64 6f 67  |ver the lazy dog|
00000030  0a 1b 4d 01 54 68 65 20  71 75 69 63 6b 20 62 72  |..M.The quick br|
00000040  6f 77 6e 20 66 6f 78 20  6a 75 6d 70 73 20 6f 76  |own fox jumps ov|
00000050  65 72 20 74 68 65 20 6c  61 7a 79 20 64 6f 67 0a  |er the lazy dog.|
00000060  1b 4d 02 54 68 65 20 71  75 69 63 6b 20 62 72 6f  |.M.The quick bro|
00000070  77 6e 20 66 6f 78 20 6a  75 6d 70 73 20 6f 76 65  |wn fox jumps ove|
00000080  72 20 74 68 65 20 6c 61  7a 79 20 64 6f 67 0a 1b  |r the lazy dog..|
00000090  4d 00 1d 56 41 03                                 |M..VA.|
00000096

Change height & width

Example

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Text of various (in-proportion) sizes */
title($printer, "Change height & width\n");
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize($i, $i);
    $printer -> text($i);
}
$printer -> text("\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}

Hexdump

00000000  1b 40 1b 21 08 0a 43 68  61 6e 67 65 20 68 65 69  |.@.!..Change hei|
00000010  67 68 74 20 26 20 77 69  64 74 68 0a 1b 21 00 1d  |ght & width..!..|
00000020  21 00 31 1d 21 11 32 1d  21 22 33 1d 21 33 34 1d  |!.1.!.2.!"3.!34.|
00000030  21 44 35 1d 21 55 36 1d  21 66 37 1d 21 77 38 0a  |!D5.!U6.!f7.!w8.|
00000040  1d 56 41 03                                       |.VA.|
00000044

Change width only (height=4)

Example

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Width changing only */
title($printer, "Change width only (height=4):\n");
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize($i, 4);
    $printer -> text($i);
}
$printer -> text("\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}

Hexdump

00000000  1b 40 1b 21 08 0a 43 68  61 6e 67 65 20 77 69 64  |.@.!..Change wid|
00000010  74 68 20 6f 6e 6c 79 20  28 68 65 69 67 68 74 3d  |th only (height=|
00000020  34 29 3a 0a 1b 21 00 1d  21 03 31 1d 21 13 32 1d  |4):..!..!.1.!.2.|
00000030  21 23 33 1d 21 33 34 1d  21 43 35 1d 21 53 36 1d  |!#3.!34.!C5.!S6.|
00000040  21 63 37 1d 21 73 38 0a  1d 56 41 03              |!c7.!s8..VA.|
0000004c

Change height only (width=4)

Example

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Height changing only */
title($printer, "Change height only (width=4):\n");
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize(4, $i);
    $printer -> text($i);
}
$printer -> text("\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}

Hexdump

00000000  1b 40 1b 21 08 0a 43 68  61 6e 67 65 20 68 65 69  |.@.!..Change hei|
00000010  67 68 74 20 6f 6e 6c 79  20 28 77 69 64 74 68 3d  |ght only (width=|
00000020  34 29 3a 0a 1b 21 00 1d  21 30 31 1d 21 31 32 1d  |4):..!..!01.!12.|
00000030  21 32 33 1d 21 33 34 1d  21 34 35 1d 21 35 36 1d  |!23.!34.!45.!56.|
00000040  21 36 37 1d 21 37 38 0a  1d 56 41 03              |!67.!78..VA.|
0000004c

Very narrow font

Example

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Very narrow text */
title($printer, "Very narrow text:\n");
$printer -> setTextSize(1, 8);
$printer -> text("The quick brown fox jumps over the lazy dog.\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}

Hexdump

00000000  1b 40 1b 21 08 0a 56 65  72 79 20 6e 61 72 72 6f  |.@.!..Very narro|
00000010  77 20 74 65 78 74 3a 0a  1b 21 00 1d 21 07 54 68  |w text:..!..!.Th|
00000020  65 20 71 75 69 63 6b 20  62 72 6f 77 6e 20 66 6f  |e quick brown fo|
00000030  78 20 6a 75 6d 70 73 20  6f 76 65 72 20 74 68 65  |x jumps over the|
00000040  20 6c 61 7a 79 20 64 6f  67 2e 0a 1d 56 41 03     | lazy dog...VA.|
0000004f

Very wide font

Example

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Very flat text */
title($printer, "Very wide text:\n");
$printer -> setTextSize(4, 1);
$printer -> text("Hello world!\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}

Hexdump

00000000  1b 40 1b 21 08 0a 56 65  72 79 20 77 69 64 65 20  |.@.!..Very wide |
00000010  74 65 78 74 3a 0a 1b 21  00 1d 21 30 48 65 6c 6c  |text:..!..!0Hell|
00000020  6f 20 77 6f 72 6c 64 21  0a 1d 56 41 03           |o world!..VA.|
0000002d

Largest possible font

Example

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Very large text */
title($printer, "Largest possible text:\n");
$printer -> setTextSize(8, 8);
$printer -> text("Hello\nworld!\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}

Hexdump

00000000  1b 40 1b 21 08 0a 4c 61  72 67 65 73 74 20 70 6f  |.@.!..Largest po|
00000010  73 73 69 62 6c 65 20 74  65 78 74 3a 0a 1b 21 00  |ssible text:..!.|
00000020  1d 21 77 48 65 6c 6c 6f  0a 77 6f 72 6c 64 21 0a  |.!wHello.world!.|
00000030  1d 56 41 03                                       |.VA.|
00000034

Double-height and doule-width

There are also commands which specifically double the width and height. These overlap with the commands covered here, but you can find them in the escpos-php documentation.

Full example

These examples were authored for escpos-php, a PHP printer driver for thermal receipt printers. The full file ships as an example with the driver, and outputs a block of ESC/POS code which can be sent a printer to give the below output.

Receipt

PHP code

<?php
/**
 * This print-out shows how large the available font sizes are. It is included
 * separately due to the amount of text it prints.
 */
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;

$connector = new FilePrintConnector("php://stdout");
$printer = new Printer($connector);

/* Initialize */
$printer -> initialize();

/* Text of various (in-proportion) sizes */
title($printer, "Change height & width\n");
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize($i, $i);
    $printer -> text($i);
}
$printer -> text("\n");

/* Width changing only */
title($printer, "Change width only (height=4):\n");
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize($i, 4);
    $printer -> text($i);
}
$printer -> text("\n");

/* Height changing only */
title($printer, "Change height only (width=4):\n");
for ($i = 1; $i <= 8; $i++) {
    $printer -> setTextSize(4, $i);
    $printer -> text($i);
}
$printer -> text("\n");

/* Very narrow text */
title($printer, "Very narrow text:\n");
$printer -> setTextSize(1, 8);
$printer -> text("The quick brown fox jumps over the lazy dog.\n");

/* Very flat text */
title($printer, "Very wide text:\n");
$printer -> setTextSize(4, 1);
$printer -> text("Hello world!\n");

/* Very large text */
title($printer, "Largest possible text:\n");
$printer -> setTextSize(8, 8);
$printer -> text("Hello\nworld!\n");

$printer -> cut();
$printer -> close();

function title(Printer $printer, $text)
{
    $printer -> selectPrintMode(Printer::MODE_EMPHASIZED);
    $printer -> text("\n" . $text);
    $printer -> selectPrintMode(); // Reset
}

Other formatting?

If text size is not what you’re after, then you can find similar examples in other posts I’ve tagged escpos.

How to compile a C++11 app on Travis CI

Thu, 09 Jun 2016 00:00:00 +0000

I have recently been adding Travis CI builds to code that I host on GitHub, so that I don’t need to host my own build infrastructure.

To users, this just means that there is a green badge at the top of the README, but not much else:

To build a simple C++ project, I added in this .travis.yml file:

langauge: cpp
sudo: false

addons:
  apt:
    packages:
      - libusb-1.0-0-dev

script:
  - make

Unfortunately the default build tools are ancient on this infrastructure, which uses Ubuntu Precise (12.04):

$ make
g++ src/missile.cpp examples/basic-sync/basic-sync.cpp -o bin/basic-sync -lpthread -lusb-1.0 -std=c++11 -Wall
cc1plus: error: unrecognized command line option ‘-std=c++11’
cc1plus: error: unrecognized command line option ‘-std=c++11’

Option 1: Update the toolchain

There is some structures you can use to install an extra repository and some named packages, instead of using apt-get directly.

langauge: cpp
sudo: false

addons:
  apt:
    sources:
    - ubuntu-toolchain-r-test
    packages:
    - gcc-4.8
    - g++-4.8
    - libusb-1.0-0-dev

script:
  - make

Because the old version was still installed, I had to refer to the exact version in the Makefile, as in:

g++-4.8 src/missile.cpp examples/basic-sync/basic-sync.cpp -o bin/basic-sync -lpthread -lusb-1.0 -std=c++11 -Wall

Option 2: Update the platform

You can also change to a more recent Ubuntu distribution. Presumably Ubuntu Precise is only the default because existing builds use it.

If you need to build C++11 apps on Travis CI, then builds will work under Ubuntu Trusty (14.04), which happens to be the newest distribution currently available:

langauge: cpp
sudo: required
dist: trusty

addons:
  apt:
    packages:
      - libusb-1.0-0-dev

script:
  - make

How to edit emulator flags in Android Studio

Thu, 26 May 2016 00:00:00 +0000

I’ve recently updated to the new IntelliJ-based Android studio 2.1.

I ran into some issues attempting to launch a “hello world” project in the emulator, which I’m writing up here for the benefit of others.

For context, I run Debian GNU/Linux Jessie on an AMD64 box, with Radeon graphics card, using the free drivers.

The OpenGL error

When attempting to launch a simple project with the emulator, the emulator died with the following message:

Cannot launch AVD in emulator.
Output:
libGL error: unable to load driver: radeonsi_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: radeonsi
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
X Error of failed request:  GLXBadContext
  Major opcode of failed request:  155 (GLX)
  Minor opcode of failed request:  6 (X_GLXIsDirect)
  Serial number of failed request:  49
  Current serial number in output stream:  48
libGL error: unable to load driver: radeonsi_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: radeonsi
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
X Error of failed request:  GLXBadContext
  Major opcode of failed request:  155 (GLX)
  Minor opcode of failed request:  6 (X_GLXIsDirect)
  Serial number of failed request:  49
  Current serial number in output stream:  48
libGL error: unable to load driver: radeonsi_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: radeonsi
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  155 (GLX)
  Minor opcode of failed request:  24 (X_GLXCreateNewContext)
  Value in failed request:  0x0
  Serial number of failed request:  33
emulator: WARNING: VM heap size set below hardware specified minimum of 228MB
emulator: WARNING: Setting VM heap size to 384MB
  Current serial number in output stream:  34
QObject::~QObject: Timers cannot be stopped from another thread

The console logs that the command being executed is:

/home/mike/Android/Sdk/tools/emulator -netdelay none -netspeed full -avd Nexus_5X_API_23

Fixing on the command-line

Things that didn’t work

Installing more libraries via apt-get did not help.

apt-get install libstdc++6 xserver-xorg-video-radeon

Pre-loading libraries also didn’t solve the issue.

LD_PRELOAD='/usr/lib64/libstdc++.so.6' /home/mike/Android/Sdk/tools/emulator -netdelay none -netspeed full -avd Nexus_5X_API_23

Attempting to understand LibGL’s debug output was also not particularly useful in my case.

LIBGL_DEBUG=verbose /home/mike/Android/Sdk/tools/emulator -netdelay none -netspeed full -avd Nexus_5X_API_23

What did work for me

The solution turned out to be this specific flag:

$ ./emulator -help | grep libstdc++
    -use-system-libs               Use system libstdc++ instead of bundled one

So launching the emulator from the CLI worked for me, using this command:

/home/mike/Android/Sdk/tools/emulator -netdelay none -netspeed full -avd Nexus_5X_API_23 -use-system-libs

Fixing in the IDE

So this was fantastic progress, but without being able to launch this emulator from Android Studio, the development environment wasn’t really integrated yet.

Apparently there used to be a feature for adding command-line flags to the emulator, but this is now gone.

So, adapting a post here, I jumped in and replaced the emulator binary with a wrapper to inject some flags.

$ cd ~/Android/Sdk/tools/
$ mv emulator emulator.0
$ touch emulator
$ chmod +x emulator

I then opened up the emulator in a text editor and punched in this:

#!/bin/sh
set -ex
$0.0 $@ -use-system-libs

Result

Success!

Blog domain migrated

Tue, 03 May 2016 00:00:00 +0000

As of today, this blog has now moved to a shorter domain name: https://mike42.me.

Hopefully, none of the content got lost in the move!

Quick guide: Running stock Debian on the Raspberry Pi 2

Fri, 22 Apr 2016 00:00:00 +0000

At the time of writing, the ‘Raspbian’ port of Debian is often used on the Raspberry Pi. It was created to match the CPU architecture, for better performance. These reasons don’t apply to the newer Raspberry Pi 2, so if you’re a Debian desktop or server user, you can do away with the fork and just run Debian Jessie armhf.

The info from Debian is: https://wiki.debian.org/RaspberryPi2

A bit more background about why this only applies to the Raspberry Pi 2:

The Raspberry Pi 1 uses ARMv6 chipset with hard floats
- The Debian armhf port requires ARMv7
- The Debian armel port doesn’t use hard floats, so is unnecessarily slow on the Pi.
- So Raspbian was created for the Raspberry Pi 1’s ARMv6 w/ hard-floats, and gets the most juice out of the CPU on the Raspberry Pi 1.
The Raspberry Pi 2 uses ARMv7 with hard floats, so the Debian armhf port is fine.

Install the image

The image is linked to from this page:

http://sjoerd.luon.net/posts/2015/02/debian-jessie-on-rpi2/

I will assume that your machine has an SD card slot. To find the device name, list out disks and look for one of the correct size, which appears when you plug in the card:

df

Download a copy of the image, extract it out, and dd the file on to the card:

wget -c https://images.collabora.co.uk/rpi2/jessie-rpi2-20150705.img.gz
gunzip jessie-rpi2-20150705.img.gz 
sudo dd if=jessie-rpi2-20150705.img of=/dev/sdX bs=4M
sudo sync
umount /media/$USER/*

Plug in the Raspberry pi, and then log in. If you are using SSH, then arp-scan is a good tool to pick up devices on the network:

sudo apt-get install arp-scan
sudo arp-scan -l
ssh root@x.y.z.w

Configure pi- Things like screen resolution and HDMI go here:

cd /boot/firmware/
nano config.txt

Perform a software upgrade:

nano /etc/apt/sources.list
apt-get update
apt-get dist-upgrade

Start fixing security defaults. Remember that this is not a clean install, so start by setting your own passwords:

passwd

Check that there are no other accounts with passwords set:

cat /etc/shadow

Regenerate all SSH Server keys (commands from here):

ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa -b 521
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa

Lastly, generate some locales:

sudo locale-gen en_US en_US.UTF-8 en_GB en_GB.UTF-8

How to resize a Windows VM image with virt-resize

Thu, 10 Mar 2016 00:00:00 +0000

I recently had a Windows 7 Virtual Machine stored on an undersized qcow2 file. This post steps through the simplest way that I know to produce a new, bigger disk and expand the filesystem onto it.

Empty out the empty space

Because the guest VM is stored on a QCOW2 file, we can recover un-used space on disk by zeroing it out now. Download the sdelete utility from Microsoft and run it on the system.

sdelete -z

One this is done, power off the guest.

Assuming the host is linux, you need the `qemu-utls` and `libguestfs-tools` packages to follow these steps. On Debian-

apt-get install libguestfs-tools qemu-utls

Move the VM image to a new filename and inspect it.

mv windows.img windows.img.bak

The file command indicates that the disk is about 30GB expanded.

$ file windows.img.bak 
windows.img.bak: QEMU QCOW Image (v3), 32212254720 bytes

The qemu-img command shows that the disk is 83% full:

$ qemu-img check windows.img.bak 
No errors were found on the image.
411337/491520 = 83.69% allocated, 5.66% fragmented, 0.00% compressed clusters
Image end offset: 26961969152

Check out your FS names, note that /dev/sda2 is the disk we want to up-size in this case

$ virt-filesystems -a windows.img -l
Name       Type        VFS   Label            Size         Parent
/dev/sda1  filesystem  ntfs  System Reserved  104857600    -
/dev/sda2  filesystem  ntfs  -                32105299968  -

Make a new, bigger disk image

Create a new disk of the desired size. In my case, 50G is sufficient:

$ qemu-img create -f qcow2 windows.img 50G
Formatting 'windows.img', fmt=qcow2 size=53687091200 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

The file command shows that this new empty disk image is larger than the old image.

$ file windows.img
windows.img: QEMU QCOW Image (v3), 53687091200 bytes

Copy the old disk to the new one. The --expand option names a partition which will be grown to fill the extra space.

virt-resize --expand /dev/sda2 windows.img.bak windows.img

The virt-resize command shows a progress bar while it works, and zero-blocks will be reclaimed as a result of the output format:

The final line of output suggests holding on to your backup until you’ve checked it, which is wise:

“Resize operation completed with no errors. Before deleting the old disk, carefully check that the resized disk boots and works correctly.”

Check that the new disk is valid and contains partitions at the expected size:

$ virt-filesystems -a windows.img -l
Name       Type        VFS   Label            Size         Parent
/dev/sda1  filesystem  ntfs  System Reserved  104857600    -
/dev/sda2  filesystem  ntfs  -                53579939840  -

Boot up the guest

When the machine boots up, you may get a disk check prompt. Because the console I was using triggered the ‘Press any key to cancel’ prompt, I had to reboot and leave the console disconnected in order for the check to start.

After booting, the C:\ drive should display at its new size:

Convert a PC to a HTPC with Debian and Kodi

Thu, 25 Feb 2016 00:00:00 +0000

I recently converted an old workstation to run as a home-theatre PC (HTPC). I’ve noted down the setup here for others who are making an installation like this. Some steps depend on using a radeon chipset, and will need to be adjusted for your computer.

Hardware

First up, Desktop ’towers’ are not a good form-factor for sitting in TV cabinets. If your PC is this sort of size, then source a small form-factor case and power supply, and load the computer’s components into it:

I also used a Logitech k400r keyboard and mouse for wireless input.

Install Debian and apps

Write the latest copy of Debian Stable to a CD or flash drive (this is version 8.3 at time of writing), and install it on the computer. Check “Debian Desktop environment” / GNOME during setup.

After installation, open a terminal, and type “su” to get root privileges.

su

Edit the software sources.

nano /etc/apt/sources.list

Update this to include ‘contrib’ and ’non-free’, as well as ‘jessie-backports’

deb http://ftp.us.debian.org/debian/ jessie main contrib non-free
deb-src http://ftp.us.debian.org/debian/ jessie main contrib non-free

deb http://security.debian.org/ jessie/updates main contrib non-free
deb-src http://security.debian.org/ jessie/updates main contrib non-free

# jessie-updates, previously known as 'volatile'
deb http://ftp.us.debian.org/debian/ jessie-updates main contrib non-free
deb-src http://ftp.us.debian.org/debian/ jessie-updates main contrib non-free

# jessie-backports
deb http://ftp.us.debian.org/debian/ jessie-backports main contrib non-free
deb-src http://ftp.us.debian.org/debian/ jessie-backports main contrib non-free

Update sources and install Kodi:

apt-get install --install-suggests kodi

Also install the firmware packages that you may need.

apt-get install firmware-linux-free firmware-amd-graphics

Tweaks

Sudo

sudo allows you to run commands as root from your regular user account. Install the package and add yourself to the sudo group:

apt-get install sudo
usermod -a -G sudo mike

To apply the change, log out and back in again. The rest of this guide assumes you are logged in as yourself, and will use sudo where necessary.

Auto-start Kodi

Open the tweak tool, and locate the list of startup programs.

gnome-tweak-tool

Add Kodi to the list, log out, log in, and Kodi will launch automatically.

For a PC attached to a TV, user permissions are not so importnat, so set the user to log in automatically.

sudo nano /etc/gdm3/daemon.conf

Un-comment this block and enter your username:

# Enabling automatic login
#  AutomaticLoginEnable = true
#  AutomaticLogin = user1

Plymouth start-up screen

Install plymouth and configure grub to change the Debian boot sequence (a menu with timeout, followed by lots of text) into a graphical splash screen. This takes a bit of configuration.

sudo apt-get install plymouth

Set it up according to these instructions:

sudo nano /etc/initramfs-tools/modules

Set drm correctly for your chipset:

# KMS
drm
radeon modeset=1

Configure grub.

sudo nano /etc/default/grub

Set these options.

...
GRUB_TIMEOUT=0
...
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
...
GRUB_GFXMODE=1920x1080
...

Update grub, set the theme in Plymouth:

sudo update-grub2
sudo /usr/sbin/plymouth-set-default-theme --list
sudo /usr/sbin/plymouth-set-default-theme joy

Run update-initramfs to apply the changes

sudo update-initramfs -u

Samba

Samba will let you share folders over your network. A basic folder with guest read/write is simple to set up:

sudo apt-get install nautilus-share samba libpam-smbpass winbind
sudo usermod -a -G sambashare mike

Log out, and back in to apply the group change, and then share the Public folder over the network by right-clicking on it and opening the “Sharing Options”:

Gnome will warn that the folder as shared if you open it:

Test the setup by typing smb://localhost into the address bar:

Overscan correction

In my case, I was able to set the TV to treat the input as a “PC” input. If that doesn’t work for you, then use xrandr in a login script:

Find the name of your input:

xrandr --query

Set underscan (get the horizontal and vertical values by trial and error):

xrandr --output HDMI-0 --set underscan on
xrandr --output HDMI-0 --set "underscan hborder" 32 --set "underscan vborder" 16

Kodi plugins

Add these as needed. The Australian catchup TV plugins repository from GitHub worked well.

Kodi RSS

The RSS feed shows Kodi updates by default, and is part of your user profile.

Edit the configuration file, and adjust the paths to your news sources of choice.

Boot speed

Readahead is the tool of choice for boot speed optimisation. Install it, and reboot.

sudo apt-get install readahead
sudo touch  /.readahead_collect
sudo reboot

Desktop Apps

If you quit Kodi, you are dropped back to the GNOME desktop. These apps are simply to improve the desktop user experience.

Google Chrome

Download the .deb file for Chrome from Google, install with dpkg, and then clean up dependencies:

dpkg -i google-chrome-stable_current_amd64.deb 
apt-get -f install

Firefox

Download and extract the Firefox for Linux tarball from Mozilla.

Move it to /usr/share, and change the owner to match other applications there.

mv firefox /usr/share/
cd /usr/share/
ls -Ahl
chown root:root firefox
chown -R root:root firefox

Find the main menu editor, and add Firefox to the menu.

Name: Firefox Web Browser
Command: /usr/share/firefox/firefox-bin
Icon: /usr/share/firefox/browser/icons/mozicon128.png

Test the new icon by searching:

Auto-clear browser profiles

Because you don’t need a password to log in to the user account, you can add this as a bit of security so that your box won’t remember any passwords or sessions.

crontab -e

This job removes the Firefox and Chrome user profiles each boot.

@reboot rm --preserve-root -Rf -- ~/.config/google-chrome ~/.cache/google-chrome ~/.mozilla/firefox ~/.cache/mozilla/firefox

VLC

For file format support, best to have another media player:

sudo apt-get install vlc

Result

You should now have a PC which boots into Kodi for media and TV, and lets you quit into a desktop to browse the web or run regular desktop apps.

On the 1GB RAM / dual-core workstation, it still took around 45 seconds from the BIOS handing over control to Kodi being ready.

Update 2017-12-29: Re-installing this setup on an SSD shortened the boot time to 21 seconds. This includes boot, login, and an application start.

How to regenerate SSH client keys

Fri, 22 Jan 2016 00:00:00 +0000

Recently, a SSH client bug was discovered that could let the server read client private keys in some situations.

If you’re affected, then follow these steps. Regular users of SSH may be familiar with the commands used.

If you use public key authentication

If you do use keys to authenticate, you should regenerate them. Start by backing up your old key. Assuming it’s at the default location, just use-

$ mv ~/.ssh/id_rsa ~/.ssh/id_rsa.old
$ mv ~/.ssh/id_rsa.pub ~/.ssh/id_rsa.pub.old

Next, create or edit ~/.ssh/config, and add the following line. This disables the roaming feature, which was part of the problem:

UseRoaming no

Generate new keys:

$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/example/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/example/.ssh/id_rsa.
Your public key has been saved in /home/example/.ssh/id_rsa.pub.

Now copy the new public key over, using the old keys. You would normally use ssh-copy-id, but it adds keys rather than replacing them, which is why we need to do it manually:

scp -i ~/.ssh/id_rsa.old ~/.ssh/id_rsa.pub user@example.com:/home/user/id_rsa.pub.new

SSH in with the old key:

ssh -i ~/.ssh/id_rsa.old

Replace all current authorized keys with the newly generated one:

mv -f id_rsa.pub.new ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Logout, log back in with the new key to make sure it works, and you’re done.

If you don’t use public keys

If you don’t use keys for login, now could be a good time to start. Public-key authentication provides much stronger protection from brute force attacks, and also prevents the need for the server to be sent your password each time you log in.

Create or edit ~/.ssh/config as above, and make sure it contains this line-

UseRoaming no

Generate a keypair and add your public key to the server-

ssh-keygen
ssh-copy-id user@example.com

Test it out:

ssh user@example.com

At his point, you should disable password-based authentication in OpenSSH server. If you don’t control the server’s SSH configuration, then just set a good random password for your account to prevent it being brute-forced:

openssl rand -base64 32
cat /dev/urandom | head -c32 | base64

Set it on your account:

passwd

Update 2017-10-22: Updated to correct errors in the commands.

Basic Cucumber feature testing

Sat, 16 Jan 2016 00:00:00 +0000

Cucumber is a Behaviour-Driven-Development (BDD) tool. You can use to automate the acceptance tests for your software systems. A real-world example of its use in a large application is the MediaWiki QA process.

This post will show you the steps to get it up-and-running to test a trivial app on a Unix (Mac, Linux) computer. I’ll assume that you already have Ruby installed.

Installation

First up, install Cucumber’s Ruby implementation:<

$ sudo gem install cucumber
$ sudo gem install rspec

If you’re on Debian, you can also achieve this with apt-get:

$ sudo apt-get install ruby-cucumber-core ruby-rspec

The versions that I got from the install were-

$ ruby --version
ruby 2.2.3p173 (2015-08-18) [x86_64-linux-gnu]
$ cucumber --version
2.3.0

Demo project

Make a “hello world” project:

$ mkdir cucumber-hello
$ cd cucumber-hello/

One of the under-celebrated features of Cucumber is the quality of its error messages. Simply attempt to run it and you will be told what to do next:

$ cucumber
No such file or directory - features. You can use `cucumber --init` to get started.

Go ahead and run cucumber --init as the program suggests:

$ cucumber --init
  create   features
  create   features/step_definitions
  create   features/support
  create   features/support/env.rb

This has set up your project. In the same vein as Test-Driven Development (TDD), the usual practice is to write the tests before you code.

Most programmers will be familiar with the expected output of a “Hello World” program. In Cucumber, this ‘feature’ would be represented in Cucumber as:

Feature: Test

  Scenario: Hello
    When I run the "hello-world" program
    Then I should see "Hello World"

Save this definition to features/hello.feature in the project directory and attempt to run Cucumber again:

$ cucumber
Feature: Test

  Scenario: Hello                        # features/hello.feature:3
    When I run the "hello-world" program # features/hello.feature:4
    Then I should see "Hello World"      # features/hello.feature:5

1 scenario (1 undefined)
2 steps (2 undefined)
0m0.007s

You can implement step definitions for undefined steps with these snippets:

When(/^I run the "([^"]*)" program$/) do |arg1|
  pending # Write code here that turns the phrase above into concrete actions
end

Then(/^I should see "([^"]*)"$/) do |arg1|
  pending # Write code here that turns the phrase above into concrete actions
end

The output helpfully contains the skeleton code for writing the test steps in Ruby. Using this as a base, I implemented a test using the rspec expectations syntax, to check if the string starts with “Hello World”.

When(/^I run the "([^"]*)" program$/) do |cmd|
  @value = `./#{cmd}`  
end

Then(/^I should see "([^"]*)"$/) do |val|
  expect(@value).to start_with(val)
end

Save these steps to features/step_definitions/hello_steps.rb

Finally, its time to code. Write up an implementation of “Hello World” in basically any programming language, and save the output to an executable file called hello-world:

Using bash:

#!/bin/sh
echo "Hello World"

$ chmod +x hello-world

Manually testing:

$ ./hello-world
Hello World

Automatically testing with Cucumber confirms that the test criteria are met:

$ cucumber
Feature: Test

  Scenario: Hello                        # features/hello.feature:3
    When I run the "hello-world" program # features/step_definitions/hello_steps.rb:1
    Then I should see "Hello World"      # features/step_definitions/hello_steps.rb:5

1 scenario (1 passed)
2 steps (2 passed)
0m0.016s

Now that you’ve completed these steps, you should have this project structure:

$ find .
.
./features
./features/support
./features/support/env.rb
./features/step_definitions
./features/step_definitions/hello_steps.rb
./features/hello.feature
./hello-world

Next steps

Our scenario for the Hello World program has users interfacing with the system entirely with non-interactive shell commands. Cucumber needs to be able to observe the external behaviour of your application, so your steps will end up including some powerful test libraries. Some that are used often are:

Command-line: Aruba
Web-based: Capybara (back-ends include Selenium WebDriver, tutorial here
Mobile: iOS & Android - Calabash

How to arrange pages for printing and cutting in LaTeX

Fri, 08 Jan 2016 00:00:00 +0000

This post covers some LaTeX examples that I recently assembled for print-outs where multiple pages per sheet will be printed for cutting.

First up, an example file. This is a document filled with random text, on C6 paper.

\documentclass{article}

\usepackage[c6paper]{geometry}
\usepackage{lipsum}

\begin{document}
\section*{Lorem Ipsum}
\lipsum[1-11]
\end{document}

It renders like so:

For the same example output in PDF format, see book-pages.pdf.

1x1 arrangement

Using the crop package, you can wrap the output pages of your document in larger pages. Using the \includepdf macro, these pages can be sourced from our example document.

This example simply renders the document on larger pages, with crop marks displayed at the document edges.

\documentclass{article}
\usepackage{pdfpages}
% Sum of dimensions of the pages. We are displaying 1x1 C6 (114 mm x 162mm) pages, which is where 114x162 comes from.
\usepackage[paperheight=162mm,paperwidth=114mm]{geometry}

% Include print/output paper size and orientation here, A4 portrait in this example (use 'landscape' option for landscape)
\usepackage[axes,cam,a4,pdftex,center]{crop}

\begin{document}

% Restrict page range, and set arrangement of pages per sheet here.
\includepdf[pages=-,nup=1x1,noautoscale]{book-pages.pdf}

\end{document}

This renders a single C6 page on A4, as below:

Alternatively see book-print-1x1.pdf for this output in PDF format.

2x1 arrangement

To lay out two images side-by-side, we need to specify the size of the document we’re wrapping (which will be twice as wide as a single C6 page, and the height of a single C6 page), and also change the orientation of the outer paper to Landscape.

The LaTeX code is:

\documentclass{article}
\usepackage{pdfpages}
% Sum of dimensions of the pages. We are displaying 2x1 C6 (114 mm x 162mm) pages, which is where 228x162 comes from.
\usepackage[paperheight=162mm,paperwidth=228mm]{geometry}

% Include print/output paper size and orientation here, A4 landscape in this example.
\usepackage[axes,cam,a4,landscape,pdftex,center]{crop}

\begin{document}

% Restrict page range, and set arrangement of pages per sheet here.
\includepdf[pages=-,nup=2x1,noautoscale]{book-pages.pdf}

\end{document}

Which renders like this (click here for PDF):

2x2 arrangement

In a similar way, we can slot four portrait C6 pages on to a piece of A4 portrait paper like so:

\documentclass{article}
\usepackage{pdfpages}
% Sum of dimensions of the pages. We are displaying 2x2 C6 (114 mm x 162mm) pages, which is where 228x324 comes from.
\usepackage[paperheight=324mm,paperwidth=228mm]{geometry}

% Include print/output paper size and orientation here, A3 portrait in this example (use 'landscape' option for landscape)
\usepackage[axes,cam,a3,pdftex,center]{crop}

\begin{document}

% Restrict page range, and set arrangement of pages per sheet here.
\includepdf[pages=-,nup=2x2,noautoscale]{book-pages.pdf}

\end{document}

Which renders as (click here for PDF):

More complex arrangement: Name tags

All of the above setups share one thing in common: they have 2 or fewer items on each side, meaning that the half-way axis lines could be used to show where each page ends.

If you are printing many small items, you might fit more than two in a direction, and need to add more axis lines so that you know where to cut!

Here is an example PDF which creates tiny name tags. All names here are randomly generated.

\documentclass{article}
% Custom page size
\usepackage[margin=4mm,paperheight=30mm,paperwidth=60mm]{geometry}

% Macro for a name tag
\newcommand{\tag}[2]{\begin{center}~\\\textbf{\Large#1}\\~\\\em{#2}\end{center}\clearpage}

\begin{document}

% Make some name tags for a conference
% Names generated via www.generatedata.com/
\tag{Flynn Craig}{ExampleCorp}
\tag{Daniel Cash}{ExampleCorp}
\tag{Griffith Durham}{ExampleCorp}
\tag{Sean Washington}{ExampleCorp}
\tag{James Simon}{ExampleCorp}
\tag{Aidan Cotton}{ExampleCorp}
\tag{Wing Sanders}{ExampleCorp}
\tag{Dean Wilcox}{ExampleCorp}
\tag{Jerome Flynn}{FooCorp}
\tag{Leonard Steele}{FooCorp}
\tag{Zane Pratt}{FooCorp}
\tag{Andrew Hartman}{FooCorp}
\tag{Roth Mccullough}{FooCorp}
\tag{Fritz Taylor}{Quux Ltd}
\tag{Beau Bray}{Quux Ltd}
\tag{Duncan Hampton}{Quux Ltd}
\tag{Daquan Witt}{Quux Ltd}
\tag{Wylie Heath}{Quux Ltd}
\tag{Thaddeus Payne}{Quux Ltd}
\tag{Carson Ayala}{Quux Ltd}
\tag{Oleg Olson}{Quux Ltd}
\tag{Herrod Gillespie}{Bar University}
\tag{Jonas Wyatt}{Bar University}
\tag{Mannix Patrick}{Bar University}
\tag{Cain Ferguson}{Bar University}
\tag{Nasim Mendez}{Bar University}
\tag{Emery Bennett}{Bar University}
\tag{Oscar Bennett}{Bar University}
\tag{William Neal}{Bar University}
\tag{Dieter Cochran}{Bar University}
\tag{Bert Schultz}{Bar University}

\end{document}

The above example renders a single name tag per page like this (click here for PDF):

Now, the trick is that crop is hard-coded to show axis lines only at the half-way points, so take a copy of crop.sty called crop-3x7.sty and find this block:

\newcommand*\CROP@marks{%
    \CROP@setmarkcolor
    \CROP@user@b
    \CROP@ulc\null\hfill\CROP@@@info\CROP@upedge\hfill\null\CROP@urc
    \vfill
    \CROP@ledge\hfill\CROP@redge
    \vfill
    \CROP@llc\null\hfill\CROP@loedge\hfill\null\CROP@lrc
}

The block uses \vfill and \hfill to make even spaces.

Edit the file to include more axis lines, like so:

%% 'crop-3x7' is not a real package for general use, it is an example only!
%%
%% It is a modified version of the 'crop' package, for rendering
%% additonal axis marks as cutting aids.
...
% Changed package name here-
\ProvidesPackage{crop-3x7}[2016/01/03 v2.0 crop marks 3x7 (mf)]
...
\newcommand*\CROP@marks{%
    % Changed from one axis on each side to 2 axes on top/lower edge, 6 on each side, for printing 3x7 panels per this example. 
    \CROP@setmarkcolor
    \CROP@user@b
    \CROP@ulc\null\hfill\CROP@@@info\CROP@upedge\hfill\CROP@upedge\hfill\null\CROP@urc
    \vfill
    \CROP@ledge\hfill\CROP@redge
    \vfill
    \CROP@ledge\hfill\CROP@redge
    \vfill
    \CROP@ledge\hfill\CROP@redge
    \vfill
    \CROP@ledge\hfill\CROP@redge
    \vfill
    \CROP@ledge\hfill\CROP@redge
    \vfill
    \CROP@ledge\hfill\CROP@redge
    \vfill
    \CROP@llc\null\hfill\CROP@loedge\hfill\CROP@loedge\hfill\null\CROP@lrc
}

Now simply prepare a page which shows a 3x7 grid of tags, using the same rules as before, and our modified crop-3x7.sty:

\documentclass{article}
% This file demonstrates the use of a customisation to the 'crop' package to render additional axes for cutting up small name labels.

\usepackage{pdfpages}
% Sum of dimensions of the pages. We are displaying 3x7 small tags (60 x 30 mm) to a page, which is where 180x210 comes from.
\usepackage[paperheight=210mm,paperwidth=180mm]{geometry}

% Include print/output paper size and orientation here, A3 portrait in this example (use 'landscape' option for landscape)
\usepackage[axes,cam,a4,pdftex,center]{crop-3x7}

\begin{document}

% Restrict page range, and set arrangement of pages per sheet here.
\includepdf[pages=-,nup=3x7,noautoscale]{labels.pdf}

\end{document}

Render the document, and see the small dashes on the sides which will show you where to cut (again, click here for PDF):

Wrap-up

Full source for the linked files can be found in my mike42/tex-examples repository on GitHub.

How to print red/black on an impact receipt printer

Sun, 27 Dec 2015 00:00:00 +0000

I recently deployed an Epson TM-U220 impact receipt printer. These printers work by striking a ribbon onto the paper, like a type-writer. One of the up-sides to using these intead of a thermal printer is the ability to install a red/black ribbon in place of the default (black) one:

I connected up my printer using a USB-parallel cable, so my previous posts (Windows, Linux) apply for the connector setup.

Using the escpos-php driver on GitHub, a line of red text is printed like this:

<?php
/*
 * Example of two-color printing, tested on an epson TM-U220 with two-color ribbon installed.
 */
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("php://stdout");
$printer = new Printer($connector);

try {
    $printer = new Escpos($connector);
    $printer -> text("Hello World!\n");
    $printer -> setColor(Printer::COLOR_2);
    $printer -> text("Red?!\n");
    $printer -> setColor(Printer::COLOR_1);
    $printer -> text("Default color again?!\n");
    $printer -> cut();
} finally {
    /* Always close the printer! */
    $printer -> close();
}

With this result:

How to connect a USB receipt printer up on Mac OS X

Thu, 05 Nov 2015 00:00:00 +0000

This post will show you how to set up a USB receipt printer on Max OS X. These steps were written on Yosemite, but should work on 10.6 onwards (ie, also Snow Leopard through to El Capitan).

This is another post in a series, which has so far covered direct USB printing on Windows and Linux. The printer tested here is this Epson TM-T20:

CUPS is the printing system that’s used on Mac, but most users would be more familiar with the system print dialog:

Enable CUPS web

In this case, we need to set up the printer via the CUPS web interface. This is accessed via a web browser at this address:

http://localhost:631

At first, you will get knocked back:

To fix this up, open up Applications → Utilities → Terminal and type in:

cupsctl WebInterface=yes

Add the printer

You can then reload the browser and click through to Administration:

Click Add Printer and log in:

Select the USB printer from the list, and optionally share it:

Click Select Another Make/Manufacturer, and select Raw → Raw Queue:

Use the defaults for the other options:

Test print

Type some junk into a file called foo.txt and attempt to print it, using the CUPS printer name:

nano foo.txt
lpr -o raw -H localhost -P EPSON_TM-T20 foo.txt

The prints will be delayed for a few moments, as CUPS spools the jobs.

Disable CUPS web

Once you’re done, for security reasons you should reset this option from before, to disable the web interface to CUPS:

cupsctl WebInterface=no

How to set up sudo on Debian GNU/Linux

Sun, 18 Oct 2015 00:00:00 +0000

If you just installed Debian, you might notice that one Linux staple, sudo is not installed by default. Here is the quick way to add it, and set yourself up as an administrative user.

1. Install sudo, using the su command to elevate privileges.

bob$ su
Password: 
root# apt-get install sudo

2. Make yourself an administrator. The default /etc/sudoers file contains this line:

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

So simply add your account (‘bob’ in this example) to the sudo group:

root# usermod -a -G sudo bob

3. Log out and back in again, in order to apply changes to your group membership.

bob$ groups
bob cdrom floppy sudo audio dip video plugdev netdev lpadmin scanner bluetooth
bob$ sudo echo test
[sudo] password for bob: 
test

Fix merge conflicts in git with Meld

Wed, 16 Sep 2015 00:00:00 +0000

When you’re writing code collaboratively, there’s plenty of situations when you need to combine two sets of changes.

This could happen, for example, if Bob and yourself both fix different bugs by making edits to the same file.

This post assumes that your source code is tracked in git.

First up, install meld. The Meld homepage has instructions for other platforms, but on Debian/Ubuntu, it’s just:

sudo apt-get install meld

Usage with git

Now tell git to use it as a tool:

git config --global merge.tool meld

Once you have a merge conflict, you can then fire up Meld like this:

git mergetool

For each file, you will get a 3-way diff. Click the arrows on the sides to move the code you want into the middle:

Once you’ve saved the file and closed Meld, you will be prompted on the command-line. You just tell it whether you’ve successfully merged the file, until it stops giving you new files to merge.

After this, commit the changes:

git commit

Done!

Standalone usage

If you don’t use git, you can simply call Meld from the command-line as well. This shows you differences between files in a similar window, and lets you move blocks of code around as well:

meld foo.c bar.c

Continuous testing in PHP with Eclipse

Thu, 13 Aug 2015 00:00:00 +0000

If you already write test cases over your code, then continuous testing is a simple idea: Get fast feedback on your code by running your project’s unit tests very frequently. From PHP, the de facto standard testing tool is PHPUnit, so running tests usually involves typing phpunit into a terminal.

This post will show you how to set up a project with the Eclipse MakeGood plugin, which is a continuous test runner for the Eclipse IDE. When it is enabled, the plugin runs a set of tests every time you save a file.

This is best used on projects which are:

Small — so your tests run in a reasonable timeframe.
Complex — reasonably high risk of introducing defects.
Well-tested — so your unit tests are reasonably likely to identify defects.

Set up your machine

Your machine needs Eclipse PDT, MakeGood and composer.

Eclipse PDT

If you’ve found this post via Google, then you probably already run Eclipse and PHP. If not, fetch the “Eclipse for PHP developers” distribution from here

You need to pick up the XDebug extension to really make use of this setup. On Debian-based distributions, this is achieved via:

sudo apt-get install php5-xdebug

If you don’t use Eclipse yet, then check whether languages or tools you’ve used before have plugins in the Eclipse Marketplace. There’s a good chance that Eclipse will do the whole lot.

MakeGood

MakeGood is simply an Eclipse plugin, so you can fetch it from the Eclipse Marketplace.

Navigate to Help → Eclipse Marketplace and search for “makegood”:

Wait for the info to be fetched:

Check all the boxes, and then click Confirm:

Accept the terms, and wait for the install:

At one point, you will be prompted to accept a developer’s certificate, which you need to do before continuing:

This feature will not work unless you have a PHP binary set up. You need to configure this under **Window → Preferences → PHP → PHP executables.**:

This is system dependent: Windows users will find a file somewhere called php-cli.exe, while Linux users might find it at /usr/bin/php.

After this, MakeGood is installed. It is configured per-project, so we’ll get to that after the next section.

Composer

Composer is a dependency manager which is widely used in the PHP world.

You can get it from [https://getcomposer.org/download/](https://getcomposer.org/download/)

Rather than use the install process which is suggested there, you can simply download the phar, make it executable, and place it somewhere like /usr/bin:

wget https://getcomposer.org/composer.phar
chmod +x composer.phar
sudo mv composer.phar /usr/bin/composer
composer --version

By using composer to load PHPUnit, MakeGood becomes much easier to configure.

Project setup

To start, make a new PHP project by clicking File → New → PHP Project. Name it something like “Example”.

We are using composer here because it provides an easy way to make sure MakeGood can see the PHPUnit code. So let’s set a few things up:

First, add a composer.json file which adds a recent version of PHPUnit as a require-dev dependency. Right-click Example project, and navigate to New → File, naming it composer.json:

{
    "require-dev": {
        "phpunit/phpunit": "4.5.*"
    }
}

Note that with the way that MakeGood constructs its command-line, newer versions of PHPUnit do not appear to work correctly at the time of writing.

Next up, install the new dependency with the help of Composer:

$ composer install
Loading composer repositories with package information
Installing dependencies (including require-dev)
  - Installing sebastian/version (1.0.6)
  - Installing sebastian/global-state (1.0.0)
  - Installing sebastian/recursion-context (1.0.1)
  - Installing sebastian/exporter (1.2.1)
  - Installing sebastian/environment (1.3.2)
  - Installing sebastian/diff (1.3.0)
  - Installing sebastian/comparator (1.2.0)
  - Installing symfony/yaml (v2.7.3)
  - Installing doctrine/instantiator (1.0.5)
  - Installing phpdocumentor/reflection-docblock (2.0.4)
  - Installing phpspec/prophecy (v1.4.1)
  - Installing phpunit/php-text-template (1.2.1)
  - Installing phpunit/phpunit-mock-objects (2.3.6)
  - Installing phpunit/php-timer (1.0.7)
  - Installing phpunit/php-token-stream (1.4.3)
  - Installing phpunit/php-file-iterator (1.3.4)
  - Installing phpunit/php-code-coverage (2.2.2)
  - Installing phpunit/phpunit (4.5.1)

sebastian/global-state suggests installing ext-uopz (*)
phpdocumentor/reflection-docblock suggests installing dflydev/markdown (~1.0)
phpdocumentor/reflection-docblock suggests installing erusev/parsedown (~1.0)
phpunit/phpunit suggests installing phpunit/php-invoker (~1.1)
Writing lock file
Generating autoload files

Set up an Example.php class with a bit of code:

<?php
/**
 * Example of some code that could do with test cases
 */
class Example {
	/**
	 * Faulty palindrome checker. A palindrome is a word which is the same when read orward and backward.
	 * 
	 * @param string $str
	 * @return boolean True if $str is a palindrome, false if it is not.
	 */
	function isPalindrone($str) {
		$halfLen = (int)(strlen($str) / 2);
		for($i = 0; $i < $halfLen; $i++) {
			// Compare characters from the front and back of the string, and move toward the middle.
			$c1 = substr($str, $i, 1);
			$c2 = substr($str, (2 * $halfLen - 1) - $i, 1);
			if($c1 != $c2) {
				return false;
			}
		}		
		return true;
	}

}

Set up a test/unit folder and create a ExampleTest class with a useful test case in it, like so:

<?php
class ExampleTest extends PHPUnit_Framework_TestCase {
	public function testIsPalindrome() {
		$e = new Example();
		$this -> assertTrue($e -> isPalindrone(""));
		$this -> assertTrue($e -> isPalindrone("a"));
		$this -> assertTrue($e -> isPalindrone("ABBA"));
		$this -> assertFalse($e -> isPalindrone("test"));
	}
}

The unit tests go in their own folder, in case you decide to add more extensive (slower) tests later. MakeGood should only be configured to run fast tests!

Now add a test/bootstrap.php for test cases to load classes (NB: At scale, you would have composer autoload your classes, and use that as your PHPUnit class loader).

<?php
require_once 'vendor/autoload.php';
require_once 'Example.php';

Optionally, also add a phpunit.xml file in your project root to configure PHPUnit with, so that you can run it on the command-line:

<phpunit
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/4.5/phpunit.xsd"
         bootstrap="test/bootstrap.php">
</phpunit>

If all goes to plan, your project should look something like this:

- Continuous testing example
  |- (.. code ..)
  |- Example.php
  |- test/
  |  |- (.. other types of tests ..) 
  |  |- unit/
  |  | |- ExampleTest.php
  |  |- bootstrap.php
  |- vendor
  |  |- (.. dependencies ..)
  |  |- autoload.php
  |- composer.json
  |- phpunit.xml

And more to the point, you should be able to use PHPUnit on the command-line to run the test case. It will look something like this:

Example$ php vendor/bin/phpunit test/unit/
PHPUnit 4.7.7 by Sebastian Bergmann and contributors.

.

Time: 71 ms, Memory: 4.50Mb

OK (1 test, 4 assertions)

Configure MakeGood

Now that the project is coming along nicely, it’s time to configure MakeGood to run the test automatically from within Eclipse.

Go to Project → Properties → MakeGood, and add the path to your unit tests, along with a link to the bootstrap.php. This bootstrap file will let MakeGood load the classes to test, along with PHPUnit itself.

Now fire up the MakeGood view. If it doesn’t come up automatically, then open it via Window → Show View → Other. Clicking the “play” button should kick off all available tests, adn if your project structure is correct, then everything will work at this point:

Once you save a file, a small box will appear in the bottom-right to indicate whether the tests worked.

As you introduce changes and save them, MakeGood will flag whether the tests are passing by changing the colour of the icon.

In this case, it is green. From the small delay that is caused when running the tests, you can see why it’s important to keep unit tests fast if you use this tool.

A test-driven bugfix cycle

Of course, Linus’ Law suggests that any code will probably have more bugs. So, say somebody reports that the palindrome “abcdedcba” is causing incorrect results. In test-driven development, you would expose this as a test case over the isPalindrome() function:

<?php
//...
	public function testIsPalindromeOddLength() {
		$e = new Example();
		$this -> assertTrue($e -> isPalindrone("abcdedcba"));
		$this -> assertFalse($e -> isPalindrone("abc"));
	}
//...

As soon as this is submitted, things start to turn red in MakeGood. In response to the broken test, you can then tweak the code.

You would then do the smallest amount of programming needed to pass all the tests. A palindrome checker which does not contain intentional bugs is as simple as this one-liner:

<?php
//...
	function isPalindrone($str) {
		return $str == strrev($str);
	}
//...

This change causes MakeGood to report that the unit tests are all passing.

Summary

A continuous test runner adds faster feedback for tedious but well-tested code. If you have to write anything in PHP which does some real computer work, then I suggest setting up MakeGood so that you can make changes to tricky code, and have a good idea of whether it’s producing sane outputs.

As this post has touched on before, slow tests should not run each time you save the code. This is why we worked in a Unit test folder. Slower integration or functional tests can still be run via the command-line in this model. An example of a project which uses this setup is mike42/escpos-php, which is a heavily test-driven PHP driver for thermal line printers.

It's time to migrate away from Outlook Express

Thu, 18 Jun 2015 00:00:00 +0000

Outlook Express is obsolete, so you need to migrate if you’re still using it. This post is a quick guide to saving your local data so that you can jump ship.

If you are keen on desktop-based email, then there are only two real contenders for a replacement mail client:

Mozilla Thunderbird (suggested).
Windows Mail.

So where is all my data?

The script below is a Windows bat script for backing up an Outlook Express setup. Just fill in the Identity variable and run it from anywhere to produce a folder containing the Outlook saved emails and contacts.

You can find your identity string as a folder name in your profile path, under Local Settings\Application Data\Identities\. The profile path for a user called bob would usually be in a folder like C:\Documents and Settings\bob or C:\Users\bob.

These files can be read by both of the mentioned replacements, so a good transition might be:

Make a backup.
Install an alternative & set it up.
Try to import as much as you can.
Once you’re happy, uninstall Outlook Express, or at least delete the shortcuts to it.

@echo off
ECHO -------------------------------------------------------------------------------
SET BACKUPDIR="%username% - Outlook Backup"
SET IDENTITY={AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA}
ECHO Backing up outlook files to: %backupdir%
ECHO --------------------------------------------------------------------------------

ECHO - Clearing backup location... (1 of 3) 
DEL /F /S /Q %backupdir% > NUL 2>&1
MKDIR %backupdir% 2> NUL
ECHO - - Done

ECHO - Copying emails... (2 of 3) 
MKDIR %backupdir%\Emails 2> NUL
XCOPY /E /H /C /R /Y "%userprofile%\Local Settings\Application Data\Identities\%IDENTITY%\Microsoft\Outlook Express" %backupdir%\Emails > NUL
ECHO - - Done

ECHO - Copying address book... (3 of 3)
MKDIR %backupdir%\AddressBook 2> NUL
XCOPY /E /H /C /R /Y "%appdata%\Microsoft\Address Book" %backupdir%\AddressBook > NUL
ECHO - - Done

ECHO - Completed 3 of 3 tasks.
ECHO -------------------------------------------------------------------------------
ECHO The backup is complete.
ECHO Please copy %BACKUPDIR% to external storage.
ECHO --------------------------------------------------------------------------------
PAUSE

How to create an animated GIF from a series of images

Thu, 11 Jun 2015 00:00:00 +0000

Sometimes, you end up with a folder full of images, which you want to animate. With the open source ImageMagick tool, this is easy on the command line:

animate *.png

This will show you all of the PNG files in the folder in quick succession, like a flip book.

ImageMagick works on just about any OS. For Linux users, it should be in your package manager.

On Debian you would install it via:

sudo apt-get install imagemagick

Or on Red Hat it is:

sudo yum install ImageMagick

But this blog post is about animated GIFs, so lets make one of those. This is a compact way to combine images (here and here for examples in context), gives you a re-usable at-a-glance illustration of something that changes over time.

Example from an older post:

The steps to make a good conversion command are below.

Check that alphabetically, your images are in order. If not, rename them:

echo *

Convert them to a GIF, and find the delay that suits you (hundredths of a second between frames). You will want to repeat this a few times to find the delay value which works for you:

convert -delay 80 *.png animated.gif

Choose an output size (width x height):

convert -resize 415x -delay 80 *.png animated.gif

Compress with -Layers Optimize for a smaller file:

convert -resize 415x -delay 80 *.png -layers Optimize animated.gif

Notes

Generated thumbnails usually take the first frame only, which is why we ask Imagemagick to resize it (Wordpress users: Choose “Full Size”).
To pause at the start of the loop for a moment, just copy the first image a few times.

Raspberry Pi KA Lite wireless deployment

Thu, 11 Jun 2015 00:00:00 +0000

Previous post in this series: How to install KA Lite on the Raspberry Pi

If you got to the end of the previous post, you would have a working install of the open source KA Lite system on the Raspberry Pi.

This post will show you how to deploy this KA Lite setup without relying on any existing network infrastructure:

To share the setup without an existing network, we will configure the wireless adapter to provide its own network for computers to connect to. If you have an existing wireless network, you can skip this post.

Before you begin

This post is intended for the widely-used “WiPi” USB WiFi adapter, so you’ll need one of those.

I used the same Tenda USB WiFi adapter as previous setups, which has a Ralink chip from the same series:

Set it up

See the official guide for these steps in compact form. These steps are the same, but expanded with extra output from my install.

Clone the raspberry pi scripts repo:

$ git clone https://github.com/learningequality/ka-lite-pi-scripts.git
Cloning into 'ka-lite-pi-scripts'...
remote: Counting objects: 66, done.
remote: Total 66 (delta 0), reused 0 (delta 0), pack-reused 66
Unpacking objects: 100% (66/66), done.

Open it up and run ./configure.sh as the root user. This requires Internet access, and will install and configure a few packages:

$ cd ka-lite-pi-scripts/
$ ls
README.md     configure_network_interfaces.py  use_wipi.sh
conf          redirect_port80_to_port8008.py
configure.sh  use_edimax.sh
$ sudo ./configure.sh
Get:1 http://mirrordirector.raspbian.org wheezy Release.gpg [490 B] 
Hit http://raspberrypi.collabora.com wheezy Release.gpg                        
Hit http://archive.raspberrypi.org wheezy Release.gpg
Get:2 http://mirrordirector.raspbian.org wheezy Release [14.4 kB]
Hit http://raspberrypi.collabora.com wheezy Release                      
Hit http://archive.raspberrypi.org wheezy Release                        
Hit http://raspberrypi.collabora.com wheezy/rpi armhf Packages           
Get:3 http://archive.raspberrypi.org wheezy/main armhf Packages [129 kB]       
Get:4 http://mirrordirector.raspbian.org wheezy/main armhf Packages [6903 kB]  
Ign http://raspberrypi.collabora.com wheezy/rpi Translation-en                 
Ign http://archive.raspberrypi.org wheezy/main Translation-en                
Get:5 http://mirrordirector.raspbian.org wheezy/contrib armhf Packages [23.6 kB]
Get:6 http://mirrordirector.raspbian.org wheezy/non-free armhf Packages [49.3 kB]
Get:7 http://mirrordirector.raspbian.org wheezy/rpi armhf Packages [592 B]     
Ign http://mirrordirector.raspbian.org wheezy/contrib Translation-en           
Ign http://mirrordirector.raspbian.org wheezy/main Translation-en              
Ign http://mirrordirector.raspbian.org wheezy/non-free Translation-en          
Ign http://mirrordirector.raspbian.org wheezy/rpi Translation-en
Fetched 6992 kB in 35s (198 kB/s)             
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be upgraded:
  cups-bsd cups-client cups-common fuse libcups2 libcupsimage2 libfuse2 raspi-config
8 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1708 kB of archives.
After this operation, 306 kB disk space will be freed.
Get:1 http://archive.raspberrypi.org/debian/ wheezy/main raspi-config all 20150131-4 [13.2 kB]
Get:2 http://mirrordirector.raspbian.org/raspbian/ wheezy/main libcupsimage2 armhf 1.5.3-5+deb7u6 [132 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ wheezy/main cups-common all 1.5.3-5+deb7u6 [904 kB]
Get:4 http://mirrordirector.raspbian.org/raspbian/ wheezy/main cups-bsd armhf 1.5.3-5+deb7u6 [44.2 kB]
Get:5 http://mirrordirector.raspbian.org/raspbian/ wheezy/main cups-client armhf 1.5.3-5+deb7u6 [174 kB]
Get:6 http://mirrordirector.raspbian.org/raspbian/ wheezy/main libcups2 armhf 1.5.3-5+deb7u6 [238 kB]
Get:7 http://mirrordirector.raspbian.org/raspbian/ wheezy/main fuse armhf 2.9.0-2+deb7u2 [70.8 kB]
Get:8 http://mirrordirector.raspbian.org/raspbian/ wheezy/main libfuse2 armhf 2.9.0-2+deb7u2 [132 kB]
Fetched 1708 kB in 3s (447 kB/s)
Preconfiguring packages ...
(Reading database ... 83674 files and directories currently installed.)
Preparing to replace libcupsimage2:armhf 1.5.3-5+deb7u5 (using .../libcupsimage2_1.5.3-5+deb7u6_armhf.deb) ...
Unpacking replacement libcupsimage2:armhf ...
Preparing to replace cups-common 1.5.3-5+deb7u5 (using .../cups-common_1.5.3-5+deb7u6_all.deb) ...
Unpacking replacement cups-common ...
Preparing to replace cups-bsd 1.5.3-5+deb7u5 (using .../cups-bsd_1.5.3-5+deb7u6_armhf.deb) ...
locale: Cannot set LC_ALL to default locale: No such file or directory
Unpacking replacement cups-bsd ...
Preparing to replace cups-client 1.5.3-5+deb7u5 (using .../cups-client_1.5.3-5+deb7u6_armhf.deb) ...
Unpacking replacement cups-client ...
Preparing to replace libcups2:armhf 1.5.3-5+deb7u5 (using .../libcups2_1.5.3-5+deb7u6_armhf.deb) ...
Unpacking replacement libcups2:armhf ...
Preparing to replace fuse 2.9.0-2+deb7u1 (using .../fuse_2.9.0-2+deb7u2_armhf.deb) ...
Unpacking replacement fuse ...
Preparing to replace libfuse2:armhf 2.9.0-2+deb7u1 (using .../libfuse2_2.9.0-2+deb7u2_armhf.deb) ...
Unpacking replacement libfuse2:armhf ...
Preparing to replace raspi-config 20150131-3 (using .../raspi-config_20150131-4_all.deb) ...
Unpacking replacement raspi-config ...
Processing triggers for man-db ...
Processing triggers for initramfs-tools ...
Setting up libcups2:armhf (1.5.3-5+deb7u6) ...
Setting up libcupsimage2:armhf (1.5.3-5+deb7u6) ...
Setting up cups-common (1.5.3-5+deb7u6) ...
Setting up cups-client (1.5.3-5+deb7u6) ...
Setting up cups-bsd (1.5.3-5+deb7u6) ...
Setting up libfuse2:armhf (2.9.0-2+deb7u2) ...
Setting up fuse (2.9.0-2+deb7u2) ...
udev active, skipping device node creation.
update-initramfs: deferring update (trigger activated)
Setting up raspi-config (20150131-4) ...
update-rc.d: warning: default start runlevel arguments (2 3 4 5) do not match raspi-config Default-Start values (S)
update-rc.d: warning: default stop runlevel arguments (0 1 6) do not match raspi-config Default-Stop values (none)
Processing triggers for initramfs-tools ...
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  dnsmasq-base libnetfilter-conntrack3
The following NEW packages will be installed:
  dnsmasq dnsmasq-base hostapd libnetfilter-conntrack3
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 822 kB of archives.
After this operation, 1653 kB of additional disk space will be used.
Get:1 http://mirrordirector.raspbian.org/raspbian/ wheezy/main libnetfilter-conntrack3 armhf 1.0.1-1 [32.2 kB]
Get:2 http://mirrordirector.raspbian.org/raspbian/ wheezy/main dnsmasq-base armhf 2.62-3+deb7u3 [356 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ wheezy/main dnsmasq all 2.62-3+deb7u3 [16.3 kB]
Get:4 http://mirrordirector.raspbian.org/raspbian/ wheezy/main hostapd armhf 1:1.0-3+deb7u2 [418 kB]
Fetched 822 kB in 4s (168 kB/s)  
Selecting previously unselected package libnetfilter-conntrack3:armhf.
(Reading database ... 83674 files and directories currently installed.)
Unpacking libnetfilter-conntrack3:armhf (from .../libnetfilter-conntrack3_1.0.1-1_armhf.deb) ...
Selecting previously unselected package dnsmasq-base.
Unpacking dnsmasq-base (from .../dnsmasq-base_2.62-3+deb7u3_armhf.deb) ...
Selecting previously unselected package dnsmasq.
Unpacking dnsmasq (from .../dnsmasq_2.62-3+deb7u3_all.deb) ...
Selecting previously unselected package hostapd.
Unpacking hostapd (from .../hostapd_1%3a1.0-3+deb7u2_armhf.deb) ...
Processing triggers for man-db ...
Setting up libnetfilter-conntrack3:armhf (1.0.1-1) ...
Setting up dnsmasq-base (2.62-3+deb7u3) ...
Setting up dnsmasq (2.62-3+deb7u3) ...
[ ok ] Starting DNS forwarder and DHCP server: dnsmasq.
Setting up hostapd (1:1.0-3+deb7u2) ...
[ ok ] Starting advanced IEEE 802.11 management: hostapd.
[ ok ] Starting DNS forwarder and DHCP server: dnsmasq[....] (already running).
update-rc.d: using dependency based boot sequencing
update-rc.d: using dependency based boot sequencing

Next, run run the ./use_wipi.sh script:

$ ./use_wipi.sh 
[ ok ] Stopping advanced IEEE 802.11 management: hostapd.
[ ok ] Starting advanced IEEE 802.11 management: hostapd.

And then the ./configure_network_interfaces.py script as root:

sudo python ./configure_network_interfaces.py

After this ran, the /etc/network/interfaces file looked like:

$ sudo nano /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
#allow-hotplug eth0
iface eth0 inet manual

auto wlan0
#allow-hotplug wlan0
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

auto wlan1
#allow-hotplug wlan1
iface wlan1 inet manual
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

iface wlan0 inet static
  address 1.1.1.1
  netmask 255.255.255.0
  pre-up sudo iptables -t nat -A PREROUTING -i wlan0 -p tcp -j DNAT --to-destination 1.1.1.1
  pre-up sudo python /home/pi/ka-lite/ka-lite-pi-scripts/redirect_port80_to_port8008.py &

And I modified /etc/default/ifplugd to only include eth0, so that the pi does not attempt to join any wireless networks itself:

$ sudo nano /etc/default/ifplugd
INTERFACES="eth0"
HOTPLUG_INTERFACES="eth0"
ARGS="-q -f -u0 -d10 -w -I"
SUSPEND_ACTION="stop"

And then reboot:

$ sudo reboot

Broadcast message from root@raspberrypi (pts/0) (Thu Jun 11 11:05:46 2015):

The system is going down for reboot NOW!

See if it’s all working

Disregard the old addresses once you’ve done the above steps, and run ifconfig to find out about addresses after rebooting.

The pi broadcasts a passwordless wireless network called “kalite”, and all DNS lookups point to 1.1.1.1.

This means in a web browser, you can just load a URL like:

http://kalite

Platforms

On a laptop in range, the network appeared once the pi booted:

The network connected without a password:

And the connection info for the client computer is being checked in the background:

Once the computer gets a lease from DHCP, the URL loads, and the interface works as expected:

Tested on Android, the player and navigation worked in a similar way:

KA Lite’s media player doesn’t seem to require anything other than a web browser. Either Chrome or Firefox seem to be suitable choices.

Debug zone

Some useful commands for debugging are shown below.

List USB devices, for debugging the wireless adapter:

lsusb

List recent errors, for debugging the same thing:

dmesg

Restart networking, run when changing configuration:

service networking restart

Note on dnsmasq

Because of the way dnsmasq hijacks lookups to provide the wireless network, the Raspberry pi cannot access the Internet over Ethernet (eg, to download content) while dnsmasq is running:

sudo service dnsmasq stop

You can access the web interface at http://1.1.1.1/ while this service is disabled.

And then start it again when done using the Internet, so that the user-friendly computer name works again:

sudo service dnsmasq start

How to install KA Lite on the Raspberry Pi

Wed, 03 Jun 2015 00:00:00 +0000

KA Lite is an open source, web-based learning package. Today I’ll run through a simple setup which will allow a Raspberry Pi to provide a wireless learning resource server for a classroom, without the Internet.

The Raspberry Pi 2 is a surprisingly powerful single-board computer. You can use it for anything which a computer can do, in a relatively cheap circuit board the size of a credit card:

This setup takes around two hours.

Materials

You will need:

Raspberry Pi (The Raspberry Pi 2 Model B was used here), with micro SD card, mouse, keyboard, monitor with HDMI.
A good Micro USB power supply - Many Android phone chargers will be suitable.
A computer which can write to micro SD cards. If your computer has an SD card slot, get an SD-to-Micro-SD adapter for it.
A wired network with Internet for the setup
A USB wireless network adapter such as WiPi for deployment.
Any WiFi-enabled device for testing.

Install Raspbian

Raspbian is an operating system for the Pi, and is the perfect choice for embedded server setups like this. You need to write the image to the Micro SD card.

Obtain the image from https://www.raspberrypi.org/downloads/
This is zipped, so extract it to get the real image file.
Write the disk image to the SD
- Windows: Fetch a copy of “Win32 Disk Imager” to write this file to the SD card.
- Linux or Mac: Use dd (found in the terminal) to write to the card — guide available here
Boot the Pi, with mouse, keyboard, HDMI, and wired network. If it doesn’t work, remove any cable converters (use HDMI direct to monitor), and if it still doesn’t work, then the SD card was not correctly imaged.
Expand root filesystem, set your password, and enable “boot to desktop”.
After reboot, you should have a desktop. Click the black terminal icon to get to work.

Install some things

From the terminal, run these commands:

sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install chromium python-m2crypto

This checks for new software (update), upgrades anything necessary (dist-upgrade), and then installs two packages:

python-m2crypto — this will help speed up KA Lite a bit by helping out with encryption.
chromium — this is the open source upstream of Google Chrome, and it’s worth installing so that you can look things up as you go, and test things locally.

If everything goes to plan, you should now be able to find Chrome on the menu and navigate the web.

Install KA Lite

These steps are a shortened version of the official guide

Once Raspbian is running, these commands will download & install KA Lite:

git clone https://github.com/learningequality/ka-lite.git
cd ka-lite/
./setup_unix.sh

The download is 150MB, so allow a few minutes. The installer then prompts for a few questions, like:

“Do you wish to download the assessment package now?” — Say yes, it fetches another 500MB.
“Do you wish to set the KA Lite server to run in the background automatically when you start this computer?” — say “y”.

At the end, you will be given a command to start the server, which you should type in as well:

/home/pi/ka-lite/bin/kalite start

Once it starts (it will take a while!), it will give you two web addresses. One of them is local to the computer (127.x), and the other will work from other computers on the network. Write both of them down!

The install & start-up together will look like this:

Test it out locally

Open up Chromium, and navigate to one of the addresses that the installer suggested.

Click on “Manage”, then “Videos”

KA Lite will prompt for registration, which lets you download content. Use the “One click registration” on the left, then wait for it to connect.

Use the Videos section to fetch something. Here we fetch something from the ancient art & culture section:

Wait a few minutes before moving to the next section: The video downloads in the background.

Access over the network

From some other device which is on the same network, enter the web address that was given at the end of the installer (ignore the address which starts with 127.x — it won’t work between two computers):

http://(IP address):8008/

From the browser, let’s test it out. Click over to Learn, and navigate to the video you just downloaded. It should display, so we can learn about those Egyptian artifacts:

This setup would be complete if you want to run on a wired network - you can simply bookmark this address on each computer which needs to use it, and manage it through the web.

Next post in this series: Raspberry Pi KA Lite wireless deployment

Howto: QR Codes on receipts with escpos-php

Sun, 26 Apr 2015 00:00:00 +0000

ESC/POS is a binary protocol for speaking to receipt printers. It contains a command for printing QR Codes on compatible printers. The PHP library escpos-php is used for generating these commands in PHP. This post will show you how to use it to generate QR codes on your receipt printer.

For printers which don’t support this command, a second option is available: sending the QR code as an image.

Getting started

First up, you need your receipt printer to be working with escpos-php. Here are some resources about how to go about that:

mike42/escpos-php on GitHub (the most up-to-date reference)
What is ESC/POS, and how do I use it?
Setting up an Epson receipt printer
Getting a USB receipt printer working on Linux (or Windows)

Option 1: Direct printing

This method sends QR codes directly. From the documentation, the syntax for this command is:

qrCode($content, $ec, $size, $model)

Print the given data as a QR code on the printer.

string $content: The content of the code. Numeric data will be more efficiently compacted.

int $ec Error-correction level to use. One of Printer::QR_ECLEVEL_L (default), Printer::QR_ECLEVEL_M, Printer::QR_ECLEVEL_Q or Printer::QR_ECLEVEL_H. Higher error correction results in a less compact code.

int $size: Pixel size to use. Must be 1-16 (default 3)

int $model: QR code model to use. Must be one of Printer::QR_MODEL_1, Printer::QR_MODEL_2 (default) or Printer::QR_MICRO (not supported by all printers).

The below code snippets are directly from the QR code printing demo, showing how the output changes with the options given.

Simple

This is the simplest use, with all default options. QR codes can be aligned in the same way as text or images on the page:

Example

PHP

// Most simple example
title($printer, "QR code demo\n");
$testStr = "Testing 123";
$printer -> qrCode($testStr);
$printer -> text("Most simple example\n");
$printer -> feed();

// Demo that alignment is the same as text
$printer -> setJustification(Printer::JUSTIFY_CENTER);
$printer -> qrCode($testStr);
$printer -> text("Same example, centred\n");
$printer -> setJustification();
$printer -> feed();

Data encoding

This is a demonstration of saving different types of data in a code. Numeric data is packed more efficiently than text. Binary data can also be stored.

Example

PHP

// Demo of numeric data being packed more densly
title($printer, "Data encoding\n");
$test = array(
	"Numeric"      => "0123456789012345678901234567890123456789",
	"Alphanumeric" => "abcdefghijklmnopqrstuvwxyzabcdefghijklmn",
	"Binary"       => str_repeat("\0", 40));
foreach($test as $type => $data) {
	$printer -> qrCode($data);
	$printer -> text("$type\n");
	$printer -> feed();
}

Error correction levels

QR codes support four levels of error correction. More error correction results in larger, but more durable codes:

Example

PHP

// Demo of error correction
title($printer, "Error correction\n");
$ec = array(
	Printer::QR_ECLEVEL_L => "L",
	Printer::QR_ECLEVEL_M => "M",
	Printer::QR_ECLEVEL_Q => "Q",
	Printer::QR_ECLEVEL_H => "H");
foreach($ec as $level => $name) {
	$printer -> qrCode($testStr, $level);
	$printer -> text("Error correction $name\n");
	$printer -> feed();
}

Code size

The defauly codes are quite small. Each pixel can be blown up, up to 16x, using the size option:

Example

PHP

// Change size
title($printer, "Pixel size\n");
$sizes = array(
	1 => "(minimum)",
	2 => "",
	3 => "(default)",
	4 => "",
	5 => "",
	10 => "",
	16 => "(maximum)");
foreach($sizes as $size => $label) {
	$printer -> qrCode($testStr, Printer::QR_ECLEVEL_L, $size);
	$printer -> text("Pixel size $size $label\n");
	$printer -> feed();
}

QR models

QR models have different appearances, storage parameters and physical sizes. The default (Model 2) is most common. The printer used here does not support micro QR codes, and used Model 2 as a fallback.

Example

PHP

// Change model
title($printer, "QR model\n");
$models = array(
	Printer::QR_MODEL_1 => "QR Model 1",
	Printer::QR_MODEL_2 => "QR Model 2 (default)",
	Printer::QR_MICRO => "Micro QR code\n(not supported on all printers)");
foreach($models as $model => $name) {
	$printer -> qrCode($testStr, Printer::QR_ECLEVEL_L, 3, $model);
	$printer -> text("$name\n");
	$printer -> feed();
}

Note

To run the snippets, you need to initialise the printer, and define a title() function to print headings, like so:

<?php
/* Demonstration of available options on the qrCode() command */
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

// ....

// Cut & close
$printer -> cut();
$printer -> close();

function title(Escpos $printer, $str) {
	$printer -> selectPrintMode(Printer::MODE_DOUBLE_HEIGHT | Printer::MODE_DOUBLE_WIDTH);
	$printer -> text($str);
	$printer -> selectPrintMode();
}

Option 2: Printing codes as images

Not all printers can generate QR codes natively. The work-around is to generate a QR code as an image on the computer, and then send that image to the printer. This is slightly slower, so if you print a lot of codes, you should consider upgrading your printer.

First up, fetch a copy of phpqrcode and generate some codes. I wont attempt to document the whole library here, but in short, it supports most of the same features as the native QR command. To generate a code, you simply use QRcode::png:

require_once("phpqrcode/qrlib.php");
QRcode::png("testing123", "test.png", 'L', 10, 0);

To print a PNG image, use the bitImage() command (the graphics command is also only available on newer printers):

require __DIR__ . '/autoload.php';
use Mike42\Escpos\EscposImage;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$img = EscposImage::load("test.png"); // Load image
$connector = new FilePrintConnector("/dev/usb/lp0"); // Add connector to your printer here
$printer = new Printer($connector);
$printer -> bitImage($img);
$printer -> feed();
$printer -> text("Code printed from image\n");

$printer -> cut();
$printer -> close();

A more sophisticated way to hack in phpqrcode would be to add this new code as a different implementaton of the qrCode function, which I’ve done before. Other improvements below are:

Use temporary files to avoid some potential issues with concurrency and writing files in the working directory
Where possible, expand the code on the printer, to send less data

<?php
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;

require_once("phpqrcode/qrlib.php");

class EscposQrImgPrinter extends Printer {
	function qrCode($content, $ec = self::QR_ECLEVEL_L, $size = 3, $model = self::QR_MODEL_2) {
		// Validate inputs
		self::validateString($content, __FUNCTION__);
		self::validateInteger($ec, 0, 3, __FUNCTION__);
		self::validateInteger($size, 1, 16, __FUNCTION__);
		$model = self::QR_MODEL_2; // Only Model 2 supported in phpqrcode, change back to it.
		$sizeMod = 0;
		if($size % 2 == 0) { // Optimisation to enlarge codes on the priner, sending 1/4 of the data.
			$size /= 2;
			$sizeMod = self::IMG_DOUBLE_HEIGHT | self::IMG_DOUBLE_WIDTH;
		}
		// Map error-correction to phpqrcode levels
		$ecMap = array(QR_ECLEVEL_L => 'L',
			QR_ECLEVEL_M => 'M',
			QR_ECLEVEL_Q => 'Q',
			QR_ECLEVEL_H => 'H');
		// Create QR code in temp file, and print it.
		$tmpfname = tempnam(sys_get_temp_dir(), "escpos-php");
		QRcode::png("testing123", $tmpfname, $ecMap[$ec], $size, 0);
		$img = new EscposImage($tmpfname);
		$this -> bitImage($img, $sizeMod);
		unlink($tmpfname);
	}
}

This new class uses phpqrcode in the background, but can be accessed with the same function calls as the parent class:

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new EscposQrImgPrinter($connector);

$testStr = "Testing 123";
$printer -> qrCode($testStr);
$printer -> text("Most simple example\n");
$printer -> feed();
$printer -> cut();

$printer -> close();

The only visible difference between the two implementations is a few pixels of spacing below the image.

Deprecated Google API's: What you need to know

Mon, 20 Apr 2015 00:00:00 +0000

In less than a day, a whole series of old Google APIs are being switched off.

If you are running any of those API’s, then whilst this is bad news, you’ve had several years to upgrade (for example, the Provisioning API features have now been replaced by the Admin SDK). However, one of the API’s not mentioned on the linked page is the old ClientLogin API, which will also stop working on April 20.

ClientLogin involves directly sending a username and password to Google, and then receiving a login token back. There are two main categories of people who use the ClientLogin API which this post is targeted at:

Developers who haven’t got a handle on OAuth yet.
Developers using client libraries for the Google Data APIs. These APIs themselves are not deprecated, but may be using a deprecated login process.

If your use case involves users actually entering their username and password, then this post won’t help (but this page will), but if you have an installed script which runs administrative functions on your domain, then read on, as it’s quite easy to retrofit your scripts with an OAuth2 service account.

Setting up a service account

Service accounts are designed for server-side use (ie, no browser), and replace using ClientLogin to authenticate as a super-admin for the domain.

First, log in to the developer console (console.developers.google.com), and create a new project, then jump to the credentials menu. Create and download a p12 private key.

Now under “Manage this Domain”, locate the advanced security options, and add a delegation for your new service account for one or more scopes. This allows the service account to access to a given API and act on behalf of users. As an example, the next section uses the Email Settings API, which has this scope:

https://apps-apis.google.com/a/feeds/emailsettings/2.0/

And it’s added like so:

Example: How to retro-fit a PHP app with OAuth 2

Assuming you already have code which sends requests to an API, all we need to change is the Authorize: header. A ClientLogin request looks like this:

Authorize: GoogleLogin auth=<ClientLogin token>

But an OAuth2 request looks like this:

Authorize: Bearer <OAuth token>

You can generate a new token for yourself at the OAuth 2.0 Playground from Google to make sure your old API’s work with the new tokens (they will: any issues are probably permission or user problems).

So how do you generate one of these tokens for a service account? The best way is to use client libraries for this. In PHP, this is the google-api-php-client.

This library also has support for newer API’s (not older Google Data API’s), which you can use to replace your implementation where applicable. For this example, we’ll just be getting an access token.

First up, we need to do some structuring. I’ve wrapped old signature code into an abstract EmailSignatureUpdater class so that I can write two versions: The old ClientLoginEmailSignatureUpdater, and the new OAuthEmailSignatureUpdater, differing only by login method and Authorize: header.

So here is some ClientLogin code:

/**
 * Functions dependent on ClientLogin, which will stop working after 2015-04-20.
 */
class ClientLoginEmailSignatureUpdater extends EmailSignatureUpdater {
    /**
     * @param array $conf Configuration for the Google login.
     */
    protected function __construct(array $conf) {
        /* Getting account */
        $account = array(
                'accountType' => 'GOOGLE',
                'Email' => $conf['user']."@".$conf['domain'],
                'Passwd' => $conf['password'],
                'service' => 'apps');

        /* Log in */
        $login = $this -> login($account);
        if(!isset($login['Auth'])) {
            if(isset($login['Error'])) {
                throw new Exception("Google returned ".$login['Error']);
            } else {
                throw new Exception("Google login failed");
            }
        }
        $this -> token = $login['Auth'];
    }

    /**
     * Perform Google ClientLogin authentication
     *
     * @param array:string $account Account details (see top for components)
     * @return string Response, including login token.
     */
    private function login($account){
        /* Log in to google apps */
        $tk_ch = curl_init();
        curl_setopt($tk_ch, CURLOPT_URL, "https://www.google.com/accounts/ClientLogin");
        curl_setopt($tk_ch, CURLOPT_POST, true);
        curl_setopt($tk_ch, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($tk_ch, CURLOPT_POSTFIELDS, $account);
        $responseTxt = curl_exec($tk_ch);
        curl_close($tk_ch);

        /* Parse response (very hack-y but works) */
        $responseLines = split('=', $responseTxt);
        $lastkey = false;
        for($i = 0; $i < count($responseLines); $i++) {
            $line = $responseLines[$i];
            if($i == 0) {
                /* First line is just a key */
                $lastkey = $line;
            } else {
                $vals = split("\n", $line);
                if($i != count($responseLines) - 1) {
                    /* Split into value and next key */
                    $nextkey = $vals[count($vals) - 1];
                    unset($vals[count($vals) - 1]);
                } else {
                    $nextkey = false;
                }
                $response[$lastkey] = implode("\n", $vals);
                $lastkey = $nextkey;
            }
        }
        return $response;
    }
    
    protected function authorize() {
        return 'Authorization: GoogleLogin auth='.trim($this -> token);
    }
}

And the part you’re probably searching for: how to get an access token out of the google-api-php-client:

/**
 * New OAuth2 authentication using google-api-php-client to get us logged in instead.
 */
class OAuthEmailSignatureUpdater extends EmailSignatureUpdater {
    protected function __construct(array $conf) {
        $this -> token = $this -> login($conf);
    }
    
    private function login(array $conf) {
        require_once(dirname(__FILE__) . "/vendor/google-api-php-client/src/Google/autoload.php");

        // Start client
        $client = new Google_Client();
        $client -> setApplicationName("Test Application");
        
        // Load key
        $key = file_get_contents($conf['key_file']);
        if(!$key) {
            throw new Exception("Key could not be loaded from file " .$conf['key_file']);
        }
    
        // Set up auth with scopes
        $gauth = new Google_Auth_AssertionCredentials(
                $conf['service_account'],
                array('https://apps-apis.google.com/a/feeds/emailsettings/2.0/'),
                $key);
        // sub- pretend to be a user
        // Should probably be the same as the user you logged in as with ClientLogin
        if(isset($conf['sub'])) {
            $gauth -> sub = $conf['sub'];
        }
        $client -> setAssertionCredentials($gauth);
        if($client -> getAuth() -> isAccessTokenExpired()) {
            $client -> getAuth() -> refreshTokenWithAssertion($gauth);
        }
        $client -> setClientId($conf['client_id']);
    
        /* Save token for later use */
        $token = $client -> getAccessToken();
        if (!$token) {
            throw new Exception("Google apps login failed!");
        }
        $arr = json_decode($token);
        if(!isset($arr -> access_token)) {
            throw new Exception("Google apps login did not return access token!");
        }
        return $arr -> access_token;
    }
    
    protected function authorize() {
        return 'Authorization: Bearer ' . $this -> token;
    }
}

In the superclass, we provide a getUpdater() function to dish out an old or new updater based on the configuration. In future releases, you could just throw an Exception instead of logging in with ClientLogin, as you know it will fail!

/**
 * Contains email signature updater methods, with no login method
 */
abstract class EmailSignatureUpdater {
    /**
     * @var string The login token.
     */
    protected $token;

    /**
     * @param array $conf The configuration to use.
     */
    protected abstract function __construct(array $conf);

    /**
     * Return authorization header
     */
    protected abstract function authorize();

    // A series of methods which make raw HTTP requests
    // to https://apps-apis.google.com/a/feeds/emailsettings/2.0/ were
    // located here, and are not relevant to this post.
    
    /**
     * Based on the configuration, return a signature updater which logs in using the correct API.
     * 
     * @param array $conf
     */
    static function getUpdater(array $conf) {
        if(isset($conf['user']) && isset($conf['domain']) && isset($conf['password'])) {
            return new ClientLoginEmailSignatureUpdater($conf);
        } else if(isset($conf['service_account']) && isset($conf['client_id']) && isset($conf['key_file'])) {
            return new OAuthEmailSignatureUpdater($conf);
        }
        throw new Exception("Configuration insufficient for loading email signatures. See config.php.example.");
    }
}

To utilise this, you need to change HTTP methods to care less about where their headers come from. I was using cURL for this, but on any library, you need to set the header using the new authorize() function before you send off the request:

curl_setopt($ch, CURLOPT_HTTPHEADER, array($this -> authorize(), ..., ...);

The idea is that users can simply alter their configuration to pick up the newly supported OAuth, allowing you to shift it from a software problem to an administrative/configuration problem:

/* Google apps domain config */
/* Old style ClientLogin usage (deprecated) */
//$config['google']['user'] = "admin";
//$config['google']['domain'] = "example.com";
//$config['google']['password'] = "...";

/* New login details for OAuth */
$config['google']['service_account'] = '...@developer.gserviceaccount.com';
$config['google']['client_id'] = '....apps.googleusercontent.com';
$config['google']['key_file'] = dirname(__FILE__) . '/google-key.p12';
$config['google']['sub'] = "admin@example.com"; // optional

Within your application, you then construct a new any-authentication object based on the configuration, and call its old methods as normal:

$updater = EmailSignatureUpdater::getUpdater($config['google']);
$updater -> doThing('param1', 'param2', etc);

I hope this helps!

How to use a Raspberry Pi as a print server

Thu, 16 Apr 2015 00:00:00 +0000

This post is designed for people who want to share a simple USB printer, such as this receipt printer, over the network.

Usually, you just connect up the printer to the computer like this:

But if you are sending the print jobs from a central server, you would instead follow these steps, and hook up a Raspberry Pi near the printer to pass on the print-outs for you:

This post will show you a very fuss-free way to do this. Because of its simplicity, if you have multiple computers printing (read: you need a server that can spool), or need two-way communication with the printer, then this setup will not be sufficient for your use case.

One-off setup

If your printer is /dev/usb/lp0, then the command to run is:

nohup nc -klp 9100 > /dev/usb/lp0 2> /dev/null&

There is quite a lot going on in this command, so I’m going to break it down into parts and explain what each one does.

nohup: Lets the command keep running after you log out.
nc -klp 9100: Listens on port 9100 (-lp), and returns to listening after each connection (-k)
> /dev/usb/lp0: Redirects any incoming data to the printer device
2> /dev/null: Suppresses errors by sending them to /dev/null
&: Runs the command in the background so that you can keep using the terminal.

Run every boot

Simply schedule the command in cron as a @reboot task.

crontab -e

And add the line:

@reboot nohup nc -klp 9100 > /dev/usb/lp0 2> /dev/null&

Note that if you reboot the printer, you will also need to reboot the raspberry pi to get it to reconnect without logging in!

Send some tests

From a computer somewhere else on the network, send a test print-out:

echo "Hello world" | nc 10.x.x.x 9100

If the target printer is a thermal receipt printer, then you could also use mike42/escpos-php to send it more elaborate commands:

<?php
$fp = fsockopen("10.x.x.x", 9100);
/* Print a "Hello world" receipt" */
$printer = new Escpos($fp);
$printer -> text("Hello World!\n");
$printer -> cut();
fclose($fp);

How to run Tetris on your Raspberry Pi

Thu, 09 Apr 2015 00:00:00 +0000

This is a simple walkthrough on how to install my Tetris clone, Blocks, on a Raspberry Pi.

On most computers running Debian (or Raspbian in the case of the Raspberry Pi), it’s as simple as clone, compile, run:

sudo apt-get install libncurses5-dev doxygen
git clone https://github.com/mike42/blocks
cd blocks
make
./bin/blocks

If you have any issues running this, then you need to fetch a newer version of GCC, as this needs C++11 support to compule (see last section for instructuins).

But if all goes to plan, you will get something like this in your terminal:

Use the keyboard to control the game:

Move: Right, down, left
Rotate: Up
Drop: Spacebar
Quit: q

Get a screen

Basically any project with graphics can benefit from one of these. Simply add on a TFT shield, such as PiTFT to create a tiny console:

Of course, this is still keyboard-controlled, but with some hacking, I’m sure you could map touch events to keyboard actions.

Troubleshooting: Update GCC

The Raspbian spftware image which many Raspberry Pi’s have is slightly too old to compile Blocks, which requires C++11 support.

Luckily, it’s very easy to upgrade from wheezy to jessie to add it. You know you need to do this if you get this error compiling:

$ git clone https://github.com/mike42/blocks
$ make
mkdir -p bin
g++ src/main.cpp src/blocks_game.cpp src/blocks_shape.cpp -o bin/blocks -lcurses -lrt -std=c++11 -Wall
cc1plus: error: unrecognized command line option ‘-std=c++11’
cc1plus: error: unrecognized command line option ‘-std=c++11’
cc1plus: error: unrecognized command line option ‘-std=c++11’
Makefile:2: recipe for target 'default' failed
make: *** [default] Error 1

Generally this means you don’t have GCC 4.8, which is not available in wheezy edition of Raspian.

$ g++ --version
g++ (Debian 4.6.3-14+rpi1) 4.6.3
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

So to summarise this thread, you need to:

nano /etc/apt/sources.list

Find this line:

deb http://mirrordirector.raspbian.org/raspbian/ wheezy main contrib non-free rpi

And change the word “wheezy” to “jessie”:

deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi

You can then update everything with:

sudo apt-get update && sudo apt-get dist-upgrade

You are now running the newer jessie release, which gives you access to the GCC 4.8 package we need:

apt-get install g++-4.8

So we can pick up where we left off, and compile the game:

make
./bin/blocks

Getting a USB receipt printer working on Windows

Wed, 08 Apr 2015 00:00:00 +0000

Note: This post is a Windows adaptation of an earlier post, Getting a USB receipt printer working on Linux, mainly in response to these questions.

In this post, I’ll step through how to get a USB thermal receipt printer appearing on Windows. The aim of this is to be able to send raw text to the printer, so that we can point a driver such as escpos-php at it. The printer tested here is once again this Epson TM-T20:

The directions below are for Windows 7, so your mileage may vary if you are on an older or newer version.

If you have issues following these steps, make sure you can locate your printer in Device Manager, and that it has “USB Print Support”.

Add the printer

Find Devices and Printers and click Add a Printer.

Add it as a Local printer, using the USB virtual port, probably USB0001:

Use the Generic / Text Only driver.

Name the printer whatever you like, and then share it under the same name:

At this point, it should pop up in the window in the background, and also prompt you to Print a test page.

The test print is plain-text, and depending on your printer, will look something like this:

Finally, you need to verify that your printer can be accessed locally, by typing \\localhost into Windows Explorer. If all goes to plan, you will see the new printer there too:

Run a command-line test print

We now know that your printer is working, and can be accessed via its share name (even locally).

Test printing from the command-line. Fire up cmd.exe and try to send it some text to verify that it’s working:

echo "Hello World" > testfile
print /D:"\\%COMPUTERNAME%\Receipt Printer" testfile
del testfile

Printing something useful

This is where you start to see real results. Receipt printers are not just for printing plain-text. Many of them support a standard called ESC/POS, which contains formatting commands.

The snippet below, from this earlier post, generates some basic ESC/POS commands.

Install PHP if you don’t have it already, and call the below code foo.php:

<?php
/* ASCII constants */
const ESC = "\x1b";
const GS="\x1d";
const NUL="\x00";

/* Output an example receipt */
echo ESC."@"; // Reset to defaults
echo ESC."E".chr(1); // Bold
echo "FOO CORP Ltd.\n"; // Company
echo ESC."E".chr(0); // Not Bold
echo ESC."d".chr(1); // Blank line
echo "Receipt for whatever\n"; // Print text
echo ESC."d".chr(4); // 4 Blank lines

/* Bar-code at the end */
echo ESC."a".chr(1); // Centered printing
echo GS."k".chr(4)."987654321".NUL; // Print barcode
echo ESC."d".chr(1); // Blank line
echo "987654321\n"; // Print number
echo GS."V\x41".chr(3); // Cut
exit(0);

You would send generated commands to the printer like this:

php foo.php > testfile
print /D:"\\%COMPUTERNAME%\Receipt Printer" testfile
rm testfile

Scaling this up

The correct ESC/POS codes are quite tricky to generate with manually, which is why I put together the escpos-php driver. You can find more information on that at:

mike42/escpos-php on GitHub
What is ESC/POS, and how do I use it?

A simple “Hello World” receipt to your Windows shared printer would be scripted as (call this one foo2.php):

<?php
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\WindowsPrintConnector;

try {
    // Enter the share name for your USB printer here
    $connector = new WindowsPrintConnector("Receipt Printer");
    $printer = new Printer($connector);

    /* Print a "Hello world" receipt" */
    $printer -> text("Hello World!\n");
    $printer -> cut();
    
    /* Close printer */
    $printer -> close();
} catch(Exception $e) {
    echo "Couldn't print to this printer: " . $e -> getMessage() . "\n";
}

This would be sent to the printer by loading it from the web, or running the script on the command-line:

php foo2.php

The full ESC/POS snippet with formatting, coded up with escpos-php, would look like this (call this one foo3.php):

<?php
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\WindowsPrintConnector;
try {
    // Enter the share name for your USB printer here
    $connector = new WindowsPrintConnector("Receipt Printer");
    $printer = new Printer($connector);

    /* Print some bold text */
    $printer -> setEmphasis(true);
    $printer -> text("FOO CORP Ltd.\n");
    $printer -> setEmphasis(false);
    $printer -> feed();
    $printer -> text("Receipt for whatever\n");
    $printer -> feed(4);

    /* Bar-code at the end */
    $printer -> setJustification(Printer::JUSTIFY_CENTER);
    $printer -> barcode("987654321");
    
    /* Close printer */
    $printer -> close();
} catch(Exception $e) {
    echo "Couldn't print to this printer: " . $e -> getMessage() . "\n";
}

And again, this could be executed by loading the page through the web, or invoking the command directly:

php foo3.php

How to empty your local user account

Sat, 04 Apr 2015 00:00:00 +0000

If you’re not going to use a user account on your computer again, but can’t delete it for some reason, then emptying it is the next best thing to do.

Word of warning: Save anything you want to keep before you start deleting things. These are destructive commands which delete all of the files and settings in the current user’s profile. If you are at all unsure, consider using a file browser to clear out the profile instead.

Windows:

cd %USERPROFILE%
del /A / F /Q /S .

Linux or Mac:

cd ~
rm -Rf .

This will make sure that the disused account no-longer wastes any disk space.

Getting a USB receipt printer working on Linux

Sun, 08 Mar 2015 00:00:00 +0000

In this post, I’ll step through how to get a thermal receipt printer with USB interface appearing on Linux. The aim of this is to be able to point a driver such as escpos-php at the device. The printer used here is an Epson TM-T20, which is very common in point-of-sale environments.

I have previously written quite a bit about how to use thermal receipt printer protocols, but the previous printer I covered had only a network interface, not USB like this one:

The directions below are for Debian, but could be adapted for any other Linux.

Find the device file

Plug in your printer, and check that usblp sees it:

dmesg
[12724.994550] usb 8-4: new full-speed USB device number 5 using ohci-pci
[12725.168956] usb 8-4: New USB device found, idVendor=04b8, idProduct=0e03
[12725.168963] usb 8-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[12725.168968] usb 8-4: Product: TM-T20
[12725.168971] usb 8-4: Manufacturer: EPSON
[12725.168975] usb 8-4: SerialNumber: ....
[12725.175114] usblp 8-4:1.0: usblp1: USB Bidirectional printer dev 5 if 0 alt 0 proto 2 vid 0x04B8 pid 0x0E03

This kernel module makes your printer visible as a device file, so that it can be accessed in the old-fashioned way. Find the new device file under /dev/usb:

ls /dev/usb

In my case, this was /dev/usb/lp1. The next step is to see if you can write to it:

echo "Hello" >> /dev/usb/lp1

Chances are, you will get a permission denied error at this point, so find out what group the printer is in:

stat /dev/usb/lp1

Which will show output something like:

File: ‘/dev/usb/lp1’
  Size: 0         	Blocks: 0          IO Block: 4096   character special file
Device: 5h/5d	Inode: 220997      Links: 1     Device type: b4,1
Access: (0660/crw-rw----)  Uid: (    0/    root)   Gid: (    7/      lp)
...

This file is owned by group lp (“line printer”). If your username was bob, you would add yourself to this group using:

sudo usermod -a -G lp bob

If you plan to build a web-based point-of-sale system with this, then also add the www-data user to that group.

Now log out and back in, and the previous test should now be working:

echo "Hello" >> /dev/usb/lp1

Troubleshooting: Check usblp

If these steps don’t work, then your computer either doesn’t have, or isn’t using usblp You’ll need to check a few things:

Install a different linux-image if the driver is not on your computer at all.
modprobe or insmod usblp
blacklist a vendor driver which has claimed the interface.
run lsusb -v and usb-devices (look for driver=)

Printing something useful

As a duplicated section from my earlier post, the printer uses ESC/POS, which means it accepts plaintext with some special commands for formatting.

A simple receipt-generator, foo.php, might look like this:

<?php
/* ASCII constants */
const ESC = "\x1b";
const GS="\x1d";
const NUL="\x00";

/* Output an example receipt */
echo ESC."@"; // Reset to defaults
echo ESC."E".chr(1); // Bold
echo "FOO CORP Ltd.\n"; // Company
echo ESC."E".chr(0); // Not Bold
echo ESC."d".chr(1); // Blank line
echo "Receipt for whatever\n"; // Print text
echo ESC."d".chr(4); // 4 Blank lines

/* Bar-code at the end */
echo ESC."a".chr(1); // Centered printing
echo GS."k".chr(4)."987654321".NUL; // Print barcode
echo ESC."d".chr(1); // Blank line
echo "987654321\n"; // Print number
echo GS."V\x41".chr(3); // Cut
exit(0);

And you would send it to the printer like this:

php foo.php > /dev/usb/lp1

Scaling this up

The codes are quite tricky to work with manually, which is why I put together the escpos-php driver. You can find it at:

mike42/escpos-php on GitHub
What is ESC/POS, and how do I use it?

The previous example would be written using escpos-php as:

<?php
require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp1");
$printer = new Printer($connector);

/* Print some bold text */
$printer -> setEmphasis(true);
$printer -> text("FOO CORP Ltd.\n");
$printer -> setEmphasis(false);
$printer -> feed();
$printer -> text("Receipt for whatever\n");
$printer -> feed(4);

/* Bar-code at the end */
$printer -> setJustification(Printer::JUSTIFY_CENTER);
$printer -> barcode("987654321");
$printer -> cut();

This would be sent to the printer by loading it from the web, or running the script on the command-line:

php foo2.php

Configure asterisk with wildcard extensions

Thu, 19 Feb 2015 00:00:00 +0000

This post covers a common use case when deploying a phone system: Somebody in the business needs to be able to record voicemail greetings for other people.

This assumes that you are already running asterisk, and that there is already something like this in your Dialplan (extensions.conf) for users to record a greeting:

; Record voicemail greeting
exten => *,1,AGI(scripts/record-voicemail-greeting.pl);

To run this script as somebody else, in the phone sense, we just need to change the CallerID before we run it. Some things you’ll need to know to do this:

An “X” in the extension matches any digit
EXTEN is a variable holding the current extension
CALLERID(num) is another variable, which holds the CallerID number
${EXTEN:2} is a “substring”, which cuts the first two letters off the extension

With that in mind, if * records your own voicemail, then **4567 would record 4567’s voicemail using this snippet:

; Record other person's voicemail greeting
exten => **XXXX,1,Set(CALLERID(num)=${EXTEN:2})
exten => **XXXX,2,Goto(*,1)

Of course, you’re not likely to want to enable this for the whole business, which is why you can also check the CallerID before you change it. This alternative snippet allows you to record any voicemail greeting, but only if you are calling from 1234.

; Record other person's voicemail greeting (if calling from phone 1234)
exten => 1234/**XXXX,1,Set(CALLERID(num)=${EXTEN:2})
exten => 1234/**XXXX,2,Goto(*,1)

Two ways to back up your Google Apps account

Wed, 11 Feb 2015 00:00:00 +0000

If you use Gmail or hosted Google Apps, you might be interested in taking a backup of your data, such as emails, Drive documents, and calendar entries. Thankfully, you can usually export copy of your account data using Google Takeout.

If your hosted Apps account has Takeout disabled, then you can do a backup, it simply has a few extra steps.

Option 1: Google Takeout

This method is nice and simple. Simply go to the Data tools - Download your data page, and select which services you want to export:

It can be a bit eye-opening to see the amount of data Google has on you (Files, conversations, location history, etc). At this point, click through to “Prepare Download”. Depending on the size of your account, this may take the duration of a coffee break, a few hours, or even an entire day.

If you check the box for it, you’ll get an email like this when your Download completes:

And this lets you fetch a single file:

The .zip file contains a series of folders, one for each service. The defaults seem to be:

Mail: A unix mbox file
Calendar: One iCal file for each calendar
Contacts: One vCard file for each group.
Drive: Exports as PDF, docx, xlsx

Option 2: Export data from each service

Sometimes, Google Takeout isn’t an option.

Mst Google services have some sort of data connection for apps to use, which can be used for export. This means if you have a new contact manager, or want to include your Google Drive in your PC backup, then there are still some options to do so.

The export formats in these examples should match the Google Takeout defaults. I’ve written up a brief guide below for each of the following services:

Mail
Contacts
Calendar
Drive

Mail

If you are not a power user, then a good option would be to set up a copy of Mozilla Thunderbird to use IMAP, and regularly using it for your email. This is a simple way to keep a clone of your inbox on your desktop computer, so that it can be included in backups.

If you are more tech-savvy, then the rest of this section will focus on helping you generate an mbox file containing a full backup of your email, the same format as Takeout uses. The best tool for that is a Linux program called getmail.

On Debian or Ubuntu Linux, issue this command to install getmail:

sudo apt-get install getmail4

For other package managers, see these directions.

First, you need to enable IMAP for your account, see Google’s article: Get started with IMAP and POP3, for the steps.

Now create a file at ~/.getmail/getmailrc, and configure it to read your email account via IMAP/SSL.

[retriever]
type = SimpleIMAPSSLRetriever
server = imap.gmail.com
port = 993
username = bob@mail.example.com
password = ....

[destination]
type = Mboxrd
path = ~/inbox

[options]
verbose = 1

Then run:

getmail

After some time, you will end up with a large mbox file at ~/inbox, containing all of your mail.

If, for some reason, you need to use POP3 instead, then see this article on Gmail backup

Contacts

Go to your contacts, and find a group. Check the box next to each name, and then find the Export button:

Select the vCard format here, as it’s the same format which Takeout would have used:

Calendar

Google provides a share-able iCal link, which you can download once, but it is only available if your calendar is public.

So, if your calendar isn’t too sensitive, click “Share” and make the calendar public:

Go to “Calendar Settings”, find the iCal link. It may take a few minutes for the link to start working, but once it does, download it, and then turn off public sharing.

There is a chance that somebody else loads your calendar while its public. An alternative which avoids making the calendar public would be to save the events individually instead.

Drive

Exporting from Google Drive is nice and simple. Select all of your files (Shift+Click):

And then find the Download button:

If you apply this to your whole drive, it may take a while, so you may wish to download it in parts if your Internet can be unreliable.

How do I use these files?

Google Drive’s files are exported in familiar formats. If you haven’t used an mbox, vCard or ics file before, then you will need to find a program which can read these for you.

Google’s support answer “Download your data: Per-service information” contains a list of files types which you’ll run into during this process, and suggests programs which can import them.

How to merge edges in GraphViz

Sun, 01 Feb 2015 00:00:00 +0000

If you are sketching out a node in graphviz which has many ancestors, a dense collection of arrow-heads can become unsightly:

The code for the above graph is:

digraph G {
	{a, b, c} -> d;
	d -> e;
}

To merge the edges together, we can instead point these three nodes to an intermediate node, using edges without arrow. This example is adapted from the GraphViz FAQ (link).

This gives us:

digraph G {
	d1 [shape=point,width=0.01,height=0.01];
	{a, b, c} -> d1 [dir=none];
	d1 -> d;
	d -> e;
}

To compile these examples, you can use an online tool such as Webgraphviz. If you have GraphViz installed, then run dot over them instead:

dot -Tpdf example.dot > example.pdf

Howto: Tethered photo capture on Linux

Thu, 15 Jan 2015 00:00:00 +0000

Have you ever wondered how professionals get photos to pop up on their computer as they snap them? Most higher-end cameras have mini USB connection, and software is available to retrieve images as they are taken.

Rather than use a GUI app, in this post I’ll use a command-line program called gphoto2 to drop the images into a folder. With large thumbnails set in your file browser, a desktop program would be redundant.

Basic process

First, you need to install the gphoto2 program. Depending on your system, one of the following commands should do the trick:

apt-get install gphoto2
yum install gphoto2

Now, plug in the camera. The command to do a “tethered capture” is:

gphoto2 --capture-tethered

Unfortunately, in most desktop environments, your file manager will mount the camera automatically. If this is the case, then the command will give you an error:

mike@mikebox:~$ gphoto2 --capture-tethered
Waiting for events from camera. Press Ctrl-C to abort.                         

*** Error ***              
An error occurred in the io-library ('Could not claim the USB device'): Could not claim interface 0 (Device or resource busy). Make sure no other program (gvfs-gphoto2-volume-monitor) or kernel module (such as sdc2xx, stv680, spca50x) is using the device and you have read/write access to the device.
*** Error (-53: 'Could not claim the USB device') ***       

For debugging messages, please use the --debug option.
Debugging messages may help finding a solution to your problem.
If you intend to send any error or debug messages to the gphoto
developer mailing list <gphoto-devel@lists.sourceforge.net>, please run
gphoto2 as follows:

    env LANG=C gphoto2 --debug --debug-logfile=my-logfile.txt --capture-tethered

Please make sure there is sufficient quoting around the arguments.

Simply find the camera and unmount it using the eject button:

Now when you type the command, it will block until a photo is taken, and then show you the name of the photo:

mike@mikebox:~$ gphoto2 --capture-tethered
Waiting for events from camera. Press Ctrl-C to abort.                         
UNKNOWN PTP Property 5007 changed
...
Downloading 'DSC_0236.JPG' from folder '/store_00010001/DCIM/100NCD90'...
Saving file as DSC_0236.JPG                                                    
Deleting 'DSC_0236.JPG' from folder '/store_00010001/DCIM/100NCD90'...

Each of the photos is loaded into the working directory after you release the shutter, so you simply close gphoto2 when you’re done — no manual downloading or SD card required.

Extending this method

Ok, so now that I’ve covered this basic use case, the real reason I suggest gphoto2 is that it will let you script just about anything to do with your camera.

Just typing gphoto2 --help shows that it can let you trigger a photo or video on a timer, download and delete folders from the camera, or hook up programs via a pipe for processing the files in realtime.

Endless possibilities.

Locking down your VOIP setup with a SIP Threat Manager

Thu, 08 Jan 2015 00:00:00 +0000

If you run a Voice over IP network which is available from the Internet, then it’s quite important to lock it down properly, so that it isn’t hijacked for relaying spam calls at your expense.

This article will cover the steps you need to deploy the SIP Threat Manager from Allo, which you can think of as a security-focused SIP proxy & firewall.

Topology

For example, you might have SIP clients both on-site with your server, and also allow people with their own devices to connect directly from home. One of the simplest ways to harden this setup is to add a specialised SIP router between your server and the Internet, to filter connections and log security-relevant events:

This post will show you how to migrate your VOIP network to this potentially more secure topology using Allo’s SIP Threat Manager.

The Allo STM Box

The STM itself is only small box, with two 100 Mbit/s Ethernet ports, and two USB ports. It is USB-powered, so one of these ports is for powering the box, and the other is for connecting external storage for log files. It is not power-hungry, and I was able to run it from a laptop USB port without any issues. Other than this, it’s externally a typical embedded network device: it has indicator lights, a power switch, a factory reset button, and a console port.

Internally, it runs a MIPS processor, which appears to host Snort on embedded Linux.

Initial Setup

Although the box advertises that it will work out-of-the-box, I found that it was easier to configure the box to match my network, than to re-work my network around the box. This section will simply show you how to get logged in and change the box’s IP address.

First up, I took a look at the console, which is accessible at a baudrate of 38400. It’s far from the most functional CLI around, so I only made use of the factoryreset function to get a clean slate. It showed a few of the open source packages running, such the dropbear sshd, lighttpd and crond among other familiar programs:

Please press Enter to activate this console. 
Starting pid 942, console /dev/ttyS2: '/usr/bin/maincli'
Shield STM Appliance Appliance
shield> factoryreset

Will output 1024 bit rsa secret key to '/etc/dropbear/dropbear_rsa_host_key'
Generating key, this may take a while...
Public key portion is:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgwCKBcVlWK+UiiELbg2CNfOt9rNmj51dmyz7d10MgRfAk9XU9x+kmlMueCFEBMTchsaywigLw0yFqeMZ
Fingerprint: md5 50:5b:c2:64:d4:87:f8:86:ab:c6:e1:59:e4:16:c2:cf
Generating a 1024 bit RSA private key
...++++++
...................++++++
writing new private key to '/etc/lighttpd/webserver.pem'
-----
ip: RTNETLINK answers: No such process
Jan  1 00:40:41 crond[875]: crond 2.3.2 dillon, started, log level 8

mount: mounting /dev/sda1 on /cf/disk failed

The CLI command show ip confirmed that the default IP of the box is 192.168.100.1, netmask 255.255.255.0. From a Linux laptop, you can change IP to something nearby and confirm that you can see the device with these commands:

# ifconfig wlan0 192.168.100.2 netmask 255.255.255.0
# arp-scan -l
Interface: wlan0, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.8.1 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
192.168.100.1	00:17:f7:00:9b:0a	CEM Solutions Pvt Ltd

The STM is then accessible via the web address https://192.168.100.1, with the default credentials admin / admin.

The IP setup is located under Device → General Setup. Change this to DHCP or a spare address on your network:

Configuration

Now that you can access the STM from any device, your first task is to change the admin password. The button for this is in the top-right:

The STM only allows one session at a time- whilst it’s a good idea not to log in twice, this was a surprising limitation. At the STM does not act as its own SIP endpoint, my server was already able to contact the Internet through it at this point.

I quickly screen captured the available settings so that you can click through them. Some of these are SIP-specific, and others of which are general firewall features. One of the more interesting features which you can’t set up with iptables is location-based IP filtering. This could, for example, block problematic SIP calls coming from fraud hotspots in areas where your organisation doesn’t operate.

Device

Security Settings

Security Alerts

Tools

So once the network is set up on the STM, no changes need to be applied to your SIP server, other than its gateway or IP.

Notes

Whilst this box works as it’s supposed to, I found it to have an un-polished user experience.

The network interface labels on the box had a label over them with the opposite information.
The box is closed on port 80: It doesn’t reply to HTTP requests, even to redirect them to HTTPS.
The command prompt wasn’t as useful as other network devices.
I couldn’t get SSH login or NTP to work, although I didn’t investigate these in great detail.
The LAN interface (but not the WAN interface) did not light up when connected to a Gigabit POE network, but did work on a 100 megabit network.

However, there are some positives: The 100 Mbit/s interfaces are more than sufficient for voice traffic, the configuration was simple, and USB is a good choice of power supply for equipment which can be connected directly to a server.

Do you really need another box?

This depends on your setup. If your VOIP server doesn’t speak to the Internet, then this box won’t fit into your topology.

If it only sees the outside world via an ISP-run SIP trunk, then this type of security is probably not necessary either. Security measures you would use instead are:

Use firewall rules to restrict connections so that only the SIP trunk can speak to your VOIP server.
Configure your VOIP server to “stay on the line” for calls (directmedia=no in Asterisk) so that the phones do not speak directly to the trunk, and disallow registration from the Internet.

If your VOIP server accepts connections from the public internet throgh SIP, then some sort of separate, SIP-aware firewall or proxy is highly advisable.

Acknowledgement

Thanks to Allo (allo.com) for sending in the box which is used for this example setup.

How to set up Asterisk in 10 minutes

Fri, 02 Jan 2015 00:00:00 +0000

Asterisk is an open-source IP PABX, meaning it lets you run a phone system over your computer network. Whilst IP telephony has been gaining the upper hand over traditional PABX’s for years, few people outside the industry realise just how easy it is to set up your own phone server.

With this guide, you can turn any Linux device into your own PABX for free. We’ll set up two SIP devices on a small network, so that they can dial each other.

Prepare the environment

First, you should get something with Linux. A virtual machine, a spare laptop, a Raspberry pi- anything.

Install asterisk with one of the following commands, depending on your distribution:

apt-get install asterisk
yum install asterisk

If you don’t have an IP phone handy, then you need a program on your computer which speaks SIP (Session Initiation Protocol). This guide uses Linphone (available for Linux and Windows among other platforms) and the Polycom 331 as examples, but any two SIP endpoints will work just as well for testing.

Make users on the server

Asterisk keeps its configuration in /etc/asterisk. The file we need to edit for this setup is users.conf. Open it up with your favourite text editor:

nano /etc/asterisk/users.conf

The syntax is similar to .ini files. Add two users to the bottom of the file:

[6001]
fullname = Example Bob
secret = 1234
hassip = yes
context = users
host = dynamic

[6002]
fullname = Example Joe
secret = 1234
hassip = yes
context = users
host = dynamic

What does this mean?

[600..]: This is the username, and will become the user’s extension on our small network. Dial 6001 for Bob or 6002 for Joe. Whilst 'bob' and 'joe' could also be used here, numeric usernames are more common.
fullname = ...: Used in the Caller ID.
secret = ...: The password used to log in. In a secure system, you would use something other than 1234!
hassip = yes: This tells Asterisk to make a SIP account for the user. Asterisk supports a few other account types, but SIP is the most widely implemented.
context = users: A context is a bit like a category for the user. The extensions which they can dial depend on this.
host = dynamic: This tells Asterisk that the users don’t have a fixed IP address. This means that they must register periodically with the SIP server so that their IP is known.

To activate these changes, save the file, and reload the configuration through the Asterisk console:

mike# asterisk -r -vvvvvvvvv
CLI> reload
CLI> sip show users
Username                   Secret           Accountcode      Def.Context      ACL  ForcerPort
6002                       1234                              users            No   Yes       
6001                       1234                              users            No   Yes

All of those v’s stand for verbose, meaning that the asterisk console will give you more information.

Configure the clients

First you should find out your server’s IP address. From the terminal, you can find this with:

ifconfig

Setting up “Example Joe” on a Linphone instance only takes a few clicks. Add a new account, with 6002 as the identity, and your asterisk server as the proxy address (eg: sip://voip.example.com). Click the image below for an example:

Meanwhile, the Polycom 331 can be configured as “Example Bob” by navigating the menus on the phone itself, or via the web (suggested). The default login is:

Username:: Polycom
Password:: 456

Your asterisk server address needs to be added under SIP -> Servers -> Server 1, while Example Bob’s identity is added under Lines -> Line1. Click the below images for an example.

Once these are saved, the two clients will register with the server. In SIP, clients periodically register so that the server knows where to find them.

In the asterisk console, you will see something like this:

-- Registered SIP '6001' at 192.168.1.4:5060
   > Saved useragent "PolycomSoundPointIP-SPIP_331-UA/3.3.3.0069" for peer 6001

If registration fails, the console will tell you why, provided that you have set the verbosity high enough

You can check which users have registered with this command:

CLI> sip show peers
Name/username              Host                                    Dyn Forcerport ACL Port     Status     
6001/6001                  192.168.1.4                              D   N             5060     Unmonitored 
6002                       (Unspecified)                            D   N             0        Unmonitored 
2 sip peers [Monitored: 0 online, 0 offline Unmonitored: 1 online, 1 offline]

Unfortunately, even after both users have registered, they aren’t ready to communicate yet.

Add extensions to the server

In the world of VOIP, an extension is not a real loop of copper, but a sequential list of things to do when a number is dialled.

This extra step is where Asterisk gets its flexibility. With your extensions.conf setup, you could set your instance to redirect numbers, or dial for 12 seconds before going to voicemail.

We haven’t told the server what to do, so if “Example Bob” makes a call, it wont work yet:

  == Using SIP RTP CoS mark 5
[Nov 15 07:59:30] NOTICE[6070]: chan_sip.c:22753 handle_request_invite: Call from '6001' (192.168.1.4:5060) to extension '6002' rejected because extension not found in context 'users'.

To add extensions, open extensions.conf up:

nano /etc/asterisk/extensions.conf

The syntax is still INI-like. Under [users], we add the steps for each extension, numbered sequentially. In this case, there is only 1 step for each extension: to dial a SIP user.

[users]
exten => 6001,1,Dial(SIP/6001)
exten => 6002,1,Dial(SIP/6002)

In the Asterisk console, type reload to activate the changes.

Now, as planned, both users on the network can dial each-other and have a chat.

More advanced setups

Ok, so you’ve built what is essentially an intercom, but not a full phone system! For a start, you need a way to dial the outside world, and let the outside world dial you. For that, you’ll need to work with hardware and service providers.

New Wordpress theme

Tue, 30 Dec 2014 00:00:00 +0000

Since the last major revision of my site setup, I’ve been including more technical content, which would be easier to read with syntax highlighting and tabs.

The most visible part of the transition is now complete:

Before

After

The old theme was Skittlish, but I decided to move to a new theme which was based on Bootstrap, so that I could use its components. The new theme is a modified version of the default twentyfourteen theme, using the visual style of morphic, with Prism.js added for code highlighting.

From SIP to speaker: Setting up a VOIP Zone Controller

Thu, 20 Nov 2014 00:00:00 +0000

In many scenarios where you find a phone system, you’ll also find a PA system. Ideally, we want the audio from the phones to be able to reach the PA system when a special number is dialled:

With analog phone systems, this is a common feature. IP-based systems can do this too, with the help of a VOIP Zone Controller.

I think of a Zone Controller as an “Ethernet to RCA” adapter. This article will show you how to get a CyberData controller set up with Asterisk.

The equipment

I settled on this CyberData 4-port Zone Controller, which is quite small, POE-powered, and is of course a reliable and configurable embedded SIP endpoint:

In a typical configuration, the controller would continually receive audio, such as background music, and then silence it when an announcement is made (this is called “night ringer”).

This device has 4 different audio outputs, and waits for a DTMF tone by default, which is then mapped to one or more outputs. I had a very simple use case, and disabled this, so that dialing the box simply caused audio output.

On the VOIP server

Hopefully, you’re using Asterisk to run your phone network! This is the usual platform for VOIP enthusiasts. If you’re dealing with a proprietary system, then you’ll need to skip this section.

First, sip.conf needs to have an entry for the zone controller. This should look like a regular phone. Without going into the intricacies of Asterisk’s SIP configuration, this snippet adds the zone controller as user 1234. It’s given the caller ID “PA System”, and is allowed it to connect from any IP address with the specified secret:

[1234]
type = peer
host = dynamic
context = users
hassip = yes
directmedia = no
fullname = PA System
callerid = PA System
secret = ... (something random here) ...
nat = no

In the extensions.conf, you can then make the device contactable by all phones by adding a line to the users context:

[users]
exten => 1234,1,Dial(SIP/1234)

Setting up the controller

First, you need to plug the audio out into some sort of speaker, and the Ethernet into a POE network with an IP phone system.

The CyberData devices have a web interface, so you need to find it on the network. I suggest filtering the output of arp-scan -l eth0 on GNU/Linux, which will find the IP address corresponding to the MAC address printed on your device.

Once you’ve found it, the configuration is dead simple. Type the IP address into a web browser, and log in as user “admin”, password “admin”.

The web interface is extensive, and shows the depth of options which are used in this niche application, such as custom audio snippets and test routines.

Of course, you may want to adjust the network configuration, which is on the “Network Configuration” page. Once you adjust anything, no changes will take effect until you save and reboot.

Remembering the login and password you set up in sip.conf, you will need to fill in the SIP configuration as well.

Once you reboot, you should see the device register from the Asterisk console, and it will then be reachable. More configuration options to explore include:

Tick “Bypass SIP DTMF Entry” in “Zone Config” if you don’t have zones.
Set the admin password (!)
Take a backup by exporting the configuration from the main page

What is ESC/POS, and how do I use it?

Wed, 12 Nov 2014 00:00:00 +0000

ESC/POS is the command set which makes receipt printers print.

Introduction

Before we begin, there’s three things you need to know about ESC/POS:

Most modern receipt printers support it in some form.
It’s dead simple to write.
Commands start with an ESC character (ASCII 27).

The most useful reference for the protocol is this Epson FAQ, which I’ve used previously to implement an ESC/POS printer driver for PHP.

Incidentally, the receipt printed in the above video is an example from the escpos-php repository. I’ll step through this print-out, as it demonstrates all of the basic ESC/POS features.

Command structure

Four specific ASCII characters make appearances in the command sequences-

Abbreviation	Name	Code (Hex)
NUL	Null	0x00
LF	Line Feed	0x0A
ESC	Escape	0x1B
GS	Group Separator	0x1D

Regular text is simply sent to the printer, separated by line-breaks. Commands begin with ESC or GS, and are followed by a printable character, and sometimes some numbers

Numbers are simply passed as a character. For example, ‘5’ is passed as 0x05.

Examples

These examples are taken from the output of demo.php.

Initialisation

When you first connect to the printer, you should initialise it. This reverts to default formatting, rather than the triple-underlined double-strike font which the previous print-out may have been using.

Command

The command to reset the formatting is:: ESC @

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);
$printer -> close();

Hexdump

00000000  1b 40                                             |.@|
00000003

“Hello world” text

This is the simplest type of receipt, and contains only unformatted text.

Example

Command

Text is simply sent to the printer.

Each line must be terminated by a line feed.

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Text */
$printer -> text("Hello world\n");
$printer -> cut();
$printer -> close();

Hexdump

00000000  1b 40 48 65 6c 6c 6f 20  77 6f 72 6c 64 0a 1d 56  |.@Hello world..V|
00000010  41 03                                             |A.|
00000012

Line feeds

The printer can quickly skip past a given number of lines with this command.

Example

Command

The commands are:: LF; ESC d [ number ]; ESC v [ number ]

The first command feeds forward, the second feeds in reverse. From the example, it can be seen that the demo printer does not support reverse paper feeding.

PHP

require_once(dirname(__FILE__) . "/escpos-php/Escpos.php");
$printer = new Escpos();

/* Line feeds */
$printer -> text("ABC");
$printer -> feed(7);
$printer -> text("DEF");
$printer -> feedReverse(3);
$printer -> text("GHI");
$printer -> feed();
$printer -> cut();
$printer -> close();

Hexdump

00000000  1b 40 41 42 43 1b 64 07  44 45 46 1b 65 03 47 48  |.@ABC.d.DEF.e.GH|
00000010  49 0a 1d 56 41 03                                 |I..VA.|
00000016

Print modes

Print modes include font height, width and boldness into a single attribute.

Example

Command

The command is:: ESC ! [number]

The font modes are made from logically OR’ing together a selection of attributes. 0 represents plan Font A text. Mode flags are:

Mode	Number
Font A (no mode)	0
Font B	1
Emphasized	8
Double height	16
Double width	32
Underline	128

The example receipt illustrates the effect of each flag.

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Font modes */
$modes = array(
	Printer:::MODE_FONT_A,
	Printer:::MODE_FONT_B,
	Printer:::MODE_EMPHASIZED,
	Printer:::MODE_DOUBLE_HEIGHT,
	Printer:::MODE_DOUBLE_WIDTH,
	Printer:::MODE_UNDERLINE);
for($i = 0; $i < 2 ** count($modes); $i++) {
	$bits = str_pad(decbin($i), count($modes), "0", STR_PAD_LEFT);
	$mode = 0;
	for($j = 0; $j < strlen($bits); $j++) {
		if(substr($bits, $j, 1) == "1") {
			$mode |= $modes[$j];
		}
	}
	$printer -> selectPrintMode($mode);
	$printer -> text("ABCDEFGHIJabcdefghijk\n");
}
$printer -> selectPrintMode(); // Reset
$printer -> cut();
$printer -> close();

Hexdump

00000000  1b 40 1b 21 00 41 42 43  44 45 46 47 48 49 4a 61  |.@.!.ABCDEFGHIJa|
00000010  62 63 64 65 66 67 68 69  6a 6b 0a 1b 21 80 41 42  |bcdefghijk..!.AB|
00000020  43 44 45 46 47 48 49 4a  61 62 63 64 65 66 67 68  |CDEFGHIJabcdefgh|
00000030  69 6a 6b 0a 1b 21 20 41  42 43 44 45 46 47 48 49  |ijk..! ABCDEFGHI|
00000040  4a 61 62 63 64 65 66 67  68 69 6a 6b 0a 1b 21 a0  |Jabcdefghijk..!.|
00000050  41 42 43 44 45 46 47 48  49 4a 61 62 63 64 65 66  |ABCDEFGHIJabcdef|
00000060  67 68 69 6a 6b 0a 1b 21  10 41 42 43 44 45 46 47  |ghijk..!.ABCDEFG|
00000070  48 49 4a 61 62 63 64 65  66 67 68 69 6a 6b 0a 1b  |HIJabcdefghijk..|
00000080  21 90 41 42 43 44 45 46  47 48 49 4a 61 62 63 64  |!.ABCDEFGHIJabcd|
00000090  65 66 67 68 69 6a 6b 0a  1b 21 30 41 42 43 44 45  |efghijk..!0ABCDE|
000000a0  46 47 48 49 4a 61 62 63  64 65 66 67 68 69 6a 6b  |FGHIJabcdefghijk|
000000b0  0a 1b 21 b0 41 42 43 44  45 46 47 48 49 4a 61 62  |..!.ABCDEFGHIJab|
000000c0  63 64 65 66 67 68 69 6a  6b 0a 1b 21 08 41 42 43  |cdefghijk..!.ABC|
000000d0  44 45 46 47 48 49 4a 61  62 63 64 65 66 67 68 69  |DEFGHIJabcdefghi|
000000e0  6a 6b 0a 1b 21 88 41 42  43 44 45 46 47 48 49 4a  |jk..!.ABCDEFGHIJ|
000000f0  61 62 63 64 65 66 67 68  69 6a 6b 0a 1b 21 28 41  |abcdefghijk..!(A|
00000100  42 43 44 45 46 47 48 49  4a 61 62 63 64 65 66 67  |BCDEFGHIJabcdefg|
00000110  68 69 6a 6b 0a 1b 21 a8  41 42 43 44 45 46 47 48  |hijk..!.ABCDEFGH|
00000120  49 4a 61 62 63 64 65 66  67 68 69 6a 6b 0a 1b 21  |IJabcdefghijk..!|
00000130  18 41 42 43 44 45 46 47  48 49 4a 61 62 63 64 65  |.ABCDEFGHIJabcde|
00000140  66 67 68 69 6a 6b 0a 1b  21 98 41 42 43 44 45 46  |fghijk..!.ABCDEF|
00000150  47 48 49 4a 61 62 63 64  65 66 67 68 69 6a 6b 0a  |GHIJabcdefghijk.|
00000160  1b 21 38 41 42 43 44 45  46 47 48 49 4a 61 62 63  |.!8ABCDEFGHIJabc|
00000170  64 65 66 67 68 69 6a 6b  0a 1b 21 b8 41 42 43 44  |defghijk..!.ABCD|
00000180  45 46 47 48 49 4a 61 62  63 64 65 66 67 68 69 6a  |EFGHIJabcdefghij|
00000190  6b 0a 1b 21 01 41 42 43  44 45 46 47 48 49 4a 61  |k..!.ABCDEFGHIJa|
000001a0  62 63 64 65 66 67 68 69  6a 6b 0a 1b 21 81 41 42  |bcdefghijk..!.AB|
000001b0  43 44 45 46 47 48 49 4a  61 62 63 64 65 66 67 68  |CDEFGHIJabcdefgh|
000001c0  69 6a 6b 0a 1b 21 21 41  42 43 44 45 46 47 48 49  |ijk..!!ABCDEFGHI|
000001d0  4a 61 62 63 64 65 66 67  68 69 6a 6b 0a 1b 21 a1  |Jabcdefghijk..!.|
000001e0  41 42 43 44 45 46 47 48  49 4a 61 62 63 64 65 66  |ABCDEFGHIJabcdef|
000001f0  67 68 69 6a 6b 0a 1b 21  11 41 42 43 44 45 46 47  |ghijk..!.ABCDEFG|
00000200  48 49 4a 61 62 63 64 65  66 67 68 69 6a 6b 0a 1b  |HIJabcdefghijk..|
00000210  21 91 41 42 43 44 45 46  47 48 49 4a 61 62 63 64  |!.ABCDEFGHIJabcd|
00000220  65 66 67 68 69 6a 6b 0a  1b 21 31 41 42 43 44 45  |efghijk..!1ABCDE|
00000230  46 47 48 49 4a 61 62 63  64 65 66 67 68 69 6a 6b  |FGHIJabcdefghijk|
00000240  0a 1b 21 b1 41 42 43 44  45 46 47 48 49 4a 61 62  |..!.ABCDEFGHIJab|
00000250  63 64 65 66 67 68 69 6a  6b 0a 1b 21 09 41 42 43  |cdefghijk..!.ABC|
00000260  44 45 46 47 48 49 4a 61  62 63 64 65 66 67 68 69  |DEFGHIJabcdefghi|
00000270  6a 6b 0a 1b 21 89 41 42  43 44 45 46 47 48 49 4a  |jk..!.ABCDEFGHIJ|
00000280  61 62 63 64 65 66 67 68  69 6a 6b 0a 1b 21 29 41  |abcdefghijk..!)A|
00000290  42 43 44 45 46 47 48 49  4a 61 62 63 64 65 66 67  |BCDEFGHIJabcdefg|
000002a0  68 69 6a 6b 0a 1b 21 a9  41 42 43 44 45 46 47 48  |hijk..!.ABCDEFGH|
000002b0  49 4a 61 62 63 64 65 66  67 68 69 6a 6b 0a 1b 21  |IJabcdefghijk..!|
000002c0  19 41 42 43 44 45 46 47  48 49 4a 61 62 63 64 65  |.ABCDEFGHIJabcde|
000002d0  66 67 68 69 6a 6b 0a 1b  21 99 41 42 43 44 45 46  |fghijk..!.ABCDEF|
000002e0  47 48 49 4a 61 62 63 64  65 66 67 68 69 6a 6b 0a  |GHIJabcdefghijk.|
000002f0  1b 21 39 41 42 43 44 45  46 47 48 49 4a 61 62 63  |.!9ABCDEFGHIJabc|
00000300  64 65 66 67 68 69 6a 6b  0a 1b 21 b9 41 42 43 44  |defghijk..!.ABCD|
00000310  45 46 47 48 49 4a 61 62  63 64 65 66 67 68 69 6a  |EFGHIJabcdefghij|
00000320  6b 0a 1b 21 00 41 42 43  44 45 46 47 48 49 4a 61  |k..!.ABCDEFGHIJa|
00000330  62 63 64 65 66 67 68 69  6a 6b 0a 1b 21 80 41 42  |bcdefghijk..!.AB|
00000340  43 44 45 46 47 48 49 4a  61 62 63 64 65 66 67 68  |CDEFGHIJabcdefgh|
00000350  69 6a 6b 0a 1b 21 20 41  42 43 44 45 46 47 48 49  |ijk..! ABCDEFGHI|
00000360  4a 61 62 63 64 65 66 67  68 69 6a 6b 0a 1b 21 a0  |Jabcdefghijk..!.|
00000370  41 42 43 44 45 46 47 48  49 4a 61 62 63 64 65 66  |ABCDEFGHIJabcdef|
00000380  67 68 69 6a 6b 0a 1b 21  10 41 42 43 44 45 46 47  |ghijk..!.ABCDEFG|
00000390  48 49 4a 61 62 63 64 65  66 67 68 69 6a 6b 0a 1b  |HIJabcdefghijk..|
000003a0  21 90 41 42 43 44 45 46  47 48 49 4a 61 62 63 64  |!.ABCDEFGHIJabcd|
000003b0  65 66 67 68 69 6a 6b 0a  1b 21 30 41 42 43 44 45  |efghijk..!0ABCDE|
000003c0  46 47 48 49 4a 61 62 63  64 65 66 67 68 69 6a 6b  |FGHIJabcdefghijk|
000003d0  0a 1b 21 b0 41 42 43 44  45 46 47 48 49 4a 61 62  |..!.ABCDEFGHIJab|
000003e0  63 64 65 66 67 68 69 6a  6b 0a 1b 21 08 41 42 43  |cdefghijk..!.ABC|
000003f0  44 45 46 47 48 49 4a 61  62 63 64 65 66 67 68 69  |DEFGHIJabcdefghi|
00000400  6a 6b 0a 1b 21 88 41 42  43 44 45 46 47 48 49 4a  |jk..!.ABCDEFGHIJ|
00000410  61 62 63 64 65 66 67 68  69 6a 6b 0a 1b 21 28 41  |abcdefghijk..!(A|
00000420  42 43 44 45 46 47 48 49  4a 61 62 63 64 65 66 67  |BCDEFGHIJabcdefg|
00000430  68 69 6a 6b 0a 1b 21 a8  41 42 43 44 45 46 47 48  |hijk..!.ABCDEFGH|
00000440  49 4a 61 62 63 64 65 66  67 68 69 6a 6b 0a 1b 21  |IJabcdefghijk..!|
00000450  18 41 42 43 44 45 46 47  48 49 4a 61 62 63 64 65  |.ABCDEFGHIJabcde|
00000460  66 67 68 69 6a 6b 0a 1b  21 98 41 42 43 44 45 46  |fghijk..!.ABCDEF|
00000470  47 48 49 4a 61 62 63 64  65 66 67 68 69 6a 6b 0a  |GHIJabcdefghijk.|
00000480  1b 21 38 41 42 43 44 45  46 47 48 49 4a 61 62 63  |.!8ABCDEFGHIJabc|
00000490  64 65 66 67 68 69 6a 6b  0a 1b 21 b8 41 42 43 44  |defghijk..!.ABCD|
000004a0  45 46 47 48 49 4a 61 62  63 64 65 66 67 68 69 6a  |EFGHIJabcdefghij|
000004b0  6b 0a 1b 21 01 41 42 43  44 45 46 47 48 49 4a 61  |k..!.ABCDEFGHIJa|
000004c0  62 63 64 65 66 67 68 69  6a 6b 0a 1b 21 81 41 42  |bcdefghijk..!.AB|
000004d0  43 44 45 46 47 48 49 4a  61 62 63 64 65 66 67 68  |CDEFGHIJabcdefgh|
000004e0  69 6a 6b 0a 1b 21 21 41  42 43 44 45 46 47 48 49  |ijk..!!ABCDEFGHI|
000004f0  4a 61 62 63 64 65 66 67  68 69 6a 6b 0a 1b 21 a1  |Jabcdefghijk..!.|
00000500  41 42 43 44 45 46 47 48  49 4a 61 62 63 64 65 66  |ABCDEFGHIJabcdef|
00000510  67 68 69 6a 6b 0a 1b 21  11 41 42 43 44 45 46 47  |ghijk..!.ABCDEFG|
00000520  48 49 4a 61 62 63 64 65  66 67 68 69 6a 6b 0a 1b  |HIJabcdefghijk..|
00000530  21 91 41 42 43 44 45 46  47 48 49 4a 61 62 63 64  |!.ABCDEFGHIJabcd|
00000540  65 66 67 68 69 6a 6b 0a  1b 21 31 41 42 43 44 45  |efghijk..!1ABCDE|
00000550  46 47 48 49 4a 61 62 63  64 65 66 67 68 69 6a 6b  |FGHIJabcdefghijk|
00000560  0a 1b 21 b1 41 42 43 44  45 46 47 48 49 4a 61 62  |..!.ABCDEFGHIJab|
00000570  63 64 65 66 67 68 69 6a  6b 0a 1b 21 09 41 42 43  |cdefghijk..!.ABC|
00000580  44 45 46 47 48 49 4a 61  62 63 64 65 66 67 68 69  |DEFGHIJabcdefghi|
00000590  6a 6b 0a 1b 21 89 41 42  43 44 45 46 47 48 49 4a  |jk..!.ABCDEFGHIJ|
000005a0  61 62 63 64 65 66 67 68  69 6a 6b 0a 1b 21 29 41  |abcdefghijk..!)A|
000005b0  42 43 44 45 46 47 48 49  4a 61 62 63 64 65 66 67  |BCDEFGHIJabcdefg|
000005c0  68 69 6a 6b 0a 1b 21 a9  41 42 43 44 45 46 47 48  |hijk..!.ABCDEFGH|
000005d0  49 4a 61 62 63 64 65 66  67 68 69 6a 6b 0a 1b 21  |IJabcdefghijk..!|
000005e0  19 41 42 43 44 45 46 47  48 49 4a 61 62 63 64 65  |.ABCDEFGHIJabcde|
000005f0  66 67 68 69 6a 6b 0a 1b  21 99 41 42 43 44 45 46  |fghijk..!.ABCDEF|
00000600  47 48 49 4a 61 62 63 64  65 66 67 68 69 6a 6b 0a  |GHIJabcdefghijk.|
00000610  1b 21 39 41 42 43 44 45  46 47 48 49 4a 61 62 63  |.!9ABCDEFGHIJabc|
00000620  64 65 66 67 68 69 6a 6b  0a 1b 21 b9 41 42 43 44  |defghijk..!.ABCD|
00000630  45 46 47 48 49 4a 61 62  63 64 65 66 67 68 69 6a  |EFGHIJabcdefghij|
00000640  6b 0a 1b 21 00 1d 56 41  03                       |k..!..VA.|
00000649

Underline

Example

Command

The command is:: ESC - [ number ]

The argument is set to 0 for no underline, 1 for underline, 2 for heavy underline.

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Underline */
for($i = 0; $i < 3; $i++) {
	$printer -> setUnderline($i);
	$printer -> text("The quick brown fox jumps over the lazy dog\n");
}
$printer -> setUnderline(0); // Reset
$printer -> cut();
$printer -> close();

Hexdump

00000000  1b 40 1b 2d 00 54 68 65  20 71 75 69 63 6b 20 62  |.@.-.The quick b|
00000010  72 6f 77 6e 20 66 6f 78  20 6a 75 6d 70 73 20 6f  |rown fox jumps o|
00000020  76 65 72 20 74 68 65 20  6c 61 7a 79 20 64 6f 67  |ver the lazy dog|
00000030  0a 1b 2d 01 54 68 65 20  71 75 69 63 6b 20 62 72  |..-.The quick br|
00000040  6f 77 6e 20 66 6f 78 20  6a 75 6d 70 73 20 6f 76  |own fox jumps ov|
00000050  65 72 20 74 68 65 20 6c  61 7a 79 20 64 6f 67 0a  |er the lazy dog.|
00000060  1b 2d 02 54 68 65 20 71  75 69 63 6b 20 62 72 6f  |.-.The quick bro|
00000070  77 6e 20 66 6f 78 20 6a  75 6d 70 73 20 6f 76 65  |wn fox jumps ove|
00000080  72 20 74 68 65 20 6c 61  7a 79 20 64 6f 67 0a 1b  |r the lazy dog..|
00000090  2d 00 1d 56 41 03                                 |-..VA.|
00000096

Cuts

Command

The command is:: ESC V [ number ]

The argument apparently represents whether to perform a ‘partial’ (65) or ‘full’ (66) cut, but has no effect on my model of printer.

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Cuts */
for($i = 0; $i < 5; $i++) {
	$printer -> cut(Printer:::CUT_PARTIAL);
	$printer -> cut(Printer:::CUT_FULL);
}
$printer -> cut();
$printer -> close();

Hexdump

00000000  1b 40 1d 56 42 03 1d 56  41 03 1d 56 42 03 1d 56  |.@.VB..VA..VB..V|
00000010  41 03 1d 56 42 03 1d 56  41 03 1d 56 42 03 1d 56  |A..VB..VA..VB..V|
00000020  41 03 1d 56 42 03 1d 56  41 03 1d 56 41 03        |A..VB..VA..VA.|
0000002e

Emphasis

Example

Command

The command is:: ESC E [ number ]

Use 1 to enable emphasis, and 0 to disable it.

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Emphasis */
for($i = 0; $i < 2; $i++) {
	$printer -> setEmphasis($i == 1);
	$printer -> text("The quick brown fox jumps over the lazy dog\n");
}
$printer -> setEmphasis(false); // Reset
$printer -> cut();
$printer -> close();

Hexdump

00000000  1b 40 1b 45 00 54 68 65  20 71 75 69 63 6b 20 62  |.@.E.The quick b|
00000010  72 6f 77 6e 20 66 6f 78  20 6a 75 6d 70 73 20 6f  |rown fox jumps o|
00000020  76 65 72 20 74 68 65 20  6c 61 7a 79 20 64 6f 67  |ver the lazy dog|
00000030  0a 1b 45 01 54 68 65 20  71 75 69 63 6b 20 62 72  |..E.The quick br|
00000040  6f 77 6e 20 66 6f 78 20  6a 75 6d 70 73 20 6f 76  |own fox jumps ov|
00000050  65 72 20 74 68 65 20 6c  61 7a 79 20 64 6f 67 0a  |er the lazy dog.|
00000060  1b 45 00 1d 56 41 03                              |.E..VA.|
00000067

Double-strike

Example

Command

The command is:: ESC G [ number ]

Use 1 to enable, or 0 to disable. On the model tested here, this appears to be identical to the “emphasis” mode above.

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Double-strike (looks basically the same as emphasis) */
for($i = 0; $i < 2; $i++) {
	$printer -> setDoubleStrike($i == 1);
	$printer -> text("The quick brown fox jumps over the lazy dog\n");
}
$printer -> setDoubleStrike(false);
$printer -> cut();
$printer -> close();

Hexdump

00000000  1b 40 1b 47 00 54 68 65  20 71 75 69 63 6b 20 62  |.@.G.The quick b|
00000010  72 6f 77 6e 20 66 6f 78  20 6a 75 6d 70 73 20 6f  |rown fox jumps o|
00000020  76 65 72 20 74 68 65 20  6c 61 7a 79 20 64 6f 67  |ver the lazy dog|
00000030  0a 1b 47 01 54 68 65 20  71 75 69 63 6b 20 62 72  |..G.The quick br|
00000040  6f 77 6e 20 66 6f 78 20  6a 75 6d 70 73 20 6f 76  |own fox jumps ov|
00000050  65 72 20 74 68 65 20 6c  61 7a 79 20 64 6f 67 0a  |er the lazy dog.|
00000060  1b 47 00 1d 56 41 03                              |.G..VA.|
00000067

Fonts

Example

Command

The command is:: ESC M [ number ]

There are three possible fonts, documented as “A”, “B” and “C”, and numbered 0, 1, and 2. Many printers, including this one, don’t have Font C.

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Fonts (many printers do not have a 'Font C') */
$fonts = array(
	Printer:::FONT_A,
	Printer:::FONT_B,
	Printer:::FONT_C);
for($i = 0; $i < count($fonts); $i++) {
	$printer -> setFont($fonts[$i]);
	$printer -> text("The quick brown fox jumps over the lazy dog\n");
}
$printer -> setFont(); // Reset
$printer -> cut();
$printer -> close();

Hexdump

00000000  1b 40 1b 4d 00 54 68 65  20 71 75 69 63 6b 20 62  |.@.M.The quick b|
00000010  72 6f 77 6e 20 66 6f 78  20 6a 75 6d 70 73 20 6f  |rown fox jumps o|
00000020  76 65 72 20 74 68 65 20  6c 61 7a 79 20 64 6f 67  |ver the lazy dog|
00000030  0a 1b 4d 01 54 68 65 20  71 75 69 63 6b 20 62 72  |..M.The quick br|
00000040  6f 77 6e 20 66 6f 78 20  6a 75 6d 70 73 20 6f 76  |own fox jumps ov|
00000050  65 72 20 74 68 65 20 6c  61 7a 79 20 64 6f 67 0a  |er the lazy dog.|
00000060  1b 4d 02 54 68 65 20 71  75 69 63 6b 20 62 72 6f  |.M.The quick bro|
00000070  77 6e 20 66 6f 78 20 6a  75 6d 70 73 20 6f 76 65  |wn fox jumps ove|
00000080  72 20 74 68 65 20 6c 61  7a 79 20 64 6f 67 0a 1b  |r the lazy dog..|
00000090  4d 00 1d 56 41 03                                 |M..VA.|
00000096

Justification

Example

Command

The command is:: ESC a [ number ]

Use 0 to justify left, 1 to centre the text, or 2 to right-align it.

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Justification */
$justification = array(
	Printer:::JUSTIFY_LEFT,
	Printer:::JUSTIFY_CENTER,
	Printer:::JUSTIFY_RIGHT);
for($i = 0; $i < count($justification); $i++) {
	$printer -> setJustification($justification[$i]);
	$printer -> text("A man a plan a canal panama\n");
}
$printer -> setJustification(); // Reset
$printer -> cut();
$printer -> close();

Hexdump

00000000  1b 40 1b 61 00 41 20 6d  61 6e 20 61 20 70 6c 61  |.@.a.A man a pla|
00000010  6e 20 61 20 63 61 6e 61  6c 20 70 61 6e 61 6d 61  |n a canal panama|
00000020  0a 1b 61 01 41 20 6d 61  6e 20 61 20 70 6c 61 6e  |..a.A man a plan|
00000030  20 61 20 63 61 6e 61 6c  20 70 61 6e 61 6d 61 0a  | a canal panama.|
00000040  1b 61 02 41 20 6d 61 6e  20 61 20 70 6c 61 6e 20  |.a.A man a plan |
00000050  61 20 63 61 6e 61 6c 20  70 61 6e 61 6d 61 0a 1b  |a canal panama..|
00000060  61 00 1d 56 41 03                                 |a..VA.|
00000066

Barcodes

Example

Command

The commands are:: GS h [ number ]; ESC k [ number ] [ text ] NUL

The first command sets the barcode height — measured in dots, while the second one prints the actual barcode. The number represents the barcode standard, which for most purposes should be “4”, representing CODE39. 6 standards are supported by the PHP driver.

You will notice that due to driver glitches or printer incompatibility, not all of the barcodes print! As above, my advice is to use CODE39 if you run into this.

PHP

require __DIR__ . '/autoload.php';
use Mike42\Escpos\Printer;
use Mike42\Escpos\PrintConnectors\FilePrintConnector;
$connector = new FilePrintConnector("/dev/usb/lp0");
$printer = new Printer($connector);

/* Barcodes */
$barcodes = array(
	Printer:::BARCODE_UPCA,
	Printer:::BARCODE_UPCE,
	Printer:::BARCODE_JAN13,
	Printer:::BARCODE_JAN8,
	Printer:::BARCODE_CODE39,
	Printer:::BARCODE_ITF,
	Printer:::BARCODE_CODABAR);
$printer -> setBarcodeHeight(80);
for($i = 0; $i < count($barcodes); $i++) {
	$printer -> text("Barcode $i " . "\n");
	$printer -> barcode("9876", $barcodes[$i]);
	$printer -> feed();
}
$printer -> cut();
$printer -> close();

Hexdump

00000000  1b 40 1d 68 50 42 61 72  63 6f 64 65 20 30 20 0a  |.@.hPBarcode 0 .|
00000010  1d 6b 00 39 38 37 36 00  0a 42 61 72 63 6f 64 65  |.k.9876..Barcode|
00000020  20 31 20 0a 1d 6b 01 39  38 37 36 00 0a 42 61 72  | 1 ..k.9876..Bar|
00000030  63 6f 64 65 20 32 20 0a  1d 6b 02 39 38 37 36 00  |code 2 ..k.9876.|
00000040  0a 42 61 72 63 6f 64 65  20 33 20 0a 1d 6b 03 39  |.Barcode 3 ..k.9|
00000050  38 37 36 00 0a 42 61 72  63 6f 64 65 20 34 20 0a  |876..Barcode 4 .|
00000060  1d 6b 04 39 38 37 36 00  0a 42 61 72 63 6f 64 65  |.k.9876..Barcode|
00000070  20 35 20 0a 1d 6b 05 39  38 37 36 00 0a 42 61 72  | 5 ..k.9876..Bar|
00000080  63 6f 64 65 20 36 20 0a  1d 6b 06 39 38 37 36 00  |code 6 ..k.9876.|
00000090  0a 1d 56 41 03                                    |..VA.|
00000095

Resources

Linked previously in this post:

And if you’ve just received an Epson printer and need to figure out how it works:

Setting up an Epson receipt printer

Update 2015-03-10: Re-wrote examples for the newer version of escpos-php.

Update 2016-04-22: Updated examples again to match the latest driver code.

Using xte to script your workflow

Fri, 07 Nov 2014 00:00:00 +0000

In the classic world of desktop automation, “macros” allow you to repeat a task easily. In general, xte is the best way of scripting this up on Linux.

First, you need to install it:

apt-get install xautomation
yum install xautomation

To use xte, you need to send it information via a ‘pipe’. The man page covers the key codes and commands in detail, but I’ll step through some basic examples below.

Examples

I’ve put together a couple of interesting examples to show some uses for this tool.

Example 1: Do a Google search

The example below uses xte to type “Hello world” into a text box.

Type the command, press enter, then quickly click on the text box:

sleep 1 && echo "Hello world" | xte

Example 2: Open a browser and search Wikipedia

To write a script which combines a few commands for xte, you could put it in a bash script. Remember to put a pause between commands so that the windowing system can catch up.

The script below will use Gnome Shells “overview mode” to launch Chromium, then open a new tab, and search Wikipedia for “cars”:

#!/bin/sh
xte << EOF
key Super_L
usleep 100000
str chromium
usleep 100000
key Return
sleep 1
keydown Control_L
key t
keyup Control_L
sleep 1
str https://en.wikipedia.org
usleep 100000
key Return
sleep 3
key Tab
usleep 100000
str cars
usleep 100000
key Return
EOF

Example 3: Draw a spiral in GIMP

When you script an operation, you can interact with any program using a set of rules. Below is a PHP script called spiral.php, which draws and labels a spiral in GIMP, switching the foreground & background colours at each step.

This requires an open GIMP window in the correct part of the screen:

The interaction in this case is quite simple for a computer, but would be tedious to do manually:

N: Open the Line tool
T: Open the Text tool
X: Swap foreground & background colours
Click & Drag: Draw a line

#!/usr/bin/env php
sleep 2
key N
usleep 100000
<?php
// Where the canvas is on-screen
$top_x = 150;
$top_y = 150;
$canvas_width = 400;

// Spiral properties
$pi = 3.14159265358979;
$centre_x = $top_x + $canvas_width / 2;
$centre_y = $top_y + $canvas_width / 2 + 100;
$spins = 5;

// Centre the mouse
echo "mousemove $centre_x $centre_y\n";
echo "usleep 100000\n";

// Draw a spiral
for($t = 0; $t < 1; $t += 0.005) {
	$angle = $spins * $t * 2 * $pi;
	$radius = $canvas_width / 2 * $t;
	$x = (int) ($centre_x + $radius * cos($angle));
	$y = (int) ($centre_y + $radius * sin($angle));
	echo "mousedown 1\n";
	echo "mousemove $x $y\n";
	echo "usleep 100000\n";
	echo "mouseup 1\n";
	echo "key X\n";
}

// Label the spiral
echo "mousemove $centre_x $top_y\n";
echo "key T\n";
echo "usleep 100000\n";
echo "mouseclick 1\n";
echo "usleep 100000\n";
echo "str Spiral\n";

After cropping, the spiral image on its own is:

Example 4: Forward emails

Google’s Gmail web interface has keyboard shortcuts for quick navigation. This example uses:

f: Forward an email
Tab: Move between fields
Ctrl+Enter: Send
j: Next email

Using these shortcuts, this script, forward.txt, will forward an email too bob@email.com and fred@example.com, then navigate to the next email:

sleep 1
key f
sleep 1
str bob@example.com
sleep 1
str fred@example.com
sleep 0.1
key Return
sleep 3
key Return
sleep 1
key Tab
sleep 0.1
keydown Control_L
key Return
keyup Control_L
sleep 3
key j
sleep 2

To send one email through xte, you could run this, and then click over to an open email in Gmail:

cat forward.txt | xte

To send the next 106 emails (e.g. in a label or search), you could instead type:

#!/bin/bash
for i in {1..106}; do echo $i; (cat forward.txt | xte); done

This is not fool-proof, so you would need to adjust the timing if your Internet connection is laggy.

When to use xte

Usually, a task which is supposed to be automated will have an API. For example, GIMP provides a python plugin interface, Gmail can be accessed via IMAP, and Google and Wikipedia searches can be done directly through HTTP. This is always the best way to do things.

However, the automation junkie should have xte in their toolkit, for situations where proper automation is not practical, such as when:

you don’t want to learn an API, to only use it for one day.
you’re doing a repetitive task in a program or website which is feature-poor.
you need to test some feature repeatedly under different circumstances.
you find a game which requires you to click fast.

Setting up an Epson receipt printer

Sat, 25 Oct 2014 00:00:00 +0000

I recently picked up one of these networked thermal receipt printers.

Being Point-of-Sale equipment, these come from a different tradition of printing, and have only a few things in common with regular laser printers. This post will cover the basic steps to getting the printer up and running.

This one has the model number TM-T82II.

Setting up the printer

Firstly, this particular printer only has an ethernet interface, which comes configured with a static IP by default, rather than DHCP. Holding the button next to the network port prints out the settings:

The IP address of the printer is shown 192.168.192.168, and subnet mask 255.255.255.0. To speak to it, we need a computer on the same subnet. In this case the last number of the IP address is the only part which needs to be different.

On GNU/Linux, this is best done with ifconfig:

sudo ifconfig eth0 192.168.192.169 netmask 255.255.255.0

If you used the correct interface, address and netmask, then you should now be able to ping the printer:

$ sudo ifconfig
eth0      Link encap:Ethernet  HWaddr ...
          inet addr:192.168.192.169  Bcast:192.168.192.255  Mask:255.255.255.0
          ...
$ ping 192.168.192.168
PING 192.168.192.168 (192.168.192.168) 56(84) bytes of data.
64 bytes from 192.168.192.168: icmp_seq=1 ttl=255 time=1.09 ms
64 bytes from 192.168.192.168: icmp_seq=2 ttl=255 time=0.506 ms
...

The printer has a web interface, and is open on two ports for printing:

$ nmap 192.168.192.168
...
PORT     STATE SERVICE
80/tcp   open  http
515/tcp  open  printer
9100/tcp open  jetdirect

The web interface will let you set different IP settings, so that you can get the printer on your network. If you mess up and can’t connect, then do a factory reset: Hold the button used before, and then reboot the printer.

Using the printer

Epson provides drivers for several platforms, which may fit your use case.

However, these printers do support ESC/POS (See Wikipedia). making it quite accessible without installed drivers.

The printer will immediately print any regular text it receives over Port 9100, line by line:

echo "Hello World" | nc 192.168.192.168 9100

ESC/POS commands allow you to format the text, print barcodes, and cut the paper. A good resource for them is this PDF reference from Epson.

I’ve included a PHP script to produce the ESC/POS commands for the below receipt, showing how to use a few of the supported features:

And the script which created it:

<?php
/* ASCII constants */
const ESC = "\x1b";
const GS="\x1d";
const NUL="\x00";

/* Output an example receipt */
echo ESC."@"; // Reset to defaults
echo ESC."E".chr(1); // Bold
echo "FOO CORP Ltd.\n"; // Company
echo ESC."E".chr(0); // Not Bold
echo ESC."d".chr(1); // Blank line
echo "Receipt for whatever\n"; // Print text
echo ESC."d".chr(4); // 4 Blank lines

/* Bar-code at the end */
echo ESC."a".chr(1); // Centered printing
echo GS."k".chr(4)."987654321".NUL; // Print barcode
echo ESC."d".chr(1); // Blank line
echo "987654321\n"; // Print number
echo GS."V\x41".chr(3); // Cut
exit(0);

This can again be sent to the printer using netcat:

php foo.php | nc 192.168.192.168 9100

Productivity: De-activate social networking on a schedule with cron

Fri, 17 Oct 2014 00:00:00 +0000

Sometimes you need to block out distractions for a set amount of time. One of these is social networking, and it’s fairly early to temporarily break it (all for productivity of course!).

Mac or Linux users can prevent themselves from accessing example.com like this:

echo "127.0.0.1 example.com www.example.com" >> /etc/hosts

This tells the computer that it hosts example.com, so it won’t load it from the internet.

And to compliment this, we have sed, which can delete lines from a file in-place based on a pattern:

sed -i '/example.com/d' /etc/hosts

Replace example.com above with your social network of choice, and have a shot. It will take a few minutes to have an effect, because of open connections and your computer’s DNS cache.

Scheduling it in

Cron is the go-to solution for scheduling any command on Unix. We’ll run this as root, as normal users don’t have permission to edit /etc/hosts:

su
crontab -e

If prompted, select nano as an editor.

If, for example, 6pm - 8pm weekdays is a distraction-free time for you, you would schedule the first command for 18:00 on days 1-5, and the second for 20:00 on days 1-5:

# m h  dom mon dow   command
0 18 * * 1-5 echo "127.0.0.1 example.com www.example.com" >> /etc/hosts
0 20 * * 1-5 sed -i '/example.com/d' /etc/hosts

If you need something more advanced, consider learning how to use squid to manage web traffic.

When you break something …

If you accidentally delete your /etc/hosts while experimenting, you can fetch its contents from /var/lib/dpkg/info/netbase.postinst (source).

How to install JOGL on Debian

Mon, 22 Sep 2014 00:00:00 +0000

JOGL is an OpenGL binding for Java (wiki) which wraps the C OpenGL API. On Debian-based systems, you might want to use the version from the package manager, and here is how to do so.

First, install it:

apt-get install libjogl2-java

Next, you need these JAR files in your build path:

/usr/share/java/jogl2.jar
/usr/share/java/gluegen2-rt.jar

In Eclipse, this is achieved by clicking Project -> Properties -> Java Build Path -> Libraries -> Add External Jar.

(If you don’t have gluegen2-rt.jar, then you need to install the libgluegen2-rt-java package as well).

How to generate professional-quality PDF files from PHP

Thu, 21 Aug 2014 00:00:00 +0000

There are a few ways to go about making PDF files from your PHP web app. Your options are basically:

Put all of your text in a 210mm column and get the user to save it as PDF.
Learn a purpose-built library, such as FPDF (free) or pdflib (proprietary).
Use PHP for generating markup which can be saved to PDF. The markup I’m suggesting is of course LaTeX

This article assumes an intermediate knowledge of both PHP and LaTeX, and that your server is running Linux.

The software mix

PHP is an open-source server package which generates HTML pages, usually based on some sort of dynamic data. It is equally good at (but less well known for) generating other types of markup.

LaTeX is an open source document typesetting system, which will take a markup file in .tex format, and output a printable document, such as a PDF. The engine I will use here is XeLaTeX, because it supports modern trimmings such as Unicode and OpenType fonts.

Naturally, this post will use PHP to populate a .tex file, and then xelatex to create a PDF for the user.

This sounds straightforward enough, but it may not work with all shared hosts. Check your setup before you read on:

Your server needs PHP, with safe mode disabled, so that it can run commands.
This server needs xelatex, or a suitable substitute such as pdflatex.

A bit about markup

We will be working with .tex templates, which will be valid LaTeX files. The basic rules are:

Define a \newcommand for every variable, so that you can compile the document without PHP.
Drop PHP code in comments, which will print out code to override those variables.

So you will end up with code like this:

% Make placeholders visible
\newcommand{\placeholder}[1]{\textbf{$<$ #1 $>$}}

% Defaults for each variable
\newcommand{\test}{\placeholder{Data here}}

% Fill in
% <?php echo "\n" . "\\renewcommand{\\test}{" . LatexTemplate::escape($data['test']) . "}\n"; ?>

Look messy? A multi-line block of PHP is a little easier to follow. This example is from the body of a table, see if you can figure out the syntax:

%<?php                                                                      /*
% */ foreach($data['invoiceItem'] as $invoiceItem) {                        /*
% */    echo "\n" . LatexTemplate::escape($invoiceItem['item']) . " & " .   /*
% */        LatexTemplate::escape($invoiceItem['qty']) . " & " .            /*
% */        LatexTemplate::escape($invoiceItem['price']) . " & " .          /*
% */        LatexTemplate::escape($invoiceItem['total']) . "\\\\\n";        /*
% */ } ?>

So what about this LatexTemplate::escape() business? In LaTeX, just about every symbol seems to be part of the syntax, so it is sadly not very simple to escape.

I have settled on the following series of str_replace() calls to sanitise information for display. It is crude but effective. Generating LaTex is much like generating SQL, HTML or LDIF from your website: it is quite important to make a habit of wrapping every piece of data with a function to prevent users from writing (‘injecting’) arbitrary code into your document:

/**
 * Series of substitutions to sanitise text for use in LaTeX.
 *
 * http://stackoverflow.com/questions/2627135/how-do-i-sanitize-latex-input
 * Target document should \usepackage{textcomp}
 */
public static function escape($text) {
    // Prepare backslash/newline handling
    $text = str_replace("\n", "\\\\", $text); // Rescue newlines
    $text = preg_replace('/[\x00-\x1F\x7F-\xFF]/', '', $text); // Strip all non-printables
    $text = str_replace("\\\\", "\n", $text); // Re-insert newlines and clear \\
    $text = str_replace("\\", "\\\\", $text); // Use double-backslash to signal a backslash in the input (escaped in the final step).

    // Symbols which are used in LaTeX syntax
    $text = str_replace("{", "\\{", $text);
    $text = str_replace("}", "\\}", $text);
    $text = str_replace("$", "\\$", $text);
    $text = str_replace("&", "\\&", $text);
    $text = str_replace("#", "\\#", $text);
    $text = str_replace("^", "\\textasciicircum{}", $text);
    $text = str_replace("_", "\\_", $text);
    $text = str_replace("~", "\\textasciitilde{}", $text);
    $text = str_replace("%", "\\%", $text);

    // Brackets & pipes
    $text = str_replace("<", "\\textless{}", $text);
    $text = str_replace(">", "\\textgreater{}", $text);
    $text = str_replace("|", "\\textbar{}", $text);

    // Quotes
    $text = str_replace("\"", "\\textquotedbl{}", $text);
    $text = str_replace("'", "\\textquotesingle{}", $text);
    $text = str_replace("`", "\\textasciigrave{}", $text);

    // Clean up backslashes from before
    $text = str_replace("\\\\", "\\textbackslash{}", $text); // Substitute backslashes from first step.
    $text = str_replace("\n", "\\\\", trim($text)); // Replace newlines (trim is in case of leading \\)
    return $text;
}

We then have a template which we can include() from PHP, or run xelatex over. Below is minimal.tex, a minimal example of a PHP-latex template in this form:

% This file is a valid PHP file and also a valid LaTeX file
% When processed with LaTeX, it will generate a blank template
% Loading with PHP will fill it with details

\documentclass{article}
% Required for proper escaping
\usepackage{textcomp} % Symbols
\usepackage[T1]{fontenc} % Input format

% Because Unicode etc.
\usepackage{fontspec} % For loading fonts
\setmainfont{Liberation Serif} % Has a lot more symbols than Computer Modern

% Make placeholders visible
\newcommand{\placeholder}[1]{\textbf{$<$ #1 $>$}}

% Defaults for each variable
\newcommand{\test}{\placeholder{Data here}}

% Fill in
% <?php echo "\n" . "\\renewcommand{\\test}{" . LatexTemplate::escape($data['test']) . "}\n"; ?>

\begin{document}
    \section{Data From PHP}
    \test{}
\end{document}

Generate a PDF on the server

Here is where the fun begins. There is no plugin for compiling a LaTeX document, so we need to directly execute the command on a file.

Looks like we need to save the output somewhere then. You would generate your filled-in LaTeX code in a temporary file by doing something like this:

/**
 * Generate a PDF file using xelatex and pass it to the user
 */
public static function download($data, $template_file, $outp_file) {
    // Pre-flight checks
    if(!file_exists($template_file)) {
        throw new Exception("Could not open template");
    }
    if(($f = tempnam(sys_get_temp_dir(), 'tex-')) === false) {
        throw new Exception("Failed to create temporary file");
    }

    $tex_f = $f . ".tex";
    $aux_f = $f . ".aux";
    $log_f = $f . ".log";
    $pdf_f = $f . ".pdf";

    // Perform substitution of variables
    ob_start();
    include($template_file);
    file_put_contents($tex_f, ob_get_clean());

The next step is to execute your engine of choice on the output files:

    // Run xelatex (Used because of native unicode and TTF font support)
    $cmd = sprintf("xelatex -interaction nonstopmode -halt-on-error %s",
            escapeshellarg($tex_f));
    chdir(sys_get_temp_dir());
    exec($cmd, $foo, $ret);

Once this is done, you can delete a lot of the extra LaTeX files, and check if a .pdf appeared as expected:

    // No need for these files anymore
    @unlink($tex_f);
    @unlink($aux_f);
    @unlink($log_f);

    // Test here
    if(!file_exists($pdf_f)) {
        @unlink($f);
        throw new Exception("Output was not generated and latex returned: $ret.");
    }

And of course, send the completed file back via HTTP:

    // Send through output
    $fp = fopen($pdf_f, 'rb');
    header('Content-Type: application/pdf');
    header('Content-Disposition: attachment; filename="' . $outp_file . '"' );
    header('Content-Length: ' . filesize($pdf_f));
    fpassthru($fp);

    // Final cleanup
    @unlink($pdf_f);
    @unlink($f);
}

The static functions escape($text) and download($data, $template_file, $outp_file) are together placed into a class called LatexTemplate for the remainder of the example (complete file on GitHub).

Putting it all together

With the library and template, it is quite easy to set up a PHP script which triggers the above code:

<?php
require_once('../LatexTemplate.php');

$test = "";
if(isset($_GET['t'])) {
    // Make the LaTeX file and send it through
    $test = $_GET['t'];
    if($test =="") {
        // Test pattern to show symbol handling
        for($i = 0; $i < 256; $i++) {
            $test .= chr($i) . " . ";
        }
    }

    try {
        LatexTemplate::download(array('test' => $test), 'minimal.tex', 'foobar.pdf');
    } catch(Exception $e) {
        echo $e -> getMessage();
    }

}
?>
<html>
<head>
<title>LaTeX test (minimal)</title>
</head>
</html>
<body>
    <p>Enter some text to be placed on the output:</p>
    <form>
        <input type="text" name="t" /><input type="submit" value="Generate" />
    </form>
</body>
</html>

The above code will show a form, which asks for input. When it gets some text, it will generate a PDF containing the text. If no text is given, it will output an ASCII table, simply to show that it can handle the symbols. Once the template code is hidden away, this powerful technique is easily applied.

Results

This is only a minimal example. In any real application, your template would be more extensive.

Compiling the template directly creates this PDF, which marks out placeholders for fields which will be populated by the user:

From the web, a form is presented to fill this single field:

Which results in this PDF containing the user data:

Tips

The text after \end{document} is not even parsed in latex. Use this area to write <?php ?> with fewer constraints.
Consult the github repository for this code to see the complete example.
Comment out the line @unlink($tex_f); of you want to preserve (for debugging, etc) the generated markup.

How to query Microsoft SQL Server from PHP

Tue, 10 Jun 2014 00:00:00 +0000

This post is for anybody who runs a GNU/Linux server and needs to query a MSSQL database. This setup will work on Debian and its relatives. As it’s a dense mix of technologies, so I’ve included all of the details which worked for me.

An obvious note: Microsoft SQL is not an ideal choice of database to pair with a GNU/Linux server, but may be acceptable if you are writing something which needs to import some data from external application which has a better reason to be using it.

A command-line alternative to this setup would be sqsh, which will let you running scheduled queries without PHP, if that’s what you’re after.

Prerequisites

Once you have PHP, the required libraries can be fetched with:

sudo apt-get install unixodbc php5-odbc tdsodbc

MSSQL is accessed with the FreeTDS driver. Once the above packages are installed, you need to tell ODBC where to find this driver, by adding the following block to /etc/odbcinst.ini:

[FreeTDS]
Description=MSSQL DB
Driver=/usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
UsageCount=1

The path is different on platforms other than amd64. Check the file list for the tdsodbc package on your architecture if you lose track of the path.

The next step requires that you know the database server address, version, and database name. Add a block for your database to the end of `/etc/odbc.ini`:

[foodb]
Driver = FreeTDS
Description = Foo Database
Trace = Yes
TraceFile = /tmp/sql.log
ForceTrace = yes
Server = 10.x.x.x
Port = 1433
Database = FooDB
TDS_Version = 8.0

Experiment with TDS_Version values if you have issues connecting. Different versions of MSSQL require different values. The name of the data source (‘foodb’), the Database, Description and Server are all bogus values which you will need to fill.

An example

For new PHP scripts, database queries are invariably performed via PHP Data Objects (PDO). The good news is, it is easy to use it with MSSQL with PDO.

The below file takes a query on standard input, throws it at the database, and returns the result as comma-separated values.

Save this as query.php and fill in your data source (‘odbc:foodb’ here), username, and password.

#!/usr/bin/env php
<?php
$query = file_get_contents("php://stdin");
$user = 'baz;
$pass = 'super secret password here';

$dbh = new PDO('odbc:foodb', $user, $pass);
$dbh -> setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$sth = $dbh -> prepare($query);
$sth -> execute();
$results = $sth -> fetchAll(PDO::FETCH_ASSOC);

/* Quick exit if there are no rows */
if(count($results) == 0) {
	return 0;
}
$f = fopen("php://stdout", "w");

/* Output header */
$a = $results[0];
$header = array();
foreach($a as $key => $val) {
	$header[] = $key;
}
fputcsv($f, $header);

/* Output rows */
foreach($results as $result) {
	fputcsv($f, $result);
}

fclose($f);

To test the new script, first make it executable:

chmod +x query.php

To run a simple test query:

echo "SELECT Name from sys.tables ORDER BY Name;" | ./query.php

Refining the setup

The above script has some cool features: It’s short, actually useful, and it sets PDO::ERRMODE_EXCEPTION. This means that if something breaks, it will fail loudly and tell you why. Hopefully, if your setup has issues, you can track down the cause with the error, and solve it by scrolling through this how-to again.

If you encounter a MSSQL database with an unknown schema, then you may want to list all rows and columns. This is achieved with:

SELECT tables.name AS tbl, columns.name AS col FROM sys.columns JOIN sys.tables ON columns.object_id = tables.object_id ORDER BY tbl, col;

The catch

I’ve run into some bizarre limitations using this. Be sure to run it on a server which you can update at the drop of a hat.

A mini-list of issues I’ve seen with this combination of software (no sources as I never tracked down the causes):

An old version of the driver would segfault PHP, apparently when non-ASCII content appeared in a text field.
Substituting non-text values fails in the version I am using, although Google suggests that updating the ODBC driver fixes this.

A Yes/No filtering tool for images

Mon, 09 Jun 2014 00:00:00 +0000

While collecting large numbers of screen captures for writing documentation, I noticed that it takes far too long to filter out the junk.

To fix this, I coded up yn (source code at mike42/yn on GitHub). Given a list of images, it will display each one to the user, which they then include by pressing Y (yes) or exclude by pressing N (no) (hence the name yn).

The images which are selected are saved as a list, so that a script can continue by processing them in some way. This could be by copying them elsewhere, or generating a document with spaces to caption them. The example in the README.md does both of these.

The code to this early version is very simple, which makes it a good example of a simple OpenCV C++ app. I’ve stepped through it below.

The code

C++ does console input and output via the iostream library:

#include <iostream>

To use OpenCV with a GUI, you need these headers. You then need to add them to your include path, and link to OpenCV for the program to compile:

#include <cv.h>
#include <highgui.h>

This line simly tells the compiler that when we say cout (console out), we mean std::cout.

using namespace std;

The only OpenCV code is in the below function. The steps are:

load an image from a file via imread(),
display it in a window with imshow(),
capture the next keypress with waitKey(), then
delete the window with destroyWindow()

/**
 * Load a file, and wait for the user to press a key.
 *
 * If the pressed key is 'y', print the filename.
 */
bool yn(string fn) {
    cv::Mat img;
    char key = 0;

    img = cv::imread(fn);
    if(!img.data) {
        cerr << "Failed to load " << fn << endl;
    } else {
        cv::namedWindow(fn);
        cv::imshow(fn, img);
        key = cv::waitKey(0);
        cv::destroyWindow(fn);
        if(key == 'y') {
            cout << fn << endl; // 'y' pressed
        } else if(key == 0x1B) {
            return false; // ESC pressed
        }
    }
    return true;
}

So with that library usage out of the way, all we need to do is get the list of files to check, and stop popping up windows when the user has pressed the escape key.

/**
 * Get list of files from command-line arguments and display them in turn
 */
int main(int argc, char** argv) {
    int i;

    /* Command-line arguments given */
    for(i = 1; i < argc; i++) {
        if(yn(string(argv[i])) == false) {
            cerr << "Quitting.\n";
            break;
        }
    }

    return 0;
}

The delay between each image appearing could be removed by loading them in a separate thread, which I may do in a future version.

Crash course: Run Windows on desktop Linux

Mon, 09 Jun 2014 00:00:00 +0000

Sometimes, you need to use a tricky windows-only program on a Linux desktop. If you have a Windows install disk and licence at your disposal, then this post will show you how to get a Windows environment running without dual-booting.

The host here is a Debian box, and the guest is running Windows 7. The instructions will work with slight modifications for any mix of Linux and Windows

On the desktop, some things are not as important as the server world. Some things are excluded for simplicity: network bridging, para-virtualised disks, migration between hosts, and disk replication.

Software setup

Everything required from the host machine can be pulled in via Debian’s qemu-kvm package.

sudo apt-get install qemu-kvm

Virtual machine setup

Prepare a disk image for Windows. The qcow2 format is suggested for the desktop as it will not expand the file to the full size until the guest uses the space:

qemu-img create -f qcow2 windows.img 30G

Launch the Windows installer in KVM with a command that looks something like this:

kvm -hda windows.img --cdrom windows-install-disc.iso -vga std -localtime -net nic,model=ne2k_pci -m 2048

Note the -m option is the number of megabytes of RAM to allocate. You can set it a little lower if you don’t have much to spare, but if it’s too low you’ll get this screen:

If you have a physical disk but no .iso of it, then using the disk drive via --cdrom /dev/cdrom will work.

Windows installer

I’ve included a walkthrough here of what the Windows 7 installer will prompt for, since it does work a bit differently to a typical Linux installer.

Select language, accept the licence agreement, choose the target disk, and let the files copy:

After reboot, enter the user details, licence key, update settings and timezone:

After another reboot, Windows is installed in the virtual machine:

Post-install tasks

The guest you have now will only run at standard VGA resolutions, and will probably not be networked. This section will show you how to fix that.

Network drivers

You will notice that we are launching the guest with -net nic,model=virtio. This means that we are using a virtual network card, rather than simulating a real one. You need to fetch a disk image with the latest binary drivers, which are best tracked down on linux-kvm.org via google.

Once you have the disk image in the same folder as your virtual machine, shut down and launch it with a CD:

kvm -hda windows.img --cdrom virtio-win-0.1-74.iso -vga std -localtime -net nic,model=ne2k_pci -m 2048

Under “My Computer” track down the “Device Manager”, find your network card, and tell Windows to update the drivers. You can then point it to the CDROM’s “Win7” directory (or other one, if you are installing a different guest). After the network adapter is recognised, you will be connected automatically.

Note that you are using “user-mode” networking, which means you are on a simulated subnet, and can only communicate via TCP and UDP (ie, ping will not work). This can be a little slow, but will work on a laptop host whether plugged in or running on Wi-Fi.

Remote desktop

The screen resolution and mouse sensitivity on the VM console is unlikely to match your host computer. The best way around this is not to fix with settings and drivers, but to enable remote desktop and log in via the network. This lets you use an arbitrary screen size, and match mouse speed to the host.

When run on localhost, so it is neigher laggy nor a security issue, and makes it possible to leverage all RDP features.

First, in the guest, enable remote desktop using these Microsoft instructions.

Then shut down and boot up with the extra -redir tcp:3389::3389 option:

kvm -hda windows.img -vga std -localtime -net nic,model=ne2k_pci -m 2048 -redir tcp:3389::3389

On the host, wait for the guest to boot, then use rdesktop to log in:

rdestkop localhost

One this works, you can shut down and boot with the extra -nographic option to leave remote desktop as the only way to interact with the guest:

kvm -hda windows.img -vga std -localtime -net nic,model=ne2k_pci -m 2048 -nographic -redir tcp:3389::3389

The rdesktop tool supports sound, file and printer redirection. It can also run fullscreen when launched with -f. All the details are in man rdesktop

If you end up using the guest operating system more, it is worth investigating USB-redirection for any peripherals (printers or mobile phones), running a virtual sound card, or running SAMBA on the host to share files.

Including git commit history in a LaTeX document

Sun, 27 Apr 2014 00:00:00 +0000

LaTeX is a document typesetting system. It allows you to mark up a document as a plain text file, so lends itself well to being used within a source-code management program such as git. I found a need to include a changelog in a document, and found this brilliant blog article by Jerel Unruh which creates a HTML log using git log.

For inclusion within a Makefile target, I re-worked this into a shell script, changes.sh, which will also auto-detect URL’s for any Github-hosted repository:

#!/bin/bash
# Find remote URL for hashes (designed for GitHub-hosted projects)
origin=`git config remote.origin.url`
base=`dirname "$origin"`/`basename "$origin" .git`

# Output LaTeX table in chronological order
echo "\\begin{tabular}{l l l}\\textbf{Detail} & \\textbf{Author} & \\textbf{Description}\\\\\\hline"
git log --pretty=format:"\\href{$base/commit/%H}{%h} & %an & %s\\\\\\hline" --reverse
echo "\end{tabular}"

The above script outputs a LaTeX table, including a hyperlink to each commit. In the pre-amble, you need:

\usepackage{hyperref}

In your Makefile, you might add something like:

changelog:
	./changes.sh > changelog.tex

The resulting changelog can then be included in the document via:

\input{changelog.tex}

QJoyPad update

Sun, 27 Apr 2014 00:00:00 +0000

As of April, there is now a qjoypad package available in the official Debian repository, see https://packages.debian.org/unstable/main/qjoypad. This program allows you to map joystick events to keyboard and mouse actions. This means that you could use, for example, a USB gamepad to browse the web.

I have previously blogged about the lack of usable, packaged programs to do this, and highly recommend getting your hands on a USB gamepad and giving this a try. Depending on your distribution, you will soon be able to just run:

sudo apt-get install qjoypad

Then connect the gamepad and type:

qjoypad

You then get a window which shows you which buttons you can map:

Clicking a button prompts you to press a key on your keyboard or select a mouse button, and each profile can be saved and named for later use.

As far as small utility programs go, I now rely on this about as much as the Gnome calculator.

An alternative: AntiMicro

I’ll also give a mention to AntiMicro, which has similar goals, and looks very promising.

As a bit of background, there is a limitation in QJoyPad around adding modifier keys: For example, you simply can’t map a joystick button to <Ctrl>+Q, making it tricky to use with most applications. The author of AntiMicro, Travis, sent me an email back in September to point out that his program both solves this issue, and has a Debian package available for download.

Whilst I was quite impressed with the functionality in AntiMicro, I found it a lot more complex to set up. I think this would be more useful for gamers, as mouse acceleration and key repeats are overkill for my use case (browsing the web and controlling VLC, MythTV, etc).

So, if you are a power user, then I suggest you give it a shot. Power users with apt-get may also consider joy2key, which is the best non-GUI tool for this, and is available in the Debian repositories.

Winning 2048 game with key-mashing?

Mon, 24 Mar 2014 00:00:00 +0000

This new, simple, addictive game is out, called 2048. You need to slide two numbers together, resulting in a bigger number, in ever-increasing powers of two. You get 2048, and you win.

I noticed that somebody already wrote neat AI for it, although it does run quite slowly. But then I also noticed a friend mashing keys in a simple pattern, and thought I should test whether this was more effective. The answer: it kinda is.

At least in the first half of the game, a simple key-mashing pattern is a much faster way to build high numbers. The PHP script below will usually get to 512 without much trouble, but rarely to 1024. I would suggest running it for a while, and then taking over with some strategy.

The script

This script produces commands which can be piped to xte for some automatic key-mashing on GNU / Linux. Save as 2048.php

#!/usr/bin/env php
mouseclick 1
<?php
for($i = 0; $i < 10; $i++) {
	move("Left", 1);
	move("Right", 1);
}

while(1) {
	move("Down", 1);
	move("Left", 1);
	move("Down", 1);
	move("Right", 1);
}

function move($dir, $count) {
	for($i = 0; $i < $count; $i++) {
		echo "key $dir\nsleep 0.025\n";
		usleep(25000);
	}
}

And then in a terminal, run this then click over to a 2048 game:

sleep 3; php 2048.php | xte

I hope this helps somebody to beat this game!

Importing myki data into GnuCash

Wed, 05 Mar 2014 00:00:00 +0000

GnuCash, despite all its bugs, is one of the best open source accounting programs going around.

Since it is not hard to export the history from a myki (public transport) card, I figured that it would be nice to track it as an account in GnuCash.

Transforming the data

The data on the statements is not quite suitable for an accounting program. Some changes that need to be done are:

Conversion of date format to YYYY-MM-DD.
Single-field descriptions for each transaction (“Top up myki money”, “Travel zone 2”), rather than multiple fields.
Entries which have a 0.00 cost need to be removed.

Once you have a CSV of your data, the script below will filter it to be ready for import:

#!/usr/bin/env php
<?php
/* Dump Myki CSV file to a file suitable for gnucash
    (c) Michael Billington < michael.billington@gmail.com >
    MIT Licence */
$in = fopen("php://stdin", "r");
$out = fopen("php://stdout", "w");
$err = fopen("php://stdout", "w");
$lc = 0;

while($line = fgets($in)) {
    $lc++;
    $a = str_getcsv($line, ',', '"');
    if(count($a) == 8) {
        $date = implode("-", array_reverse(explode("/", substr($a[0], 0, strpos($a[0], " ")))));
        $credit = $a[5] == "-" ? "" : $a[5];
        if($credit != "") { // Probably a top-up or reimbursement
            $description = $a[1];
        } else if($a[3] == "-") { // Probably buying myki pass
            $description = trim($a[1], "*");
        } else { // Probably travel charges
            $description = "Travel: " . $a[2] . ", Zone " . $a[3];
        }
        $debit = $a[6] == "-" ? "" : $a[6];
        $balance = $a[7] == "-" ? "" : $a[7];
        if($balance != "") { // Ignore non-charge entries
            fputcsv($out, array($date, $description, $credit, $debit, $balance));
        }
    }
}
fclose($in);
fclose($out);
fclose($err);

Why would you track it as an account?

Here are a few of reasons why a public transport card is account-like enough to put into GnuCash:

Money is not spent until you touch on and off — note that no GST is payable until the card balance is used. This means that if you record a top-up under Expenses, it’s not quite correct.
You can cancel a myki and have its balance moved to another card.
A card can be handed in, and the balance paid back to you as cash.

Update 2014-05-04: All of these myki-related scripts are now available at mike42/myki-import on GitHub.

Posts on Mike's Software Blog

Farewell, wordpress!

Why Wordpress had to go

Editing experience

The AI elephant in the room

Making the switch

Preserving URLs

Exporting the content

The 80/20 rule

Remaining issues

Dark mode and transparency

Comments

Subscribers

Analytics

Search

Wrap-up

Switching between VFIO-enabled virtual machines

The problem

Creating an API

Web UI

GNOME extension

Result and next steps

Going all-in on GPU passthrough for software development

Background and previous setup

Hardware

Inspecting IOMMU groups

Graphics

Audio output

Audio input

USB

File share

Power management

Recap of unexpected issues

Wrap-up and future ideas

Building a 1U quiet NAS

The problem

Parts list

Closer look at the Athena Power RM1UC138

Closer look at the Topton N6005 Mini ITX

Custom power supply adapter plate

Custom fan controller

Final assembly

Completed build

Software

Wrap-up

Controlling computer fans with a microcontroller

The problem

First attempt

But can it control fans?

Porting to the ATtiny85

Writing fuses

Writing code

Updates

Next steps & lessons learned

Converting my 65C816 computer project to 3.3 V

Re-cap: Why 3.3 V is useful

ROM

UART

Clock, reset and address decode

SD card

Modifications

Wrap-up

Porting the Amiga bouncing ball demo to the NES

How it works

Pre-rendered 3D

NES implmentation

Physics and sound

Development setup

Wrap-up

Let’s make a bootloader – Part 2

Getting more data

Higher memory addresses

New bootloader

A placeholder kernel

Result

Speed increase

A debugging detour

Alternative boot method

Wrap-up

Let's make a bootloader - Part 1