Switching between VFIO-enabled virtual machines

Posted on

For the past few months, I’ve been using a setup for software development where I pass the mouse, keyboard and graphics card to a virtual machine via Linux VFIO. The setup is described in detail in its own blog post.

In short though, the idea is to provision a separate virtual machine for each project, so that I can install whatever tools I need to solve the problem at hand, with no real consequences if I break my install.

The problem

I have multiple virtual machines which are configured to use the same hardware, so I can only run one at a time. Before making the changes outlined in this blog post, the method I used to switch to a different VM was unwieldy: I needed is to shut down the current VM, switch monitor inputs to the host graphics, start a different VM, then switch monitor inputs back the the VM graphics.

I intend to do all of my work in VM’s, so my basic goal was to make a ‘switch VM’ mechanism available from within a VM, to remove any need to interact with the host operating system directly.

Creating an API

My plan for this was to run an agent on the host, which will accept a request to switch VM’s. It would then shut down the current VM, and start up a different one instead.

This was the first time I tried using the libvirt Python bindings before, and I was able to quickly make a script which toggled between two VM’s, which are nested within the VM I’m using for development, because of course they are.

Toggling which VM is active, using the Python libvirt API in PyCharm.

I then extended this to use an orderly shutdown, and to work with an arbitrary number of VM’s. CirrOS was not completing its boot in this environment, so I switched the test VM’s to Debian, and also passed through a USB device to each of them so that I would get an error if I ever tried to boot both VM’s simultaneously.

Last, I exposed this over HTTP as a simple API.

For the first iteration, I installed this on the host, and placed some scripts in a folder which is shared between all of the VM’s. The scripts use curl commands to call the API. To trigger a VM switch (or a shutdown of the host), I could go to a folder, and run one of these scripts.

Web UI

With that proof-of-concept out of the way, I made a web interface, so that I can use this instead of some bash scripts in a folder. This is a very simple Angular app.

GNOME extension

For the most part, I’m working on Debian using the GNOME desktop environment, which gives me some options for integrating this further into my desktop.

I first considered making a search provider, but settled instead for making a simple standalone extension to display an indicator instead, based on the example app shown running here.

As if nested virtualization was not mind-bending enough, I started testing this using a GNOME session within a GNOME session.

The extension is written in JavaScript, though the JavaScript API for GNOME extensions was very unfamiliar to me. There some obvious quality issues in the code, but it’s not being published to a repository, and it fires off the correct HTTP requests, so I’ll call it a success.

Deploying a GNOME extension manually is as simple as placing some files my home directory.

Result and next steps

I’ve bookmarked this URL in each of my VM’s.

And I’ve also installed this GNOME extension on the VM that runs on boot.

The idea of this setup is to have one virtual machine per project, so a good next step would be to make it easier to kick of the install process.

I’ve put the code for this project online at mike42/vfio-vm-switcher on GitHub.

Controlling computer fans with a microcontroller

Posted on

I’m currently working on building a small computer, and want to add some 4-pin computer fans, running quietly at a fixed speed.

This blog post is just a quick set of notes from prototyping, since it covers a few topics which I haven’t written about before.

The problem

The speed of 4-pin computer fans can be controlled by varying the duty cycle on a 25 KHz PWM signal. This signal normally comes from a 4-pin case fan header on the motherboard, which will not be available in this situation. Rather than run the fans at 100%, I’m going to try to generate my own PWM signal.

Two main considerations led me to choose to use a microcontroller for this:

  • I’ll need some way to adjust the PWM duty cycle after building the circuit, because I don’t know which value will give the best trade-off between airflow and noise yet.
  • Fans need a higher PWM duty cycle to start than they do to run. If I want to run the fans at a very low speed, then I’ll need them to ramp up on start-up, before slowing to the configured value.

It’s worth noting that I’m a complete beginner with microcontrollers. I’ve run some example programs on the Raspberry Pi Pico, but that’s it.

First attempt

I already had MicroPython running on my Raspberry Pi Pico, so that was my starting point.

For development environment, I installed the MicroPython plugin for PyCharm, since I use JetBrains IDE’s already for programming. Most guides for beginners suggest using Thonny.

There are introductory examples on GitHub at raspberrypi/pico-micropython-examples which showed me everything I needed to know. I was able to combine an ADC example and a PWM example within a few minutes.

import time
from machine import Pin, PWM, ADC, Timer

level_input = ADC(0)
pwm_output = PWM(Pin(27))
timer = Timer()

# 25 KHz
pwm_output.freq(25000)


def update_pwm(timer):
    """ Update PWM duty cycle based on ADC input """
    duty = level_input.read_u16()
    pwm_output.duty_u16(duty)


# Start with 50% duty cycle for 2 seconds (to start fan)
pwm_output.duty_u16(32768)
time.sleep(2)

# Update from ADC input after that
timer.init(mode=Timer.PERIODIC, period=100, callback=update_pwm)

On my oscilloscope, I could confirm that the PWM signal had a 25 KHz frequency, and that the code was adjusting the duty cycle as expected. When the analog input (above) is set to a high value, the PWM signal has a high duty cycle.

When set to a low value, the PWM signal has a low duty cycle.

But can it control fans?

I wired up a PC fan to 12 V power, and also sent it this PWM signal, but the speed didn’t change. This was expected, since I knew that I would most likely need to convert the 3.3 V signal up to 5 V.

I ran it through a 74LS04 hex inverter with VCC at 5 V, which did the trick. I could then adjust a potentiometer, and the fan would speed up or slow down.

I captured the breadboard setup in Fritzing. Just note that there are floating inputs on the 74LS04 chip (not generally a good idea) and that the part is incorrectly labelled 74HC04, when the actual part I used was a 74LS04.

This shows a working setup, but it’s got more components than I would like. I decided to implement it a second time, on a simpler micro-controller which can work at 5 V directly.

Porting to the ATtiny85

For a second attempt, I tried using an ATtiny85. This uses the AVR architecture (of Arduino fame), which I’ve never looked at before.

This chip is small, widely available, and can run at 5 V. I can also program it using the TL-866II+ rather than investing into the ecosystem with Arduino development boards or programmers.

I found GPL-licensed code by Marcelo Aquino for controlling 4-wire fans.

After a few false starts trying to compile manually, I followed this guide to get the ATtiny85 ‘board’ definition loaded into the Arduino IDE.

From there I was able to build and get an intel hex file, using the “Export compiled binary” feature.

Writing fuses

The code assumes an 8 MHz clock. The Attiny85 ships from the factory with a “divide by 8” fuse active. This needs to be turned off, otherwise the clock will be 1 MHz. This involves setting some magic values, separate to the program code.

I found these values using a fuse calculator.

The factory default for this chip is:

lfuse 0x62, hfuse 0xdf, efuse 0xff.

To disable the divide-by-8 clock but leave all other values default, this needs to be:

lfuse 0xe2, hfuse 0xdf, efuse 0xff.

I am using the minipro open source tool to program the chip, via a TL-866II+ programmer. First, to get the fuses:

$ minipro -p ATTINY85@DIP8 -r fuses.txt -c config
Found TL866II+ 04.2.126 (0x27e)
Warning: Firmware is out of date.
  Expected  04.2.132 (0x284)
  Found     04.2.126 (0x27e)
Chip ID: 0x1E930B  OK
Reading config... 0.00Sec  OK

This returns the values expected in a text file.

lfuse = 0x62
hfuse = 0xdf
efuse = 0x00
lock = 0xff

I then set lfuse = 0xe2, and wrote the values back with this command.

$ minipro -p ATTINY85@DIP8 -w fuses.txt -c config
Found TL866II+ 04.2.126 (0x27e)
Warning: Firmware is out of date.
  Expected  04.2.132 (0x284)
  Found     04.2.126 (0x27e)
Chip ID: 0x1E930B  OK
Writing fuses... 0.01Sec  OK
Writing lock bits... 0.01Sec  OK

Writing code

Now the micro-controller is ready to accept the exported binary containing the program.

minipro -s -p ATTINY85@DIP8 -w sketch_pwm.ino.tiny8.hex -f ihex
Found TL866II+ 04.2.126 (0x27e)
Warning: Firmware is out of date.
  Expected  04.2.132 (0x284)
  Found     04.2.126 (0x27e)
Chip ID: 0x1E930B  OK
Found Intel hex file.
Erasing... 0.01Sec OK
Writing Code...  1.09Sec  OK
Reading Code...  0.45Sec  OK
Verification OK

With the chip fully programmed, I wired it up on a breadboard with 5 V power.

Checking the output again, the signal was the correct amplitude this time, but the frequency does move around a bit. This is likely because I’m using the internal RC timing on the chip, which is not very accurate. My understanding is that anything near 25 KHz will work fine.

Updates

I made only one change to Marcelo’s code, which is to spin the fan at 50% for a few seconds before using the potentiometer-set value. This is to avoid any issues where the fans fail to start because I’ve set them to run at a very low PWM value.

/*
 *                         ATtiny85
 *                      -------u-------
 *  RST - A0 - (D 5) --| 1 PB5   VCC 8 |-- +5V
 *                     |               |
 *        A3 - (D 3) --| 2 PB3   PB2 7 |-- (D 2) - A1  --> 10K Potentiometer
 *                     |               | 
 *        A2 - (D 4) --| 3 PB4   PB1 6 |-- (D 1) - PWM --> Fan Blue wire
 *                     |               |      
 *              Gnd ---| 4 GND   PB0 5 |-- (D 0) - PWM --> Disabled
 *                     -----------------
 */

// normal delay() won't work anymore because we are changing Timer1 behavior
// Adds delay_ms and delay_us functions
#include <util/delay.h>    // Adds delay_ms and delay_us functions

// Clock at 8mHz
#define F_CPU 8000000  // This is used by delay.h library

const int PWMPin = 1;  // Only works with Pin 1(PB1)
const int PotPin = A1;

void setup()
{
  pinMode(PWMPin, OUTPUT);
  // Phase Correct PWM Mode, no Prescaler
  // PWM on Pin 1(PB1), Pin 0(PB0) disabled
  // 8Mhz / 160 / 2 = 25Khz
  TCCR0A = _BV(COM0B1) | _BV(WGM00);
  TCCR0B = _BV(WGM02) | _BV(CS00); 
  // Set TOP and initialize duty cycle to 50%
  OCR0A = 160;  // TOP - DO NOT CHANGE, SETS PWM PULSE RATE
  OCR0B = 80; // duty cycle for Pin 1(PB1)
  // initial bring-up: leave at 50% for 4 seconds
  _delay_ms(4000);
}

void loop()
{
  int in, out;
  // follow potentiometer-set speed from there
  in = analogRead(PotPin);
  out = map(in, 0, 1023, 0, 160);
  OCR0B = out;
  _delay_ms(200);
}

The wiring of the breadboard is shown below. The capacitor is 0.1 µF for decoupling.

This is far more compact than the Raspberry Pi Pico prototype. I could also miniaturise it further by simply using surface-mount versions of the same components, where using an RP2040 microcontroller from the Pico directly on a custom board would incur some design effort.

Next steps & lessons learned

Although this project is simple, I had to learn quite a few things to prototype it successfully. Using a generic chip programmer like the TL-866II+ appears to be uncommon in the AVR world, and most online guides instead suggest repurposing an Arduino development board or using an Arduino ICSP programmer to program the Attiny85. I was glad to confirm that I could use my existing hardware instead of buying ecosystem-specific items which I would have no other use for. I find the development experience to be far better with the Raspberry Pi Pico, and that’s what I would be choosing for a more complex project.

I also captured the breadboard wiring in Fritzing for this blog post. The diagrams are clearer than a photo of a breadboard, but I’m not confident that they communicate information about the circuit as well as alternative approaches. For future posts, I’ll return to using KiCad EDA for schematic capture, unless there is some reason to highlight the physical layout of a breadboard prototype.

As a next step, I’ll be building a simple break-out PCB for a specific computer case to power the fans and supply a PWM signal, based on the ATtiny85 prototype shown here.

How to export a Maildir email inbox as a CSV file

Posted on

I’ve often thought it would be useful to have an “Export as CSV” button on my inbox, and recently had an excuse to implement one.

I needed to parse the Subject of each email coming from an automated process, but the inbox was in the Maildir format, and I needed something more useful for data interchange.

So I wrote a utility, mail2csv, to export the Maildir as CSV file, which many existing tools can work with. It produces one row per email, and one column per email header:

The basic usage is:

$ mail2csv example/
Date,Subject,From
"Wed, 16 May 2018 20:05:16 +0000",An email,Bob <bob@example.com>
"Wed, 16 May 2018 20:07:52 +0000",Also an email,Alice <alice@example.com>

You can select any header with the --headers field, which accepts globs:

$ mail2csv example/ --headers Message-ID Date Subject
Message-ID,Date,Subject
<89125@example.com>,"Wed, 16 May 2018 20:05:16 +0000",An email
<180c6@example.com>,"Wed, 16 May 2018 20:07:52 +0000",Also an email

If you’re not sure which headers your emails have, then you can make a CSV with all the headers, and just read the first line:

$ mail2csv  example/ --headers '*' | head -n1

You can find the Python code for this tool on GitHub here.

Make better CLI progress bars with Unicode block characters

Posted on

As a programmer, you might add a progress bar so that the user has feedback while they wait for a slow task.

If you are writing a console (CLI) application, then you need to make your progress bars from text. A good command-line progress bar should update in small increments, like this example:

This uses Unicode Block Elements to give the progress bar a higher resolution.

Code Character
U+2588 █
U+2589 ▉
U+258A ▊
U+258B ▋
U+258C ▌
U+258D ▍
U+258E ▎
U+258F ▏

A lot of applications use plain ASCII in their progress bars. The progress bar in wget, for example, uses ===> characters only, like this:

Progress bars made from ASCII characters like = and # signs are very common, most likely because of the historical portability issues around non-ASCII text. Nowadays, UTF-8 support is ubiquitous, and it’s pointless to adhere to such limitations.

Example: Better progress bars in python

The animation at the top of this blog post is a simple python script.

import sys
import time

from progress_bar import ProgressBar

"""
Example usage of ProgressBar class
"""
print("Doing work\n")

with ProgressBar(sys.stdout) as progress:
    for i in range(0,800):
        progress.update(i / 800)
        time.sleep(0.05)
print("\nDone.\n");

The script progress_bars.py, written for Python 3, contains a class that allows the progress bar to be created and drawn on different types of terminals.

import abc
import math
import shutil
import sys
import time

from typing import TextIO

"""
Produce progress bar with ANSI code output.
"""
class ProgressBar(object):
    def __init__(self, target: TextIO = sys.stdout):
        self._target = target
        self._text_only = not self._target.isatty()
        self._update_width()

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        if(exc_type is None):
            # Set to 100% for neatness, if no exception is thrown
            self.update(1.0)
        if not self._text_only:
            # ANSI-output should be rounded off with a newline
            self._target.write('\n')
        self._target.flush()
        pass

    def _update_width(self):
        self._width, _ = shutil.get_terminal_size((80, 20))

    def update(self, progress : float):
        # Update width in case of resize
        self._update_width()
        # Progress bar itself
        if self._width < 12:
            # No label in excessively small terminal
            percent_str = ''
            progress_bar_str = ProgressBar.progress_bar_str(progress, self._width - 2)
        elif self._width < 40:
            # No padding at smaller size
            percent_str = "{:6.2f} %".format(progress * 100)
            progress_bar_str = ProgressBar.progress_bar_str(progress, self._width - 11) + ' '
        else:
            # Standard progress bar with padding and label
            percent_str = "{:6.2f} %".format(progress * 100) + "  "
            progress_bar_str = " " * 5 + ProgressBar.progress_bar_str(progress, self._width - 21)
        # Write output
        if self._text_only:
            self._target.write(progress_bar_str + percent_str + '\n')
            self._target.flush()
        else:
            self._target.write('\033[G' + progress_bar_str + percent_str)
            self._target.flush()

    @staticmethod
    def progress_bar_str(progress : float, width : int):
        # 0 <= progress <= 1
        progress = min(1, max(0, progress))
        whole_width = math.floor(progress * width)
        remainder_width = (progress * width) % 1
        part_width = math.floor(remainder_width * 8)
        part_char = [" ", "▏", "▎", "▍", "▌", "▋", "▊", "▉"][part_width]
        if (width - whole_width - 1) < 0:
          part_char = ""
        line = "[" + "█" * whole_width + part_char + " " * (width - whole_width - 1) + "]"
        return line

Aside from the use of box drawing characters, this script includes a few other things which a good progress bar should implement:

  • Resize the progress bar when you resize the terminal
  • Simplify the progress bar on very small terminals
  • Don’t print ANSI terminal codes if the script is not connected to a terminal
  • Round off to 100% once the use of the progress bar completes without error

After writing this, I discovered that the progress pypi package can also use these box characters, so I haven’t packaged this code up. I haven’t used progress before, but you might like to evaluate it for your own applications.