How to query Microsoft SQL Server from PHP

This post is for anybody who runs a GNU/Linux server and needs to query a MSSQL database. This setup will work on Debian and its relatives. As it’s a dense mix of technologies, so I’ve included all of the details which worked for me.

An obvious note: Microsoft SQL is not an ideal choice of database to pair with a GNU/Linux server, but may be acceptable if you are writing something which needs to import some data from external application which has a better reason to be using it.

A command-line alternative to this setup would be sqsh, which will let you running scheduled queries without PHP, if that’s what you’re after.

Prerequisites

Once you have PHP, the required libraries can be fetched with:

sudo apt-get install unixodbc php5-odbc tdsodbc

MSSQL is accessed with the FreeTDS driver. Once the above packages are installed, you need to tell ODBC where to find this driver, by adding the following block to /etc/odbcinst.ini:

[FreeTDS]
Description=MSSQL DB
Driver=/usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
UsageCount=1

The path is different on platforms other than amd64. Check the file list for the tdsodbc package on your architecture if you lose track of the path.

The next step requires that you know the database server address, version, and database name. Add a block for your database to the end of /etc/odbc.ini:

[foodb]
Driver = FreeTDS
Description = Foo Database
Trace = Yes
TraceFile = /tmp/sql.log
ForceTrace = yes
Server = 10.x.x.x
Port = 1433
Database = FooDB
TDS_Version = 8.0

Experiment with TDS_Version values if you have issues connecting. Different versions of MSSQL require different values. The name of the data source (‘foodb’), the Database, Description and Server are all bogus values which you will need to fill.

An example

For new PHP scripts, database grunt-work is invariably done via PHP Data Objects (PDO). The good news is, it is easy to use it with MSSQL from here.

The below file takes a query on standard input, throws it at the database, and returns the result as comma-separated values.

Save this as query.php and fill in your data source (‘odbc:foodb’ here), username, and password.

#!/usr/bin/env php
<?php
$query = file_get_contents("php://stdin");
$user = 'baz;
$pass = 'super secret password here';

$dbh = new PDO('odbc:foodb', $user, $pass);
$dbh -> setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$sth = $dbh -> prepare($query);
$sth -> execute();
$results = $sth -> fetchAll(PDO::FETCH_ASSOC);

/* Quick exit if there are no rows */
if(count($results) == 0) {
	return 0;
}
$f = fopen("php://stdout", "w");

/* Output header */
$a = $results[0];
$header = array();
foreach($a as $key => $val) {
	$header[] = $key;
}
fputcsv($f, $header);

/* Output rows */
foreach($results as $result) {
	fputcsv($f, $result);
}

fclose($f);

To test the new script, first make it executable:

chmod +x query.php

To run a simple test query:

echo "SELECT Name from sys.tables ORDER BY Name;" | ./query.php

Refining the setup

The above script has some cool features: It’s short, actually useful, and it sets PDO::ERRMODE_EXCEPTION. This means that if something breaks, it will fail loudly and tell you why.

Hopefully, if your setup has issues, you can track down the cause with the error, and solve it by scrolling through this how-to again.

If you encounter a MSSQL datablase with an unknown schema, then you may want to list al rows and columns. This is achieved with:

SELECT tables.name AS tbl, columns.name AS col FROM sys.columns JOIN sys.tables ON columns.object_id = tables.object_id ORDER BY tbl, col;

The catch

I’ve run into some bizarre limitations using this. Be sure to run it on a server which you can update at the drop of a hat.

A mini-list of issues I’ve seen with this combination of software (no sources as I never tracked down the causes):

  • An old version of the driver would segfault PHP, apparently when non-ASCII content appeared in a text field.
  • Substituting non-text values fails in the version I am using, although Google suggests that updating the ODBC driver fixes this.

Recovering auto-saved files in MySQL Workbench

MySQL workbench is an open source tool for designing databases. As version 6.0.8, it is one of those programs where you need to save often, because the window you are working in will vanish every couple of hours.

Bug #1: Can’t recover files that weren’t saved

I was unlucky enough to have forgotten to save my work when it crashed today, and found this nasty flaw in the auto-recover feature:

Auto-save model interval: An open model that has not been saved will automatically be saved after this period. On loading a model file, MySQL Workbench will notify the user if the file was not previously saved correctly, due to a crash or power failure. MySQL Workbench can then attempt to recover the last auto-saved version. For automatic recovery to be available for a new file, it will have to have been saved at least once by the user.

Uh oh! The file hadn’t been saved yet, so it’s gone right? According to wb_model_file.cpp, this is not the case. The auto-save file is always written, but the recovery process wont be started until you try to use it again (which will never happen if you don’t have an old saved version):

/* Auto-saving
 *
 * Auto-saving works by saving the model document file (the XML) to the expanded document folder
 * from time to time, named as document-autosave.mwb.xml. The expanded document folder is
 * automatically deleted when it is closed normally.
 * When a document is opened, it will check if there already is a document folder for that file
 * and if so, the recovery function will kick in, using the autosave XML file.
 */

So under ~/.mysql/workbench/, I found a newmodel.mwbd folder. Workbench files are .zip files in disguise, so I compared it to a test file. It had all the same content, but with a document-autosave.xml, rather than a document.xml (see test file below):

Test archive

Cool, so I’d just rename the file, compress the whole lot and make it a .mwb? No such luck.

Bug #2: File Roller can’t compress ‘@’ files

Possibly because of the -@ command-line option in the zip command, File Roller refused to work with these files.

File Roller bug

Luckily, the document.mwb.xml file alone is enough for the file to be recognised and recovered from the auto-saved files:

File is recovered

The take-away from this? Save your work. In 2014, you still can’t count on auto-save to do this!

Transforming between SQL dialects

I recently found myself Googling for some data voodoo. I have a web app which I want to work with locally, and the RDBMS requirements are a little bit incompatible.

Unfortunately, neither this impressive sed script nor this eloquent mix of sed, ruby and perl could do this with <100 syntax errors in the output, so I had to get creative. Here is what I have learned:

  • Simplify the problem. I decided to convert the structure to SQLite manually, as it is not likely to change. The parts you will need to convert often (thousands of INSERT statements) are the parts which are more important to have a script for. The extra mysqldump options for getting the data only, without nasty `backticks` were:
    --compatible=ansi --skip-extended-insert --compact --no-create-info
  • Use sed to fix the escaping. MySQL escapes single quotes with ‘, and double quotes with ” but SQLite uses ” and “. This one-liner made the conversion:
    sed -e "s/\'/''/g" -e 's/\"/"/g' db.sql > db.sqlite

The resulting file could have the structure cat‘d on to the start and imported into SQLite.

Loading Asterisk CDR into a database

In the interests of accurate accounting, Asterisk creates a Master.csv, logging all calls and a few things about them.

This page on wiki.asterisk.org has a good breakdown of what all the fields mean.

I put together this MySQL table so that the data can be processed for data-analysis, accounting, or whatever it is that the data is needed for.

-- Code to create MySQL table for Asterisk CDR data
-- Field descriptions from:
--   https://wiki.asterisk.org/wiki/display/AST/CDR+Fields
CREATE TABLE cdr (
    accountcode VARCHAR(256) COMMENT 'What account number to use, (string, 20 characters)',
    src VARCHAR(80) COMMENT 'CallerID number',
    dst VARCHAR(80) COMMENT 'Destination extension',
    dcontext VARCHAR(80) COMMENT 'Destination context',
    clid VARCHAR(80) COMMENT 'CallerID with text',
    channel VARCHAR(80) COMMENT 'Channel used',
    dstchannel VARCHAR(80) COMMENT 'Destination channel if appropriate',
    lastapp VARCHAR(80) COMMENT 'Last application if appropriate',
    lastdata VARCHAR(80) COMMENT 'Last application data (arguments)',
    tsstart DATETIME COMMENT 'Start of call (date/time)',
    tsanswer DATETIME COMMENT 'Answer of call (date/time)',
    tsend DATETIME COMMENT 'End of call (date/time)',
    duration INT(11) COMMENT 'Total time in system, in seconds (integer), from dial to hangup',
    billsec INT(11) COMMENT 'Total time call is up, in seconds (integer), from answer to hangup',
    disposition ENUM('ANSWERED', 'NO ANSWER', 'BUSY') COMMENT 'What happened to the call',
    amaflags ENUM('DOCUMENTATION', 'BILL', 'IGNORE') COMMENT 'What flags to use, specified on a per channel basis like accountcode.',
    uniqueid VARCHAR(32) COMMENT 'Unique Channel Identifier',
    userfield VARCHAR(256) COMMENT 'user field: A user-defined field, maximum 255 characters'
) ;

The second-last field is not present on all installations, but it is on mine. The version I’m on is:

asterisk:~# asterisk -V
Asterisk 1.8.10.1~dfsg-1ubuntu1