While the Acoustid database is reimporting, I wanted to do something fun, so tried to integrate Acoustid links into the MusicBrainz website. I wrote a simple user script that calls the Acoustid API, loads all IDs linked to a recording and displays the data right next to the list of PUIDs. If you install the script, the “PUIDs” page will now look like this:

It links back to the Acoustid track pages, which you can see for example here. The database is still very small, so you will probably not see the links too often, but that should get better soon.

The next step, which is probably not going to be this easy, will be adding the “Remove” links to the pages also for Acoustid tracks and actually making them work.

If you just want to have a local copy of the MB NGS database, don’t want to mess with CPAN or compiling PostgreSQL extensions, you can try the NGS version of mbslave. You only need Python, psycopg2 and PostgreSQL. The number of steps you have to do manually is probably larger than with the official way, but on any Unix-based OS it should be just copy&paste from the README file and it should really just work, with the benefit of being able to customize everything.

Download:

• Windows Installer (9.0M)
• Source Code Tarball (1.4M)

Changes since the last release:

• Changed Picard icon license to CC by-sa.
• Small UI consistency changes.
• Albums with tracks linked to more than one file are never marked as “completed”.
• Fixed matching of scanned files to tracks while the album is still loading.
• Support for properly embedded FLAC pictures.
• Existing embedded images in APE and ASF files are removed only if there are new images to replace them.
• More strict tagger script validation.
• Fixed the $truncate tagger script function.
• Proper rounding of track durations.
• Fixed a bug with saving images larger than 64k to WMA files.
• Added a $swapprefix tagger script function.
• Release events with a date are preferred over the ones without a date.
• Files that are being saved as marked as pending.
• Updated .desktop file to allow opening Picard with multiple files.
• Handle the “open file” event on Mac OS X.
• Added timeouts to the HTTP web service client.
• Fixed a bug with albums missing the expand/collapse icons.

It has scripts for importing the database dumps as well as applying replication packets, so it can be used creating a MB database from scratch as well as updating an existing one. Instructions for setting up a new database can be found in the README file.

There are a few differences between these scripts and mb_server:

1. Depends only on Python and psycopg2. Should be easy to set it up on non-Linux platforms, but I’ve not tested that.
2. Doesn’t require additional space for extracting the database dump tarballs. It decompresses and imports the files on-the-fly.
3. Can be configured to import the MB database into any schema, not just “public”.
4. Can be configured to not replicate specific tables (e.g. direct search indexes, PUIDs).
5. Regrouped transactions. The MB replication packet consists of multiple transactions (each of which has multiple database operations). The original replication code in mb_server imports the transactions exactly as they were executed in the master database. My code applies the whole packet in a single transaction. This is a small disadvantage, but from an external point of view (external scripts, triggers), there is no practical difference.
6. New code, possibly buggy.

The code currently assumes that you know how to setup the MusicBrainz database, so it doesn’t do many checks. It it turns out it’s useful also to somebody else, I can make it more robust.

• Source code (104K)
• API documentation

The following things were changed in this version:

• Use Requires.private instead of Requires for dependencies in libmusicbrainz3.pc
• Support for reading and submitting ISRCs
• Support for reading ratings for all entity type
• Full release group and label support
• C wrappers for extractFragment and extractUuid
• Support for filtering by any Lucene query

mkdir -p ~/.quodlibet/plugins/events/
cd ~/.quodlibet/plugins/events/
wget http://dl.dropbox.com/u/5215054/mbtagsubmit.py

After you enable it and let it know your MusicBrainz username and password (Music → Plugins), it will watch the songs you listen and if any of them has a “musicbrainz_trackid” and at least one “genre” tag, it will use the MusicBrainz web service to submit them. The submission normally happens only every half an hour, but if you change many files in a short time, it will submit them in batches of 20 tracks (the maximum number allowed by the web service) every 2 minutes.

MusicBrainz uses random-based UUIDs (version 4) for all it’s new IDs, so the first idea was to implement the same. I know I can’t use this code in the end, because I need a good pseudo-random number generator, but I couldn’t resist to write it anyway. Messing with bits in high-level languages is always fun :) Here is the result (because of the use of the random() function, don’t use the code for anything serious):

CREATE OR REPLACE FUNCTION generate_uuid_v4() RETURNS uuid
    AS $$
DECLARE
    value VARCHAR(36);
BEGIN
    value =          lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || '-';
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || '-';
    value = value || lpad((to_hex((ceil(random() * 255)::int & 15) | 64)), 2, '0');
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || '-';
    value = value || lpad((to_hex((ceil(random() * 255)::int & 63) | 128)), 2, '0');
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || '-';
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    value = value || lpad(to_hex(ceil(random() * 255)::int), 2, '0');
    RETURN value::uuid;
END;
$$ LANGUAGE 'plpgsql';

It turned out that we need deterministic IDs to be generated from the script, so V4 was out of question. That was good, because we would need a better PNRG for the final version.

The next idea was to create the URL on which the new rows will be server and generate name-based UUIDs using the URL namespace. The idea is to concatenate a namespace and a name, calculate a cryptographic hash of the result, and use it’s bits to generate the UUID.  There are two options for hashing, either MD5 (version 3) or SHA-1 (version 5). SHA-1 is preferred by the RFC, but PostgreSQL only has a built-in function for MD5, so the decision for us was easy. The code doesn’t depend any random numbers, so it’s good enough to use in production.

CREATE OR REPLACE FUNCTION from_hex(t text) RETURNS integer
    AS $$
DECLARE
    r RECORD;
BEGIN
    FOR r IN EXECUTE 'SELECT x'''||t||'''::integer AS hex' LOOP
        RETURN r.hex;
    END LOOP;
END
$$ LANGUAGE plpgsql IMMUTABLE STRICT;

CREATE OR REPLACE FUNCTION generate_uuid_v3(namespace varchar, name varchar) RETURNS uuid
    AS $$
DECLARE
    value varchar(36);
    bytes varchar;
BEGIN
    bytes = md5(decode(namespace, 'hex') || decode(name, 'escape'));
    value = substr(bytes, 1+0, 8);
    value = value || '-';
    value = value || substr(bytes, 1+2*4, 4);
    value = value || '-';
    value = value || lpad(to_hex((from_hex(substr(bytes, 1+2*6, 2)) & 15) | 48), 2, '0');
    value = value || substr(bytes, 1+2*7, 2);
    value = value || '-';
    value = value || lpad(to_hex((from_hex(substr(bytes, 1+2*8, 2)) & 63) | 128), 2, '0');
    value = value || substr(bytes, 1+2*9, 2);
    value = value || '-';
    value = value || substr(bytes, 1+2*10, 12);
    return value::uuid;
END;
$$ LANGUAGE 'plpgsql' IMMUTABLE STRICT;

This code should be easy enough to modify to generate UUIDv5, if you have a way to calculate SHA-1 hashes. To use the function, you need to pass it a namespace and a name. The namespace itself is a UUID, it can be anything, but there are a few well-known options:

• URL
  '6ba7b8119dad11d180b400c04fd430c8'
• DNS (fully-qualified domain name)
  '6ba7b8109dad11d180b400c04fd430c8'
• ISO OID
  '6ba7b8129dad11d180b400c04fd430c8'
• X.500 DN (in DER or a text output format)
  '6ba7b814-9dad-11d1-80b4-00c04fd430c8'

The URL one is probably the most useful. So, to generate UUIDv3 for http://www.example.com/foo/1, you can use the following:

SELECT generate_uuid_v3('6ba7b8119dad11d180b400c04fd430c8', 'http://www.example.com/foo/1');