$ gst-launch-0.10 -t filesrc location=/path/to/audio/file ! decodebin ! audioconvert ! chromaprint duration=60 ! fakesink sync=0 | grep 'chromaprint fingerprint'

I also wrote a simple Python script that uses the GStreamer element to generate fingeprints and look them up using the Acoustid web service. It’s nice to see how little code is necessary to have a fully working fingerprinting application.

The only disadvantage is that it’s too slow. :) I had some GStreamer code in Picard for decoding audio files, but I eventually removed it because it was significantly slower compared to the other decoding options (FFmpeg and DirectX). It seems that the situation is still the same, because even compared to calling the ffmpeg binary in a subprocess (which I used in acoustid-tools), GStreamer is about three times slower. It’s not too bad for applications that rarely use the functionality, but I wouldn’t use it in a tagger that needs to identify many files as fast as possible.

Anyway, the project is on Launchpad, where you can download the source code using bzr or a tarball. It depends on the development version of libchromaprint.

In the past few days I’ve been setting up a new server from Hetzner. It’s still not powerful enough for hosting the “final” service, but it should be fine to allow testing of the web service. With the new hosting, I’m also able to make the data dumps publicly available for download. It will take me some time to have an easy way to import/export them, but exporting a simple CSV file with all the submissions I got should be simple enough.

Finally, I hacked up some scripts in Python to demonstrate how to use libchromaprint and Acoustid to do audio file lookups. For now it requires the latest version of libchromaprint from bzr, but I’m planning to make a new release soon.

I must say that I’m really happy about the project. It started as a very simple experiment, but it’s becoming more and more realistic that it can end up being useful. I was also very surprised how much interest is there. I know about some people who are experimenting with chromaprint, I’ve been contacted by a few small companies that are interested in the project (way too early for that!), somebody even added it to Wikipedia next to the big commercial fingerprinting systems. :)

So the problem was to find an appropriate compression algorithm for the fingerprints. The usual integer compression algorithms (Elias codes, Golomb coding, Rice coding) work well for small numbers. The most common use case for these algorithms are posting lists in an inverted index, where the document IDs can be sorted and simple delta encoding then produces small numbers that can be nicely compressed. The problem is that I had very large numbers and I couldn’t sort them anyway. Using subtraction to calculate deltas wouldn’t help me much, because the resulting numbers would be still too high. After some failed experiments with the standard approaches, I had one of those “light bulb moments”: it occurred to me that even though the fingerprint numbers are high, they change slowly in terms in bit changes. If I use XOR instead of subtraction for the delta encoding, I will still get high numbers, but they will usually not have many bits set. What if I just take the bits that are set and encode those as a sequence of numbers between 1 and 32. It turns out that this works pretty well. Let’s say we have a tiny fingerprint with four 32-bit numbers:

0x7E65BC44 | 0x7E25B444 | 0x7E259414 | 0x6625D424

After applying the XOR encoding, we will get the following results:

0x7E65BC44 | 0x400800 | 0x2050 | 0x18004030

The next step is to extract individual bits from these numbers:

3 7 11 12 13 14 16 17 19 22 23 26 27 28 29 30 31 | 12 23 | 5 7 14 | 5 6 15 28 29

This doesn’t seem like we achieved much, but if you look closer, you will notice that the numbers both small and increasing. Having large and randomly ordered numbers was the main reason for not being able to use traditional integer compression techniques. So, let’s delta-encode the numbers:

3 4 4 1 1 1 2 1 2 3 1 3 1 1 1 1 1 | 12 11 | 5 2 7 | 5 1 9 13 1

The result is a lot of very small numbers. That’s exactly the right situation variable length integer encoding. I decided to use the PFOR-like algorithm for this I’d still like to re-evaluate other variable length encodings, but PFOR was very simple to implement and worked better than any other alternative I’d tried. The basic idea is to use constant n-bit for encoding all the numbers, with an “exception” block after the main block for numbers greater or equal to 2ⁿ-1. My version is different in that instead of storing the offset to the exception in the main block, I decided to always put the maximum possible value there. The current implementation in Chromaprint uses 3 bits, so the main block would contain these numbers, each of them encoded in 3 bits:

3 4 4 1 1 1 2 1 2 3 1 3 1 1 1 1 1 0 7 7 0 5 2 7 0 5 1 7 7 1 0

And the exception block would only have numbers that were greater or equal to 7, with the 7 subtracted from the to fit in 5 bits each:

5 4 0 2 6

In order to store this we need 31 * 3 + 5 * 5 = 118 bits, which is already a little less than the original binary encoding with 4 * 32 = 128 bits and it only gets better for larger fingerprints. In many cases, increasing the size of elements in main block to 4-bits also helps, so I’m planning to dynamically determine the ideal size and store in the fingerprint.

Here is a few examples of compression ratios on 120 seconds long fingerprints:

I have implemented this algorithm first in C++ and then also ported to Java to use it on the server side.

There isn’t much functionality. You copy&paste your Acoustid API key, select which directories to scan and let the application do its job. It will go through the directories, look for audio files that it hasn’t seen before, read MBID tags, calculate fingerprints and submit them to Acoustid. It remembers which files to ignore on the next run by using a simple text file with the list of processed files.

This is not an official release, so I don’t have any packages yet (you can download the source code using bzr if you are interested), but I’d love if somebody could give the Windows version a try. I’m sure there will be some bugs, but after fixing them and releasing the fingerprinter app, I can finally ask non-technical people for help with submitting data and focus on the server side again.

Here is a minimal template that can be used for starting a Qt/CMake project from scratch:

cmake_minimum_required(VERSION 2.6)

project(myproject)

find_package(Qt4 COMPONENTS QtCore QtGui QtNetwork REQUIRED)
include(${QT_USE_FILE})

set(myproject_HEADERS someclass.h)
set(myproject_SOURCES someclass.cpp main.cpp)
set(myproject_UIS myproject.ui)
set(myproject_RESOURCES myproject.qrc)

qt4_wrap_cpp(myproject_MOC ${myproject_HEADERS})
qt4_wrap_ui(myproject_UIS_H ${myproject_UIS})
qt4_add_resources(myproject_RESOURCES_CPP ${myproject_RESOURCES})

include_directories(${CMAKE_CURRENT_BINARY_DIR})

add_executable(myproject
    ${myproject_SOURCES}
    ${myproject_MOC}
    ${myproject_UIS_H}
    ${myproject_RESOURCES_CPP}
)

target_link_libraries(myproject ${QT_LIBRARIES})

New website

At first I wanted to do everything on the server side in Java, but as I was delaying it, I realized I better quickly just hack something up in PHP and then maybe write a better solution later. So I spent last weekend working on the website and came up with this. Nothing fancy, but the I needed the website only for API key management.

It allows you to log in with either OpenID or your MusicBrainz username and password. Once you are logged in, it will show you your user API key that can be used for submitting data to the database. Additionally, users can register applications and get their application API keys. I’m thinking about adding some statistics once there is more data, but that’s probably all the website has to do.

Web service is now (partially) online

The Jave web service code has also been changed. I think I’m finally reaching a point where I’m happy with the design of both code and database structure. The web service is documented on wiki, but the XML for results with MusicBrainz metadata will probably change. Ideas on how to integrate the MusicBrainz XML schema are welcome.

Anyway, since I had the website only and I was happy with the submission API, I’ve decided to put it online as well, so http://api.acoustid.org/submit is now accepting submissions as described on the wiki. It is hosted on a cheap VPS for now, but it should be enough for collecting the test data.

Some tweaking based on the collected data will have to be done before the lookup API can be enabled, but I hope it will follow shortly. Data dumps will be released as soon as I figure out a good way to do that (and there data to release of course!).

Chromaprint API

A very important part of Chromaprint’s algorithm is machine-learned configuration. When I asked people to send me test fingerprints in July, it was using a configuration that was learned on a random selection from my music collection. That wasn’t very good, so later I downloaded many 30 second audio snippets spanning many music genres from eMusic and other music websites and did some experiments with using this as the training data. It producer a little better results, but I still wanted something better. The problem was that I never had the time to do anything better. So I decided to take the best configuration I had and run another round of tests on real data.

Once I’ve done this decision, I needed to make a public API that can be used from external applications. The result is here. It’s a plain C API that I hope should be possible to use from any programming language that can load a shared library. There is still no official Chromaprint release, but there will be one as soon as I successfully use the API to build the graphical fingerprint submission tool that I was planning to. I don’t expected the API will change much though.

Help needed!

Now I need to build a large database of fingerprint/MBID data to see how this all works in action. The last round helped me a lot, but I’ve made a big mistake of not collecting MusicBrainz identifiers along with the fingerprints. This has been fixed and the fpcollect tool from Chromaprint now extracts MB track IDs from files and writes it to the log file. If you are on Linux (or another platform that makes compiling from source code easy), please download and compile Chromaprint and run the fingerprinter on your music collection. On Debian/Ubuntu, something like this should do the trick:

sudo apt-get install bzr cmake libfftw3-dev libavcodec-dev libavformat-dev libtag1-dev libboost-dev
bzr branch lp:chromaprint
cd chromaprint
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TOOLS=ON .
make
./tools/fpcollect /path/to/your/music/library >fpcollect.log

When you have the log file, you can run a simple Python application that is also available in Chromaprint to submit the data via the Acoustid web service. For this, you will have to first log in to the Acoustid website and get an API key. Once you have the key, you can run a command like this to submit the data (replace XXX with your API key):

./tools/fpsubmit.py -a XXX fpcollect.log

I know the process isn’t very easy, but I’m hoping to have a better and properly packaged submission tool soon.

First, you need the mingw32 package, which contains the compiler toolkit plus all the required WinAPI libraries. For cross-compiling CMake projects, you also need a file with paths to the MinGW toolkit:

set(CMAKE_SYSTEM_NAME Windows)
set(CMAKE_C_COMPILER i586-mingw32msvc-gcc)
set(CMAKE_CXX_COMPILER i586-mingw32msvc-g++)
set(CMAKE_FIND_ROOT_PATH /usr/i586-mingw32msvc /home/lukas/projects/win32build)
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

I have a special directory for installing compiled binaries, which makes it easier to use previously compiled libraries, so that’s included in the file. To compile a Windows version of libmusicbrainz, which uses CMake, you can use a serie of command like this:

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=/home/lukas/projects/mingw.cmake -DCMAKE_INSTALL_PREFIX=/home/lukas/projects/win32build/
make
make install

To build libmusicbrainz I also needed neon, which uses autotools. Fortunately, cross-compiling autotools projects is usually also quite simple, but neon doesn’t support MinGW out of the box, so one patch is necessary. After applying the patch, it can be compiled like this:

./configure --host=i586-mingw32msvc --disable-debug --disable-webdav --prefix=/home/lukas/projects/win32build/
make
make install

So, let me introduce Acoustid (named by herojoker), the server side component for audio fingerprint lookups. As I mentioned before, it uses PostgreSQL’s GIN indexes with the intarray implemented to do the hard work. Java is used as a glue language that provides an web-based service for interacting with the database. I originally considered using C++ for this (using the POCO libraries), but there is significantly more tools in Java for an application like this, so even though I don’t particularly like Java, it seemed like the best language for this.

Chromaprint fingerprints are basically timed sequences of 32-bit numbers (“subfingerprints”). They are extracted from the audio stream, using a technique which is not important here, every 0.1238 seconds and cover about 1.9814 seconds of the following audio data. This means that to represent a minute of audio, you need around 460 of such numbers. For two audio files representing the same song in different file formats, these numbers should be very similar in terms of bit errors. Acoustid’s job is to take these numbers and find a fingerprint in the database that has the least bit errors compared to the query. Theoretically, this could be used to identify songs based on a few seconds of the original audio, but that would require indexing whole fingerprints, because we don’t which part of the original fingerprint does the query match. This requires a lot of RAM to perform efficiently, so I decided to collect the data that is necessary to perform searches like this, but not implement the feature for now.

Instead, Acoustid’s primary task is to identify audio files and for this purpose it can assume that the query is extracted from the beginning of the audio. It indexes only ~15 (currently from 0:25 to 0:40) seconds of audio and uses this index to search for possible matches. Currently it assumes that at least one number from the range matches exactly. This can yield some false positive matches, so the next step is to compare the full query to the full fingerprint. As the fingerprints might not be ideally aligned, it first finds the best alignment and then calculates a score as the number of subfingerprints with less than 2 bit errors. This matching and scoring is done completely on the database server, using the intarray extension and a custom C extension for calculating the score.

This seem to all work fine from technical point of view. What I’m fighting with is the API for this. How to pass the fingerprint data efficiently, how to authenticate users for submissions, whether to have the concept of users at all. The main problems currently are:

1. The fingerprints are quite large and it would be ideal to send them as binary data. This conflicts with most existing standards for web services. I’ve implemented a compression algorithm that makes the size of base64-encoded fingerprint similar to the size of an uncompressed binary fingerprint, but I’d still like to not waste the bandwidth if I can avoid it. I’ve considered bencode-formatted requests, but that is probably too non-standard. I’ve also considered accepting “Content-Encoding: gzip” in HTTP requests, which everybody seems to be using only for responses, but it seems usable also for requests.
2. I’d like to track who submitted which fingerprints. I don’t need any private data, but I’d like to be able to delete all fingerprints from one source if I find out that there is something wrong with the data. Maybe I’m paranoid, but I’m afraid that some people might try to submit wrong data in order to reduce the usefulnes of the database.
3. If I want to track users, where should the user accounts come from. I definitely don’t want people to have to register and I’d like the service to be integrated with the MusicBrainz database, so accepting MB usernames/passports would be one option. Another option would be to go use any OpenID account and use something like OAuth for API authentication. I guess some people would prefer to not use their Google/Yahoo/etc. OpenID accounts, so perhaps having MB as an OpenID provider would help?
4. There is also the question of how to authenticate the users. If I accept MB username/passwords I could use standard HTTP Digest authentication, but normally that means that applications would have to send most authenticated HTTP requests twice. Only submissions would have to be authenticated and these will have a lot of data in the HTTP body, it would be a waste to send them twice. Other options are OAuth or a custom token-based method.
5. How to handle metadata? Should it rely only on MusicBrainz identifiers? That would make it less usable for taggers that prefer other metadata databases. Should it allow submissions that are not in MusicBrainz?

Regarding licensing, the source code is distributed under the GPLv3 license. I’ve considered more permissible licenses, but while there isn’t much code yet, I’d like to be sure that any improvements stay open source. The service is not yet live, so perhaps it’s too soon to speak about data licensing, but I meant to use something like CC BY-SA for the data (including database dumps), but not allow commercial usage of the free service. If there will be a commercial application interested in the service in the future, it should either run their own mirror or help paying the hosting costs so that the service can stay free for free applications.

Anyway, the source code is on Launchpad and there is a mailing list on Google Groups. There is also a wiki, which will eventually contain documentation for the project. If you would like to help me with finishing this project or are just interested in its future, please join the mailing list.

It all started in February this year, when when I got my Google Alerts mail for “musicbrainz picard” which included a link to this paper (“Waveprint: Efficient wavelet-based audio fingerprinting” (2007) by Shumeet Baluja and Michele Covell). I’ve never paid much attention to how audio identification systems are actually implemented, but I found it interesting that a paper published by Google researchers cited Picard, especially because Picard doesn’t implement any fingerprinting algorithm, just uses libofa. Anyway, I’ve read the paper and realized that maybe it’s not that hard to implement such a system. No tough DSP stuff or scary mathematics (of course there is some DSP stuff and mathematics, but mostly basics). The system described in the paper seemed quite straight-forward to implement, so I become curious and decided to give it a try. Later on I realized that it’s perhaps not the best system and that even the authors published new papers describing different approaches. That, combined with the fact that I was still officially an university student and had free access to all the papers from most organizations like ACM or IEEE, caused that I started reading more and more papers on the topic, learning about the history, how the systems evolved, and so on.

Many ideas were based on a paper by Yan Ke, Derek Hoiem, and Rahul Sukthankar called “Computer Vision for Music Identification” (2005). In fact, even the Last.fm fingerprinter uses the code published by the authors of this paper. This is where I learned that audio identification is more about machine learning that it is about DSP. Many useful methods for extracting interesting features from audio streams are well-known and the problem is more about how to apply and index them the best way. The basic idea here is to treat audio as a spectral image and index the content of the image. I’ll explain this in more detail and how Chromaprint uses this in a following post.

Another important paper for me was “Pairwise Boosted Audio Fingerprint” (2009) by Dalwon Jang, Chang D. Yoo, Sunil Lee, Sungwoong Kim and Ton Kalker (Ton Kalker is a co-author of a historically important paper “Robust Audio Hashing for Content Identification” (2001) published by Philips Research), which combined previous experiments of the authors with audio identification based on spectral centroid features and the indexing approach similar to the one suggested by Y. Ke, D. Hoiem and R. Sukthankar. For a long time this was the best solution I had and since it was actually not very hard to implement, the most time I spent on tweaking the configuration to get the best results.

The last major change came after I learned about “chroma” features by reading the “Efficient Index-Based Audio Matching” (2008) by Frank Kurth and Meinard Müller. I’ve read more papers about chroma features later, but this was the first and also the most important one for me and some ideas about processing the feature vectors from it are implemented in Chromaprint. Chroma features are typically used for music identification, as opposed to audio file identification, but I tried to use them with the approach I already had implemented and it nicely improved the quality of the fingerprinting function and actually reduced complexity which allowed me to use much larger training data sets.

Anyway, this is more or less how I got to this point. As I mentioned, I’ll try to describe in more detail how Chromaprint works and where are the exact ideas from in an another post later. The code is not finished yet, but the core ideas are already implemented and tested. The work that has to be done is mostly about cleaning the code, tweaking the configuration, running the learning algorithm on a better training data set (as I used only random selections of my music collection so far) and building some API that can be used by external applications.

The code is written in C++, but I plan the public API to be in plain C. Except for a FFT library (either FFTW3 or FFmpeg), it has no external dependencies. It’s released under the LGPL 2.1 license, so there should be no problem integrating it into a commercial application, assuming FFmpeg is used for FFT calculations (using FFTW3 would require the binary to be GPL compatible). The project is hosted on Launchpad using Bazaar for development. I’m sorry I didn’t include the complete development history there, but it’s just full of junk commits, so you will not miss much. :)

What I’d really like is to start actually working also on the fingerprint lookup service, for which I need as many fingerprints as possible. I have a proof of concept written in Java, using PostgreSQL and the intarray extension, which allows me to search the fingerprints using GIN indexes. This works fine on a database with tens of thousands of fingerprints, but I’m not sure if it will scale to much higher numbers. If you would like to help and you are running Debian/Ubuntu Linux, please run these commands and email me the compressed fpcollect.log file:

sudo apt-get install bzr cmake libfftw3-dev libavcodec-dev libavformat-dev libtag1-dev libboost-dev
bzr branch lp:chromaprint
cd chromaprint
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TOOLS=ON .
make
./tools/fpcollect /path/to/your/music/library >fpcollect.log

I’m sure the recipe should be easy enough to modify for other Linux distributions. I’ll try to build a Windows binary in the next days.

UPDATE: I’ve compiled a Windows version of fpcollect. If you would like to help, please download it, change the fpcollect.bat file to point to your collection and run it. It should should produce a file called fpcollect.log like the Linux example above.

So that’s all for now. My plans for near future are to clean up the library and build some simple GUI application for collecting fingerprints. Once this is done, I can ask non-programmers to help me build a test database of fingerprints (including MusicBrainz IDs) and work primarily on the server component. There are some things that could be improved on the client library from functional point of view, but I think it’s good enough for now, so the server part seems more important at the moment.

Btw, I always considered “Chromaprint” to be a temporary name for the client library, not meant to be its final name. If you can think of a better name, ideally something that could be used also for the service/server, please let me know!

This StackOverflow answer is a nice example of the problem:

$mondayStr = "last monday";
if (date('N') !== '1') {  // it's not Monday today
    $mondayStr .= " last week";
}

$monday = strtotime($mondayStr);
echo date('r', $monday);    // Mon, 22 Feb 2010 00:00:00 +1000

$sunday = $monday + 86400 * 7 - 1;
echo date('r', $sunday);    // Sun, 28 Feb 2010 23:59:59 +1000

The code seems logical. Get the timestamp of the last Monday, add 60 * 60 * 24 * 7 – 1 seconds and you have the end of Sunday. Works fine most of the time. Although, if the $monday happens to be Mon, 22 Mar 2010 00:00:00, the date that was supposed to be Sun, 28 Mar 2010 23:59:59 will actually be Mon, 29 Mar 2010 00:59:59. Why? Because 28 March 2010 has only 23 hours.

Never use timestamps to do calendar calculations. It’s hard to get it right. If you really have to, at least use GMT timestamps.