blog.humaneguitarist.org

MXMLiszt version 0.9.0 is now available for download.

MXMLiszt is a web-based delivery and search/retrieval environment for MusicXML files and their manifestations.

MXMLiszt was created in order to complete a Master’s in Library and Information Science at the University of Alabama under the direction of Dr. Steven L. MacCall.

The documentation and source-code download links are available here.

The accompanying research paper, “Beyond Images: Encoding Music for Access and Retrieval” can be accessed here.

As of June, 2010 the live demo of MXMLiszt can be accessed at:

http://opensourcelibrarian.org/MXMLiszt

MXMLiszt is licensed under the BSD software license.

For some reason I feel obligated to point out that I haven’t blogged in a while for a few reasons:

1. Christmas break from school/work at the University of Alabama
2. the desire not to blog for the sake of blogging
3. and …

I’ve been working on something huge – at least for me. It’s a piece of software called AudioRegent that harnesses XML to create derivative "clips" of regions within WAV audio files. A region is simply a user-defined segment within an audio file, like a track on a Compact Disc.

Besides writing the program in Python, which I pretty much finished in December, I had to also develop the XML format which I call SimpleADL (Simple Audio Decision List) that AudioRegent looks at and then makes derivative audio clips by leveraging SoX, the Sound Exchange command line audio editor. AudioRegent and SimpleADL can also be used to sync audio to text, like transcripts.

Actually, the programming and devising SimpleADL were the easy part. The hard stuff was the documentation and deciding on a license for the software.

I tried to find a balance in documenting the software: being thorough without writing a novel. I’m not sure I succeeded, but I can always improve it with time.

I used the W3C’s Amaya editor to write the documentation in XHTML. Sure, you can use OpenOffice to export a document to XHTML, but man is it bloated and messy. Amaya writes really clean XHTML.

As for the license, I chose the BSD license. As I understand it, this allows one to use the source code at will in future open or closed-source applications as long as you maintain the credits for AudioRegent. I was tempted to use the Mozilla Public License (MPL) which, again from what I can tell, is similar to the BSD license except that any source derived from AudioRegent would have to stay open-source though any peripheral code can be closed-source. I absolutely decided against the GNU General Public License which is viral and imposes its philosophy perpetually on all subsequent code, even peripheral code. Some have even argued that it works against its own objectives and is less "open" than the MPL.

Now I realize that, practically speaking, a skilled programmer could write better code from scratch in 30 minutes as opposed to the some 30 hours I needed, but I wanted to go about this quasi-professionally. And I learned more about licensing, which was cool.

Anyway, rather than try and explain the software itself and how to get it, I’d be better off pointing you to the documentation if you have any interest …

One of the nice things about an emerging standard, namely MusicXML, having a command center (Recordare LLC) is having a central place to learn about what’s new.

On Friday, I was looking at Recordare’s page of MusicXML related software for software that worked from the command line and noticed something new and really interesting: MusicSQL.

According the the Goodle Code page that hosts this project, MusicSQL is:

a system for conducting complex searches of symbolic music databases. The database can import and export MusicXML files. In the current version searches are constructed using a command line interface or through simple Python scripting tools.

Basically, at least as I understand it, MusicSQL is a Python program that sits on top of a MySQL database – now I really hope Oracle doesn’t kill MySQL if it buys Sun.

I was so excited to get MusicSQL working that I didn’t notate all the little problems I had along the way. The documentation for MusicSQL is very good and is written for Windows, Mac, and Linux (Ubuntu) users. But I’m inconceivably impatient, so I just mowed through the installation with little care for remembering what I was doing.

I do remember that I had to install Python 2.5, whereas I already have Python 2.6 installed – now I have both. I put/installed all the dependencies in my Python 2.5 directory just to compartmentalized everything – the exception being MySQL, which I installed wherever the default is.

So far, I only ran the first query in the documentation that uses "scientific" musical notation in the form Nx, where "N" is the alphabetical note name, say C, and "x" is an integer that denotes what octave the note is a member of. In other words, a C-Major scale would be "Cx Dx Ex Fx Gx Ax Bx Cx+1", something like "C5 D5 … B5 C6", etc. You can place an integer before the note name to denote its duration.

Running the query from the command line, I was really happy with the speed and the output of MusicSQL for the test query.

One problem I did have, though, is I kept getting errors for another great feature of MusicSQL. Basically, after you run your query, you can see a PDF of the results (i.e. the music excerpt pertaining to the query results). The PDF is made by Lilypond, a text-based notation software that produces – in my opinion – the absolute best looking engraving out there, that’s why I use it (and yes, it’s free).

Now Lilypond doesn’t natively read MusicXML, it uses its own encoding. So MusicSQL takes advantage of a Python script that comes with the Lilypond install called "xml2ly" that converts MusicXML to Lilypond format. I left a message on the project forum for MusicSQL, so I’m hoping I can figure out what I need to do to get the Lilypond outout of the query results to work. At any rate, I do wonder how effective it can be since the conversion from MusicXML to Lilypond can sometimes get ugly.

I wonder if an alternative solution is to use the command line options for the MuseScore notation software to generate a PDF of the query results. Musescore can also convert MusicXML to other graphics formats (PNG) and even audio (WAV, FLAC, OGG), so theoretically it could be leveraged to make audio files for the corresponding query results.

At any rate, I’m really looking forward to the future developments of MusicSQL.

And as for using MuseScore’s command line in conjunction with MusicXML and how it can add value to a web collection of MusicXML docs – there will be more to that later …

A while ago, I posted about my first experience with XQuery and how I’d used the .NET version of the Saxon processor on my local Windows machine.

Obviously, I want to extend that experience to running XQueries online. So far, I know this can easily be done with a native XML database server like eXist. That is to say, when I installed eXist on my local machine and made it go live: bam! – instant XML server with built-in XQuery functionality, accessible from anywhere.

Another way is to use one of the more popular web-scripting languages to execute XQuery syntax. I nearly killed myself this weekend trying to install the Zorba PHP binding for XQuery (i.e. run XQuery natively from within a PHP script). I just couldn’t get all the dependencies successfully installed on my virtual install of Ubuntu Netbook Remix (BTW: I use Sun’s VirtualBox for virtualization). Perhaps I’ll be able to make it work another time.

Now, even though it makes all the sense in the world to stick with a native XML server like eXist if I want to make a large collection of searchable XML documents online (and I do), I’m feeling non-sensical. What I decided to try was to run my computer as a more traditional server using the X-Apache-MySql-PHP model, specifically WampServer.

From there, I placed my sample document, "books.xml" and my "test.xquery" file from last time:

<ul>
{
for $x in doc("books.xml")/bookstore/book/title
order by $x
return <li>{data($x)}</li>
}
</ul>

in WAMP’s "www" directory – i.e. the directory which is accessible from the browser, the place where one would put all their .html files, etc. for the world to see.

Of course, I was still missing the actual XQuery processor at this point. What I tried is to put the relevant executables for Saxon in the "www" directory as well.

… Now, I’m sure this is totally unsafe or something, but I was just testing and I only make my computer "go live" as a server for short periods of time.

Anyway, from there I used PHP to call the Saxon processor and to display the results in the browser. It actually worked!

Here’s the code:

<?php
echo "Hi. I’m going to use XQuery to list the books alphabetically.";
exec("query.exe test.xquery !indent=yes", $results);
foreach ($results as $value)
    {
       echo "$value";
    }
?>

You can see that the PHP "exec" command called the Saxon executable named "query.exe" and executes the "test.xquery" file.

It saves the results in a variable called "results" and then prints each value of the results in the browser.

…

For now, I’m OK with this, but I need to eventually do the same thing with the Java version of Saxon, I suppose, if I’m ever going to run this on a Linux server. It shouldn’t present any new hurdles, but I need to try it to make sure, of course.

If anyone out there has any thoughts or recommendations on a more elegant method to achieve these results – and what the security risks of the approach I’ve outlined presents (since I didn’t use a CGI-bin), please speak up. I’m all ears.

UPDATE, 11/15/2009: No problems with using the Java version of Saxon. I did place it in a "bin" directory on my local server so that the Java file wouldn’t reside in the directory that users have direct access to.

I usually try to post to this blog once a week on average. That forces me to try to learn something or learn about something during the week that I can post about.

Sometimes though I have to resort to things that I’ve been sitting on for a while, like this Python script I wrote called MyMusicReporter.

This program makes use of the id3reader module available here. In other words, the program won’t function without this module.

For those unfamiliar with Python, it supports the use of modules, which – as the name implies – is basically allowing the current Python program to take advantage of another, pre-existing Python program. At least that’s how I understand it for now …

Anyway, MyMusicReporter does the following:

1. it asks you to point it to a root directory on your computer,
2. it then finds all the MP3s in that folder and its subfolders,
3. it makes an XML file in the same directory as MyMusicReporter called MyMusicReport.xml which contains the Title, Performer, Track #, Year, and relative file path for each MP3,
4. it makes an XML stylesheet in the same directory as MyMusicReporter called MyMusicReport.xsl which makes MyMusicReport.xml look pretty in your web browser. 

Here’s an example of how the XML looks with my home-grown schema:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="MyMusicReport.xsl"?>

<MyMusicReport>
    <mp3>
        <title>R2-DVD2</title>
        <performer>Nitin Arora</performer>
        <track>1</track>
        <year>2008</year>
        <path>r2_dvd2.mp3</path>
    </mp3>
</MyMusicReport>

Anyway, here’s the source code for MyMusicReporter.py.

You assume all responsibility for use.

Just a short post on the XSPF playlist format.

I’m not going to get into how to use it, but I do want to share a demo of how it could be used to lets users more easily navigate an audio rendering of a poem or transcript, etc. using the JW Player.

The demo uses XSPF to allow a user to jump to the first 3 stanzas of The Midnight Ride of Paul Revere as read by Bridget Rafferty for LibriVox.  Just use the fast forward button to jump to the next stanza.

Conversely, if the stanzas were segmented into multiple MP3s, etc. then the playlist could be used in the opposite manner. That’s to say that the playlist could be used to play the various segments back as a single continuity so the user doesn’t have to manually start the playback of each stanza/MP3.

Here’s the XSPF playlist if you’d like to see it.

Here’s the demo.

Earlier today,  I posted about my first experience with XQuery. I’d mentioned that I wanted to get my feet wet before I started trying to run queries on MusicXML documents.

Well, I’m an incredibly impatient person.

I couldn’t wait to take a couple of simple queries for a test run, especially after reading the following paper from the 2008 International Conference on Music Information Retrieval hosted by ISMIR, the International Society for Music Information Retrieval:  

Using XQuery on MusicXML Databases for Musicological Analysis
Joachim Ganseman, Paul Scheunders and Wim D’haes

Now, I’ve known for a while the tests have been done using XQuery on MusicXML documents, but this paper was getting at something that’s been on my mind for a long time now: the day we can have digital libraries of sheet music, not as image files, but as encoded documents, allowing musicians and the like to have the same online ability to query music in the way that users of prose and literary documents now take for granted.

Anyway, on to my first XQuery and MusicXML experience …

For testing, I used a very silly little ditty I wrote called "MusicXML: I Heart Thee".

Here are its various manifestations:

• MusicXML

• JPEG

• MP3

The first query demonstrated in the paper (see page 3) is one to count the total notes in a digital library, in this case the Wikifonia collection of MusicXML docs.

I couldn’t get it to work as written even after I adjusted the query to work on my test document. This is likely due to my own ignorance, but in the end it was a good thing because it forced me to write my own, simpler queries.

I’m using the Saxon query processor as described in my earlier post.

1. This query (in red) counts all the notes in my piece:

<ul>
{
for $i in doc("i_heart_thee.xml")/score-partwise
let $j :=count($i/part/measure/note)
return $j
}
</ul>

A line-by-line translation:

Open an unordered list.

Open the query syntax with the "{" character.

Let there be a variable called "i" that will start at the root element, <score-partwise>, of the document "i_heart_thee.xml".

Let there be a variable, "j", that executes the Count function on "i" for the <note> element which is a child of <measure> and a grandchild of <part>.

Print the value of "j".

Close the query syntax with the "}" character.

Close the unordered list.

2. This query (in red) counts all the notes in the vocal part (there are 3 parts altogether: voice, guitar, bass):

<ul>
{
for $i in doc("i_heart_thee.xml")/score-partwise
let $j :=count($i/part[@id='P1']/measure/note)
return $j
}
</ul>

A line-by-line translation:

Open an unordered list.

Open the query syntax with the "{" character.

Let there be a variable called "i" that will start at the root element, <score-partwise>, of the document "i_heart_thee.xml".

Let there be a variable, "j", that executes the Count function on "i" for the <note> element which is a child of <measure> and a grandchild of <part>, where the "ID" attribute of <part> is = to "P1". This is the vocal part of the score.

Print the value of "j".

Close the query syntax with the "}" character.

Close the unordered list.

If you run the first query you get the result "137" as in 137 notes, including rests – even the hidden rests in measures 1,5, and 9 that exist because both voices in the guitar part have rests, though it only displays as one rest each time on the image version of the score.

If you run the second query, you get 43 notes including rests and the tied notes.

I’m sure there are ways to subtract rests and tied notes, but I have to start somewhere, right?

Obviously, I've been playing around with XSLT lately. So naturally, the next logical step was to delve into XQuery, the XML query language de jure. Eventually I want to run queries on MusicXML documents, but I need to start small.

While the W3C tutorial on XQuery is a great introduction, there's one little problem.

It doesn't really tell you how to implement XQuery: i.e. how to actually run a query and retrieve results.

So after some random perusing and downloading, I - like the fool I am – was made aware by Dr. Michael Kay's "Learn XQuery in 10 Minutes: An XQuery Tutorial"  that the Saxon XSLT processor I was already using for XSLT transformations already had an XQuery engine built in.

That's to say that the .NET version of Saxon has 2 command line executables:

1. Transform.exe, which I'd already used for XSLT transformations
2. Query.exe, which allows one to run XQuery queries

So much for paying attention to what I download …

From there, it was a simple matter to use XQuery for the first time.

Here are the steps:

1. I downloaded the books.xml file provided by the W3C and place it into the "bin" directory of Saxon on my drive. This is same directory where the 2 afformentioned executables reside.
2. Using the kick-tail text editor jEdit, I copy/pasted/saved this query example from the W3C as "test.xquery" (also saved in the "bin" directory):

<ul>
{
for $x in doc("books.xml")/bookstore/book/title
order by $x
return <li>{data($x)}</li>
}
</ul>

This query simply lists all the titles from "books.xml" in alphabetical order.

3. Then using jEdit's command line plug-in called "Console", I set Console to the Saxon "bin" directory where "query.exe", "books.xml", and "test.xquery" reside. The easiest way to set the directory in Console is to type:

cd "C:\Documents and Settings\nitin\Desktop\saxon\bin"

Of course, you might extract Saxon elsewhere, but the important thing is to type cd + opening quotation mark + full path to Saxon's "bin" folder + ending quotation mark.

4. Now I was in the correct folder and could run the XQuery with the following command line syntax:

query test.xquery

And my results look like this:

I know what you're thinking: no line breaks! Sure, the computer doesn't care, but this is really hard for humans to read!

Yes, that's true. But I went ahead and pasted the following:

<?xml version="1.0" encoding="UTF-8"?><ul><li>Everyday Italian</li><li>Harry Potter</li><li>Learning XML</li><li>XQuery Kick Start</li></ul>

into a new document in jEdit anyway.

We're gonna take care of those line breaks now …

5. One of the many great things about jEdit is the ability to run Beanshell commands, which despite my attempts to sound authoritative, I only learned about roughly 30 minutes ago. This means that a search and replace can be done in jEdit using simple Java syntax to fix that line break issue. The first step is identifying where to insert the line break. I needed it in between > and <. Specifically, I needed a line break between all the red and green colored brackets:

<?xml version="1.0" encoding="UTF-8"?><ul><li>Everyday Italian</li><li>Harry Potter</li><li>Learning XML</li><li>XQuery Kick Start</li></ul>

 So I just invoked the jEdit search/replace box and did the following:

This simply says:

Find all instances of

><

and

Replace it with

>

<

- i.e. the text between the quotation marks. The n is, by the way, the line break syntax.

When I hit "Replace All", this was the result:

<?xml version="1.0" encoding="UTF-8"?>
<ul>
<li>Everyday Italian</li>
<li>Harry Potter</li>
<li>Learning XML</li>
<li>XQuery Kick Start</li>
</ul>

Problem solved.

6. Now I simply saved this document as "test.html" and opened it in a browser.

Anyway, that's my very simple start to XQuery, but I'm feeling pretty good about it nonetheless.

As part of my coursework for the University of Alabama SLIS program, I took a database class last year. Long story short, one of assignments was to create a Microsoft Access dbase based on Medline records.

The records were already provided for us as well as Java-based script to parse the information into a tab-delimited format prior to import into Access.

For extra credit, we were given another script that would parse records from an Ovid database. If we could find access to an Ovid dbase (I couldn't as they were all password protected, understandably), we could run the script, parse the records and bring them into Access for additional credit.

But there was a way to use a free source, Pubmed, and still get the job done.

How? Well, Pubmed allows article information to be exported as XML.

Once in XML, there was no need for a script to parse the information. From there it was simple to bring the information into Access. I found it easier to import it into Excel, clean it up, and then import that Excel data source into Access.

But what if you have OpenOffice?

I'm not aware of a simple way to import XML documents into OpenOffice Calc (their spreadsheet app) or Base (their dbase app).

But by using XSLT, there's a way around this issue.

Here are the steps:

1. Conduct searches in Pubmed.
2. Send your articles to the Clipboard.
3. Set display to "XML".
4. Send the results to "File" (see image below).
5. Save the file as "pubmed_results.txt".
6. Change the file's extension from "txt" to "xml".
7. Open the document in a text editor.
8. Above the DTD (i.e. <!DOCTYPE PubmedArticleSet PUBLIC … ">), add the following line:

<?xml-stylesheet type="text/xsl" href="pubmed_xslt.xsl"?>

9. Re-save the file.
10. Then, download this file to the same directory as your "pubmed_results.xml" file.
11. Now click on "pubmed_results.xml" ; your browser should now display select data in an HTML tabular format.
12. From here, simply copy/paste the tabular data into OpenOffice Calc, clean it up as desired, save it as a ".ods" file, hook it up to OpenOffice Base, and design your queries, etc.

And now you've got a totally Free (minus the cost of a laptop, internet connexion, etc.) desktop dbase of Medline results.

* Note that the XML stylesheet I provided only displays certain info. You can always open the stylesheet in a text editor and set it to display more information, such as Abstract, etc.

A few months ago, my department head had encouraged us to learn about XML stylesheets and XSLT transformations. After picking at it here and there, I finally had my breakthrough with it this weekend. Of course, were I more patient, I could have gotten paid to do this at work tomorrow.

As usual, the majority of the work is in finding examples and explanations that speak to me. This thread was particularly helpful.

One of the biggest breakthroughs – as embarrassing as it is to admit – was my realization that one needed an XSLT processor to actually create a new XML document based on the instructions provided in the stylesheet.

I’ve been experimenting with both the Saxon and Microsoft processors. Rather than run them from the Windows command prompt, I’ve been using the command line interface in the jEdit text editor. There’s a built in XSLT processor plug-in with jEdit, but I couldn’t get it to work, hence the use of the afformentioned methods.

If I understand corrently, one of the uses of this will be to take XML data about audio files generated from the JSTOR/Harvard Object Validation Environment (JHOVE) and map the pertinent information to another schema/XML document. That’s a bit out of my league right now, but a modest start is yet a start.

I’ll also be interested in using transformations to make customized XML documents from MusicXML sources and Zotero exports. Admittedly, I have no real ideas as to what I’d need to do this for, but I simply have a hankering to think of related projects. Maybe pulling the lyrics out of a MusicXML document into a TEI verse document?