blog.humaneguitarist.org

Just a quick morning post today …

The last 10 days or so I’ve been struggling with some major problems that arose in trying to implement effective XQuer-ies on my web demo.

1. Dublin Core doesn’t allow me to differentiate creator "types", so I was limited to searching across the DC:creator element for all creators, be they Composer, Lyricist, or Arranger. MusicXML does differentiate these types, so essentially Dublin Core was making me "dumb down" some information. I want people to be able to search creator specifically by their role: Composer, Lyricist, or Arranger.
2. I needed a way to iterate an XQuery over all the MusicXML documents and I needed it to be relatively fast. A demo is a demo, but impatience is impatience and I just can’t accept slow query processing.
3. The XQuery processor I was using didn’t support some XQuery functions that would allow a searcher to type in "Bach" and retrieve documents for which the creator was "J.S Bach", "Johann Sebastian Bach", "Bach, J.S", "P.D.Q Bach", etc. This really was limiting the search/query coolness factor and I wasn’t at all happy about it.

Here are my solutions (details to follow in a few days or so):

1. Ditch Dublin Core and switch to MODS, which does allow me to specify the role of a creator. Last week, I made a MusicXML to MODS XSL transformation for descriptive metadata and it’s working well.
2. Steal an idea from Using XQuery on MusicXML Databases for Musicological Analysis so that rather than iterate one query (say for the number of notes in a piece) across multiple MusicXML docs, I just concatenated all the MusicXML documents. The original files are left alone, but a "super" MusicXML file gets created so that one can just query that one file, hence no need for lengthy iteration. I’m not sure how those fellows did it, but I just automated it via PHP using the following format:

 <hyperMXML>

<hypoMXML file="foo1.xml">

1st MusicXML document

</hypoMXML>

<hypoMXML file="foo2.xml">

2nd MusicXML document

</hypoMXML>

</hyperMXML>

3. Switch XQuery processors! I’ll go into the ones that didn’t support the function I needed another time, but I will say that BaseX did the trick. Below is the query that searches for creators with "Bach" somewhere in MusicXML’s <creator> element. For the deliverable demo, I won’t be querying these big MusicXML documents for simple descriptive metadata like Creator, that’s what the MODS is for. But this is just an example. The "ftcontains" syntax is what allows for retrieval of values where "Bach" is somewhere within the element, but isn’t necessarily equivalent to the entire element value.

for $i in doc("../temp/concat/concatMXML.xml")/hyperMXML/hypoMXML/score-partwise
where $i/identification/creator ftcontains "Bach"
return ($i/work/work-title)

This blog post is part of a semester-long investigation into digital encoding of symbolic music representation (SMR), its context in libraries, web-based delivery, preservation and metadata, and search and retrieval technologies.

All posts regarding this project will appear in the LS-598 category.

This is my final semester at the School of Library and Information Studies at the University of Alabama.

For my final credits, I’ll be researching the digital encoding of symbolic music representation (SMR), i.e. "sheet music", its background, and the benefits it offers over simply thinking of SMR as an image - all within the context of libraries, including avenues of web-based delivery, preservation and metadata, and search and retrieval technologies. My research will be directed by Dr. Steven MacCall.

In addition to a paper addressing these issues, I’m required to deliver what is essentially a "demo" of a MusicXML web-based delivery system that could serve to demonstrate to librarians the possibilities that arise with the usage of XML-encoded musical information.

I’ll be blogging along this semester as part of a modular approach to constructing the paper. Currently, I’m plugging away at the demo which utilizes open-source server-side music applications.

The first idea behind the demo is that one could "drop" MusicXML documents on their server and - via automation - PDF, audio, and preliminary Dublin Core metadata are generated. So far, I’ve got all that covered but I need to polish the output and I really need to comment-up my PHP code as I’m even forgetting at times what I’ve done and why. I’ll be sharing the code as well as the XSL transformations that are used to generate the Dublin Core metadata from the MusicXML documents.

As time allows, I’ll try and add some cool features. For example, this week I implemented libmusicxml so that the user can generate a PDF of a musical score in a different key than the original. I’m guessing this is the same way that the Wikifonia site offers this transposition option. I’ll have to ask them to make sure.

The second idea behind the demo is to implement a search/retrieval mechanism using XQuery. This will prove the biggest challenge as I don’t know XQuery well, but I have some excellent sources on querying MusicXML documents from which I can learn. Implementing XQuery on the site has - for the most part - already been dealt with in terms of scripting. In other words, the hard, hard work is done in terms of the demo, but I’m not quite there yet …

ps: Unfortunately, the server-side software required to run the demo can’t be added to a free server like the one I use for this blog. But while I can’t host the demo, I will at the end of the semester offer it as a download-able package so that one can run it on their personal computer as server.

This blog post is part of a semester-long investigation into digital encoding of symbolic music representation (SMR), its context in libraries, web-based delivery, preservation and metadata, and search and retrieval technologies.

All posts regarding this project will appear in the LS-598 category.

For some reason I feel obligated to point out that I haven’t blogged in a while for a few reasons:

1. Christmas break from school/work at the University of Alabama
2. the desire not to blog for the sake of blogging
3. and …

I’ve been working on something huge - at least for me. It’s a piece of software called AudioRegent that harnesses XML to create derivative "clips" of regions within WAV audio files. A region is simply a user-defined segment within an audio file, like a track on a Compact Disc.

Besides writing the program in Python, which I pretty much finished in December, I had to also develop the XML format which I call SimpleADL (Simple Audio Decision List) that AudioRegent looks at and then makes derivative audio clips by leveraging SoX, the Sound Exchange command line audio editor. AudioRegent and SimpleADL can also be used to sync audio to text, like transcripts.

Actually, the programming and devising SimpleADL were the easy part. The hard stuff was the documentation and deciding on a license for the software.

I tried to find a balance in documenting the software: being thorough without writing a novel. I’m not sure I succeeded, but I can always improve it with time.

I used the W3C’s Amaya editor to write the documentation in XHTML. Sure, you can use OpenOffice to export a document to XHTML, but man is it bloated and messy. Amaya writes really clean XHTML.

As for the license, I chose the BSD license. As I understand it, this allows one to use the source code at will in future open or closed-source applications as long as you maintain the credits for AudioRegent. I was tempted to use the Mozilla Public License (MPL) which, again from what I can tell, is similar to the BSD license except that any source derived from AudioRegent would have to stay open-source though any peripheral code can be closed-source. I absolutely decided against the GNU General Public License which is viral and imposes its philosophy perpetually on all subsequent code, even peripheral code. Some have even argued that it works against its own objectives and is less "open" than the MPL.

Now I realize that, practically speaking, a skilled programmer could write better code from scratch in 30 minutes as opposed to the some 30 hours I needed, but I wanted to go about this quasi-professionally. And I learned more about licensing, which was cool.

Anyway, rather than try and explain the software itself and how to get it, I’d be better off pointing you to the documentation if you have any interest …

It seems that about a year ago, the creators of South Park started freely offering their episodes online at southparkstudios.com.

The move was made, in part, to offer a legal way for fans without television access or even televisions (that’s me!) to watch the show. As the creators stated, they got:

"really sick of having to download our own show illegally all the time.  So we gave ourselves a legal alternative."

This was great for me because I admittedly was downloading those shows as well. One of the other great things about it is the Random Episode link which, you guessed it, launches a random episode of the show.

The only thing that makes it less than perfect is that I can’t set it to repeatedly play more random episodes.

Until now.

I decided to sit down and write the most important code I may ever write: a program that leverages the Random Episode feature to launch successive random episodes of the great social commentary, courtesy of Matt and Trey.

Here’s the core of the code, from a Python perspective:

import sys, webbrowser, time

url = ‘http://www.southparkstudios.com/episodes/random.php’

i=1

while i <= 5:
      webbrowser.open(url)
      time.sleep(1440) # 1440 seconds = 24 minutes, the average length of an episode plus ads.
       i=i+1
sys.exit()

Basically this is a loop that keeps launching a episode every 24 minutes while the variable "i" is less than or equal to 5. After every episode, "i" grows by one so when "i" equals 6, the program shuts down. BTW: after the program starts, the user does have to manually initiate playback of the first episode, but the last 4 episodes play automatically.

Now, this works fine and I even embellished the code so that the user can select from 2 to 10 episodes to launch. But I was still thinking, "This would be great to share with my friends, but they aren’t going to run software from the command line!"

So I wrote a version using Microsoft’s Visual Studio Express, a free version of their software that allows one to create GUI applications with a mixture of visual tools and the Visual Basic language. It’s a really great tool that I hope to use more in the future although I do worry about the extra time required just to make software have a graphical interface.

Anyway, here’s a ZIP file that contains both the installer for the Windows version of South Park Launcher as well as the Python version for Windows and other platforms. Make sure to read the ReadMe.txt file first!

You assume all responsibility for use.

One of the nice things about an emerging standard, namely MusicXML, having a command center (Recordare LLC) is having a central place to learn about what’s new.

On Friday, I was looking at Recordare’s page of MusicXML related software for software that worked from the command line and noticed something new and really interesting: MusicSQL.

According the the Goodle Code page that hosts this project, MusicSQL is:

a system for conducting complex searches of symbolic music databases. The database can import and export MusicXML files. In the current version searches are constructed using a command line interface or through simple Python scripting tools.

Basically, at least as I understand it, MusicSQL is a Python program that sits on top of a MySQL database - now I really hope Oracle doesn’t kill MySQL if it buys Sun.

I was so excited to get MusicSQL working that I didn’t notate all the little problems I had along the way. The documentation for MusicSQL is very good and is written for Windows, Mac, and Linux (Ubuntu) users. But I’m inconceivably impatient, so I just mowed through the installation with little care for remembering what I was doing.

I do remember that I had to install Python 2.5, whereas I already have Python 2.6 installed - now I have both. I put/installed all the dependencies in my Python 2.5 directory just to compartmentalized everything - the exception being MySQL, which I installed wherever the default is.

So far, I only ran the first query in the documentation that uses "scientific" musical notation in the form Nx, where "N" is the alphabetical note name, say C, and "x" is an integer that denotes what octave the note is a member of. In other words, a C-Major scale would be "Cx Dx Ex Fx Gx Ax Bx Cx+1", something like "C5 D5 … B5 C6", etc. You can place an integer before the note name to denote its duration.

Running the query from the command line, I was really happy with the speed and the output of MusicSQL for the test query.

One problem I did have, though, is I kept getting errors for another great feature of MusicSQL. Basically, after you run your query, you can see a PDF of the results (i.e. the music excerpt pertaining to the query results). The PDF is made by Lilypond, a text-based notation software that produces - in my opinion - the absolute best looking engraving out there, that’s why I use it (and yes, it’s free).

Now Lilypond doesn’t natively read MusicXML, it uses its own encoding. So MusicSQL takes advantage of a Python script that comes with the Lilypond install called "xml2ly" that converts MusicXML to Lilypond format. I left a message on the project forum for MusicSQL, so I’m hoping I can figure out what I need to do to get the Lilypond outout of the query results to work. At any rate, I do wonder how effective it can be since the conversion from MusicXML to Lilypond can sometimes get ugly.

I wonder if an alternative solution is to use the command line options for the MuseScore notation software to generate a PDF of the query results. Musescore can also convert MusicXML to other graphics formats (PNG) and even audio (WAV, FLAC, OGG), so theoretically it could be leveraged to make audio files for the corresponding query results.

At any rate, I’m really looking forward to the future developments of MusicSQL.

And as for using MuseScore’s command line in conjunction with MusicXML and how it can add value to a web collection of MusicXML docs - there will be more to that later …

A while ago, I posted about my first experience with XQuery and how I’d used the .NET version of the Saxon processor on my local Windows machine.

Obviously, I want to extend that experience to running XQueries online. So far, I know this can easily be done with a native XML database server like eXist. That is to say, when I installed eXist on my local machine and made it go live: bam! - instant XML server with built-in XQuery functionality, accessible from anywhere.

Another way is to use one of the more popular web-scripting languages to execute XQuery syntax. I nearly killed myself this weekend trying to install the Zorba PHP binding for XQuery (i.e. run XQuery natively from within a PHP script). I just couldn’t get all the dependencies successfully installed on my virtual install of Ubuntu Netbook Remix (BTW: I use Sun’s VirtualBox for virtualization). Perhaps I’ll be able to make it work another time.

Now, even though it makes all the sense in the world to stick with a native XML server like eXist if I want to make a large collection of searchable XML documents online (and I do), I’m feeling non-sensical. What I decided to try was to run my computer as a more traditional server using the X-Apache-MySql-PHP model, specifically WampServer.

From there, I placed my sample document, "books.xml" and my "test.xquery" file from last time:

<ul>
{
for $x in doc("books.xml")/bookstore/book/title
order by $x
return <li>{data($x)}</li>
}
</ul>

in WAMP’s "www" directory - i.e. the directory which is accessible from the browser, the place where one would put all their .html files, etc. for the world to see.

Of course, I was still missing the actual XQuery processor at this point. What I tried is to put the relevant executables for Saxon in the "www" directory as well.

… Now, I’m sure this is totally unsafe or something, but I was just testing and I only make my computer "go live" as a server for short periods of time.

Anyway, from there I used PHP to call the Saxon processor and to display the results in the browser. It actually worked!

Here’s the code:

<?php
echo "Hi. I’m going to use XQuery to list the books alphabetically.";
exec("query.exe test.xquery !indent=yes", $results);
foreach ($results as $value)
    {
       echo "$value";
    }
?>

You can see that the PHP "exec" command called the Saxon executable named "query.exe" and executes the "test.xquery" file.

It saves the results in a variable called "results" and then prints each value of the results in the browser.

…

For now, I’m OK with this, but I need to eventually do the same thing with the Java version of Saxon, I suppose, if I’m ever going to run this on a Linux server. It shouldn’t present any new hurdles, but I need to try it to make sure, of course.

If anyone out there has any thoughts or recommendations on a more elegant method to achieve these results - and what the security risks of the approach I’ve outlined presents (since I didn’t use a CGI-bin), please speak up. I’m all ears.

UPDATE, 11/15/2009: No problems with using the Java version of Saxon. I did place it in a "bin" directory on my local server so that the Java file wouldn’t reside in the directory that users have direct access to.

A few months ago at work I spearheaded the standardization of our digitization log spreadsheets - simple Excel files that let students track what they digitized, its unique identifier, number of scans per item, date of scan, notes, etc.

The reason for this was twofold:

1. A standard template would hopefully make the students a little more productive in that they would not have to re-acclimate themselves to a ‘new look’ whenever they starting working on a different digitization project.
2. A standard template would allow us to quickly merge all the spreadsheets into a temporary master Excel file and import that into Microsoft Access so we could run queries and make quick monthly reports. We used a Visual Basic for Applications (VBA) script called RDBMerge to make the master file. It’s a fantastically well written piece of work.

Of course, it took several days to standardize the numerous existing spreadsheets but we got there. Now, all spreadsheets had the same naming convention (using the collection’s unique identifier) and were placed in the same folder - which essentially made it a quasi-database that could be queried with the search functions inherent to the OS. We considered using an actual database, but honestly that causes its own set of problems with training, support, and other stuff - hence the decision to stick with Excel and employ RDBMerge/Access.

Now, over time with standardized spreadsheets one naturally begins to think, "Ah! We should have also added a field for … !"

Well, yes that’s a pain. In a true database that’s no big deal, but with individual files who wants to go through myriad spreadsheets and manually add a column or a formula, etc?

Not me.

I did it once and it took a few hours even using some minor automation in the form a macro I recorded in Excel.

Once was enough.

But the great - I mean truly great - thing though about recording macros in Microsoft Office is that the resultant VBA code - code that the act of recording a macro generates - is visible. And, thus, editable. Not to mention it’s a great way to learn VBA syntax. In other words, if you record yourself doing a task in Excel, you can then actually see/edit the line item code that equates to your actions. Seems like a great tool for teaching some programming skills …

Anyway, I started to ask the lazy question, "How can I iterate a recorded macro/s over all spreadsheets in a given folder?"

After much digging, I found a great template script here. The script basically is a wrapper that can encapsulate the VBA script for one’s recorded macros. A little know-how and tweaking is required, but that’s better than the alternative: learning from-scratch Visual Basic on the job is something I really just don’t have the time for right now. At any rate, I was able to use the template script and automate stuff across all of our digitization log files in no time.

By the way, if you use the script, note the snippet below shows a line break that isn’t present in the blogger’s post, perhaps due to their blogging software’s formatting.

‘will start LOOP until all files in folder sPath have been looped through

Set oWbk = Workbooks.Open(sPath & "" & sFil)

p.s. in my experience, batch VBA scripts break on Excel files for which sharing is turned on (simultaneous multi-user editing). Those files need to be identified* and temporarily isolated into another folder or the sharing needs to temporarily be turned off. Also, sheets need to be ‘unprotected’ for the code to work, but it’s quite simple to add a line in your macro that unprotects the sheet prior to performing whatever macro you’ve created.

* It’s a good idea to use some sort of suffix to identify shared files … something like "filename.share.xlsx". In fact, if one does so, a VBA script could be run to turn off sharing for all shared files (*.share.xls*). Unfortunately, trying to turn sharing off for a non-shared Excel file will break the script … which, again, is why it’s best to identify shared documents through a filenaming convention.

I usually try to post to this blog once a week on average. That forces me to try to learn something or learn about something during the week that I can post about.

Sometimes though I have to resort to things that I’ve been sitting on for a while, like this Python script I wrote called MyMusicReporter.

This program makes use of the id3reader module available here. In other words, the program won’t function without this module.

For those unfamiliar with Python, it supports the use of modules, which - as the name implies - is basically allowing the current Python program to take advantage of another, pre-existing Python program. At least that’s how I understand it for now …

Anyway, MyMusicReporter does the following:

1. it asks you to point it to a root directory on your computer,
2. it then finds all the MP3s in that folder and its subfolders,
3. it makes an XML file in the same directory as MyMusicReporter called MyMusicReport.xml which contains the Title, Performer, Track #, Year, and relative file path for each MP3,
4. it makes an XML stylesheet in the same directory as MyMusicReporter called MyMusicReport.xsl which makes MyMusicReport.xml look pretty in your web browser. 

Here’s an example of how the XML looks with my home-grown schema:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="MyMusicReport.xsl"?>

<MyMusicReport>
    <mp3>
        <title>R2-DVD2</title>
        <performer>Nitin Arora</performer>
        <track>1</track>
        <year>2008</year>
        <path>r2_dvd2.mp3</path>
    </mp3>
</MyMusicReport>

Anyway, here’s the source code for MyMusicReporter.py.

You assume all responsibility for use.

Last week I posted a simple Python script and alluded to what I feel is the need for LIS programs to address computer programming.

The LIS field - or industry? - is so immersed in the usage and advocating of technology that it seems logical, per my logic at least, to offer library students an introductory class that allows them to learn about what lies behind the surface.

That said, I think LIS programs that do offer such classes need to really think about what language(s) should be taught to students who will - perhaps more often than not - have virtually no prior programming experience.

And while it would be neither professional - or perhaps even nice - of me to get into specifics, let me just throw out this question:

Why Java?

Sure, it’s critically important as a language, but that isn’t the point. The point is "is it appropriate for what will likely be a class of beginner programmers majoring in library science?"

Now I’m not a real programmer, an educator, or even a ‘librarian’ but I’m going to answer the question anyway.

No.

Why?

For starters, library students - who are not computer science majors - need to learn a language that is syntactically simple enough to more or less stay out of their way as they learn to think about design - knowing where you want to go and figuring out how to get there.

For an example of the problems in wading through syntactical issues when you’re starting out, here’s an excerpt from John M. Zelle’s Python as a First Language regarding the ubiquitous "Hello World!" that constitutes many people’s first program.

Python supports the use of functions and classes but does not force it. Simple programs really are simple. For example, consider the ubiquitous “Hello World” program in Python:

print "Hello World!"

C++ requires this program to be wrapped in a function and preceded by a preprocessor directive:

#include <iostream.h>
int main()
{
   cout << "Hello World!";
}

In Java, the situation is even worse, as all code must be inside of a class:

public class helloWorld
{
   public static void main(String [] args)
   {
     System.out.println("Hello World!");
   }
}

Sure you can find plenty of Java advocates too, but keep in mind that Zelle is writing in terms of appropriate first languages for computer science majors, not librarians. That seems to only bolster the notion the Java might not be the best thing for LIS students - at least not for a first programming class.

Secondly, I think one needs to also consider the execution environment. PHP, for instance, can be executed via the command line, but also can so easily be executed in the most widespread GUI we’ve ever known: the web browser. All one needs is something like XAMPP, an Apache distribution with PHP that installs in minutes if not seconds. From there one can make web forms and database queries to their heart’s desire in addition to running backend programs that manipulate files and folders, etc.

Lastly, there’s the matter of salability. I’m not saying Java skills aren’t salable - that would be ridiculous - but will people really develop salable skills in one semester? PHP is salable, I think, in the library world just because it has, I’d argue, a lot of name brand recognition even amongst administrators who are less familiar with a lot of technologies. I see it on library job descriptions all the time (preferred qualifications).

By the way, here’s "Hello World!" in PHP; it’s almost as simple as Python.

<?php
echo ‘Hello World!’;
?>

I guess what I’m saying is that for a beginner what matters isn’t the language itself, it’s the language behind the language and I’m not sure Java is the best way for LIS students to become conversant in the latter.

I haven’t hidden the fact that this blog exists, in part, as a resume builder. And I think that programming skills are a valuable asset in digital libraries. Frankly, LIS programs need to address this to keep their graduates competitive. But that’s another post altogether …

Anyway, at work yesterday (Friday) I made a little Windows .bat file that creates a folder named after every line in a given plain text file.

When I came home, I napped until nearly midnight and woke up in an odd, creative state: when I’m mostly mind and hardly any body. Whether I’m hardly anybody is a matter of opinion.

I decided to make a fancier version of the batch file using Python, which is named after Monty Python’s Flying Circus. In all seriousness, that’s a huge part of the reason I decided to start learning Python.

The program can do one of the following:

1. Ask you to supply a path and then mimick the folder structure (all new folders are created inside a folder called "MyFolderListFolder" to prevent accidental messes).
2. Read from a plain text file of yours called "MyFolderList.txt" and make folders named for each line in the text file (all new folders are created inside a folder called "MyFolderListFolder" to prevent accidental messes).

Subfolders can be made using this format (Unix/Linux users need to use forward slashes, Windows users can use either):

folder1

folder1\subfolder

folder2

3. Take you to this page if you’d simply like to see the source code and learn more about it. Of course, .py files can be opened in a text editor so this is only really useful for the Windows .exe version I compiled with cx_Freeze.

The program has some error checking built in. For example, if your text file has blank rows or duplicates, it will alert you under most situations that there’s a problem you need to fix.

The source code link is below if anyone is interested … real programmers likely will - and should - laugh, but I’m still learning.

A text file of the program (i.e. source code) is available here.

You assume all responsibility for use.