XML at blog.humaneguitarist.org

Archive for the ‘XML’ Category

fun with lxml, part 2

Just following up on a previous post from about a month ago …

Per a request, I need to tweak some software of mine to allow a user to specify a parent element in an XML document and in turn retrieve child element values. Big deal. That's what XSLT is for – blah, blah, blah. But this is particularly for PubMed XML exports and turning those into Excel files.

Anyway, the value of a given child element needs to be able to be specified (i.e. by position) and placed into an Excel cell. Alternatively, all children values need to be able to be placed into one cell separated by a delimiter.

So before I try and tinker with the software I want to work a solution out using test code:

from lxml import etree

##### Step 1
    # make an XML example
xml = '<a>  \
            <b>  \
                <c>cee1</c>  \
                <d>dee1</d>  \
                <c>cee2</c>  \
                <d>dee2</d>  \
            </b>  \
            <b>bee</b>  \
            <c>cee3</c> \
        </a>'

##### Step 2
    # parse the XML example
parseXML = etree.XML(xml)

##### Step 3
    # make a list of the first (i.e. the Zero-th) <b> element
b_list = parseXML.findall('.//b')[0]

##### Step 4
    # get a list of all the children in that first <b> element
b_childList = b_list.getchildren()

##### Step 5
    # make a new list called "c_list" with only <c> elements
    # that are children of our first <b> element

c_list = [] # make an empty list to put things in and
# place into that list only element *values* for child elements
# of first <b> element from children that are <c> elements only
for child in b_childList:
    if child.tag == 'c':
        c_list.append(child.text)

##### Step 6
    # print desired results

for c in c_list: #print all values, one per line
    print (c)

print ('-'*4) # print dash line for reading ease
print ('; '.join(c_list)) # print all values on one line with delimeter

print ('-'*4)
print (c_list[1]) #print only the second <c> element value

Here are the results:>>> cee1 cee2 ---- cee1; cee2 ---- cee2

--------------

fun with lxml

leave a comment

First off, I don't consider myself a programmer. I just know enough to dabble even though I try and learn new stuff all the time in the hope that I – as someone in digital libraries – can occasionally write something that can serve the needs of others, rather than serving my ego. Don't get me started on people who try and write software that has no utility other than patting themselves on the back …

Anyway, that's another post for another time.

So, the other day I got some questions/feature requests for PubMed2XL and so I started thinking about ways to tackle a few of the issues. It kinda makes me feel like a real programmer when people in the real world are asking about the software – but only for a few minutes before I make myself come back down to earth.

Currently, the software places into a spreadsheet cell the value of one XML element, the position of which is defined by the user in the setup file. But there may potentially be a need it seems to be able to concatenate ALL the values for a given element into one spreadsheet cell. So I wrote a little function to help me get started with that.

The code uses this simple restaurant-based XML file from W3Schools and uses the awesome lxml Python library.

When run, it yields the following:

>>> Calories for the first entree: 650 Calories for all entrees: 650; 900; 900; 600; 950

And here's the code:

#import required modules (lxml is non-standard; it likely needs to be installed)
import urllib #makes it easy to read documents from the web!
from lxml import etree #great XML parser and more!
                       #see: http://lxml.de/

#retrieve values from an XML file
def ElementCherryPicker(xpathArg, positionArg):
    '''
    This places all the element values for the element passed as the
    "xpathArg" argument into a list called "elementBox". It then returns
    the list item preceeding the one specified by the "positionArg" argument.
    This means passing a "1" equates to the first item in the list instead
    of the traditional "0". If "0" is passed then the entire list will be
    returned as a string with a delimiter of '; '.
    '''
    positionArg = positionArg - 1
    elements = parseUrl.findall(xpathArg) #make list of all matching elements
    elementBox = [] #create empty list
    for element in elements:
        elementBox.append(element.text) #place element values into the list
    if positionArg != -1:
        try:
            elementBox = elementBox[positionArg]
        except:
            elementBox = [] #if no element at stated position exists,
                            #then make the list empty again
    else:
        delimiter = '; '
        elementBox = delimiter.join(elementBox)
    return elementBox

#define, open, read, and parse an XML file
url= 'http://www.w3schools.com/xml/simple.xml'
readUrl = urllib.urlopen(url).read()
parseUrl = etree.XML(readUrl)

#print header and the values returned from ElementCherryPicker()
print 'Calories for the first entree:'
print ElementCherryPicker('.//calories', 1)
print 'Calories for all entrees:'
print ElementCherryPicker('.//calories', 0)

--------------

of ADLs and SMIL and stuff

leave a comment

Even more than usual – this post is me thinking out loud. So some of the stuff at the bottom might not make sense since it refers to some software of mine that really only I use.

This morning I played around a little with Kino, an open-source video editor, and Adobe Audition, Adobe's flagship audio editor – which is based on their acquisition of Cool Edit.

The reason I wanted to play around with Kino is because it can export the project timeline to SMIL. I was mainly interested in seeing if it could be used as a pseudo audio editor – the idea being it could be a quick and dirty SMIL exporter. Well, it doesn't seem to support importing audio formats. I couldn't get it to import WAV or OGG files. It's still a cool application though.

The session exports from Audition are, as expected, pretty dense. For people like me who work in libraries there are issues involved in terms of setting limits for how much can and should be done in digital audio "preservation" (funny, I don't remember ordering jam and bread …). Well, at least I think there need to be limits, lest libraries want to start being creators, too, and admit that in doing so they are donating material of their own editorial designs back onto themselves. Anyway, by imposing limits I'm not sure XML session exports of thousands of lines for simple edits are a good idea.

I'd like to see other session formats without downloading demos for all kinds of audio editing software (some more expensive packages don't even seem to offer demos). For a small fee, there's always AATranslator.

But getting back to SMIL, I'm wondering how to use it in conjunction with AudioRegent without writing more code into the application – for now.

It would seem pretty easy to create a SMIL to SimpleADL XSLT and set up a chain to create derivative files.

Specifically, say I have a source file called source.wav. And I have two SMIL files as such:

source-1.smil.xml

<?xml version="1.0"?>
<smil xmlns="http://www.w3.org/2001/SMIL20/Language">
  <body>
    <seq>
      <audio src="source.wav" clipBegin="00:00:00.000" clipEnd="00:00:30.000."/>
    </seq>
  </body>
</smil>

and source-2.smil.xml

<?xml version="1.0"?>
<smil xmlns="http://www.w3.org/2001/SMIL20/Language">
  <body>
    <seq>
      <audio src="source.wav" clipBegin="00:00:30.000" clipEnd="00:00:50.000."/>
    </seq>
    <seq>
      <audio src="source.wav" clipBegin="00:01:00.000" clipEnd="00:02:00.000."/>
    </seq>
  </body>
</smil>

For both, the assumption is that two clips are to be made from source.wav: source-1 and source-2.

All I'd need to do is then setup a chain as such:

Do source-1.smil.xml to temp.adl.xml via XSLT.
Have AudioRegent make source.ogg by pointing it, via the command line options, to the source file, source.wav, and the SimpleADL file, temp.adl.xml.
Rename source.ogg to source-1.ogg – i.e. with the same prefix as the corresponding SMIL file.
Do source-2.smil.xml to temp.adl.xml via XSLT, overwriting temp.adl.xml.
Have AudioRegent make source-2.ogg by pointing it, via the command line options, to the source file, source.wav, and the SimpleADL file, temp.adl.xml.
Rename source.ogg to source-2.ogg – i.e. with the same prefix as the corresponding SMIL file.

Here's what temp.adl.wav would look like initially (step 1):

<?xml version="1.0" encoding="UTF-8"?>
<audioDecisionList filename="source.wav">
  <region id="_01">
    <in unit="seconds">0</in>
    <duration unit="seconds">30</duration>
  </region>
  <outputAsTracks>false</outputAsTracks>
</audioDecisionList>

And then it would look like this during the second pass (step 4):

<?xml version="1.0" encoding="UTF-8"?>
<audioDecisionList filename="source.wav">
  <region id="_01">
    <in unit="seconds">30</in>
    <duration unit="seconds">20</duration>
  </region>
  <region id="_02">
    <in unit="seconds">60</in>
    <duration unit="seconds">60</duration>
  </region>
  <outputAsTracks>false</outputAsTracks>
</audioDecisionList>

By the way, since the SimpleADL files are temporary, I don't see why – rather than converting time format to seconds – I couldn't just use something like this:

<in unit="time">00:01:00.000</in>
<duration unit="time">00:01:00.000</duration>

or something …

--------------

MXMLiszt version 0.9.1 released

leave a comment

I've made some minor changes to MXMLiszt to address a bug that began to appear after months of trouble-free performance.

So here are the changes I made to address the issue related to the display of MODS metadata:

Created mods.css file to display MODS on a transparent background.
Changed displayMODS.php to display MODS files via an <iframe>. The previous version was using the mods.xsl stylesheet to parse the MODS element values in real-time.

You can read the documentation and download the source code for version 0.9.1 here.

--------------

PubMed to Excel: PubMed2XL version 0.9

3 comments

I've released the first Beta version of PubMed2XL, a Windows application that converts article lists from pubmed.gov into Microsoft Excel files.

If you'd like to use the software you can download it. Yes, it's free.

Here's a little video tutorial on installing and using the software:

PubMed2XL: Basic Installation and Use from nitin arora on Vimeo.

PubMed2XL's documentation is available at: blog.humaneguitarist.org/projects/pubmed2xl/.

The documentation includes a download link to the program files.

--------------

MXMLiszt release 0.9.0

4 comments

MXMLiszt version 0.9.0 is now available for download.

MXMLiszt is a web-based delivery and search/retrieval environment for MusicXML files and their manifestations.

MXMLiszt was created in order to complete a Master’s in Library and Information Science at the University of Alabama under the direction of Dr. Steven L. MacCall.

The documentation and source-code download links are available here.

The accompanying research paper, “Beyond Images: Encoding Music for Access and Retrieval” can be accessed here.

~~As of June, 2010 the live demo of MXMLiszt can be accessed at:~~

~~http://opensourcelibrarian.org/MXMLiszt~~

--------------

segmenting audio with AudioRegent, SoX and XML

leave a comment

For some reason I feel obligated to point out that I haven’t blogged in a while for a few reasons:

Christmas break from school/work at the University of Alabama
the desire not to blog for the sake of blogging
and …

I’ve been working on something huge – at least for me. It’s a piece of software called AudioRegent that harnesses XML to create derivative "clips" of regions within WAV audio files. A region is simply a user-defined segment within an audio file, like a track on a Compact Disc.

Besides writing the program in Python, which I pretty much finished in December, I had to also develop the XML format which I call SimpleADL (Simple Audio Decision List) that AudioRegent looks at and then makes derivative audio clips by leveraging SoX, the Sound Exchange command line audio editor. AudioRegent and SimpleADL can also be used to sync audio to text, like transcripts.

Actually, the programming and devising SimpleADL were the easy part. The hard stuff was the documentation and deciding on a license for the software.

I tried to find a balance in documenting the software: being thorough without writing a novel. I’m not sure I succeeded, but I can always improve it with time.

I used the W3C’s Amaya editor to write the documentation in XHTML. Sure, you can use OpenOffice to export a document to XHTML, but man is it bloated and messy. Amaya writes really clean XHTML.

As for the license, I chose the BSD license. As I understand it, this allows one to use the source code at will in future open or closed-source applications as long as you maintain the credits for AudioRegent. I was tempted to use the Mozilla Public License (MPL) which, again from what I can tell, is similar to the BSD license except that any source derived from AudioRegent would have to stay open-source though any peripheral code can be closed-source. I absolutely decided against the GNU General Public License which is viral and imposes its philosophy perpetually on all subsequent code, even peripheral code. Some have even argued that it works against its own objectives and is less "open" than the MPL.

Now I realize that, practically speaking, a skilled programmer could write better code from scratch in 30 minutes as opposed to the some 30 hours I needed, but I wanted to go about this quasi-professionally. And I learned more about licensing, which was cool.

Anyway, rather than try and explain the software itself and how to get it, I’d be better off pointing you to the documentation if you have any interest …

--------------

MusicSQL: initial thoughts

one comment

One of the nice things about an emerging standard, namely MusicXML, having a command center (Recordare LLC) is having a central place to learn about what’s new.

On Friday, I was looking at Recordare’s page of MusicXML related software for software that worked from the command line and noticed something new and really interesting: MusicSQL.

According the the Goodle Code page that hosts this project, MusicSQL is:

… a system for conducting complex searches of symbolic music databases. The database can import and export MusicXML files. In the current version searches are constructed using a command line interface or through simple Python scripting tools.

Basically, at least as I understand it, MusicSQL is a Python program that sits on top of a MySQL database – now I really hope Oracle doesn’t kill MySQL if it buys Sun.

I was so excited to get MusicSQL working that I didn’t notate all the little problems I had along the way. The documentation for MusicSQL is very good and is written for Windows, Mac, and Linux (Ubuntu) users. But I’m inconceivably impatient, so I just mowed through the installation with little care for remembering what I was doing.

I do remember that I had to install Python 2.5, whereas I already have Python 2.6 installed – now I have both. I put/installed all the dependencies in my Python 2.5 directory just to compartmentalized everything – the exception being MySQL, which I installed wherever the default is.

So far, I only ran the first query in the documentation that uses "scientific" musical notation in the form Nx, where "N" is the alphabetical note name, say C, and "x" is an integer that denotes what octave the note is a member of. In other words, a C-Major scale would be "Cx Dx Ex Fx Gx Ax Bx Cx+1", something like "C5 D5 … B5 C6", etc. You can place an integer before the note name to denote its duration.

Running the query from the command line, I was really happy with the speed and the output of MusicSQL for the test query.

One problem I did have, though, is I kept getting errors for another great feature of MusicSQL. Basically, after you run your query, you can see a PDF of the results (i.e. the music excerpt pertaining to the query results). The PDF is made by Lilypond, a text-based notation software that produces – in my opinion – the absolute best looking engraving out there, that’s why I use it (and yes, it’s free).

Now Lilypond doesn’t natively read MusicXML, it uses its own encoding. So MusicSQL takes advantage of a Python script that comes with the Lilypond install called "xml2ly" that converts MusicXML to Lilypond format. I left a message on the project forum for MusicSQL, so I’m hoping I can figure out what I need to do to get the Lilypond outout of the query results to work. At any rate, I do wonder how effective it can be since the conversion from MusicXML to Lilypond can sometimes get ugly.

I wonder if an alternative solution is to use the command line options for the MuseScore notation software to generate a PDF of the query results. Musescore can also convert MusicXML to other graphics formats (PNG) and even audio (WAV, FLAC, OGG), so theoretically it could be leveraged to make audio files for the corresponding query results.

At any rate, I’m really looking forward to the future developments of MusicSQL.

And as for using MuseScore’s command line in conjunction with MusicXML and how it can add value to a web collection of MusicXML docs – there will be more to that later …

--------------

running XQuery online

2 comments

A while ago, I posted about my first experience with XQuery and how I'd used the .NET version of the Saxon processor on my local Windows machine.

Obviously, I want to extend that experience to running XQueries online. So far, I know this can easily be done with a native XML database server like eXist. That is to say, when I installed eXist on my local machine and made it go live: bam! – instant XML server with built-in XQuery functionality, accessible from anywhere.

Another way is to use one of the more popular web-scripting languages to execute XQuery syntax. I nearly killed myself this weekend trying to install the Zorba PHP binding for XQuery (i.e. run XQuery natively from within a PHP script). I just couldn't get all the dependencies successfully installed on my virtual install of Ubuntu Netbook Remix (BTW: I use Sun's VirtualBox for virtualization). Perhaps I'll be able to make it work another time.

Now, even though it makes all the sense in the world to stick with a native XML server like eXist if I want to make a large collection of searchable XML documents online (and I do), I'm feeling non-sensical. What I decided to try was to run my computer as a more traditional server using the X-Apache-MySql-PHP model, specifically WampServer.

From there, I placed my sample document, "books.xml" and my "test.xquery" file from last time:

<ul>
{
for $x in doc("books.xml")/bookstore/book/title
order by $x
return <li>{data($x)}</li>
}
</ul>

in WAMP's "www" directory – i.e. the directory which is accessible from the browser, the place where one would put all their HTML files, etc. for the world to see.

Of course, I was still missing the actual XQuery processor at this point. What I tried is to put the relevant executables for Saxon in the "www" directory as well.

… Now, I'm sure this is totally unsafe or something, but I was just testing and I only make my computer "go live" as a server for short periods of time.

Anyway, from there I used PHP to call the Saxon processor and to display the results in the browser. It actually worked!

Here's the code:

<?php
echo "Hi. I'm going to use XQuery to list the books alphabetically.";
exec("query.exe test.xquery !indent=yes", $results);
foreach ($results as $value)
    {
       echo "$value";
    }
?>

You can see that the PHP "exec" command called the Saxon executable named "query.exe" and executes the "test.xquery" file.

It saves the results in a variable called "results" and then prints each value of the results in the browser.

For now, I'm OK with this, but I need to eventually do the same thing with the Java version of Saxon, I suppose, if I'm ever going to run this on a Linux server. It shouldn't present any new hurdles, but I need to try it to make sure, of course.

If anyone out there has any thoughts or recommendations on a more elegant method to achieve these results – and what the security risks of the approach I've outlined presents (since I didn't use a CGI-bin), please speak up. I'm all ears.

…

Update, November, 2009: No problems with using the Java version of Saxon. I did place it in a "bin" directory on my local server so that the Java file wouldn't reside in the directory that users have direct access to.

--------------

an ID3 tag reporter: MyMusicReporter.py

2 comments

I usually try to post to this blog once a week on average. That forces me to try to learn something or learn about something during the week that I can post about.

Sometimes though I have to resort to things that I've been sitting on for a while, like this Python script I wrote called MyMusicReporter.

This program makes use of the id3reader module available here. In other words, the program won't function without this module.

For those unfamiliar with Python, it supports the use of modules, which – as the name implies – is basically allowing the current Python program to take advantage of another, pre-existing Python program. At least that's how I understand it for now …

Anyway, MyMusicReporter does the following:

it asks you to point it to a root directory on your computer,
it then finds all the MP3s in that folder and its subfolders,
it makes an XML file in the same directory as MyMusicReporter called MyMusicReport.xml which contains the Title, Performer, Track #, Year, and relative file path for each MP3,
it makes an XML stylesheet in the same directory as MyMusicReporter called MyMusicReport.xsl which makes MyMusicReport.xml look pretty in your web browser.

Here's an example of how the XML looks with my home-grown schema:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="MyMusicReport.xsl"?>
<MyMusicReport>
    <mp3>
        <title>R2-DVD2</title>
        <performer>Nitin Arora</performer>
        <track>1</track>
        <year>2008</year>
        <path>r2_dvd2.mp3</path>
    </mp3>
</MyMusicReport>

Anyway, here's the source code for MyMusicReporter.py.

You assume all responsibility for use.

--------------

blog.humaneguitarist.org

Archive for the ‘XML’ Category

fun with lxml, part 2

Related Content:

fun with lxml

Related Content:

of ADLs and SMIL and stuff

Related Content:

MXMLiszt version 0.9.1 released

Related Content:

PubMed to Excel: PubMed2XL version 0.9

Related Content:

MXMLiszt release 0.9.0

Related Content:

segmenting audio with AudioRegent, SoX and XML

Related Content:

MusicSQL: initial thoughts

Related Content:

running XQuery online

Related Content:

an ID3 tag reporter: MyMusicReporter.py

Related Content:

Archives

Pages

Categories

Recent Posts

Recent Comments

Tags