Funny. Not too many folks seem all that concerned with when things are.

Anyway …

At work, we have a database with all the libraries we serve and their addresses. And the other week we needed to quickly make a map with all their locations.

If necessity if the mother of invention, laziness is it's favorite uncle.

Enter BatchGeo. We were able to take those values from our database and get a map generated in minutes. But it gets better.

One of the nice things about this process is that in addition to a map, you also get a KML file download option. Taking this little XML file, it's a simple process (via XSL or other) to make a delimited file containing the inputted names of institutions and their latitude and longitude (altitude is also available).

From there, it's not brain surgery to get those coordinates into a database and using an SQL JOIN to be able to push out an institution's name and now its coordinates, too, whenever.

Just in case someone wants/needs to do something similar with an address book or a list of businesses, etc.

But there is cake, too.

Anyway, for a current project I want to place some XSL inside an XML config file.

But of course, you can't just drop XML inside XML without coating it in something.

So for another project, PubMed2XL, I did something like this:

<cell>{{?xml version="1.0" encoding="UTF-8"?}}
  {{xsl:stylesheet version="1.0" encoding="UTF-8" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"}}
  {{xsl:output method = "text" /}}
  {{xsl:template match="/"}}
    {{xsl:value-of select="//PMID" /}}
  {{/xsl:template}}
  {{/xsl:stylesheet}}
</cell>

Putting XSL inside double curly brackets works just fine, but now I know a better way: just put it inside a CDATA section!

<map name="LibriVox">
  <XSLT>./XSLT/LibriVox_to_Solr.xsl</XSLT>
  <nextXSL>
  <![CDATA[
  <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output method="text"/>
    <xsl:template match="/">
      <xsl:variable name="baseURL" select="'%s'" />
      <xsl:variable name="URL_params" select="'%s'" />
      <xsl:variable name="offset_" select="substring-after($URL_params,'=')" />
      <xsl:variable name="offset" select="substring-before($offset_,'&amp;')" />
      <xsl:variable name="limit_" select="substring-after($URL_params,'&amp;')" />
      <xsl:variable name="limit" select="substring-after($limit_,'=')" />
      <xsl:variable name="output">
        <xsl:value-of select="$baseURL" />
        <xsl:text>?offset=</xsl:text>
        <xsl:value-of select="$offset+50" />
        <xsl:text>&amp;limit=</xsl:text>
        <xsl:value-of select="50" />
      </xsl:variable>
      <xsl:value-of select="$output" />
    </xsl:template>
  </xsl:stylesheet>
  ]]>
  </nextXSL>
</map>

Duh. I guess like a good XML parser that I always just ignored anything inside a CDATA section. Never thought I'd need to use one.

Putting the XSL inside a CDATA section worked like a charm in terms of being able to read it with a script and perform an XSLT with it.

Luckily, the PubMed2XL script can use either the CDATA way of embedding XSL or my Curly solution – not that I knew that when I wrote it!

It's certainly easier to cut/paste the XSL in the CDATA block without having to replace the brackets with curly quotes or vice versa. It's also just easier to read, which makes it easier to edit and troubleshoot. And it tastes better, too.

I couldn't find any native lxml solutions to make my XML look pretty. All I found were some functions on various code sites written by people to pretty print the XML using a bunch of regular expressions. Yuck.

So I thought, "Why not use XSLT to pretty print my XML?" and I found an XSL written by none other than Michael Kay on this page (see comment #4).

And it seems to work just fine as a function to return pretty XML, not to mention it's super short and sweet.

Anyway, here's an example of using the XSL for pretty printing.

from lxml import etree
from lxml.etree import XSLT

def prettify(someXML):
  #for more on lxml/XSLT see: http://lxml.de/xpathxslt.html#xslt-result-objects
  xslt_tree = etree.XML('''\
    <!-- XSLT taken from Comment 4 by Michael Kay found here:
    http://www.dpawson.co.uk/xsl/sect2/pretty.html#d8621e19 -->
    <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output method="xml" indent="yes" encoding="UTF-8"/>
      <xsl:strip-space elements="*"/>
      <xsl:template match="/">
        <xsl:copy-of select="."/>
      </xsl:template>
    </xsl:stylesheet>''')
  transform = etree.XSLT(xslt_tree)
  result = transform(someXML)
  return unicode(result)

myXML = etree.XML('<a><b><c><d/></c></b></a>')
print prettify(myXML)

The example above would output the following:

>>>
<?xml version="1.0"?>
<a>
  <b>
    <c>
      <d/>
    </c>
  </b>
</a>


By the way I don't even need to see the XML I'm processing most of the time, so why all the pretty printing fuss?

Well, because it bothers me …

And all good XML should look like an X-wing starfighter. If it doesn't your probably doing something wrong or your schema just sucks.

It isn't called an X-wing for no reason.

For most things, that's fine. I mean for indexing titles, creators, and descriptions, etc. I just need to index the value of a given element like <title> so that I can search for that element's value.

But for timed text, it's different. Or at least, it can be.

Say I have this DFXP snippet for an audio file with an "id" value of "XYZ".

<p begin="10.0s" end="30.0s">Hello world!</p>

I would need the user to search for the string "Hello world!" or part of it but I would also need to index at least the value of the "begin" attribute so that I can pass that to a page that will play the file "XYZ" starting at the 10 second mark – if the user clicks on the "Hello world!" line in their search result. And I don't want the "10" second value to be something they search against since they might be searching for the string "10" within the text itself.

So I'm wondering how to do that with Solr.

Maybe when I learn more I'll discover a better way to do this, but for now I'm thinking I could do the following:

First, I would pretty much index the timed text twice in Solr.

<doc>
  <field name="id">XYZ</field>
...
  <field name="timedText-stripped">Hello world!</field>
  <field name="timedText">Hello World! {10}</field>
</doc>


After indexing the "id" of the audio file this would index:

• just the text "Hello world!"
• the text of "Hello world!" with the "begin" attribute value in curly quotes.

I guess this way the user could be made to search across the "timedText-stripped" field but, via the XSL that can be passed to Solr to display results, the "timedText" field could be displayed in a manner that would make the text "Hello World!" linked to whatever file will play file "XYZ" starting at the 10 second mark. Basically, by planting the "begin" value in curly quotes, I can parse the string for the text and the "begin" value as separate things.

So, here's a really crappy XSL snippet that would do something like that. It assumes a variable "$id" exists that equals "XYZ", the identifier for the example audio file.

<xsl:for-each select="//field[@name='timedText']">
  <xsl:variable name="whole">
    <xsl:value-of select="."/>
    <!-- Gets entire element string -->
  </xsl:variable>
  <xsl:variable name="text">
    <xsl:value-of select="substring-before($whole,'{')"/>
    <!-- Gets text prior to seconds -->
  </xsl:variable>
  <xsl:variable name="begin">
    <xsl:value-of select="substring-before(substring-after($whole,'{'),'}')"/>
    <!-- Gets seconds value from end of string -->
  </xsl:variable>
  <a href="someMediaPlayer.php?id={$id)&amp;begin={$begin}">
    <xsl:value-of select="$text"/>
  </a>
  <!-- So, I'm saying that
  "someMediaPlayer.php?id=XYZ&start=10"
  would launch a player that would start file XYZ at the 10 seconds mark.
  -->
</xsl:for-each>

The search output would be some HTML code like so:

<a href="someMediaPlayer.php?id=XYZ&amp;begin=10>Hello World!</a>

It seems weird to index something twice, more or less, but as user Erick says in the nabble.com thread, "You've gotta take off your DB hat and not worry about duplicating data."

But now as I write this, I'm wondering if I can't just index as follows:

  <field name="text">Hello world!</field>
  <field name="begin">10</field>

and trust that for each "text" field, there will be a matching "begin" field and that they can't just be used in tandem to create the same HTML link as above. Sounds like I need to play around some more.

I like the idea of a one-stop-shop for all kinds of XML processing, but I think unless I had a specific need to use it I'd probably use a Python script or something to sequentially do some batch XML work on a given document. That's exactly what XProc is a solution against, but I guess it all depends on one's needs. I should certainly think about it in terms of doing things with MusicXML though.

Anyway, I've only been through one slide – and it's long at about 170 slides, but I found it well done and easy to understand.

Also, there's a list of XProc implementations here – Java, Java, Java …

Apparently, there used to be a Python implementation on GitHub, but it's pulling a 404. Bummer. Well, at least GitHub's 404 message is a cool homage to Star Wars!

Lastly, this daisy-pipeline for Daisy talking books looks interesting, too.

So is this post just a fancy way for me to save bookmarks for my future use or what?

Per a request, I need to tweak some software of mine to allow a user to specify a parent element in an XML document and in turn retrieve child element values. Big deal. That's what XSLT is for – blah, blah, blah. But this is particularly for PubMed XML exports and turning those into Excel files.

Anyway, the value of a given child element needs to be able to be specified (i.e. by position) and placed into an Excel cell. Alternatively, all children values need to be able to be placed into one cell separated by a delimiter.

So before I try and tinker with the software I want to work a solution out using test code:

from lxml import etree

##### Step 1
    # make an XML example
xml = '<a>  \
            <b>  \
                <c>cee1</c>  \
                <d>dee1</d>  \
                <c>cee2</c>  \
                <d>dee2</d>  \
            </b>  \
            <b>bee</b>  \
            <c>cee3</c> \
        </a>'

##### Step 2
    # parse the XML example
parseXML = etree.XML(xml)

##### Step 3
    # make a list of the first (i.e. the Zero-th) <b> element
b_list = parseXML.findall('.//b')[0]

##### Step 4
    # get a list of all the children in that first <b> element
b_childList = b_list.getchildren()

##### Step 5
    # make a new list called "c_list" with only <c> elements
    # that are children of our first <b> element

c_list = [] # make an empty list to put things in and
# place into that list only element *values* for child elements
# of first <b> element from children that are <c> elements only
for child in b_childList:
    if child.tag == 'c':
        c_list.append(child.text)

##### Step 6
    # print desired results

for c in c_list: #print all values, one per line
    print (c)

print ('-'*4) # print dash line for reading ease
print ('; '.join(c_list)) # print all values on one line with delimeter

print ('-'*4)
print (c_list[1]) #print only the second <c> element value

Here are the results:
>>>
cee1
cee2
----
cee1; cee2
----
cee2

Anyway, that's another post for another time.

So, the other day I got some questions/feature requests for PubMed2XL and so I started thinking about ways to tackle a few of the issues. It kinda makes me feel like a real programmer when people in the real world are asking about the software – but only for a few minutes before I make myself come back down to earth.

Currently, the software places into a spreadsheet cell the value of one XML element, the position of which is defined by the user in the setup file. But there may potentially be a need it seems to be able to concatenate ALL the values for a given element into one spreadsheet cell. So I wrote a little function to help me get started with that.

The code uses this simple restaurant-based XML file from W3Schools and uses the awesome lxml Python library.

When run, it yields the following:

>>>
Calories for the first entree:
650
Calories for all entrees:
650; 900; 900; 600; 950

And here's the code:

#import required modules (lxml is non-standard; it likely needs to be installed)
import urllib #makes it easy to read documents from the web!
from lxml import etree #great XML parser and more!
                       #see: http://lxml.de/

#retrieve values from an XML file
def ElementCherryPicker(xpathArg, positionArg):
    '''
    This places all the element values for the element passed as the
    "xpathArg" argument into a list called "elementBox". It then returns
    the list item preceeding the one specified by the "positionArg" argument.
    This means passing a "1" equates to the first item in the list instead
    of the traditional "0". If "0" is passed then the entire list will be
    returned as a string with a delimiter of '; '.
    '''
    positionArg = positionArg - 1
    elements = parseUrl.findall(xpathArg) #make list of all matching elements
    elementBox = [] #create empty list
    for element in elements:
        elementBox.append(element.text) #place element values into the list
    if positionArg != -1:
        try:
            elementBox = elementBox[positionArg]
        except:
            elementBox = [] #if no element at stated position exists,
                            #then make the list empty again
    else:
        delimiter = '; '
        elementBox = delimiter.join(elementBox)
    return elementBox

#define, open, read, and parse an XML file
url= 'http://www.w3schools.com/xml/simple.xml'
readUrl = urllib.urlopen(url).read()
parseUrl = etree.XML(readUrl)

#print header and the values returned from ElementCherryPicker()
print 'Calories for the first entree:'
print ElementCherryPicker('.//calories', 1)
print 'Calories for all entrees:'
print ElementCherryPicker('.//calories', 0)

This morning I played around a little with Kino, an open-source video editor, and Adobe Audition, Adobe's flagship audio editor – which is based on their acquisition of Cool Edit.

The reason I wanted to play around with Kino is because it can export the project timeline to SMIL. I was mainly interested in seeing if it could be used as a pseudo audio editor – the idea being it could be a quick and dirty SMIL exporter. Well, it doesn't seem to support importing audio formats. I couldn't get it to import WAV or OGG files. It's still a cool application though.

The session exports from Audition are, as expected, pretty dense. For people like me who work in libraries there are issues involved in terms of setting limits for how much can and should be done in digital audio "preservation" (funny, I don't remember ordering jam and bread …). Well, at least I think there need to be limits, lest libraries want to start being creators, too, and admit that in doing so they are donating material of their own editorial designs back onto themselves. Anyway, by imposing limits I'm not sure XML session exports of thousands of lines for simple edits are a good idea.

I'd like to see other session formats without downloading demos for all kinds of audio editing software (some more expensive packages don't even seem to offer demos). For a small fee, there's always AATranslator.

But getting back to SMIL, I'm wondering how to use it in conjunction with AudioRegent without writing more code into the application – for now.

It would seem pretty easy to create a SMIL to SimpleADL XSLT and set up a chain to create derivative files.

Specifically, say I have a source file called source.wav. And I have two SMIL files as such:

source-1.smil.xml

<?xml version="1.0"?>
<smil xmlns="http://www.w3.org/2001/SMIL20/Language">
  <body>
    <seq>
      <audio src="source.wav" clipBegin="00:00:00.000" clipEnd="00:00:30.000."/>
    </seq>
  </body>
</smil>

and source-2.smil.xml

<?xml version="1.0"?>
<smil xmlns="http://www.w3.org/2001/SMIL20/Language">
  <body>
    <seq>
      <audio src="source.wav" clipBegin="00:00:30.000" clipEnd="00:00:50.000."/>
    </seq>
    <seq>
      <audio src="source.wav" clipBegin="00:01:00.000" clipEnd="00:02:00.000."/>
    </seq>
  </body>
</smil>

For both, the assumption is that two clips are to be made from source.wav: source-1 and source-2.

All I'd need to do is then setup a chain as such:

1. Do source-1.smil.xml to temp.adl.xml via XSLT.
2. Have AudioRegent make source.ogg by pointing it, via the command line options, to the source file, source.wav, and the SimpleADL file, temp.adl.xml.
3. Rename source.ogg to source-1.ogg – i.e. with the same prefix as the corresponding SMIL file.
4. Do source-2.smil.xml to temp.adl.xml via XSLT, overwriting temp.adl.xml.
5. Have AudioRegent make source-2.ogg by pointing it, via the command line options, to the source file, source.wav, and the SimpleADL file, temp.adl.xml.
6. Rename source.ogg to source-2.ogg – i.e. with the same prefix as the corresponding SMIL file.

Here's what temp.adl.wav would look like initially (step 1):

<?xml version="1.0" encoding="UTF-8"?>
<audioDecisionList filename="source.wav">
  <region id="_01">
    <in unit="seconds">0</in>
    <duration unit="seconds">30</duration>
  </region>
  <outputAsTracks>false</outputAsTracks>
</audioDecisionList>

And then it would look like this during the second pass (step 4):

<?xml version="1.0" encoding="UTF-8"?>
<audioDecisionList filename="source.wav">
  <region id="_01">
    <in unit="seconds">30</in>
    <duration unit="seconds">20</duration>
  </region>
  <region id="_02">
    <in unit="seconds">60</in>
    <duration unit="seconds">60</duration>
  </region>
  <outputAsTracks>false</outputAsTracks>
</audioDecisionList>

By the way, since the SimpleADL files are temporary, I don't see why – rather than converting time format to seconds – I couldn't just use something like this:

<in unit="time">00:01:00.000</in>
<duration unit="time">00:01:00.000</duration>

or something …

So here are the changes I made to address the issue related to the display of MODS metadata:

• Created mods.css file to display MODS on a transparent background.
• Changed displayMODS.php to display MODS files via an <iframe>. The previous version was using the mods.xsl stylesheet to parse the MODS element values in real-time.

You can read the documentation and download the source code for version 0.9.1 here.

If you'd like to use the software you can download it. Yes, it's free.

Here's a little video tutorial on installing and using the software:

PubMed2XL: Basic Installation and Use from nitin arora on Vimeo.

PubMed2XL's documentation is available at: blog.humaneguitarist.org/​projects/pubmed2xl/.

The documentation includes a download link to the program files.