blog.humaneguitarist.org

Let's forget the fact I've blogged more this month than I intend to in a whole year …

What I really want to mention is that I'm reading Dracula by Bram Stoker and noticed these very interesting bits (or should I say 'bites'?) in Chapter 17.

In this chapter the character of Mina Harker is becoming acquainted with a friend of her now dead friend, Lucy. This friend, Dr. Seward, uses a phonograph to record his patient notes, much as my dad used to use a micro-cassette back in the late 1970's and 1980's. Mina, on the other hand, uses her cutting edge writing tool, the typewriter, to make her diary entries easily readable.

The funny thing is that Seward confesses to Mina that he doesn't have a way to get to specific points within each recording, i.e. he doesn't have a way to denote and retrieve audio at a specific time with advanced knowledge of what passages exist at those points. Um, sound familiar?

MINA HARKER'S JOURNAL

29 September.

Again he paused, and I could see that he was trying to invent an excuse. At length, he stammered out, "You see, I do not know how to pick out any particular part of the diary."

…

I could not but smile, at which he grimaced. "I gave myself away that time!" he said. "But do you know that, although I have kept the diary for months past, it never once struck me how I was going to find any particular part of it in case I wanted to look it up?"

Mina goes on to transcribe his recordings so that the text can be compared with other diary entries by principal characters as they try to formulate the totality of Dracula's agenda.

DR. SEWARD'S DIARY

30 September.

Harker has gone back, and is again collecting material. He says that by dinner time they will be able to show a whole connected narrative. He thinks that in the meantime I should see Renfield, as hitherto he has been a sort of index to the coming and going of the Count. I hardly see this yet, but when I get at the dates I suppose I shall. What a good thing that Mrs. Harker put my cylinders into type! We never could have found the dates otherwise.

Update (or "later" as in the novel): It might be a nice homage to sync the transcript to the audio of Orson Welles' radio play based on the book.

Geolocation/geocoding is so "hip" these days. Everyone's so obsessed where where they and other things are. There's almost a comparison with 3-D filmmaking …

Funny. Not too many folks seem all that concerned with when things are.

Anyway …

At work, we have a database with all the libraries we serve and their addresses. And the other week we needed to quickly make a map with all their locations.

If necessity if the mother of invention, laziness is it's favorite uncle.

Enter BatchGeo. We were able to take those values from our database and get a map generated in minutes. But it gets better.

One of the nice things about this process is that in addition to a map, you also get a KML file download option. Taking this little XML file, it's a simple process (via XSL or other) to make a delimited file containing the inputted names of institutions and their latitude and longitude (altitude is also available).

From there, it's not brain surgery to get those coordinates into a database and using an SQL JOIN to be able to push out an institution's name and now its coordinates, too, whenever.

Just in case someone wants/needs to do something similar with an address book or a list of businesses, etc.

Related Content:

• simple point and search with a maps API
• dynamic menus with TKinter

Last night I installed lxml on my Amazon Linux AMI (ami-31814f58) and it was just as not-straightforward as when my co-worker and I put it on our CentOS server a few weeks ago.

So by referring to the yum log, I think the following covers what I needed to install with yum:

gcc-4.4.5-6.35.amzn1.i686 #for building lxml
python26-devel-2.6.7-1.36.amzn1.i686
libxslt-1.1.26-2.6.amzn1.i686 #this can't be necessary given the line below, right?
libxslt-devel-1.1.26-2.6.amzn1.i686
libxml2-devel-2.7.6-1.9.amzn1.i686

… and then I used easy_install (which already was on the system 'far as I know) to install lxml a la: easy_install lxml.

For whatever reason, the easy_install part took several, several minutes. But I was watching "Season of the Witch" so I didn't mind.

Related Content:

• here ye, here ye, get your free AWS account today
• pretty printing XML with Python, lxml, and XSLT
• a Halloween treat: Burn, Witch, Burn

I've totally over-blogged this month, but just as a note to self …

I've noticed that when using a syntax highlighter with WordPress – I'm using Syntax Highlighter Compress – that using single quotes can sometimes cause the highlighting to break down.

In the example below I have identical code written twice. Well, almost identical. In the first, I'm surrounding "data-startTime" with single quotes and in the second I'm using doubles. The first has some highlighting problems where single quotes are.

Oh well, I guess I'll just have to keep an eye on that.

function jAUs_3(this_currentTime,thisAudioTag_stopTime,i){

  if (thisAudioTag_stopTime){
    //if there's a data-stopTime attribute then ...
    if (this_currentTime > thisAudioTag_stopTime){
      //... reset audio to data-startTime when data-stopTime is reached.
      audioTagArray[i].currentTime = audioTagArray[i].getAttribute('data-startTime');
      audioTagArray[i].pause();
    }
  }
  else if (audioTagArray[i].ended == true){
    //if there's no data-stopTime, move back to data-startTime when playback has ended.
    audioTagArray[i].currentTime = audioTagArray[i].getAttribute('data-startTime');
    audioTagArray[i].pause();
  }
}

function jAUs_3(this_currentTime,thisAudioTag_stopTime,i){

  if (thisAudioTag_stopTime){
    //if there's a data-stopTime attribute then ...
    if (this_currentTime > thisAudioTag_stopTime){
      //... reset audio to data-startTime when data-stopTime is reached.
      audioTagArray[i].currentTime = audioTagArray[i].getAttribute("data-startTime");
      audioTagArray[i].pause();
    }
  }
  else if (audioTagArray[i].ended == true){
    //if there's no data-stopTime, move back to data-startTime when playback has ended.
    audioTagArray[i].currentTime = audioTagArray[i].getAttribute("data-startTime");
    audioTagArray[i].pause();
  }
}

Related Content:

• a switch to Syntax Highlighter Compress, a WordPress plugin
• syntax highlighting with WordPress: that's not that!
• syntax highlighting with WordPress

I'm currently working with some folks on a pilot project to build a shared index of digital collection metadata from libraries in North Carolina. My part entails harvesting the metadata and indexing it with Solr.

Since most of the stuff is North Carolina centric, I thought it might be neat to use the API that the index will have to drag a marker on a map and then use the marker location to send a search to the index. My co-worker also wants to do something similar so people can search using a map marker for a project of hers. So, I thought I'd investigate.

I wanted to see how easy it was to do this with the Google Maps API and, well, it is pretty easy. Especially, since I found this marker dragging example.

There's lot of stuff that can be done, like dynamically populating the page with a set of results via AJAX but for now I'm just using the city name plus a pre-written string to create a search string that the user has to manually click.

In the little example I made (below), I'm using my home state of South Carolina. If you move the marker within SC and drop it, a simple search string is created for sending a search, ironically, to Bing Maps for the term "restaurants" and that city.

I think a search like this is really for fun, but it might be a nice way to search and learn a little geography along the way. I've also got an idea for a game using this kind of search that's sort of a spin on Concentration for digital collections, but I'll write that up later. I'm hungry and need to start enjoying my Saturday. Looking at these restaurants searches is making me even hungrier.

For better or worse, the one application of mine that people actually use is the one I wrote pretty casually with Python over a couple weekends from bed because I was too lazy or hungover to get moving on those days.

That software, PubMed2XL, lets people do a few things with downloaded citations from PubMed.gov that isn't currently offered directly from the site. I've gotten some nice feedback from librarians, researchers, and information-y people at companies that have found it useful.

This post isn't a plug though; it's more an acknowledgement of something that I didn't really realize in full at the time. And that is when one writes software that people go on to actually use, one better be prepared to support it. Now, the software's simple enough that there haven't been real bugs save one, but it does eat at me that I can't offer a simple way for it to work on multiple platforms.

While the Windows version is really easy to setup – thanks to py2exe and Inno Setup – getting it running on Linux is a bit more work, given all the distro variations and dependency installation. But getting it running on a Mac – particularly with an easy to use installer – isn't going to be possible unless I can find someone to compile it for a Mac who will also test it and compile future versions. Sure, there's the possibility of using Wine, but that's still asking a lot from end users.

Normally, I wouldn't care. Apple doesn't make it easy for people to develop for Macs unless you fork over the change for a Mac – and I ain't buying a copy of OSX and doing the Hackintosh bit. But, since the software is ultimately about health-related research, I do care.

Unfortunately I made – with the advantage of hindsight – two coding decisions that create problems.

First, I chose PyQT as the GUI toolkit for the software simply because it looks prettier than Python's native Tkinter. My reasoning at the time was the people were more likely to trust better looking software even though it's just a small window with some basic menu options. Eventually, I added a progress bar, too, so downgrading to Tkinter has become less of an option.

Second (and this is the big one), I used lxml since the PubMed2XL setup files employ XSL to tell the software what data to put in a spreadsheet cell. Granted, lxml is freakin' fantastic, but since it's not a pure Python module I can't just distribute it in a folder and import the module locally. Not that I had much of a choice: there's no built in XSLT-capable module that ships with Python 'far as I know.

So I've been asking myself how to make the serpent (Python) and the apple (OSX) get along.

I've consider just making PubMed2XL a web-app, but that will entail expenses for me that simply offering people a desktop app doesn't entail.

So, I think the solution lies in a cup of Joe. That's to say that a Java app is the obvious solution, specifically using Jython.

That would leave me to replace PyQT with Swing. I'm fine with that. It's not like PyQT is all that Pythonic in the first place. There's a nice Jython/Swing tutorial here.

And as for the XSLT component, this tutorial on XSLT with Jython and native Java libraries should help immensely.

So, I should be able to use Jython to make a cross-platform version of PubMed2XL. I don't necessarily want to, but given the type of research I'd like to help facilitate (in a very small way, I know), I think I probably should.

Related Content:

• PubMed2XL 1.0 available
• PubMed2XL 0.9.1 available
• dealing with a PubMed2XL bug
• and yet more PubMed to Excel news
• PubMed to Excel: PubMed2XL version 0.9

I really should be packing for an overseas vacation that begins tomorrow, but I wanted to jot some stuff down before I forget – and I intend to forget a lot!

Anywho, in a previous post I wrote about putting XSL inside a CDATA block inside an XML config file. I had the following example:

<map name="LibriVox">
  <XSLT>./XSLT/LibriVox_to_Solr.xsl</XSLT>
  <nextXSL>
  <![CDATA[
  <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output method="text"/>
    <xsl:template match="/">
      <xsl:variable name="baseURL" select="'%s'" />
      <xsl:variable name="URL_params" select="'%s'" />
      <xsl:variable name="offset_" select="substring-after($URL_params,'=')" />
      <xsl:variable name="offset" select="substring-before($offset_,'&amp;')" />
      <xsl:variable name="limit_" select="substring-after($URL_params,'&amp;')" />
      <xsl:variable name="limit" select="substring-after($limit_,'=')" />
      <xsl:variable name="output">
        <xsl:value-of select="$baseURL" />
        <xsl:text>?offset=</xsl:text>
        <xsl:value-of select="$offset+50" />
        <xsl:text>&amp;limit=</xsl:text>
        <xsl:value-of select="50" />
      </xsl:variable>
      <xsl:value-of select="$output" />
    </xsl:template>
  </xsl:stylesheet>
  ]]>
  </nextXSL>
</map>

This is part of this pOAIndexter script I'm working on for, well, work.

The XML code above is from one of the config files where the <XSLT> element points to an XSL file used to process metadata retrieved from a website. In this case, the point is to make a Solr-compatible XML document that can be used for indexing purposes. The second element, <nextXSL>, is used to return to pOAIndexter the URL for the next batch of metadata for a given feed, i.e. the next page or the next set within a collection, etc.

And as you can see there are two weird looking variables at the top:

      <xsl:variable name="baseURL" select="'%s'" />
      <xsl:variable name="URL_params" select="'%s'" />

The reason being that the pOAIndexter script actually populates these with the actual base URL for the batch just retrieved and the parameters, respectively, before the XSL within the <nextXSL> element is run, returning the string of the next URL.

I chose XSL because I think, as a librarian, it seems to be common to a lot of metadata and digital library folk and such people could extend the capabilities of pOAIndexter without having to know Python. But all along I wanted people to be able to process the metadata and return the next URL with whatever scripting language they want, provided the interpreter exists on their system.

So, say for example you like PHP instead. Instead of the using XSL you could use something like this:

<map name="LibriVox">
  <PHP>./PHP/LibriVox_to_Solr.php</PHP>
  <nextPHP>
  <![CDATA[
  <?php
  $baseURL=%s;
  $URL_params=%s;

  //some PHP code here

  echo $output; //where $output is the next URL ...
  ?>
  ]]>
  </nextPHP>
</map>

That way PHP could be used to make the Solr-XML file and to return to pOAIndexter the next URL string so that the next batch of metadata from a feed could be processed/transformed. Of course, you could mix and match, too – XSLT for making the Solr-XML file and PHP just for getting the next URL.

That's actually pretty simply to do with the common scripting languages like Python, PHP, Perl, Ruby, etc. But what I really wanted to support was JavaScript because, well, it would just be cool, but also because that's another one of those common languages that a lot of people might know even though there might be great variation amongst the other scripting languages they know when compared to a lot of their colleagues.

But I didn't know how to execute Javascript via the command line so that pOAIndexter can capture the next URL via the standard output stream.

Well, enter PhantomJS.

That is all. Time to pack.

Related Content:

• bidi bidi bidi and more on pOAIndexter-ing metadata
• pOAIndexter: grabbing and indexing online metadata

I'll keep this short.

For me version control is just zipping up files after I make a change and then versioning the ZIP files via filename.

But I probably should look into version control of some sort. But I want something easy to learn.

Git seems all the rage these days, but honestly I'd have to read the documentation a few times to learn it properly … and that ain't happening. Also, it's more than I need given that what I'll be working on are very small projects.

But I really think Fossil is very promising. It's super easy to learn and the built in web interface – replete with a Wiki and ticketing system – is pretty sweet.

I'll still be zipping up my files, but I think they'll probably be making their way into Fossil repositories from now on, too.

… and as for the blog title, it's a bit of a play on words of course. Did you "Git" it? Do you "dig" Fossil like I do?

Update, November 3, 2011: In my excitement and haste, it seems I didn't read carefully enough. The free year only applies to "micro" instances and may even only apply applies to Amazon's own Linux distro at that. Please read the terms more carefully than I did before you proceed. My apologies. If it makes anyone feel better, my penance is a $10.00 bill from Amazon.

Update, November 3, 2011: Thanks to Eileen of Amazon for removing the charge upon learning that I'm an idiot who doesn't read carefully!

…

Yesterday I stumbled upon the news that Amazon is offering one year free of use – for new users – of it's Elastic Computer Cloud (EC2):

Free Tier*

As part of AWS’s Free Usage Tier, new AWS customers can get started with Amazon EC2 for free. Upon sign-up, new AWS customers receive the following EC2 services each month for one year:

• 750 hours of EC2 running Linux/Unix Micro instance usage
• 750 hours of Elastic Load Balancing plus 15 GB data processing
• 10 GB of Amazon Elastic Block Storage (EBS) plus 1 million IOs and 1 GB snapshot storage
• 15 GB of bandwidth out aggregated across all AWS services
• 1 GB of Regional Data Transfer

That's pretty cool because I could really use something that allowed me to demo stuff that my five-bucks-a-month GoDaddy hosting account won't allow me to do.

Truth be told, I already have a Windows VPS with KickAssVPS.com but at $20.00 per month I always remind myself that in the course of a year I could have paid for a small laptop/netbook and used that as a server for special projects, but I really don't want to do that because I don't want to be thinking along the lines of "Man, this storm is really bad. I should unplug everything even if it kills the server" or "Man, this day is really beautiful. I think I'll take the netbook to the local coffee shop", etc. I just don't want to have to think about that kind of stuff.

Anyway, KickAssVPS has totally lived up to their name over the last year and half I've used them. Their service and response time is great but I need it for so very little that I can't justify the expense. Especially not for a year now that I've got a Windows Server 2003 and Ubuntu AMI (Amazon Machine Image) up and running through Amazon.

I'm not hosting anything live at the moment through AWS/EC2 but I will be. I'll be moving the MXMLiszt demo over to the Windows AMI though I think I can get it successfully ported over to Ubuntu. But what this also allows me to do is offer web services, applications, and demos that I simply can't run on the Godaddy account but that I've been playing around with lately – things like pOAIndexter. Also, the Ubuntu AMI I installed has Python 2.6 and I can install my favorite Python modules like libxml. In fact, I've got root access so I can install whatever I need to. Can't do that with my current GoDaddy plan.

So why am I writing this other than to be an unpaid fanboy?

Well, I wanted to share a few things that made the process easier for me such as:

• this YouTube video on the basic, first-time stuff with EC2. It's a little out-of-date but it's still useful. Ignore the stuff about a FireFox plugin because AWS now has a web-based management panel.
• this post on using WinSCP for connecting, via SSH, to an AMI.
  • see this related post, too.
• and these particular 32-bit AMI identifiers that are working for me:
  • ami-f11ff098 (Windows 2003)
  • ami-1a837773 (Ubuntu 10.10)

I had trouble getting some of the other AMIs to work and those two are working just fine it seems. Note that as reported per the Ubuntu AMI link, a reboot does seem necessary before connecting via SSH is possible. For the Windows one, I just use Remote Desktop.

By the way, if you don't need or want Ubuntu, Amazon offers their own Amazon Linux, Suse, and Red Hat AMIs. I found the AMIs made by Amazon seem to work without any hassle.

I should also point out a few things with using WinSCP with that particular Ubuntu AMI. The easiest way for me to do this is show a screen shot of my WinSCP login settings:

Note the user name of "ubuntu". The default user for AMI accounts is "root" but not for this one.

Also, I manually seemed to have to set the File Protocol to "SCP" otherwise I was running into error messages like "Received too large … SFTP packet. Max supported packet size is 102400 B" and "Error skipping startup message. Your shell is probably incompatible with the application (BASH is recommended)".

I should mention I just like WinSCP for the GUI file/folder viewer and the drag-and-drop stuff. I don't like the terminal that's built into WinSCP at all so I use Putty for the terminal. Luckily, WinSCP let's you select "Open in Putty" from the "Commands" menu.

Well, that's it for now. I guess I really wrote this as a note-to-self but I also thought that it might be useful news for any of my librarian-programmer friends who might also benefit from a having a free year to host more advanced applications and demos.

Related Content:

• installing lxml on my Amazon Linux instance

I've been casually learning Solr in order to easily create a faceted search/retrieval interface for some digital collections with OAI feeds but now I kinda wish I was learning to do that with Microsoft's PivotViewer instead.

I won't do that, at least for now, but I probably won't be able to resist in the near future.

Here's a cool demo of it with some Netflix OData.

For a description of how it's done, check out this post: Pivot, OData, and Windows Azure: Visual Netflix Browsing.

Adding this follow-up thought the next day: Actually, Pivot would be a really good way to make eBooks and eAudio titles discoverable … all those book covers. Sheet music (first page) would be cool, too.

Oh, the possibilities.

Related Content:

• HammerFlicks
• in the can: another HammerFlix update
• Netflix: thumbs down?
• HammerFlix 2: Terror of the lost API Keys
• HammerFlix