blog.humaneguitarist.org

discoveries in digital audio, music notation, and information encoding

Archive for the ‘Python’ tag

getting real-time values from imported modules with a Python GUI

leave a comment

Situation: Over a year ago I wrote a Python script to allow one to convert XML citations from Pubmed.gov to a Microsoft spreadsheet.

I know some people are using it here and there, so I wanted to make it better.

The main problem with the old script is it's just sloppy (I knew even less back then than I do now!). It's also a script that wraps everything into one: non-GUI and GUI. If you don't pass command line options, then it launches the GUI, etc. Anyway, that makes the code hard to read for me since it intermixes data parsing with command line option stuff and GUI stuff.

So, for the next version I'm working on, I started with the premise to write it as a Python library so that it can be imported and one can use the function to make a spreadsheet inside another Python script a la:

import pubmed2xl
pubmed2xl.makeSheet("pubmed.xml", "pubmed.xls") #pass input and output (Excel file to be written)

It's also setup to make it easy to use command line options a la:

python pubmed2xl.py pubmed.xml pubmed.xls

The function and command line options also support showing the progress of completion while the spreadsheet is being made. This can be called as such:

pubmed2xl.makeSheet("pubmed.xml", "pubmed.xls", showProgress=True)

or

python pubmed2xl.py pubmed.xml pubmed.xsl --verbose

The problem for me, then, was how to show the progress inside a GUI application. Essentially, I needed the value of a the progress counter "variable" that was created inside a loop and updated each time the loop occurred – i.e. updating the progress counter. But I couldn't figure out how to retrieve the value of the progress counter variable in real time as the loop occurred. And I need it in real time so my GUI could show the progress update to the user – in real time!

I spent way too much time following leads that got me nowhere. I tried threads, running the python script as a sub-process, etc.  but I could never access the variable "progressValue" that equates to the percentage of task completion as citations are getting processed into a spreadsheet.

So, somehow I found my way to realizing that if my original script had a class and my second script added a method to the class then I could get the value of "progressValue" in real time.

Anyway, I've got two scripts below. The "first.py" script emulates a progress calculator by simply counting to 100. The script also has a class, "callback" and a global dictionary "_CALLBACK_DICT" into which I can place key/value pairs for whatever variables I want to retrieve during the loop.

The function "canYouSeeMe()" inside "first.py" also tries to execute the method "_CALLBACK.callback()" during the loop. In other words, if the method's there, run it, otherwise just ignore it.

The second script "second.py" is a little TKinter GUI app. It imports the first module and the instantiated class("_CALLBACK"). It also has a function called "getCallback()" that does what I want: i.e. retrieve the progress count in real time and show it in the GUI in real time. I then I equate "getCallback()" to the "_CALLBACK.callback()" method. So now, when I run the "second.py" script, the loop in "first.py" can give me the data I want to show in "second.py" in real time. Make sense? I hope so because it seems to be working OK.

Here's a screenshot from running "second.py" and below are the scripts themselves. I'd love any feedback on better ways of doing this, by the way.

Tkinter callback example

first.py

##### "first.py"

class callback():
  pass
_CALLBACK = callback()
_CALLBACK_DICT = {}

rangers = range(0, 101, 10)
def canYouSeeMe():
  for ranger in rangers:
    _CALLBACK_DICT["this_ranger"] = str(ranger)
    try:
      _CALLBACK.callback()
    except:
      pass

second.py

##### "second.py"

#import first module
import first
from first import _CALLBACK

#import Tkinter
from Tkinter import *

#create function and add as method to class "_CALLBACK"
def getCallback():
    importedValue = first._CALLBACK_DICT["this_ranger"]
    t.insert(END, importedValue + "%\n")
    if importedValue == "100":
        t.insert(END, "\nDone.")
    t.see(END)
    t.update_idletasks()
_CALLBACK.callback = getCallback #adding method to class

#create GUI buttons
class buttons():
   
    def __init__(self, root):
 
        #make frame/button
        frame = Frame(root)
        frame.pack()
       
        buttonText = "go"
        buttonAction = self.go
        self.makeButton = Button(frame, text=buttonText, command=buttonAction)
        self.makeButton.pack()
       
    #run go()
    def go(self):
      first.canYouSeeMe()

#create GUI
root = Tk()
buttons = buttons(root)
t = Text(root, background="black", foreground="blue")
t.pack()
geo = ("150x250")
root.geometry(geo)
root.mainloop()
--------------

Related Content:

Written by nitin

February 7th, 2013 at 3:00 pm

Posted in scripts

Tagged with , , ,

pixelation: custom XSLT functions with Python and lxml

leave a comment

I'll be brief.

Because the Python "lxml" module doesn't support XSLT 2.0 functions, I was looking at support for EXSLT

… but then stumbled on how to write my own functions and call them from stylesheets.

Freakin' cool.

I like calling it "pxslt" for "Python XSLT" and pronouncing it like "pixelate".

:P

Example below of the "module" I made;  the script that calls it, and the results.

Told you I'd be brief.

Module:

#pxslt.py

def underscore(context, word):
  '''Replace whitespace with underscore.'''
  out = word[0].replace(' ', '_')
  return out

def multiply(context, int_val, int2_val):
  '''Multiply two integers.'''
  int_val, int2_val = int(int_val[0]), int(int2_val[0])
  return int_val * int2_val

def libraryThing(context, isbn):
  '''Get language for a work based on ISBN using LibraryThing API.'''
  isbn = isbn[0]
  import urllib
  res = urllib.urlopen('http://www.librarything.com/api/thingLang.php?isbn=' + isbn)
  res_r = res.read()
  return res_r

##### DO NOT EDIT
##### makes it possible to call the above functions with XSLT
def pxslt():
  myFunctions = []
  gbs = globals()
  from inspect import isfunction
  for gb in gbs:
    if isfunction(gbs[gb]) and gb != 'pxslt':
      #print gb
      myFunctions.append(gbs[gb])

  from lxml import etree
  #see: http://lxml.de/extensions.html
  ns = etree.FunctionNamespace('file://libs/pxslt.py')
  ns.prefix = 'pxsl'
  for myFunction in myFunctions:
    name = str(myFunction.func_name)
    ns[name] = myFunction
  return ns

Usage example:

from lxml import etree

#####
myXML = etree.XML('''\
<a>
  <b>Hello. This will appear with whitespaces replaced by underscores.</b>
  <c>3</c>
</a>''')

myXSL = etree.XSLT(etree.XML('''\
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:pxslt="file://libs/pxslt.py">
  <xsl:output method="text" version="1.0" />
  <xsl:template match="a">
    <xsl:variable name="isbn">9955081260</xsl:variable>
    <xsl:value-of select="pxslt:libraryThing($isbn)" />
    <xsl:text>\n</xsl:text> <!-- Python will line break here -->
    <xsl:value-of select="pxslt:underscore(b/text())" />
    <xsl:text>\n</xsl:text> <!-- Python will line break here -->
    <xsl:call-template name="mathFunc">
    </xsl:call-template>
  </xsl:template>
  <xsl:template name="mathFunc">
    <xsl:variable name="myNum">10</xsl:variable>
    <xsl:value-of select="pxslt:multiply(c/text(), $myNum)" />
  </xsl:template>
</xsl:stylesheet>'''))

import pxslt
pxslt.pxslt() #get all set up with namespaces and function stuff

print(myXSL(myXML))

#myXSL_file = etree.XSLT(etree.parse('foo.xsl')) #for testing with a real XSL file
#print(myXSL_file(myXML))

Output:

>>>
lit
Hello._This_will_appear_with_whitespaces_replaced_by_underscores.
30

--------------

Related Content:

Written by nitin

November 2nd, 2012 at 5:28 pm

just goofin’ with a little Python CSV function and a limerickesque

leave a comment

This is probably a total waste of anyone's time but  …

The other night, after I'd worked on the Aguado Rondo (Op. 2, #2), I started goofing around with a little Python function that would turn a delimited text file into a Python dictionary, allowing me to target a specific cell in a delimited file, i.e. a "spreadsheet", without using the CSV module.

It works (well, at least I hope it does) this way:

films = csv2dict("films.txt", ";") #pass filename and delimiter
print films["title"] #print "title" column sans the header
print films["title"][-1] #prints last cell in the "title" column

It seems helpful to be able to do this. I've tested it with a UTF-8 file to write to file with some accent markings on people's names, etc. but I'm pretty sure it'll break on stuff I haven't thought of.

Anyway, maybe it'll come in handy to me for something.

Here's the function:

def csv2dict(fileName, delimiter):	
  f = open(fileName, "r") #open file
  lines = f.read() #read file
  f.close() #close file
  
  rows = lines.split("\n") #put lines in list
  headers = rows[0].split(delimiter) #put header titles in list
  rows.pop(0) #remove header from "rows" list

  i = 0
  worksheet = {}
  for header in headers: #for each header, i.e. each column
    columnCells = []
    for row in rows: #for each non-header row in delimited file
      if row != "":
        rowCells = row.split(delimiter) #get cells in row
        columnCells.append(rowCells[i]) #put column's cells in list
    worksheet[header] = columnCells #set header as KEY and set "columnCells" list as VALUE
    i = i + 1

  return worksheet

And for any guitarists out there, here's a little silly rhyme I wrote when living in my home eternal, Charleston, SC. I know the Sor as "warrior against the French" thing isn't accurate, but it helps the punchline.

:P

The title is, of course, that as Sor's famous duet.

"Le Deux Ami"

Fernando Sor,
who served in the war,
fought Bonaparte in the army.

But Dionisio Aguado,
who lacked such bravado,
preferred to run home to his mommy.
--------------

Related Content:

Written by nitin

February 16th, 2012 at 8:44 pm

AudioRegent 1.3.1 released

leave a comment

I've updated AudioRegent to version 1.3.1.

You can read an overview of the software and get the download link to the new version here.

The only reason I updated the software is because, as I've mentioned before, I've been having problems with Windows (and only recently at that) in terms of calling executables from the command line.

What seems to have helped is to no longer pass a command as a string a la:

RunSoxString = SoxPath + " ./outWavs/" + OggArray[cnt] + ws + "--comment-file comment.txt ./outOggs/" + str(OggArray[cnt])[:-4] + "." + outputType + ws + SoxOptions
RunSox = subprocess.Popen([RunSoxString], shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
RunSox.wait() #wait until the subprocess finishes

Now, it seems I have to pass it as a Python list (aka an array):

RunSoxString = SoxPath + " ./outWavs/" + OggArray[cnt] + ws + "--comment-file comment.txt ./outOggs/" + str(OggArray[cnt])[:-4] + "." + outputType + ws + SoxOptions
import shlex
RunSoxList = shlex.split(RunSoxString)
RunSox = subprocess.Popen(RunSoxList, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
RunSox.wait() #wait until the subprocess finishes

By the way, I totally haven't tested this new version enough to distribute it and I haven't tested it at all on a Linux box. But since no one's using it, I'm not too worried.

--------------

Related Content:

Written by nitin

July 28th, 2011 at 6:31 pm

Posted in digital audio,scripts

Tagged with , ,

fun with lxml

leave a comment

First off, I don't consider myself a programmer. I just know enough to dabble even though I try and learn new stuff all the time in the hope that I – as someone in digital libraries – can occasionally write something that can serve the needs of others, rather than serving my ego. Don't get me started on people who try and write software that has no utility other than patting themselves on the back …

Anyway, that's another post for another time.

So, the other day I got some questions/feature requests for PubMed2XL and so I started thinking about ways to tackle a few of the issues. It kinda makes me feel like a real programmer when people in the real world are asking about the software – but only for a few minutes before I make myself come back down to earth.

:/

Currently, the software places into a spreadsheet cell the value of one XML element, the position of which is defined by the user in the setup file. But there may potentially be a need it seems to be able to concatenate ALL the values for a given element into one spreadsheet cell. So I wrote a little function to help me get started with that.

The code uses this simple restaurant-based XML file from W3Schools and uses the awesome lxml Python library.

When run, it yields the following:

>>>
Calories for the first entree:
650
Calories for all entrees:
650; 900; 900; 600; 950

And here's the code:

#import required modules (lxml is non-standard; it likely needs to be installed)
import urllib #makes it easy to read documents from the web!
from lxml import etree #great XML parser and more!
                       #see: http://lxml.de/

#retrieve values from an XML file
def ElementCherryPicker(xpathArg, positionArg):
    '''
    This places all the element values for the element passed as the
    "xpathArg" argument into a list called "elementBox". It then returns
    the list item preceeding the one specified by the "positionArg" argument.
    This means passing a "1" equates to the first item in the list instead
    of the traditional "0". If "0" is passed then the entire list will be
    returned as a string with a delimiter of '; '.
    '''
    positionArg = positionArg - 1
    elements = parseUrl.findall(xpathArg) #make list of all matching elements
    elementBox = [] #create empty list
    for element in elements:
        elementBox.append(element.text) #place element values into the list
    if positionArg != -1:
        try:
            elementBox = elementBox[positionArg]
        except:
            elementBox = [] #if no element at stated position exists,
                            #then make the list empty again
    else:
        delimiter = '; '
        elementBox = delimiter.join(elementBox)
    return elementBox

#define, open, read, and parse an XML file
url= 'http://www.w3schools.com/xml/simple.xml'
readUrl = urllib.urlopen(url).read()
parseUrl = etree.XML(readUrl)

#print header and the values returned from ElementCherryPicker()
print 'Calories for the first entree:'
print ElementCherryPicker('.//calories', 1)
print 'Calories for all entrees:'
print ElementCherryPicker('.//calories', 0)


	

	
--------------

Related Content:

Written by nitin

March 7th, 2011 at 11:13 am

Posted in scripts,XML

Tagged with , ,

on adding an update checker to my Python programs

leave a comment

As I've been writing a few Python applications to share, I've been wondering the last day or so about how to have the application check, upon startup, for updates. This is a pretty common feature in a lot of programs, so I've been wondering how to do it.

Here's what I came up for now. It should work.

Say I have an application called FooPython and its project folder is http://blog.humaneguitarist.org/uploads/FooPython/.

I can just maintain a small XML file in that directory called update.xml that would look like this:

<versionInfo>
    <currentVersion>0.9.1</currentVersion>
    <message>Version 0.9.1 is now available!</message>
</
versionInfo>

Now all I'd have to do is add code similar to that below in my application to alert the user of an update:

import urllib
from lxml import etree #see: http://codespeak.net/lxml/

url = "http://blog.humaneguitarist.org/uploads/FooPython/update.xml" #not a real URL
update = urllib.urlopen(url).read()
root = etree.XML(update)
currentVersion = root.find(".//currentVersion")
currentVersionValue = currentVersion.text
message = root.find(".//message")
messageValue = message.text

if currentVersionValue != "0.9.0": #assuming this code is for version 0.9.0
   print messageValue

All this does is go to the update.xml file and read the element values. If the <currentVersion> doesn't match the version specified within the source code itself then it will print the value of the <message> element.

--------------

Related Content:

Written by nitin

November 5th, 2010 at 8:59 pm

Posted in scripts

Tagged with ,

South Park Launcher: Python and VB editions

leave a comment

It seems that about a year ago, the creators of South Park started freely offering their episodes online at southparkstudios.com.

The move was made, in part, to offer a legal way for fans without television access or even televisions (that's me!) to watch the show. As the creators stated, 

"(we got) really sick of having to download our own show illegally all the time.  So we gave ourselves a legal alternative."

This was great for me because I admittedly was downloading those shows as well. One of the other great things about it is the Random Episode link which, you guessed it, launches a random episode of the show.

The only thing that makes it less than perfect is that I can't set it to repeatedly play more random episodes.

Until now.

I decided to sit down and write the most important code I may ever write: a program that leverages the Random Episode feature to launch successive random episodes of the great social commentary, courtesy of Matt and Trey.

Here's the core of the code, from a Python perspective:

import sys, webbrowser, time

url = 'http://www.southparkstudios.com/episodes/random.php'

i=1

while i <= 5:
      webbrowser.open(url)
      time.sleep(1440) # 1440 seconds = 24 minutes, the average length of an episode plus ads.
      i=i+1
sys.exit()

Basically this is a loop that keeps launching a episode every 24 minutes while the variable "i" is less than or equal to 5. After every episode, "i" grows by one so when "i" equals 6, the program shuts down. BTW: after the program starts, the user does have to manually initiate playback of the first episode, but the last 4 episodes play automatically.

Now, this works fine and I even embellished the code so that the user can select from 2 to 10 episodes to launch. But I was still thinking, "This would be great to share with my friends, but they aren't going to run software from the command line!"

So I wrote a version using Microsoft's Visual Studio Express, a free version of their software that allows one to create GUI applications with a mixture of visual tools and the Visual Basic language. It's a really great tool that I hope to use more in the future although I do worry about the extra time required just to make software have a graphical interface.

Anyway, here's a ZIP file that contains both the installer for the Windows version of South Park Launcher as well as the Python version for Windows and other platforms. Make sure to read the ReadMe.txt file first!

You assume all responsibility for use.

Update, March 14, 2010: Looks like the South Park site now requires one to click the "play" button to start each and every episode, so South Park Launcher isn't very useful anymore.

Oh well. It was fun while it lasted.

Update, September, 2011: I've removed the link to the ZIP file as it wasn't a good program nor should I have been distributing code so early in my coding experience.  

--------------

Related Content:

Written by nitin

November 27th, 2009 at 9:12 am

MusicSQL: initial thoughts

one comment

One of the nice things about an emerging standard, namely MusicXML, having a command center (Recordare LLC) is having a central place to learn about what’s new.

On Friday, I was looking at Recordare’s page of MusicXML related software for software that worked from the command line and noticed something new and really interesting: MusicSQL.

According the the Goodle Code page that hosts this project, MusicSQL is:

… a system for conducting complex searches of symbolic music databases. The database can import and export MusicXML files. In the current version searches are constructed using a command line interface or through simple Python scripting tools.

Basically, at least as I understand it, MusicSQL is a Python program that sits on top of a MySQL database – now I really hope Oracle doesn’t kill MySQL if it buys Sun.

I was so excited to get MusicSQL working that I didn’t notate all the little problems I had along the way. The documentation for MusicSQL is very good and is written for Windows, Mac, and Linux (Ubuntu) users. But I’m inconceivably impatient, so I just mowed through the installation with little care for remembering what I was doing.

I do remember that I had to install Python 2.5, whereas I already have Python 2.6 installed – now I have both. I put/installed all the dependencies in my Python 2.5 directory just to compartmentalized everything – the exception being MySQL, which I installed wherever the default is.

So far, I only ran the first query in the documentation that uses "scientific" musical notation in the form Nx, where "N" is the alphabetical note name, say C, and "x" is an integer that denotes what octave the note is a member of. In other words, a C-Major scale would be "Cx Dx Ex Fx Gx Ax Bx Cx+1", something like "C5 D5 … B5 C6", etc. You can place an integer before the note name to denote its duration.

Running the query from the command line, I was really happy with the speed and the output of MusicSQL for the test query.

One problem I did have, though, is I kept getting errors for another great feature of MusicSQL. Basically, after you run your query, you can see a PDF of the results (i.e. the music excerpt pertaining to the query results). The PDF is made by Lilypond, a text-based notation software that produces – in my opinion – the absolute best looking engraving out there, that’s why I use it (and yes, it’s free).

Now Lilypond doesn’t natively read MusicXML, it uses its own encoding. So MusicSQL takes advantage of a Python script that comes with the Lilypond install called "xml2ly" that converts MusicXML to Lilypond format. I left a message on the project forum for MusicSQL, so I’m hoping I can figure out what I need to do to get the Lilypond outout of the query results to work. At any rate, I do wonder how effective it can be since the conversion from MusicXML to Lilypond can sometimes get ugly.

I wonder if an alternative solution is to use the command line options for the MuseScore notation software to generate a PDF of the query results. Musescore can also convert MusicXML to other graphics formats (PNG) and even audio (WAV, FLAC, OGG), so theoretically it could be leveraged to make audio files for the corresponding query results.

At any rate, I’m really looking forward to the future developments of MusicSQL.

And as for using MuseScore’s command line in conjunction with MusicXML and how it can add value to a web collection of MusicXML docs – there will be more to that later …

--------------

Related Content:

Written by nitin

November 15th, 2009 at 3:54 pm

an ID3 tag reporter: MyMusicReporter.py

2 comments

I usually try to post to this blog once a week on average. That forces me to try to learn something or learn about something during the week that I can post about.

Sometimes though I have to resort to things that I've been sitting on for a while, like this Python script I wrote called MyMusicReporter.

This program makes use of the id3reader module available here. In other words, the program won't function without this module.

For those unfamiliar with Python, it supports the use of modules, which – as the name implies – is basically allowing the current Python program to take advantage of another, pre-existing Python program. At least that's how I understand it for now …

Anyway, MyMusicReporter does the following:

  1. it asks you to point it to a root directory on your computer,
  2. it then finds all the MP3s in that folder and its subfolders,
  3. it makes an XML file in the same directory as MyMusicReporter called MyMusicReport.xml which contains the Title, Performer, Track #, Year, and relative file path for each MP3,
  4. it makes an XML stylesheet in the same directory as MyMusicReporter called MyMusicReport.xsl which makes MyMusicReport.xml look pretty in your web browser. 

Here's an example of how the XML looks with my home-grown schema:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="MyMusicReport.xsl"?>
<MyMusicReport>
    <mp3>
        <title>R2-DVD2</title>
        <performer>Nitin Arora</performer>
        <track>1</track>
        <year>2008</year>
        <path>r2_dvd2.mp3</path>
    </mp3>
</MyMusicReport>

Anyway, here's the source code for MyMusicReporter.py.

You assume all responsibility for use.

--------------

Related Content:

Written by nitin

October 17th, 2009 at 9:12 pm

Posted in scripts,XML

Tagged with , , ,

something completely different: MyFolderMaker.py

leave a comment

I haven't hidden the fact that this blog exists, in part, as a resume builder. And I think that programming skills are a valuable asset in digital libraries. Frankly, LIS programs need to address this to keep their graduates competitive. But that's another post altogether …

Anyway, at work yesterday (Friday) I made a little Windows .bat file that creates a folder named after every line in a given plain text file.

When I came home, I napped until nearly midnight and woke up in an odd, creative state: when I'm mostly mind and hardly any body. Whether I'm hardly anybody is a matter of opinion.

I decided to make a fancier version of the batch file using Python, which is named after Monty Python's Flying Circus. In all seriousness, that's a huge part of the reason I decided to start learning Python.

The program can do one of the following:

  1. Ask you to supply a path and then mimick the folder structure (all new folders are created inside a folder called "MyFolderListFolder" to prevent accidental messes).
  2. Read from a plain text file of yours called "MyFolderList.txt" and make folders named for each line in the text file (all new folders are created inside a folder called "MyFolderListFolder" to prevent accidental messes). Subfolders can be made using this format (Unix/Linux users need to use forward slashes, Windows users can use either):

folder1

folder1\subfolder

folder2

  1. Take you to this page if you'd simply like to see the source code and learn more about it. Of course, .py files can be opened in a text editor so this is only really useful for the Windows .exe version I compiled with cx_Freeze.

The program has some error checking built in. For example, if your text file has blank rows or duplicates, it will alert you under most situations that there's a problem you need to fix.

The source code link is below if anyone is interested … real programmers likely will – and should – laugh, but I'm still learning.

A text file of the program (i.e. source code) is available here.

You assume all responsibility for use.

--------------

Related Content:

Written by nitin

September 26th, 2009 at 2:34 pm

Posted in scripts

Tagged with , , , ,

Switch to our mobile site