blog.humaneguitarist.org

North Carolina grants, Google App Engine, and pie ... mmm.

[Tue, 01 May 2012 14:42:29 +0000]
I took April off from blogging after realizing I was over blogging, as opposed to over logging [http://www.southparkstudios.com/full-episodes/s12e06-over-logging]. I'll keep this short. Well, I'll try. I'm shacked up in the apartment due to some unexpected circumstances and yesterday I decided to try and be a little productive and learn something I could potentially use in the workplace. I learned a little about Google App Engine [https://developers.google.com/appengine/]. I was drawn to it because of the Python support and because it gives me a free environment where I can deploy Python apps using the ever-elusive lxml [http://lxml.de/] library. While I wrote some silly stuff using lxml and data available from the Business.gov API [http://www.sba.gov/api] I ended uploading a simple app - if you can call it that - that parses a CSV file from North Carolina's (USA) NCOpenBook [http://www.ncopenbook.gov/NCOpenBook/]. I didn't use the csv [http://docs.python.org/library/csv.html] module because the CSV file I used has like three lines at the top that aren't headers (people: don't do that!). I don't know if there's a way to handle that with the csv module (there probably is) but I wasn't interested in digging around. Instead, I used a modified version of this [http://blog.humaneguitarist.org/2012/02/16/just-goofin-with-a-little-python-csv-function-and-a-limerickesque/] code I wrote previously. The CSV file lists grantees who've received funding by North Carolina and the app pulls out the top ten since 2007 based on cumulative grant totals. The app uses Google Chart Tools [https://developers.google.com/chart/] to make a pie chart of the top ten recipients. I'm not so sure about the colors in the pie chart - it's hard to see the difference between some of the colors associated with each grantee - but it's a simple start. Here's a screenshot: IMAGE: "Top Ten NC Grants by Grantee"[http://blog.humaneguitarist.org/uploads/top-ten-nc-totals-by-grantee.png] .. and here's the link to the app online: http://top-ten-nc-totals-by-grantee.appspot.com [http://top-ten-nc-totals-by-grantee.appspot.com/]. I've also pasted the app.yaml file, my Python code, and the Jinja/HTML template below if anyone's interested. YAML: application: top-ten-nc-totals-by-grantee version: 1 runtime: python27 api_version: 1 threadsafe: true handlers: - url: /stylesheets static_dir: stylesheets - url: /.* script: nctotals.app libraries: - name: jinja2 version: latest Python: #import modules import urllib import webapp2 import jinja2 import os jinja_environment = jinja2.Environment( loader=jinja2.FileSystemLoader(os.path.dirname(__file__))) ##### #see: http://stackoverflow.com/a/2827664 class Object(object): pass #my CSV parser def csv2dict(fileName, delimiter): f = urllib.urlopen(fileName) #open file lines = f.read() #read file rows = lines.split("\n") #put lines in list #cut out non-header rows at top of this particular CSV file for i in range(0,3): rows.pop(0) #shorten the CSV data to 10 rows (there were too many damn rows in the CSV file!) for i in range(12,len(rows)+1): rows.pop(-1) headers = rows[0].split(delimiter) #put header titles in list rows.pop(0) #remove header from "rows" list i = 0 worksheet = {} for header in headers: #for each header, i.e. each column columnCells = [] #print header #test line for row in rows: #for each non-header row in delimited file if row != "": #!!!you need to also add a test for lines that don't split on the delimeter (i.e. notes) rowCells = row.split(delimiter) #get cells in row columnCells.append(rowCells[i].strip()) #put column's cells in list worksheet[header] = columnCells #set header as KEY and set "columnCells" list as VALUE i = i + 1 return worksheet ##### class MainPage(webapp2.RequestHandler): def get(self): parsed = csv2dict("http://data.osbm.state.nc.us/openbook/comma_grant_cumulative_awards_and_annual_disbursements_by_grantee.csv", '","') #pass filename and delimiter topTen = range(0,len(parsed['"Non-Profit Name (*)'])) #i.e. range is 1 to 10, or 0 to 9 depending on your p.o.v. for i in topTen: #add attributes to each of the ten agencies in the CSV file topTen[i] = Object() topTen[i].name = parsed['"Non-Profit Name (*)'][i].replace('"','') topTen[i].total = parsed['Cumulative Total Award'][i] raw_total = parsed['Cumulative Total Award'][i] raw_total = raw_total.replace('$','') raw_total = raw_total.replace(',','') topTen[i].raw_total = raw_total #data for the Jinja template template_values = { 'topTen': topTen} template = jinja_environment.get_template('index.html') self.response.out.write(template.render(template_values)) #write data to the index.html template app = webapp2.WSGIApplication([('/', MainPage)],debug=True) Template: <!DOCTYPE HTML> <html> <head> <title> Top Ten NC Grants by Grantee (since 2007) </title> <link type="text/css" rel="stylesheet" href="/stylesheets/style.css" /> <script type="text/javascript" src="http://www.google.com/jsapi"></script> <script type="text/javascript"> google.load('visualization', '1', {packages: ['imagepiechart']}); </script> <script type="text/javascript"> function drawVisualization() { // Create and populate the data table. var data = new google.visualization.DataTable(); data.addColumn('string', 'name'); data.addColumn('number', 'raw_total'); data.addRows([ {% for topper in topTen %} ["{{ topper.name }} - {{ topper.total }}", {{ topper.raw_total }}], {% endfor %} ]); // Create and draw the visualization. new google.visualization.ImagePieChart(document.getElementById('visualization')). draw(data, null); } google.setOnLoadCallback(drawVisualization); </script> </head> <body> <h3>Top Ten <a href="http://www.ncopenbook.gov/NCOpenBook/GrantsHome.jsp">NC Grants</a> by Grantee (cumulative totals since 2007)</h3> <p>see the source CSV file <a href="http://data.osbm.state.nc.us/openbook/comma_grant_cumulative_awards_and_annual_disbursements_by_grantee.csv">here</a></p> <div id="visualization"></div> <p>Made with:</p> <ul> <li><a href="https://developers.google.com/appengine/docs/python/gettingstartedpython27/">Google App Engine (Python 2.7)</a></li> <li><a href="https://developers.google.com/chart/">Google Chart Tools</a></li> </ul> <p>More info (blog post):</p> <ul> <li><a href="http://blog.humaneguitarist.org/2012/05/01/north-carolina-grants-google-app-engine-and-pie-mmm/">North Carolina grants, Google App Engine, and pie ... mmm.</a></li> </ul> </body> </html>