I took April off from blogging after realizing I was over blogging, as opposed to over logging.
I'll keep this short. Well, I'll try.
I'm shacked up in the apartment due to some unexpected circumstances and yesterday I decided to try and be a little productive and learn something I could potentially use in the workplace.
I learned a little about Google App Engine. I was drawn to it because of the Python support and because it gives me a free environment where I can deploy Python apps using the ever-elusive lxml library.
While I wrote some silly stuff using lxml and data available from the Business.gov API I ended uploading a simple app – if you can call it that – that parses a CSV file from North Carolina's (USA) NCOpenBook.
I didn't use the csv module because the CSV file I used has like three lines at the top that aren't headers (people: don't do that!). I don't know if there's a way to handle that with the csv module (there probably is) but I wasn't interested in digging around. Instead, I used a modified version of this code I wrote previously.
The CSV file lists grantees who've received funding by North Carolina and the app pulls out the top ten since 2007 based on cumulative grant totals. The app uses Google Chart Tools to make a pie chart of the top ten recipients. I'm not so sure about the colors in the pie chart – it's hard to see the difference between some of the colors associated with each grantee – but it's a simple start.
Here's a screenshot:
.. and here's the link to the app online: http://top-ten-nc-totals-by-grantee.appspot.com.
I've also pasted the app.yaml file, my Python code, and the Jinja/HTML template below if anyone's interested.
YAML:
application: top-ten-nc-totals-by-grantee version: 1 runtime: python27 api_version: 1 threadsafe: true handlers: - url: /stylesheets static_dir: stylesheets - url: /.* script: nctotals.app libraries: - name: jinja2 version: latest
Python:
#import modules
import urllib
import webapp2
import jinja2
import os
jinja_environment = jinja2.Environment(
loader=jinja2.FileSystemLoader(os.path.dirname(__file__)))
#####
#see: http://stackoverflow.com/a/2827664
class Object(object):
pass
#my CSV parser
def csv2dict(fileName, delimiter):
f = urllib.urlopen(fileName) #open file
lines = f.read() #read file
rows = lines.split("\n") #put lines in list
#cut out non-header rows at top of this particular CSV file
for i in range(0,3):
rows.pop(0)
#shorten the CSV data to 10 rows (there were too many damn rows in the CSV file!)
for i in range(12,len(rows)+1):
rows.pop(-1)
headers = rows[0].split(delimiter) #put header titles in list
rows.pop(0) #remove header from "rows" list
i = 0
worksheet = {}
for header in headers: #for each header, i.e. each column
columnCells = []
#print header #test line
for row in rows: #for each non-header row in delimited file
if row != "": #!!!you need to also add a test for lines that don't split on the delimeter (i.e. notes)
rowCells = row.split(delimiter) #get cells in row
columnCells.append(rowCells[i].strip()) #put column's cells in list
worksheet[header] = columnCells #set header as KEY and set "columnCells" list as VALUE
i = i + 1
return worksheet
#####
class MainPage(webapp2.RequestHandler):
def get(self):
parsed = csv2dict("http://data.osbm.state.nc.us/openbook/comma_grant_cumulative_awards_and_annual_disbursements_by_grantee.csv", '","') #pass filename and delimiter
topTen = range(0,len(parsed['"Non-Profit Name (*)'])) #i.e. range is 1 to 10, or 0 to 9 depending on your p.o.v.
for i in topTen: #add attributes to each of the ten agencies in the CSV file
topTen[i] = Object()
topTen[i].name = parsed['"Non-Profit Name (*)'][i].replace('"','')
topTen[i].total = parsed['Cumulative Total Award'][i]
raw_total = parsed['Cumulative Total Award'][i]
raw_total = raw_total.replace('$','')
raw_total = raw_total.replace(',','')
topTen[i].raw_total = raw_total
#data for the Jinja template
template_values = {
'topTen': topTen}
template = jinja_environment.get_template('index.html')
self.response.out.write(template.render(template_values)) #write data to the index.html template
app = webapp2.WSGIApplication([('/', MainPage)],debug=True)
Template:
<!DOCTYPE HTML>
<html>
<head>
<title>
Top Ten NC Grants by Grantee (since 2007)
</title>
<link type="text/css" rel="stylesheet" href="/stylesheets/style.css" />
<script type="text/javascript" src="http://www.google.com/jsapi"></script>
<script type="text/javascript">
google.load('visualization', '1', {packages: ['imagepiechart']});
</script>
<script type="text/javascript">
function drawVisualization() {
// Create and populate the data table.
var data = new google.visualization.DataTable();
data.addColumn('string', 'name');
data.addColumn('number', 'raw_total');
data.addRows([
{% for topper in topTen %}
["{{ topper.name }} - {{ topper.total }}", {{ topper.raw_total }}],
{% endfor %}
]);
// Create and draw the visualization.
new google.visualization.ImagePieChart(document.getElementById('visualization')).
draw(data, null);
}
google.setOnLoadCallback(drawVisualization);
</script>
</head>
<body>
<h3>Top Ten <a href="http://www.ncopenbook.gov/NCOpenBook/GrantsHome.jsp">NC Grants</a> by Grantee (cumulative totals since 2007)</h3>
<p>see the source CSV file <a href="http://data.osbm.state.nc.us/openbook/comma_grant_cumulative_awards_and_annual_disbursements_by_grantee.csv">here</a></p>
<div id="visualization"></div>
<p>Made with:</p>
<ul>
<li><a href="https://developers.google.com/appengine/docs/python/gettingstartedpython27/">Google App Engine (Python 2.7)</a></li>
<li><a href="https://developers.google.com/chart/">Google Chart Tools</a></li>
</ul>
<p>More info (blog post):</p>
<ul>
<li><a href="http://blog.humaneguitarist.org/2012/05/01/north-carolina-grants-google-app-engine-and-pie-mmm/">North Carolina grants, Google App Engine, and pie ... mmm.</a></li>
</ul>
</body>
</html>
--------------
