when my coworker, Dan, woke up this morning, i’m pretty sure that “learning linear algebra” wasn’t on his todo list for the day. linear algebra certainly wasn’t on my mind either when he came to me for help on a problem.
Dan has to do basic faculty support and training for the university’s course management system (which we use, but don’t develop ourselves). when something’s broken, he usually has to be the one to file the bug report with the developers. today he was working on trying to understand a bug in the gradebook so he could give them a useful report of exactly what was going wrong.
the trouble seemed to be that the grades it was calculating weren’t quite right. for the class in question, there were only three grades per student that were supposed to be weighted 30%, 30%, and 40% respectively. Dan had imported the grades into a spreadsheet and created a formula to generate the correct grades and display them next to the incorrect ones that the gradebook was coming up with. we spent a couple minutes staring at it and noticed that there seemed to be a rough correlation between the first column and the average. when the grade in the first column was higher than the average, the gradebook’s result was also higher and vice versa. that gave us the suspicion that the gradebook was weighting the columns incorrectly and, in particular, weighting the first column too heavily.
Dan started plugging in different weights into his formula trying to guess the gradebook’s weights and verify that that was indeed what was going wrong. i watched him do that for a few minutes and realized that it could take him all day to stumble on the right set of weights. if we were right about the weights being off, then it’s just a system of linear equations with three unknowns (the three weights) which should be fairly straightforward to solve with Gaussian Elimination. So the spreadsheet looked like this:
well, there were a bunch more rows, but three is enough to see what’s going on and actually just enough to solve the system. anyway, that just corresponds to these three equations:
Gaussian elimination is just a somewhat mechanical algorithm for taking a system like that and coming out with values for Wa, Wb, and Wc.
i started walking Dan through it on paper but we started getting some bogus looking numbers and i remembered why it was such a pain. the process is pretty mechanical, but it involves a lot of arithmetic and generally just has a lot of places where you can make a mistake and mess the whole thing up. if we’d had a copy of Matlab or even my trusty old TI-85, it would have been no problem.
luckily though, python has some nice linear algebra packages so i went and punched in this code:
from Numeric import *
from LinearAlgebra import solve_linear_equations
a = array([[ 78.33, 85.3 , 95. ],
[ 68.33, 80.3 , 89. ],
[ 70. , 88.7 , 86. ]])
b = array([ 85.07, 77.45, 79.24])
print solve_linear_equations(a,b)
and got the result [0.46242474, 0.23079146, 0.30696587], which we plugged back into the spreadsheet and saw that it indeed matched what the gradebook was calculating for all the rows. now, why the gradebook code was using those weights instead of [.30, .30, .40] we have no idea; but that’s the developers’ problem not ours.
i guess my point is that math shows up in unexpected places so it pays to 1) know enough to recognize it and know how to approach solving a math problem and 2) it helps to know how to get a computer to solve the problem for you. so pay attention in calculus class, kids.
<p>every day i’m becoming more and more convinced that we’re really living in a movie. i’m not sure yet if it’s sci-fi or horror. </p>
<p>dogs aren’t having all the fun though. scientists have also been <a href="http://www.kuro5hin.org/story/2005/6/20/111815/063">extracting video from cat brains</a>. now we just need to combine the two and extract video from zombie animal brains. or combine everything and pull the video out of cockroach controlled zombie animals.</p>
<p>clearly, the next evolutionary step will be cockroaches controlling mice. </p>
<p>ladies and gentlemen, the human race’s days are numbered. prepare to welcome your new cockroach controlled rodent overlords.</p>
i use SQLObject for all my programming that involves a database. i think it hits a real sweet spot on the simplicity/power curve.
using it basically involves creating some classes that inherit from SQLObject for each table in the database and giving them each attributes that correspond to the columns of the tables. it doesn’t really let you forget that you’re using a database to store your stuff, but it saves you from having to hand code all the simple SELECT and UPDATE type SQL queries. i really like this approach; it’s more of a representation of database tables and rows as objects than an attempt to force fit objects into a relational database.
it’s got a powerful enough query syntax to cover about 95% of anyone’s needs. for four of the remaining five percent, the ability to pass in raw SQL for the WHERE clause covers you. so you can do LEFT OUTER JOINs to your heart’s content when you really need to.
what it’s missing out of the box though, is the ability to specify which columns get fetched. since it returns a list of objects instead of raw results, it will always fetch the columns it needs to build those objects. 99% of the time that’s fine. but it prevents you from using a lot of SQL’s built in functions like COUNT(), MAX(), or AVG(), which are useful for various reporting type queries and for doing really complex queries with GROUP BY and HAVING constructs.
but all hope is not lost!
i had to dig around in the source code a bit to figure out exactly how to do it, but it’s actually pretty easy to directly access SQLObject’s internal database connection methods and make raw queries directly. it just looks like:
:::python
# Node is a class that inherited from SQLObject
results = Node._connection.queryAll(
"""SELECT [some really complex and nasty SQL
that needs aggregate functions, etc.]""")
# results will then be a list of tuples,
# one tuple per row returned.
there’s also a queryOne() method that just returns one tuple if that’s all you want.
the only thing you don’t want to do is use that stuff for updates, inserts, or deletes since it bypasses SQLObject’s caching and you could end up with weird inconsistencies between what’s in the database and what’s in memory. use SQObject’s regular interface for anything that modifies the database.
anyway, this post is here so google will find it and other people wanting to do raw queries won’t have to dig through the source code (although it’s pretty easy to understand if you’re so inclined).
i’ve been kind of busy in the world of plone lately.
one project that i contributed to is out now: Wicked, “wiki with no wiki aftertaste”. it’s pretty cool but might not make much sense to you unless you’re already immersed in plone development.
another one that my coworker, Jonah, and i are building is PloneStickies. it’s not really release ready yet, but should be soon. it basically implements “sticky notes” (“post-its” is trademarked) for plone. Jonah’s doing most of the actual plone part and i’ve been concentrating on the frontend javascript and CSS part. as a taste of what’s to come, here’s my stickies test page which has the latest version of the frontend stuff which will hopefully make it into the upcoming PloneStickies release.
i’m rather proud of that bit of code. it’s fast, robust, full-featured, fairly clean and it even works in IE i’m told. i’m trying to keep it general enough that at least parts of it will be useful outside Plone.
they’re quite elaborate and, as he points out, quite secure. over the last thousand years, they’ve done a pretty good job of preventing just about any kind of voter fraud that could be imagined.
what really gets me about the whole thing is that with all of this obvious thought having been put into ensuring fairness and transparency, it’s clear that the cardinals don’t trust each other very far (or at least historically haven’t). all of those cardinals are supposed to be holy, moral, ethical men, but if there had never been problems, no one would have bothered coming up with and codifying all these elaborate safeguards. so clearly at least a few bad apples have made it through. yet, once one of them is elected pope, he is elevated to the holiest office in the world and is considered to speak directly for god.
it reminds me of Bill Hicks’ observation on the popemobile and its inch thick bulletproof plexiglass: “that’s faith in action, folks.”
i’ve been using mpd as my music player lately. it’s a neat little program; very minimalist but it does a good job with my music library and is a lot less flakey than yammi has been lately.
mpd doesn’t really keep track of the songs you’ve played though. i really like archiving that kind of data for some reason. i have to keep my packrat tendencies under control when it comes to physical stuff since my apartment would fill up, but with data, i figure disk space is cheap. so if there’s any chance i might someday find it useful, i try to figure out how to put it in a database.
so, a couple lines of python later, i had a little script that would detect song changes and fire off an http request with the info.
with the help of subway and a few more lines of python, i had a nice web interface to the database of what songs i’d listened to when. it’s open to the public too; the curious can check out the last 50 songs i’ve listened to updated in real time.
i recently installed trac on my work machine to manage my little personal coding projects like the engine behind this site.
trac is a pretty sweet combination of a subversion repository, a wiki, and a bug database. i’m quite impressed with it so far. it was a little painful to get it setup and configured, but it’s been nice to me since.
it’s up here if you’re curious. if you encounter bugs with this site or have suggestions for new features, the best thing to do is to add a ticket through trac.
posting this from my new laptop, a nice refurbished thinkpad T23 that i picked up for $500 (1.1GHz, i spent some extra cash to upgrade it to 1GB of RAM, 30GBdrive, 14” XGA screen, 5.3lbs, and a PCMCIA wireless card that i picked up for $12.99. not the sexiest laptop out there, but for the price, i’m quite pleased with it.)
i’d been hearing a lot about the Ubuntu linux distribution lately. at the plone sprint i went to in january, there were a bunch of happy ubuntu users and people have been raving about how well optimized it is for laptops, so i burned an ubuntu install CD and put it on the laptop when it arrived today.
i’ve been a pretty happy gentoo user for a while now and i love its package manager, portage, more than toast. for a seasoned linux user who wants to keep on the bleeding edge and tweak the hell out of their system, gentoo is hard to beat. but today, i just wanted to play with my new toy as soon as possible and make sure everything worked properly and gentoo takes a long time to install.
ubuntu seems to have been a good choice. it took about 15 minutes in all to get it up and running with a nice Gnome desktop and all the hardware working and configured properly. ubuntu aims to be “linux for humans” and the founder, Mark Shuttleworth (remember the billionaire who bought his way onto a russian space launch?), has some strong opinions about how software should Just Work. well, ubuntu pretty much does. the install process pretty much consisted of booting off the CD, hitting enter a few times to select the defaults, entering a username and password and then rebooting. when it was done copying stuff off the CD and installing itself, everything pretty much just worked. the wireless networking was turned off by default but all i had to do was turn it back on; i didn’t have to configure it at all.
i’m quite impressed that after installing linux on this machine, i actually have no idea what models the videocard or soundcard are. it just autodetected them and they work. granted, thinkpads have a reputation for being extremely well supported by linux distributions, but it’s still a fairly novel experience for me. knoppix usually does a good job with the hardware detection too, but installing from knoppix is clumsy at best and the user experience lacks a lot of polish. ubuntu seems well integrated and planned. the install process doesn’t include any package selection step; it just comes with a base set defaulting to a nicely customized Gnome desktop. once it’s installed, it’s basically a standard Debian system so apt-get will pull down anything else you need.
i’m going to see how long i can live with ubuntu on here before the tweaking urges take over and i install gentoo on it. it probably depends on whether it becomes my main machine at home. if it does, i’ll probably want to do things that apt-get doesn’t let me sooner rather than later and that will be the end of ubuntu. but if it mostly becomes a travel machine that i only pull out every once in a while and want everything to work perfectly without having to think about, i think ubuntu will probably work out.
at any rate, unless i encounter anything really heinous in the next couple days of playing, i think i have a new distro to recommend to people who are looking to try out linux. the install process was almost identical to OS X’s (except that it runs in console mode and doesn’t really look very pretty; but the number of steps and the choices you’re presented with are about the same) and the polish of the resulting system is pretty close to what Apple gives you (and catching up quickly). so i would also recommend ubuntu to Mac fans who have some spare x86 hardware and want to run something nicer than windows on it.