here’s the full report on the trip to mexico with lani and anastasiya.
no one was killed or kidnapped and overall it was a good time. Copper Canyon and Mexico in general (outside the border towns and spring break type areas) are highly recommended for vacation spots. the main downside to the trip was that we spent a lot of time travelling.
lani’s mom is a flight attendent so we are able to fly pretty cheaply. the catch is that we have to fly standby and that means that things don’t always work out like planned.
the plan was for me to fly down to austin on friday afternoon (with connections in DC and atlanta) where lani and anastasiya were already waiting, then we would fly to El Paso on saturday morning and take a bus across the border to Juarez and then a 10 hour bus down to Creel. i managed to get on a 3:30 flight to DC. the last flight from atlanta to austin was at 10pm. to catch it, i had to get on a flight to atlanta that left DC before 7. didn’t get on the 5:00. didn’t get on the 6:00. didn’t get on the 7:00. the 8:00 was delayed till 8:30 but i finally got on it. since i’d missed the last flight out of atlanta, lani’s mom picked me up at the airport (she lives outside atlanta) and i spent the night giving her guitar lessons. first thing in the morning, i got on a flight to austin.
we spent saturday in austin while lani finished up stuff at her lab. there were 5:30 and 7:30 flights from austin to el paso (with a connection in dallas). we made it to the airport at about 5pm. one of the other catches with flying standby on the airline’s dime is that there is a dress code. no jeans. no open toe shoes. etc. lani and i have flown standby many times so we’re used to it. this was anastasiya’s first time though. since it had never affected lani or i before, we were unaware that lip rings are also against the dress code. to make a long story short, i had to extract it in the airport with some needle-nose pliers. by the time that was taken care of though, we’d missed the 5:30 flight. the ticket agent assured us that the 7:30 had plenty of empty seats though. 2 hours later, we discovered that there were exactly 2 empty seats on the plane (and 3 of us).
flights out of austin the next morning were all oversold (although the dallas to el paso leg was fine). so we had no choice but to rent a car and drive to dallas overnight. none of us had really slept more than a few hours in the last few days, but somehow we made it without killing ourselves (though the overnight drive through texas certainly had its share of David Lynch moments).
from there on it was pretty smooth sailing. we got to El Paso and got on the bus to cross the border. i had my first injury of the trip then. the bus had little TVs hanging down from the ceiling with nice sharp corners. one of which put a nice dent in my skull and some blood in my hair. but it didn’t hurt too much and we made it across the border and got onto a sweet bus to Chihuahua. probably the nicest bus i’ve ever been on. the seats were comfy, there was plenty of legroom, and LCD television screens (that were nowhere near my head).
we got into Chihuahua at about 6:30 at night only to find that there were no more buses to Creel until the next morning. so we found a dirty $10 hotel (the Lonely Planet guide recommended it as about the best in town) with no hot water, had some dinner and wandered around town for a bit (but it was sunday night and there wasn’t much open).
the bus to Creel wasn’t nearly as nice and turned out to be a local bus so it stopped at every little town on the way. it even stopped to pick up people on the side of the highway. 6 hours later, we finally made it to Creel. dropped off our stuff at the hotel (the same one that every other Lonely Planet reader in town was staying at). got some lunch. then decided to rent some mountain bikes and explore the sites around town.
it only took about 20 minutes for that to turn out to be a really bad idea. i was riding behind lani and anastasiya, we were going downhill at a pretty good clip. they stopped. i squeezed the brakes as hard as i could. the front brakes apparently worked really well–the front wheel stopped completely but the rest of the bike and my body continued on at full speed. after flying over my handlebars at full speed, according to lani, i did a pretty impressive aikido style roll on the pavement (tucker would be proud). the roll seemed to have prevented any kind of head injury which was good because they didn’t rent us helmets with our bikes. still though, a high speed somersault on pavement isn’t recommended. i left a nice amount of palm and knee skin on the pavement and couldn’t really move my arms through their full range.
adrenaline’s a hell of a drug though and i was feeling surprisingly spry after my little dive so we kept riding. unfortunately lani’s bike started having some mechanical issues so we didn’t make it too far. after a few hours though, i started feeling more and more pain in my wrists, elbows, and generally my entire body. it was starting to get dark, we had to ride a few miles back along a winding mountain road with no real visibility, our bikes didn’t seem to have any sort of reflectors (i started looking after that and never saw a single bicycle with a reflector for the entire trip) and i was wearing my usual “invisible pedestrian” costume (all black), so we turned around and headed back. my wrists and arms kept getting worse and by the time we were back in town, i couldn’t move my left arm at all or put any weight on it and my left wrist was massively swollen. my right arm was only a little better but i was able to use it to steer the bike (with some pain).
we got some bandages and antiseptic and cleaned my wounds up and immobilized my left arm the best we could. at dinner that night and all the next day, i could barely get the fork to my mouth and drinking the last half of a bottle of beer was painful (but i did it dammit!). on tuesday i was way too sore to go anywhere but anastasiya took a bus to Batopilas (a town at the bottom of the canyons; Creel is up near the top in the mountains). lani and i just stayed in Creel and had a nice siesta to catch up on the sleep we missed with all the traveling, walked around town a bit, and spent the evening at the town’s only bar drinking tequilla.
by wednesday my right arm had more or less returned to usefulness. lani and i went on a hike up to the hill above town, then we took a bus to Divisadero, which is a nice high point that you can see most of the canyons from. the bus there cost about $3 and was a 1 hour ride over twisting turning mountain roads. the view there was absolutely stunning. Copper Canyon is four times the size of the grand canyon and even deeper. from Divisadero, you can pretty much see it all at once.
unfortunately, we didn’t realize that the bus we came to Divisadero on was the last one going back to Creel for the day. so instead of taking the bus back, we had to wait about 2 and a half hours for the train that (luckily) was heading that way. the train took about 2 hours to cover the same distance and cost over $10.
thursday morning we took a bus back to Chihuahua (another annoying local bus) and spent the afternoon there wandering around the city taking pictures, eating street food and spent some time online. we tried a different hotel that night and it was even more special than the one before. this one had hot water but the toilet was missing a seat. after anastasiya reported seeing cockroaches in her room, lani was so freaked out that she insisted we sleep with the light on. somehow we survived the night without being eaten alive by killer cockroaches or anything and got on a bus back to Juarez and El Paso first thing in the morning. we didn’t even have any trouble crossing the border.
anastasiya had actually purchased a ticket out of El Paso, so she got right on a plane and out of texas. there were no more free seats on planes for lani and i though so we got a hotel room and explored the strip malls of El Paso on foot. not too exciting. we finally managed to escape El Paso the next morning and made it back to our respective cities.
i can more or less move my arms now but they still hurt like crazy if i try to put too much weight on them. i hope to someday be able to do pushups again. i’ve also still got some massive, ugly bruises on my elbows and the skin hasn’t fully grown back on my knee yet. but really, it was a fun vacation. i highly recommend it. the people were all really friendly and helpful, the scenery was gorgeous, and the food was fantastic (and even edible for a couple vegetarians). if you go, you may want to skip the mountain bikes though if you have my coordination and ability to injure yourself.
in Chihuahua now. all members of the party are still alive and unkidnapped. back in the US tomorrow. mexico rocks but mexican pavement is unpleasant at high speeds. i’ll explain when i get back.
in austin right now. leaving in a few minutes for copper canyon with lani and her russian friend. we don’t really know how we’re getting there, what we’re doing when we get there and none of us speak any spanish. should be interesting.
there was a great tragedy today. lani and i went out to get some lunch. my camera was still in my bag so i didn’t bring it with me for once. naturally, on the way to lunch, we spotted a ninja on a bicycle. i’m totally serious. but no one will ever believe us because i have no photographic evidence.
oh, and yesterday while i was in airport limbo, i saw Jon Stewart (from the Daily Show) on CNN’s Crossfire up on the DC airport screens (with a weird lag of a few seconds between the audio and video. it was awesome. he just tore into the hosts for their part in the downfall of american TV journalism. if you missed it, the transcript is worth a read.
anyway, i’ll be back in a week if we don’t get kidnapped by banditos.
watching the presidential and vice-presidential debates with all the rules, time limits, lights, and buzzers, it occurs to me that our presidential election has been turned into a televised game show.
today my friend julia dragged me out to queens to go look at graffiti. we ended up at 5points, which is an old factory whose owners have given it up to graffiti artists and turned it into a sort of living gallery. we were lucky enough to see a bunch of artists out painting, and on the roof there were some photographers and a model doing a fashion shoot.
it was a nice day and we saw some great art. my photos are here.
my friend marc has recently started an advertising master’s program. for one of his class assignments, he analyzed the ‘thraxil’ brand. here’s a nice diagram of his results (warning: pdf).
at work the other day, i gave a little mini presentation on my experience using python’s profiler. afterwards i wrote it up and posted it to our internal weblog. i thought it might be worth preserving here as well with a little editing.
one of the applications i work on is the “Image Annotation Tool” (or IAT), which allows users to manage collections of images and annotate them. there’s a lot more to it than that, but for our purposes here, that’s enough to understand.
the IAT had been functioning fine in the sandbox where i’ve been doing development. last week i migrated the data from last semester to the format that the new version needed and updated the IAT code on production. when i tried loading the page that shows every single image in my library (607 of them) on the production server, it was working but taking several minutes to load. my sandbox library with 60 images loaded in about 2 seconds. clearly there were some scalability issues that needed to be looked into. i decided it was time to figure out python’s profiler.
a profiler is a program (or library in python’s case) that will examine a running program and collect statistics on how many times functions are called and how long they took. there are other profilers that will track things like memory usage, files opened, and other resources, but for now, i’m just interested in CPU time.
the conventional wisdom is that premature optimization is the root of all evil and that code should be written in the most straightforward, maintainable fashion without much regard paid towards performance until it’s demonstrable that the code doesn’t run fast enough. at that point, a profiler should be used to find the bottlenecks in the code and the bare minimum of optimizations should be done to get the code running within acceptable limits. the rationale is that most optimizations make code harder to understand, maintain, and modify; since programmer time is far more expensive and limited than CPU time, they should only be a last resort. the other part of the reasoning is that programmers are notoriously bad at guessing where the bottlenecks are in a piece of code (and notoriously confident that they are good at guessing). it’s far too common to see programmers spend time complicating their code adding optimizations that speed up a part of the code that’s only responsible for half a percent of the overall execution time anyway. you don’t have to be a software engineer to understand this logic; it’s Cost Benefit Analysis 101.
this is how i’ve tried to write code for years now. my experience so far with web application and database development has been that optimizations are almost never necessary at all. i just write things in the most straightforward way i can think of, using convenient abstractions and high level interpreted languages planning on optimizing another day. the other day rarely comes. at work we’re usually not working with large enough amounts of data, thousands of simultaneous users or doing complicated enough things that the backend code ever approaches being the bottleneck (which is usually just network latency and browser rendering time). on the rare occasions that speed became a factor, i didn’t really bother using a profiler; i would just sprinkle a bunch of print statements in the code that would print labels and timestamps to the logs. then i could dig through the logs and get a rough idea of where things were bogging down. that strategy works fairly well if the code is simple enough that you have a good idea where the bottleneck might be. i’ve been pretty lucky with my guesses in the past and things have generally worked out, but i’ve chased a bottleneck up the wrong tree enough times that i’m now willing to accept that profilers could be useful.
for the IAT bottleneck, i of course had my guesses as to what was slowing things down. i’m using SQLObject as an ORM(Object Relational Mapper) and my experience with ORMs is that when you fetch a large collection of objects from the database, they sometimes decide to fetch the data for each object in seperate SQL SELECTs. so for 607 images, i wasn’t going to be surprised if i saw 607 seperate database calls being made instead of one SELECT returning the data for all 607 in one pass.
python’s profiler runs in two stages. the first stage collects the data from running the app and stores it in a log. the second stage parses that data and lets you generate reports from it. the two reports i chose to generate were a list of the function calls sorted by the time spent in each function (“stats.txt“) and a list of which functions called which other functions (“callees.txt“) which is useful for tracing through and making sense of the results of the first report.
the first run gave me this for stats.txt (i’m just going to show the top few lines for the sake of brevity):
each line has the number of times that a particular function was called, the total time spent executing that function (not including time spent in functions that it calls), the average time for each call of the function, the total cumulative time spent in that function (including time spent in functions that it calls) and the average cumulative time per call.
to make sense of a profiler report, you have to be a little familiar with the code that you’re profiling and the libraries that it calls. eg, it’s important to know that DBConnection.py, Cache.py and SQLObject.py are part of the SQLObject library and htmltmpl.py is the templating library that i use to generate the output. iat.py is the main file with IAT specific code.
so the report tells us that it took 220 seconds to run the code. we can also see that a lot of time (157 seconds) was spent in the _queryAll() function in DBConnection.py which was called 6454 times. that’s the function that does a select from the database. so the code is basically doing over 6000 database calls. no wonder it’s running slow! the next step is to figure out why. so far, it seems to support my hypothesis that the ORM isn’t doing things in the most efficient manner. looking at the cumulative time column, we can see that there are a couple big ones being called from in iat.py (the mysterious 220 second <string>:1(?) line can be ignored; it’s just how the profiler reports string conversions. since the outermost layer of the code returns a string, it picks up all others in the cumulative time, but you can see that it only uses a second and a half total time itself). data() on lines 632 and 845 of iat.py seem to account for a lot of the cumulative time.
line 632 is the data() method of the Image class. since there are 607 images that we need to get the info on, it’s not too surprising that that method is called a lot. the reason that it’s called 1001 times instead of 607 times is probably that the page has a sidebar with a list of libraries, collections and image counts, so that probably accounts for some extra calls beyond the 607.
but line 845 is a little more mysterious. it’s the data() method of the Annotation class. (the basic hierarchy of the IAT is Library -> Image -> Group -> Annotation -> Shape). so Annotations are subcomponents of Images. the question is why are we fetching Annotations’ data when we’re just making a page with an index of thumbnails. digging into the callees.txt report, we see:
Image.data() is calling Group.data() (line 774) 469 times and Group.data() is calling <code>Annotation.data()</code> 3035 times. Group.data() calling Annotation.data() makes sense; pretty much any time you want the data for a group of annotations, you’re going to want the data for the annotations within that group as well. but why would Image.data() be calling Group.data() at least in this case? we have no use for the groups and annotations data here.
so i take a closer look at Image.data() and see that sure enough there’s a line:
groups = [g.data(user=user) for g in self.groups]
in it. so Image.data() is fetching the data for each group attached to that image. the interesting thing is that ‘groups’ is never used after that point. so the line is totally superfluous! it serves no purpose but seems to be responsible for a lot of database calls. i don’t know why i wrote it; i probably had an idea for something, got part of it in there, then got distracted and forgot about it. it happens. on the sandbox library, it wasn’t slowing things down enough to be be noticable so it never got removed.
removed the offending line and ran the profiler again.
down from 220 seconds to 64 seconds (about a 70% improvement) with a one-line change. not too shabby. while my guess about the ORM slowing things down with a lot of DB calls was partially right, we’ve already seen that it wasn’t the primary bottleneck at all. without the profiler, i probably would have caught the problem eventually, but i probably would have rewritten a lot of code first only to see it not speed things up very much. profiler: 1, anders: 0.
ok. a 70% improvement is pretty sweet, but 64 seconds to load a single page is still not really acceptable. the top line now seems to be a single call to htmltmpl.py‘s process() function which is taking about 32 seconds. htmltmpl is the templating library that i use to seperate the business logic from the display logic. it’s generating a pretty big page from a lot of data, but 32 seconds still sounds like a long time. a coworker and i have argued about the usefulness of templating libraries versus their overhead many times before with my claim being that they would probably never be the bottleneck in one of our applications. was i going to have to eat some serious crow here?
at this point, i actually started re-reading the profiler documentation, sure that i was misinterpreting the numbers somehow. i couldn’t imagine how it would be taking up so much time. i was especially surprised, since, as one of the developers for htmltmpl.py, i could remember merging a patch sent in that had some performance enhancements and the informal benchmarks i’d done at the time using much larger datasets didn’t take anywhere near that long to run. so i started digging into the copy of htmltmpl.py installed on our production server. the problem was immediately clear. i had never upgraded htmltmpl on that machine to the version with that patch; it was a much older version installed. so i did an upgrade and re-ran the profiler.
that cut it in half again. down to under 2 seconds to process the templates. for a monster page of 600 thumbnails, that’s good enough performance from the templating engine. without the profiler, i may never have noticed that htmltmpl.py was a version behind. i never would have suspected it of being a bottleneck. so we’ve sped things up by almost an order of magnitude now by removing a single line of code and upgrading a library. both optimizations that i wouldn’t have come up with on my own in the same amount of time. profiler: 2, anders: 0.
actually the score should be even higher for the profiler. during this process, there were actually a number of times where, despite the profiler data sitting right in front of me, i was tempted by my intuition and made changes here and there that i thought would at least speed things up a few seconds. in every case, using the profiler confirmed that my intuitions were wrong and resulted in negligable improvements at best. all of those changes were backed out and took a little bit of my ego with them.
of course 34 seconds still wasn’t quite fast enough for me (i’d be happy with something under 10 seconds. loading a page of 607 thumbnails will bring many browsers to their knees, so it’s not something that we expect users to be doing that frequently and if the backend takes less than 10 seconds, the bottleneck will then clearly be back to network latency and page rendering time, which we can’t do anything about). since Image.data() is taking 27 seconds of the time, it looks like we’re now at the point where convincing the ORM to do things more efficiently will make for a significant improvement.
i’ll spare you the profiler output, but with a few more passes, i’d gotten it down to between 8 and 12 seconds (depending on the server load). once again, my intuitions weren’t quite right.
we highlight thumbnails for images that have annotations attached to them and it turned out to be the code that counted how many annotations each image has that was running slow. the Image class has a method called num_annotations() which would return that count. num_annotations looked like:
def num_annotations(self):
cnt = 0
for g in self.groups:
cnt += g.num_annotations()
return cnt
so again, we were looping over all the groups for every image in the library. my first pass at fixing it was to do:
ie, to let SQLObject do one query (with a relatively complex join) for each image instead of looping. well, imagine my surprise when i ran the profiler again and found that that actually worsened performance by a few seconds. since most images only had 1 or 2 groups at the most, the old way was really only doing 2 or 3 very simple queries for each image while the newer one was doing 1 complex query. apparently the joins were bad enough that it was slower than a couple simple queries.
my next approach actually worked. i got rid of num_annotations altogether and at a higher level, where we were getting the list of images in the library, did a single complex query to get a list of every annotation in the library and then looped over the list of images and marked them as annotated if they appeared in that list. in that case, the expense of one really complicated query was much lower than 600 - 1200 or so simple queries.
score yet another point for the profiler.
this is probably good enough for me. the backend takes around 10 seconds to load a page with 607 thumbnails but the browser takes at least that long just to render the page (more the first time since it has to fetch all the thumbnails). other pages take a second or two (so the gathering of sidebar data isn’t taking too long on large libraries). i could probably get it a bit faster, but the gains would be smaller and smaller and the costs (in terms of complexity and time invested) would keep increasing.
so the bottom line is that if you are optimizing code and not using a profiler, you’re probably wasting a lot of time. if you are optimizing code before verifying that there is even a problem, you’re definitely wasting time.
Philips has a pretty neat new camera which they are advertising as “wearable”.
i’ve had a strong interest in the field of wearable computing for a long time now so i was naturally interested. unfortunately, their idea of “wearable” is that it goes on your keychain. one of the attributes in wearable computing is that it should be unmonopolizing, ie. it shouldn’t require the user to actively do anything to use it. with most cameras, including this one, you have to pull it out, point it at your subject and click a button to take the picture. all that this camera really has as an advantage at that point is that it’s small enough to always carry with you.
what i’d like to see is a really wearable camera in the wearable computing sense. with a bit of hacking on the hardware, this camera might be a good candidate. if you could modify it so it clips to your shirt pocket and just snaps a photo every 10 seconds or so, that would be pretty useful. you would have an automatic record of your day. you would get 99% boring, random, dark, and out of focus pictures, but every once in a while it might capture something interesting that you never would have shot if you’d had to pull it out of your pocket, aim, and shoot.
the reassuring thing about this camera is that in terms of size and power, we’re really almost there. the camera can be set to take low-res 640x480 photos and has 128MB of memory. that’s about 1000 pictures, which is about 1 picture per minute for 16 hours. so if you synched it up every night, you’d be fine. i’m sure that within a year or two, you could buy a version that had 1GB of space which pretty much gets you to the 10 seconds between shots point. with a bit more intelligence built in, you could probably also have the camera detect that light levels are too low or that nothing is in focus and not bother taking a bunch of the pictures that you would end up throwing away anyway. within 5 years, a device this size for a reasonable price could take decent quality video of your entire day.
probably the limiting factor on the camera is the battery life. it might have space for 1000 pictures, but i doubt the battery could handle staying on for 16 hours. haven’t been able to find any info on its battery life though.
i’m really tempted to buy one of these, glue a clip on the back of it, and rip it open and see if i could add some circuitry to automatically snap pictures once a minute. could be a fun project. need to find out more about the battery life though to know if it would be worth trying.
Emmanuel Goldstein, from 2600 was one of the many arrested during the RNC protests this week.
<p>apparently, he was arrested for trying to film one of the protests. </p>
<p>i have nothing to say really except that i’m currently reading <a href="http://www.amazon.com/exec/obidos/tg/detail/-/0375422307/">Persepolis</a>, which is the story of a girl growing up in Iran during the Islamic Revolution. she recalls people being arrested for the crime of simply taking pictures of the protests that led up to the revolution. </p>
<p>something is really wrong with a government when simply documenting real events becomes a crime.</p>
as wonderful as RSS and Atom feeds are, there are some basic questions about its scalability that are becoming increasingly pressing as aggregators become more popular. one of the proposed solutions is to build on top of a protocol that fits the syndication model better than HTTP(Hyper Text Transfer Protocol). one called NNTP(Network News Transport Protocol) already exists and is in widespread use as the foundation for network newsgroups. its scalability is proven (if it can support the whole alt.binaries.* hierarchy, weblog feeds should be nothing) and there are plenty of battle-tested server implementations and client libraries already written. there’s been a lot of talk about building some sort of RSS -> NNTP bridge, but not much action.
anyway. i’ve modified thraxil’s templates to support his test schema, so once i get it registered, you should be able to read thraxil in gnus or trn or whatever you use for reading newsgroups.