04.27.09

LOLcats ISO code

Posted in Random thoughts, Too Much Information at 11:15 am by ducky

I’ve been working on LOLcat subtitles for Sita Sings The Blues, and a friend asked me what the ISO 639 language code for LOLcat was.

It turns out that the ISO 639 language code LOL does exist, and is for the central African language Mongo.  Who knew that the kittehs were African!?!?

Update: My buddy Luther points out that the African Wildcat is the ancestor of the domesticated housecat.  As he pointed out, “Duh.”

Update2: Luther also asks, “I can haz bank scam?”

03.30.09

Dream date

Posted in Random thoughts, Too Much Information at 1:18 pm by ducky

I’ve had a constant question running in the back of my head for a long time, “Who would you have dinner with if you could?”  (I talked about this a little in the post about Steve Wozniak.)

I have expanded it a little to which three people I would like to be at the same table with, or even just to witness.  I’ve decided that my dream dinner team is Randall Munroe of xkcd, Nate Silver of fivethirtyeight.com, and Jon Stewart of The Daily Show.

How to go about pulling that off is a trickier matter.  Step one would probably be to do something really interesting, such that I would be a desirable dinner date…

02.17.09

Dangers of email

Posted in Email, Too Much Information at 4:59 pm by ducky

This happened back in late 1994 or early 1995, but thinking about it still makes me laugh…

Ben Johnson, a graduate student at the University of Illinois developed a wonderful, wonderful webmail tool called @ATS to help NCSA provide email support for Mosaic, the first graphical Web browser.  I was good friends with Mitch, who was in charge of Macintosh tech support for Mosaic, so I heard about it.  I could see that it would be immensely helpful answering all the email questions I got in the course of my job as www.uiuc.edu webmaster, so talked NCSA/Ben into letting me use it also.  Ben warned me that it was still quite new and under development, but I didn’t care — it made my life much easier even in that early stage.

Back in those days, there wasn’t nearly as much spam.  The very first commercial Usenet spam had only come out in January 1994.  Thus I was a little surprised to get a message to the webmaster account with an extremely pornographic subject line, and a body that was a word salad of dirty words.

I forwarded the message to Mitch, and asked in the body of the message, “Do you get messages like this frequently?”  I got back a message from Mitch saying that yes, he did sometimes get messages like to the Mosaic support email address.  Fine.  I forgot about it.

The next day, I saw Ben.  He said that he had needed to do some troubleshooting of @ATS, and had to go into my account to check something out.  I had developed a webmail system myself, so empathized.  This seemed perfectly understandable and reasonable to me.  He continued by assuring me that he was only there for a moment and emphasized vigorously that he did not look in any of the messages, only at the subject lines.

I think he was puzzled but relieved when I burst out laughing.

I did explain why Mitch sent me a message with the subject

Subject: Re: verb your feminine-noun

and let Ben know that he didn’t need to inform the higher-ups about potential sexual harassment. 🙂

11.30.08

Programming persistence

Posted in Hacking, Maps, Too Much Information at 11:21 pm by ducky

Warning: this is a long and geeky post.

From time to time in the past few years, I have mentioned that I was a little puzzled as to why more people didn’t render tiles on-the-fly for Google Maps, as I do in my U.S. Census Bureau/Google Maps mashup.

I have reappraised my attitude.  I have been redoing my mapping framework to make it easier to use.  I have reminded myself of all the hurdles I had to overcome, and discovered a number of annoying new ones.

First pass

I originally wrote my mapping framework in an extreme hurry.  It was a term project, and a month before the end of the term, I realized that it would be good for personal reasons to hand it in a week early.  The code functioned well enough to get me an A+, but I cut a huge number of corners.

Language/libraries/database choice

It was very important to minimize risk, so I wrote the framework in C++.  I would have liked to use a scripting language, but I knew that I would need to use a graphics library and a library to interpret shapefiles.  The only ones I found that looked reasonable were C-based libraries (Frank Warmerdam’s Shapelib library andThomas Boutell’s gd library).   I knew it was possible using a tool called SWIG, but I hadn’t ever used SWIG and had heard that it was touchy.  Doing it in C++ was guaranteed to be painful, but I knew what the limits of that pain were.  I didn’t know what the limits of pain of using SWIG would be.

Projection

I also had problems figuring out how to convert from latitude/longitude to pixel coordinates in the Google tile space.  At the time (December 2005), I had a hard time simply finding out what the mathematics of the Mercator transformation were.  (It is easier to find Mercator projection information now.)  I was able to figure out something that worked most of the time, but if you zoomed out past a certain level, there would be a consistent error in the y-coordinates.  The more you zoomed out, the bigger the error.  I’m pretty sure it’s some sort of rounding error.  I looked at it several times, trying to figure out where I could possibly have a roundoff error, but never did figure it out.  I just restricted how far people could zoom out.  (It also took a very long time to render tiles if you were way zoomed out, so it seemed reasonable to restrict it.)

Polygon intersection

I remember that I spent quite a lot of time on my polygon intersection code. I believe that I looked around the Web and didn’t find any helpful code, so developed it from scratch on little sleep. (Remember, I was doing this in a frantic, frantic hurry.) I ended up with eight comparisons that needed to be done for each polygon in the database for every tile. More on this later.

Rendering bug

The version I handed in had a bug where horizontal lines would show up at the bottom of tiles frequently, as you can see in the bottom left tile:

It was pretty obvious that the bug was my fault, as gd is a very mature and well-used graphics library.  My old office partner Carlos Pero had used it way back in 1994 to develop Carlos’ Coloring Book, so it was clear to me that the problem was my fault.

After I handed in my project, I spent quite a lot of time going through my code trying to figure out where the problem was with no luck.  Frustrated, I downloaded and built gd so that I could put breakpoints into the gd code.  Much to my surprise, I discovered that the bug was in the gd library!  I thus had to study and understand the gd code, fix it, report the bug (and patch), and of course blog about it so that other people wouldn’t have the same problem.

Pointing my code to the fixed gd

Then, in order to actually get the fix, I had to figure out how to statically link gd into my binaries. I like my ISP (Dreamhost) and wasn’t particularly interested in changing, but that meant I couldn’t use the system-installed gd libraries.  Statically linking wasn’t a huge deal, but it took me at least several hours to figure out which flag to insert where in my makefile to get it to build statically.  It was just one more thing.

Second pass

I have graduated, but haven’t found a job yet, so I decided to revamp my mapping framework. In addition to the aesthetic joy of making nice clean code:

  • It would be an opportunity to learn and demonstrate competence in another technology.
  • I had ideas for how I could improve the performance by pre-computing some things.
  • With a more flexible framework, I would be able to do some cool new mashups that I figured would get me more exposure, and hence lead to some consulting jobs.

Language/libraries/database choice

Vancouver is a PHP town, so I thought I’d give PHP a shot. I expected that I might have to rewrite my code in C++ eventually, but that I could get the basics of my improved algorithms shaken out first.  (I’m not done yet, but so far, I have been very very pleased with that strategy.)

I also decided to use MySQL.  While the feeling in the GIS community is that the Postgres‘ GIS extensions (PostGIS) are better than the GIS extensions to MySQL, I can’t run Postgres on my ISP, and MySQL is used more than Postgres.

I had installed PHP4 and MySQL 4 on my home computer some time ago, when I was working on Mapeteria.  However, I recently upgraded my home Ubuntu installation to Hardy Heron, and PHP4 was no longer supported.  That meant I need to install a variety of packages, and I went through a process of downloading, trying, discovering I was missing a package, downloading/installing, discovering I was missing a package, lather, rinse, repeat.  I needed to install  mysql-server-5.0,  mysql-client-5.0, php5, php5-mcrypt, php5-cli, php5-gd, libgd2-xpm-dev, php5-mysql, and php5-curl.  I also spent some time trying to figure out why php5 wouldn’t run scripts that were in my cgi-bin directory before realizing/discovering that with mod_php, it was supposed to run from everywhere but the cgi-bin directory.

Note that I could have done all my development on my ISP’s machines, but that seemed clunky.  I knew I’d want to be able to develop offline at some point, so wanted to get it done sooner rather than later.  It’s also a little faster to develop on my local system.

I did a little bit of looking around for a graphics library, but stopped when I found that PHP had hooks to the gd library.  I knew that if gd had not yet incorporated my horizontal lines bug fix, then I might have to drop back to C++ in order to link in “my” gd, but I figured I could worry about that later.

Projection

I made a conscious decision to write my Mercator conversion code from scratch, without looking at my C++ code.  I did this because I didn’t want to be influenced in a way that might lead me to get the same error at way-zoomed-out that I did before.  I was able to find equations on the Wikipedia Mercator page for transforming Mercator coordinates to X-Y coordinates, but those equations didn’t give a scale for the X-Y coordinates!  It took some trial and error to sort that out.

Data

For the initial development, I decided to use country boundaries instead of census tract boundaries. The code wouldn’t care which data it was using, and it would be nice to have tiles that would render faster when way-zoomed-out. I whipped up a script read a KML file with country boundaries (that I got from Valery Hronusov and used in my Mapeteria project) and loaded it into MySQL.  Unfortunately, I had real problems with precision.  I don’t remember whether it was PHP or MySQL, but I kept losing some precision in the latitude and longitude when I read and uploaded it.  I eventually converted to uploading integers that were 1,000,000 times the latitude and longitude, and so had no rounding difficulties.

One thing that helped me enormously when working on the projection algorithm was to gather actual data via Google.  I found a number of places on the Google maps where three territories (e.g. British Columbia, Alberta, and Montana) came together.  I would determine the latitude/longitude of those points, then figure out what the tile coordinates, pixel X, and pixel Y of that point were for various zoom levels.  That let me assemble high-quality test cases, which were absolutely essential in figuring out what the transformation algorithm should be, but it was very slow, boring, and tedious to collect that data.

Polygon intersection

When it came time to implement my polygon bounding box intersection code again, I looked at my old polygon intersection code again, saw that it took eight comparisons, and thought to myself, “That can’t be right!”  Indeed, it took me very little time to come up with a version with only four comparisons, (and was now able to find sources on the Web that describe that algorithm).

Stored procedures

One thing that I saw regularly in employment ads was a request for use of stored procedures, which became available with MySQL 5.  It seemed reasonable that using a stored procedure to calculate the bounding box intersection would be even faster, so I ran some timing tests.  In one, I used PHP to generate a complex WHERE clause string from eight values; in the other, I passed eight values to a stored procedure and used that in the WHERE clause.  Much to my suprise, it took almost 20 times more time to use the stored procedure!  I think I understand why, but it was interesting to discover that it was not always faster.

GIS extensions

My beloved husband had been harping on me to use the built-in GIS extensions.  I had been ignoring him because a large part of the point of this exercise was to learn more about MySQL, including stored procedures, but now that I found that the stored procedure was slow, it was time to time the built-in bounding box intersection routine.  If I stored the bounding box as a POLYGON type instead of as two coordinate pairs, then calculating the intersection took half the time.  Woot!

Rendering

I discovered that despite my having reported the horizontal lines bug fifteen months ago, the gd team hasn’t done anything with it yet.  Needless to say, this means that the version of libgd.a on Dreamhost has the bug in it. I thought about porting back to C++. I figured that porting back would probably take at minimum a week, and would raise the possibility of nasty pointer bugs, so it was worth spending a few days trying to get PHP to use my version of gd.

It is possible to compile your own version of PHP and use it, though it means using the CGI version of PHP instead of mod_php. I looked around for information on how to do that, and found a Dreamhost page on how to do so.. but failed utterly when I followed the directions. I almost gave up at that point, but sent a detailed message to Dreamhost customer support explaining what I was trying to do, why, and what was blocking me. On US Thanksgiving Day, I got a very thoughtful response back from Robert at Dreamhost customer support which pointed me at a different how-to-compile-PHP-on-Dreamhost page that ultimately proved successful.  (This is part of why I like Dreamhost and don’t really want to change ISPs.)

Compiling unfamiliar packages can be a real pain, and this was no different.  The Dreamhost page (on their user-generated wiki) had a few scripts that would do the install/build for me, but they weren’t the whole story.  Each of the scripts downloaded a number of projects (like openSSL, IMAP, CURL, etc) in compressed form, extracted the files, and built them.  The scripts were somewhat fragile — they would just break if something didn’t work right.  They were sometimes opaque — they didn’t always print an error message if something broke.  If there was a problem, they started over from the beginning, removing everything that had been downloaded and extracted.  Something small — like if the mirror site for mcrypt was so busy that the download timed out — would mean starting from scratch.  (I ended up iteratively commenting out large swaths of the scripts so that I wouldn’t have to redo work.)

There was some problem with the IMAP build having to do with SSL.  I finally changed one of the flags so that IMAP built without SSL — figuring that I was unlikely to be using this instance of PHP to do IMAP, let alone IMAP with SSL — but it took several false starts, each taking quite a long time to go through.

Finally, once I got it to build without my custom gd, I tried folding in my gd.  I uploaded my gd/.libs directory, but that wasn’t enough — it wanted the gd.h file.  I suppose I could have tried to figure out what it wanted, where it wanted it, but I figured it would be faster to just re-build gd on my Dreamhost account, then do a make install to some local directory. Uploading my source was fast and the build was slow but straightforward. However, I couldn’t figure out how to specify where the install should go. The makefiles were all autogenerated and very difficult to follow. I tried to figure out where in configure the install directory got set, but that too was hard to decipher. Finally, I just hand-edited the default installation directory. So there. That worked. PHP built!

Unfortunately, it wouldn’t run. It turned out that the installation script had a bug in it:

cp ${INSTALLDIR}/bin/php ${HOME}/${DOMAIN}/cgi-bin/php.cgi

instead of

cp ${INSTALLDIR}/bin/php.cgi ${HOME}/${DOMAIN}/cgi-bin/php.cgi

But finally, after all that, success!

Bottom line

So let me review what it took to get to tile rendering on the second pass:

  1. Choose a database and figure out how to extract data from it, requiring reading and learning.
  2. Find and load boundary information into the database, requiring trial and error.
  3. Choose a graphics library and figure out how to draw coloured polygons with it, requiring reading and learning.
  4. Gather test cases for converting from latitude/longitude into Google coordinate system, requiring patience.  A lot of patience.
  5. Figure out how to translate from latitude/longitude pairs into the Google coordinate system, requiring algorithmic skills.
  6. Diagnose and fix a bug in a large-ish C graphics library, requiring skill debugging in C.
  7. Download and install PHP and MySQL, requiring system administration skills.
  8. Figure out how to build a custom PHP, requiring understanding of bash scripts and makefiles.

So now, I guess it isn’t that easy to generate tiles!

Note: there is an entirely different ecosystem for generating tiles, one that comes from the mainline GIS world, one that descends from the ESRI ecosystem. I expect that I could have used PostGIS and GeoTools with uDig look like fine tools, but they are complex tools with many many features.  Had I gone that route, I would have had to wade through a lot of documentation of features I didn’t care about.  (I also would have had to figure out which ISP to move to in order to get Postgres.)  I think that it would have taken me long enough to learn / install that ecosystem’s tools that it wouldn’t have been worth it for the relatively simple things that I needed to do.  Your milage may vary.

04.22.08

really strange phishing

Posted in Random thoughts, Too Much Information at 9:19 am by ducky

I got email from a “Carlie Martindaley” today that sure looks like phishing:

Ducky <at> webfoot.com,   (the email address was properly formatted in the message I got)

Hello Ducky,

I am from your Middle School years and finally got your email.

I have fallen in love with your shoes and I just wanted to know, did you spray paint them? They are so shiny, like fresh glass on a mirror, I cannot resist sending this email. Please tell your shoes, I love them, and thier laces are the most beautiful things.

From,

Annoymous

  1. Starting off with my email address before my name looks fishy, like a computer generated it.
  2. I don’t know any Carlie.
  3. I don’t know any Martindaley.
  4. I didn’t go to Middle School.
  5. I haven’t painted any shoes.
  6. I haven’t sent any old pals email.

The strange thing is that there was no call to action in the message!  The only links were a mailto URL attached to my email address (I took it out for the purposes of this post) and a generic Yahoo ad at the bottom of the page.

Strange.  Maybe the spambots have gotten lonely?

01.19.08

Western medicine

Posted in Canadian life, Random thoughts, Too Much Information at 9:11 pm by ducky

Western medicine is amazingly good in some ways. They can sometimes cure things you didn’t even know were wrong with you.

The docs discovered my mom’s PMP on a CAT scan they did looking at what they think was diverticulitis, an annoying but generally easy to treat disease. I believe that she is totally recovered thanks to that early diagnosis. Me, I went to the doctor because I had a bump on my arm, and they ended up checking me out for cancer. (It wasn’t, but it could have been.)

I went to the doctor for three pretty innocuous things. I might not have gone if there were only one, but three together pushed me over some sort of tipping point.

  • The most important thing was that a bump on my arm — which the docs had told me was fine but to keep an eye on — looked different. The skin around it was peeling slightly.
  • One was that my urine output didn’t seem as “forceful” as it should. My brother-in-law had had fibroids in his urinary tract, and the thought crossed my mind that I might have something similar.
  • The last one was so trivial that I honestly can’t remember what it was.

They said my bump was infected slightly. They said that when the infection died down, they could take it off if I wanted. I did and they did.

They seemed far more interested in my urine output, and that ended up causing a cascade of diagnostic tests which culminated in them taking out a polyp six weeks ago. While it turned out to be nothing, there was a non-zero chance that it could have been cancer, where early detection probably would have saved my life.

And that underperforming urine stream?  That thing which seemed too trivial for a visit to the doctor on its own?  It got robust again all on its own.

I am just astounded at how random life is. In only a slightly different version of the universe, I could be saying, “A bump on my arm saved my life.”

the brain is really strange

Posted in Canadian life, Random thoughts, Too Much Information at 8:22 pm by ducky

The brain is really strange. Or maybe I should say, “my brain is really strange”.

The surgery that I mentioned in my last posting was to remove a tiny little uterine polyp. While polyps are almost always benign, I knew that uterine cancer was really nasty. (The Wikipedia article on uterine cancer seems to indicate that it’s usually only nasty if you are post-menopause, but I didn’t read that article until I researched this posting.)

So five months ago, when their diagnostics first surfaced the possibility of a polyp, I could have been really freaked out about it. Fortunately, I am really good at denial for health/safety issues: I once hid away a fear of heights, I was unfazed by a good friend’s 7 cm breast cancer tumour, and I took my mother’s PMP in stride.

Unfortunately, I am not good at denial when it comes to bureaucracy. I was actually quite anxious about the bureaucratic aspect of the prospect of uterine cancer. I was worried that if I got cancer, I would be disqualified from getting Canadian Permanent Residency. I’d have to leave Canada when I graduated, and that would put me in the US without health insurance and with a history of cancer. This seemed absolutely horrible to me.

Intellectually, I realized that it was rather stranger to be worried about losing a visa than about losing my life, but that’s how my brain worked.

Perhaps partly this is because I have seen a lot of friends and family have really seriously hugely awful bad things happen to them, and almost all of them pulled through. The friend who had that 7cm breast cancer tumour five years ago is not just alive but very active. Mom had surgery that required 40 stitches and is — as far as anyone can tell — completely recovered. A high school friend got multiple meyeloma, which is one of the deadliest, deadliest forms of cancer there is. One friend got throat cancer three years ago and is still talking.   Another friend got leukemia, was in remission for three years, and has been fighting again for about two years. Even cousin Ellen was in remission for three years after (criminally) late treatment of her breast cancer.

On the other hand, I’ve seen lots of snafus with paperwork. Constantly. All the time. (Like how the Canadian government couldn’t figure out for the longest time that I spell “Kaitlin” with a “K” and not a “C”!) So in some ways, it is easier for me to believe that bureaucracies would destroy me than that cancer would destroy me.

01.18.08

nausea or euphoria? hmm…

Posted in Random thoughts, Too Much Information at 10:56 pm by ducky

I had minor surgery recently that I wasn’t particularly looking forward to.  It required a general anesthetic, and in my two previous experiences with general anesthetic, I had real trouble with nausea. I basically woke up, rolled over, and threw up.

This time, however, when I woke up, I had no nausea at all.  I felt surprisingly good, and was actually even chatty when I woke up.  I continued to be chatty and happy and only a little tired for the next few hours.

In fact, on the drive home, at one point, Jim remarked, “Honey, you’re high!”  Surprised, I took stock, and had to agree.  I can’t actually speak from experience — I don’t drink, and I have never taken any controlled substances that weren’t prescribed.  But I could tell that I was in some kind of an altered state: I was gregarious, garrulous, and euphoric.  (For those of you who know me, I should say more gregarious and more garrulous.)

What was going on?

One of the nice things about living at Green College is that there are people who can answer just about any question.  Chris, one of the med students here, explained that anesthesiologists usually use usually a mixture of IV and gaseous drugs to knock people out.  Sometimes, the gas can diffuse into the tissues  surgery at the time of application and then come out later.  (He said that about one percent of patients pass out again in the recovery room!)

One of the gases that they commonly use is nitrous oxide.  So basically, I was high on laughing gas!  It only lasted a few hours, but was much more pleasant than the throwing up I usually do.

11.26.07

scalding

Posted in Canadian life, Too Much Information, University life at 7:09 pm by ducky

On the day before US Thanksgiving, I was at my lab late, waiting for my beloved husband to get done with an opera chorus rehearsal. For reasons that don’t matter, I didn’t go home for dinner. Instead, I prepared some yummy cup-of-soup for dinner, adding boiling water from the electric kettle in our lab. I immediately grabbed the cup in my right hand to take back to my desk… and slopped boiling-hot soup on my hand. It landed in the little triangle of skin between the thumb and index finger, where there is a little depression — and saw no good reason to leave. This hurt, so my body acted quickly to throw the water off of the web of skin.

Unfortunately, that meant that soup jumped out of the cup onto the floor, onto my shirt, onto the desk, onto the back of my hand, and even one little splatter onto my forehead. YOWCH!

I quickly grabbed my waterbottle and poured it over the back of my hand (watering one of the office plants in the process). It hurt, but not so much that I thought I couldn’t clean up some of the mess before going 50m to the closest washroom to run more cold water over my hand. So it was a minute or two before I ran more cold water over the burn.

It hurt, but it didn’t hurt that badly. I had never scalded myself before, but I certainly had burned myself before, and it wasn’t painful in that same league. So I ate what was left of the cup of soup and went back to debugging.

After a while, the scald started to hurt. It hurt more and more as time went on. This was really, really strange. I wondered if scalds were somehow different from “regular” burns. It was hurting enough that I was having trouble focusing on my work. “How long would it keep getting worse?” I wondered.

What really made me nervous was that I was supposed to jump in a car and go down to the US in a few hours. I have travellers’ insurance so that I can get medical treatment in the US if needed, but it would probably be a big logistical hassle. Finally, I decided to go to the emergency room a few blocks away. I felt kind of foolish for doing so, but I had never had the experience of a burn’s pain getting worse and worse as time goes on.

As soon as I got outside, my hand started feeling better. This made sense, given the cold and moist air on my bare skin. At the emergency room, they saw me quickly, told me I was fine, and sent me home. I felt relieved but somewhat foolish, but it had been really strange for the pain to increase.

I went back to the lab and commenced working again. I started typing and mousing and typing and mousing again. With my hands. Especially my right hand. And what do you think happened? The pain started increasing again in my right hand!

I felt really, really stupid at that point. It was hurting more and more because I was stretching the scalded area. Duh.

05.27.06

company perks

Posted in Technology trends, Too Much Information at 10:29 am by ducky

Google is famous for its perks, and rightfully so. They have the best perks of any place I’ve ever worked, even better than SGI in its heyday. It seems like Google really wants to eliminate anything that might distract people from doing great things.

It’s also possible that they just want to be nice to their employees, and I found an argument for that: heated toilet seats.

While I haven’t done a rigorous statistical analysis to determine distribution, almost all of the toilets at the Googleplex that I have sampled are high-tech megafunction toilets: the kind that can spray your privates clean and then blow them dry.

The first time I walked into a stall, I rolled my eyes at how over-the-top the toilet was. I mean, how necessary is it to have megafunction toilets?

Then I sat down and discovered the seat was heated, and to my surprise, I found that I had a very visceral response. It was comforting. I suspect that the seat is set to body temperature, and I bet that I have very strong associations of comfort attached to body heat on my butt. I wanted to just sit on that nice warm seat for hours. This argues against the perks being there to improve productivity.

I haven’t tried the toilets’ wash and dry cycle yet — I’m afraid to. After all, I do need to get some work done.

« Previous entries Next Page » Next Page »