ruminations

I read this fantastic blog post by Brent Simmons, the author of NetNewsWire about all the decisions that go into making even a “tiny” feature. You can read his blog post here:

http://inessential.com/2009/07/30/anatomy_of_a_feature

It explains all the scenarios you have to think through when designing a feature and how those decisions can make or break the user experience.

This post really resonated with me because it’s exactly the kind of thing I have to do on a day to day basis as a PM at Microsoft.

Brent’s job is a bit different in the sense that he’s both the PM & the Dev. He only has to convince himself that his idea is correct and then go and implement it. When you’re working as part of a larger team, you have to also convince other people (including your manager) about the decisions you’ve made.

Today was the last day of the week long geek fest also known as WWDC. It was a pretty mind melting marathon of sessions, labs and meeting other Mac developers.

Before I went, I read the WWDC 09 survival guide. The information there is super useful, especially for a WWDC newb like me:

http://northisup.com/blog/wwdc-survival-guide-2009-with-additions/

Here’s a bunch of people eagerly waiting in line to get into the keynote (when I arrived at Moscone @ 7:00am, the line was already around 500 people long)

Obviously I can’t talk about any of the session contents because NDA blah blah, but suffice to say that it was a very iPhone centric conference. It was pretty obvious that a lot of people there were eager to get in on the new Gold Rush. Besides the technical sessions, the most enjoyable talks for me were the lunch time talks. I attended 3:

Smule is a company that creates musical instruments you can play on your iPhone. They make great use of all the hardware sensors that are available on the iPhone. The first demo they showed was of a lighter app, except that this was not just ANY lighter app. It had realistic physics, i.e., if you tilt the phone, the flame will keep pointing up. If you turn it 90 degrees, the phoen will make a scorching noise, mimicing what would happen if you held a real lighter to your iPhone. You can also turn the lighter into a blow torch to light up another iPhone that has the app.

The second demo was of the flute app. You hold the iphone like a sandwich and blow into the mic and adjust the tone by pressing down on various areas of the iPhone surface. The presenter demoed the app by playing the Final Fantasy theme song. Big applause considering the audience. What’s amazing is that the app reports back to their server the location of the user. They were able to plot this on a globe and it was really interesting to see which areas of the world lit up, so to speak.

Then the presenter went on to demo Sonic Vox, a real time voice modulator on your iPhone and Leaf Trombone, a Guitar Hero’ish app for the iPhone.

This talk was about technoarchaeology. Not really Mac related, but really fascinating none the less. Basically, they got a hold of a bunch of pictures that the Lunar Orbiter (1986) took on tapes. The project was to attempt to restore these images, which was no trivial task considering how old the tapes were and that no one produced the drives necessary to play them back. Their home base was a recently closed McDonalds. They wanted to use their freezers apparently Anyway, here is one of the images they restored:

The last lunch session I attended as given by the CEO (Neil Young, no relation to the musician) of ngmoco. It was similar to the Smule talk in that he chronicled the life of the company. He was quite candid and shared some very interesting numbers that I bet a lot of people in the iPhone gaming business would die to see. Unfortunately the slides are not going to be available. There was also a brief section on raising money which was not all that revealing. The focus was having the right team, which a) is a no brainer and b) was quite easy for them to pull of since the CEO spent 15 years at EA so was quite knowledgeable about the gaming market.  Key take aways for me were that the iPhone gaming business is really thriving and that you can really build a sustainable business doing it. ngomoco is not only focused on writing hit games however. They have a deeper vision of the market and are building all their games ontop of their gaming platform, which the CEO hopes to be a “lovechild of Xbox Live & Facebook”. Services like Achievements and recommendations are already implemented and he promised more was to come.

I’m quite thankful that my company sponsored my trip and I’m already looking forward to the next one.

TextExpander is a incredibly useful Pref Pane plugin for OSX. The basic premise is that you define “snippets” and a delimiter. When you’re typing, you type a keyword + the delimiter and TextExpander will automatically insert the snippet for you. For example, I often have to sign my emails with different signatures. In Entourage, when I am composing an email, I can just type: “sig1′” and that automatically gets translated to “Cheers, Yi”.

One problem I’ve had with TextExpander so far is that it doesn’t offer syncing of your snippets unless you have a MobileMe subscription. Fortunately for us, it stores all the snippets and settings in a simple XML plist file in ~/Library/Application\ Support/TextExpander/Settings.textexpander.

All you have to do to sync your snippets & settings across all your machines is to copy that file somewhere into your Dropbox directory and make a symbolic link (ln -s <src> <dst>) from the Dropbox location to your Application Support folder.

Yesterday I spent 2 hours trying to figure out how to wrap a single Objective-C class using the PyObjC bridge. After looking high and low and browsing through the minimal pyobjc documentation, I came up empty handed.
The closest I got was how to extract the classes from a nib, or a framework. To do that, you

import objc
objc.loadBundle(bundle_path=pathToYourBundle)

After spending another hour trying to figure out how that worked so I could wrap a single class, I gave up and emailed the pyobjc-devel list. To my surprise, I got a response within a couple of hours and was totally embarrased by the answer:

“If you’re using the PyObjC templates from Xcode, and if your objc code and python code are in the same program/project, you can get to your ObjC class(es) simply by importing Foundation in the python module — your class and methods will be found at runtime via PyObjC’s introspection mechanism. Easy!
”

In the end, the solution was to do absolutely nothing!

Disclaimer: I work for the Macintosh Business Unit but had nothing to do with this release

MacBU just opened the public beta for the latest and greatest version of Entourage. It’s dubbed Entourage for Exchange Web Service because it uses a new protocol to talk to Exchange (2007 and above). This version finally does notes, tasks and category syncing.

Get it while it’s hot:
http://www.microsoft.com/mac/itpros/entourage-ews.mspx

I just started my new job at Microsoft in Mountain View. I’m a program manager (sounds fancier than it actually is) on the Mac team here. It’s been a pretty exciting month, arriving in San Francisco with no idea where I’d be living. Lucikly I was able to find a gorgeous townhouse in Potrero Hill, so I’m taking the Caltrain to work everyday. I’ll post pictures of my new pad + workplace soon

A pretty big rewrite of the worker part of dsage landed in trac now. It’s ticket #3600 on the Sage trac and is eagerly awaiting review. Here’s a quick rundown of the major changes:

1. Workers no longer poll the server for new jobs.

It’s been a while since I’ve blogged about dsage so here’s a big braindump on what’s been happening…

I was really happy by the amount of activity and interest in distributed computing at Sage Devel Days 1. I think the major participants were William Stein, Glenn Tarbox and Bill Furnish. 

As the week went on it became clear that dsage as it stands right now does not fulfill the needs of some (maybe even many, or most) users. As a result, right now I am aware of several “next gen” dsage proposals such as dsageNG and some stuff that Bill wrote (i think by the time dev1 was over, Bill was at dsage4 Bill and I chatted a bit about the architecture of dsage currently and the major problem he saw was that dsage workers used polling and there could be a significant (i.e., seconds) delay between when a new job arrived and when a job was processed by the workers. 

I thought about this for a while and I think that he’s absolutely right. Therefore, I’ve rewritten parts of the worker/server code so now workers will start on a job immediately. This should bring down the overhead of running dsage jobs considerably. 

The rewrite is also more Twisted by using twisted’s async process communication. It was actually surprisingly easy to write a worker pool using the existing tools in the framework.

I also added a convenience function called eval_function() that allows you to submit a live function as a job. This works for any function that can be pickled (its arguments as well of course). For example:

sage: def f(n):
...     return n*n
...
sage: j = d.eval_function(f, ((25,),{}), job_name='square')
sage: j
625
sage: print j.wall_time
0:00:00.144780

This is much, much faster than the performance before the rewriting of the workers.

Having eval_function als makes it really easy to the map part of map reduce (it’s also being referred to as scatter/gather). For example:

sage: jobs = d.map(f, [10..20])
sage: jobs
[100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400]

Also, dsage now supports the new @parallel convenience decorator that William wrote:

sage: P = parallel(p_iter = d.parallel_iter)
sage: @P
....: def f(n,m):
....:     return n+m
....:
sage: f([(1,2), (5, 10/3)])
[((1, 2), 3), ((5, 10/3), 25/3)]

William and I both agree that the strategy for Sage should be to have something that is very fast on local multicore machines (multiprocessing module comes to mind) while also having something that will work both on local clusters and WANs (dsage).

If anyone is interested in helping me to test out the new version and making it more robust, please let me know!

I’m hoping that this stuff will make it into the next major release of Sage since I will going on vacation for 6 weeks (hooray) on July 14th.

I discovered freehg.org today, which is a free mercurial repo hosting site. In spirit it’s very akin to github, but obviously lacks many of the features and much of the polish. However, this particular purpose, it’s more than sufficient.

If you access many *nix machines on a regular basis, you’ve probably have been annoyed that your custom configuration files are not immediately available. I used to scp config files around all the time, but that gold old quick. I’m going to show you my current setup for making sure that all the dotfiles (zshrc, vimrc, etc) that I use are version controlled and are easily accessible.

First, collect all your dotfiles in one directory and make it an hg repository. I use ~/.dotfiles, you can use whatever you like.  Here is what my .dotfiles looks like:

iapetus:~/.dotfiles> ls -l
total 54k
-rw-r--r-- 1 yqiang staff   85 2008-07-04 10:20 ackrc
-rw-r--r-- 1 yqiang staff   50 2008-05-29 18:24 bash_profile
-rw-r--r-- 1 yqiang staff 2.0k 2008-05-29 18:24 bashrc
-rw-r--r-- 1 yqiang staff  569 2008-07-04 14:09 create_symlinks.py
drwx------ 3 yqiang staff  102 2008-06-13 15:52 gtk-2.0
-rw-r--r-- 1 yqiang staff  624 2008-07-04 09:48 gvimrc
-rw-r--r-- 1 yqiang staff   48 2008-06-17 15:46 hgignore
-rw-r--r-- 1 yqiang staff  454 2008-07-04 10:30 hgrc
drwx------ 5 yqiang staff  170 2008-06-29 17:55 irssi
-rw-r--r-- 1 yqiang staff  403 2008-06-22 11:20 pdbrc
-rw-r--r-- 1 yqiang staff  642 2008-06-29 10:49 screenrc
-rw-r--r-- 1 yqiang staff 6.8k 2008-07-04 13:39 vimrc
-rw-r--r-- 1 yqiang staff 5.8k 2008-06-29 17:43 zshrc

Then create a freehg.org account and initialize a public repo there. You can find mine at:

http://freehg.org/u/yqiang/dotfiles/

Now, when you access a new machine, to get all your dotfiles in order, just do:

veritas:~ > hg clone http://freehg.org/u/yqiang/dotfiles/ .dotfiles
requesting all changes
adding changesets
adding manifests
adding file changes
added 22 changesets with 37 changes to 15 files
15 files updated, 0 files merged, 0 files removed, 0 files unresolved
veritas:~ > cd .dotfiles
veritas:~/.dotfiles > python create_symlinks.py
Symlinking /home/yqiang/.dotfiles/zshrc to /home/yqiang/.zshrc
Symlinking /home/yqiang/.dotfiles/gvimrc to /home/yqiang/.gvimrc
Symlinking /home/yqiang/.dotfiles/bash_profile to /home/yqiang/.bash_profile
Symlinking /home/yqiang/.dotfiles/hgignore to /home/yqiang/.hgignore
Symlinking /home/yqiang/.dotfiles/bashrc to /home/yqiang/.bashrc
Symlinking /home/yqiang/.dotfiles/hgrc to /home/yqiang/.hgrc
Symlinking /home/yqiang/.dotfiles/pdbrc to /home/yqiang/.pdbrc
Symlinking /home/yqiang/.dotfiles/ackrc to /home/yqiang/.ackrc
Symlinking /home/yqiang/.dotfiles/screenrc to /home/yqiang/.screenrc
Symlinking /home/yqiang/.dotfiles/vimrc to /home/yqiang/.vimrc

create_symlink.py is a simple python script that will create the symlinks for you. Here is the code for it:

#!/usr/bin/env python
import os
home = os.path.abspath(os.environ['HOME'])
path = os.path.join(home, '.dotfiles')
excludes = ['gtk-2.0', 'create_symlinks.py']
for f in os.listdir(path):
    if f.startswith('.'):
        continue
    if f not in excludes:
        dst = os.path.join(home, '.' + f)
        src = os.path.abspath(f)
        try:
            print "Symlinking %s to %s" % (src, dst)
            os.symlink(src, dst)
        except Exception, msg:
            print "Failed to symlink %s to %s " % (src, dst)
            print msg

Tada. All your config files are in place now. If you’re really into it, you can run a cron script that automatically does an hg pull so you don’t even have to think about it.

I haven’t looked at sage -h for a while and was surprised to see many useful convenience features that have been added. I will highlight some ones that I’ve been using constantly that makes Sage development more convenient.

sage  -b [branch]   -- switch to and build SAGE branch in devel/sage-branch
sage  -br [branch]  -- switch to, build, and run SAGE branch in devel/sage-branch
sage  -clone [new branch] -- clone and run a new branch of the SAGE library from current branch
sage  -python       -- run the python interpreter
sage  -sh           -- run $SHELL (/opt/local/bin/zsh) with SAGE environment variables set
sage  -t [-optional] [-verbose] [-long] -- test examples in .py, .pyx, .sage or .tex files
                   -optional -- include examples with 'optional' and 'package'
                   -long     -- include lines with the phrase 'long time'
                   -verbose  -- print debuging output during the test

In particular, sage -sh is really useful for setting all the shell variables.

You can get a list of all the command line options by doing

sage -advanced