Further digging reveals that it doesn't just use Wifi SSID, it uses some form of fingerprint (possibly the MAC address of the base station which, I think, is going to be visible regardless of encryption etc, tha's why iStumbler etc can tell what make an access point is without any connection). Definitely doesn't use the IP address as that would require connection to that WiFi network.
The iPhones (and iPod Touch) all use
Skyhook Wireless (
http://www.gpsbusinessnews.com/index.php?action=article&numero=603), who have a database of WiFi fingerprints and locations.
So that would explain why turning WiFi off would slow down location finding, as it uses the location info from Skyhook to work out roughly which GPS satellites it should be looking for.
Well, that's my take on it anyway
ETA: That also explains why my location showed up wrong when I moved my accesspoint. I changed the SSID name by adding an extra digit, but the broadcast MAC address would have stayed the same.