He might have gotten away doing some casual scraping himself, but publishing the tool is asking for trouble... Friendster certainly doesn't have bandwidth to spare for this guy, so I'm pretty sure they'll take steps to block his robot.

If friendster was cool and wanted to encourage this sort of thing, they might just open their MySQL port to the public (I think K5 may do this? someone does...)

Whoa, I'm not so sure that was a good idea. When I wrote my friendster spider so I could graph my network I purposely didn't post the code in public (a few people got copies, including Ben iirc) because I was worried about the amount of traffic it would generate if everyone decided to crawl their network with it.

It seemed the only way to let everyone have at their own network without overloading the already beleaguered Friendster servers is to do some sort of shared p2p database away from the official site so that data already scraped by someone doesn't have to get scraped again. Of course, that gets into legal issues of using their data for a different 'service' (plus up-to-date issues for the data) so I never went forward with it.

I hope the servers hold up....

Edited by author 08-13-2003 04:59 PM
This sort of thing is always fascinating! I've fooled around with similar stuff based on Livejournal's "friends" network; see here for a graph I made of my own friends network near the beginning of the year. (requires SVG)

Also, this is a pretty cool Java program for interactively graphing LJ friend networks.

The coolest thing about this was the link to the graphviz
site. I like graphing tools.

A couple months ago I started working on my own spider, but the purpose was to point out to Freindster and their users that their sandbox was still in the middle of a dangerous world. A quick series of extensive data grabs can be done by any markting department, cultural aggregator, trend research group, nefarious advertising agency to cull and sell a huge treasure trove of personal data that most users would never make public. I ran out of time to finish it up, but now almost anyone could build their own cultural aggregator and better target products to you you don't want.

I don't think that overloading Friendster's server is an issue currently, because of the slow response. 30 sec timeout wasn't long enough, i had to push it to 90! That means that the spider spends a long time waiting for each page - no worse than a real person trying to click around their network.

Now, if the spider tried to put in multiple simultaneous requests to try to go faster, then THAT would be obnoxious and inadvisable to distribute..

Edited by author 08-14-2003 02:19 PM
I just posted the source to a friendster scraper if anyone's interested. It's single threaded, so it doesn't totally whack their servers. I used it to gather the data for my touchgraph-based browser. And to the p2p/friendster caching idea: why not export it to FOAF?

You guys don't get it. It doesn't matter if the spider is single threaded. Mine was single threaded and it was even nice enough to pause between each request. The problem is that by releasing a spider so that anybody can just download it there could potentially be a huge net increase in traffic to the site. Especially, for that matter, something as easy to use as a windows EXE, where even a non-geek can just kick off a spider process (...well, at least they aren't coming with installers yet). Imagine if thousands of people are running these spiders to map their own networks. Sure the spider isn't doing anything other than clicking on links. But it's clicking on every linked friendster it sees, incessantly which is not normal human behavior. Seriously, how do you not see that this is going to be a giant increase in traffic (on an already overloaded set of servers)? It took me days to scrape just a small fraction of my network, by the way.

And FOAF is fine, however it doesn't address the only important issue with caching the data somewhere off of friendsters databases, which is the legal issue of using the company's data for non-approved purposes.

Maybe this is the right approach though. If all the scraping causes problems it may force Friendster to open a net interface to their data so people can build apps on top of it.

Dav, i think we simply disagree. I believe:
1. adding an additional pause, when it already takes 3-20 seconds to retrieve a page, seems ineffective.
2. the number of people that are really into graphing their network is smaller than you might think.
3. the number of friendsters that it is worthwhile to graph is not that large - after a few hundred people, people are too unrelated to be interesting, so spider won't run that long.
4. it's not clicking on every link it sees, but moves gradually outward into the network, gently
5. yes, 'encouraging' Friendster to open a net interface would be a nice side-effect of this stuff

Dav has a point. Does this follow the robots exclusion protocol?
(Not that www.friendster.com has a robots.txt now, but they
might someday.) How long does it wait between page hits? If
it takes 90 seconds to get a page, the site is clearly
overloaded, and a spider should wait an hour or more before
getting the next page. If the response time is speedy, one
page every minute to five minutes is my rule of thumb.

Regarding B.Mindful's warning of "extensive data grabs can be done by any markting department", i feel this is not a big issue because of the anonymous nature of friendster. No real names or email addresses are exposed, so the marketing uses of the data are forced to be rather benign. I also fail to see what "treasure trove of personal data that most users would never make public" is on Friendster. I mean, favorite books and movies? Posting such things anonymously hardly seems like a sensitive disclosure.

Edited by author 08-14-2003 03:19 PM
(corrected spelling mistake)

1. adding an additional pause, when it already takes 3-20 seconds to retrieve a page, seems ineffective.

Yes indeed. I put the pause in before I ever ran the spider just in case. But my point was exactly that it didn't matter, jsut as being single threaded didn't matter.

2. the number of people that are really into graphing their network is smaller than you might think.

Yes I disagree. There are many more reasons to get a copy of a friendster network other than making pretty pictures.

3. the number of friendsters that it is worthwhile to graph is not that large - after a few hundred people, people are too unrelated to be interesting, so spider won't run that long.

Again, you're thinking only of how you intended to use it for your own purposes.

4. it's not clicking on every link it sees, but moves gradually outward into the network, gently

What do you mean? I didn't say every link, I said "every linked friendster." And certainly your spider is doing that? I can see how that could be misunderstood though, so to be clear the behavior I'm talking about is traversing the friend links.

5. yes, 'encouraging' Friendster to open a net interface would be a nice side-effect of this stuff

Crossing my fingers, but not holding my breath :)

/m15: i feel this is not a big issue because of the anonymous nature of friendster. No real names or email addresses are exposed, so the marketing uses of the data are forced to be rather benign

OK, I didn't want to let this out but I feel like I need to make a point. My cousin is going to be producing an event this fall and I want to help her publicize it. I'm going to write a spider that will traverse my friendster network (it will actually start from a fakester account) and send a friendster email to every person who lists a bay area city as their location. I'm going to spam friendster.

This isn't a terribly clever idea, and I'm sure I'm not the only person who's thought of it.

> send a friendster email to every person who lists a
> bay area city as their location.

Ah. The Friendster 'Bulletin Board' feature is supposed to already provide this capability: announcements to your personal network,
although clumsy. I can see your use of the spider in this way as somewhat valid, since it works around the following failings of
Friendster:

1. They need to make the 'Bulletin Board' more powerful, so that you can specify how many hops you want to sent do, and geographically filter.
2. They need to give users control of who they want to hear from, ie. how close a user has to be, either geographically or # of hops, to be allowed to send a Friendster message. This is necessary to balance out the power of the other features, and limit the appeal of spam-like uses. The current default of "any # of hops" is clearly wide open to mis-use.
To summarize, i think that when Friendster adds the safeguards that it clearly needs, the potential misuses and the appeal of large spidering will decrease.

-Ben

On another (related) subject: Fakesters

I feel strongly that both the pro-fakester people and the anti-fakester people (esp. Jonathan of Friendster) are both making a big deal out of nothing. The obvious solution is to simply add a flag for fakesters. This could be either voluntary, or a review process the way it currently is, except that fakesters are flagged rather than removed. Then, each person can choose whether they want to see their Friendster network with or without fakesters in it, and everyone is happy.

Personally, i have a "fake" flag in my spider output format, though of couse it can't be gathered, but manually filled in, so that visualization can toggle those nodes on or off depending on whether you want to see the "Real" social network or not.

re: fakesters: Fakester Manifesto

Dav: a combo applet of your blogosphere + my touchgraph would be neat.

I've also experimented with some derived measures of my friendster network like ranking users by the number of single people of the opposite sex in their immediate network, if they are single ("mackness").

Friendster really needs to come up with reasons to keep people around (besides dating) after they've collected all their real life friends. Something like the contact list your describe is probably the way to go. I know they've pulled a few fake accounts like "Austin Parties" that are being used for this purpose. I bet they want to add support for it formally and charge for the service in some way.