what to use to write a scraper

Live forum: http://forum.freeipodguide.com/viewtopic.php?t=77654


23-10-2008 10:12:49

I'd like to copy the text from a particular section of a website that's spread over 185 pages. Rather than copying and pasting - anyone have a suggestion on where I can find, or write a scraper?


23-10-2008 12:05:29

There's Firefox extensions
[u620795065e]iMacros[=https//addons.mozilla.org/en-US/firefox/addon/3863]iMacros[/u620795065e] or
I'm not sure how to script what you're asking. I was gonna try this also to scrape all the torrent names from several pages, but haven't gotten around to doing it (read lazy)


23-10-2008 16:12:04

Two approaches. There are some good off-the-shelf tools out there (I think one was called WebScraper Plus.)

A company called Mozenda (http//www.mozenda.com/) was telling me they planned to offer a SaaS scraping tool. I haven't used it, but it looks interesting.

Third is to have Turkers do it. You could probably get it done right for a few bucks.

Let me know what you settle on. I'd be happy to offer more help if you'd like.


23-10-2008 20:07:42

Thanks Keith. In the short term, because I'd like the data sooner than later, I'm tempted to go the Turk route, though I've never used it before; have you? How do you determine prices/fees? How much specificity can you give in the description?

I intend to do the same thing at other websites as well, so I'll look into the two tools you recommended as well.


24-10-2008 06:17:30

I've used it. For what you're asking, you'd probably be fine paying $.03 or $.04 per site.

Shoot me an email with more detail and I'll help you figure it out.


24-10-2008 06:40:54

Awesome - thanks!