doylnea
23-10-2008 10:12:49
I'd like to copy the text from a particular section of a website that's spread over 185 pages. Rather than copying and pasting - anyone have a suggestion on where I can find, or write a scraper?
ajasax
23-10-2008 12:05:29
KeithA
23-10-2008 16:12:04
Two approaches. There are some good off-the-shelf tools out there (I think one was called WebScraper Plus.)
A company called Mozenda (http//www.mozenda.com/) was telling me they planned to offer a SaaS scraping tool. I haven't used it, but it looks interesting.
Third is to have Turkers do it. You could probably get it done right for a few bucks.
Let me know what you settle on. I'd be happy to offer more help if you'd like.
doylnea
23-10-2008 20:07:42
Thanks Keith. In the short term, because I'd like the data sooner than later, I'm tempted to go the Turk route, though I've never used it before; have you? How do you determine prices/fees? How much specificity can you give in the description?
I intend to do the same thing at other websites as well, so I'll look into the two tools you recommended as well.
KeithA
24-10-2008 06:17:30
I've used it. For what you're asking, you'd probably be fine paying $.03 or $.04 per site.
Shoot me an email with more detail and I'll help you figure it out.