I'd like to copy the text from a particular section of a website that's spread over 185 pages. Rather than copying and pasting - anyone have a suggestion on where I can find, or write a scraper?
Two approaches. There are some good off-the-shelf tools out there (I think one was called WebScraper Plus.)
A company called Mozenda (http//www.mozenda.com/) was telling me they planned to offer a SaaS scraping tool. I haven't used it, but it looks interesting.
Third is to have Turkers do it. You could probably get it done right for a few bucks.
Let me know what you settle on. I'd be happy to offer more help if you'd like.
Thanks Keith. In the short term, because I'd like the data sooner than later, I'm tempted to go the Turk route, though I've never used it before; have you? How do you determine prices/fees? How much specificity can you give in the description?
I intend to do the same thing at other websites as well, so I'll look into the two tools you recommended as well.
I've used it. For what you're asking, you'd probably be fine paying $.03 or $.04 per site.
Shoot me an email with more detail and I'll help you figure it out.