10-24-2011, 08:44 AM
Hi, I have a site which is similar to dmoz. It's a directory with thousands of URLs (all different).
Is it possible to scrape all of these URL automatically?
Been trying site:www.url.com, but it only scrapes the directory's URL not the sites present in the directory.
Thanks, V
Is it possible to scrape all of these URL automatically?
Been trying site:www.url.com, but it only scrapes the directory's URL not the sites present in the directory.
Thanks, V