10-15-2011, 12:38 PM
I want to scrape one particular domain to submit all their URLs (about 19,000 pages) for indexing.
The domain apparently hasn't got a sitemap (at least I couldn't find them) and if I put site:http://www.rootdomain.com in the footprint window, use proxies and harvest, scapebox deletes 99% results because apparently "the keywords were maybe too" similar.
Should I use the search site differently or is there another way to scrape one domain only?
Can you help?
K
The domain apparently hasn't got a sitemap (at least I couldn't find them) and if I put site:http://www.rootdomain.com in the footprint window, use proxies and harvest, scapebox deletes 99% results because apparently "the keywords were maybe too" similar.
Should I use the search site differently or is there another way to scrape one domain only?
Can you help?
K