Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Grab links from very large sites
#1
Hi,

is it possible to scrape URLs of larger sites like Am*zon ? 

I tried it with the Grab Links function, but it ends with 168730 URLs.
Reply
#2
Sure. You mean internal or external urls or ? Probably your going to fast and its getting you blocked, or maybe thats the max crawl depth.
Reply




Users browsing this thread: 1 Guest(s)