The Blueprint Training

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Grab Links Problem
I want to grab these links with company data:
I'm using "Grab Links by crawling a site"

Works fine, but I get only get 2200. Any chance to get the entire 20.000?
Try loading in diffferent pages for starting points. ITs probably just running out of crawl depth. So if you load in several differnet pages, like the first page and last and something in the middle or different sections of the site it will crawl thru different areas, if that makes sense.
The problem is the pagination. URL stays the same. Do you have a solution for this ?
You can look at the requests in a debugger when you browsing thru the pages in a web browser. Often times you can see there is pagination but its just not displayed in your browser.

Now if its a matter of javascript it won't work, but sometimes you can pick out the pagination structure in debugger and then just fabricate those urls and they will work.

Http debugger pro is the program I use and it has a free fully featured trial.

Users browsing this thread: 1 Guest(s)