09-04-2012, 08:18 AM
Google limits to access only 1000 SERPs (Search Engine Result Pages) for any queries. It shows the number of links or results it found but doesn't allow to access all.
To scrape profile pages you need to identify the unique user id #, based on which the site fetches the profile from db. See the example.
www.domain.com/profile.php?uid=111
www.domain.com/profile.php?uid=112
www.domain.com/profile.php?uid=56
www.domain.com/profile.php?uid=456
If you get such pattern in url then you can easily target all profiles for scraping.
To scrape profile pages you need to identify the unique user id #, based on which the site fetches the profile from db. See the example.
www.domain.com/profile.php?uid=111
www.domain.com/profile.php?uid=112
www.domain.com/profile.php?uid=56
www.domain.com/profile.php?uid=456
If you get such pattern in url then you can easily target all profiles for scraping.