06-07-2012, 11:36 AM
Good afternoon, colleagues and laypersons)
It appeared a very interesting nuance in the parsing of Google ... I am interested in collecting user profiles pages on a particular site. In order for your favorite Google gave what I'm looking for I type the following query:
As its known, Google, like most other search engines, gives only 1000 results maximum for 1 query (but usually finds much more...) But, after I write this request I see next:
And... Google shows only one result, and "gently" asks: "Do you want to see another N number of similar results?" - Agree and click, after which it is seen that G ate 872k profiles from this site (minus a couple thousands of pages of garbage caught here also.) BUT, scrolling down the SERP - and seeing only 50 (fifty, damn!) 50 (!) results from a 872k, O_O WTF, gentlemen?
In fact, brewing question: "How do I can scrape all of these (or most, but not 50 ...) URLs profile from the site?" How to deal with such "bad thing" from Google?
It appeared a very interesting nuance in the parsing of Google ... I am interested in collecting user profiles pages on a particular site. In order for your favorite Google gave what I'm looking for I type the following query:
Code:
"Просмотр профиля *" site:http://forum.guitarplayer.ru/index.p...ion=profile;u= -inurl:topic
And... Google shows only one result, and "gently" asks: "Do you want to see another N number of similar results?" - Agree and click, after which it is seen that G ate 872k profiles from this site (minus a couple thousands of pages of garbage caught here also.) BUT, scrolling down the SERP - and seeing only 50 (fifty, damn!) 50 (!) results from a 872k, O_O WTF, gentlemen?
In fact, brewing question: "How do I can scrape all of these (or most, but not 50 ...) URLs profile from the site?" How to deal with such "bad thing" from Google?