Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Stopping after 300 results?
#1
Hi Guys, 

I am trying to scrape LinkedIn profiles for a certain job titles. When I run it, it stops at 300 results every time. The search that I am trying to scrape Google says has 2 million results, but I can only get to 300 reults total. 

Here are the details: 

URL I am scraping: site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" Note: the forum post is converting "colon salaries" to an emoji.
Proxies: Using proxies harvested through SB proxy manager. Currently have 168 proxies that passed the Google Proxy test. 

I've tried using the detailed harvest and the custom harvester. Same result. I also reloaded the custom harvester settings to see if anything had changed. 

Anyone have any suggestions?
Reply
#2
Google soft caps at 300 to 600 results for more advanced queries, clearly 300 with a query like that.

So just add on keywords like

site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" a
site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" b
site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" 1
site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" 2
site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" purple
site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" car
etc..

then just remove duplicates when your done. This makes google return different sets of results from its database.
[-] The following 1 user says Thank You to loopline for this post:
  • rightintwo
Reply
#3
Oh wow- didn't know that. Good to know!

Thank you Loopline!
Reply
#4
Your welcome, have a great day!
Reply




Users browsing this thread: 1 Guest(s)