The Ultimate Managed Hosting Platform

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Stopping after 300 results?
Hi Guys, 

I am trying to scrape LinkedIn profiles for a certain job titles. When I run it, it stops at 300 results every time. The search that I am trying to scrape Google says has 2 million results, but I can only get to 300 reults total. 

Here are the details: 

URL I am scraping: (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" Note: the forum post is converting "colon salaries" to an emoji.
Proxies: Using proxies harvested through SB proxy manager. Currently have 168 proxies that passed the Google Proxy test. 

I've tried using the detailed harvest and the custom harvester. Same result. I also reloaded the custom harvester settings to see if anything had changed. 

Anyone have any suggestions?
Google soft caps at 300 to 600 results for more advanced queries, clearly 300 with a query like that.

So just add on keywords like (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" a (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" b (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" 1 (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" 2 (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" purple (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" car

then just remove duplicates when your done. This makes google return different sets of results from its database.
Oh wow- didn't know that. Good to know!

Thank you Loopline!
Your welcome, have a great day!

Users browsing this thread: 1 Guest(s)

Looplines Scrapebox List