+- ScrapeBox Forum (https://www.scrapeboxforum.com)
+-- Forum: ScrapeBox Main Discussion (https://www.scrapeboxforum.com/Forum-scrapebox-main-discussion)
+--- Forum: Scrapebox Footprints (https://www.scrapeboxforum.com/Forum-scrapebox-footprints)
+--- Thread: Stopping after 300 results? (/Thread-stopping-after-300-results)
Stopping after 300 results? - rightintwo - 07-14-2019
Hi Guys,
I am trying to scrape LinkedIn profiles for a certain job titles. When I run it, it stops at 300 results every time. The search that I am trying to scrape Google says has 2 million results, but I can only get to 300 reults total.
Here are the details:
URL I am scraping: site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurl
alaries -inurl:dir -inurl:jobs "Director of IT" Note: the forum post is converting "colon salaries" to an emoji. Proxies: Using proxies harvested through SB proxy manager. Currently have 168 proxies that passed the Google Proxy test.
I've tried using the detailed harvest and the custom harvester. Same result. I also reloaded the custom harvester settings to see if anything had changed.
Anyone have any suggestions?
RE: Stopping after 300 results? - loopline - 07-16-2019
Google soft caps at 300 to 600 results for more advanced queries, clearly 300 with a query like that.
So just add on keywords like
site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" a
site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" b
site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" 1
site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" 2
site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" purple
site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" car
etc..
then just remove duplicates when your done. This makes google return different sets of results from its database.
RE: Stopping after 300 results? - rightintwo - 07-16-2019
Oh wow- didn't know that. Good to know!
Thank you Loopline!
RE: Stopping after 300 results? - loopline - 07-17-2019
Your welcome, have a great day!