02-16-2016, 01:29 PM
(02-12-2016, 02:49 AM)loopline Wrote: But if you try it in a browser its the same thing, if you choose not to omit your only going to get max 1000 anyway, and even then they may not sure that many. In most cases when using an advanced operator google figures if you haven't found what you want by result 600, give or take a little, then your not going to and they just cut you off.
You add in keywords and then remove duplicate urls to solve it.
site:domain.com car
site:domain.com green
site:domain.com 1
site:domain.com a
etc..
That forces google to respond with different sets of results from its database and then just remove duplicates when done. the keyword scraper has an option to append a-z to help get you started.
I ended up at this conclusion myself! I am still unable to scrape everything regardless of the queries I have constructed but I guess some is better than none at all.
Thanks for all your input on this thread loopline, greatly appreciated!