+- ScrapeBox Forum (https://www.scrapeboxforum.com)
+-- Forum: ScrapeBox Main Discussion (https://www.scrapeboxforum.com/Forum-scrapebox-main-discussion)
+--- Forum: General ScrapeBox Talk (https://www.scrapeboxforum.com/Forum-general-scrapebox-talk)
+--- Thread: Tips For Scraping Google? Even If It's Has To Be Done Slowly (/Thread-tips-for-scraping-google-even-if-it-s-has-to-be-done-slowly)
Tips For Scraping Google? Even If It's Has To Be Done Slowly - msj484 - 09-17-2016
Hey All,
Just wondering if any of you have tips for scraping Google with SB these days without getting blacklisted within 20secs. How are you all doing it?
I am trying to search for basic queries like:
"keyword" + "specific footprint"
Thanks ahead of time for your help.
RE: Tips For Scraping Google? Even If It's Has To Be Done Slowly - loopline - 09-20-2016
Yes, I have a 22 min video dedicated to just that
https://www.youtube.com/watch?v=GadX5AXiW34
RE: Tips For Scraping Google? Even If It's Has To Be Done Slowly - msj484 - 09-20-2016
Of course, I've seen this and almost all of your other videos, but something I didn't think about doing was to drastically slow down how fast I crawling. Thanks for the reminder to look at your video to find the "sweet spot". Take care
RE: Tips For Scraping Google? Even If It's Has To Be Done Slowly - loopline - 09-20-2016
Your welcome. The sweet spot is a great thing.
Also in the latest versions of Scrapebox there is now an option under settings >> connections timeouts and other settings >> more harvester options >> proxy change interval - this now allows you to specify how often a proxy is changed.
Typically I would suggest a lower amount for private proxies and a higher amount for public proxies. But you can experiment and see what works best for you.
RE: Tips For Scraping Google? Even If It's Has To Be Done Slowly - msj484 - 09-20-2016
Thanks for letting me know that tip also. At this point, my detailed harvester has been running for almost 24hrs with a 20sec delay, but it's been using the same port via back-connect proxies.
Now that I can do a proxy change interval... I can probably speed that up. I'm kind of scared to stop this current scrape though. Lol. Does this interval change apply to detailed harvester as well?
I just read it and noticed that it only applies to the custom harvester. Hmm... I'm not really sure how the number set in the intervals works. Currently it's set to 3, so what does that mean?
RE: Tips For Scraping Google? Even If It's Has To Be Done Slowly - loopline - 09-27-2016
The interval change is only for the harvester. But if its using a back connect proxy port it won't matter as its changing the ip on the back end. I mean unless it only changes the ip like every 5 mins and you have multiple ports or soemthing, then you can force the change ever X often.
that means every 3 requests it changes the proxy. So if your doing 100 results per page and you set it to 1000 results for a keywords, results 100-300 get pulled with 1 ip and then it changes to anther ip for the next 300/3 requests.
RE: Tips For Scraping Google? Even If It's Has To Be Done Slowly - fatbusybee - 10-04-2016
Hi guys
im new to SB and just going through on how to best maximize this tool.
my issue is im trying to use public proxy for now because i actually do not now very much about proxy, connections and such stuffs
how do i know how many proxies to use for just my 1 site if i need to do some manual blog commenting?
and how do i set a better connection delay with a public proxy....
i also discovered that i cant access my SB page rank checker addon - seems not included in mine (so it seems) not included in my addons menu nor in the premium menu (i hope its not a premium addon?)
im still watching some loopline videos i downloaded but most of the details still sound new to me...
RE: Tips For Scraping Google? Even If It's Has To Be Done Slowly - loopline - 10-05-2016
(10-04-2016, 06:43 AM)fatbusybee Wrote: Hi guys
im new to SB and just going through on how to best maximize this tool.
my issue is im trying to use public proxy for now because i actually do not now very much about proxy, connections and such stuffs
how do i know how many proxies to use for just my 1 site if i need to do some manual blog commenting?
and how do i set a better connection delay with a public proxy....
i also discovered that i cant access my SB page rank checker addon - seems not included in mine (so it seems) not included in my addons menu nor in the premium menu (i hope its not a premium addon?)
im still watching some loopline videos i downloaded but most of the details still sound new to me...
Google shut down page rank values so no tool on the planet can get page rank values.
You should use at least 1 private/shared private proxy for posting anything you can get 1 or 5 for like $5 to $10. You need a lot of proxeis for harvesting, but even 1 is fine for posting.
RE: Tips For Scraping Google? Even If It's Has To Be Done Slowly - fatbusybee - 10-05-2016
(10-05-2016, 03:23 AM)loopline Wrote:(10-04-2016, 06:43 AM)fatbusybee Wrote: Hi guys
im new to SB and just going through on how to best maximize this tool.
my issue is im trying to use public proxy for now because i actually do not now very much about proxy, connections and such stuffs
how do i know how many proxies to use for just my 1 site if i need to do some manual blog commenting?
and how do i set a better connection delay with a public proxy....
i also discovered that i cant access my SB page rank checker addon - seems not included in mine (so it seems) not included in my addons menu nor in the premium menu (i hope its not a premium addon?)
im still watching some loopline videos i downloaded but most of the details still sound new to me...
Google shut down page rank values so no tool on the planet can get page rank values.
You should use at least 1 private/shared private proxy for posting anything you can get 1 or 5 for like $5 to $10. You need a lot of proxeis for harvesting, but even 1 is fine for posting.
Thanks Loopline for clearing that out
according to one of your videos you mentioned that its best to post to HPR sites...so how do i get to now if a site has a high page rank (or did i missed something there) ...is PR no more a factor for ranking?
Also can i mix the proxies - lots of public proxies and just 1 or 2 private proxies to mae post
as im using a public proxy how best would you suggest i timed my connection delays (im using a not so fast internet connection)
thanks loopline
RE: Tips For Scraping Google? Even If It's Has To Be Done Slowly - moobs - 10-07-2016
I have 24 private proxies that I use for scraping Google. I've been using a lot of advanced operators with my searches so I've been running at 1 thread using the detailed harvester with a 60 second delay. If I scrape 24/7 how low do you think I can drop the delay to without having to worry about getting hit with captcha blocks?