+- ScrapeBox Forum (https://www.scrapeboxforum.com)
+-- Forum: ScrapeBox Main Discussion (https://www.scrapeboxforum.com/Forum-scrapebox-main-discussion)
+--- Forum: General ScrapeBox Talk (https://www.scrapeboxforum.com/Forum-general-scrapebox-talk)
+--- Thread: Are there still working Proxy Providers for scraping Google?? (/Thread-are-there-still-working-proxy-providers-for-scraping-google)
Are there still working Proxy Providers for scraping Google?? - DevilHands - 06-10-2016
I just got my scrapebox and I can't believe its so hard to look for a proxy provider to work with Scrapebox. At first I went with buyproxies and got their 10 shared proxy package which was being recommended. I haven't even used the proxy yet and they were banned by Google from the very start. They immediately replace the batch, however only about half of them were working. They admitted that they don't know why this is happening, even their fresh proxies weren't working. I love their customer support as they respond in minutes however I told them that I would like to try others out.
I then went with a proxy provider in BHW that offers 15000 rotating proxies. These proxies gives a lot of errors and it gave a lot less results!
I then went with another BHW provider, this time he offers U.S. private proxies but only after 2 mins of scraping it stopped working and all gave 302 errors. T_T
I was using a site: operator scraping it at 1 thread with 30 second delay. Isn't the delay already more than the recommended setting? Not sure what to do anymore. Are there other good providers out there that I can get below $15? I only have a small project and don't want to blow out my budget.
RE: Are there still working Proxy Providers for scraping Google?? - loopline - 06-11-2016
Some blocks of Ips get perma banned by google, but typically its a case of that a ip can take 48 hours to unban after being banned.
When they give you ips, those ips may have been being used by someone else an hour ago, so they are just still blocked. This has happened to me with multiple providers before, if you just wait 48 hours or less they will typically all be unblocked.
If you have made no requests to google in 48 hours and an ip is still blocked you can ask them to replace that specific ip.
I have hundreds of proxies from buyproxies and largely they scrape google just fine so long as I go slow enough for the number of proxies I have loaded.
Well your 2 mins of scraping and then 302 errors means You are getting them blocked by going to fast.
You might need more then 30 seconds. I know one person that was using quite advanced query. He had to use 100 proxies with a 5 minute delay and then all was well.
So "it just depends, you will need to test" is the best answer anyone can give you.
Clearly if 30 seconds got you blocked then it wasn't enough. Your $15 budget will mean you will trade speed for a low price. Basically it means that the more you spend, the more proxies you get, the more proxies you get the faster you can go.
Using 10 proxies is fine, you just have to slow down, maybe way down. But you might be surprised at how many results you can get when you just go slow and let it hum along while you sleep.
further note that there are over 20 engines in scrapebox, you don't have to use google for everything, maybe not at all. Bing is much more gracious on ip bans. Also deeperweb and google api are google powered and there is yahoo etc..
I have a video
https://www.youtube.com/watch?v=GadX5AXiW34
RE: Are there still working Proxy Providers for scraping Google?? - DevilHands - 06-12-2016
Thanks loopline! What do you think about sequential ips? Does using them make them go banned faster?