12-25-2016, 10:49 PM
10054 is a terminated connection. If a back end proxy goes bad etc... or public proxies it can happen a lot. Also if you are hitting a wall and no more results will scrape untill you shut it down and then scrape again that consistent with hitting a threshold for security software and then that software is killing all the connections.
So make sure you whitelist scrapebox in all anti-virus, malware checkers and firewalls.
Also you should note that whitelisting is key, because just disabling such software only stops new rules from forming in many cases, existing rules still fire.
AS for the rotation, you may be pulling from different pools of ips with back connect providers. So if everyone in the 5 minute pool is getting the ips blocked, its going to be worse then maybe the 1 minute pool or the every request pool because they don't have a chance to get blocked by others before they switch so they aren't blocked when you get them. Im guessing, but you could ask your provider how the pools work.
So make sure you whitelist scrapebox in all anti-virus, malware checkers and firewalls.
Also you should note that whitelisting is key, because just disabling such software only stops new rules from forming in many cases, existing rules still fire.
AS for the rotation, you may be pulling from different pools of ips with back connect providers. So if everyone in the 5 minute pool is getting the ips blocked, its going to be worse then maybe the 1 minute pool or the every request pool because they don't have a chance to get blocked by others before they switch so they aren't blocked when you get them. Im guessing, but you could ask your provider how the pools work.