+- ScrapeBox Forum (https://www.scrapeboxforum.com)
+-- Forum: ScrapeBox Main Discussion (https://www.scrapeboxforum.com/Forum-scrapebox-main-discussion)
+--- Forum: General ScrapeBox Talk (https://www.scrapeboxforum.com/Forum-general-scrapebox-talk)
+--- Thread: Proxy Speed & SB crashing (/Thread-proxy-speed-sb-crashing)
Proxy Speed & SB crashing - scrapingby - 05-10-2017
Hey,
i´m scraping emails from domain lists at around 100k per 5 min.
Any idea how much faster it will be with proxies? Do 10 proxies mean 10x speed?
Also for some reason SC seems to crash/become nonrespondant after a while..
i think it might have to do with a browser doing parallel work.. can that be or
are there other issues?
RE: Proxy Speed & SB crashing - loopline - 05-16-2017
No, it may be a tad faster or a tad slower. I mean if you can up the connections by 10 times then it would be faster, and if you had a nice enough mix of domains then it would be fine. But if your scraping all from the same domain that many connections may be very bad.
So are you processing thru 100K domains/urls per every 5 mins?
And are these all from the same domain or a nice mix of domains?
RE: Proxy Speed & SB crashing - scrapingby - 05-19-2017
(05-16-2017, 09:34 PM)loopline Wrote: >So are you processing thru 100K domains/urls per every 5 mins?
And are these all from the same domain or a nice mix of domains?
Yes and no.
So do you mean that if up the connections quantity and add proxies it, it won´t help if it´s from the same domain?
That seems a bit couterintuitive.
RE: Proxy Speed & SB crashing - loopline - 05-20-2017
To a point it can help, but you can easily run more connections then inexpensive hosting can handle, so you can take down a site.
The short answer is "it depends". Try like 2 or 3 times the proxies. So you can try and see, there are a lot of variables that could affect it.
RE: Proxy Speed & SB crashing - scrapingby - 05-22-2017
(05-20-2017, 11:34 PM)loopline Wrote: To a point it can help, but you can easily run more connections then inexpensive hosting can handle, so you can take down a site.
The short answer is "it depends". Try like 2 or 3 times the proxies. So you can try and see, there are a lot of variables that could affect it.
Ok..but if the site is running on a CDN/by a big company with scalable server capabilities, that shouldn´t be a problem, right?
RE: Proxy Speed & SB crashing - loopline - 05-25-2017
Yes, if the site can handle the queries, then its fine.