+- ScrapeBox Forum (https://www.scrapeboxforum.com)
+-- Forum: ScrapeBox Main Discussion (https://www.scrapeboxforum.com/Forum-scrapebox-main-discussion)
+--- Forum: General ScrapeBox Talk (https://www.scrapeboxforum.com/Forum-general-scrapebox-talk)
+--- Thread: meta scraper (/Thread-meta-scraper)
meta scraper - scrapingby - 08-08-2017
I´m trying to scrape thousands of meta descriptions. My storm proxy 40 threads seems not to be able to handle it.
I get alot of Failed: Connection timed out
Am i doing something wrong or is the number of threads too low?
RE: meta scraper - loopline - 08-10-2017
Dont use proxies when meta scraping. I mean unless you are just working all with the same domain. But with storm proxies you want to use the 5 minute proxies or something like that. The standard proxies that change every ip are ok for mass, but not good accuracy.
But if its a nice mix of domains then you don't even need proxies. Because your just loading the page and thats perfectly normal, thats the entire reason that the page exists is to be viewed.