01-10-2017, 12:21 AM
Randomize Url List on Depth 2-15 of email grabbing...
Basically, when Email Grabber runs, the first level it visits just the single page, but when you get further into the scrape, the urls become consecutive. This causes your proxies to slam the same site(domain) page after page, this causes a 421 or 403 ban for me sometimes. Is there a way to find the Depth URL list when doing a scrape or is there a randomize function?
When the scrapper turns over from level 1 to level 2 scraping, it already knows the urls it is going to scrape for the entire second level, but they are in succession.
If someone has a solution, let me know. Otherwise, Mods will you please pass the suggestion on to dev. Thank you!
Basically, when Email Grabber runs, the first level it visits just the single page, but when you get further into the scrape, the urls become consecutive. This causes your proxies to slam the same site(domain) page after page, this causes a 421 or 403 ban for me sometimes. Is there a way to find the Depth URL list when doing a scrape or is there a randomize function?
When the scrapper turns over from level 1 to level 2 scraping, it already knows the urls it is going to scrape for the entire second level, but they are in succession.
If someone has a solution, let me know. Otherwise, Mods will you please pass the suggestion on to dev. Thank you!