+- ScrapeBox Forum (https://www.scrapeboxforum.com)
+-- Forum: ScrapeBox Main Discussion (https://www.scrapeboxforum.com/Forum-scrapebox-main-discussion)
+--- Forum: General ScrapeBox Talk (https://www.scrapeboxforum.com/Forum-general-scrapebox-talk)
+--- Thread: *Real* site crawling with Scrapebox? (/Thread-real-site-crawling-with-scrapebox)
*Real* site crawling with Scrapebox? - DigitalMu - 01-05-2020
So I see SB has the site crawler where you can crawl one domain at a time (a deathly slow process)....or you can use link extractor (over and over and over and over) in an inefficient way to grab most links at a time) or you can use the "search google" method to see what pops up....but
Isn't there just like a regular ol' webcrawler? You give it a list of urls, it spidered through all the links, pulls out the links on a domain, and ding - turkey's done.
Am I missing something obvious? It seems like such an obvious tool that should be included. Instead, I find myself now poling aroudn the web to find some alternative.
Any ideas?
Thanks!!
RE: *Real* site crawling with Scrapebox? - loopline - 01-08-2020
The link extractor can be just as effecient as this, in fact more so. Because what your asking for could use so much memory it would routinely crash and thus is one of the reasons it no in scrapebox. Where as with the link extractor method this is not the case.
Also you can use the link extractor method with the automator to make it automated.
video
https://www.youtube.com/watch?v=Ed3SGP_ch3Q