+- ScrapeBox Forum (https://www.scrapeboxforum.com)
+-- Forum: ScrapeBox Main Discussion (https://www.scrapeboxforum.com/Forum-scrapebox-main-discussion)
+--- Forum: General ScrapeBox Talk (https://www.scrapeboxforum.com/Forum-general-scrapebox-talk)
+--- Thread: Harvesting over 1M URLs at a time (/Thread-harvesting-over-1m-urls-at-a-time)
Harvesting over 1M URLs at a time - Gnikf - 02-17-2011
When you harvest URLs and it gets lucky to harvest over 1M, it seems that cuts them off just to 1M in the harvested window later.
So is there a way around this, so you are able to harvest over 1M urls at once?
Maybe some kind of export before that?
RE: Harvesting over 1M URLs at a time - googlealchemist - 02-19-2011
would like to know as well...sucks spending all that time scraping just to find out you were using to many keywords or in my case a site: search on a bunch of domains
subbed
RE: Harvesting over 1M URLs at a time - Gnikf - 02-19-2011
I found that it saves them in Harvester_Sessions so you can get all the urls there in those batch001.txt files, it just makes batch002.txt after the first 1M and so on. Pretty cool
p.s.
I was also doing site: when went over 1M btw :>
RE: Harvesting over 1M URLs at a time - seogawd - 02-21-2011
(02-19-2011, 09:17 AM)Gnikf Wrote: I found that it saves them in Harvester_Sessions so you can get all the urls there in those batch001.txt files, it just makes batch002.txt after the first 1M and so on. Pretty cool
p.s.
I was also doing site: when went over 1M btw :>
yep!!
RE: Harvesting over 1M URLs at a time - Refresh - 02-21-2011
Don't forget to reimport the not completed keywords until you harvest all of them, after that just use their newest plugin, dupe remover to sort out your list and a txt file splitter to split them into 20k batches