mindmaster Wrote:However I suggest you split your file in 50.000 or 100.000 urls and work with each one.
Here's how:
Go to addons and use the 'dupe remover'.
On that new window, you will see that this addon has 3 features:
*First is to merge multiple files in one.
*Second is to remove duplicate urls or duplicate domains from a given list.
***And third feature is to split a file.
So load your list there, select how many lines (urls) you want each new file to have and hit run.
thanks I have been looking for this info on spliting files.