12-23-2019, 06:19 AM
(12-23-2019, 03:23 AM)loopline Wrote: First I recommend you try this approach.
https://www.youtube.com/watch?v=bZRh6sZZyz0
If that doesn't work you can try settings >> connections timeouts and other settings >> other >> link checker min threads.
That attempts to force close all connections when you reach the threshold. I like 20%. So if you are using 100 threads on link checking then I would set it to 20 for link checker min threads. Note that if you set the link checker min threads equal to or greater then your total link checker connections that checking will end instantly when you push start on the link checker.
Believe or not scrapebox actually can't just stop the threads. The thing is scrapebox uses threads/sockets and these are a global convention. Scrapebox did not design them, they are used by programs around the world. These are built in such a way where windows literally has control over them.
Coding isn't as cut and dry as it seems. So windows literally has control over the threads and if windows locks a thread or any 3rd party software locks a thread then scrapebox is forced to wait till its unlocked.
Windows doesn't really "intentionally" just lock threads probably, unless it doesn't like the content on a url, but windows just was not built for scrapebox, so windows just has "brain farts" if you will and it just locks threads sometimes.
On my servers I combat this by working in ultra small chunks. I run the automator and first split my source urls into like 100 url chunks, dump them in a folder.
Then have the automator link check 1, and output results and then run a custom script that copies in the next file in the folder and then loop the automator to link check it.
This works well, even google runs everything in small chunks and small servers as its more effective then larger machines etc..
I take it a step further, which is probably overkill for you, but I want stuff to run 24/7/365. So I have another script that monitors the link checker and arbitrarily force closes it every 12 hours. I lose a few links every 12 hours this way, but you can always load them back in to check again, which your doing anyway.
When the link checker is force closed I kill it, scrapebox/automator the works and then the script pauses a few seconds and restarts it scrapebox and the automator which then kicks off the link checker again.
So it runs year round seamlessly, for the most part. There is always anomyles (no idea how to spell that, lol) of course.
Anyway link checker min threads is probably your best bet. Also you can fire up another instance of scrapebox and slit your list in half, that way if 1 locks the other completes you can only lose half the work. you can run unlimited instances of scrapebox on a single machine.
As for windows having control over the threads, it sucks, but its the best option out there, so its a case of its better then the alternative. But you can work around it. Small list size and more then 1 instance is how I do it.
Thanks for your lengthy and thoughtful reply. I freely admit that I don't understand the behind-the-scenes coding controlling such threads. The model I have in my mind, which must be incorrect, can't comprehend the notion of being unable to terminate threads. I concede to my ignorance.
I'm not opposed to setting up automator scripts and doing things in smaller batches and I've been increasingly doing things just like that. It makes sense to do smaller batches for a variety of reasons, but I'm also often impatient and love the feeling to doing big jobs quickly haha
I will follow your advice and try other things though, but probably not the first piece as I really would like to keep my monitor on my desk