(03-21-2015, 09:25 AM)gopo2k Wrote: Yes. You have too few proxies and too many threads.
Get 100 proxies and use 5 threads at once.

(03-21-2015, 09:24 PM)loopline Wrote:

(03-21-2015, 09:25 AM)gopo2k Wrote: Yes. You have too few proxies and too many threads.
Get 100 proxies and use 5 threads at once.

In ratio to what he is using now, he has about 100 proxies at 3 connections so your suggestion would actually make the problem worse, but you are on right track, just not with the number of proxies and threads persay.

~~~~~~~~~~~~~~~`

OP your proxies are getting banned. Thats why the errors. When you paused some of them got unblocked and thus the reason it worked fine for 20 mins and then started over.

The google test only tests against basic keyewords, not advanced operators. You can have proxies that work fine for 1 query and are blocked for the next. I have a video on it here:

https://www.youtube.com/watch?v=P9CbGhfc1aY


You could build a custom test with your exact query and then you would see exact results.

You would just need more proxies, perhaps as much as 50+ total with 1 connection for advanced operators.

Also I would recommend using the detailed harvester, it still only harvests at 1 connection but will give you more details and has the option to add a delay.

So you could wait 48 hours for your ips to be unbanned and then add a 2-5 second delay.

Or you could try some back connect proxies for harvesting, they change the ip every 10 mins on the back end.

Recommendations here:
http://scrapeboxfaq.com/scrapebox-proxies

The bar at the bottom is percentage of total keywords completed.

Scrapebox will have no problem with that volume of keywords, as long as you are using the 64bit version.

The overall effect is to bear in mind that V2 is highly optimized for Scraping and its VERY fast at scraping google, so the concept of using 1 connection on 30 proxies in V1 is probably fine as the harvester is slower. The concept in general just keeps things slow enough you don't get your ips banned, but V2 is so fast and optimized its rotating thru the ips quicker, hence the ban. So you have to add a delay or more proxies to slow things back down to a speed where you won't get banned. That or use back connect proxies that just rotate so you don't have to worry about bans as much.

(03-24-2015, 02:57 AM)scanner737 Wrote:

(03-21-2015, 09:24 PM)loopline Wrote:

(03-21-2015, 09:25 AM)gopo2k Wrote: Yes. You have too few proxies and too many threads.
Get 100 proxies and use 5 threads at once.

In ratio to what he is using now, he has about 100 proxies at 3 connections so your suggestion would actually make the problem worse, but you are on right track, just not with the number of proxies and threads persay.

~~~~~~~~~~~~~~~`

OP your proxies are getting banned. Thats why the errors. When you paused some of them got unblocked and thus the reason it worked fine for 20 mins and then started over.

The google test only tests against basic keyewords, not advanced operators. You can have proxies that work fine for 1 query and are blocked for the next. I have a video on it here:

https://www.youtube.com/watch?v=P9CbGhfc1aY


You could build a custom test with your exact query and then you would see exact results.

You would just need more proxies, perhaps as much as 50+ total with 1 connection for advanced operators.

Also I would recommend using the detailed harvester, it still only harvests at 1 connection but will give you more details and has the option to add a delay.

So you could wait 48 hours for your ips to be unbanned and then add a 2-5 second delay.

Or you could try some back connect proxies for harvesting, they change the ip every 10 mins on the back end.

Recommendations here:
http://scrapeboxfaq.com/scrapebox-proxies

The bar at the bottom is percentage of total keywords completed.

Scrapebox will have no problem with that volume of keywords, as long as you are using the 64bit version.

The overall effect is to bear in mind that V2 is highly optimized for Scraping and its VERY fast at scraping google, so the concept of using 1 connection on 30 proxies in V1 is probably fine as the harvester is slower. The concept in general just keeps things slow enough you don't get your ips banned, but V2 is so fast and optimized its rotating thru the ips quicker, hence the ban. So you have to add a delay or more proxies to slow things back down to a speed where you won't get banned. That or use back connect proxies that just rotate so you don't have to worry about bans as much.

Good advice, Loopline, thanks. I was away for the weekend and I turned off my compy, so I'll try starting it over with the detailed harvester and a delay as you recommend.

*Edit*

So I set the detailed harvester up with a delay of 5 seconds. Set at only one connection still to be safe and slow. It's at around 750,000 URLs after about 15 hours (unfortunately there's no timer on the detailed harvester yet). Every few thousand searches I'll notice there's a group/cluster of searches with 0 results and it still says "harvesting with proxy..." but never seems to complete those queries to get results. Are those instances of where it banned the proxies for awhile again?

Is there a way to save harvesting progress, as in I wanted to turn off my computer for the night then pick it back up the next day in the same spot? That would be HUGE. Then I can swap out proxies if I need to, as well, without losing my progress or can just not have to worry about having to leave my PC on constantly.

I was also going to ask which Back Connect Proxies package you'd recommend, but it seems like it's really just all about how much I want to spend vs. how long I want to wait to get complete results when dealing with lots and lots of advanced searches.

Thanks for the help again!

(03-25-2015, 03:47 PM)scanner737 Wrote: I think I was mostly concerned about power outages. I should invest in a UPC to cover short, second long cutouts because that's a lot of harvesting (progress) lost in that situation. I also don't like to leave my PC on when I'm away for an extended period of time, like a weekend away or vacation.

I'm not sure how realistic from a practical point of view a save feature (even an auto save feature like every 60 minutes or something) would be, but figured I'd mention that might make a nice addition.

I will say though I just realized (my brain works on occasion) that a workaround is to simply observe which keyword the harvester is up to when you want to stop it, then, because the harvester goes in order top to bottom of your keyword list when harvesting, you can just start with that part of your keyword list the next time you boot it up if you did want to actively stop the harvester midway through. Of course you'd have to build your own master harvested URL list because otherwise it will be broken into multiple folders, but yeah that would work just fine for me come to think of it.

By the way, I believe you're correct; each cluster that's returning 0 results is from the same main advance operator (plus all of my base keywords). Even when it's not, when it's in a cluster of other keywords which are returning a handful of results, when I do a manual search for the keyword returning 0 then I can confirm it in my own Google search that there are no results for it. So that's good to know that it's not my proxies so far. Now I'm wondering if my initial, non detailed harvester run was counting "0" return searches as "errors" or if those were indeed banned IPs. I like the detailed for the added detail as you have a better idea of what's going on, though I'd still be great to have a specific notification that proxies are banned on a search and maybe SB could give them some time to breathe before testing them again.

This is great though learning more about SB and getting a better handle on it.