+- ScrapeBox Forum (https://www.scrapeboxforum.com)
+-- Forum: ScrapeBox Main Discussion (https://www.scrapeboxforum.com/Forum-scrapebox-main-discussion)
+--- Forum: General ScrapeBox Talk (https://www.scrapeboxforum.com/Forum-general-scrapebox-talk)
+--- Thread: Set Timespan? (/Thread-set-timespan)
Pages:
1
2
Set Timespan? - handsun - 07-20-2016
I cannot find a way to set timespan for scraping (I am looking for current blog posts within the last week.) The only way I see how to do it is to go to google and set the custom date search , than copy that string and add it to Google in the advanced screen. There used to be a Time option and I cant find it anywhere. Can someone please help? Thanks!!
I forgot to add I need to use Detailed Harvester because I am using private proxies, google scraping with advanced operators, so I have to set delay
RE: Set Timespan? - loopline - 07-20-2016
Well there is already a google engine in the harvester engine list that has past week, so you could just use that. If you don't see it scroll to teh bottom of the list of engines. If you still don't see it go to settings >> harvester engine configuration >> import >> download default engines.
But you can add your own date range as well, I have a video
https://www.youtube.com/watch?v=72bC56R_4-M
RE: Set Timespan? - handsun - 07-23-2016
That was it!! Thank you for your help. I have an associated question, I am using the Detailed Harvester because I am scraping google with private proxies, I set the dealy to 180 seconds and the results were coming in pretty fast, and now proxies got banned (I just have to wait 24 hours)
Under settings I have set threads to 1 for harvester and now I just set delay to 60 in Harvester Engine Settings (it won't let me set higher) . Have I set enough constraints like this? Since I scrape google with private proxies, I don't mind it going slow, just prefer not to lose the proxies every time I scrape!
RE: Set Timespan? - loopline - 07-24-2016
The detailed harvester automatically uses only 1 thread, the settings menu is for custom harvester.
What do you mean it won't let you set more then 60 delay? I just set a 500 delay, it should let you set a really high delay. I know some people have super advanced queries and use like 5 min delays.
RE: Set Timespan? - handsun - 07-25-2016
I was referring to the Settings menu. I will try 500 in detailed harvester, I just wanted to be sure I had it covered, 180 still got my proxies knocked off!
RE: Set Timespan? - loopline - 07-26-2016
Ok, you are in the right place.
Part of it is the query your using. But part of it is the history of those proxies in relation to getting banned. It seems the more often google bans a proxy and the more times it has been banned, the quicker and longer they ban it in the future.
So a different set or proxies might be different. But all in all, with any proxy, there is a balance where they won't get blocked. Once you find it, then you can let it harvest round the clock.
My proxy provider gives me a new set proxies every 30 days, I have it worked out where thats my bottle neck. I can literally start harvesting and its not proxies being blocked that causes the harvester to stop, its when they get swapped out weeks later.
RE: Set Timespan? - handsun - 08-01-2016
Hi Loopline, I sure do appreciate your help, I am confused, I got my proxies replaced, set delay to 300 (5 minutes) seconds, 15 results (just so I could test my new footprint) and the results literally came back in 1 second.
http://screencast.com/t/mpA91KPV
http://screencast.com/t/STXmToW7jI7
(I removed the data from the screen in this one, but you can see it is set to Detailed Harvester, 15 results.
There must be another setting somewhere because for 15 results shouldn't it have take 75 minutes at a 5 minute delay? I just don't want to lose my proxies!!
Thanks!
RE: Set Timespan? - loopline - 08-03-2016
Well the first keyword will come back right away and then a delay. Is that what happened?
RE: Set Timespan? - handsun - 08-06-2016
all the 15 results came back in a flash!
RE: Set Timespan? - handsun - 08-08-2016
Hi Loopline, the results all came back within a couple of seconds (15) I thought with one thread that it would take 5 minutes per result