+- ScrapeBox Forum (https://www.scrapeboxforum.com)
+-- Forum: ScrapeBox Main Discussion (https://www.scrapeboxforum.com/Forum-scrapebox-main-discussion)
+--- Forum: General ScrapeBox Talk (https://www.scrapeboxforum.com/Forum-general-scrapebox-talk)
+--- Thread: Google Maps Custom Harvester (/Thread-google-maps-custom-harvester)
Pages:
1
2
Google Maps Custom Harvester - deepx - 01-12-2014
I'm trying to harvest url's from Google Maps using the custom harvester.
The default settings for Google Maps engine are not working and I get this error when using the test option.
![[Image: 213ons2.jpg]](http://i42.tinypic.com/213ons2.jpg)
I also tried to create my own version of Google Maps Engine but I got the same results.
![[Image: 2hfkwoz.jpg]](http://i43.tinypic.com/2hfkwoz.jpg)
Also, when I do a simple harvesting test for keyword "dentist" using the Google engine in custom harvester, I don't get any results. AOL, Yahoo and others work perfectly.
I tried to find a solution online before posting this thread but there are so little in-depth tutorials and informational posts on Scrapebox usage.
Can someone please create a detailed tutorial and explain how to scrape Google Maps for url's or emails?
Thank you!
RE: Google Maps Custom Harvester - loopline - 01-13-2014
(01-12-2014, 07:40 PM)deepx Wrote: I'm trying to harvest url's from Google Maps using the custom harvester.
The default settings for Google Maps engine are not working and I get this error when using the test option.
I also tried to create my own version of Google Maps Engine but I got the same results.
Also, when I do a simple harvesting test for keyword "dentist" using the Google engine in custom harvester, I don't get any results. AOL, Yahoo and others work perfectly.
I tried to find a solution online before posting this thread but there are so little in-depth tutorials and informational posts on Scrapebox usage.
Can someone please create a detailed tutorial and explain how to scrape Google Maps for url's or emails?
Thank you!
Google maps works fine for me. All of your errors point to the fact that your proxies/ips are blocked.
Go to settings >> use multi threaded harvester - and uncheck this. Also under that same settings menu uncheck the use custom harvester. Then try and harvest, there will be a status column. What errors are you getting in the status column?
Here is a video that goes over how to do this and other troubleshooting for the harvester that might be helpful. http://youtu.be/2QaLWgTXsRo
"so little in-depth tutorials and informational posts on Scrapebox usage." Have you looked at my youtube channel? I have over 14 hours of video on how to use scrapebox, in fact the most recent is The Art of Harvesting, which is 24 mins long. Have you mastered everything in it yet? I have a LOT of indepth tutorials.
http://www.youtube.com/user/looplinescrapebox/videos
How to use google maps urls? Easy just tick off google maps in the custom harvester and harvest like normal.
What do you mean how to use it for email? You can use the email grabber to grab emails from urls, but the url that your going to get from google maps is probably a home page, which doesn't likely have emails on it.
You would want to grab your urls with the maps scraper
Trim them to root if they aren't already
do a
site:domain.com
on the regular google engine I mean
Then take the results from that and use the link extractor to pull internal links and then maybe take those results and pull internal 1 more time. Then mash them all together and remove duplicates and then you can use the email grabber to grab emails.
Also under options you can turn on the option to save the url with the mail address, if thats a help to you.
RE: Google Maps Custom Harvester - deepx - 01-13-2014
You were so right!
I'm so sorry for not educating myself enough by watching all of your videos before posting this thread.
I did watch the art of harvesting video and it all looked so simple when your were doing it. I bought my scrapebox license long time ago but I was not using it at the time when I was watching your video.
![[Image: 2hob3mv.png]](http://i44.tinypic.com/2hob3mv.png)
I didn't thought that my IP was banned because I started using a new VPS server that same day and I only scraped for 1-2 thousands to find some free proxies.
Question: Will I solve my IP block issue by using good proxies? If I buy 10 shared proxies that are being shared between max 3 users, will I be able to scrape for like 10k url's each day without a problem. Or do I need to buy private proxies?
After looking at the art of harvesting video where I saw the Google maps in custom harvester settings, I had to give it a try.
Why? Just because of the irrelevancy of highly targeted results in Google search.
If I would let's say do a search for the term "web design company in London, UK", even the top 100 results are not relevant enough. And I would always get sites like Wikipedia, LinkedIn and others which are hard to filter out.
I figured out how to avoid this filter problem by scraping Facebook pages for the same term. None of the email harvesters that I tried were able to pull out the emails efficiently from FB pages.
The email is clearly visible in the info section of each facebook page (link/info) but the harvester is just not grabbing it.
That's why I need Google Maps results. They are highly relevant, even the first few thousands. You told me to use the normal search engine but that won't work.
If I do a search for site:maps.google.com "dentist" "london", I don't get the same results as I would get with maps.google.com search engine. The results are really weird and not that related to each other.
I hope that I'll be able to make the custom harvester work with Google Maps by using shared proxies.
Would you mind sharing your opinion on this? How would you search for such keywords? I'm sure there are some special tricks that you use but many are not aware of.
Thank you for taking the time to answer all these questions. Your goodwill and professional customer support are easily recognizable online.
RE: Google Maps Custom Harvester - loopline - 01-14-2014
(01-13-2014, 08:09 PM)deepx Wrote: You were so right!
I'm so sorry for not educating myself enough by watching all of your videos before posting this thread.
I did watch the art of harvesting video and it all looked so simple when your were doing it. I bought my scrapebox license long time ago but I was not using it at the time when I was watching your video.
I didn't thought that my IP was banned because I started using a new VPS server that same day and I only scraped for 1-2 thousands to find some free proxies.
Question: Will I solve my IP block issue by using good proxies? If I buy 10 shared proxies that are being shared between max 3 users, will I be able to scrape for like 10k url's each day without a problem. Or do I need to buy private proxies?
After looking at the art of harvesting video where I saw the Google maps in custom harvester settings, I had to give it a try.
Why? Just because of the irrelevancy of highly targeted results in Google search.
If I would let's say do a search for the term "web design company in London, UK", even the top 100 results are not relevant enough. And I would always get sites like Wikipedia, LinkedIn and others which are hard to filter out.
I figured out how to avoid this filter problem by scraping Facebook pages for the same term. None of the email harvesters that I tried were able to pull out the emails efficiently from FB pages.
The email is clearly visible in the info section of each facebook page (link/info) but the harvester is just not grabbing it.
That's why I need Google Maps results. They are highly relevant, even the first few thousands. You told me to use the normal search engine but that won't work.
If I do a search for site:maps.google.com "dentist" "london", I don't get the same results as I would get with maps.google.com search engine. The results are really weird and not that related to each other.
I hope that I'll be able to make the custom harvester work with Google Maps by using shared proxies.
Would you mind sharing your opinion on this? How would you search for such keywords? I'm sure there are some special tricks that you use but many are not aware of.
Thank you for taking the time to answer all these questions. Your goodwill and professional customer support are easily recognizable online.
Glad you got the ip ban figured out, google is ban happy these days.
I would definitely just use the google maps option, its going to be the best bet.
You can solve your IP ban issues with proxies. As for the shared proxies, I use a good deal of shared proxies. If you have a good provider and you tell them what your using them for, scraping google, they will try and match you up with people not using them for that. Buyproxies.org has always been great for this with me, but I have others that do good as well.
Else private proxies solves the issue as you are the only one using them.
You should watch the how to safely harvest google with private proxies. http://www.youtube.com/watch?v=Ri6cci288vk
RE: Google Maps Custom Harvester - deepx - 01-19-2014
I managed to fix the issue by buying 10 private proxies but that didn't work as I expected.
I have an unusual problem, my proxies are working for normal Google scraping, but they don't work for Google Maps scraping. Any idea what the problem could be?
My 10 private proxies were working but only on the first day. I was even able to scrape 1-2k urls daily on Google Maps for a few days without using proxies at all. As you already know, my VPS IP is was blocked by Google long time ago so I'm totally confused right now. I'm not able to scrape Google Maps with our without proxies anymore...
Maybe my proxies got banned by Google Maps server and not by Google Search Engine server? If that's the case, is it possible that my 10 private proxies got banned right away by Google Maps. I used 1 connection and 5 sec delay.
I hope that you'll be able to help me
Thank you
RE: Google Maps Custom Harvester - loopline - 01-24-2014
(01-19-2014, 04:05 PM)deepx Wrote: I managed to fix the issue by buying 10 private proxies but that didn't work as I expected.
I have an unusual problem, my proxies are working for normal Google scraping, but they don't work for Google Maps scraping. Any idea what the problem could be?
My 10 private proxies were working but only on the first day. I was even able to scrape 1-2k urls daily on Google Maps for a few days without using proxies at all. As you already know, my VPS IP is was blocked by Google long time ago so I'm totally confused right now. I'm not able to scrape Google Maps with our without proxies anymore...
Maybe my proxies got banned by Google Maps server and not by Google Search Engine server? If that's the case, is it possible that my 10 private proxies got banned right away by Google Maps. I used 1 connection and 5 sec delay.
I hope that you'll be able to help me
Thank you
Yes most likely your proxies are banned by google maps. Ive been doing some google maps scraping and I can tell you for a fact it is EXCESSIVELY AGGRESSIVE at banning IPs. Much more then regular google scraping. Not sure why, but it is.
Given that maps is centered around images and scrapebox and any other socket based scraper thats multi threaded would not work with images or javascript, then google knows its being scraped because the images, which is the key point of google maps, isn't being used.
The only alternative to this is using a single threaded scraper and running it in a browser mode, which is slow and a pain at best.
So I am working with very small batches of keywords and very low connections and/or solid delays. You could up the delay or even go to random. RND and then you can set the random range under settings.
RE: Google Maps Custom Harvester - dorkus - 01-25-2014
i have the same problem. i think there must be sthm wrong with the google maps harvester. i can access the mpas url via webbrowser and proxy with no problems, but the maps harvester always returns 0 results.
i used 3 proxies from different hosting companies and i highly doubt they are all blocked by google....
could you please verify? we really need the maps harvester.
is there any other way to harvest google maps results?
RE: Google Maps Custom Harvester - dorkus - 01-25-2014
i have the same problem with google maps harvester. none of my 10 private proxies from different hosting companies is working. it must have sth. to do with scrapebox.
when i access the url via proxy on the web it works flawlessly.
could you please look into it? thanks!
RE: Google Maps Custom Harvester - loopline - 01-26-2014
Scrapebox Support (not me) is looking into it. Its working flawlessly for them, so its something thats only happening to some people. So they are trying to replicate it so they can fix it.
RE: Google Maps Custom Harvester - dorkus - 01-26-2014
great. let me know if i can be of assistance.