Is it ok to use Broken Links Checker and Backlinks Checker addons without proxies?

Seems to me neither of those two hit Google or any other search engine and so there wouldn't be risks of blocking local ip address.

Just wanted to confirm.

thanks

Hello to everyone.

While scraping Google SERP I came across a problem. When the process of scraping is done I receive only the list of scraped URLs. But how to match them with the keywords these URLs came from? 
I got around 40k keywords so I received 4,000,000 URLs. Do I now have to process them manually to match the urls with the keywords?

Hi Guys,
 
Yesterday I ran a campaign for harvesting a specific niche and it scrapped 150 K records. Now when I cleaned them up was only left with 70 records and it was all full of duplicates. Now my question is can I somehow prevent this from happening so that duplicates are not harvested , like I could set a condition based on emails or name or website , that I only want 1 record per name or website or address. Otherwise it was a total waste of time and wouldn't want the same to happen again. As I am after a unique email from a specific company / organization.

Also for harvesting , please share a footprint that will ensure that the keywords that I harvest won't have duplicate url's or duplicate emails.

Thanks,

So this happened twice , I had been sraping yelp and got almost 10 K results after almost about a day's harvesting , then the scraper hanged , I had to stop the process. Now the session had to be closed which I did through task manager after waiting for hours for it to close (same thing happened with proxy harvestor or email graber ' when I stop it ' the threads get stuck at two and the application doesn't closes) So now I went back to check the harvested data its in the autosave folder, but how do I use that ? where to open it as thats in text file format and I need that in excel sheet form.


Please advise.


thanks,

Hi There,

Was just wondering if there is a way we could scrape , FB posts or Twitter Tweets for certain keywords based on say last 24 hours . I did find a google 24 hrs search engine while harvesting as an option , but it didn't give me any results at all. So any sugggestion in this regard would be welcome.

Thanks,

Hi there,

Loopline shared a great video of using proxies on auto from the file. Now I am using Yelp Scrapper and is there a way we could do the same thing , as I saw , if I  put load from file , it doesn't pick fresh proxies from that file , just picks it once that were there in that file at the time of start , but during the run it doesn't keep refreshing like the standard harvester does ..

Maybe I was missing something.

Hi there,

I am sure there would be an option but am not sure which one or how it would be done. Like we harvest lots of url's , and the process is we remove duplicates.

Then I want to remove the url's with certain words like ;

youtube.
wiki
cnn
bbc


So what I want is perhaps create a file or I did find a blacklist word and edited , put  those words in it , and removed those but those url's still remained , so maybe there is something wrong with how I am doing it.

Also would be great to know if you guys could guide how I can harvest so that these url's containing those stop words are not harvested.

Thanks again

Hi,

is it possible to scrape URLs of larger sites like Am*zon ? 

I tried it with the Grab Links function, but it ends with 168730 URLs.

I buy ScrapeBox for fwgg#pisem.net email. Pay money by paypal when i found what is software is shit i try to make money back, they say it impossible - use it, but they cancel my license and two month i cant activate software what i purchase - what THEY say - use it - you buy it.
THIS IS SHIT.
ACTIVATE MY SOFTWARE OR RETURN MONEY!
this is not a deal for software seller!

Hey Scrapers!

Just a questions. 

Is it possible for scrapebox to crawl through links and check their html to see if they have the facebook pixel installed on the code. 

And save the URL if they dont?

Mick