06-01-2011, 08:40 PM
I'm scraping a site for specific pages. This site is enormous with probably millions of pages. The results i'm getting in google are sometimes 40k, 80k...etc with my queries. I know you can only scrape 1000 results in SB. Even with additional operators in the keyword box i'm not sure i can get all the results i need for my query.
Here is the query i wrote for what i'm trying to scrape:
intitle:review "*@*" inurlite.com intitle:appliances|appliance|photo|photographic|audio|visual|video|computer|computers|electronic|electronics|camera|cameras|laptop|laptops -consultants -repair -repairs -training -schools -courses -networks -systems -alerts -youtube -software -developers -guidelines -entertainment
The *@* is to only display pages with email addresses in case you're wondering.
Anyone have an idea how to get all the results I need for this query? I could break it up for each category (appliances/electronics...etc)...but either way i need to get the 40k that google says exists not the 1000 that SB can only scrape.
Any help is greatly appreciated!
Here is the query i wrote for what i'm trying to scrape:
intitle:review "*@*" inurlite.com intitle:appliances|appliance|photo|photographic|audio|visual|video|computer|computers|electronic|electronics|camera|cameras|laptop|laptops -consultants -repair -repairs -training -schools -courses -networks -systems -alerts -youtube -software -developers -guidelines -entertainment
The *@* is to only display pages with email addresses in case you're wondering.
Anyone have an idea how to get all the results I need for this query? I could break it up for each category (appliances/electronics...etc)...but either way i need to get the 40k that google says exists not the 1000 that SB can only scrape.
Any help is greatly appreciated!