+- ScrapeBox Forum (https://www.scrapeboxforum.com)
+-- Forum: ScrapeBox Main Discussion (https://www.scrapeboxforum.com/Forum-scrapebox-main-discussion)
+--- Forum: General ScrapeBox Talk (https://www.scrapeboxforum.com/Forum-general-scrapebox-talk)
+--- Thread: [Email Scraper Plugin - URL Generator] 2 distinct PAGENUM (/Thread-email-scraper-plugin-url-generator-2-distinct-pagenum)
[Email Scraper Plugin - URL Generator] 2 distinct PAGENUM - PixAtom - 07-25-2018
Hi everyone,
I'm using the URL Generator in the Email Scraper to scrape a directory. The URLs look like this
Code:
https://domaine.fr/page?keyword&from=1&to=20
https://domaine.fr/page?keyword&from=21&to=40
https://domaine.fr/page?keyword&from=41&to=60
...
https://domaine.fr/page?keyword&from=1781&to=1800So I tried
https://domaine.fr/page?keyword&from=1&to={pagenum}
but the website crashes when the pagenum is too high.
I want to do something like
https://domaine.fr/page?keyword&from={pagenum-20+1}&to={pagenum}
I tried with
https://domaine.fr/page?keyword&from={keyword}&to={pagenum}
with 1, 21, 41, 61, etc... as {keyword}, (from an Excel) but I have thousand of useless URLs...
Can someone help me?
Thanks!
RE: [Email Scraper Plugin - URL Generator] 2 distinct PAGENUM - loopline - 07-26-2018
I don't think I have a good solution for you. If the numbers sometimes don't get used or go to high. You could run them thru the alive checker first, to get an idea.
If the numbers are consistent, thats easy enough. If they are random, its just dealing with mass. You just have to work thru it.