Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[Email Scraper Plugin - URL Generator] 2 distinct PAGENUM
#1
Hi everyone,

I'm using the URL Generator in the Email Scraper to scrape a directory. The URLs look like this

Code:
https://domaine.fr/page?keyword&from=1&to=20
https://domaine.fr/page?keyword&from=21&to=40
https://domaine.fr/page?keyword&from=41&to=60
...
https://domaine.fr/page?keyword&from=1781&to=1800


So I tried 
https://domaine.fr/page?keyword&from=1&to={pagenum}
but the website crashes when the pagenum is too high.

I want to do something like
https://domaine.fr/page?keyword&from={pagenum-20+1}&to={pagenum}

I tried with
https://domaine.fr/page?keyword&from={keyword}&to={pagenum}
with  1, 21, 41, 61, etc... as {keyword}, (from an Excel) but I have thousand of useless URLs...

Can someone help me?

Thanks!
Reply
#2
I don't think I have a good solution for you. If the numbers sometimes don't get used or go to high. You could run them thru the alive checker first, to get an idea.

If the numbers are consistent, thats easy enough. If they are random, its just dealing with mass. You just have to work thru it.
Reply




Users browsing this thread: 1 Guest(s)
Ultimate List-Building Plugin for WordPress