I'm working on a few hundred pages of amazon search results for some data analysis, and using hap to parse out the results:
This only returns the first four li's of the result list though when there are more. I've checked and double checked and I'm sure this is right - am I doing something very wrong? I can't see why the results are limited to just 4. Typical page content might be: https://www.amazon.com/s/?url=search-alias%3Daps&field-keywords=100+percent+barstow
Using other search results pages give the same problem and do not give the results on the page, but always significantly less than what's there.
Give this a try:
No guarantees for other pages though because I'm only basing on the link you gave