How to scrape data in a page with jquery button click using HtmlAgility pack

html-agility-pack web-crawler web-scraping

Question

I am trying to scrape data from a page with similar content(Shopping website) using HtmlAgility pack.

There is a button to load more items designed of tag. On click it loads more items on same page.

If it is designed using tag then I will get the next items using the href attribute URL in tag and also I will be loading new page for the new next items, So no problem.

But here no new URL and items loaded on same page.

So is there any way to get this functionality implemented? How to trigger that load more button to get more items?

Accepted Answer

HtmlAgilityPack is an html parser alone, it knows only to parse a static html document. what you want may be accomplished using selenium web driver.

Another possibility is - if the number of item load actions is so that you can complete the loading manually - do so and save the resulting html locally, and only afterwards use HtmlAgiliyPack to parse the static html you stored locally (instead of parsing the http response).

Share the link of the site you are talking about so I can add some code snippets to exemplify.



Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why