HtmlAgilityPack download webpage which loaded asynchronously by javascript

c# html-agility-pack javascript web-scraping

Question

i am using HtmlAgilityPack and trying to load some webpages. some webpages are javascript based and loads asynchoronously. is there any way to load web page after x seconds or after making sure page is completely loaded

Popular Answer

Html Agility Pack is not mimicking the client side calls to dynamically load content into the DOM. It is a headless scraper that is downloading the static page given by the server; if you want that content, you will have to mimic the calls made by the client browser. If you do not want to try to emulate the calls a browser would make, instead of using a headless scraper, you can use something like Selenium to do this for you, the down side being, the browser will be opened on the host machine.



Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why