HtmlAgilityPack download webpage which loaded asynchronously by javascript

c# html-agility-pack javascript web-scraping

Question

i am using HtmlAgilityPack and trying to load some webpages. some webpages are javascript based and loads asynchoronously. is there any way to load web page after x seconds or after making sure page is completely loaded

1
1
8/27/2016 7:47:22 PM

Popular Answer

Html Agility Pack is not mimicking the client side calls to dynamically load content into the DOM. It is a headless scraper that is downloading the static page given by the server; if you want that content, you will have to mimic the calls made by the client browser. If you do not want to try to emulate the calls a browser would make, instead of using a headless scraper, you can use something like Selenium to do this for you, the down side being, the browser will be opened on the host machine.

0
8/27/2016 8:03:43 PM


Related Questions





Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow