Loading web page and parse its html after asyncronous javascript completes

c# html html-agility-pack html-parsing javascript

Question

For a personal use .net project i'm trying to load and parse a skyscanner.net search results page. I know Skyscanner has APIs for these kind of tasks but it seems there isn't a free personal license to use.

The problem is Skyscanner needs a lot of seconds to complete a search process, so using HtmlAgilityPack to load the document results in a page without the content i'm looking for.

I tried to use WebBrowser object and its event DocumentCompleted, but it seems the event is triggered before the actual search results are loaded into the page.

So, is there any way to load the page, wait for the javascript asyncronous scripts completely fill the page and then get the html to parse?

Popular Answer

You could use of the PhantomJs. I had this Issue, but don't found any solution for my problem. I use of the PhantomJs in This Article and get loaded page after 10 seconds. In my opinion, best solution for your issue is that Article.




Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why