How can I scrape data from a Webpage after searching desired data using HTML Agility
asp.net c# html html-agility-pack
Question
I want to get information from this website
into my ASPX page using the HTML Agility Pack. But I cannot do that, since the data is loaded after I search the data in the webpage.
I need some data to be done continuously after an interval of 5 mins.
Popular Answer
Results for searches in the website you've mentioned are rendered dynamically using Javascript and the data comes as Json response via Ajax. HtmlAgilityPack is intended to parse Html, not Json.
Consider using Selenium or iMacros drivers for .Net, or WebBrowser class provided Microsoft Framework. These tools run a browser in background, so they can run Javascript code in that page and render Html you want to scrape.
Just need to set up proper time out, so they will keep waiting until search results appear onto the page.
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow