How do i Screen scrap html page generated by javascript

c# html-agility-pack screen-scraping

Question

How do i screen scrap a html page generated by javascript? Ive tried something like this: 

        HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
        HtmlWeb hw = new HtmlWeb();
        doc = hw.Load("http://stats.nba.com/scores.html?gameDate=04/11/2014");
        HtmlNodeCollection nodes = doc.DocumentNode.SelectNodes("//div[@id='scoreboards']");

This doesn't work however because the content is generated by javascript. Is there a way to scrape the page after the javascript has generated the html? Maybe using something other then agility pack?

1
0
4/12/2014 5:09:56 PM

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge

Popular Answer

Use a webbrowser control to get content and element witch loded by js or ajax 

private void LoadHtmlWithBrowser(String url)
{
    webBrowser1.ScriptErrorsSuppressed = true;
    webBrowser1.Navigate(url);

    waitTillLoad(this.webBrowser1);

    HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
    var documentAsIHtmlDocument3 = (mshtml.IHTMLDocument3)webBrowser1.Document.DomDocument; 
    StringReader sr = new StringReader(documentAsIHtmlDocument3.documentElement.outerHTML); 
    doc.Load(sr);
}

private void waitTillLoad(WebBrowser webBrControl)
{
    WebBrowserReadyState loadStatus;
    int waittime = 100000;
    int counter = 0;
    while (true)
    {
        loadStatus = webBrControl.ReadyState;
        Application.DoEvents();
        if ((counter > waittime) || (loadStatus == WebBrowserReadyState.Uninitialized) || (loadStatus == WebBrowserReadyState.Loading) || (loadStatus == WebBrowserReadyState.Interactive))
        {
            break;
        }
        counter++;
    }

    counter = 0;
    while (true)
    {
        loadStatus = webBrControl.ReadyState;
        Application.DoEvents();
        if (loadStatus == WebBrowserReadyState.Complete && webBrControl.IsBusy != true)
        {
            break;
        }
        counter++;
    }
}
2
4/12/2014 7:02:21 PM

Related Questions

HTML Agility Pack, C#, parsing HTML page

Following this example, I can find the LI sections....Html Agility Pack - Parsing <li>...However, I only want the LI items that reside inside the div with an id of "res"....How do I do that?
c# html html-agility-pack
asked by Eric Burdo

Get an html fragment from an html document using the Html Agility Pack.

I think you need to do it in pieces....you can do selectnodes of document for body or html as follows...doc.DocumentNode.SelectSingleNode("//body") // returns body with entire contents :) ...then you can check for null values for criteria and if that is provided, you can take the string as it is. ...Hope it helps :)
c# html html-agility-pack
asked by Manish Pansiniya

Scrape data from a wiki page in C# (screen-scraping)

I want to scrape a Wiki page. Specifically, ...this one....My app will allow users to enter the registration number of the vehicle (for example, SBS8988Z) and it will display the related information (which is on the page itself)....For example, if the user enters SBS8988Z into a text field in my application, it should look for the line on that wiki...
c# html-agility-pack screen screen-scraping
asked by ryanswj

get all the divs ids on a html page using Html Agility Pack

Yo can get the collection of div by passing xpath syntax ...Like this ...HtmlAgilityPack.HtmlDocument htmlDoc = new HtmlAgilityPack.HtmlDocument(); htmlDoc.OptionFixNestedTags=true; htmlDoc.Load(filePath); foreach(HtmlNode div doc.DocumentElement.SelectNodes("//div")) { ///.. code here }
c# html-agility-pack
asked by HatSoft

get the web page source with the rendered html from javascript

If I use this ...WebClient client = new WebClient(); String htmlCode = client.DownloadString("http://test.net"); ...I am able to use the agility pack to scan the html and get most of the tags that I need but its missing the html that is rendered by the javascript....My question is, how do I get the final rendered page source using c#. Is there some...
asp.net c# html-agility-pack vb.net
asked by Hello-World

Loading web page and parse its html after asyncronous javascript completes

For a personal use .net project i'm trying to load and parse a skyscanner.net search results page. I know Skyscanner has APIs for these kind of tasks but it seems there isn't a free personal license to use....The problem is Skyscanner needs a lot of seconds to complete a search process, so using HtmlAgilityPack to load the document results in a pag...
c# html html-agility-pack html-parsing javascript
asked by Luca Clavarino

how to get javascript code too with the actual source with Html Agility Pack

For that you would need a real browser. Consider automating a browser (which can be headless - see ...PhantomJS...) with the help of ...selenium.......See also:...Running Scripts in HtmlAgilityPack
c# html-agility-pack javascript parsing xpath
asked by alecxe

how can i get html that is generated via AJAX with HTML Agility Pack?

I am trying parse Web Page. part of that page generated via AJAX, ...WebClient.DownloadString... I can get whole HTML except that code which is generated via AJAX? can you someone help me please? ...My code is: ...var client = new WebClient(); client .Headers.Add(HttpRequestHeader.UserAgent, "UserAgent,Mozilla/5.0 (Windows NT 5.1; rv:14.0) Gecko/...
ajax asp.net c# html html-agility-pack
asked by Temo Gelashvili

html agility pack is returning javascript code except the actual Html

i want to get the links using c# console from a website using html agility pack but there is java script code written in li and href tag why java script changes code on click i don't know please tell me the solution how t get actual code ...<li onmouseover="activate_menu('top-menu-61', 61); void(0);" onmouseout="deactivate_menu('top-menu-61', 61);"...
c# html-agility-pack javascript parsing
asked by Muhammad Mateen

HTML generate rowspan in html table string

I have the following code, which adds rowspan to a datatable in an html string, but I want it not to receive as a parameter a datatable, but instead to receive as a parameter an html string in C#....protected void Page_Load(object sender, EventArgs e) { DataTable dt = new DataTable(); dt.Columns.Add("Subject"); dt.Columns.Add("Code"); ...
c# html html-agility-pack html-table
asked by Loaiza Diego

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
View more on Stack Overflow

Prime Library

Performance

Expression Evaluator

More Projects...

Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow