How do i Screen scrap html page generated by javascript

c# html-agility-pack screen-scraping

Question

How can I screen scrape a javascript-generated html page? I've tried a similar thing:

        HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
        HtmlWeb hw = new HtmlWeb();
        doc = hw.Load("http://stats.nba.com/scores.html?gameDate=04/11/2014");
        HtmlNodeCollection nodes = doc.DocumentNode.SelectNodes("//div[@id='scoreboards']");

However, because javascript is used to produce the content, this is ineffective. After javascript has produced the html, is there a method to crawl the page? maybe utilizing something different from the agility pack?

1
0
4/12/2014 5:09:56 PM

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge

Popular Answer

To get material and elements that are loaded using js or ajax, use a web browser control.

private void LoadHtmlWithBrowser(String url)
{
    webBrowser1.ScriptErrorsSuppressed = true;
    webBrowser1.Navigate(url);

    waitTillLoad(this.webBrowser1);

    HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
    var documentAsIHtmlDocument3 = (mshtml.IHTMLDocument3)webBrowser1.Document.DomDocument; 
    StringReader sr = new StringReader(documentAsIHtmlDocument3.documentElement.outerHTML); 
    doc.Load(sr);
}

private void waitTillLoad(WebBrowser webBrControl)
{
    WebBrowserReadyState loadStatus;
    int waittime = 100000;
    int counter = 0;
    while (true)
    {
        loadStatus = webBrControl.ReadyState;
        Application.DoEvents();
        if ((counter > waittime) || (loadStatus == WebBrowserReadyState.Uninitialized) || (loadStatus == WebBrowserReadyState.Loading) || (loadStatus == WebBrowserReadyState.Interactive))
        {
            break;
        }
        counter++;
    }

    counter = 0;
    while (true)
    {
        loadStatus = webBrControl.ReadyState;
        Application.DoEvents();
        if (loadStatus == WebBrowserReadyState.Complete && webBrControl.IsBusy != true)
        {
            break;
        }
        counter++;
    }
}
2
4/12/2014 7:02:21 PM

Related Questions

HTML Agility Pack, C#, parsing HTML page

I can locate the LI parts by using this instance as a guide....Parsing li> in HTML Agility Pack...But I only want the LI elements that are included in the div with the id "res."...How can I go about doing that?
c# html html-agility-pack
asked by Eric Burdo

Get an html fragment from an html document using the Html Agility Pack.

I believe you should do it in sections....You may do the following selectnodes for the document's body or html:...doc.DocumentNode.SelectSingleNode("//body") // returns body with entire contents :) ...If no null values are given, you may accept the string as is after checking for them....Hope it's useful:
c# html html-agility-pack
asked by Manish Pansiniya

Scrape data from a wiki page in C# (screen-scraping)

A Wiki page must be scraped, please. in particular, ...namely this....Users of my app will be able to input the vehicle's registration number (for instance, SBS8988Z), and it will show the relevant information (which is on the page itself)....For instance, my program should search for the line if the user types SBS8988Z into a text field and find i...
c# html-agility-pack screen screen-scraping
asked by ryanswj

get all the divs ids on a html page using Html Agility Pack

You may provide xpath syntax to get the collection of divs....like as...HtmlAgilityPack.HtmlDocument htmlDoc = new HtmlAgilityPack.HtmlDocument(); htmlDoc.OptionFixNestedTags=true; htmlDoc.Load(filePath); foreach(HtmlNode div doc.DocumentElement.SelectNodes("//div")) { ///.. code here }
c# html-agility-pack
asked by HatSoft

get the web page source with the rendered html from javascript

If I employ this...WebClient client = new WebClient(); String htmlCode = client.DownloadString("http://test.net"); ...I can use the agility pack to scan the html and extract the majority of the elements I need, but the javascript-rendered html is not included....How can I get the final displayed page source in C# is the query I have. Is there anyth...
asp.net c# html-agility-pack vb.net
asked by Hello-World

Loading web page and parse its html after asyncronous javascript completes

I'm attempting to load and analyze a skyscanner.net search results page for a personal.net project. I'm aware that Skyscanner provides APIs for this sort of work, however it doesn't look like a free personal license is available....HtmlAgilityPack is used to load the document, however since Skyscanner takes a long time to finish a search, the page ...
c# html html-agility-pack html-parsing javascript
asked by Luca Clavarino

how to get javascript code too with the actual source with Html Agility Pack

You would want a proper browser for that. Think of using ...selenium... to automate a browser (which can be headless - see ...PhantomJS...)....also see...Script Execution in HTML Agility Pack
c# html-agility-pack javascript parsing xpath
asked by alecxe

how can i get html that is generated via AJAX with HTML Agility Pack?

I'm attempting to parse a website. the page's AJAX-generated content,...WebClient.DownloadString... I can access everything HTML, but not the AJAX-generated code. Could someone please assist me?...My code is...var client = new WebClient(); client .Headers.Add(HttpRequestHeader.UserAgent, "UserAgent,Mozilla/5.0 (Windows NT 5.1; rv:14.0) Gecko/201001...
ajax asp.net c# html html-agility-pack
asked by Temo Gelashvili

html agility pack is returning javascript code except the actual Html

I want to use a c# console to get links from a website that uses the HTML Agility Pack, however there is java script code placed in the li and href tags. If you know why this happens and how to fix it, please let me know. how to get the code...<li onmouseover="activate_menu('top-menu-61', 61); void(0);" onmouseout="deactivate_menu('top-menu-61', 61...
c# html-agility-pack javascript parsing
asked by Muhammad Mateen

HTML generate rowspan in html table string

I have the following C# code that adds rowspan to a datatable in an html string, but I would want for it to take an html text as a parameter rather than a datatable....protected void Page_Load(object sender, EventArgs e) { DataTable dt = new DataTable(); dt.Columns.Add("Subject"); dt.Columns.Add("Code"); dt.Columns.Add("Test"); ...
c# html html-agility-pack html-table
asked by Loaiza Diego

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
View more on Stack Overflow

Prime Library

Performance

Expression Evaluator

More Projects...

Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow