How to read JavaScript object with XPath/HTMLAgilityPack

c# html-agility-pack javascript xpath

Question

For my crawler project, I need to get product details from JavaScript object.

How can I effectively get object details from the following JavaScript? I Use XPath and HTMLAgilityPack.

<script type="text/javascript">
    var product = {
        identifier: '2051189775',     //PRODUCT ID
        fn: 'Fit- Whiskered Dark Wash Skirt',
        category: ['sale'],
        brand: 'Brand Name',
        price: '22.90',  // this would be the discount price
        amount: '31.80',  // this would be the original price
        currency: 'USD',
        //List can me even more.
    };
</script>

I've not tried getting details from JavaScript objects before. I was getting details directly from HTML for other crawlers.

1
1
7/19/2013 9:56:02 AM

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge

Accepted Answer

Since the HTML Agility Pack doesn't evaluate any of the contents of the HTML, the javascript code should just be considered plain text. Use the SelectSingleNode method to find the piece of Javascript, then just grab the InnerHtml to get to the contents.

Either find a C# javascript parser (Iron JS for example) or write a parser using standard text manipulation techniques (String.* or Regex to extract the bits you're after.

Once you have the bits between the curly brackets you could parse them using a before mentioned parser or a library like Json.NET, since the pieces between the curly brackets seems to be valid json.

1
7/19/2013 2:52:11 PM

Related Questions

HtmlAgilityPack's javascript function is called.

If you trust the source, it looks to me like you'd be better off invoking the ...WebBrowser control.... HtmlAgilityPack does not provide a scripting engine.
c# html-agility-pack javascript winforms
asked by spender

The error "Object reference not assigned to an instance of an object" is returned by HtmlAgilityPack WebGet.Load.

The actual problem is in HtmlAgilityPack internals. The page not working has this meta content type: ...<META http-equiv="Content-Type" content="text/html; charset=8859-9">... where ...charset=8859-9... seems to be incorrent. The HAL internals tries to get an appropriate encoding for this string by using something like ...Encoding.GetEncoding("8859...
c# html-agility-pack
asked by Alex

To identify all elements with innertext containing a certain phrase or words, use Xpath and HtmlAgilityPack.

I am trying to build a simple search-engine using HtmlAgilityPack and Xpath with C# (.NET 4). I want to find every node containing a userdefined searchword, but I can't seem to get the XPath right. For Example:...<HTML> <BODY> <H1>Mr T for president</H1> <div>We believe the new president should be</div> <div>the awsome Mr T</div> <div>...
html-agility-pack xpath
asked by user1161569

Parsing HTML using Xpath with Javascript

In .NET there is a lovely library that allows me to easily parse an external html page using xpath queries (HTML Agility Project) - the problem is I have to do that client-side, so only javascript. Is there any way to do that?
html-agility-pack html-parsing javascript jquery xpath
asked by Rafał Saltarski

Does HtmlAgilityPack have the ability to use regular expressions in its XPATH selector?

I would like to be able to create a collection of nodes where the text starts with a word and then a number. For example, given the following:...<p>FINDTHIS 1</p> <p>FINDTHIS SOMETEXT</p> <p>FINDTHIS 2</p> ...I would like to be able to create a collection consisting of two paragraph nodes: FINDTHIS 1 and FINDTHIS 2....One possible approach would b...
c# html-agility-pack regex xpath
asked by Arthur Edmunds

Parsing javascript HTML using HTMLAgilityPack

You can't parse javascript with HTML Agility Pack, it only supports HTML parsing. You can get to the script you need with an XPATH like this:...doc.DocumentNode.SelectSingleNode("//script[contains(text(), 'Blablabla')]").InnerHtml ...But you'll need to parse the javascript with another method (regex, js grammar, etc.)
c# html html-agility-pack parsing
asked by Simon Mourier

HtmlAgilityPack - Remove child nodes but retain inner text for the main node without XPath

You can create extension method for HtmlNode...public static class HtmlHelper { public static string InnerText(this HtmlNode node) { var sb = new StringBuilder(); foreach (var x in node.ChildNodes) { if (x.NodeType == HtmlNodeType.Text) sb.Append(x.InnerText); if (x.NodeType =...
c# html-agility-pack
asked by maxlego

Site uses Javascript and I am having trouble accessing it using htmlagilitypack

It is not possible to scrape the data with HtmlAgilityPack since it is loaded asynchronously. What you get when querying the page is its skeleton....What you could try is watch the network calls and see if a public webservice is called. Look out for json or xml data
c# html-agility-pack web-scraping windows-phone-8
asked by samy

HtmlAgilityPack download webpage which loaded asynchronously by javascript

Html Agility Pack is not mimicking the client side calls to dynamically load content into the DOM. It is a headless scraper that is downloading the static page given by the server; if you want that content, you will have to mimic the calls made by the client browser. If you do not want to try to emulate the calls a browser would make, instead of us...
c# html-agility-pack javascript web-scraping
asked by Collin Stevens

Htmlagilitypack - reading two different nodes from the same line and add them to listview

Ok so I did some quick googling on ListView...here are the results....Imports HtmlAgilityPack Public Class Form1 Private Sub Form1_Load(sender As Object, e As EventArgs) Handles Me.Load ListView2.Columns.Add("Participant's contact - Full Name", 250, HorizontalAlignment.Left) ListView2.Columns.Add("Brand Name", 100, HorizontalAlignment.Lef...
html html-agility-pack html-table selectnodes vb.net
asked by CruleD

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
View more on Stack Overflow

Prime Library

Performance

Expression Evaluator

More Projects...

Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow