Issue parsing html using powershell and xpath

html html-agility-pack html-parsing powershell xpath

Question

This is a follow up question to one I asked last week, posted here. I have gotten past the original issue but now I am running into a slightly different issue.

I am now able to get the attribute of an item I'm interested in if the html tags aren't nested using the GetAttributeValue method, here it is the data-pid but I am now having trouble grabbing the attribute of an item that is in nested tags, in my code snippet it is the date. I am using xpath and the HtmlAgility pack to parse the html here but in the example below the same date gets returned over and over.

Here is what the $item object looks like:

Attributes           : {class, data-pid}
ChildNodes           : {#text, a, #text, span...}
Closed               : True
ClosingAttributes    : {}
FirstChild           : HtmlAgilityPack.HtmlTextNode
HasAttributes        : True
HasChildNodes        : True
HasClosingAttributes : False
Id                   : 
InnerHtml            :  <a href="/mod/4175126893.html" class="i"><span class="price">$20</span></a> <span class="star"></span> <span class="pl"> <span class="date">Nov 
                       30</span>  <a href="/mod/4175126893.html">Unlock Any GSM Cell Phone Today!</a> </span> <span class="l2"> <span class="price">$20</span>  <span 
                       class="pnr"> <small> (Des Moines)</small> <span class="px"> <span class="p"> </span></span> </span>  <a class="gc" href="/mod/" 
                       data-cat="mod">cell phones - by dealer</a> </span> 
InnerText            :  $20   Nov 30  Unlock Any GSM Cell Phone Today!   $20    (Des Moines)      cell phones - by dealer  
LastChild            : HtmlAgilityPack.HtmlTextNode
Line                 : 305
LinePosition         : 5408
Name                 : p
NextSibling          : HtmlAgilityPack.HtmlTextNode
NodeType             : Element
OriginalName         : p
OuterHtml            : <p class="row" data-pid="4175126893"> <a href="/mod/4175126893.html" class="i"><span class="price">$20</span></a> <span class="star"></span> 
                       <span class="pl"> <span class="date">Nov 30</span>  <a href="/mod/4175126893.html">Unlock Any GSM Cell Phone Today!</a> </span> <span class="l2"> 
                       <span class="price">$20</span>  <span class="pnr"> <small> (Des Moines)</small> <span class="px"> <span class="p"> </span></span> </span>  <a 
                       class="gc" href="/mod/" data-cat="mod">cell phones - by dealer</a> </span> </p>
OwnerDocument        : HtmlAgilityPack.HtmlDocument
ParentNode           : HtmlAgilityPack.HtmlNode
PreviousSibling      : HtmlAgilityPack.HtmlTextNode
StreamPosition       : 18733
XPath                : /html[1]/body[1]/article[1]/section[1]/div[1]/div[2]/p[11]

Attributes           : {class, data-pid}
ChildNodes           : {#text, a, #text, span...}
Closed               : True
ClosingAttributes    : {}

I want to pull out data from the outerhtml value.

OuterHtml            : <p class="row" data-latitude="41.5937565437255" data-longitude="-93.6437636649079" data-pid="4184719674"> <a href="/mod/4184719674.html" class="i"></a> 
               <span class="star"></span> <span class="pl"> <span class="date">Nov 27</span>  <a href="/mod/4184719674.html">iPhone and other Cell Phone Unlocks</a> 
               </span> <span class="l2">   <span class="pnr"> <small> (Des Moines)</small> <span class="px"> <span class="p"> <a href="#" class="maptag" 
               data-pid="4184719674">map</a></span></span> </span>  <a class="gc" href="/mod/" data-cat="mod">cell phones - by dealer</a> </span> </p>

I can grab the data-pid no problem. Here is what the current code looks like:

ForEach ($item in $results) {

    # This is working
    $ID = $item.GetAttributeValue("data-pid", "")

    # This is looping over the same item
    $Date = $item.SelectSingleNode("//span[@class='date']").InnerText
}

What I want to do is to be able to grab attributes from the different tags that are contained in the outerhtml object using my xpath statements but I can't figure out how to do that. Is that the best way to go about the problem or should I just be using some regex to get the value I want?

Let me know what other details I need to post.

1
2
5/23/2017 12:07:20 PM

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge

Accepted Answer

I haven't used the HTML Agility Pack, but AFAICS built-in tools should suffice anyway:

$url = 'http://www.example.com/path/to/some.html'

$html = (Invoke-Webrequest $url).ParsedHTML

$html.getElementsByTagName('p') | ? { $_.className -eq 'row' } | % {
  $ID   = $_.getAttributeNode('data-pid').value
  $Date = $_.getElementsByTagName('span') | ? { $_.className -eq 'date' } |
          % { $_.innerText }

  # do stuff with $ID and $Date
  "{0}: {1}" -f $ID, $Date
}

Note that Invoke-Webrequest requires PowerShell v3. Use the Internet Explorer COM object if your limited to PowerShell v2:

$ie = New-Object -COM InternetExplorer.Application
$ie.Navigate($url)
while ($ie.ReadyState -ne 4) { sleep 100 }
$html = $ie.Document

If your HTML file is a local file, replace the Invoke-Webrequest line with something like this:

$htmlfile = 'C:\path\to\some.html'

$html = New-Object -COM HTMLFile
$html.write((Get-Content $htmlfile | Out-String))
7
9/3/2014 3:44:29 PM

Popular Answer

I'm way too late but here's your mistake. You've been using absolute paths.

ForEach ($item in $results) {

    # This is working
    $ID = $item.GetAttributeValue("data-pid", "")

    # This is looping over the same item
    $Date = $item.SelectSingleNode("//span[@class='date']").InnerText

    # This is looping over the different items (i.e. this is what what you want)
    $Date = $item.SelectSingleNode(".//span[@class='date']").InnerText
}
Tomalak

Related Questions

HTML Agility Pack, C#, parsing HTML page

Following this example, I can find the LI sections....Html Agility Pack - Parsing <li>...However, I only want the LI items that reside inside the div with an id of "res"....How do I do that?
c# html html-agility-pack
asked by Eric Burdo

HtmlAgilityPack-Xpath and RegExp are used to parse HTML documents.

I try parse image url from html with HtmlAgilityPack. In html doc I have img tag :...<a class="css_foto" href="" title="Fotka: MyKe015"> <span> <img src="http://213.215.107.125/fotky/1358/93/v_13589304.jpg?v=6" width="176" height="216" alt="Fotka: MyKe015" /> </span> </a> ...I need get from this img tag atribute src. I need...
html-agility-pack xpath
asked by Tom

Issue with HTMLAgilityPack parsing HTML using C#

If you look at the page source for that URL, there's not actually an element with ...id=indu_table.... It appears to be generated dynamically (i.e. in javascript); the html that you get when loading directly from the server will not reflect anything that's changed by client script. This is probably why it's not working.
c# html-agility-pack xpath
asked by Jamie Treworgy

Parsing HTML using Xpath with Javascript

jQuery also supports xPath selector as well CSS, you can get more information from the link below....http://docs.jquery.com/DOM/Traversing/Selectors
html-agility-pack html-parsing javascript jquery xpath
asked by Onur Topal

Can Html Agility Pack be used to parse HTML fragments?

Absolutely, that is what it excels at. ...In fact, many web pages you'll find in the wild could be described as HTML fragments, due to missing ...<html>... tags, or improperly closed tags....The HtmlAgilityPack simulates what the browser has to do - try to make sense from what is sometimes a jumble of mismatched tags. An imperfect science, but Html...
.net c# html html-agility-pack parsing
asked by RedFilter

Parse HTML Table in PowerShell V3

I have the following HTML table ...Link To the HTML...I want to parse it and convert it to XML/CSV/PS Object, I tried to do with HtmlAgilityPack.dll but no success. Can anybody give me any directions to do it?...I want to convert the table to a PSObject and export it to csv, I currently have just the beginning of the code, and access to the lines...
html-agility-pack powershell
asked by OhadH

C# HTML Parsing using XPath - Namespace

I'm trying to parse a HTML page using the HTML Agility Pack. I used a Firefox extension named XPath Checker and I'm rather sure that the expression is correct. However when I run my code, .NET tells me that "Namespace Manager or XsltContext needed. This query has a prefix, variable, or user-defined function."...That is fine except I have no idea wh...
asp.net-mvc-4 c# html-agility-pack xpath
asked by Astaar

C# parse html with xpath

I'm trying to parse out stock exchange information whit a simple piece of C# from a HTML document. The problem is that I can not get my head around the syntax, the ...tr class="LomakeTaustaVari"... gets parsed out but how do I get the second bit that has no tr-class? ...Here's a piece of the HTML, it repeats it self whit different values. ...<tr ...
c# html-agility-pack xpath
asked by user2886603

Parse HTML by breaklines using HTML AgilityPack

Something like this should work...[Test] public void Test() { var x = ReadTillTwoBr(GetDivClass()).ToList(); } public HtmlNode GetDivClass() { var html = @"<html><div class=""PlainText""> DATE: 2013-10-28 20:00:43 -0500 <br/> Item 1: Text1 <br/> Item 1: Text1 <br/> Item 1: Text1 <br/> ...
c# html-agility-pack parsing
asked by devshorts

Parse HTML Table using HTML Agility Pack - No ID's for Table

Here is my Html Code...<center> <table cellspacing="0" cellpadding="0" border="0"> <tbody><tr><td><img src="/someimages/images/dot_t.gif" hspace="20"></td> <td><font face="Arial Rounded MT Bold, Arial, Helvetica" size="5" color="#000088"> Marks Sheet Page</font></td> </tr> </tbody></table> <table> <tbody><tr> <...
c# html-agility-pack
asked by CoolArchTek

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
View more on Stack Overflow

Prime Library

Performance

Expression Evaluator

More Projects...

Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow