Unable to get Child Categories inside
    using HtmlAgilityPack C# ASP.NET

asp.net c# html-agility-pack web-scraping

Question

I am new in Webscraping and trying to get data from a website with HTMLAgilityPack using ASP.NET C#. HTML structure which I am trying to parse is:

<li class='subsubnav' id='new-women-clothing'>
    <span class='cat-name'>CLOTHING</span>

    <ul>
        <li><a href="/womenswear/womens-just-in" id="just-in">Just In</a></li>

        <li><a href="/womenswear/new-season-exclusives" id="exclusives">Exclusives</a></li>

        <li><a href="/womenswear/new-season-dresses" id="dresses-&-gowns">Dresses & Gowns</a></li>

        <li><a href="/womenswear/new-season-coats" id="coats">Coats</a></li>

        <li><a href="/womenswear/new-season-jackets" id="jackets">Jackets</a></li>

        <li><a href="/womenswear/new-season-shirts-and-blouses" id="shirts-&-blouses">Shirts & Blouses</a></li>

        <li><a href="/womenswear/new-season-tops" id="tops">Tops</a></li>

        <li><a href="/womenswear/new-season-knitwear" id="knitwear">Knitwear</a></li>

        <li><a href="/womenswear/new-season-sweatshirts" id="sweatshirts">Sweatshirts</a></li>

        <li><a href="/womenswear/new-season-skirts-and-shorts" id="skirts-&-shorts">Skirts & Shorts</a></li>

        <li><a href="/womenswear/new-season-trousers" id="trousers">Trousers</a></li>

        <li><a href="/womenswear/new-season-jumpsuits" id="jumpsuits">Jumpsuits</a></li>

        <li><a href="/womenswear/new-season-jeans" id="jeans">Jeans</a></li>

        <li><a href="/womenswear/new-season-swimwear" id="swimwear">Swimwear</a></li>

        <li><a href="/womenswear/new-season-lingerie" id="lingerie">Lingerie</a></li>

        <li><a href="/womenswear/new-season-nightwear" id="nightwear">Nightwear</a></li>

        <li><a href="/womenswear/sportswear" id="sportswear">Sportswear</a></li>

        <li><a href="/womenswear/ski-wear" id="ski-wear">Ski Wear</a></li>

    </ul>

</li>

I am getting the parent categories which in this case is CLOTHING perfectly but i am unable to get elements inside ul.

here is my c# code:

var html = new HtmlDocument();
html.LoadHtml(new WebClient().DownloadString("http://www.harrods.com/men/t-shirts?icid=megamenu_MW_clothing_t_shirts"));
var root = html.DocumentNode;
var nodes = root.Descendants();
var totalNodes = nodes.Count();
var dt = root.Descendants().Where(n => n.GetAttributeValue("class", "").Equals("cat-name"));

foreach(var x in dt)
{
    foreach (var element in x.Descendants("ul"))
    {
        child_data.Add(new cat_childs(element.InnerText));
    }

    data.Add(new Categories(x.InnerText,child_data));
}

test.DataSource = data;
test.DataBind();

So how can I get the link and text of anchor tags inside <ul>?

1
1
3/15/2016 7:36:24 AM

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge

Accepted Answer

If you want to base the iteration on span with class='cat-name', then the target ul relation to the span is following sibling instead of descendant. You can use SelectNodes() to get following sibling elements from current span, like so :

foreach (var x in dt)
{
    foreach (var element in x.SelectNodes("following-sibling::ul/li/a"))
    {
        child_data.Add(new cat_childs(element.InnerText));
    }

    data.Add(new Categories(x.InnerText,child_data));
}

UPDATE :

It seems that the actual problem is in child_data variable being declared outside the outer loop. It means that you're keep adding item to the same child_data instance. Try to declare it inside the outer loop, right after foreach (var x in dt){. Alternatively, you can write the entire codes as a LINQ expression, something like this :

var data = (from d in dt
            let child_data = x.SelectNodes("following-sibling::ul/li/a")
                              .Select(o => new cat_childs(o.InnerText))
                              .ToList()
            select new Categories(x.InnerText, child_data)
           ).ToList();
2
3/9/2016 12:08:56 PM

Popular Answer

Using this xpath. It will get all the <li> that contain a <span> that has a class='cat-name'. After which it picks all the <a>s that are enclosed by <li>.

//If the span has no influence on what you want you can simply use: 
//HtmlNodeCollection hNC = htmlDoc.DocumentNode.SelectNodes("//ul/li/a");

HtmlNodeCollection hNC = htmlDoc.DocumentNode.SelectNodes("//li/span[@class='cat-name']/parent::*/ul/li");
foreach (HtmlNode h in hNC)
{
    Console.Write(h.InnerText+" ");
    Console.WriteLine(h.GetAttributeValue("href", ""));
}
Kent Kostelac

Related Questions

Creating an htmltable from a table in asp.net vb (htmlagilitypack)

1) You only have one TableRow and one TableCell. You will need to create a new one for each row/cell. You can re-use the variables but you will need to "New" an object into them....2) You might need to select ..../tr... and ..../td... to get only rows and cells in the current table / row.
asp.net html-agility-pack html-table vb.net web-scraping
asked by Hans Kesting

ASP.net / HTMLAgilityPack : Getting Value of span within div

I think you are looking for this. ...var myNode = myPage.DocumentNode.SelectSingleNode("//div[@id='olpDivId']/span[2]/a"); if (myNode != null) { string value = myNode.InnerText.Trim(); } ...I tested this code with the below html snippet...<div id="olpDivId"> <span class="blah"><a href="fsdfs1">1</a></span> <span class="blah"><a href="fsdfs...
asp.net html-agility-pack
asked by Anil Vangari

ASP.NET => Compiler Error Message: CS0246: The type or namespace name 'HtmlAgilityPack'

Had the same problem, my .NET project was targeting .NET Framework 4.0 but HtmlAgilityPack.dll was build with .NET Framework 4.5....The solution was to reference the HtmlAgilityPack.dll build with .NET Fx 4.0....The download package for ...HtmlAgilityPack... offers versions for Net20, Net40, Net45 and even WinRT.
.net asp.net c# html-agility-pack
asked by Vasile Tomoiaga

C# HtmlAgilityPack parse <ul>

Something like this might suit your needs....//Assumes your document is loaded into a variable named 'document' List<string> dataAttribute = new List<string>(); //This will contain the long # in the data attribute List<string> spanText = new List<string>(); //This will contain the text between the <span> tags HtmlNodeCollection nodeCollection...
c# html html-agility-pack parsing
asked by Ichabod Clay

how parse asp.net mvc razor view (cshtml) like html parser in c#

I want to parse razor view file in c# . I have also used Html Agility Pack to parse razor view file but it failed to save correct file contents....Basically i want to change some html elements inner html by server side using c#...<div id="content1"> <p>this contents i want to change </p> <span>contes</span> </div> ...i want to change content1 inner...
asp.net-mvc-3 c# html-agility-pack html-parsing razor
asked by Shoaib Ijaz

VB.net Getting the InnerText of href using HtmlAgilityPack

You shoud use the attribute ...href... instead, also note that .NET is case-sensitive by default...For Each link As HtmlNode In htmlDoc.DocumentNode.SelectNodes("//a[@href]") Dim href = link.Attributes("href").Value If href.IndexOf("test", StringComparison.OrdinalIgnoreCase) >= 0 Then ListBox1.Items.Add(href) ' or ListBox...
href html-agility-pack innertext vb.net
asked by Tim Schmelter

Retrieve all text nodes of element including children using HtmlAgilityPack in C#

You should use a relative XPath expression, that is, relative to your ...el... context node...HtmlAgilityPack.HtmlNode el = htmlDoc.DocumentNode.SelectSingleNode("(//div[@class='TableContainer'])[" + index + "]"); if (el != null) { foreach (HtmlNode node in el.SelectNodes(".//text()")) { Debug.WriteLine("text=" + node.InnerText.Repl...
.net .net-2.0 c# html-agility-pack xpath
asked by paul trmbrth

Getting the <ul> <li> and <b> tags in htmlAgilityPack VB.NET

Consider the following annotations: ...<li> <b>Username:</b> <!-- Use XPath to obtain this element. --> " user" <!-- This is the next sibling. --> </li> ...With these annotations in mind:...You can obtain the ...<b>... element using XPath as you have done already and then access the username or password text by means of the ...NextSibling..
.net html-agility-pack vb.net
asked by User 12345678

Getting the value of the 'href' inside of a div in HTMLAgilityPack in C#

You can filter down the objects in question with more accurate xpath as long as you don't need any of the other nodes inside the s_newsbox div.... var list = page.DocumentNode.SelectNodes("//div[@class='s_newsbox']/a[string-length(@href)>0]"); foreach (var obj in list) { var url = obj.SelectSingleNode(".").Attributes[...
c# href html-agility-pack xpath
asked by Brent D

HtmlAgilityPack getting all value inside a <div>

Using the provided HTML snippet, you can use a XPath selector to get the text nodes directly:...var html = @" <body onload='initialize()' id='dashboard'> <div id='header'>...</div> <div id='dashboard_container'> <div id='comm' class='comm_expanded'> <div id='pl_header_tab'>...</div> <div id='pltext_tab'>...</div> <div id='plex_container'> <div id='...
c# html html-agility-pack xpath
asked by kuujinbo

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
View more on Stack Overflow

Prime Library

Performance

Expression Evaluator

More Projects...

Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow