extracting just page text using HTMLAgilityPack

.net c# html-agility-pack xml-parsing xpath

Question

Ok so i am really new to XPath queries used in HTMLAgilityPack.

So lets consider this page http://health.yahoo.net/articles/healthcare/what-your-favorite-flavor-says-about-you. What i want is to extract just the page content and nothing else.

So for that i first remove script and style tags.

Document = new HtmlDocument();
        Document.LoadHtml(page);
        TempString = new StringBuilder();
        foreach (HtmlNode style in Document.DocumentNode.Descendants("style").ToArray())
        {
            style.Remove();
        }
        foreach (HtmlNode script in Document.DocumentNode.Descendants("script").ToArray())
        {
            script.Remove();
        }

After that i am trying to use //text() to get all the text nodes.

foreach (HtmlTextNode node in Document.DocumentNode.SelectNodes("//text()"))
        {
            TempString.AppendLine(node.InnerText);
        }

However not only i am not getting just text i am also getting numerous /r /n characters.

Please i require a little guidance in this regard.

Popular Answer

If you consider that script and style nodes only have text nodes for children, you can use this XPath expression to get text nodes that are not in script or style tags, so that you don't need to remove the nodes beforehand:

//*[not(self::script or self::style)]/text()

You can further exclude text nodes that are only whitespace using XPath's normalize-space():

//*[not(self::script or self::style)]/text()[not(normalize-space(.)="")]

or the shorter

//*[not(self::script or self::style)]/text()[normalize-space()]

But you will still get text nodes that may have leading or trailing whitespace. This can be handled in your application as @aL3891 suggests.




Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why