Convert (render) HTML to Text with correct line-breaks

c# html-agility-pack

Question

I must transform an HTML string into plain text (preferably using HTML Agility pack). with appropriate white spaces, particularly correct line breaks.

By "correct line-breaks," I refer to the following code:

<div>
    <div>
        <div>
            line1
        </div>
    </div>
</div>
<div>line2</div>

need should be transformed as

line1
line2

I.e., line break at zzz-25 zzz.

The majority of the fixes I've seen just convert everything<div> <br> <p> links to\n this is clearly terrible.

Any recommendations for displaying logic in plaintext from HTML in C#? Even if the answer isn't the whole code, common sense solutions like "replace all closing DIVs with line-breaks, but only if the next sibling is not a DIV also" will be quite helpful.

Attempts I made included acquiring the.InnerText property (clearly incorrect), regex (painful, sluggish, requires many hacks, and is also 12 times slower than HTMLAgilityPack - I measured it), this solution , and comparable expressions (returns more line-breaks then required)

1
36
5/7/2015 8:30:57 AM

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge

Accepted Answer

With the example supplied, the code below works well and even deals with odd things like<div><br></div> Although there is room for improvement, the fundamental notion is there. Look at the remarks.

public static string FormatLineBreaks(string html)
{
    //first - remove all the existing '\n' from HTML
    //they mean nothing in HTML, but break our logic
    html = html.Replace("\r", "").Replace("\n", " ");

    //now create an Html Agile Doc object
    HtmlDocument doc = new HtmlDocument();
    doc.LoadHtml(html);

    //remove comments, head, style and script tags
    foreach (HtmlNode node in doc.DocumentNode.SafeSelectNodes("//comment() | //script | //style | //head"))
    {
        node.ParentNode.RemoveChild(node);
    }

    //now remove all "meaningless" inline elements like "span"
    foreach (HtmlNode node in doc.DocumentNode.SafeSelectNodes("//span | //label")) //add "b", "i" if required
    {
        node.ParentNode.ReplaceChild(HtmlNode.CreateNode(node.InnerHtml), node);
    }

    //block-elements - convert to line-breaks
    foreach (HtmlNode node in doc.DocumentNode.SafeSelectNodes("//p | //div")) //you could add more tags here
    {
        //we add a "\n" ONLY if the node contains some plain text as "direct" child
        //meaning - text is not nested inside children, but only one-level deep

        //use XPath to find direct "text" in element
        var txtNode = node.SelectSingleNode("text()");

        //no "direct" text - NOT ADDDING the \n !!!!
        if (txtNode == null || txtNode.InnerHtml.Trim() == "") continue;

        //"surround" the node with line breaks
        node.ParentNode.InsertBefore(doc.CreateTextNode("\r\n"), node);
        node.ParentNode.InsertAfter(doc.CreateTextNode("\r\n"), node);
    }

    //todo: might need to replace multiple "\n\n" into one here, I'm still testing...

    //now BR tags - simply replace with "\n" and forget
    foreach (HtmlNode node in doc.DocumentNode.SafeSelectNodes("//br"))
        node.ParentNode.ReplaceChild(doc.CreateTextNode("\r\n"), node);

    //finally - return the text which will have our inserted line-breaks in it
    return doc.DocumentNode.InnerText.Trim();

    //todo - you should probably add "&code;" processing, to decode all the &nbsp; and such
}    

//here's the extension method I use
private static HtmlNodeCollection SafeSelectNodes(this HtmlNode node, string selector)
{
    return (node.SelectNodes(selector) ?? new HtmlNodeCollection(node));
}
20
5/11/2015 2:16:20 PM

Popular Answer

Concerns:

  1. Unnoticeable tags (script, style)
  2. blocks of tags
  3. embedded tags
  4. Br tag
  5. enclosable places (leading, trailing and multi whitespaces)
  6. Hard areas
  7. Entities

Algebraic choice

  plain-text = Process(Plain(html))

  Plain(node-s) => Plain(node-0), Plain(node-1), ..., Plain(node-N)
  Plain(BR) => BR
  Plain(not-visible-element(child-s)) => nil
  Plain(block-element(child-s)) => BS, Plain(child-s), BE
  Plain(inline-element(child-s)) => Plain(child-s)   
  Plain(text) => ch-0, ch-1, .., ch-N

  Process(symbol-s) => Process(start-line, symbol-s)

  Process(start-line, BR, symbol-s) => Print('\n'), Process(start-line, symbol-s)
  Process(start-line, BS, symbol-s) => Process(start-line, symbol-s)
  Process(start-line, BE, symbol-s) => Process(start-line, symbol-s)
  Process(start-line, hard-space, symbol-s) => Print(' '), Process(not-ws, symbol-s)
  Process(start-line, space, symbol-s) => Process(start-line, symbol-s)
  Process(start-line, common-symbol, symbol-s) => Print(common-symbol), 
                                                  Process(not-ws, symbol-s)

  Process(not-ws, BR|BS|BE, symbol-s) => Print('\n'), Process(start-line, symbol-s)
  Process(not-ws, hard-space, symbol-s) => Print(' '), Process(not-ws, symbol-s)
  Process(not-ws, space, symbol-s) => Process(ws, symbol-s)
  Process(not-ws, common-symbol, symbol-s) => Process(ws, symbol-s)

  Process(ws, BR|BS|BE, symbol-s) => Print('\n'), Process(start-line, symbol-s)
  Process(ws, hard-space, symbol-s) => Print(' '), Print(' '), 
                                       Process(not-ws, symbol-s)
  Process(ws, space, symbol-s) => Process(ws, symbol-s)
  Process(ws, common-symbol, symbol-s) => Print(' '), Print(common-symbol),
                                          Process(not-ws, symbol-s)

HtmlAgilityPack and System.Xml.Linq C# decision:

  //HtmlAgilityPack part
  public static string ToPlainText(this HtmlAgilityPack.HtmlDocument doc)
  {
    var builder = new System.Text.StringBuilder();
    var state = ToPlainTextState.StartLine;

    Plain(builder, ref state, new[]{doc.DocumentNode});
    return builder.ToString();
  }
  static void Plain(StringBuilder builder, ref ToPlainTextState state, IEnumerable<HtmlAgilityPack.HtmlNode> nodes)
  {
    foreach (var node in nodes)
    {
      if (node is HtmlAgilityPack.HtmlTextNode)
      {
        var text = (HtmlAgilityPack.HtmlTextNode)node;
        Process(builder, ref state, HtmlAgilityPack.HtmlEntity.DeEntitize(text.Text).ToCharArray());
      }
      else
      {
        var tag = node.Name.ToLower();

        if (tag == "br")
        {
          builder.AppendLine();
          state = ToPlainTextState.StartLine;
        }
        else if (NonVisibleTags.Contains(tag))
        {
        }
        else if (InlineTags.Contains(tag))
        {
          Plain(builder, ref state, node.ChildNodes);
        }
        else
        {
          if (state != ToPlainTextState.StartLine)
          {
            builder.AppendLine();
            state = ToPlainTextState.StartLine;
          }
          Plain(builder, ref state, node.ChildNodes);
          if (state != ToPlainTextState.StartLine)
          {
            builder.AppendLine();
            state = ToPlainTextState.StartLine;
          }
        }

      }

    }
  }

  //System.Xml.Linq part
  public static string ToPlainText(this IEnumerable<XNode> nodes)
  {
    var builder = new System.Text.StringBuilder();
    var state = ToPlainTextState.StartLine;

    Plain(builder, ref state, nodes);
    return builder.ToString();
  }
  static void Plain(StringBuilder builder, ref ToPlainTextState state, IEnumerable<XNode> nodes)
  {
    foreach (var node in nodes)
    {
      if (node is XElement)
      {
        var element = (XElement)node;
        var tag = element.Name.LocalName.ToLower();

        if (tag == "br")
        {
          builder.AppendLine();
          state = ToPlainTextState.StartLine;
        }
        else if (NonVisibleTags.Contains(tag))
        {
        }
        else if (InlineTags.Contains(tag))
        {
          Plain(builder, ref state, element.Nodes());
        }
        else
        {
          if (state != ToPlainTextState.StartLine)
          {
            builder.AppendLine();
            state = ToPlainTextState.StartLine;
          }
          Plain(builder, ref state, element.Nodes());
          if (state != ToPlainTextState.StartLine)
          {
            builder.AppendLine();
            state = ToPlainTextState.StartLine;
          }
        }

      }
      else if (node is XText)
      {
        var text = (XText)node;
        Process(builder, ref state, text.Value.ToCharArray());
      }
    }
  }
  //common part
  public static void Process(System.Text.StringBuilder builder, ref ToPlainTextState state, params char[] chars)
  {
    foreach (var ch in chars)
    {
      if (char.IsWhiteSpace(ch))
      {
        if (IsHardSpace(ch))
        {
          if (state == ToPlainTextState.WhiteSpace)
            builder.Append(' ');
          builder.Append(' ');
          state = ToPlainTextState.NotWhiteSpace;
        }
        else
        {
          if (state == ToPlainTextState.NotWhiteSpace)
            state = ToPlainTextState.WhiteSpace;
        }
      }
      else
      {
        if (state == ToPlainTextState.WhiteSpace)
          builder.Append(' ');
        builder.Append(ch);
        state = ToPlainTextState.NotWhiteSpace;
      }
    }
  }
  static bool IsHardSpace(char ch)
  {
    return ch == 0xA0 || ch ==  0x2007 || ch == 0x202F;
  }

  private static readonly HashSet<string> InlineTags = new HashSet<string>
  {
      //from https://developer.mozilla.org/en-US/docs/Web/HTML/Inline_elemente
      "b", "big", "i", "small", "tt", "abbr", "acronym", 
      "cite", "code", "dfn", "em", "kbd", "strong", "samp", 
      "var", "a", "bdo", "br", "img", "map", "object", "q", 
      "script", "span", "sub", "sup", "button", "input", "label", 
      "select", "textarea"
  };

  private static readonly HashSet<string> NonVisibleTags = new HashSet<string>
  {
      "script", "style"
  };

  public enum ToPlainTextState
  {
    StartLine = 0,
    NotWhiteSpace,
    WhiteSpace,
  }

}

Examples:

// <div>  1 </div>  2 <div> 3  </div>
1
2
3
//  <div>1  <br/><br/>&#160; <b> 2 </b> <div>   </div><div> </div>  &#160;3</div>
1

  2
 3
//  <span>1<style> text </style><i>2</i></span>3
123
//<div>
//    <div>
//        <div>
//            line1
//        </div>
//    </div>
//</div>
//<div>line2</div>
line1
line2
Serj-Tm

Related Questions

Html Agility Pack extracts every text from html.

var root = doc.DocumentNode; var sb = new StringBuilder(); foreach (var node in root.DescendantNodesAndSelf()) { if (!node.HasChildNodes) { string text = node.InnerText; if (!string.IsNullOrEmpty(text)) sb.AppendLine(text.Trim()); } } ...Although it satisfies your needs, I'm not sure whether this is the ideal...
c# html-agility-pack
asked by Dyppl

Get an html fragment from an html document using the Html Agility Pack.

I believe you should do it in sections....You may do the following selectnodes for the document's body or html:...doc.DocumentNode.SelectSingleNode("//body") // returns body with entire contents :) ...If no null values are given, you may accept the string as is after checking for them....Hope it's useful:
c# html html-agility-pack
asked by Manish Pansiniya

Can HtmlAgilityPack produce html from an xml file that includes an xsl file?

The XML and XSLT output should be displayed. You previously downloaded the XML, which is what you need to perform this. To find the XSL reference, parse the XML next. The XSL must then be downloaded and applied to the XML document....Those links might be helpful....How to use.NET 3.5 to read processing instructions from an XML file...How to use Vis...
c# html-agility-pack
asked by Brian Lyttle

The HTML Agility Pack eliminates the break tag.

This occurs as a result of the unique handling of the BR by the HTML Agility Pack. It continues to support the outdated HTML 3.2 syntax, which is still used on the internet today and allows the BR to be stated without using any closing tags (browsers also still handle it gracefully by the way...)....You must alter the default behavior to change it....
asp.net html-agility-pack vb.net
asked by Simon Mourier

Html Agility Pack extracting inner content from HTML BODY node

What about...Return htmldoc.DocumentNode.SelectSingleNode("//body").InnerText
c# html html-agility-pack vb.net
asked by Jeff Mercado

get the web page source with the rendered html from javascript

You need a javascript engine in addition to the HTML Agility Pack to achieve your goals. To do that, you may want to check out something like Geckofx, which will allow you to embed a fully functional web browser into your application, and than allow you to programatically access the contents of the dom after the page has rendered....http://code.goo...
asp.net c# html-agility-pack vb.net
asked by javram

How do I replace line breaks with valid html but not when in an html element already

As text nodes don't have children, it seems that you simply need to analyze top-level HTML text nodes:...var html = "Dear Person,\r\nHello and welcome to this example.\r\nTodo: <ul><li>item 1</li>\r\n<li>item 2</li>\r\nThanks."; var doc = new HtmlDocument(); doc.LoadHtml(html); var textNodes = doc.DocumentNode.ChildNodes .OfType<HtmlTextNode>()...
c# html-agility-pack
asked by Alex

how to use html agility pack to extract all url from html text

the following...var doc = new HtmlDocument(); doc.LoadHtml(html); var images = doc.DocumentNode.Descendants("img") .Where(i => i.GetAttributeValue("src", null) != null) .Select(i => i.Attributes["src"].Value); ...This selects all of the...<img>... the document's components that have...src... returning these URLs with the property set.
c# html-agility-pack
asked by Alex

Create Html links from in-text Urls using Html Agility Pack

Although I haven't tried it, I'm very confident it's doable....Here's how to insert links in a document to replace a fixed string:...Find keyword in text when keyword match certain conditions - C#...Learn how to regex for urls here....regular expression for url...Combining them ought to make it doable....Pseudocode...select all text nodes...for eac...
c# html-agility-pack
asked by user1778606

Convert HTML table to CSV using Html Agility Pack

For each table cell, a new line is created. Replace with the following code to make each table row have its own line:...var myTableValues = myTable.Descendants("td"); foreach (var tdV in myTableValues) { PlayerFile.WriteLine(tdV.InnerText); Console.WriteLine(tdV.InnerText); } ...with...var myTableRows = myTable.Descendants("tr").Where(tr =>...
c# csv html-agility-pack linq
asked by Raidri supports Monica

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
View more on Stack Overflow

Prime Library

Performance

Expression Evaluator

More Projects...

Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow