Find all elements with text string?

Question

Find all elements with text string?

c# html html-agility-pack xpath

Question

I am trying to remove all html-elements (tags) that contain a specific text-string. I have 2376 html-documents, all with different doctype standards. Some even don't have a doctype (might be irrelevant to this question).

So, I am looking for a text string that says "How to cite this paper", and I've found that it is enclosed within either a <p>-tag, <h4>-tag or a <legend>-tag.

The <p>-tag often looks like this,

<p style="text-align : center; color : Red; font-weight : bold;">How to cite this paper:</i></p>

The <h4>-tag often looks like this,

<h4>How to cite this paper:</h4>Antunes, P., Costa, C.J. &amp; Pino, J.A. (2006).

The <legend>-tag looks like this,

<legend style="color: white; background-color: maroon; font-size: medium; padding: .1ex .5ex; border-right: 1px solid navy; border-bottom: 1px solid navy; font-weight: bold;">How to cite this paper</legend>

The task at hand is to find these tags and remove them from the file, and then save the file again. I do have more tags to remove, but need some help understanding HAP and XPath, and how to locate specific tags based on their values or other unique data.

So far I have come up with this code in C#, it's a console application. This is my Main (sorry for bad indentation),

//Variables
string Ext = "*.html";
string folder = @"D:\websites\dev.openjournal.tld\public\arkivet\";
IEnumerable<string> files = GetHTMLFiles(folder, Ext);
List<string> cite_files = new List<string>();            
var doc = new HtmlDocument();

//Loop to match all html-elements to query
foreach (var file in files)
{
 try
   {
      doc.Load(file);
      cite_files.Add(doc.DocumentNode.SelectNodes("//h4[contains(., 'How to cite this paper')]").ToString()); 

     cite_files.Add(doc.DocumentNode.SelectNodes("//p[contains(., 'How to cite this paper')]").ToString());
   }                
                    catch (Exception Ex)
                    {
                        Console.WriteLine(Ex.Message);
                    }
                }

                //Counts numbers of hits and prints data to user
                int filecount = files.Count();
                int citations = cite_files.Count();            
                Console.WriteLine("Number of files scanned: " + filecount);
                Console.WriteLine("Number of citations: {0}", citations);

                // Program end
                Console.WriteLine("Press any key to close program....");
                Console.ReadKey();

And this is the private method that looks through directories for files,

//List all HTML-files recursively and return them to a list 
 public static IEnumerable<string> GetHTMLFiles(string directory, string Ext)
    {
        List<string> files = new List<string>();

        try
        {
            files.AddRange(Directory.GetFiles(directory, Ext, SearchOption.AllDirectories));
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
        }            
        return files;
    }

The unique thing seems to be "How to cite this paper" so I am trying to find all specific tags that contain these exact words, and then remove them. My Notepad shows that there should be 1094 files with this phrase, so I am trying to get them all. :)

Any help greatly appreciated! :)

Christer Johansson

View more on Stack Overflow

kuujinbo · Accepted Answer · 2/7/2018 8:21:11 PM

Html Agility Pack supports LINQ selectors, which is very convenient in this case. Given some HTML based on your example above:

var html =
@"<html><head></head><body>

<!-- selector match: delete these nodes -->
<p style='text-align: center; color: Red; font-weight: bold;'>How to cite this paper:</i></p>
<h4> How to cite this paper:</h4> Antunes, P., Costa, C.J. & amp; Pino, J.A. (2006).
<legend style='color: white; background-color: maroon; font-size: medium; padding: .1ex .5ex; border-right: 1px solid navy; border-bottom: 1px solid navy; font-weight: bold;'>How to cite this paper </legend>
<div><p><i><b>How to cite this paper (NESTED)</b></i></p></div>

<!-- no match: keep these nodes -->
<p>DO NOT DELETE - How to cite</p>
<h4>DO NOT DELETE - cite this paper:</h4>
<legend>DO NOT DELETE</legend>

</body></html>";

You can create a collection of tags that should be searched, select matching nodes, and then and remove them like this:

var tagsToDelete = new string[] { "p", "h4", "legend" };
var nodesToDelete = new List<HtmlNode>();

var document = new HtmlDocument();
document.LoadHtml(html);
foreach (var tag in tagsToDelete)
{
    nodesToDelete.AddRange(
        from searchText in document.DocumentNode.Descendants(tag)
            where searchText.InnerText.Contains("How to cite this paper")
            select searchText
    );
}

foreach (var node in nodesToDelete) node.Remove();

document.Save(OUTPUT);

With the the following result:

<html><head></head><body>

<!-- XPath match: delete these nodes -->

 Antunes, P., Costa, C.J. & amp; Pino, J.A. (2006).

<div></div>

<!-- no match, keep these nodes -->
<p>DO NOT DELETE - How to cite</p>
<h4>DO NOT DELETE - cite this paper:</h4>
<legend>DO NOT DELETE</legend>

</body></html>

Find all elements with text string?

Question

Fastest Entity Framework Extensions

Accepted Answer

Performance

Expression Evaluator

Related