Question
I am trying to remove all html-elements (tags) that contain a specific text-string. I have 2376 html-documents, all with different doctype standards. Some even don't have a doctype (might be irrelevant to this question).
So, I am looking for a text string that says "How to cite this paper", and I've found that it is enclosed within either a <p>-tag, <h4>-tag or a <legend>-tag.
The <p>-tag often looks like this,
<p style="text-align : center; color : Red; font-weight : bold;">How to cite this paper:</i></p>
The <h4>-tag often looks like this,
<h4>How to cite this paper:</h4>Antunes, P., Costa, C.J. & Pino, J.A. (2006).
The <legend>-tag looks like this,
<legend style="color: white; background-color: maroon; font-size: medium; padding: .1ex .5ex; border-right: 1px solid navy; border-bottom: 1px solid navy; font-weight: bold;">How to cite this paper</legend>
The task at hand is to find these tags and remove them from the file, and then save the file again. I do have more tags to remove, but need some help understanding HAP and XPath, and how to locate specific tags based on their values or other unique data.
So far I have come up with this code in C#, it's a console application. This is my Main (sorry for bad indentation),
//Variables
string Ext = "*.html";
string folder = @"D:\websites\dev.openjournal.tld\public\arkivet\";
IEnumerable<string> files = GetHTMLFiles(folder, Ext);
List<string> cite_files = new List<string>();
var doc = new HtmlDocument();
//Loop to match all html-elements to query
foreach (var file in files)
{
try
{
doc.Load(file);
cite_files.Add(doc.DocumentNode.SelectNodes("//h4[contains(., 'How to cite this paper')]").ToString());
cite_files.Add(doc.DocumentNode.SelectNodes("//p[contains(., 'How to cite this paper')]").ToString());
}
catch (Exception Ex)
{
Console.WriteLine(Ex.Message);
}
}
//Counts numbers of hits and prints data to user
int filecount = files.Count();
int citations = cite_files.Count();
Console.WriteLine("Number of files scanned: " + filecount);
Console.WriteLine("Number of citations: {0}", citations);
// Program end
Console.WriteLine("Press any key to close program....");
Console.ReadKey();
And this is the private method that looks through directories for files,
//List all HTML-files recursively and return them to a list
public static IEnumerable<string> GetHTMLFiles(string directory, string Ext)
{
List<string> files = new List<string>();
try
{
files.AddRange(Directory.GetFiles(directory, Ext, SearchOption.AllDirectories));
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
return files;
}
The unique thing seems to be "How to cite this paper" so I am trying to find all specific tags that contain these exact words, and then remove them. My Notepad shows that there should be 1094 files with this phrase, so I am trying to get them all. :)
Any help greatly appreciated! :)
Accepted Answer
Html Agility Pack supports LINQ selectors, which is very convenient in this case. Given some HTML based on your example above:
var html =
@"<html><head></head><body>
<!-- selector match: delete these nodes -->
<p style='text-align: center; color: Red; font-weight: bold;'>How to cite this paper:</i></p>
<h4> How to cite this paper:</h4> Antunes, P., Costa, C.J. & amp; Pino, J.A. (2006).
<legend style='color: white; background-color: maroon; font-size: medium; padding: .1ex .5ex; border-right: 1px solid navy; border-bottom: 1px solid navy; font-weight: bold;'>How to cite this paper </legend>
<div><p><i><b>How to cite this paper (NESTED)</b></i></p></div>
<!-- no match: keep these nodes -->
<p>DO NOT DELETE - How to cite</p>
<h4>DO NOT DELETE - cite this paper:</h4>
<legend>DO NOT DELETE</legend>
</body></html>";
You can create a collection of tags that should be searched, select matching nodes, and then and remove them like this:
var tagsToDelete = new string[] { "p", "h4", "legend" };
var nodesToDelete = new List<HtmlNode>();
var document = new HtmlDocument();
document.LoadHtml(html);
foreach (var tag in tagsToDelete)
{
nodesToDelete.AddRange(
from searchText in document.DocumentNode.Descendants(tag)
where searchText.InnerText.Contains("How to cite this paper")
select searchText
);
}
foreach (var node in nodesToDelete) node.Remove();
document.Save(OUTPUT);
With the the following result:
<html><head></head><body>
<!-- XPath match: delete these nodes -->
Antunes, P., Costa, C.J. & amp; Pino, J.A. (2006).
<div></div>
<!-- no match, keep these nodes -->
<p>DO NOT DELETE - How to cite</p>
<h4>DO NOT DELETE - cite this paper:</h4>
<legend>DO NOT DELETE</legend>
</body></html>