Convert HTML table to CSV using Html Agility Pack

Question

Convert HTML table to CSV using Html Agility Pack

c# csv html-agility-pack linq

Question

I want to extract every element from a table with the id statsTable so that I can read the information into a CSV file.

What I have thus far is as follows:

// Create a request for the URL. 
WebRequest request = WebRequest.Create("http://www.pgatour.com/stats/stat.120.html");
Console.WriteLine("Requesting data from: http://www.pgatour.com/stats/stat.120.html");

// If required by the server, set the credentials.
request.Credentials = CredentialCache.DefaultCredentials;

WebResponse response = request.GetResponse();

using (Stream stream = response.GetResponseStream())
{
    StreamReader reader = new StreamReader(stream);

    // covert html to string
    String responseString = reader.ReadToEnd();

    HtmlDocument doc = new HtmlDocument();

    doc.LoadHtml(responseString);

    var desktopFolder = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory);
    var fullFileName = Path.Combine(desktopFolder, "GolfStats.csv");

    using (var PlayerFile = new StreamWriter(fullFileName))
    {
        PlayerFile.WriteLine("Data downloaded: " + DateTime.Now);

        var myTable = doc.DocumentNode
                        .Descendants("table")
                        .Where(table => table.Attributes.Contains("id"))
                        .SingleOrDefault(table => table.Attributes["id"].Value == "statsTable");

        var myTableValues = myTable.Descendants("td");

        foreach (var tdV in myTableValues)
        {
            PlayerFile.WriteLine(tdV.InnerText);
            Console.WriteLine(tdV.InnerText);
        }

        PlayerFile.Flush();
    }
}

The issue is that my csv only picks up an ad that is put in the database and lists the data in a single column (see url in the webRequest). It would be great if you could assist me with exporting the data in tabular format.

Matt D. Webb

Related Questions

View more on Stack Overflow

Raidri supports Monica · Accepted Answer · 3/8/2014 11:02:38 PM

For each table cell, a new line is created. Replace with the following code to make each table row have its own line:

var myTableValues = myTable.Descendants("td");
foreach (var tdV in myTableValues)
{
    PlayerFile.WriteLine(tdV.InnerText);
    Console.WriteLine(tdV.InnerText);
}

with

var myTableRows = myTable.Descendants("tr").Where(tr => tr.Attributes.Contains("id"));
foreach (var tr in myTableRows)
{
    string line = string.Join(";", tr.Descendants("td").Select(td => td.InnerText));
    PlayerFile.WriteLine(line);
    Console.WriteLine(line);
}

The .Where(tr => tr.Attributes.Contains("id")) the table row containing the advertisement is excluded since it lacks an id but all player rows have.

Convert HTML table to CSV using Html Agility Pack

Question

Fastest Entity Framework Extensions

Accepted Answer

Related Questions

Fastest Entity Framework Extensions

Performance

Expression Evaluator

Related