Get web page using HtmlAgilityPack.NETCore

.net-core c# html-agility-pack

Question

I used the HtmlAgilityPack for work with html pages. Previously I did this:

HtmlWeb web = new HtmlWeb();
HtmlDocument document = web.Load(url);
var nodes = document.DocumentNode.SelectNodes("necessary node");

but now i need to use the HtmlAgilityPack.NETCore where HtmlWeb is absent. What should i use instead HtmlWeb to have the same result?

Accepted Answer

Use the HttpClient as a new way to interact with remote resources via http.

As for your solution, you probably need to use the async methods here for non-blocking your thread, instead of .Result usage. Also note that HttpClient was meant to be used from different threads starting from .Net 4.5, so you should not recreate it each time:

// instance or static variable
HttpClient client = new HttpClient();

// get answer in non-blocking way
using (var response = await client.GetAsync(url))
{
    using (var content = response.Content)
    {
        // read answer in non-blocking way
        var result = await content.ReadAsStringAsync();
        var document = new HtmlDocument();
        document.LoadHtml(result);
        var nodes = document.DocumentNode.SelectNodes("Your nodes");
        //Some work with page....
    }
}

Great article about async/await: Async/Await - Best Practices in Asynchronous Programming by @StephenCleary | March 2013


Popular Answer

I had the same problem in Visual Studio code with netcoreapp1.0. Ended up using HtmlAgilityPack version 1.5.0-beta5 instead.

Remenber:

using HtmlAgilityPack;
using System.Net.Http;
using System.IO;

I did it like this:

HttpClient hc = new HttpClient(); 
HttpResponseMessage result = await hc.GetAsync($"http://somewebsite.com"); 
Stream stream = await result.Content.ReadAsStreamAsync(); 
HtmlDocument doc = new HtmlDocument(); 
doc.Load(stream); 
HtmlNodeCollection nodes = doc.DocumentNode.SelectNodes("//div[@class='whateverclassyouarelookingfor']");


Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why