How to check if it is 404 error page(page does not exist) using HtmlAgilityPack

You will have to create aPostRequestHandler occasion on theHtmlWeb For instance, it will rise after each document download, and you'll have access to theHttpWebResponse object. It has a feature for theStatusCode .

 HtmlWeb web = new HtmlWeb();
 HttpStatusCode statusCode = HttpStatusCode.OK;
 web.PostRequestHandler += (request, response) =>
 {
     if (response != null)
     {
         statusCode = response.StatusCode;
     }
 }

 var doc = web.Load(completeUrl)
 if (statusCode == HttpStatusCode.OK)
 {
     // received a read document
 }

It's much easy when you look at the HtmlAgilityPack's code on GutHub.HtmlWeb possesses a qualityStatusCode which has the value "" set:

var web = new HtmlWeb();
var document = web.Load(completeurl);

if (web.StatusCode == HttpStatusCode.OK)
{
    var urls = document.DocumentNode.Descendants("img")
          .Select(e => e.GetAttributeValue("src", null))
          .Where(s => !String.IsNullOrEmpty(s)).ToList();
}

Update

The AgilityPack API has been updated. The method remains the same:

var htmlWeb = new HtmlWeb();
var lastStatusCode = HttpStatusCode.OK;

htmlWeb.PostResponse = (request, response) =>
{
    if (response != null)
    {
        lastStatusCode = response.StatusCode;
    }
};