HtmlAgilityPack and Authentication

c# html-agility-pack networkcredentials

Question

If given a certain url, I have a technique to get the ids and xpaths. How can I scrape a url that asks for a login and password if I don't know how to put in the username and password with the request?

using HtmlAgilityPack;

_web = new HtmlWeb();

internal Dictionary<string, string> GetidsAndXPaths(string url)
{
    var webidsAndXPaths = new Dictionary<string, string>();
    var doc = _web.Load(url);
    var nodes = doc.DocumentNode.SelectNodes("//*[@id]");
    if (nodes == null) return webidsAndXPaths;
    // code to get all the xpaths and ids

Should I make a web call to get the source code of the page before passing it to the aforementioned method?

var wc = new WebClient();
wc.Credentials = new NetworkCredential("UserName", "Password");
wc.DownloadFile("http://somewebsite.com/page.aspx", @"C:\localfile.html");
1
6
4/25/2014 5:04:06 PM

Accepted Answer

HtmlWeb.Load contains many overloads, and each of them will take either an instance ofNetworkCredential alternatively you may immediately enter a username and password.

Name // Description 
Public method Load(String) //Gets an HTML document from an Internet resource.  
Public method Load(String, String) //Loads an HTML document from an Internet resource.  
Public method Load(String, String, WebProxy, NetworkCredential) //Loads an HTML document from an Internet resource.  
Public method Load(String, String, Int32, String, String) //Loads an HTML document from an Internet resource. 

You are not need to pass aWebProxy for illustration, or you may enter the system default.

As an alternative, you might connect theHtmlWeb.PreRequest and set up the request's credentials.

htmlWeb.PreRequest += (request) => {
    request.Credentials = new NetworkCredential(...);
    return true;
};
3
4/5/2018 8:33:01 PM


Related Questions





Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow