Htmlagilitypack after login to Https Website with HttpWebRequest?

c# html-agility-pack https httpwebrequest


I want to parse certain HTML websites, such as pluralsight, for example ( How can I first connect to the website programmatically (without using webbrowser control) before calling another url, such as (Pluralsight), obtain a response, and then parse it using the HTML Agility Pack?

I have a login code, but I am unsure of what to do next.

public class Login
    private CookieContainer Cookies = new CookieContainer();

public void SiteLogin(string username, string password)
    Uri site = new Uri("");

    HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(site);
    wr.Method = "Post";
    wr.ContentType = "application/x-www-form-urlencoded";
    wr.Referer = "";
    wr.CookieContainer = Cookies;
    var parameters = new Dictionary<string, string>{
    {"realm", "vzw"},
    {"gx_charset", "UTF-8"},
    {"IDToken1", username},
    {"IDToken2", password}

    string input = string.Empty;
    using (var requestStream = wr.GetRequestStream())
    using (var writer = new StreamWriter(requestStream, Encoding.UTF8))

    using (var response = (HttpWebResponse)wr.GetResponse())

        if (response.StatusCode == HttpStatusCode.OK)
            //but I do not know the next step.

private string ParamsToFormEncoded(Dictionary<string, string> parameters)
    return string.Join("&", parameters.Select(kvp => Uri.EscapeDataString(kvp.Key).Replace("%20", "+")
    + "=" + Uri.EscapeDataString(kvp.Value).Replace("20%", "+")
5/23/2016 7:34:34 AM

Popular Answer

You must first load the content using the HttpWebResponse.GetResponseStream command to get the response stream.Load the HtmlDocument's method.

var doc = new HtmlAgilityPack.HtmlDocument();
//further processing...
5/24/2016 4:55:02 PM

Related Questions


Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow