Calling a web service with that needs cookies c# html-agility-pack web-services


I'm trying to use the HTML Agility Pack to scrape the list of lawyers at The site uses Ajax to populate the names, so I haven't had any luck doing that server side. I noticed the site is calling a web service at so I'm trying to call the same web service from my application.

I'm using the following code, but it's returning an Internal Server Error 500, which I suspect is because I need a session cookie? (as the site doesn't work if I disable cookies)

Can anyone help?

private static string WebServiceCall()

    WebRequest webRequest = WebRequest.Create("");
    HttpWebRequest httpRequest = (HttpWebRequest)webRequest;
    httpRequest.Method = "POST";
    httpRequest.ContentType = "text/xml; charset=utf-8";
    httpRequest.ProtocolVersion = HttpVersion.Version11;
    Stream requestStream = httpRequest.GetRequestStream();
    //Create Stream and Complete Request             
    StreamWriter streamWriter = new StreamWriter(requestStream, Encoding.ASCII);
    string request = "<soap:Envelope xmlns:xsi='' xmlns:xsd='' xmlns:soap=''><soap:Body><FindAttorneys xmlns='FCWSite.FCWSite.DlaPiperWS'><scFirstName></scFirstName><scLastName>A</scLastName><scKeyword></scKeyword><scOfficesGUID></scOfficesGUID><scSpokenLanguagesGUID></scSpokenLanguagesGUID><scServicesGUID></scServicesGUID><scRegionGUID></scRegionGUID><scSchoolGUID></scSchoolGUID><scAdmissionGUID></scAdmissionGUID><scLevelGUID></scLevelGUID><strLanguageGUID>7483b893-e478-44a4-8fed-f49aa917d8cf</strLanguageGUID><strCountry>global</strCountry><returnUntranslated>true</returnUntranslated><sortBy>name</sortBy><page>0</page></FindAttorneys></soap:Body></soap:Envelope>";

    //Get the Response    
    HttpWebResponse wr = (HttpWebResponse)httpRequest.GetResponse();
    StreamReader srd = new StreamReader(wr.GetResponseStream());
    string resulXmlFromWebService = srd.ReadToEnd();
    return resulXmlFromWebService;

Accepted Answer

As I stated in my comment.. I added the Attorney.asmx url as a web reference in my project.

When doing so, I let the web reference have the namespace of the reference be 'com.dlapiper.www'.

The following is in, and represents a standard default page. The class com.dlapiper.www.Attorneys represents the context of the connection.

Note that Attorneys.FindAttorney returns a string value..

Also, you can see from my buffer and array.convertAll attempt in the code, that I was trying different encodings.. if you attempt to invoke FindAttorney without adding the cookie container and the following cookies.. you'll incorrectly get a SOAP exception stating the localizationGUID was not in a proper input.

I left this somewhat irrelevant portion in the code because I'm a fan of contravariant programming.

the truth is, that parameter can't be empty, but isn't actually used by function.. the cookie name:value is.


Public Class _Default
    Inherits System.Web.UI.Page

    Dim AttorneyList As com.dlapiper.www.Attorneys = New com.dlapiper.www.Attorneys()
    Dim attys As String

    Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
        Dim buffer As Byte()
        buffer = Array.ConvertAll(Of Char, Byte)("7483b893e47844a48fedf49aa917d8cf".ToUpper.ToCharArray(), New Converter(Of Char, Byte)(AddressOf CharToBuffer))

        Dim cookieLanguage As System.Net.Cookie = New System.Net.Cookie
        cookieLanguage.Domain = ""
        cookieLanguage.Name = "Language"
        cookieLanguage.Value = UTF8Encoding.ASCII.GetString(buffer)
        cookieLanguage.Path = "/"

        Dim cookieLocalization As System.Net.Cookie = New System.Net.Cookie
        cookieLocalization.Domain = ""
        cookieLocalization.Name = "Localization"
        cookieLocalization.Value = "TimeZone=0&UsesDaylightSavings=False&TimeZoneAbbrev=IDLW&Persists=True"
        cookieLocalization.Path = "/"

        Dim cookieCulture As System.Net.Cookie = New System.Net.Cookie
        cookieCulture.Domain = ""
        cookieCulture.Name = "DefaultCulture"
        cookieCulture.Value = "en-US"
        cookieCulture.Path = "/"

        Dim cookieHideNotice As System.Net.Cookie = New System.Net.Cookie
        cookieHideNotice.Domain = ""
        cookieHideNotice.Name = "hide-cookie-notice"
        cookieHideNotice.Value = "1"
        cookieHideNotice.Path = "/"

        Dim cookieMode As System.Net.Cookie = New System.Net.Cookie
        cookieMode.Domain = ""
        cookieMode.Name = "Mode"
        cookieMode.Value = "1"
        cookieMode.Path = "/"

        Dim cookieNavId As System.Net.Cookie = New System.Net.Cookie
        cookieNavId.Domain = ""
        cookieNavId.Name = "NavId"
        cookieNavId.Value = "1074"
        cookieNavId.Path = "/"

        Dim cookiePortletId As System.Net.Cookie = New System.Net.Cookie
        cookiePortletId.Domain = ""
        cookiePortletId.Name = "PortletId"
        cookiePortletId.Value = "12601"
        cookiePortletId.Path = "/"

        Dim cookieSERVER_PORT As System.Net.Cookie = New System.Net.Cookie
        cookieSERVER_PORT.Domain = ""
        cookieSERVER_PORT.Name = "SERVER_PORT"
        cookieSERVER_PORT.Value = "80"
        cookieSERVER_PORT.Path = "/"

        Dim cookieSiteId As System.Net.Cookie = New System.Net.Cookie
        cookieSiteId.Domain = ""
        cookieSiteId.Name = "SiteId"
        cookieSiteId.Value = "1039"
        cookieSiteId.Path = "/"

        Dim cookieZoneId As System.Net.Cookie = New System.Net.Cookie
        cookieZoneId.Domain = ""
        cookieZoneId.Name = "ZoneId"
        cookieZoneId.Value = "8"
        cookieZoneId.Path = "/"

        Dim cookieEventingStatus As System.Net.Cookie = New System.Net.Cookie
        cookieEventingStatus.Domain = ""
        cookieEventingStatus.Name = "EventingStatus"
        cookieEventingStatus.Value = "1"
        cookieEventingStatus.Path = "/"

        AttorneyList.CookieContainer = New System.Net.CookieContainer()


        attys = AttorneyList.FindAttorneys("B", "", "", "", "", "", "", "", "", "", UTF8Encoding.ASCII.GetString(buffer), "global", False, "name", 0)
    End Sub

    Function CharToBuffer(ByVal character As Char) As Byte
        Return Convert.ToByte(character)
    End Function
End Class

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why