HTML AGILITY PACK: Using Proxies?

html-agility-pack

Question

How do I use proxies with HAP? This is what I have so far...(with no luck).

IPCHICKEN is just used to test the ip address. it is showing my ip address and not the ip address of my proxy

Function GetPrice(ByVal AmazonURL As String, ByVal Delay As Integer)
    Dim aHtml As New HtmlWeb
        Dim ChromeAgent As String = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11"
    aHtml.UserAgent = ChromeAgent


    Dim proxy As New System.Net.WebProxy
    Dim proxyAddress As New Uri("http://111.111.111/")


    Dim aDoc As HtmlDocument = aHtml.Load("http://www.ipchicken.com", "GET", proxy, System.Net.CredentialCache.DefaultCredentials)

    Dim aNode As HtmlAgilityPack.HtmlNode
    aNode = aDoc.DocumentNode.SelectSingleNode("//div[@id='olpDivId']/span[2]")

    If aNode.InnerText Is Nothing Then

    End If

    Dim UsedPrice1 As String = aNode.InnerText
    Dim i As Integer = UsedPrice1.IndexOf("$")
    Dim UsedPrice As Integer = UsedPrice1.Substring(i + 1)

    System.Threading.Thread.Sleep(Delay)

    Return UsedPrice
End Function

Popular Answer

Maybe try something like the following. Depending on the page you mightn't need the UTF8 conversion.

WebProxy proxy = new WebProxy("proxyname", 8080);
proxy.Credentials = CredentialCache.DefaultCredentials;

WebClient client = new WebClient();
client.Proxy = proxy;

string baseHtml = "";

byte[] pageContent = client.DownloadData("your target url");

UTF8Encoding utf = new UTF8Encoding();
baseHtml = utf.GetString(pageContent);

HtmlDocument pageHtml = new HtmlDocument();
pageHtml.LoadHtml(baseHtml);



Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why