capturing redirct in C# htmlagility pack

c# html-agility-pack

Question

HI all, this one is really simple (I hope). I'm using htmlagility pack to do my webcrawling. So what happens if I input url whatever, that then directs me to a new url, how do I capture that new redirected URL?

If htmlagility pack doesnt have a way, can someone suggest another method?

1
2
6/4/2011 8:51:01 PM

Accepted Answer

When you create your HttpWebRequest you can set AllowAutoRedirect property to true and it will automatically follow any redirects you have.

HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create("http://www.contoso.com");  
myHttpWebRequest.MaximumAutomaticRedirections=1;
myHttpWebRequest.AllowAutoRedirect=true;
HttpWebResponse myHttpWebResponse=(HttpWebResponse)myHttpWebRequest.GetResponse(); 

you can find more info at msdn

5
6/4/2011 8:58:21 PM

Popular Answer

Using the HtmlWeb class that comes with the Html Agility Pack, you can tweak the request before it's actually executed, like this:

    HtmlWeb web = new HtmlWeb();
    web.PreRequest = OnPreRequest;
    HtmlDocument doc = web.Load("http://wwwblablahh.com");


private static bool OnPreRequest(HttpWebRequest request)
{
    request.AllowAutoRedirect = true;
    return true;
}


Related Questions





Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow