C# htmlagility pack, capturing redirct

c# html-agility-pack

Question

HI all, this one is really simple (I hope). I'm using htmlagility pack to do my webcrawling. So what happens if I input url whatever, that then directs me to a new url, how do I capture that new redirected URL?

If htmlagility pack doesnt have a way, can someone suggest another method?

Accepted Answer

When you create your HttpWebRequest you can set AllowAutoRedirect property to true and it will automatically follow any redirects you have.

HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create("http://www.contoso.com");  
myHttpWebRequest.MaximumAutomaticRedirections=1;
myHttpWebRequest.AllowAutoRedirect=true;
HttpWebResponse myHttpWebResponse=(HttpWebResponse)myHttpWebRequest.GetResponse(); 

you can find more info at msdn


Popular Answer

Using the HtmlWeb class that comes with the Html Agility Pack, you can tweak the request before it's actually executed, like this:

    HtmlWeb web = new HtmlWeb();
    web.PreRequest = OnPreRequest;
    HtmlDocument doc = web.Load("http://wwwblablahh.com");


private static bool OnPreRequest(HttpWebRequest request)
{
    request.AllowAutoRedirect = true;
    return true;
}



Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why