How can i add a webRequest to set timeout when loading a url to document in HtmlAgilityPack?
c# html-agility-pack
Question
I have this code when loading the url:
private List<string> test(string url, int levels,DoWorkEventArgs eve)
{
HtmlWeb hw = new HtmlWeb();
List<string> webSites;
try
{
this.Invoke(new MethodInvoker(delegate { Texts(richTextBox1, "Loading The Url: " + url + "..." , Color.Red); }));
doc = hw.Load(url);
this.Invoke(new MethodInvoker(delegate { Texts(richTextBox1, "Done " + Environment.NewLine, Color.Red); }));
Sometimes when its Loading the url its taking a lot fo time since the website in the url variable is not responding. I want to add a timeout so lets say after X seconds it will throw a message like " there was a timeout".
Now HtmlAgilityPack does not have any timeout property or class. So i thought to create a new function in my Form1 that will use webrequest and webresponde and set a timeout in this new function then calling this function before loading the url.
Can someone show me how to make the new function to work with my code ? Also with a timeout.
Thanks.
Accepted Answer
I don't know anything about html agility's timeout. but I'm using it like this.
Maybe it's helpful to you.
Goodluck.
String Data = GetURLData(url);
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(Data);
public static string GetURLData(string URL)
{
try
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(URL);
request.UserAgent = "Omurcek";
request.Timeout = 4000;
WebResponse response = request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
return reader.ReadToEnd();
}
catch (Exception ex )
{
LogYaz("Receive DATA Error : " + URL + ex.ToString());
return "";
}
}
Popular Answer
Source: http://blog.jongallant.com/2012/07/htmlagilitypack-set-timeout.html#.VBY-_fmSz3Q
var web = new HtmlWeb();
web.PreRequest = delegate(HttpWebRequest webRequest)
{
webRequest.Timeout = 4;
return true;
};
var doc = web.Load("http://www.msn.com/");