HTMLAgilityPack and timeouts on load

.net c# html-agility-pack vb.net

Question

I'm using HTMLAgilityPack in a parser that I have up on a server, but I'm having issues with one of the websites that I'm parsing: Every day around 6am they tend to shut down their servers for maintenance, which throws off the Load() method for HTMLWeb, and makes my app crash. Do any of you guys have a more secure way of loading a website into HTMLAgilityPack, or maybe some way to do error checking in C# to prevent my app from crashing? (my c# is a little rusty). Here is my code right now:

HtmlWeb webGet = new HtmlWeb();
HtmlDocument document = webGet.Load(dealsiteLink); //The Load() method here stalls the program because it takes 1 or 2 minutes before it realizes the website is down

Thank you!

Popular Answer

Just surround the call with a try-catch:

HtmlWeb webGet = new HtmlWeb();

HtmlDocument document;
try
{
    document = webGet.Load(dealsiteLink); 
}
catch (WebException ex)
{
    // Logic to retry (maybe in 10 minutes) goes here
}

The exact retry logic will depend on how your application is structured - you will probably find that the try-catch block needs to be placed higher up in your application needs to go much higher up than this.

I think WebException is the exception you should catch, but I can't be sure because I can't find the documentation. You might find that you also need to catch TimeoutException.




Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why