Question
This pertains to cookies set inside a script (maybe inside a script tag).
System.Windows.Forms.HtmlDocument executes those scripts and the cookies set (like document.cookie=etc...) can be retrieved through its Cookies property.
I assume HtmlAgilityPack.HtmlDocument doesn't do this (execution). I wonder if there is an easy way to emulate the System.Windows.Forms.HtmlDocument capabilities (the cookies part).
Anyone?
Popular Answer
When I need to use Cookies and HtmlAgilityPack together, or just create custom requests (for example, set the User-Agent property, etc), here is what I do:
- Create a class that encapsulates the request/response. Let's call this class
WebQuery - Have a private CookieCollection (in your case public) property inside that class
- Create a method inside the class that does manually the request. The signature could be:
...
public HtmlAgilityPack.HtmlDocument GetSource(string url);
What do we need to do inside this method?
Well, using HttpWebRequest and HttpWebResponse, generate the http request manually (there are several examples of how to do this on Internet), create an instance of a HtmlDocument class using the constructor that receives an stream.
What stream do we have to use? Well, the one returned by:
httpResponse.GetResponseStream();
If you use HttpWebRequest to make the query, you can easily set the CookieContainer property of it to the variable you declared before everytime you access a new page, and that way all cookies set by the sites you access will be properly stored in the CookieContainer variable you declared in your WebQuery class, taking in count you're using only one instance of the WebQuery class.
Hope you find useful this explanation. Take in count that using this, you can do whatever you want, no matter if HtmlAgilityPack supports it or not.