C# Using HTML Agility Pack and Need To programmatically Login To Website

c# html html-agility-pack login web-scraping

Question

Using HTML Agility Pack, I can get the HTML document of the non-member version of the site and parse what I want from it.

var web = new HtmlWeb();
var financialPage = web.Load("http://www.gurufocus.com/financials/ko");

I'm a member of the site. I have a username and password and want to programmatically login in and scrape all the data that is only available to members.

I've spent a lot of time reading answers like this... How to programmatically log in to a website to screenscape? and I just can't figure it out. Can't someone please explain to me what I need to do get the member's version of the HTML document? Thank you.

Popular Answer

It took me a long time to come to grips with the fact that this doesn't work with HTML Agility Pack. Just use Selenium with PhantomJS. It solved all my problems and it's possible to use CSS selectors. Don't waste as much time as I did on this.



Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why