登录后Htmlagilitypack

c# facebook html-agility-pack httprequest

我想解析一些像facebook这样的html网站,
例如(www.facebook.com/somePage)
如果我想将此链接粘贴到我的资源管理器,它会重定向我首先登录我的帐户。所以我不看那个页面。所以我不能使用Htmlagilitypack来获得响应。
那么我怎样才能首先登录网站programmaticaly(不使用webbrowser控件)然后调用该facebook页面并获得响应并使用Htmlagility包进行解析。我知道如何使用HtmlAgility包并且我知道使用Httprequest设置cookie我使用以下代码来设置cookie但是之后如何解析somePage

CookieCollection cookies = new CookieCollection();
        try
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.facebook.com");
            request.CookieContainer = new CookieContainer();
            request.CookieContainer.Add(cookies);
            //Get the response from the server and save the cookies from the first request..
            HttpWebResponse response = (HttpWebResponse)request.GetResponse();
            cookies = response.Cookies;
            response.Close();
        }
        catch (WebException)
        {
            MessageBox.Show("error");
        }

        string getUrl = "https://www.facebook.com/login.php?login_attempt=1";
        string postData = String.Format("email={0}&pass={1}", "xxxx@hotmail.com", "xxxxx");
        HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
        getRequest.CookieContainer = new CookieContainer();
        getRequest.CookieContainer.Add(cookies); //recover cookies First request
        getRequest.Method = WebRequestMethods.Http.Post;
        getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
        getRequest.AllowWriteStreamBuffering = true;
        getRequest.ProtocolVersion = HttpVersion.Version11;
        getRequest.AllowAutoRedirect = true;
        getRequest.ContentType = "application/x-www-form-urlencoded";

        byte[] byteArray = Encoding.ASCII.GetBytes(postData);
        getRequest.ContentLength = byteArray.Length;
        Stream newStream = getRequest.GetRequestStream(); //open connection
        newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
        newStream.Close();
        //How I parse (www.facebook.com/somePage) here?
       HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
       HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
       using (StreamReader sr = new StreamReader(getResponse.GetResponseStream(), Encoding.GetEncoding("windows-1251")))
        {

            doc.LoadHtml(sr.ReadToEnd());

        }

        foreach (HtmlNode link in doc.DocumentNode.SelectNodes("//a[@href]"))
        {
            listBox1.Items.Add(link.InnerHtml);
        }

一般承认的答案

在您的HttpWebRequest调用GetResponse方法。这将生成一个WebResponse对象,您可以在其上调用GetResponseStream()来获取内容。

由于Facebook几乎没有返回任何HTML内容(它们为浏览器发送大量Javascript以生成文档),HtmlAgilityPack将无法真正帮助您。它将下载Javascript,但无法执行它,因此您会遇到难以解释的文档。

其他Html软件包,如AwesoniumPhantomJS ,实际上可以执行Javascript并返回解释的HtmlDomDocument。这些不需要你运行整个浏览器,它们都可以运行Headless(因为它被称为运行浏览器而没有UI)。

或者,使用Facebook Graph API访问Facebook上的数据而无需解析HTML,它更稳定,并且是为了与Facebook上的数据交互的精确目的而构建的。



Related

许可下: CC-BY-SA with attribution
不隶属于 Stack Overflow
许可下: CC-BY-SA with attribution
不隶属于 Stack Overflow