無論如何使用“BrowserSession”下載文件? C#

c# cookies download html-agility-pack

我有一個網站,需要登錄才能下載文件。目前我正在使用BrowserSession類登錄並執行所需的所有抓取(至少在大多數情況下)。

發布底部的BrowserSession類源:

下載鏈接顯示在文檔節點上。但我不知道如何向該類添加下載功能,如果我嘗試使用webclient下載它們失敗,我已經不得不大量修改BrowserSession類,(我應該將其修改為部分但不是' t)所以我真的不想改變使用BrowserSession類。

我相信它使用htmlAgilityPack.HtmlWeb來下載和加載網頁。

如果沒有簡單的方法來修改BrowserSession,是否有人使用它的CookieCollection與Webclient?

PS:我需要登錄才能下載文件,否則鏈接會重定向到登錄界面。這就是為什麼我無法簡單地使用WebClient,並且需要修改BrowserSession類才能下載,或者在獲取頁面之前修改WebClient以使用cookie。

我承認我不太了解cookie(我不確定它們是否在每次使用GET時使用,或者它只是在POST時使用),但到目前為止BrowserSession已經處理了所有這些。

PPS:我發布的BrowserSession不是我添加的東西,但核心功能都是一樣的。

public class BrowserSession
{
private bool _isPost;
private HtmlDocument _htmlDoc;

/// <summary>
/// System.Net.CookieCollection. Provides a collection container for instances of Cookie class 
/// </summary>
public CookieCollection Cookies { get; set; }

/// <summary>
/// Provide a key-value-pair collection of form elements 
/// </summary>
public FormElementCollection FormElements { get; set; }

/// <summary>
/// Makes a HTTP GET request to the given URL
/// </summary>
public string Get(string url)
{
    _isPost = false;
    CreateWebRequestObject().Load(url);
    return _htmlDoc.DocumentNode.InnerHtml;
}

/// <summary>
/// Makes a HTTP POST request to the given URL
/// </summary>
public string Post(string url)
{
    _isPost = true;
    CreateWebRequestObject().Load(url, "POST");
    return _htmlDoc.DocumentNode.InnerHtml;
}

/// <summary>
/// Creates the HtmlWeb object and initializes all event handlers. 
/// </summary>
private HtmlWeb CreateWebRequestObject()
{
    HtmlWeb web = new HtmlWeb();
    web.UseCookies = true;
    web.PreRequest = new HtmlWeb.PreRequestHandler(OnPreRequest);
    web.PostResponse = new HtmlWeb.PostResponseHandler(OnAfterResponse);
    web.PreHandleDocument = new HtmlWeb.PreHandleDocumentHandler(OnPreHandleDocument);
    return web;
}

/// <summary>
/// Event handler for HtmlWeb.PreRequestHandler. Occurs before an HTTP request is executed.
/// </summary>
protected bool OnPreRequest(HttpWebRequest request)
{
    AddCookiesTo(request);               // Add cookies that were saved from previous requests
    if (_isPost) AddPostDataTo(request); // We only need to add post data on a POST request
    return true;
}

/// <summary>
/// Event handler for HtmlWeb.PostResponseHandler. Occurs after a HTTP response is received
/// </summary>
protected void OnAfterResponse(HttpWebRequest request, HttpWebResponse response)
{
    SaveCookiesFrom(response); // Save cookies for subsequent requests
}

/// <summary>
/// Event handler for HtmlWeb.PreHandleDocumentHandler. Occurs before a HTML document is handled
/// </summary>
protected void OnPreHandleDocument(HtmlDocument document)
{
    SaveHtmlDocument(document);
}

/// <summary>
/// Assembles the Post data and attaches to the request object
/// </summary>
private void AddPostDataTo(HttpWebRequest request)
{
    string payload = FormElements.AssemblePostPayload();
    byte[] buff = Encoding.UTF8.GetBytes(payload.ToCharArray());
    request.ContentLength = buff.Length;
    request.ContentType = "application/x-www-form-urlencoded";
    System.IO.Stream reqStream = request.GetRequestStream();
    reqStream.Write(buff, 0, buff.Length);
}

/// <summary>
/// Add cookies to the request object
/// </summary>
private void AddCookiesTo(HttpWebRequest request)
{
    if (Cookies != null && Cookies.Count > 0)
    {
        request.CookieContainer.Add(Cookies);
    }
}

/// <summary>
/// Saves cookies from the response object to the local CookieCollection object
/// </summary>
private void SaveCookiesFrom(HttpWebResponse response)
{
    if (response.Cookies.Count > 0)
    {
        if (Cookies == null)  Cookies = new CookieCollection(); 
        Cookies.Add(response.Cookies);
    }
}

/// <summary>
/// Saves the form elements collection by parsing the HTML document
/// </summary>
private void SaveHtmlDocument(HtmlDocument document)
{
    _htmlDoc = document;
    FormElements = new FormElementCollection(_htmlDoc);
}
}

FormElementCollection類:

/// <summary>
/// Represents a combined list and collection of Form Elements.
/// </summary>
public class FormElementCollection : Dictionary<string, string>
{
/// <summary>
/// Constructor. Parses the HtmlDocument to get all form input elements. 
/// </summary>
public FormElementCollection(HtmlDocument htmlDoc)
{
    var inputs = htmlDoc.DocumentNode.Descendants("input");
    foreach (var element in inputs)
    {
        string name = element.GetAttributeValue("name", "undefined");
        string value = element.GetAttributeValue("value", "");
        if (!name.Equals("undefined")) Add(name, value);
    }
}

/// <summary>
/// Assembles all form elements and values to POST. Also html encodes the values.  
/// </summary>
public string AssemblePostPayload()
{
    StringBuilder sb = new StringBuilder();
    foreach (var element in this)
    {
        string value = System.Web.HttpUtility.UrlEncode(element.Value);
        sb.Append("&" + element.Key + "=" + value);
    }
    return sb.ToString().Substring(1);
}
}

一般承認的答案

我設法使用BrowserSession和修改後的webClient使其正常工作:

首先關閉將_htmlDoc更改為Public以訪問文檔節點:

public class BrowserSession
{
    private bool _isPost;
    public string previous_Response { get; private set; }
    public HtmlDocument _htmlDoc { get; private set; }
}

其次將此方法添加到BrowserSession:

 public void DownloadCookieProtectedFile(string url, string Filename)
    {
        using (CookieAwareWebClient wc = new CookieAwareWebClient())
        {
            wc.Cookies = Cookies;
            wc.DownloadFile(url, Filename);
        }
    }
//rest of BrowserSession

第三,在某處添加此類,允許將Cookies從BrowserSession傳遞到WebClient。

public class CookieAwareWebClient : WebClient
{
    public CookieCollection Cookies = new CookieCollection();
    private void AddCookiesTo(HttpWebRequest request)
    {
        if (Cookies != null && Cookies.Count > 0)
        {
            request.CookieContainer.Add(Cookies);
        }
    }

    protected override WebRequest GetWebRequest(Uri address)
    {
        WebRequest request = base.GetWebRequest(address);
        HttpWebRequest webRequest = request as HttpWebRequest;
        if (webRequest != null)
        {
            if (webRequest.CookieContainer == null) webRequest.CookieContainer = new CookieContainer();
            AddCookiesTo(webRequest);
        }
        return request;
    }
}

這應該讓你能夠像往常一樣使用BrowserSession,當你需要獲取一個只能訪問的文件時如果你已登錄,只需調用BrowserSession.DownloadCookieProtectedFile()就像它是一個WebClient一樣,只設置Cookies像這樣:

Using(wc = new CookieAwareWebClient())
{
    wc.Cookies = BrowserSession.Cookies
    //Download with WebClient As normal
    wc.DownloadFile();
}

熱門答案

登錄和下載WebPages並不容易。我最近遇到過同樣的問題。如果您找到了這個解決方案,請提供。

現在我做的是我使用Selenium和PhantomJS。使用Selenium,我可以與我選擇的webbrowser進行交互。

此外,Browser類不使用Html Agility Pack,它是通過nuget提供的第三方庫。

我想向您推薦這個問題 ,我已經創建了一個如何使用Selenium以及如何下載HtmlDocument並使用xpath過濾掉必要信息的完整示例。



Related

許可下: CC-BY-SA with attribution
不隸屬於 Stack Overflow
許可下: CC-BY-SA with attribution
不隸屬於 Stack Overflow