HTMLAgility Pack:屏幕刮痧無法在類名中找到帶有連字符的Div?

c# html-agility-pack screen-scraping

這是一種學習練習,但也很有趣。基本上,我試圖在C#控制台應用程序中解析“陽台”州會議室價格(目前為1039美元)的價格。網址是:

http://www.carnival.com/BookingEngine/Stateroom/Stateroom2/?embkCode=PCV&itinCode=SC0&durDays=8&shipCode=SH&subRegionCode=CS&sailDate=08082015&sailingID=68791&numGuests=2&showDbl=False&isOver55=N&isPastGuest=N&stateCode=&isMilitary=N&evsel=&be_version=1

我有上面的網址加載正常:

var document = getHtmlWeb.Load(web_address);

Balcony價格的容器是一個帶有' col '類的div,是column-container clearfix類中的第3個div。我認為我所需要的只是按類別對所有div進行罰款:

var lowest_price = document.DocumentNode.SelectNodes("//div[@class='col-bottom']");

然後選擇第3個節點以獲得陽台價格。但是lowest_price變量仍然返回null。我知道文檔本身已加載,如果我選擇' col ',我可以在' col '中看到。是否是col-bottom的連字符阻止了div的發現?

任何替代方法來達到這個目的?正如我所說,這主要是一個學習練習。但是我不得不創建一些需要屏幕抓取的自定義監控解決方案,因此它並非都很有趣。

謝謝!

編輯包含相關信息的HTML片段:

    <div class="col">
      <h2 data-cat-title="Balcony" class="uk-rate-title-OB"> Balcony </h2>   <p>&nbsp;</p>
        <div class="col-bottom">
        <h3> From</h3>
         <strong> $1,039.00* <span class="rate-compare-strike"> </span> </strong><a metacode="OB" href="#" class="select-btn">Select</a> </div>
    </div>

熱門答案

對於有效html的屬性名稱或值中的連字符沒有任何問題,你的來源問題是他們在客戶端上使用javascript來呈現html,以驗證你可以下載html頁面,你會注意到你的元素正在尋找不存在。

要解析首先需要執行javascript的頁面,可以使用Web瀏覽器控件然後將html傳遞給HAP。

這是一個關於如何使用WinForms Web瀏覽器控件的簡單示例:

private void ParseSomeHtmlThatRenderedJavascript(){
        var browser = new System.Windows.Forms.WebBrowser() { ScriptErrorsSuppressed = true };

        string link = "yourLinkHere";

        //This will be called when the web page loads, it better be a class member since this is just a simple demonstration
        WebBrowserDocumentCompletedEventHandler onDocumentCompleted = new WebBrowserDocumentCompletedEventHandler((s, evt) => {
            //Do your HtmlParsingHere
            var doc = new HtmlAgilityPack.HtmlDocument();
            doc.LoadHtml(browser.DocumentText);
            var someNode = doc.DocumentNode.SelectNodes("yourxpathHere");
        });

        //subscribe to the DocumentCompleted event using our above handler before navigating
        browser.DocumentCompleted += onDocumentCompleted;

        browser.Navigate(link);
    }

您還可以查看Awesomium和其他一些嵌入式WebBrowser控件。

此外,如果你想在控制台應用程序中運行WebBrowser,這裡有一個示例,如果你沒有得到它使用windows窗體,這個示例是藉助於這個SO答案WebBrowser Control在一個新線程

    using System;
    using System.Text;
    using System.Threading;
    using System.Threading.Tasks;
    using System.Windows.Forms;
    using HtmlAgilityPack;
    namespace ConsoleApplication276
    {

        // a container for a url and a parser Action
        public class Link
        {
            public string link{get;set;}
            public Action<string> parser { get; set; }
        }

        public class Program
        {

            // Entry Point of the console app
            public static void Main(string[] args)
            {
                try
                {
                    // download each page and dump the content
                    // you can add more links here, associate each link with a parser action, as for what data should the parser generate create a property for that in the Link container

                    var task = MessageLoopWorker.Run(DoWorkAsync, new Link() { 
                        link = "google.com", 
                        parser = (string html) => {

                            //do what ever you need with hap here
                            var doc = new HtmlAgilityPack.HtmlDocument();
                            doc.LoadHtml(html);
                            var someNodes = doc.DocumentNode.SelectSingleNode("//div");

                        } });


                    task.Wait();
                    Console.WriteLine("DoWorkAsync completed.");
                }
                catch (Exception ex)
                {
                    Console.WriteLine("DoWorkAsync failed: " + ex.Message);
                }

                Console.WriteLine("Press Enter to exit.");
                Console.ReadLine();
            }

            // navigate WebBrowser to the list of urls in a loop
            public static async Task<Link> DoWorkAsync(Link[] args)
            {
                Console.WriteLine("Start working.");

                using (var wb = new WebBrowser())
                {
                    wb.ScriptErrorsSuppressed = true;

                    TaskCompletionSource<bool> tcs = null;
                    WebBrowserDocumentCompletedEventHandler documentCompletedHandler = (s, e) =>
                        tcs.TrySetResult(true);

                    // navigate to each URL in the list
                    foreach (var arg in args)
                    {
                        tcs = new TaskCompletionSource<bool>();
                        wb.DocumentCompleted += documentCompletedHandler;
                        try
                        {
                            wb.Navigate(arg.link.ToString());
                            // await for DocumentCompleted
                            await tcs.Task;
                            // after the page loads pass the html to the parser 
                            arg.parser(wb.DocumentText);
                        }
                        finally
                        {
                            wb.DocumentCompleted -= documentCompletedHandler;
                        }
                        // the DOM is ready
                        Console.WriteLine(arg.link.ToString());
                        Console.WriteLine(wb.Document.Body.OuterHtml);
                    }
                }

                Console.WriteLine("End working.");
                return null;
            }

        }

        // a helper class to start the message loop and execute an asynchronous task
        public static class MessageLoopWorker
        {
            public static async Task<Object> Run(Func<Link[], Task<Link>> worker, params Link[] args)
            {
                var tcs = new TaskCompletionSource<object>();

                var thread = new Thread(() =>
                {
                    EventHandler idleHandler = null;

                    idleHandler = async (s, e) =>
                    {
                        // handle Application.Idle just once
                        Application.Idle -= idleHandler;

                        // return to the message loop
                        await Task.Yield();

                        // and continue asynchronously
                        // propogate the result or exception
                        try
                        {
                            var result = await worker(args);
                            tcs.SetResult(result);
                        }
                        catch (Exception ex)
                        {
                            tcs.SetException(ex);
                        }

                        // signal to exit the message loop
                        // Application.Run will exit at this point
                        Application.ExitThread();
                    };

                    // handle Application.Idle just once
                    // to make sure we're inside the message loop
                    // and SynchronizationContext has been correctly installed
                    Application.Idle += idleHandler;
                    Application.Run();
                });

                // set STA model for the new thread
                thread.SetApartmentState(ApartmentState.STA);

                // start the thread and await for the task
                thread.Start();
                try
                {
                    return await tcs.Task;
                }
                finally
                {
                    thread.Join();
                }
            }
        }
    }



許可下: CC-BY-SA with attribution
不隸屬於 Stack Overflow
這個KB合法嗎? 是的,了解原因
許可下: CC-BY-SA with attribution
不隸屬於 Stack Overflow
這個KB合法嗎? 是的,了解原因