YouTube HTML Agility Pack C#

c# html html-agility-pack html-parsing

Question

I am trying to retrieve all the video ids off the YouTube's search results page.

Each result has this code:

<a href="/watch?v=aYIC-ebAD3o" class="ux-thumb-wrap result-item-thumb">
  <span class="video-thumb ux-thumb-128 ">
    <span class="clip">
      <img onload="tn_load(5)" alt="Thumbnail" src="//i2.ytimg.com/vi/aYIC-ebAD3o/default.jpg" >
    </span>
  </span>
  <span class="video-time">4:16</span>
  <span dir="ltr" class="yt-uix-button-group addto-container short video-actions" data-video-ids="aYIC-ebAD3o" data-feature="thumbnail">
    <button type="button" class="start master-sprite  yt-uix-button yt-uix-button-short yt-uix-tooltip" onclick=";return false;" title="" data-button-action="yt.www.addtomenu.add" role="button" aria-pressed="false">
      <img class="yt-uix-button-icon yt-uix-button-icon-addto" src="//s.ytimg.com/yt/img/pixel-vfl3z5WfW.gif" alt="">
        <span class="yt-uix-button-content">
          <span class="addto-label">Add to</span>
        </span>
    </button>
    <button type="button" class="end  yt-uix-button yt-uix-button-short yt-uix-tooltip yt-uix-button-empty" onclick=";return false;" title="" data-button-menu-id="shared-addto-menu" data-button-action="yt.www.addtomenu.load" role="button" aria-pressed="false">
      <img class="yt-uix-button-arrow" src="//s.ytimg.com/yt/img/pixel-vfl3z5WfW.gif" alt="">
    </button>
  </span>
  <span class="video-in-quicklist">Added to queue    </span>
</a>
<div class="result-item-main-content"> 

And I am trying to parse out the "data-video-ids" class data. Whats the best way to do this with the HTML Agility Pack?

I have tried this:

foreach(HtmlNode node in doc.DocumentNode.
    SelectNodes("//span[@class='data-video-ids']"))
{
    string text = node.InnerText;
    lblTest2.Text += text + Environment.NewLine;
}

Any ideas?

Accepted Answer

The 'data-video-ids' you're trying to filter out is not a class but an attribute - please try out the following expression in SelectNodes:

"//span[@data-video-ids]"

To retrieve the attribute value you could try this approach (since HtmlAgilityPack doesn't support attribute selection you have to get an element first and then select the actual attribute):

foreach(HtmlNode node in doc.DocumentNode.
    SelectNodes("//span[@data-video-ids]"))
{
    var videoIds = node.Attributes["data-video-ids"];
    if (videoIds == null) continue;

    string text = videoIds.Value;
    lblTest2.Text += text + Environment.NewLine;
}

Popular Answer

I think you will be better off in the longrun if you use one of YouTube's APIs.

I would only use web requests and HtmlAgilityPack as a last resort when no API exists. The main reason for this is if YouTube ever changes their page, it breaks your code. Open APIs are generally geared to be backwards compatible so your application should work indefinitely in most cases.

Here is a code example from Youtube's API:

YouTubeQuery query = new YouTubeQuery(YouTubeQuery.DefaultVideoUri);

//order results by the number of views (most viewed first)
query.OrderBy = "viewCount";

// search for puppies and include restricted content in the search results
// query.SafeSearch could also be set to YouTubeQuery.SafeSearchValues.Moderate
query.Query = "puppy";
query.SafeSearch = YouTubeQuery.SafeSearchValues.None;

Feed<Video> videoFeed = request.Get<Video>(query);

printVideoFeed(videoFeed);

Looks simple, right?




Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why