Reading Specific text from a website

c# html html-agility-pack xpath


I am trying to make a database, but i need to get info from a website. Mainly the Title, Date, Length and Genre from the IMDB website. I have tried like 50 different things and it is just not working. Here is my code.

    public string GetName(string URL)
    HtmlWeb web = new HtmlWeb();
    HtmlDocument doc = web.Load(URL);

    var Attr = doc.DocumentNode.SelectNodes("//*[@id=\"overview - top\"]/h1/span[1]@itemprop")[0];

    return Name;

When I run this it just gives me a XPathException. I just want it to return the Title of a movie. I am now just using this movie for a example and testing but, I want it to work with all movies I am using the HtmlAgilityPack.

12/6/2015 12:05:51 PM

Accepted Answer

The last bit of your XPath is not valid. Also to get only single element from HtmlDocument() you can use SelectSingleNode() instead of SelectNodes() :

HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load("");

var xpath = "//*[@id='overview-top']/h1/span[@class='itemprop']";
var span = doc.DocumentNode.SelectSingleNode(xpath);
var title = span.InnerText;


output :

The 40-Year-Old Virgin

demo link : *

*) the demo shows that the correct title is printed, along with an error specific to .NET Fiddle (you can safely ignore the error).

12/6/2015 12:45:27 PM

Popular Answer

I making something familiar and this is my code which gets info from website.:

string html = getUrlData(imdbUrl + "combined");
            Id = match(@"<link rel=""canonical"" href=""\d{7})/combined"" />", html);
            if (!string.IsNullOrEmpty(Id))
                status = true;
                Title = match(@"<title>(IMDb \- )*(.*?) \(.*?</title>", html, 2);
                OriginalTitle = match(@"title-extra"">(.*?)<", html);
                Year = match(@"<title>.*?\(.*?(\d{4}).*?\).*?</title>", html);
                Rating = match(@"<b>(\d.\d)/10</b>", html);
                Genres = matchAll(@"<a.*?>(.*?)</a>", match(@"Genre.?:(.*?)(</div>|See more)", html));
                Directors = matchAll(@"<td valign=""top""><a.*?href=""/name/.*?/"">(.*?)</a>", match(@"Directed by</a></h5>(.*?)</table>", html));
                Cast = matchAll(@"<td class=""nm""><a.*?href=""/name/.*?/"".*?>(.*?)</a>", match(@"<h3>Cast</h3>(.*?)</table>", html));
                Plot = match(@"Plot:</h5>.*?<div class=""info-content"">(.*?)(<a|</div)", html);
                Runtime = match(@"Runtime:</h5><div class=""info-content"">(\d{1,4}) min[\s]*.*?</div>", html);
                Languages = matchAll(@"<a.*?>(.*?)</a>", match(@"Language.?:(.*?)(</div>|>.?and )", html));
                Countries = matchAll(@"<a.*?>(.*?)</a>", match(@"Country:(.*?)(</div>|>.?and )", html));
                Poster = match(@"<div class=""photo"">.*?<a name=""poster"".*?><img.*?src=""(.*?)"".*?</div>", html);
                if (!string.IsNullOrEmpty(Poster) && Poster.IndexOf("") > 0)
                    Poster = Regex.Replace(Poster, @"_V1.*?.jpg", "_V1._SY200.jpg");
                    PosterLarge = Regex.Replace(Poster, @"_V1.*?.jpg", "_V1._SY500.jpg");
                    PosterFull = Regex.Replace(Poster, @"_V1.*?.jpg", "_V1._SY0.jpg");
                    Poster = string.Empty;
                    PosterLarge = string.Empty;
                    PosterFull = string.Empty;
                ImdbURL = "" + Id + "/";
                if (GetExtraInfo)
                    string plotHtml = getUrlData(imdbUrl + "plotsummary");

//Match single instance
    private string match(string regex, string html, int i = 1)
        return new Regex(regex, RegexOptions.Multiline).Match(html).Groups[i].Value.Trim();

    //Match all instances and return as ArrayList
    private ArrayList matchAll(string regex, string html, int i = 1)
        ArrayList list = new ArrayList();
        foreach (Match m in new Regex(regex, RegexOptions.Multiline).Matches(html))
        return list;

Maybe you will find something useful

Related Questions


Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow