HTML Agility Pack-Get always the first element details

c# html-agility-pack

Question

I'm using HTML Agility Pack to fetch element's details from this url:Link

I'm using this code in C# (windows Form Application):

var webGet = new HtmlWeb();
    doc = webGet.Load("http://www.trendyol.com/Butik/Liste/Kadin");

    HtmlNodeCollection butiks = doc.DocumentNode.SelectNodes("//div[contains(@class,'butik')]");
    richTextBox1.Text = butiks.Count().ToString();

    if (butiks != null)
    {
        foreach (HtmlNode element in butiks)
        {
            var butikUrl = element.SelectSingleNode("//div[@class='butik-large-image']/a").GetAttributeValue("href", null);
            var butikTitle = element.SelectSingleNode("//div[@class='butik-large-image']/a").GetAttributeValue("title", null);
            var butikImg = element.SelectSingleNode("//div[@class='butik-large-image']//a/img").GetAttributeValue("src", null);
            var butikEndTime = element.SelectSingleNode("//div[@class='butik-name']/div[@class='butikTimeLine']/a/div[@class='timelineMain']/h1").GetAttributeValue("enddate", null);
            dataGridView1.Rows.Add("", butikUrl, butikTitle, butikImg, butikEndTime);
        }

    }
    else
    {
        MessageBox.Show("Null Obeject...!");
    }

This code always return me the element details. Can you help?

I also have used the following code but the following error occurs:

var butikUrl = element.SelectSingleNode(".//div[@class='butik-large-image']/a").GetAttributeValue("href", null);
                        var butikTitle = element.SelectSingleNode(".//div[@class='butik-large-image']/a").GetAttributeValue("title", null);
                        var butikImg = element.SelectSingleNode(".//div[@class='butik-large-image']//a/img").GetAttributeValue("src", null);
                        var butikEndTime = element.SelectSingleNode(".//div[@class='butik-name']/div[@class='butikTimeLine']/a/div[@class='timelineMain']/h1").GetAttributeValue("enddate", null);

This error is for var butikUrl = element.SelectSingleNode(".//div[@class='butik-large-image']/a").GetAttributeValue("href", null);

Error: Additional information: Object reference not set to an instance of an object.

1
0
12/11/2016 8:30:02 PM

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge

Accepted Answer

The XPath predicate to populate butiks variable seems too general. contains(@class,'butik') expression will also match butik-large-image, butik-name, etc. which don't have certain descendant element you're trying to access in the foreach loop body, that's possibly the cause of the exception. Try to use a more specific predicate, for example by matching div having class exactly equals 'butik large' (XPath tested in Firefox's FirePath) :

doc.DocumentNode.SelectNodes("//div[@class='butik large']");
1
9/12/2015 4:28:39 PM

Popular Answer

Change

HtmlNodeCollection butiks = doc.DocumentNode.SelectNodes("//div[contains(@class,'butik')]");

To

HtmlNodeCollection butiks = doc.DocumentNode.SelectNodes("//div[contains(@class,'butik-large-image')]");

This should return the 20 stacked advertisement elements.

You can then grab another NodeCollection of the other advertisements with

HtmlNodeCollection butiks2 = doc.DocumentNode.SelectNodes("//div[contains(@class,'butik small left')]");

I have some HtmlAgilityPack web scrapping code at home, that I can shoot your way they may help as well.

Edit: You can join the two lists with LINQ

butiks.Union(butiks2);
ertdiddy

Related Questions

Get an html fragment from an html document using the Html Agility Pack.

I think you need to do it in pieces....you can do selectnodes of document for body or html as follows...doc.DocumentNode.SelectSingleNode("//body") // returns body with entire contents :) ...then you can check for null values for criteria and if that is provided, you can take the string as it is. ...Hope it helps :)
c# html html-agility-pack
asked by Manish Pansiniya

How can I attach an element to the top of the Head element in HTML Agility Pack?

Realizing that this is an old question, there is also the possibility of prepending child elements that might not have existed then....// Load content as new Html document HtmlDocument html = new HtmlDocument(); html.LoadHtml(oldContent); // Wrapper acts as a root element string newContent = "<div>" + someHtml + "</div>"; // Create new node from ...
.net asp.net-mvc-2 c# html-agility-pack
asked by Nenotlep

With HTML Agility Pack for ASP.NET, how can I delete certain components from HTML? (vb)

Something along the lines of (sorry my code is C# but I hope it will help nonetheless)...HtmlDocument doc = new HtmlDocument(); doc.LoadHtml("some html markup here"); HtmlNodeCollection links = doc.DocumentNode.SelectNodes("//a[@name]"); foreach(HtmlNode link in links) { link.Remove(); } //then one of the many doc.Save(...) overrides to act...
asp.net html-agility-pack vb.net
asked by CyberDude

Selecting the current element in a loop using Html Agility Pack (XPATH)

Use ...var test = e.SelectNodes(".//a[@class='head']"); ...instead. Your current code ( ...//a[]...) searches all a elements starting from the root node. If you prefix it with a dot instead (....//a[]...) only the descendants of the current node will be considered. Since it is a direct child in your case you could of course also do:...var test = e....
.net c# html html-agility-pack xpath
asked by BrokenGlass

How to parse a webresponse and obtain a specific html element in C# using the Html Agility Pack

This console application code:... HtmlDocument doc = new HtmlDocument(); doc.Load(yourHtml); foreach (HtmlNode node in doc.DocumentNode.SelectNodes("//div[@class = 'labels']")) { Console.WriteLine(node.NextSibling.InnerText.Trim()); } ...will output this:...04:28:24 06:04:41
c# html html-agility-pack parsing
asked by Simon Mourier

Using Html Agility Pack to get content between two HTML elements

So, what you really want as a result is the div around the h1-Tag? If yes, then this should work....helpDocument.DocumentNode.SelectSingleNode("//h1/a[contains(@name, '"+sectionName+"')]/ancestor::div"); ...Also works with ...SelectNodes... depending on your Html. Like this:...helpDocument.DocumentNode.SelectNodes("//h1/a[starts-with(@name,'_Toc'...
.net c# html-agility-pack
asked by shriek

Html Agility Pack SelectSingleNode giving always same result in iteration?

By writing ...node.SelectSingleNode("//span[@itemprop='name']").InnerText...it's like you writing ...doc.DocumentNode.SelectSingleNode("//span[@itemprop='name']").InnerText...To do what you want to do you should write it like this: ...node.SelectSingleNode(".//span[@itemprop='name']").InnerText.... ...This .......dot / period tells make a search on...
c# html html-agility-pack selectsinglenode
asked by a1204773

Html Agility Pack cannot find element using xpath but it is working fine with WebDriver

WebDriver will always rely on the target browser when working with XPATH. Technically, it's just a fancy bridge to the browser (whether the browser is Firefox or Chrome - IE up to 11 does not support XPATH)...Unfortunately the DOM (elements and attributes structure) that reside in browser memory ...is not the same... as the DOM that you probably pr...
.net html-agility-pack webdriver xpath
asked by Simon Mourier

How to select elements using HTML agility pack include selector element?

You can keep using ...//div[@class='bla']... selector and get HTML markup of the corresponding ...div...s from ...OuterHtml... property, for example :...var html = @"<div> <div class='bla'> <p>Some text1</p> </div> <div class='bla'> <p>Some text2</p> </div> </div>"; var doc = new HtmlDocument(); doc.LoadHtml(html); var ...
c# html-agility-pack
asked by har07

html agility pack url scraping-- getting full html link

You can check the HREF value if it's relative URL or absolute. Load the link into a ...Uri... and test whether it is relative If it relative convert it to absolute will be the way to go....static void Main(string[] args) { List<string> linksToVisit = ParseLinks("https://www.facebook.com"); } public static List<string> ParseLinks(st...
c# html-agility-pack url web-crawler web-scraping
asked by Vikash Rathee

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
View more on Stack Overflow

Prime Library

Performance

Expression Evaluator

More Projects...

Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow