How do I correctly grab the images I scrape with HtmlAgilityPack?

c# html-agility-pack web-scraping xpath

Question

I am currently working on a project and I am learning HAP as I go. I get the basics of it and it seems like it could be very powerful.

I'm having an issue right now, I am trying to scrape a product on this one website and get the links to the images but I dont know how to extract the link from the xpath. I used to do this with Regex which was alot easier but I am moving on this HAP.

This is my current code I dont think it will be very useful to see but i'll out it in either way.

    private static void HAP()
    {
        var url = "https://www.dhgate.com/product/brass-hexagonal-fidget-spinner-hexa-spinner/403294406.html#gw-0-4|ff8080815e03d6df015e9394cc681f8a:ff80808159abe8a5015a3fd78c5b51bb";
        // HtmlWeb - A Utility class to get HTML document from http
        var web = new HtmlWeb();
        //Load() Method download the specified HTML document from an Internet resource.
        var doc = web.Load(url);

        var rootNode = doc.DocumentNode;

        var divs = doc.DocumentNode.SelectNodes(String.Format("//IMG[@src='{0}']", "www.dhresource.com/webp/m/100x100/f2/albu/g5/M00/14/45/rBVaI1kWttaAI1IrAATeirRp-t8793.jpg"));
        Console.WriteLine(divs);
        Console.ReadLine();
    }

This is the link I am scraping from

https://www.dhgate.com/product/2017-led-light-up-hand-spinners-fidget-spinner/398793721.html#s1-0-1b;searl|4175152669

And this should be the xPath of the first image.

//IMG[@src='//www.dhresource.com/webp/m/100x100s/f2-albu-g5-M00-6E-20-rBVaI1kWtmmAF9cmAANMKysq_GY926.jpg/2017-led-light-up-hand-spinners-fidget-spinner.jpg']
1
1
9/20/2017 11:47:07 AM

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge

Popular Answer

I create a helper method for this. I had to get the node, and then the attribute and then cycle through the attribute to get all the links.

private static void HAP()
        {
            //Declare the URL
            var url = "https://www.dhgate.com/product/brass-hexagonal-fidget-spinner-hexa-spinner/403294406.html#gw-0-4|ff8080815e03d6df015e9394cc681f8a:ff80808159abe8a5015a3fd78c5b51bb";
            // HtmlWeb - A Utility class to get HTML document from http
            var web = new HtmlWeb();
            //Load() Method download the specified HTML document from an Internet resource.
            var doc = web.Load(url);

            var rootNode = doc.DocumentNode;

            var nodes = doc.DocumentNode.SelectNodes("//img");
            foreach (var src in nodes)
            {
                var links = src.Attributes["src"].Value;
                Console.WriteLine(links);
            }
            Console.ReadLine();
        }
0
9/20/2017 12:59:51 PM

Related Questions

Scrape data from a wiki page in C# (screen-scraping)

What's wrong with the code? To be blunt, everything. :P...The page is not formatted in the way you are reading it. You can't hope to get the desired contents that way....The contents of the page (the part we're interested in) looks something like this:...<h2> <span id="Deployments" class="mw-headline">Deployments</span> </h2> <p> <!-- ... --...
c# html-agility-pack screen screen-scraping
asked by Jeff Mercado

Grab data from an HTML table using HtmlAgilityPack.

My program uses HtmlAgilityPack and grabs a HTML web page, stores it in a variable and I'm trying to get from the HTML two tables which are under specific Div Class tags (boardcontainer). With my current code it searches through the whole web page for every table and displays them but when a cell is empty it throws an exception:..."NullReferenceExc...
c# html html-agility-pack screen-scraping
asked by Joe Chatterton

Using C# and the.NET Framework, do screen scraping, web scraping, web harvesting, web data extraction, and more.

I am working on a Microsoft .NET Application in C# for Web Harvesting, Web Scraping, Web Data Extraction, Screen Scraping, etc. whatever you want to call it. For parsing HTML, I'm attempting to incorporate HTML Agility Pack but it's not as easy as I thought it would be. I have included some specifications and images of what I have so far and was ho...
.net c# html-agility-pack visual-studio web-scraping
asked by John

Better option for web scraping (HTMLAgilityPack or Python+beautifulsoup) for C# programmer

I'm a .NET programmer. I need to work on a web scraping project. I want to get an idea on HTMLAgilityPack vs BeautifulSoup. ...Many people say, BeautifulSoup is much better than HTMLAgilityPack. But for this, I need to learn Python....So, my question is, Is it reasonable for me to learn Python and BeautifulSoup or continue with C# and HTMLAgilityPa...
beautifulsoup c# html-agility-pack python
asked by user1752009

Using htmlAgilityPack scraping all inner text from <a> tag

I want to scrap all the word from the link ...http://search.freefind.com/siteindex.html?id=59478474&ltr=10240&fwr=0&pid=i&ics=1... I tried something like this:...HtmlWeb web = new HtmlWeb(); HtmlAgilityPack.HtmlDocument doc = web.Load("http://search.freefind.com/siteindex.html?id=59478474&ltr=10240&fwr=0&pid=i&ics=1"); HtmlNodeCollection nodes = d...
.net c# html-agility-pack web-scraping xpath
asked by ImonBayazid

HtmlAgilityPack.HtmlWeb - scraping from a url with a dot in

Use the ...Uri... class to create your URL....Uri uri = new Uri("http://blahblah.com/over-under-2.5"); HtmlWeb htmlWeb = new HtmlWeb(); var page = htmlWeb.Load(uri.AbsoluteUri); Console.WriteLine(page.DocumentNode.SelectSingleNode("//*[@id=\"currentpage\"]").InnerText); ...which will output "BLAH" from the web page.
.net c# html-agility-pack
asked by Daniel B

Visual Basic HTML Agility Pack How to Grab Images from Table Cells

So, presumably the images look something like:...<img src="whatever.jpg"/> ...in the markup, right? ...HAP will allow you to grab image nodes with something like ...... .SelectNodes("./img") ...And for the paths:...... .Attributes("src").Value() ...From there, I'm not aware of any particular HAP features that allow you to actually perform any HTT...
extract html-agility-pack image vb.net
asked by ttrmw

How to scrape <div class="content" id ="getSch"> using HtmlAgilityPack C#

From what I gather you are trying to get the Film name? if so the below code should do this:... foreach (HtmlNode heading in htmlNode.SelectNodes("//ul[@class='col_row']") { var heading = heading.SelectSingleNode(".//li[@class='col cb']/a").InnerText; //I Presume you want other fields here? }
asp.net c# html html-agility-pack
asked by BMac

HtmlAgilityPack scraping - extracting specific nodes from html document

Here is how to get the HTML for one day of matches using Selenium. Rest is HtmlAgilityPack. The site uses self signed certificates so I had to configure the driver to accept self signed certificates. Have fun.... var ffOptions = new FirefoxOptions(); ffOptions.BrowserExecutableLocation = @"C:\Program Files (x86)\Mozilla Firefox\firef...
c# html html-agility-pack web-scraping xpath
asked by Kent Kostelac

How to identify if tweet is original or retweet in scraping with HtmlAgilityPack?

Scraping Twitter 101...Get all Tweets from a page (which comes in handy tables ...<table class='tweet '>...)...HtmlWeb p = new HtmlWeb(); var doc = p.Load(@"https://twitter.com/dailygametips"); var nodes = doc.DocumentNode.SelectNodes("//table[@class='tweet ']"); ...Look in nodes for the ...<span class='context'>... to indicated that this tweet i...
c# filter html-agility-pack tweetr web-scraping
asked by Smartis

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
View more on Stack Overflow

Prime Library

Performance

Expression Evaluator

More Projects...

Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow