How do I correctly grab the images I scrape with HtmlAgilityPack?

c# html-agility-pack web-scraping xpath

Question

I'm presently working on a project and picking up HAP along the way. I get the gist of it, and it seems to have a lot of potential.

I'm attempting to scrape a product from this one website and obtain the URLs to the photos, but I'm having trouble since I don't know how to extract the link from the xpath. It was much simpler for me to perform this using Regex in the past, but I'm switching to this HAP now.

I currently use this code. Although I don't believe it will be particularly helpful to see, I'll still provide it.

    private static void HAP()
    {
        var url = "https://www.dhgate.com/product/brass-hexagonal-fidget-spinner-hexa-spinner/403294406.html#gw-0-4|ff8080815e03d6df015e9394cc681f8a:ff80808159abe8a5015a3fd78c5b51bb";
        // HtmlWeb - A Utility class to get HTML document from http
        var web = new HtmlWeb();
        //Load() Method download the specified HTML document from an Internet resource.
        var doc = web.Load(url);

        var rootNode = doc.DocumentNode;

        var divs = doc.DocumentNode.SelectNodes(String.Format("//IMG[@src='{0}']", "www.dhresource.com/webp/m/100x100/f2/albu/g5/M00/14/45/rBVaI1kWttaAI1IrAATeirRp-t8793.jpg"));
        Console.WriteLine(divs);
        Console.ReadLine();
    }

This is the URL that I'm copying.

https://www.dhgate.com/product/2017-led-light-up-hand-spinners-fidget-spinner/398793721.html#s1-0-1b;searl|4175152669

And this ought to be the first image's xPath.

//IMG[@src='//www.dhresource.com/webp/m/100x100s/f2-albu-g5-M00-6E-20-rBVaI1kWtmmAF9cmAANMKysq_GY926.jpg/2017-led-light-up-hand-spinners-fidget-spinner.jpg']
1
1
9/20/2017 11:47:07 AM

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge

Popular Answer

For this, I write a helper method. To acquire all the links, I had to retrieve the node, then the attribute, and finally cycle over the attribute.

private static void HAP()
        {
            //Declare the URL
            var url = "https://www.dhgate.com/product/brass-hexagonal-fidget-spinner-hexa-spinner/403294406.html#gw-0-4|ff8080815e03d6df015e9394cc681f8a:ff80808159abe8a5015a3fd78c5b51bb";
            // HtmlWeb - A Utility class to get HTML document from http
            var web = new HtmlWeb();
            //Load() Method download the specified HTML document from an Internet resource.
            var doc = web.Load(url);

            var rootNode = doc.DocumentNode;

            var nodes = doc.DocumentNode.SelectNodes("//img");
            foreach (var src in nodes)
            {
                var links = src.Attributes["src"].Value;
                Console.WriteLine(links);
            }
            Console.ReadLine();
        }
0
9/20/2017 12:59:51 PM

Related Questions

Scrape data from a wiki page in C# (screen-scraping)

Why is the code flawed? To put it simply, everything....The way you are viewing the page is not how it is structured. That is not how you may expect to get the necessary contents....The section of the page's content that interests us looks like this:...<h2> <span id="Deployments" class="mw-headline">Deployments</span> </h2> <p> <!-- ... --> ...
c# html-agility-pack screen screen-scraping
asked by Jeff Mercado

Grab data from an HTML table using HtmlAgilityPack.

My software utilizes HTMLAgilityPack to take a web page in HTML format, puts it in a variable, then attempts to extract two tables from the HTML that are contained inside particular Div Class elements (boardcontainer). My current code shows every table found by searching the whole web page and raises an exception if a column is empty:..."Object ref...
c# html html-agility-pack screen-scraping
asked by Joe Chatterton

Using C# and the.NET Framework, do screen scraping, web scraping, web harvesting, web data extraction, and more.

Whatever you want to call it—web harvesting, web scraping, web data extraction, screen scraping, etc.—I'm working on a Microsoft.NET application in C#. I'm trying to integrate HTML Agility Pack for HTML parsing, but it's more difficult than I anticipated. I've included some requirements and examples of what I currently have, and I'd need your feedb...
.net c# html-agility-pack visual-studio web-scraping
asked by John

Better option for web scraping (HTMLAgilityPack or Python+beautifulsoup) for C# programmer

I write.NET programs. I have a web scraping project I need to work on. To compare HTMLAgilityPack with BeautifulSoup, please....Many claim that BeautifulSoup outperforms HTMLAgilityPack. But I have to learn Python for this....Therefore, I would want to know whether learning Python and BeautifulSoup is realistic or if I should stick with C# with HTM...
beautifulsoup c# html-agility-pack python
asked by user1752009

Using htmlAgilityPack scraping all inner text from <a> tag

I want to remove every word from the ...http://search.freefind.com/siteindex.html?id=59478474&ltr=10240&fwr=0&pid=i&ics=1... link. I made the following attempt:...HtmlWeb web = new HtmlWeb(); HtmlAgilityPack.HtmlDocument doc = web.Load("http://search.freefind.com/siteindex.html?id=59478474&ltr=10240&fwr=0&pid=i&ics=1"); HtmlNodeCollection nodes = d...
.net c# html-agility-pack web-scraping xpath
asked by ImonBayazid

HtmlAgilityPack.HtmlWeb - scraping from a url with a dot in

Apply the...Uri... for creating your URL....Uri uri = new Uri("http://blahblah.com/over-under-2.5"); HtmlWeb htmlWeb = new HtmlWeb(); var page = htmlWeb.Load(uri.AbsoluteUri); Console.WriteLine(page.DocumentNode.SelectSingleNode("//*[@id=\"currentpage\"]").InnerText); ...This causes the web page to print "BLAH."
.net c# html-agility-pack
asked by Daniel B

Visual Basic HTML Agility Pack How to Grab Images from Table Cells

Therefore, it is likely that the visuals are similar to:...<img src="whatever.jpg"/> ...presumably in the markup?...HAP enables you to seize picture nodes using tools like...... .SelectNodes("./img") ...Regarding the paths:...... .Attributes("src").Value() ...I'm not aware of any specific HAP capabilities that would enable you to execute any HTTP...
extract html-agility-pack image vb.net
asked by ttrmw

How to scrape <div class="content" id ="getSch"> using HtmlAgilityPack C#

From what I gather you are trying to get the Film name? If so, the code below should execute as follows:... foreach (HtmlNode heading in htmlNode.SelectNodes("//ul[@class='col_row']") { var heading = heading.SelectSingleNode(".//li[@class='col cb']/a").InnerText; //I Presume you want other fields here? }
asp.net c# html html-agility-pack
asked by BMac

HtmlAgilityPack scraping - extracting specific nodes from html document

Here's how to use Selenium to get the HTML for a single day of games. HtmlAgilityPack is the rest. I have to set up the driver to allow self signed certificates since the website utilizes them. A fantastic time.... var ffOptions = new FirefoxOptions(); ffOptions.BrowserExecutableLocation = @"C:\Program Files (x86)\Mozilla Firefox\fir...
c# html html-agility-pack web-scraping xpath
asked by Kent Kostelac

How to identify if tweet is original or retweet in scraping with HtmlAgilityPack?

101 on Twitter scraping...Get every tweet from a page (which is available in useful tables)....<table class='tweet '>... )...HtmlWeb p = new HtmlWeb(); var doc = p.Load(@"https://twitter.com/dailygametips"); var nodes = doc.DocumentNode.SelectNodes("//table[@class='tweet ']"); ...Look for the in nodes...<span class='context'>... to make it clear ...
c# filter html-agility-pack tweetr web-scraping
asked by Smartis

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
View more on Stack Overflow

Prime Library

Performance

Expression Evaluator

More Projects...

Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow