Retrieving all links on a google search

c# html-agility-pack selenium

Question

I'm trying to retrieve all links from a google search but without success ...

Im using Selenium + HTML Agility Pack

HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(pageSource);

foreach (HtmlNode link in doc.DocumentNode.SelectNodes("//*[@id='rso']/div/div/div[1]/div/div/h3/a"))
{
     string href = link.GetAttributeValue("data-href", string.Empty);

     list.Add(href);
}

foreach (var item in list)
{
     Console.WriteLine(item);
}

Google HTML

<a href="/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=2&amp;cad=rja&amp;uact=8&amp;ved=0ahUKEwizrv_f06TWAhXLD5AKHa4pB1AQFgg3MAE&amp;url=http%3A%2F%2Fwww.rtl.de%2Fcms%2Fdsds-2017-das-sind-die-votingergebnisse-der-14-staffel-4112962.html&amp;usg=AFQjCNF_sFs_fpNAeBAPLitsVZbShMAhiw" onmousedown="return rwt(this,'','','','2','AFQjCNF_sFs_fpNAeBAPLitsVZbShMAhiw','','0ahUKEwizrv_f06TWAhXLD5AKHa4pB1AQFgg3MAE','','',event)" data-href="http://www.rtl.de/cms/dsds-2017-das-sind-die-votingergebnisse-der-14-staffel-4112962.html">DSDS 2017: Das sind die Votingergebnisse der 14. Staffel - RTL.de</a>

I would like to extract the date-href but without success

1
0
9/14/2017 12:26:33 PM

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge

Popular Answer

Isn't the problem that in this way you are finding all links.

When you add a[@data-href] to the XPath of SelectNodes you'll select only the links with an actual data url attribute in them.

So it will become :

foreach (HtmlNode link in doc.DocumentNode.SelectNodes("//*[@id='rso']/div/div/div[1]/div/div/h3/a[@data-href]")) 
{ 
    string href = link.GetAttributeValue("data-href", string.Empty); 
    list.Add(href); 
}
0
9/16/2017 1:24:20 PM

Related Questions

The inner text of any other child node is not searched for by XPath.

You need to use ...text()... to get text nodes. To get all text nodes in the document, use ...//text().......From the ...specification...:...text()... matches any text node.
c# html html-agility-pack xpath
asked by lonesomeday

Obtaining the site's position in Google search results

I want to code some algorithm or parser which should get site position in google search results. The issue is every time google page layout will change I should correct/change the algorithm. How do you think guys is will really often change? Is there any techniques/advices/tricks about determining Google' site position? ...How can I make robust pos...
.net c# google-search html-agility-pack parsing
asked by kseen

Get HtmlAgilityPack Node using exact HTML search or Converting HTMLElement to HTMLNode

I have created a HTMLElement picker (DOM) by using the default .net WebBrowser. The user can pick (select) a HTMLElement by clicking on it....I want to get the HtmlAgilityPack.HTMLNode corresponding to the HTMLElement....The easiest way (in my mind) is to use doc.DocumentNode.SelectSingleNode(EXACTHTMLTEXT) but it does not really work (because the...
browser c# html-agility-pack
asked by BlasterGod

Getting text between all tags in a given html and recursively going through links

Regex is not a good choice for parsing HTML files.....HTML is not strict nor is it regular with its format.....Use ...htmlagilitypack...This extracts all the links from the web page...public List<string> getAllLinks(string webAddress) { HtmlAgilityPack.HtmlWeb web = new HtmlAgilityPack.HtmlWeb(); HtmlDocument newdoc=web.Load(webAddress); ...
c# html html-agility-pack web-crawler
asked by Anirudha

I want to retrieve the width and height of html element with HtmlAgilityPack

Basead in this page: ...page...public sealed class UtilParserHTML { //Private Fields private Uri Uri; private Stream StreamPage; private HttpWebRequest HttpRequest; private HttpWebResponse HttpResponse; //Public Fields public HtmlDocument HtmlDocument { private set; get; } public UtilParserHTML() { if ...
asp.net c# html-agility-pack
asked by Gus

Scraping HTML from Google Translate

This is a bad idea. As commenters have pointed out, Google offers a programmatic interface as a paid service. Google surely has security features in place to try to block exactly what you are doing, and that is why it isn't working. Perhaps you could get it working somehow, but even then you would always be in danger of Google improving its secu...
.net c# google-translate html-agility-pack winforms
asked by user1919238

VB.net extract links from google-search using HtmlAgilityPack

I would use ...Html Agility Pack... to extract the links as below ...Dim links As New List(Of String)() Dim htmlDoc As New HtmlAgilityPack.HtmlDocument() htmlDoc.LoadHtml(WebSource) For Each link As HtmlNode In htmlDoc.DocumentNode.SelectNodes("//a[@href]") Dim att As HtmlAttribute = link.Attributes("href") If att.Value.Contains("/forums/"...
google-search html-agility-pack vb.net
asked by Damith

Retrieve all text nodes of element including children using HtmlAgilityPack in C#

You should use a relative XPath expression, that is, relative to your ...el... context node...HtmlAgilityPack.HtmlNode el = htmlDoc.DocumentNode.SelectSingleNode("(//div[@class='TableContainer'])[" + index + "]"); if (el != null) { foreach (HtmlNode node in el.SelectNodes(".//text()")) { Debug.WriteLine("text=" + node.InnerText.Repl...
.net .net-2.0 c# html-agility-pack xpath
asked by paul trmbrth

parse google search results

Sample code: ... string result = @"<div class=""srg""> <li class=""g"">...</li> <li class=""g"">...</li> <li class=""g"">...</li> <li class=""g"">...</li> <li class=""g"">...</li> <li class=""g"">...
c# html-agility-pack parsing
asked by TH Todorov

How can I scrape data from a Webpage after searching desired data using HTML Agility

Results for searches in the website you've mentioned are rendered dynamically using Javascript and the data comes as Json response via Ajax. HtmlAgilityPack is intended to parse Html, not Json....Consider using ...Selenium... or ...iMacros... drivers for .Net, or ...WebBrowser... class provided Microsoft Framework. These tools run a browser in back...
asp.net c# html html-agility-pack
asked by derloopkat

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
View more on Stack Overflow

Prime Library

Performance

Expression Evaluator

More Projects...

Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow