Html Agility Pack - Knowledge Base (KB)
59 results in tag: web-scraping
HTML Agility Pack Screen Scraping XPATH isn't returning data
I'm attempting to write a screen scraper for Digikey that will allow our company to keep accurate track of pricing, part availability and product replacements when a part is discontinued. There seems to be a discrepancy between the XPATH that I'm seeing ...
asked by Matthias
Html Agility Pack: Find Comment Node
I am scraping a website that uses Javascript to dynamically populate the content of a website with the Html Agility pack....Basically, I was searching for the XPATH ..."\\div[@class='PricingInfo']"..., but that div node was being written to the DOM via Ja...
asked by Abe
Using HTML Agility Pack to get text next to image?
I have this bit of html that I need to parse though...<p class="feature_list"> <img src="candy.gif" alt="candy" title="candy"/> x 3 <img src="lollies.gif" alt="lollies" title="lollies"/> ...
asked by Diskdrive
Masking your web scraping activities to look like normal browser surfing activities?
I'm using the Html Agility Pack and I keep getting this error. "The remote server returned an error: (500) Internal Server Error." on certain pages....Now I'm not sure what this is, as I can use Firefox to get to these pages without any problems....I have...
asked by Diskdrive
How to parse a FORM from WebResponse into a POST body of a WebRequest
I'm new to this, this is my virgin voyage, the task at hand is to create a transaction in C# that will navigate through a page flow of a web app via WebRequest/WebResponse. I got the Request/Response mechanism working, cookies and all (I can successfully ...
asked by Jessica Jones
Html Agility Pack C#: Expression must evaluate to a node-set
I'm using Html Agility Pack to fetch a webpage. I want to collect all the TEXT I AM LOOKING FOR of the following form:...<li><a href="/deal/map/4087664" class="show-location" title="bla bla" data-address="TEXT I AM LOOKING FOR"></a></li> ...I tried this c...
asked by Elad Benda
Extracting table from html into htmltable in asp.net vb (htmlagilitypack)
I am trying to grab a html table from a remote page and display the contents of this table in a htmltable on my site. I am using htmlagility pack. So far here is my code:...Imports HtmlAgilityPack Partial Class ContentGrabExperiment Inherits System.We...
asked by Phil
HTML Scraping with HTML Agility Pack
Can some one tell me the best way to get the contents using HTMLAgilityPack that i mention below from the html....In the HTML provided i need to scrape ...value... of the ID "img" and set the values for ...x... and ...y... for them to be used in another f...
asked by user671805
Screen Scraping, Web Scraping, Web Harvesting, Web Data Extraction, etc. using C# and the .NET Framework
I am working on a Microsoft .NET Application in C# for Web Harvesting, Web Scraping, Web Data Extraction, Screen Scraping, etc. whatever you want to call it. For parsing HTML, I'm attempting to incorporate HTML Agility Pack but it's not as easy as I thoug...
asked by John
HTML Agility Pack - Filter Href Value Results
I'm working on a web scraper. The following text shows the results of the code given at the end of this question, which gets the values of all hrefs from a page. ...I only want to get values that contain ...docid=...index.php?pageid=a45475a11ec72b843d7495...
asked by guitarPH
Html Agility Pack. Load and scrape webpage
Is this the ...best... way to get a webpage when scraping?...HttpWebRequest oReq = (HttpWebRequest)WebRequest.Create(url); HttpWebResponse resp = (HttpWebResponse)oReq.GetResponse(); var doc = new HtmlAgilityPack.HtmlDocument(); doc.Load(resp.GetRespons...
asked by thatsIT
Removing useless TextNodes in HtmlAgilityPack
I'm scraping a number of websites using HtmlAgilityPack. The problem is that it seems to insist on inserting TextNodes in most places which are either empty or just contain a mass of \n, whitespaces and \r....They tend to cause me issues when I'm counting...
asked by Aabela
Extracting a table with Html Agility Pack
I am trying to extract a table from a webpage using Html Agility Pack. So far I have managed to do a little of progress with it. This is my code so far... Dim web As New HtmlAgilityPack.HtmlWeb() Dim htmlDoc As HtmlAgilityPack.HtmlDocument = web.Lo...
asked by Rob Schneider
Scraping data using HTMLAgilityPack
In HTMLAgailityPack, how to get the data from the website which is not coming in the innerhtml method of it. For example, if in the link below:...https://www.theice.com/productguide/ProductSpec.shtml?specId=1496#expiry...The table starting with contract s...
asked by nishant jain
How to scrape a flash based site?
We are using ...Html Agility Pack... to scrape data for HTML-based site; is there any DLL like ...Html Agility Pack... to scrape flash-based site?
asked by Noor
Html Agility Pack, Web Scraping, and spoofing in C#
Is there a way to spoof a web request from C# code so it doesn't look like a bot or spam hitting the site? I am trying to web scrape my website, but keep getting blocked after a certain amount of calls. I want to act like a real browser. I am using thi...
asked by Mike Flynn
How to get table from Wikipedia
I want to put one table from Wikipedia into xml file and then parse it to C#. Is it possible? If yes, can I save in xml only ...Title... and ...Genre... column?...HtmlWeb web = new HtmlWeb(); HtmlDocument doc = web.Load("http://en.wikipedia.org/wiki/2012_...
asked by a1204773
HTML Agility Pack and LINQ
I want to use HAP to scrape data from a table on a website, loop through the rows to find a value in a column that matches a predefined string and then store only that row that matches. Then I will have a dictionary with the column header as the key, and ...
asked by jrob42
Using htmlAgilityPack scraping all inner text from <a> tag
I want to scrap all the word from the link ...http://search.freefind.com/siteindex.html?id=59478474<r=10240&fwr=0&pid=i&ics=1... I tried something like this:...HtmlWeb web = new HtmlWeb(); HtmlAgilityPack.HtmlDocument doc = web.Load("http://search.free...
asked by ImonBayazid
HtmlAgilityPack SelectNodes, Disposing
I am trying to do some screen scraping using HtmlAgilityPack using ...SelectNodes... and getting some values from each node returned...Here is the code ...private readonly HtmlDocument _document = new HtmlDocument(); public void ParseValues(string html) ...
asked by Roman Ratskey
Page 1 of 3