HttpClient doesn't get full website html source
c# html-agility-pack http web-scraping win-universal-app
Question
I try to scrap offers from http://olx.pl/ site, I'm using HttpClient, problem is that site retrived from client is way diffrient and doesn't contain offers list like it is in source code accessed directly from browser. Any idea? Here's my code:
string url = "http://olx.pl/oferty/q-diablo/?search%5Bdescription%5D=1";
HttpClient client = new HttpClient();
string result = await client.GetStringAsync(url);
Accepted Answer
HttpClient wont load content that is generated from javascript. Instead you can use WebView that will run js. I ran both, HttpClient result had length of 235507 and WebView result lenght of 464476.
WebView wv = new WebView();
wv.NavigationCompleted += Wv_NavigationCompleted;
wv.Navigate(new Uri(url));
private async void Wv_NavigationCompleted(WebView sender, WebViewNavigationCompletedEventArgs args)
{
string wvresult = await sender.InvokeScriptAsync("eval", new string[] { "document.documentElement.outerHTML;" });
}
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow