HttpClient doesn't get full website html source

Question

HttpClient doesn't get full website html source

c# html-agility-pack http web-scraping win-universal-app

Question

I try to scrap offers from http://olx.pl/ site, I'm using HttpClient, problem is that site retrived from client is way diffrient and doesn't contain offers list like it is in source code accessed directly from browser. Any idea? Here's my code:

  string url = "http://olx.pl/oferty/q-diablo/?search%5Bdescription%5D=1";
  HttpClient client = new HttpClient();
  string result = await client.GetStringAsync(url);

Chiro

View more on Stack Overflow

Accepted Answer

HttpClient wont load content that is generated from javascript. Instead you can use WebView that will run js. I ran both, HttpClient result had length of 235507 and WebView result lenght of 464476.

    WebView wv = new WebView();
    wv.NavigationCompleted += Wv_NavigationCompleted;
    wv.Navigate(new Uri(url));

    private async void Wv_NavigationCompleted(WebView sender, WebViewNavigationCompletedEventArgs args)
    {
        string wvresult = await sender.InvokeScriptAsync("eval", new string[] { "document.documentElement.outerHTML;" });
    }

HttpClient doesn't get full website html source

Question

Accepted Answer

Performance

Expression Evaluator

Related