I am trying to fetch the website content using `httpclinet as you can see here
public async Task<List<NewsContent>> parsing(string newsArchive)
{
List<NewsContent> lstResult=new List<NewsContent>();
HttpClient http = new HttpClient();
var response = await http.GetByteArrayAsync(newsArchive);
String source = Encoding.GetEncoding("utf-8").GetString(response, 0, response.Length - 1);
source = WebUtility.HtmlDecode(source);
HtmlDocument resultat = new HtmlDocument();
resultat.LoadHtml(source);
List<HtmlNode> toftitle = resultat.DocumentNode.Descendants().Where
(x => (x.Name == "div" && x.Attributes["class"] != null && x.Attributes["class"].Value.Contains("news_list"))).ToList();
var li = toftitle[0].Descendants().Where
(x => (x.Name == "div" && x.Attributes["class"] != null && x.Attributes["class"].Value=="news_item")).ToList();
foreach (var item in li)
{
NewsContent newsContent = new NewsContent();
newsContent.Url = item.Descendants("a").ToList()[0].GetAttributeValue("href", null);
newsContent.Img = item.Descendants("img").ToList()[0].GetAttributeValue("src", null);
newsContent.Title = item.Descendants("h2").ToList()[0].InnerText;
//finding main news content
var response1 = await http.GetByteArrayAsync("http://www.nsfund.ir/news" + newsContent.Url);
String source1 = Encoding.GetEncoding("utf-8").GetString(response1, 0, response1.Length - 1);
source1 = WebUtility.HtmlDecode(source1);
HtmlDocument resultat1 = new HtmlDocument();
resultat1.LoadHtml(source1);
newsContent.Content = resultat1.DocumentNode.SelectSingleNode("//div[@class='news_content_container']").InnerText;
}
return lstResult;
}
As you can see i used async
method to get the data .here :
var response = await http.GetByteArrayAsync(newsArchive);
But the problem is when i call my async
function :
News newagent = new News();
Task<List<NewsContent>> lst = newagent.parsing("http://www.nsfund.ir");
Task.WaitAll(lst);
List<NewsContent> enresult = lst.Result;
I don't get any result.so i decided to convert this async
function to a normal function ,what kind of code should be replaced with this :
var response = await http.GetByteArrayAsync(newsArchive);
I guess i found the problem with your code.
You're not adding the NewsContent
object to your List
.
in the foreach
loop please add it to the List
lstResult.Add(newsContent)
Hope it solves the issue with your async
strategy
But the problem is when i call my async function :
Task.WaitAll(lst);
List<NewsContent> enresult = lst.Result;
Yes, that's a problem, all right. Two problems, actually: Task.WaitAll
and Result
. They should both be replaced with a single await
:
List<NewsContent> enresult = await lst;
The core problem is a deadlock scenario that I explain in full on my blog. In summary, await
will capture the current context and use that to resume the async
method. But ASP.NET only allows one thread within its request context at a time. So when parsing
is first called, it executes until it hits its await
and then returns. The calling method then blocks; this is where the problem is, because by blocking, the calling method is keeping a thread in that ASP.NET request context.
Later, when the await
inside parsing
is done, it attempts to resume the parsing
method in that ASP.NET request context but it cannot, because there's a thread stuck in that context and ASP.NET only allows one thread at a time. The calling method is waiting for parsing
to complete, and parsing
is waiting for the context to be free. Classic deadlock.