Question
New to C# here and to running asynchronous tasks.
I'm trying to scrape some music album info from a website. The webpage's search produces a JSON object in plaintext, but I can't seem to access any DOM info. Here's what I tried (using HtmlAgilityPack):
using HtmlAgilityPack;
using System;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace WindowsFormsApp1 {
public partial class Form1 : Form {
public Form1() {
InitializeComponent();
}
public async Task<String> AlbumScraper(string albumname) {
HtmlWeb web = new HtmlWeb();
string albumurl = Uri.EscapeUriString("https://www.metal-archives.com/search/ajax-album-search/?field=title&query=" + albumname);
Console.Write(albumurl);
var albumdoc = await Task.Factory.StartNew(() => web.Load(albumurl));
string albumjson = "";
if (albumdoc.DocumentNode != null) {
albumjson = albumdoc.DocumentNode.InnerText;
}
return albumjson;
}
private async void Form1_Load(object sender, EventArgs e) {
string rawtext = await AlbumScraper("rust+in+peace");
Console.Write(rawtext);
}
}
}
How can I grab the JSON text generated? I can see it clearly when I load the "albumurl" URL...
Accepted Answer
Firstly you don't need HtmlAgilityPack.
Secondly,try:
using Newtonsoft.Json.Linq;
string albumurl = Uri.EscapeUriString("https://www.metal-archives.com/search/ajax-album-search/?field=title&query=rust+in+peace");
string doc = "";
using (System.Net.WebClient client = new System.Net.WebClient()) // WebClient class inherits IDisposable
{
doc = client.DownloadString(albumurl);
}
then you could deserialize it (@itikhomi)
AlbumSearchResponse data = JsonConvert.DeserializeObject<AlbumSearchResponse>(doc);
You could also parse it manually
JObject json = JObject.Parse(doc);
string error= Convert.ToString(json["error"]);
. . .
string aaData= Convert.ToString(json["aaData"]);
JArray arr = JArray.Parse(aaData);
foreach(JToken token in arr)
{
string[] strarr = token.ToObject<string[]>();
}
Popular Answer
You can generate it by online tool http://json2csharp.com/ Then add generated class to your code
public class AlbumSearchResponse
{
public string error { get; set; }
public int iTotalRecords { get; set; }
public int iTotalDisplayRecords { get; set; }
public int sEcho { get; set; }
public List<List<string>> aaData { get; set; }
}
convert your response text to class
var data = JsonConvert.DeserializeObject<AlbumSearchResponse>(response);
foreach (var item in data.aaData)
{
//do whatever your want with data
}
also you need add newtonsoft json package from nuget to get JsonConvert working
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow