VB.net extraire des liens de google-search en utilisant HtmlAgilityPack

google-search html-agility-pack vb.net

Question

J'ai maintenant mis à jour mon code en tant que test. Je souhaite répertorier toutes les URL du mot index.php, mais il affiche également d'autres éléments.

Voici mon code de travail:

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

    Dim webClient As New System.Net.WebClient
    Dim WebSource As String = webClient.DownloadString("http://www.google.com/search?lr=&cr=countryCA&newwindow=1&hl=fil&as_qdr=all&biw=1366&bih=667&tbs=ctr%3AcountryCA&q=index.php&oq=index.php&gs_l=serp.12..0l10.520034.522335.0.525032.9.9.0.0.0.0.497.3073.1j1j2j0j5.9.0....0...1c.1.25.serp..5.4.884.J4smY262XgY")
    RichTextBox1.Text = WebSource

    ListBox1.Items.Clear()


    Dim htmlDoc As New HtmlAgilityPack.HtmlDocument()
    htmlDoc.LoadHtml(WebSource)

    For Each link As HtmlNode In htmlDoc.DocumentNode.SelectNodes("//cite")

        If link.InnerText.Contains("index.php") Then
            ListBox1.Items.Add(link.InnerText)
        End If

    Next

End Sub

la sortie attendue ne devrait être que les sites Web sur lesquels il y a index.php, comme ceci:

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

    Dim webClient As New System.Net.WebClient
    Dim WebSource As String = webClient.DownloadString("http://www.google.com/search?lr=&cr=countryCA&newwindow=1&hl=fil&as_qdr=all&biw=1366&bih=667&tbs=ctr%3AcountryCA&q=index.php&oq=index.php&gs_l=serp.12..0l10.520034.522335.0.525032.9.9.0.0.0.0.497.3073.1j1j2j0j5.9.0....0...1c.1.25.serp..5.4.884.J4smY262XgY")
    RichTextBox1.Text = WebSource

    ListBox1.Items.Clear()


    Dim htmlDoc As New HtmlAgilityPack.HtmlDocument()
    htmlDoc.LoadHtml(WebSource)

    For Each link As HtmlNode In htmlDoc.DocumentNode.SelectNodes("//cite")

        If link.InnerText.Contains("index.php") Then
            ListBox1.Items.Add(link.InnerText)
        End If

    Next

End Sub

Mais le problème est qu'il ne s'arrête que jusqu'à index.php, les autres parties du lien ne sont pas incluses.

par exemple l'URL complète est

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

    Dim webClient As New System.Net.WebClient
    Dim WebSource As String = webClient.DownloadString("http://www.google.com/search?lr=&cr=countryCA&newwindow=1&hl=fil&as_qdr=all&biw=1366&bih=667&tbs=ctr%3AcountryCA&q=index.php&oq=index.php&gs_l=serp.12..0l10.520034.522335.0.525032.9.9.0.0.0.0.497.3073.1j1j2j0j5.9.0....0...1c.1.25.serp..5.4.884.J4smY262XgY")
    RichTextBox1.Text = WebSource

    ListBox1.Items.Clear()


    Dim htmlDoc As New HtmlAgilityPack.HtmlDocument()
    htmlDoc.LoadHtml(WebSource)

    For Each link As HtmlNode In htmlDoc.DocumentNode.SelectNodes("//cite")

        If link.InnerText.Contains("index.php") Then
            ListBox1.Items.Add(link.InnerText)
        End If

    Next

End Sub

le programme affiche seulement

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

    Dim webClient As New System.Net.WebClient
    Dim WebSource As String = webClient.DownloadString("http://www.google.com/search?lr=&cr=countryCA&newwindow=1&hl=fil&as_qdr=all&biw=1366&bih=667&tbs=ctr%3AcountryCA&q=index.php&oq=index.php&gs_l=serp.12..0l10.520034.522335.0.525032.9.9.0.0.0.0.497.3073.1j1j2j0j5.9.0....0...1c.1.25.serp..5.4.884.J4smY262XgY")
    RichTextBox1.Text = WebSource

    ListBox1.Items.Clear()


    Dim htmlDoc As New HtmlAgilityPack.HtmlDocument()
    htmlDoc.LoadHtml(WebSource)

    For Each link As HtmlNode In htmlDoc.DocumentNode.SelectNodes("//cite")

        If link.InnerText.Contains("index.php") Then
            ListBox1.Items.Add(link.InnerText)
        End If

    Next

End Sub

ou il aurait cassé des points comme

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

    Dim webClient As New System.Net.WebClient
    Dim WebSource As String = webClient.DownloadString("http://www.google.com/search?lr=&cr=countryCA&newwindow=1&hl=fil&as_qdr=all&biw=1366&bih=667&tbs=ctr%3AcountryCA&q=index.php&oq=index.php&gs_l=serp.12..0l10.520034.522335.0.525032.9.9.0.0.0.0.497.3073.1j1j2j0j5.9.0....0...1c.1.25.serp..5.4.884.J4smY262XgY")
    RichTextBox1.Text = WebSource

    ListBox1.Items.Clear()


    Dim htmlDoc As New HtmlAgilityPack.HtmlDocument()
    htmlDoc.LoadHtml(WebSource)

    For Each link As HtmlNode In htmlDoc.DocumentNode.SelectNodes("//cite")

        If link.InnerText.Contains("index.php") Then
            ListBox1.Items.Add(link.InnerText)
        End If

    Next

End Sub

Réponse acceptée

Je voudrais utiliser Html Agility Pack pour extraire les liens comme ci-dessous

Dim links As New List(Of String)()
Dim htmlDoc As New HtmlAgilityPack.HtmlDocument()
htmlDoc.LoadHtml(WebSource)
For Each link As HtmlNode In htmlDoc.DocumentNode.SelectNodes("//a[@href]")
    Dim att As HtmlAttribute = link.Attributes("href")
    If att.Value.Contains("/forums/") Then
        links.Add(att.Value)
    End If
Next

si c'est le résultat de recherche google essayez quelque chose comme ci-dessous

Dim links As New List(Of String)()
Dim htmlDoc As New HtmlAgilityPack.HtmlDocument()
htmlDoc.LoadHtml(WebSource)
For Each link As HtmlNode In htmlDoc.DocumentNode.SelectNodes("//a[@href]")
    Dim att As HtmlAttribute = link.Attributes("href")
    If att.Value.Contains("/forums/") Then
        links.Add(att.Value)
    End If
Next



Sous licence: CC-BY-SA with attribution
Non affilié à Stack Overflow
Est-ce KB légal? Oui, apprenez pourquoi
Sous licence: CC-BY-SA with attribution
Non affilié à Stack Overflow
Est-ce KB légal? Oui, apprenez pourquoi