C# How to loop through a list, update the same list, and continue looping?

c# for-loop html-agility-pack loops web-scraping

Question

A webscraper that adds certain urls to a list is what I'm working on.

using HtmlAgilityPack;

List<string> mylist = new List<string>();
var firstUrl = "http://example.com";
HtmlWeb web = new HtmlWeb();
HtmlDocument document = web.Load(firstUrl);

HtmlNodeCollection nodes = document.DocumentNode.SelectNodes("//div[contains(@class,'Name')]/a");
            foreach (HtmlNode htmlNode in (IEnumerable<HtmlNode>)nodes)
            {
                if (!mylist.Contains(htmlNode.InnerText))
                {
                    mylist.Add(htmlNode.InnerText);

                }

            }

At this point, I want to loop through "mylist," do the same action, and essentially never stop. Newly parsed URLs should be added to the list by the code. What would be the simplest method to do this?

After the last loop, I attempted to create a for loop. But the list does not seem to be updated. It will perpetually cycle through the same things that are already on the list (since I will always be less than mylist.Count)

for (int i = 0; i < mylist.Count; i++)
            {
                //the items in mylist are added to the url
                var urls = "http://example.com" + mylist[i];

                HtmlWeb web = new HtmlWeb();
                HtmlDocument document = web.Load(urls);

                HtmlNodeCollection nodes = document.DocumentNode.SelectNodes("//div[contains(@class,'Name')]/a");
                foreach (HtmlNode htmlNode in (IEnumerable<HtmlNode>)nodes)
                {
                    if (!mylist.Contains(htmlNode.InnerText))
                    { 
                        mylist.Add(htmlNode.InnerText);
                    }

                }


            }

Thanks!

1
1
6/22/2018 3:38:19 AM

Accepted Answer

Queue suitable for your needs.

Queue<string> mylist = new Queue<string>();

Initial pass:

using HtmlAgilityPack;

Queue<string> mylist = new Queue<string>();
var firstUrl = "http://example.com";
HtmlWeb web = new HtmlWeb();
HtmlDocument document = web.Load(firstUrl);

HtmlNodeCollection nodes = document.DocumentNode.SelectNodes("//div[contains(@class,'Name')]/a");
            foreach (HtmlNode htmlNode in (IEnumerable<HtmlNode>)nodes)
            {
                if (!mylist.Contains(htmlNode.InnerText))
                {
                    mylist.Enqueue(htmlNode.InnerText);

                }
            }

the subsequent pass

while (mylist.Count > 0)
            {
                var url = mylist..Dequeue();
                //the items in mylist are added to the url
                var urls = "http://example.com" + url;

                HtmlWeb web = new HtmlWeb();
                HtmlDocument document = web.Load(urls);

                HtmlNodeCollection nodes = document.DocumentNode.SelectNodes("//div[contains(@class,'Name')]/a");
                foreach (HtmlNode htmlNode in (IEnumerable<HtmlNode>)nodes)
                {
                    if (!mylist.Contains(htmlNode.InnerText))
                    { 
                        mylist.Enqueue(htmlNode.InnerText);
                    }

                }


            }
1
6/22/2018 8:39:18 AM

Popular Answer

Go to NuGet and then enter "System.Interactive"

var found = new HashSet<string>();

var urls =
    EnumerableEx
        .Expand(
            new[] { "http://example.com" },
            url =>
            {
                var web = new HtmlWeb();
                var document = web.Load(url);
                var nodes = document.DocumentNode.SelectNodes("//div[contains(@class,'Name')]/a");
                return
                    nodes
                        .Cast<HtmlNode>()
                        .Select(x => x.InnerText)
                        .Where(x => !found.Contains(x))
                        .Do(x => found.Add(x))
                        .Select(x => "http://example.com" + x);
            });


Related Questions





Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow