I was reading an article the other day and I came across the term "web crawler". The context in which it was used got me a little curious about the design of a web crawler. A web crawler is a simple program that scans or "crawls" through web pages to create an index of the … Continue reading How To Build A Web Crawler?