Introduction to this course.

Let’s set some goals of what we are going to learn.

What is web scraping and why do we use it? A small introduction to web scraping before we begin.

Before we start this course, we need to talk about legal and ethical issues. You can’t go scraping all the websites you want. There are rules, terms of service, and even laws.

We are going to make a simple class that holds all the information we will scrape from the websites. Each time we get the information from an online source we need to store the information somewhere. For now, that is in-memory.

The very first thing we need to do is to grab the HTML from the online source we want to scrape. I will start with the page that shows all the blogs on my website.

In the previous chapter, we have retrieved the HTML. Now we need to inspect that HTML and only extract that information we need. We do this with the package HTML Agility Pack, which is built to inspect HTML with XPath.

We now have the data from the website. Let’s show this information on the screen.

Now that we know the basics of building one scraper, let’s add another one! But from a different website. The goal is to only build a new scraper and not change the whole application. Both scrapers (the previous one and the new one) should work along side each other.

The strategy pattern can be used to simply switch between different algorithms without changing the code’s logic. Ideal if you have multiple scrapers and you don’t want huge if-statements.

Request headers store vital information, such as authentication/authorization, the method (POST, GET, PUT, DELETE), the content type, and much more. In this chapter, we will add the user-agent header since this is mostly required by websites.

If you want to scrape a website with different pages you are launching a lot of requests to that one website and/or server. At one point the website and/or server is going to block you. To avoid this we are going to build a mechanism that restricts the number of requests within a specific time window. A throttling system will add delays between the requests.

We are going to add a retry mechanism to our code. We will add a retry counter, check the request response and retry the request when the status code is 429, and repeat this routine until the maximum number of retries is reached.

Grabbing the information from the web pages is cool, but storing the data somewhere would be convenient. Let’s store some information in a SQLite database and use that stored information in our application.

Time to add some performance changes. The code works, no worries there, but there are a few parts we can make better. In this chapter, we will look at socked exhaustion, pipelining, and the HttpClientFactory.

A small recap and some small notes after the course.