[quick note] Web Scraper 101 - 1
Basic rule: Create Sitemap -> add new selector -> set interval and delay -> start scrape.
Some of the grammars need to know:
Steps: Example: https://movie.douban.com/top250?start=[0-225:25]&filter= [0-225:25] means from 0 to 255 with the step of 25.
Break out: add this after the selector content ":nth-of-type(-n+number)" without the quote mark. For example, if the selector is: span.title:nth-of-type(1) and you want 100 lines of data, then change the selector to this: span.title:nth-of-type(1):nth-of-type(-n+100)
Lesson 1: How to select multiple elements on the page and scrape 'em all? USE THE CONTAINER!
A container is a block that contains a group of elements on the page you may want to scrape. Typically a block's attribute is "class".
For example, on the top 250 movie page, one container includes all the information about ONE movie, the poster, name, director name, scores and so on. In the element viewing window it shows like this:
To select multiple elements on the page, the rule is to select multiple containers which contain these elements, then in the container, select individual elements. Remember, do not select multiple elements that are in different containers individually. Instead, select multiple containers, add individual elements you want to scrape into the container!
For example, the douban movie top 250. If you want to scrape the name, score, the number of comments, and so on, you'll need to MULTIPLE select each block of the movie as a container, then in the container, select the elements without multiple select. Here's how it goes:
a. create sitemap b. create a "container" selector, type: element, multi-select all containers, meaning all the blocks with each movie as a block, that is everything between <li class="">...</li>
c. go in the "container" selector, add all the selectors you want. Remember to deselect the "multiple select" here cuz you're now in the "container" selector, one container has only one movie name, or score, and such.
You could go to the " selector graph" to check the structure of all your selectors.
