HTML is almost intuitive. CSS is a great advancement that cleanly separates the structure of a page from its look and feel. JavaScript adds some pizazz. That's the theory. The real world is a little different.
In this tutorial, you'll learn how the content you see in the browser actually gets rendered and how to go about scraping it when necessary. In particular, you'll learn how to count Disqus comments. Our tools will be Python and awesome packages like requests, BeautifulSoup, and Selenium.
Web scraping is the practice of automatically fetching the content of web pages designed for interaction with human users, parsing them, and extracting some information (possibly navigating links to other pages). It is sometimes necessary if there is no other way to extract the necessary information. Ideally, the application provides a dedicated API for accessing its data programmatically. There are several reasons web scraping should be your last resort:
Let's understand what we are up against, by looking at the output of some common web application code. In the article Introduction to Vagrant, there are some Disqus comments at the bottom of the page:

In order to scrape these comments, we need to find them on the page first.
Every browser since the dawn of time (the 1990s) has supported the ability to view the HTML of the current page. Here is a snippet from the view source of Introduction to Vagrant that starts with a huge chunk of minified and uglified JavaScript unrelated to the article itself. Here is a small portion of it:

Here is some actual HTML from the page:

This looks pretty messy, but what is surprising is that you will not find the Disqus comments in the source of the page.
It turns out that the page is a mashup, and the Disqus comments are embedded as an iframe (inline frame) element. You can find it out by right-clicking on the comments area, and you'll see that there is frame information and source there:

That makes sense. Embedding third-party content as an iframe is one of the primary reasons to use iframes. Let's find the <iframe> tag then in the main page source. Foiled again! There is no <iframe> tag in the main page source.
The reason for this omission is that view page source shows you the content that was fetched from the server. But the final DOM (document object model) that gets rendered by the browser may be very different. JavaScript kicks in and can manipulate the DOM at will. The iframe can't be found, because it wasn't there when the page was retrieved from the server.
Static scraping ignores JavaScript. It fetches web pages from the server without the help of a browser. You get exactly what you see in "view page source", and then you slice and dice it. If the content you're looking for is available, you need to go no further. However, if the content is something like the Disqus comments iframe, you need dynamic scraping.
Dynamic scraping uses an actual browser (or a headless browser) and lets JavaScript do its thing. Then, it queries the DOM to extract the content it's looking for. Sometimes you need to automate the browser by simulating a user to get the content you need.
Let's see how static scraping works using two awesome Python packages: requests for fetching web pages and BeautifulSoup for parsing HTML pages.
Install pipenv first, and then: pipenv install requests beautifulsoup4
This will create a virtual environment for you too. If you're using the code from gitlab, you can just pipenv install.
Fetching a page with requests is a one liner: r = requests.get(url)
The response object has a lot of attributes. The most important ones are ok and content. If the request fails then r.ok will be False and r.content will contain the error. The content is a stream of bytes. It is usually better to decode it to utf-8 when dealing with text:
>>> r = requests.get('http://www.c2.com/no-such-page')
>>> r.ok
False
>>> print(r.content.decode('utf-8'))
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /ggg was not found on this server.</p>
<hr>
<address>
Apache/2.0.52 (CentOS) Server at www.c2.com Port 80
</address>
</body></html>
If everything is OK then r.content will contain the requested web page (same as view page source).
The get_page() function below fetches a web page by URL, decodes it to UTF-8, and parses it into a BeautifulSoup object using the HTML parser.
def get_page(url):
r = requests.get(url)
content = r.content.decode('utf-8')
return BeautifulSoup(content, 'html.parser')
Once we have a BeautifulSoup object, we can start extracting information from the page. BeautifulSoup provides many find functions to locate elements inside the page and drill down deep nested elements.
Tuts+ author pages contain multiple tutorials. Here is my author page. On each page, there are up to 12 tutorials. If you have more than 12 tutorials then you can navigate to the next page. The HTML for each article is enclosed in an <article> tag. The following function finds all the article elements on the page, drills down to their links, and extracts the href attribute to get the URL of the tutorial:
def get_page_articles(page):
elements = page.findAll('article')
articles = [e.a.attrs['href'] for e in elements]
return articles
The following code gets all the articles from my page and prints them (without the common prefix):
page = get_page('https://tutsplus.com/authors/gigi-sayfan')
articles = get_page_articles(page)
prefix = 'https://code.tutsplus.com/tutorials'
for a in articles:
print(a[len(prefix):])
Output:
building-games-with-python-3-and-pygame-part-5--cms-30085
building-games-with-python-3-and-pygame-part-4--cms-30084
building-games-with-python-3-and-pygame-part-3--cms-30083
building-games-with-python-3-and-pygame-part-2--cms-30082
building-games-with-python-3-and-pygame-part-1--cms-30081
mastering-the-react-lifecycle-methods--cms-29849
testing-data-intensive-code-with-go-part-5--cms-29852
testing-data-intensive-code-with-go-part-4--cms-29851
testing-data-intensive-code-with-go-part-3--cms-29850
testing-data-intensive-code-with-go-part-2--cms-29848
testing-data-intensive-code-with-go-part-1--cms-29847
make-your-go-programs-lightning-fast-with-profiling--cms-29809Static scraping was good enough to get the list of articles, but as we saw earlier, the Disqus comments are embedded as an iframe element by JavaScript. In order to harvest the comments, we will need to automate the browser and interact with the DOM interactively. One of the best tools for the job is Selenium.
Selenium is primarily geared towards automated testing of web applications, but it is great as a general-purpose browser automation tool.
Type this command to install Selenium: pipenv install selenium
Selenium needs a web driver (the browser it automates). For web scraping, it usually doesn't matter which driver you choose. I prefer the Chrome driver. Follow the instructions in this Selenium guide.
In some cases you may prefer to use a headless browser, which means no UI is displayed. Theoretically, PhantomJS is just another web driver. But, in practice, people reported incompatibility issues where Selenium works properly with Chrome or Firefox and sometimes fails with PhantomJS. I prefer to remove this variable from the equation and use an actual browser web driver.
Let's do some dynamic scraping and use Selenium to count Disqus comments on Tuts+ tutorials. Here are the necessary imports.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.expected_conditions import (
presence_of_element_located)
from selenium.webdriver.support.wait import WebDriverWaitThe get_comment_count() function accepts a Selenium driver and URL. It uses the get() method of the driver to fetch the URL. This is similar to requests.get(), but the difference is that the driver object manages a live representation of the DOM.
Then, it gets the title of the tutorial and locates the Disqus iframe using its parent id disqus_thread and then the iframe itself:
def get_comment_count(driver, url):
driver.get(url)
class_name = 'content-banner__title'
name = driver.find_element_by_class_name(class_name).text
e = driver.find_element_by_id('disqus_thread')
disqus_iframe = e.find_element_by_tag_name('iframe')
iframe_url = disqus_iframe.get_attribute('src')
The next step is to fetch the contents of the iframe itself. Note that we wait for the comment-count element to be present because the comments are loaded dynamically and not necessarily available yet.
driver.get(iframe_url)
wait = WebDriverWait(driver, 5)
commentCountPresent = presence_of_element_located(
(By.CLASS_NAME, 'comment-count'))
wait.until(commentCountPresent)
comment_count_span = driver.find_element_by_class_name(
'comment-count')
comment_count = int(comment_count_span.text.split()[0])
The last part is to return the last comment if it wasn't made by me. The idea is to detect comments I haven't responded to yet.
last_comment = {}
if comment_count > 0:
e = driver.find_elements_by_class_name('author')[-1]
last_author = e.find_element_by_tag_name('a')
last_author = e.get_attribute('data-username')
if last_author != 'the_gigi':
e = driver.find_elements_by_class_name('post-meta')
meta = e[-1].find_element_by_tag_name('a')
last_comment = dict(
author=last_author,
title=meta.get_attribute('title'),
when=meta.text)
return name, comment_count, last_commentWeb scraping is a useful practice when the information you need is accessible through a web application that doesn't provide an appropriate API. It takes some non-trivial work to extract data from modern web applications, but mature and well-designed tools like requests, BeautifulSoup, and Selenium make it worthwhile.
Additionally, don’t hesitate to see what we have available for sale and for study in the Envato Market, and don't hesitate to ask any questions and provide your valuable feedback using the feed below.
Create Modern Vue Apps Using Create-Vue and Vite
/
5Pros and Cons of Using WordPress
/
-4How to Fix the “There Has Been a Critical Error in Your Website” Error in WordPress
/
1How To Fix The “There Has Been A Critical Error in Your Website” Error in WordPress
How to Create a Privacy Policy Page in WordPress
/
8WordPress Website Maintenance Guide For Beginners
/
6How Long Does It Take to Learn JavaScript?
/
-1How Long Does it Take to Learn JavaScript
/
1The Best Way to Deep Copy an Object in JavaScript
/
-2
4
-5Adding and Removing Elements From Arrays in JavaScript
/
-2Create a JavaScript AJAX Post Request: With and Without jQuery
/
75 Real-Life Uses for the JavaScript reduce() Method
/
-9How to Enable or Disable a Button With [removed] jQuery vs. Vanilla
/
3How to Enable or Disable a Button With [removed] jQuery vs Vanilla
/
-4Confirm Yes or No With JavaScript
/
-1How to Change the URL in [removed] Redirecting
/
-315+ Best WordPress Twitter Widgets
/
-227 Best Tab and Accordion Widget Plugins for WordPress (Free & Premium)
/
1021 Best Tab and Accordion Widget Plugins for WordPress (Free & Premium)
/
-1130 HTML Best Practices for Beginners
/
-831 Best WordPress Calendar Plugins and Widgets (With 5 Free Plugins)
/
525 Ridiculously Impressive HTML5 Canvas Experiments
/
-8
1How to Implement Email Verification for New Members
/
3How to Create a Simple Web-Based Chat Application
/
530 Popular WordPress User Interface Elements
/
-5Top 18 Best Practices for Writing Super Readable Code
/
2Best Affiliate WooCommerce Plugins Compared
/
-1018 Best WordPress Star Rating Plugins
/
1410+ Best WordPress Twitter Widgets
/
320+ Best WordPress Booking and Reservation Plugins
/
5Working With Tables in React: Part Two
/
-4Best CSS Animations and Effects on CodeCanyon
/
-1630 CSS Best Practices for Beginners
/
4How to Create a Custom WordPress Plugin From Scratch
/
510 Best Responsive HTML5 Sliders for Images and Text… and 3 Free Options
/
-416 Best Tab and Accordion Widget Plugins for WordPress
/
-418 Best WordPress Membership Plugins and 5 Free Plugins
/
325 Best WooCommerce Plugins for Products, Pricing, Payments and More
/
310 Best WordPress Twitter Widgets
1 /
1512 Best Contact Form PHP Scripts for 2020
/
-220 Popular WordPress User Interface Elements
/
-210 Best WordPress Star Rating Plugins
/
-312 Best CSS Animations on CodeCanyon
/
112 Best WordPress Booking and Reservation Plugins
/
220 Best WordPress Calendar Plugins and Widgets
/
-212 Elegant CSS Pricing Tables for Your Latest Web Project
/
724 Best WordPress Form Plugins for 2020
/
-214 Best PHP Event Calendar and Booking Scripts
/
-1Create a Blog for Each Category or Department in Your WooCommerce Store
8 Best WordPress Booking and Reservation Plugins
/
-18Best Exit Popups for WordPress Compared
/
19Best Exit Popups for WordPress Compared
/
-611 Best Tab & Accordion WordPress Widgets & Plugins
/
-1012 Best Tab & Accordion WordPress Widgets & Plugins
1New Course: Practical React Fundamentals
/
3Short Course: Better Angular App Architecture With Modules
/
2Preview Our New Course on Angular Material
/
5Object-Oriented PHP With Classes and Objects
/
3Best Practices for ARIA Implementation
/
-2Accessible Apps: Barriers to Access and Getting Started With Accessibility
/
-3Dramatically Speed Up Your React Front-End App Using Lazy Loading
/
1915 Best Modern JavaScript Admin Templates for React, Angular, and Vue.js
/
-1415 Best Modern JavaScript Admin Templates for React, Angular and Vue.js
/
219 Best JavaScript Admin Templates for React, Angular, and Vue.js
/
5New Course: Build an App With JavaScript and the MEAN Stack
/
10Hands-on With ARIA: Accessibility Recipes for Web Apps
/
810 Best WordPress Facebook Widgets
13 /
-5Hands-on With ARIA: Accessibility for eCommerce
/
6New eBooks Available for Subscribers
/
-5Hands-on With ARIA: Homepage Elements and Standard Navigation
/
10
-9Site Accessibility: Getting Started With ARIA
/
2How Secure Are Your JavaScript Open-Source Dependencies?
/
-2New Course: Secure Your WordPress Site With SSL
/
-3Testing Components in React Using Jest and Enzyme
/
-8Testing Components in React Using Jest: The Basics
/
-415 Best PHP Event Calendar and Booking Scripts
/
-1Create Interactive Gradient Animations Using Granim.js
/
10How to Build Complex, Large-Scale Vue.js Apps With Vuex
1 /
1Examples of Dependency Injection in PHP With Symfony Components
/
-4Set Up Routing in PHP Applications Using the Symfony Routing Component
1 /
11A Beginner’s Guide to Regular Expressions in JavaScript
/
12Introduction to Popmotion: Custom Animation Scrubber
/
-4Introduction to Popmotion: Pointers and Physics
/
-10New Course: Connect to a Database With Laravel’s Eloquent ORM
/
-5How to Create a Custom Settings Panel in WooCommerce
/
-12Building the DOM faster: speculative parsing, async, defer and preload
1 /
120 Useful PHP Scripts Available on CodeCanyon
3 /
49How to Find and Fix Poor Page Load Times With Raygun
/
2Introduction to the Stimulus Framework
/
-14Single-Page React Applications With the React-Router and React-Transition-Group Modules
612 Best Contact Form PHP Scripts
1 /
-4Getting Started With the Mojs Animation Library: The ShapeSwirl and Stagger Modules
/
2Getting Started With the Mojs Animation Library: The Shape Module
/
-4Getting Started With the Mojs Animation Library: The HTML Module
/
3Project Management Considerations for Your WordPress Project
/
-98 Things That Make Jest the Best React Testing Framework
/
3Creating an Image Editor Using CamanJS: Layers, Blend Modes, and Events
/
-3New Short Course: Code a Front-End App With GraphQL and React
Creating an Image Editor Using CamanJS: Applying Basic Filters
/
-14Creating an Image Editor Using CamanJS: Creating Custom Filters and Blend Modes
/
1Challenge: Create a To-Do List in React
1Deploy PHP Web Applications Using Laravel Forge
/
-1Getting Started With the Mojs Animation Library: The Burst Module
/
6
-9A Gentle Introduction to Higher-Order Components in React: Best Practices
/
-4Challenge: Build a React Component
/
-4A Gentle Introduction to HOC in React: Learn by Example
/
5A Gentle Introduction to Higher-Order Components in React
/
-6Creating Pretty Popup Messages Using SweetAlert2
/
5Creating Stylish and Responsive Progress Bars Using ProgressBar.js
/
1718 Best Contact Form PHP Scripts for 2022
/
-3How to Make a Real-Time Sports Application Using Node.js
/
1Creating a Blogging App Using Angular & MongoDB: Delete Post
/
-17Set Up an OAuth2 Server Using Passport in Laravel
/
1Creating a Blogging App Using Angular & MongoDB: Edit Post
/
-2Creating a Blogging App Using Angular & MongoDB: Add Post
/
-1Creating a Blogging App Using Angular & MongoDB: Show Post
/
8Creating a Blogging App Using Angular & MongoDB: Home
/
10
2Creating a Blogging App Using Angular & MongoDB: Login
/
13Creating Your First Angular App: Implement Routing
/
13Persisted WordPress Admin Notices: Part 4
/
-4Creating Your First Angular App: Components, Part 2
/
3Persisted WordPress Admin Notices: Part 3
/
3How Laravel Broadcasting Works
/
1
9Persisted WordPress Admin Notices: Part 2
/
-3Create Your First Angular App: Storing and Accessing Data
/
2Persisted WordPress Admin Notices: Part 1
/
-3Error and Performance Monitoring for Web & Mobile Apps Using Raygun
/
6Using Luxon for Date and Time in JavaScript
7 /
-5How to Create an Audio Oscillator With the Web Audio API
/
-6How to Cache Using Redis in Django Applications
/
220 Essential WordPress Utilities to Manage Your Site
/
-5Introduction to API Calls With React and Axios
/
18Beginner’s Guide to Angular 4: HTTP
/
11Rapid Web Deployment for Laravel With GitHub, Linode, and RunCloud.io
/
8Beginners Guide to Angular 4: Routing
/
-9Beginner’s Guide to Angular 4: Components
/
1Creating a Drop-Down Menu for Mobile Pages
/
2Introduction to Forms in Angular 4: Writing Custom Form Validators
/
510 Best WordPress Booking & Reservation Plugins
/
3Getting Started With Redux: Connecting Redux With React
/
15Getting Started With Redux: Learn by Example
/
15Getting Started With Redux: Why Redux?
/
9Understanding Recursion With JavaScript
/
-2How to Auto Update WordPress Salts
/
-8How to Download Files in Python
/
-4
-22Eloquent Mutators and Accessors in Laravel
1 /
17
-11
1210 Best HTML5 Sliders for Images and Text
/
5Site Authentication in Node.js: User Signup
/
-4Creating a Task Manager App Using Ionic: Part 2
/
2Creating a Task Manager App Using Ionic: Part 1
/
1
3Introduction to Forms in Angular 4: Template-Driven Forms
/
-124 Essential WordPress Utilities to Manage Your Site
/
-125 Essential WordPress Utilities to Manage Your Site
/
-12Get Rid of Bugs Quickly Using BugReplay
1 /
-13Manipulating HTML5 Canvas Using Konva: Part 1, Getting Started
/
710 Must-See Easy Digital Downloads Extensions for Your WordPress Site
22 Best WordPress Booking and Reservation Plugins
/
-5Understanding ExpressJS Routing
/
-515 Best WordPress Star Rating Plugins
/
12Inheritance and Extending Objects With JavaScript
/
-7Introduction to the CSS Grid Layout With Examples
1 /
2
8Performant Animations Using KUTE.js: Part 5, Easing Functions and Attributes
/
-8Performant Animations Using KUTE.js: Part 4, Animating Text
/
-12Performant Animations Using KUTE.js: Part 3, Animating SVG
/
-8New Course: Code a Quiz App With Vue.js
/
-1Performant Animations Using KUTE.js: Part 2, Animating CSS Properties
/
3
9Performant Animations Using KUTE.js: Part 1, Getting Started
/
410 Best Responsive HTML5 Sliders for Images and Text (Plus 3 Free Options)
/
-3Single-Page Applications With ngRoute and ngAnimate in AngularJS
/
-12Deferring Tasks in Laravel Using Queues
/
-5
4Site Authentication in Node.js: User Signup and Login
/
11Working With Tables in React, Part Two
/
-16Working With Tables in React, Part One
/
6How to Set Up a Scalable, E-Commerce-Ready WordPress Site Using ClusterCS
/
-2New Course on WordPress Conditional Tags
/
4
-11Building With Vue.js 2 and Firebase
6 /
10
-6
1Best Unique Bootstrap JavaScript Plugins
/
-10
-5Essential JavaScript Libraries and Frameworks You Should Know About
/
7
2Vue.js Crash Course: Create a Simple Blog Using Vue.js
/
-13Build a React App With a Laravel RESTful Back End: Part 1, Laravel 5.5 API
/
5API Authentication With Node.js
/
-4Beginner’s Guide to Angular: HTTP
/
7Beginner’s Guide to Angular: Routing
/
-4Beginners Guide to Angular: Routing
/
-6Beginner’s Guide to Angular: Services
/
5Beginner’s Guide to Angular: Components
/
-1How to Create a Custom Authentication Guard in Laravel
/
7Learn Computer Science With [removed] Part 3, Loops
/
-2Build Web Applications Using Node.js
/
3Learn Computer Science With [removed] Part 4, Functions
/
3Learn Computer Science With [removed] Part 2, Conditionals
/
12Learn Computer Science With [removed] Part 1, The Basics
/
-5Create Interactive Charts Using Plotly.js, Part 5: Pie and Gauge Charts
/
-8Create Interactive Charts Using Plotly.js, Part 4: Bubble and Dot Charts
/
18Create Interactive Charts Using Plotly.js, Part 3: Bar Charts
/
-14Awesome JavaScript Libraries and Frameworks You Should Know About
/
-4Create Interactive Charts Using Plotly.js, Part 2: Line Charts
/
-8Bulk Import a CSV File Into MongoDB Using Mongoose With Node.js
/
-4Build a To-Do API With Node, Express, and MongoDB
/
-6Getting Started With End-to-End Testing in Angular Using Protractor
/
10TypeScript for Beginners, Part 4: Classes
/
-10Object-Oriented Programming With JavaScript
/
610 Best Affiliate WooCommerce Plugins Compared
/
9Stateful vs. Stateless Functional Components in React
/
11Make Your JavaScript Code Robust With Flow
/
-6Build a To-Do API With Node and Restify
/
6Testing Components in Angular Using Jasmine: Part 2, Services
/
1Testing Components in Angular Using Jasmine: Part 1
/
-2Creating a Blogging App Using React, Part 6: Tags
/
-11React Crash Course for Beginners, Part 3
/
2React Crash Course for Beginners, Part 2
/
12React Crash Course for Beginners, Part 1
/
2Set Up a React Environment, Part 4
1 /
8Set Up a React Environment, Part 3
/
-13New Course: Get Started With Phoenix
/
-11Set Up a React Environment, Part 2
/
2Command Line Basics and Useful Tricks With the Terminal
/
3How to Create a Real-Time Feed Using Phoenix and React
/
-8Build a React App With a Laravel Back End: Part 2, React
/
2Build a React App With a Laravel RESTful Back End: Part 1, Laravel 9 API
/
-7Creating a Blogging App Using React, Part 5: Profile Page
/
-5Pagination in CodeIgniter: The Complete Guide
/
22JavaScript-Based Animations Using Anime.js, Part 4: Callbacks, Easings, and SVG
/
-1JavaScript-Based Animations Using Anime.js, Part 3: Values, Timeline, and Playback
Learn to Code With [removed] Part 1, The Basics
/
510 Elegant CSS Pricing Tables for Your Latest Web Project
/
-6Getting Started With the Flux Architecture in React
/
-2Getting Started With Matter.js: The Composites and Composite Modules
/
4Getting Started With Matter.js: The Engine and World Modules
/
1510 More Popular HTML5 Projects for You to Use and Study
/
-24Understand the Basics of Laravel Middleware
/
-6Iterating Fast With Django & Heroku
/
8Creating a Blogging App Using React, Part 4: Update & Delete Posts
/
3Creating a jQuery Plugin for Long Shadow Design
/
-9How to Register & Use Laravel Service Providers
2 /
-13Unit Testing in React: Shallow vs. Static Testing
/
5Creating a Blogging App Using React, Part 3: Add & Display Post
/
15Creating a Blogging App Using React, Part 2: User Sign-Up
20 /
-5Creating a Blogging App Using React, Part 1: User Sign-In
/
109 More Popular HTML5 Projects for You to Use and Study
/
18Creating a Grocery List Manager Using Angular, Part 2: Managing Items
/
99 Elegant CSS Pricing Tables for Your Latest Web Project
/
1Dynamic Page Templates in WordPress, Part 3
/
-2Angular vs. React: 7 Key Features Compared
/
6Creating a Grocery List Manager Using Angular, Part 1: Add & Display Items
/
2New eBooks Available for Subscribers in June 2017
/
7Create Interactive Charts Using Plotly.js, Part 1: Getting Started
/
11The 5 Best IDEs for WordPress Development (And Why)
/
233 Popular WordPress User Interface Elements
/
2New Course: How to Hack Your Own App
/
-10How to Install Yii on Windows or a Mac
/
5What Is a JavaScript Operator?
/
-5How to Register and Use Laravel Service Providers
/
9
waly Good blog post. I absolutely love this…