Screaming Frog Simplified

Screaming Frog is a tool used to crawl a website and identify its technical issues which, when resolved, can improve SEO. Even though it is a complex application, we would like to spotlight some features you can start using today. Screaming Frog has both a free and paid version. Beware that the free version has a limit of 500 links to crawl while the paid version has an unlimited amount of links to crawl.

When you download and open Screaming Frog, on the main navigation bar you can see the “Mode” menu. It has three items: Spider, List, and SERP.

Screenshot of title mode in Screaming Frog

SERP Mode

SERP mode is handy when you have a list of pages and you want to check their Title lengths. The list must be in XLSX or CSV format and have “URL” as the label for the URL column, “Title” as the label for the Title column, and “Description” as the label for the Description column.

Let’s use one of our blogs with a lengthy title, “How to Keep Staging and Sandbox Traffic out of Google Analytics,” as an example. While you would normally use any mode of Screaming Frog to review a large number of links simultaneously, we will just use one link in our example. Here is our CSV file with all the required labels:

Examples of proper csv file for Screaming Frog

To begin, let’s switch to SERP Mode in Screaming Frog:

Screenshot of SERP Mode in Screaming Frog

Next, click Upload File:

Screenshot of where to upload files on Screaming Frog

After we selected our file, we got a message that Screaming Frog could read the file successfully. Now we can click OK:

Example of Message from Screaming Frog after reading SERP file

After clicking OK, this is what our screen looks like:

Screenshot of Title Length on Screaming Frog

We can see that the Title for the blog is 63 characters including spaces. It is recommended to keep Titles 50-70 characters long for SEO; although the Title for our blog appears lengthy, it doesn’t exceed the limit.

List Mode

You can use List mode in Screaming Frog to quickly check the status and indexability of specific pages. For this mode, you can upload a file or enter the links manually. When you enter the links manually, this window will appear:

Screenshot of Screaming Frog URL List uploads

Click Next. A Reading File window will appear; click OK. Now we can see in the Status column that our links have 404, 301, and 200 statuses:

Screenshot of Screaming Frog Status Column

Spider Mode

Spider mode is the most popular mode to work with in Screaming Frog. You will easily find online resources about it—from beginner’s guides to advanced information about its capabilities. Nevertheless, let us share a couple of our tips:

You can crawl all the links on a site by entering the website’s address and clicking the Start button:

Screenshot of Screaming Frog Start Button

If you are crawling a large site but get only a few links in results (or if you don’t get any at all), go to the Configuration menu on the main navigation bar and select Spider:

Screenshot of Screaming Frog Spider Configuration

In the Spider Configuration settings, the “Follow Internal ‘nofollow’” checkbox must be selected like this:

Screenshot of Screaming Frog Spider Mode No Follow Option

Next, go to Configuration >  robots.txt > Settings:

Screenshot of Screaming Frog Robot.txt Settings

In the Robots Setting window, in the drop-down list, select “Ignore robots.txt”:

Screenshot of Screaming Frog Robots.txt Settings indicating Ignore robots.txt

Now you should get all the links. You can use Filter to search for a specific type of content:

Screenshot of Screaming Frog Filters

Once you’ve compiled the links you want to work with, you can review the information in the Status, Indexability, Title, Title length, and other columns. In addition, you can use the Export button to create CSV or Excel files containing the links:

Screenshot of Screaming Frog Export Button

Finally, you can use the created files for reports or for feeding a plugin the data. To learn more about Screaming Frog and its capabilities, check out their blog.