Screaming Frog is a tool used to crawl a website and identify its technical issues which, when resolved, can improve SEO. Even though it is a complex application, we would like to spotlight some features you can start using today. Screaming Frog has both a free and paid version. Beware that the free version has a limit of 500 links to crawl while the paid version has an unlimited amount of links to crawl.
When you download and open Screaming Frog, on the main navigation bar you can see the “Mode” menu. It has three items: Spider, List, and SERP.
SERP Mode
SERP mode is handy when you have a list of pages and you want to check their Title lengths. The list must be in XLSX or CSV format and have “URL” as the label for the URL column, “Title” as the label for the Title column, and “Description” as the label for the Description column.
Let’s use one of our blogs with a lengthy title, “How to Keep Staging and Sandbox Traffic out of Google Analytics,” as an example. While you would normally use any mode of Screaming Frog to review a large number of links simultaneously, we will just use one link in our example. Here is our CSV file with all the required labels:
To begin, let’s switch to SERP Mode in Screaming Frog:
Next, click Upload File:
After we selected our file, we got a message that Screaming Frog could read the file successfully. Now we can click OK:
After clicking OK, this is what our screen looks like:
We can see that the Title for the blog is 63 characters including spaces. It is recommended to keep Titles 50-70 characters long for SEO; although the Title for our blog appears lengthy, it doesn’t exceed the limit.
List Mode
You can use List mode in Screaming Frog to quickly check the status and indexability of specific pages. For this mode, you can upload a file or enter the links manually. When you enter the links manually, this window will appear:
Click Next. A Reading File window will appear; click OK. Now we can see in the Status column that our links have 404, 301, and 200 statuses:
Spider Mode
Spider mode is the most popular mode to work with in Screaming Frog. You will easily find online resources about it—from beginner’s guides to advanced information about its capabilities. Nevertheless, let us share a couple of our tips:
You can crawl all the links on a site by entering the website’s address and clicking the Start button:
If you are crawling a large site but get only a few links in results (or if you don’t get any at all), go to the Configuration menu on the main navigation bar and select Spider:
In the Spider Configuration settings, the “Follow Internal ‘nofollow’” checkbox must be selected like this:
Next, go to Configuration > robots.txt > Settings:
In the Robots Setting window, in the drop-down list, select “Ignore robots.txt”:
Now you should get all the links. You can use Filter to search for a specific type of content:
Once you’ve compiled the links you want to work with, you can review the information in the Status, Indexability, Title, Title length, and other columns. In addition, you can use the Export button to create CSV or Excel files containing the links:
Finally, you can use the created files for reports or for feeding a plugin the data. To learn more about Screaming Frog and its capabilities, check out their blog.
Subscribe via RSS
