Screaming Frog, crawl and audit your web (Version 14.1)

We’re already going through version 14.1 of Screaming Frog. They don’t stop surprising me and helping a lot of SEOs around the world. The desktop crawler par excellence.

Why? Because we need to monitor every aspect of a website, analyze and study every detail to ensure the pillars of our website’s SEO strategy. There is a wide variety of tools to achieve this, although there are some that stand out in this group due to their efficiency and effectiveness. Screaming Frog is one of those applications that you should have in your basic suite. Next, I will give you all the information about this tool so you can find out why.

What is Screaming Frog and what is it for?

Screaming Frog is a web crawler, an essential tool for analyzing and auditing websites. It is an application that is available for Windows, Linux, or Mac and allows us to crawl links on a website.

Learn how to download and configure it

The opportunities offered by Screaming Frog are very varied, so the benefit we can get from this tool depends on how we handle those options and the analysis we make of the results. But, to start we must download the tool. We can get a free version on the official website of Screaming Frog that, although it will have certain limitations, it works very well for analyzing small project pages.

Before reading on if you don’t have it, go ahead and download it.

Similarly, we will also find the paid version, which I assure you is worth it if we have the possibility of acquiring it.

Let’s go there! The first thing we must do is open the application and enter the web domain in the corresponding place, then we start the configuration of the crawling of this page.

In the configuration section we can enter the options that we are interested in analyzing, among the options that this tool allows us to crawl we have: external and internal links, CSS, images, Java Script, among others. Finally, when we select what we want to crawl we can press the Start button and the crawling process will start, we must wait for it to finish and then extract the information we are interested in from the results. It should be noted that this procedure is extremely basic, we can always go a little deeper to get even more detailed information.

First approach to a screaming frog report

Status codes

In the tab “Response Codes” you can get a list of all the web addresses, here you can filter the result according to the answer code they give. This way, you can analyze all the results that do not have a good response code and carry out the necessary proceedures to improve your website.
With the results of this tab you can check if the URLs you want are blocked by robots.txt. You can also check that your page doesn’t have “no response” pages, broken pages that give 404 error and the quality of the redirects.

URI

The URI section shows all the information about the URLs of the website, especially about the way they are written. Here you can filter the information in this way:

• Non ASCII Characters: URLs that have non-ASCII characters.
• Underscores: URLs that contain underscores
• Uppercase: URLs with capital letters.
• Duplicate: Identical URLs are detected here.
• Parameters: URLs that have parameters, such as blocking by robots, noindex, among others, are shown here.
• Over 115 Characters: very long URLs are shown here.

Titles

In the Titles panel we will be able to see information about the “title” of each page, here we can see if it exists or not, if they are duplicated, if they are too long or short and also if there are multiple, that is to say, if there are several titles in the same URL.

Meta-description

The Meta-Description panel is shown next to the Titles panel, so the filters and information we can get is exactly the same. We can see if the description is empty, duplicated, if it is very short or long and if it is multiplied within the same page.

Images

The section of images is one of the easiest, in this we can filter the result to obtain the images that weigh more than 100 kb, images that lack ALT attribute or those that have this very long aspect. These three aspects are very important to take care of when looking for SEO optimization.

Spider Configuration

Content

Within the Configuration – Spider option, we can choose how we analyze and process the content of the web page:

• Text Only
• Old AJAX Crawling Scheme
• Javascript

The Old AJAX Crawling Scheme option is the default, it emulates a Google system that crawls AJAX content. This method has been obsolete for few years, however, it’s still one of the closest methods to crawling today. On the other hand, if you select the Text Only option, you can only obtain the HTML code and the information related to it. Whereas the JavaScript option crawls the code of this type, which is undoubtedly the tool’s heaviest process, so it is normal that it takes a little longer.

Robots.txt

The next option, robots.txt, shows us the following three options within the Settings option:

• Ignore robots.txt
• Show internal URLs blocked by robots.txt
• Show external URLs blocked by robots.tx

Depending on the option we select we can ignore all the robots.txt files on the web, show the URLs blocked by the robots.txt -the most recommended option- or show only the external URLs that are blocked.

Protocol

If you have migrated your page from http to https, you will be interested in the information provided by the Protocol tab, which allows you to differentiate between the two types of address (http and https). 

Sitemap XML

Screaming Frog allows us to easily create an XML Sitemap, for this we just have to click on the option Create XML Sitemap and configure the options we want: include pages without index, PDFs, canonical, among others.

We can leave each of the options with its default configuration and select Next. Then, we will get the map completely ready, which can easily be saved.

Spider mode, list mode and SERP mode

If we activate the spider mode the tool will act by crawling the URL information in URLs, always following each of the internal links of the web page. In the list mode, it will crawl the URLs indicated in a list, which must be done manually. This mode allows us to practically analyze the URLs in a Sitemap and see if they have errors or are damaged. In this option, we also have to load the URLs manually, but this time in a CSV document.

Why Use Screaming Frog

Las opciones que nos ofrece la herramienta Screaming Frog son muy diversas, ya que no solo podemos aplicarlas a una página web, sino que podemos también analizar la competencia y analizar otros detalles referentes al rendimiento de la web.

Analyze your own website

Screaming Frog is a must-have application for optimizing your own website, since you can get very important information that will help you improve the quality of your website. You will be able to solve errors and facilitate the indexing of information by the different web engines.

Analysis of the competition

One of the great benefits Screaming Frog offers you is that you can also analyze your competition. Through the results on the tags, titles, descriptions, and so on, you can discover the keywords they are using the most.

Linkbuilding management

Screaming Frog allows us to analyze external or outgoing links, which allows us to analyze the sites where competitors’ pages are linking to.

Discover URLs with no traffic

This tool allows you to see the organic flow that certain specific URLs receive, so you can pay more attention to them and optimize the content to further improve your website.

Latest changes v. 14.1

• Dark mode
• Exporting documents to Google Spreadsheet
• HTTP headers
• Storage of cookies through crawling
• Number of discovered URLs per directory (in tree view)
• New advanced configuration options: ignore non-indexable URLs for on-page filters and ignore paginated URLs for duplicate filters.
• Alt attribute filter not found in the images tab.
• Small updates and improvements.