Screaming Frog is a very useful tool with a bit of a strange name. It’s excellent for having an overall look at a web site for on-page SEO audits. It’s especially useful to find the right KPIs in your web site. The free version allows crawling of 500 internal + external URLs, while the premium version removes this limitation and adds options such as saving results, data exports, reports creation, sitemap creation and some more.
This is how the program interface looks like:
You add an URL to spider, and after it finishes crawling a web site, the interface changes to:
The first and second tab (“Internal” and “External”) give you an overall idea on the URLs crawled. Internal refers to the internal URLs of the web site and external to the URLs from other domains (it’s useful to know, for example, if you link out to web sites which give errors). I will not be focusing on these two tabs, as the other tabs within the software tend to be much more relevant.
“Protocol” is the first tab we’ll focus on:
“Status Code” is useful to find out which are the response codes sent by your server. I include below an infographic which can help you better understand what response codes are:
“Response Codes” tab also shows information on the number of internal links a URL has (the higher the number, the better). “Response Time” gives you an indication on how fast a web site loads. “Redirect URI” is useful to use to find out, if you have a redirection in place, what is the new, redirected URL.
“Page Titles” tab shows information about the (so-called, although incorrectly, “meta”) title tag. There can be more than one appearance of the tag within a page, and you can see this in the “Occurrences” column. Now, how large can a meta title be? You can measure this in characters, and for this you will find use of “Title 1 Length”, or you can measure the pixel size of a specific text (letters “W” and “I” have a totally different pixel length, a title containing ALL CAPS text will likely be larger than a text with lots of “i”s and “l”s). The recommended size for title length varies from once source to another, but 65 characters are generally safe, 70 is a bit pushy, while texts over 75 characters are considered too large. For Pixel Width, the recommended size is 512 pixels. Also, it’s best to test these yourself, as Google, on rare occasions, changes these recommendations.
“Meta Description” tab includes some similar pieces of information as the “Meta Titles” tag. In here, the recommended sizes for “Length” tab is generally 155 characters, perhaps 160 even, but no larger than 170 characters (Google has been testing some larger sizes, also).
I advise against using meta keywords tag within a web site. The meta tag hasn’t been used by the search engines for its original purpose for years (it might be used against you, if you abuse that tag, you might get a penalty flag). Thus, I generally use the information on “Meta Keywords” tab just to find out which pages contain that tag, so I can remove them.
“H1” tab just gives you information on the number of times the tag occurs and its length. I don’t have an optimal suggested size for length, just note that very large H1s shouldn’t exist on a web site. What is “very large”, I leave you to decide, it’s not an exact science. More problematic is the number of occurrences. I’ve seen quite a few web sites who put H1 tag on the logo. It shouldn’t be there. Others use H1 tag not as “title of the page”, as it was intended, but as “this is a large, bolded, text”. Thus, in this case it is a bit more difficult to make the change to the template used on the web site. I would advise to always use just one H1 tag, for the title of the pages. The homepage generally shouldn’t have a H1 tag.
The “H2” tab refers to, what else about?, the H2 tags on your web site. My advice on using this tag is to use it only for very large pages on your web site, for sub-titles. Also, for online stores, in category listings, you can add it to product names. Unfortunately, this tag is also misused, I’ve seen plenty of times other things than the recommended ones being used on H2 tag. As it is a less important issue, you can either solve it or just ignore it, it’s not that important.
The “images” tab shows some details about the images. There is a column visible right away which shows the image size. I recommend to consider compressing the images which are larger than 120, even 100 KB. If you click on an image, you will see the details of the ALT text of that image. This is an important element, it should contain the title of the image, very useful for search engines, as they can’t easily decipher what an image is about. For the same purpose, the name of the image (you can see it at the end of the URL) should also use plain English, rather than contain a random set of letters and numbers. The separator for more than one word in the URL of an image should be “-“.
The “Directives” tab lets you know in an instant if there are meta robots tags such as noindex / no follow (you should use them, but with great care, and only if you really know what you are doing, do some research prior to using them). “Meta Refresh” is a poor way of doing redirects, and should be avoided. “Canonical Link Element” is a very useful tag to be used on HTML pages (so, not on images, but on destination URLs for HTML pages). I would consider adding it to all the pages of a web site, to avoid issues with duplicate content. Again, do some research prior to implementing it. Rel=”next”/”prev” is a useful tag to be used on pages with multipagination, for example, the multiple pages of a category listing. X Robots tag is rarely used on a web site, but you can specify in there, for example, if a page should have noindex or not.
The “Hreflang” tag lets you understand in an instant what translation of the current page you associate with. So, if you have a page in English, you can associate it with different other translated versions of the page. This tab helps you check if you have set the right associations.
The final two tabs, “Analytics” and “Search Console” help check whether the site has the tags for verification for both Google Analytics and Google Search Console.
Note: when you click on a URL in Screaming Frog, you will see much more details about that URL, depending on the tab you are in (it’s different for images, for examples). This is how the information might look like:
Note: Also see the Yahoo! Group on which I present similar issues: IMRo. To join, email firstname.lastname@example.org and reply to the confirmation email.