SEO and Screaming Frog

Easy SEO anyone can do

Did you arrive at the Fireplace Aquarium by way of a search engine like google or bing?  How search engines choose which sites to show and in which order can be influenced by ‘search engine optimisation’ or SEO.  An important aspect of SEO, in addition to having interesting and informative content, is having the technical aspects of a website sorted out such that the website facilitates the content instead of getting in the way.  Currently, I’m working on the following pieces:

1. No broken links, missing pages or ‘404’ errors or other obvious sources of user frustration
2. Informative meta descriptions for web pages so search engines don’t have to guess what to say
3. All images have an ‘ALT’ text description to aid the visually impaired and help search engines understand the messages images are conveying
4. Verifying that search engines know about all the different pages they might decide to index using Google Search Console (or similar)
5. Add headings that match questions Google searchers have asked^* and provide helpful answers to those questions.

How to use Screaming Frog for SEO

A really great tool to help accomplish the above is the Screaming Frog SEO spider.  You download and run the app^**, then point it at your website where it then ‘crawls’ all the interconnected pages exactly like the search engines do, collecting information you can query in a user-friendly way.  For example, you can click a button to see a list of all the broken links in your website (or ideally, see that there aren’t any broken links).  Similarly you can query which pages have meta descriptions, which images have ALT descriptions, and what those descriptions say.  It’s also easy to generate a list of all the web pages, and indications as to which can be indexed by search engines, and then to compare that list with pages actually indexed by google or bing etc.  Where the search engines have missed some pages, you can point the engines to those pages specifically.

How to make a Screaming Frog sitemap

One of the cool things you can do is to generate visual sitemaps, either based on the directory structure, or the route in which the SEO spider crawls the site, and see those as either a tree graph or force-directed graph.  The graphs can be exported in either SVG or HTML format; unfortunately, WordPress doesn’t play well with SVG, so I converted the SVG saved output to PNG using Gimp and you can see two versions of the results below.

Making the sitemap is easy.  Launch up Screaming Frog and type the base address of the website into the ‘Enter URL to spider’ address bar at the top, press ‘Start’ and wait for the scan to complete.  Then select your visualisation of choice from the Visualisations menu.  Once the visualisation comes up, have a play with the settings by clicking on the small gear icon in upper right corner of the visualisation window.  When it’s looking how you want it to, click the diskette icon to the immediate left of the gear icon to bring up the save screen, pick where you want the visualisation to be saved, select either ‘svg’ or ‘html’ format from the dropdown menu at the bottom right of the save screen, and then press ‘Save’.

^*Real world example:  I clicked the ‘Performance’ tab in my Google Search Console and found I was getting queries for “screaming frog visual sitemap”.  I then did a google search for “screaming frog visual sitemap” to see what people wanted, and the search result page helpfully said:

People also ask:

• How do I make a screaming frog sitemap?
• How do you use a screaming frog for SEO?  [really, that’s what they ask]

So I made level 2 headings corresponding to those two questions, and then (hopefully) provided some helpful answers.  If you found these answers were helpful, you could thank me by linking to my site from your website, ideally with a “do follow” link. 😀

^**Screaming Frog is a UK-based outfit (which is nice) and provides their SEO spider free of charge for use with smaller websites (less than 500 URLs).  They claim versions for Windows, macOS and Ubuntu, however, I develop Fireplace Aquarium on a Chromebook running Debian linux (the testing repository) and much to my surprise (and delight), the Ubuntu .deb file installed using gdebi without incident and seems to run just fine under Debian.