Using the Google Search Console

About This Document

This document introduced the Google Search Console – an essential tool for those managing public websites if you want your content to be found using Google.

Why Do I Need the Google Search Console?

Is this what you think?

• You’ve launched your website and Google will automatically index the content so people will easily find the website.
• You’ve had a website for a long time and Google will have indexed the old pages and will quickly index new content which is added.

You may be right 🙂 But it’s also possible that none of your pages have been indexed and are therefore invisible to Google!

A recent survey:(Google Coverage of Morris Federation Team Websites) found that 7 websites appeared to have no pages visible in Google and 40 had only a single page which was available in Google.

The Google Search Console is a free-to-use web-based service which lets you see how many pages on your website have been indexed by Google and provide information on pages which have not been indexed (for a variety of reasons) as well as giving you an indication of Google search traffic over time and the search terms used to deliver traffic to your website.

Before Using the Google Search Console

The Google Search Console requires some technical expertise before you can use it. However you can find out details about your website by simply using Google.

A quick way of checking Google coverage for your (or indeed any) website is to use the inurl: or site: Google command. Examples for the (1) Morris Federation (morrisfed.org.uk) and (2) the Weald of Kent Morris websites are given below:

These examples illustrate:

• Differing results can be obtained (and note that results can vary at different times of the day)
• If only a small number of results are shown, there may be a message saying “we have omitted some entries very similar to the 4 already displayed“. If you notice this you should click on the link saying “you can repeat the search with the omitted results included” to get a full set of results.

If you use this approach for your website and notice a disparity between the number of pages indexed by Google and the number of pages your site contains, you may find your website has indexing problems. The Google Search Console can identify problems.

IMPORTANT NOTE: When you look at the results for your website you should note that the initial numbers can significantly differ from the actual numbers. You should scroll to the bottom of the results and follow any links to further hits and also click to see the final page of results.

When you visit the final page of results you may find that the number of results is significantly lower than the number shown initially – this may be due to pages having been deleted or are no longer indexed, and Google initially displaying an estimate of the number of pages. This is another reason why use of the Google Search Console is recommended.

What Does Google Search Console Provide?

As described on the Google website:

“Google Search Console is a free service offered by Google that helps you monitor, maintain, and troubleshoot your site’s presence in Google Search results. You don’t have to sign up for Search Console to be included in Google Search results, but Search Console helps you understand and improve how Google sees your site.”

Search Console offers tools and reports for the following actions:

• Confirm that Google can find and crawl your site.
• Fix indexing problems and request re-indexing of new or updated content.
• View Google Search traffic data for your site: how often your site appears in Google Search, which search queries show your site, how often searchers click through for those queries, and more.
• Receive alerts when Google encounters indexing, spam, or other issues on your site.
• Show you which sites link to your website.
• Troubleshoot issues for AMP, mobile usability, and other Search features.”

Setting up Google Search Console

As described in Getting started with Search Console document to use Google Search Console you will need to:

• Sign up for Search Console (note it’s free)
• Add and verify ownership of your site. You’ll need to prove that you are the owner of your website, because Search Console shows information about your site that only site owners should have access to, and allows you to make changes that can affect how your site appears on Google.

Using Google Search Console

Once you have signed up and verified your ownership of your website you will be able to understand how Google is indexing your resources, how users use Google to find pages on your website, find problem areas and, possibly, fix the problem.

Example

To illustrate the key features of the Google Search Console the following screenshots are provided.

When you login you should see an Overview page for your website which gives a summary of the numbers of visits to your website from a Google search and the numbers of pages on your website which Google knows about:

The search results page allows you to see how much traffic your website has received (you can change the period for periods between the current day, last week and other period up to the last 16 months). It can be useful to see if your website received significant additional traffic on May Day. It can also be useful to look at the queries which people used in Google to find your website.

The third important page we’ll mention is ‘Coverage’ which shows the pages which Google has indexed. In addition you can also see the pages which Google may know about but has not indexed. However, by definition, the Search Console will not show the pages which Google does not know about!

If you notice that there is a difference between the number of pages indexed and the number of pages you know your website holds you should follow the trouble-shooting section.

Trouble-Shooting

If you visit the Coverage page, in addition to seeing the Valid section (which has details of pages “Indexed, not submitted in sitemap” and “Indexed, not submitted in sitemap”) there is also an Excluded section which includes details of pages:

• Excluded by ‘noindex’ tag
• Discovered – currently not indexed
• Crawled – currently not indexed
• Page with redirect
• Not found (404)
• Duplicate (various types)

Pages may be listed in the Excluded section for valid reasons: for example, you may wish to minimise duplicated results for pages which can be found in a number of ways (by author, category, date, etc.) and therefore tag the pages as ‘noindex’ so they will be listed as “Excluded by ‘noindex’”.

The categories in the Excluded section have a links to a Learn More page, as shown below.

The Learn More link takes you to the Index Coverage report help page, which is valuable in tracking down problems.

Identify and Fixing Problems

The survey (Google Coverage of Morris Federation Team Websites) showed that 7 websites appeared not to be indexed by Google and 40 had only a single page visible in Google.

If you have noticed disparity between the number of pages indexed by Google and the number of pages your site contains you may find the problem is caused by one of the following issues:

• Areas blocking by the robots.txt file: The robots.txt file can be used to instruct Google (and other search engines) not to index certain areas of the website. This is normally done (a) to block search engines before a website is officially launched; (b) to block areas from being found by Google (although the pages can be accessed by following links) or (c) to block search engines from accessing system files (e.g. log files). It may be that websites have some lines in the robots.txt file which block search engines from accessing key areas of the website.
• Misconfigured robots.txt file: It was noticed that at least one website uses a now obsolete crawl-delay command which may have resulted in only 1 page being indexed.
• Website configuration: There may be other options in the content management system (e.g. WordPress, Wix, etc) to prevent the site being indexed before it is launched, and this option was not changed after the launch.
• Web server configuration: Other options provided by the webserver hosting agency may prevent the site from being indexed.
• Frames and other HTML technologies: We noticed that at least one website used HTML FRAMES, which can act as a barrier to indexing.
• Dynamic web pages: Web pages which are created from content held in a backend database (often characterised by a URL containing a ‘?’ such as index.php?album=20170607 as used for photographs which have been indexed from the Weald of Kent Morris website) can be a barrier to indexing.
• Complexities in use of http: and https: protocols: Your site may have moved from use of http:// to the more secure https:// Normally this should work transparently (and since 2015 Google has been indexing HTTPS pages by default). However website owners sometimes report problems.
• Advanced Web technologies: Advanced web technologies (such as AJAX) can act as a barrier to indexing.

If your team’s website has such problems you should initially use the Google Search Console URL Inspection feature and paste in a URL which appears to be missing and then submit it for indexing.

If you are not able to immediately fix the problems, get in touch with the author of this document with the URL of the website details of the web technologies used and we will see if we can suggest a solution. Note that we can’t guarantee that we can help, but if we can find a simple solution we will let you know.

We welcome feedback on this document. In addition we would like to hear from morris, sword and other traditional dance sides who use IT and would be willing to share their experiences. Would you like to contribute a case study?

Status of this Document

Document published: 21 Feb 2022
Document updated: 15 Apr 2022 (link to Google Coverage of Morris Federation Team Websites added)

Author

Brian Kelly, Comms and IT Volunteer. Email: comms-it@morrisfed.org.uk

Licence for this Document

This document is available with a Creative Commons Sharealike (CC-BY) licence. In brief, this means you can copy and make changes to this document provided you give acknowledgements to the author/publisher. A suggested wording for acknowledgements is:

This document is based on the "Using the Google Search Console" document by Brian Kelly, Comms and IT volunteer with the Morris Federation.

#MorrisFedITResources #MFITResources