Guide: How to Perform Keyword Analysis

The terms that a person puts into a search engine (like Google) are commonly referred to as key terms, keywords or key phrases.

Keyword research is the name of the process that we go through to find keywords, collect information about them and analyse what the user intent of those search terms might be in order that we provide the best content possible to service the users need.

How the Keyword Groups SEO Research Process Works

The process typically works in some key steps:

Finding the kind or types of keywords and key phrases that refer to the particular service or product that you are concerned with – a good way to begin the process is to build a “master list” or “seed list” of starting words and then try to identify the different core concepts or themes that exist within your original list.
Grouping keywords into the core ideas, themes and concepts that exist within your list based on the similarity, the intent the user may have or through other conceptual frameworks that might emerge.
Analysing the intention of the search term and the possible behaviour of the users searching for these various core ideas and concepts.
Ordering based On Value the groups and subsequent terms based on the volume and potential opportunities presented by a search term but tempered by the likelihood of ranking for the term based on how competitive it is.
Utilising your findings to inform your content marketing tactics and strategy

How to Use the Data From Your Keyword Research Analysis

The important information you glean from your keyword groups SEO research and analysis will inform various of stages of your content marketing strategy:

It will determine for you the demand within the market and help guide your positioning in the market
It will help you establish a calendar to develop your content strategy
It will allow you to prioritise technical and content fixes to your website.

All things considered the data you collect from this process is essentially to saving money in over the longer term of developing and implementing an effecient content strategy for the years to come in your business.

How Keyword Research Evolved Over Time

Search engines have always taken keywords as input. They have varied in the degree to which they have exposed how many people are searching for different things and in how much they have revealed about the ranking algorithms that they use to display results based on those keywords.

There are three main primary sources of keyword data:

Search engines themselves – releasing either aggregate data (like Google Keyword Planner) or data specific to websites that owners receive directly (through Google Search Console for example)
Websites’ own log files / analytics – if the search engine passes information to the destination site about the searcher’s query then websites can analyze their own search traffic
Aggregators of analytics across multiple websites – a number of popular software services and plugins that are used across significant portions of major websites have released aggregated data gathered from various sources. One excellent paid resource that offers a limited free trial is SEMRush.

Most of the other retail sources of keyword data buy or gather their data from one of these sources.

Changes to Keyword Data Availability

In late 2011, Google began moving from the vast majority of queries happening in the clear, over http (non-secure) connections to almost all queries happening via https (secure) connections. This automatically resulted in search queries being hidden from ISPs as the URLs being requested were obscured within encrypted requests.

At the same time, they changed the information passed to the destination website in the so-called ‘referrer’ tag which normally shows what URL someone was on before they clicked a link to your website. Google obscured the keyword data in the referrer field so website analytics (and aggregators of website analytics) could no longer analyse the search terms used by most of their visitors from search. In place of the old keyword data, Google started passing (not provided). It also became more difficult for website owners when Google stopped providing keyword volumes on their keyword planner unless you were spending sifnicant amounts on Google ads.

Between these changes, the current state of play is that keyword search data is only available from:

Search engines themselves as before
Aggregators relying on old data from before these changes
Paid tools like SEMrush and Ahrefs

Other Inputs Enhance Search Results

The other major change to impact how we think about keywords is the increasing diversity of inputs considered by the algorithm. In addition to the words entered by the user, Google in particular will consider a number of implicit aspects of the query in order to customize the results. Among others, these can include:

Geographical location – both country / state information (for US users) as well as fine-grained location (such as city/town) can impact search results
Device
Search history and other implicit preferences
Social connections and social sharing

It used to be that search engines would return the exact same set of results in the same order for any user for a given input. Now that is no longer true, we need to pay even more attention to the intent of the user and their context.

Probably the best example of this is the increasing utility of very bland searches. Searches like “cinema” would not have returned useful results in the past. These days a search like this is likely to be treated with implicit geographical, social and preference inputs as a search for something much more specific and you are likely to get recommendations of restaurants nearby.

How Does this Change the Uses of Keyword Data?

It would definitely be nice to have access to more granular data than is now available, but the main effect of the changes is to encourage a focus on qualitative keyword research over exact mathematical calculations. In particular:

Much more attention paid to user intent
Additional focus on the confounding variables (location, time, preferences)
More thematic grouping and less focus on individual keywords

After digging a little deeper into the data quality, we’ll start learning more about the process itself.

Data Availability and Skew

In an ideal world, for any conceivable theme, we would have access to:

Search volumes
Breakdown by geography, demographic and intent
Seasonality trends
Levels of competition

We do have some information in all of these areas, but it’s not perfect. In particular, since categories 1-3 are now almost exclusively provided by the search engines and paid search tools it is hard to sense-check the information provided and there are some generalities that can be used to make judgments but figuring this out is not an exact science.

The primary source of keyword data from Google is the Keyword Planner. It ostensibly provides volume (if you pay for ads, otherwise it is vague) and seasonality data, but we now have little way of sense-checking it.

For some of the other data pieces, we have to turn to any source we can find for keyword data. SEMrush is probably the best paid tool on the marketplace for keyword data across the globe.

There are other free keyword research tools available but you’ll find that you are are very limited on the data you can pull from them.

As a general rule, there is a natural tendency for Google to over-emphasize commercial searches – if for no other reason than its utility largely coming from its use in keyword advertising which is naturally commercially-focused. For this, and other similar reasons, it is important to treat Google-provided keyword data as directional and relative as well as using it for idea-mining rather than as an input into traffic modelling unless you are prepared to work around the caveats.

The Long Tail Keyword

At Google, the top 20% of searches account for much less than 80% of the search volume. The incredible volume in the “long tail” of searches (each of which may be executed only once a month or even less frequently) is mind-boggling. It’s hard for us to comprehend, but one illustration that can help is to think about the trillions of unique search queries Google has seen up to this point – and to realise that 20-25% of all searches are brand new each day.

The upshot of all of this is that keyword data is naturally limited in its ability to analyze all the incredible variety of ways that people search. Not only are individual keywords less interesting than aggregated search volume across a wide range of related long-tail keywords, but the fact that many of these long-tail keywords have never been searched-for before makes them exceptionally hard to analyze.

This is yet one more reason why our approach is therefore to seek themes, concepts and groups rather than to analyze each keyword individually

The Different Types of Keyword

Searcher Intent

There are a variety of ways of classifying different types of search by intent, but SEO people generally refer to the following segmentation:

Navigational: when the searcher knows exactly what they are seeking and simply uses the search engine to find the location. An example of a navigational search is “Facebook” as a keyword shows the intention of looking for the Facebook website.
Informational: when the searcher is researching something and seeks answers, information or resources. An example of an informational search could be “Dollars to pounds exchange rate?”.
Transactional: when the searcher is seeking to do something – whether that be to buy something or to carry out some other kind of action An example of a transactional search could be “buy 4K TV”.

It is generally not possible to measure intent exactly, but rather we must seek to infer it and understand more about the kind of thing the searcher is seeking. Our ultimate goal, of course, is to drive profitable user action by aligning what the website is offering with the user’s desires.

Branded vs. Unbranded Search

In all three of the categories identified above, it can be useful to classify searches additionally as branded or unbranded. Branded searches are those that mention or seek a specific brand or website.

Mentions are simple to understand – a search for “Instagram” for example has straight-forward intent. You may choose to count certain other kinds of searches as branded – for example those mentioning specific unique product names, or other identifiers.

Let’s look at examples of each kind of keyword in both branded and unbranded form.

Branded examples:

Branded navigational searches are those like “Instagram” which are just shortcuts for typing “facebook.com” and also searches like “Sky Sports” which are seeking specific pages
Branded informational searches are those like “bbc chicken pie recipe” which are seeking a general thing from a specific site or brand
Branded transactional searches are those like “Samsung S9 Phone” which are precursors to a transaction targeted at a specific site or brand

Unbranded examples:

Unbranded navigational are the rarest and typically exist only where a fair degree of consumer education has taken place such that a generic term is actually associated with a specific brand. Examples include “app store” which would generally mean “apple app store” (even though other companies like Amazon have app stores). In Amazon’s favor, some searches like those for specific books are almost unbranded navigational searches for Amazon because searchers are so confident of finding Amazon at #1 or worst case above the fold
Unbranded informational searches are those like “Ireland Presidential Election Results” that don’t seek to return only pages from a particular site or brand
Unbranded transactional searches are typically very valuable rankings – searches like “Professional SEO Services” would fall into this category

Identifying Keyword Types

Although in general, we need to remember that a given keyword can mean different things to different searchers (think about all the different potential meanings for a search for “monopoly” for example), we know our business and it is unusual for us to find keywords that our audience uses in multiple different ways.

So step one of any identification process is to put ourselves in the shoes of our audience (or potential audience) performing the search.

It is generally then – assuming we know our business well – easy to assign any given keyword into one of the six buckets outlined above (branded or unbranded, navigational, informational or transactional). What is harder, is making that process scale to dozens, hundreds, or thousands of keywords.

To do this effectively, we must find rules of thumb and heuristics to help us. Typically these rules take the form of Excel or Google Sheets formulae.

Here are some of the rules and heuristics that we have found useful:

Word or stem matching – typically most useful for branded / non-branded analysis
Look-ups – for example against product lists
Filters on keyphrase length – to make manual review possible by making lists a manageable size – and noting that longer searches tend to be informational (and, to a lesser degree, transactional)

Aligning Website Content to Search Intent

Beyond identifying new opportunities, a key goal of keyword research is to match your website content with the intent of the searcher. This is important for a number of reasons:

It’s the key to profitable outcomes since you need users who do reach your site to stay, engage with your content and ultimately convert
It’s critical for long-term search success because if your content is not aligned with searcher intent, that will be reflected in poor user metrics, high bounce rates from search, and lower levels of link growth

You can identify intent mismatches by seeking pages and sections with high bounce rates from organic search in analytics.

Of course, it’s impossible to control all the things that a page ranks for, so sometimes you will find high bounce rates where pages rank for an accidental term. Outside of those situations, where you find bounce rates indicating a mismatch, you can use the framework above to modify page content to suit.

The most common mismatch we see is websites that put commercial, conversion-oriented content in front of searchers performing informational queries. Of course, you need to consider the profitable outcomes you are seeking, but if users don’t even stay around to consume any content, you have no chance of converting them. In these situations, you may need to consider smaller conversion steps such as email subscription sign-ups.

Keyword Brainstorming

You can get a long way by bringing together people who understand the underlying business with people who are used to thinking about searcher behavior. You can do this face-to-face or virtually, but the key is to ask good questions. For example, for a B2B website, think about questions like:

How would our customers describe our business?
What questions would they need to answer while making their decisions?
When earlier in the process – as they have identified problems but are not yet necessarily seeking your services as a solution, what issues might they be researching?

At this stage, you are not necessarily seeking to structure your output particularly carefully, but can just make an unordered list of ideas.

User Data Sources

As you begin to exhaust your creativity in brainstorming, you should turn to additional resources for ideas and evidence of how your customers and potential customers think about your services.

Useful insights may be found from:

Support documentation
Inbound support inquiries
Frequently Asked Questions (FAQs)
Site searches performed on your existing website
Comments and other user-generated content
Speaking to product, sales and support staff

And any other places you can find website users and customers writing in their own words.

Exploring and Finding Keywords

Before working with the list you are building, you will want to gather data from a variety of sources to identify edge cases and also to enhance your master list with whatever other data you can find.

Keyword Research Tools and Techniques

Now that you have a seed list of keywords, you need to expand them into a list that is as comprehensive as possible. Below are a few techniques that you can learn to do this.

Google Suggest

Google Suggest can be a great way to expand your seed keyword list and provoke more ideas for you to explore. You won’t be given search volume for the keywords but Google Suggest is largely powered by user intent which means that it gives you a good indicator of what people search for.

Ubersuggest

Ubersuggest was created by famous internet marketer Neil Patel and has expanded it’s functionality out to give keywords and rough volumes. Ubersuggest basically scrapes Google Suggest which means you don’t have to manually enter each possible keyword and note down the results. You can start with a simple search and get a long list of suggestions.

Why You Should Never Second Guess Neil Patel! — Neil Patel

You can select your target language and country as well as the type of results you want such as web, news, images etc. You could make use of these because you can often get very different results when selecting different types of results.

You only need to run a few variations of searches on Ubersuggest to get a few thousand suggestions very quickly. Once you’ve copied all of these into your spreadsheet, you will need to de-dupe (remove duplicates) them as there will be crossover sometimes.

Gathering additional data

In order to help prioritise your activities, it is valuable to expand your raw keyword list with additional data. You might seek data like:

Search volume
Geographic differences in searcher behavior
Seasonality of search volume through the year

Before you start using other tools to add this extra data, you should make sure that your master list of keywords is tidy and free from duplicates.

Keyword Search Volume and Seasonality

To gather search volume and seasonality data for your list of keywords, you can use the Google AdWords keyword planner tool.

You can see data for the following:

Competition – an estimate based on number of other people bidding on these keywords using AdWords. It is NOT based on organic search competition.
Suggested bid – this is an estimate of how much you’d have to pay in order to appear on page 1 for this keyword if you were using AdWords. Again, this isn’t related to organic search but it can be an indicator of how valuable a keyword is i.e. the more someone is willing to bid on it, the more likely it is to be a well-converting keyword.
Ad impression share – this only applies if you’re running AdWords campaigns already and will show you opportunities for bidding on more keywords or bidding higher on keywords.

These data points are somewhat useful in determining competition but you need to bear in mind that it’s purely competition from an AdWords perspective, not an organic search perspective.

You can then download all of this data back into a new CSV file so that you have your core list of keywords but with lots of additional bits of useful data.

Again, this process is a lot easier when using a paid tool like SEMRush, like we do at SEO Services NI.

Geographic Differences in Keyword Search Volume

Many keyword tools also allows you to define a geographic location, like Ubersuggest and SEMRush so that you can see how search volume differs by country:

This can be really useful for a couple of reasons:

If you’re thinking of expanding into new countries and want to get an idea of which one may have the most potential for traffic, you can get the data to show it
You can get an idea of subtle differences in terminology used in different locations

At the end of this process, you should have a list of relevant keywords along with search volume and you should have a strong idea of their seasonality and geographical differences.

Keyword Themes and Concepts

We already know the importance of keyword classification based on searcher intent. Once we are armed with an actual keyword list, we can also slice this data in other ways. One key way that helps integrate your keyword data into a publishing schedule or content calendar is to consider themes and concepts.

Defining Keyword Themes and Concepts

While the exact terminology isn’t important, it is important to separate two different ways of grouping keywords:

Syntactic groupings: this is what we would consider a keyword theme – grouping keywords by their shared language i.e. phrases containing the same words
Semantic relationships: this is what we would consider a keyword concept – grouping keywords by shared intent i.e. different ways of phrasing the same thing

Techniques for Grouping Keywords

You can define groups “top down” by creating the groups based on external information such as website sections or folders or “bottom up” by seeking emergent themes in the data.

Either way, you can start to define rules for classifying your keywords – typically either based on lookup relationships to words in the query or by using intermediate lookups that first map words to synonyms and then use lookups to perform the classification.

You can choose how narrow or broad to make your themes depending on the use to which you are putting the data. To guide a website’s information architecture, you may seek fewer, broader categories while a content strategy or calendar may need more fine detail.

Analysing Competitor Keyword Information

Competitor information serves two main purposes during keyword research:

Finding gaps or opportunities that you have failed to uncover via any other route
Determining the level of competition across a market in a given topic area

Understanding Competitor Keyword Targeting

At a high level, you can often determine major keyword groupings simply by looking at a competitor’s menu structure and major categories. To dive deeper, you will likely need to gather more data – normally through crawling the site in question or through external data sources like SEO Quake, Scrapebox, Sitebulb or Screaming Frog. The kinds of data that you can seek out include:

Number of pages in each category
Number of pages by keyword theme
Successful pages in category / by theme based on:
- Engagement metrics (comments etc.)
- External popularity metrics (links, social shares)
Analysis of copy on key pages to extract common words, phrases and theme

It is valuable to perform this kind of analysis both for real-world competitors and for those sites that appear in the search results for words and phrases you are targeting.

For example, it is typically worth checking out:

Media sites that cover your industry
Government or public sector sites that provide information to your customers
Companies with different business models that serve similar segments – such as affiliates, aggregators etc.

Below are a few techniques for finding keywords that your competitors are targeting.

Crawling Competitor Websites For Keyword Information

You can use a crawler such as Screaming Frog to find on-page keyword targeting from elements of the page such as META data and header tags.

You can export all of these titles into a spreadsheet and look for the most used keywords and their frequency.

Using third party tools

There are tools available that can give you insight into the keywords that competitors rank well for. The two most common ones are Searchmetrics and SEMrush. These tools gather a lot of ranking data for a wide range of keywords which means that you can see a sample of keywords that your competitor is ranking for.

This can be useful in finding competitors who you may not have previously considered.

Determining Levels of Keyword Competition

There are two ways of looking at competition levels:

Publishing volume and amount of quality content on a topic
Strength and authority of the sites that are ranking for the words and themes you’d like to target

You can often get a good sense of both of these areas by simply searching around the themes and topics you are targeting, but there are a range of tools that are useful. For example SEMrush gives an approximate measure of the percentage of traffic a website is getting for a given keyword, this information can be very useful for analysing competitor performance.

You can use the data you gather from this research, combined with an understanding of the authority of your own website to inform your own content strategy. In particular, if you find yourself in an area of high competition and with your own website being lower authority than the established players, you may find it more effective to target the gaps and seek areas that the bigger players haven’t targeted.

How to Determine the Value a Keyword

In order to work out how much value might accrue to your business from targeting a keyword or theme, you need to combine three kinds of estimation:

The average value of an individual visitor from a particular keyword / theme
The number of people performing the searches you would like to target
What kind of performance you believe you can achieve in capturing those searchers

How many people search for any given keyword?

Historically, search volume data has come from two main sources:

Directly from search engines themselves
From ISPs and plugins who sell anonymised data about searches traversing their network or terminating on websites containing their code

The ISP data has gradually gone away and the remaining data that comes from the search engines themselves comes with its own downsides:

Smaller search engines don’t necessarily have representative data
Google doesn’t always have the incentives to share transparent data

The data we get from Google is important and the best we can really do is take the information we get from paid tools, understanding that they are not always accurate.

What kind of performance can you expect?

After factoring in the competitiveness of the keywords and the strength of your own site, you can begin to build models for the potential performance you might see when you build out content targeting the phrases you’ve chosen. This is not an exact science, of course, but by building the model, you can understand the sensitivity to the different assumptions you make along the way.

Once you have a performance model, you can layer on top an estimate of clickthrough rates to build a model for the search traffic that you may be able to capture. As Google’s user interface becomes increasingly complicated, with rich snippets, answer boxes, knowledge graphs etc., these steps are also getting harder. If you have an existing website covering some of the areas you plan to target, you can sense-check your calculations by comparing the output to the organic search traffic that existing pages are already capturing.

Keyword Research and Website Architecture

When you are planning the structure of a new website or seeking to improve the structure of an existing website, you are typically trying to understand the kinds of thing that your visitors are going to be looking for. This is where the themes and categories discovered through wide-ranging keyword research can come in useful.

By targeting themes in this way, you not only create sensible classifications for the content you have or that you want, but you are also creating user-centric paths to discovering the content that your users really want.

Content Strategy

There are a whole bunch of inputs that are useful in determining the kind of content that you choose to invest in and create, but demand is a key element.

Of course, you will also want to consider brand messaging, the competitive landscape, and your ability to build an audience or drive engagement, but by using your keyword research, you ensure that you are building realistic models and that you are aligning your strategy with the opportunities in the market.

Keyword Targeting

Of course one of the primary uses of keyword research data is to guide the tactical creation of individual pieces of content. Typically, once you have chosen the subject and the target keyword, it is straight-forward to integrate the specific words and phrases that your demand research has identified (if it’s not, that typically points to a mismatch in your content targeting).

Conclusion

Keyword research can be a daunting and time consuming process. That’s why most companies have dedicated in-house teams for performing these types of marketing activities or they employ outside professional SEO services. If you are planning to do it yourself, we really recommend using SEMRush as it’s user interface is the most user friendly, the data is excellent and the price is very reasonable for what you get. Plus they are regularly updating the features of their service. Check out the free 7-day pro trial here.