How to Monitor 404 Errors

404 errors happen when someone requests a page or file that doesn’t exist. Actually, it’s more accurate to say the webserver can’t find the file.

404 errors can happen when:

1. Someone types the wrong address
2. Someone links to the wrong address
3. The file or page moves to a different address
4. The page or file is deleted
5. The page or file never existed

404 errors might sound scary, but they are mostly harmless. It is easy to monitor 404 errors and fix them, thanks to Rank Math.

1 What is a 404 Error?

When you visit a website through a link, or by typing in an address, you aim to find and consume content. Instead, if you’re hit with a 404 error, not only will you not be able to consume the content, but your experience will also be less than satisfactory.

It doesn’t matter if it’s your fault, the website owner’s fault, or someone else’s. Maybe you clicked on a broken link caused by a typing mistake on another website. Maybe you cut-and-pasted a URL wrong. Or, you possibly made a spelling mistake when you typed a URL manually.

404 errors are so common that it is unlikely that you haven’t experienced them. And, as any Internet user will tell you, they are definitely an inconvenience.

When you operate your website, you make a lot of effort to give your users an excellent experience. You create amazing content, you use the best WordPress themes, you design your content, you make your website as fast as possible, and many other things. All that effort can go to waste if your user can’t even reach your content.

You might think 404 errors only affect a very small percentage of your audience, and you will eventually fix them—so there is no need to be on top of them all the time. But you’d be wrong. What if a large publication found your content useful and decided to link to you, sending you thousands of visitors for years to come, but made a mistake while linking to you?

From your end, nothing would be wrong. You might see better traffic, but all those visitors will quickly bounce off your website—costing you thousands of subscribers, followers, and lifelong fans. Not to mention the wasted SEO value of the link you received.

Fortunately, Rank Math’s 404 Monitor can save you from all sorts of SEO trouble. Below, we will show you how to set up and monitor your 404 errors using Rank Math. Let’s check it out.

2 How to Monitor 404 Errors Using Rank Math

2.1 Enable the 404 Monitor Module

The first step to monitoring 404 errors is to enable the Rank Math 404 Monitor module. From your WordPress dashboard, head over to Rank Math SEO → Dashboard and enable the 404 Monitor module, as shown below.

Note: You need to be in Advanced Mode to enable the 404 Monitor module from Rank Math’s dashboard.

2.2 Navigate to the 404 Monitor

Once you have enabled the 404 Monitor module, head over to Rank Math SEO → 404 Monitor from your WordPress dashboard. 

You will be presented with your 404 Monitor.

When a visitor encounters a 404 error on your site, the details of the error will be recorded and presented to you on this page. In the below image, notice that it says, “The 404 error log is empty”. Well, of course, it is because we keep our site clean.

An empty log is great, but it doesn’t show you much. So, for this example, we’ll introduce a few errors. After introducing the errors, here is how the 404 Monitor will look.

Let’s understand the data that Rank Math is sharing with us. If you notice, there are 3 fields in the 404 Monitor.

• URI
• Hits
• Access Time

These fields help you to identify the source of your 404 errors and how quickly you need to solve them.

2.2.1 URI

The URI (Uniform Resource Identifier) represents the URL that the user tried to access. The only purpose of this field is to see if the error is caused by an obvious typo. If it is, you can fix it and not think about it anymore.

2.2.2 Hits

Hits represent the number of times an error-causing URI was accessed. This is an important number as you would want to fix errors with the most number of hits first.

2.2.3 Access Time

The Access Time field captures the time of the last hit on the error-causing URL. This detail, combined with the detail presented in the Hits field, is pretty useful in solving issues.

3 How to Use the 404 Monitor Information

Using the 404 Monitor information is quite straightforward. By viewing the URI and the number of hits, you can prioritize the URIs with the most errors and start fixing them by following the instructions in Rank Math’s guide on fixing 404 errors.

But, there is an important limitation with this data. This data does tell you about the URL with the error, but it does not tell you if the error originated from your website or an external website.

The distinction is important. You can fix errors on your own website quickly and easily, but if the cause of the error is on another website, then your potential solutions to the problem are limited.

But first, you have to find the source of the problem, for which, you can use the Advanced 404 Monitor available in Rank Math.

4 Advanced 404 Monitor

To activate the Advanced 404 Monitor, head over to Rank Math SEO → General Settings → 404 Monitor from your WordPress dashboard. Once done, navigate to Mode and select Advanced, as shown below.

Before getting into what the Advanced 404 Monitor is, let us give you some basic guidelines on how to use and how not to use the 404 Monitor.

The Advanced 404 Monitor captures a lot more details about your 404 errors. This will help you to better identify the source of your errors. But it will also increase the number of errors displayed in the 404 Monitor.

With the Simple 404 Monitor, errors are grouped by their hit count. What that means is that a single error that occurs a hundred times will show up in the logs as a single error with a hit count of 100.

The Advanced 404 Monitor is different. Due to the nature of the information it captures, errors, even on the same URL, are captured as individual errors. But there is a catch, and you can still isolate the grouped errors, and we’ll show you how to do that later in this article.

If you have tons of errors on your website, then that can become a problem. Our recommendation is that you only use the Advanced 404 Monitor to fix 404 errors that you aren’t able to fix with the Simple 404 Monitor.

Now that you understand, let us see what the Advanced 404 Monitor actually does and how it is different from the Simple 404 Monitor.

4.1 Difference Between the Simple and Advanced 404 Monitors

If you enable the Advanced 404 Monitor, two new fields: Referer and User-Agent, will be included in your 404 Monitor, as shown below.

Obviously, the new fields will be empty for the previously captured errors, as the Advanced 404 Monitor was not enabled at the time they were logged. To explain the importance of the two fields, let us introduce a few errors again.

Here is how the 404 Monitor looks after we introduced some new errors. In the below image, you will notice that the Referer and User-Agent fields are now populated.

With that said, allow us to explain the importance of the data in the Referer and User-Agents fields and why you may need them.

4.2 Referer in Advanced 404 Monitor

This Referer field contains the URL that led visitors to your 404 page. This field’s information is crucial when a bad internal or external URL causes a 404 error. For example, if another website links to an important post on your website but with an error, this field will help you identify the source of the error. Using that data, you can contact the site linking to you and ask them to correct their link.

In some of the errors, we simply typed in an incorrect address in the address bar, causing the referer field to be empty. But, for the last example, we created a post and added a broken link. Once we clicked the link, Rank Math captured the URL of the post in the referer field, as you can see in the image.

It’s important to know that the referer link can be an internal link or an external link. That means that if you make a mistake in linking internally to your post, Rank Math will capture that for you as well.

The Hits column indicates the number of times an unavailable URL on your website has been accessed. And, there is more to it — the 404 errors grouped in the Advanced 404 Monitor can be accessed by simply clicking the Hits count.

All the 404 errors resulting from the specific URI will be filtered. This makes it easy for you to analyze the referer and user agent for an unavailable URI, especially when your 404 error log includes hundreds or even thousands of 404 errors.

4.3 User-Agent in Advanced 404 Monitor

Wikipedia defines a user agent as, “In computing, a user agent is a software (a software agent) that is acting on behalf of a user, such as a web browser that retrieves, renders and facilitates end-user interaction with Web content. An email reader is a mail user agent.”

In this context, the User-Agent field contains the details of the browser your visitor used to access the 404 page. This is extremely powerful information for many reasons, and here is an example.

If you have or know about affiliate sites, you’ll know that they spend a lot of time creating and managing links. Many affiliate networks offer scripts that can convert regular URLs to affiliate URLs. Imagine a scenario where a script has an issue with a particular browser which causes the URL to throw an error. Without the User-Agent data provided by Rank Math, you’ll keep on trying to reproduce the error but will fail. Only when you see the User-Agent data will you be able to understand the issue.

Note: The Advanced 404 Monitor collects only the details of the user-agent, such as the browser and operating system, and no personally identifiable information of the end-user (such as IP address) is stored.

With that said, let us walk you through the 404 Monitor settings available in Rank Math and how you can use them to decide how your 404 errors are identified and logged. 

5 404 Monitor Settings

The 404 Monitor settings allow you to decide what should be considered a 404 error and how the error details should be logged.

To access the 404 Monitor settings, navigate to Rank Math SEO → General Settings → 404 Monitor from your WordPress dashboard.

Your 404 Monitor settings will be presented to you, as shown below.

Now, we will go through the available options one after another.

5.1 Mode

The Mode setting presents you with two options, Simple and Advanced.

We have previously explained both options, but we will touch them again briefly.

5.1.1 Simple

If you select Simple, your 404 Monitor will display three pieces of information:

• URI -The link to the webpage your visitor encountered the 404 error.
• Hits – How many times your visitors accessed the 404 webpage.
• Access Time – The last time a visitor accessed the 404 webpage.

5.1.2 Advanced

When you select Advanced, your 404 Monitor will display the details available in the Simple 404 Monitor along with two new details:

• Referer – The link to the webpage that directed visitors to your 404 webpage.
• User-Agent – The browser the visitor used to access your 404 webpage.

Note: While the Advanced 404 Monitor provides detailed information about a 404 error, it can quickly fill your 404 Monitor. You may only want to enable it when investigating the source of a 404 error.

5.2 Log Limit

The Log Limit setting allows you to set the maximum number of 404 errors displayed in the 404 Monitor.

The Log Limit is set to 100 by default, but you can increase or decrease it as required. You can also disable it by setting it to zero 0.

Note: When the number of errors logged in the 404 Monitor exceedes the number set in the Log Limit, the 404 Monitor will be automatically cleared and begin counting afresh.

5.3 Exclude Paths

The Exclude Paths setting allows you to prevent a 404 error from being logged in the 404 Monitor.

The Exclude Paths setting contains a field and dropdown menu you will use to specify the condition a URI must meet in order to avoid being logged in the 404 Monitor. We will discuss the Exclude Paths field and dropdown in more detail below. 

5.3.1 Exclude Paths Field

In the Exclude Paths field, you will enter the URI or keyword you want to exclude from being logged.

Note: You should ensure the keyword or URI does not begin with a forward slash /, else the setting will not work as expected. 

5.3.2 Exclude Paths Dropdown Menu

In the Exclude Paths dropdown menu, you will select the condition the URI or keyword you entered in the Exclude Paths field must meet in order to be excluded from being logged.

The Exclude Paths dropdown menu contains five options, as listed below.

• Exact
• Contains
• Starts With
• End With
• Regex

We will now explain these options in detail.

a Exact

When the Exact option is selected, a 404 error will not be logged when a visitor encounters a 404 error on the URI listed in the Exclude Paths field.

b Contains

When the Contains option is selected, a 404 error will not be logged when a visitor encounters a 404 error on a URI that contains the keyword or URI listed in the Exclude Paths field. For example, if the Exclude Paths field is set to rank-in, a 404 error encountered at https://yourdomain.com/how-to-rank-in-search will not be logged. 

c Starts With

When the Starts With option is selected, a 404 error will not be logged when a visitor encounters a 404 error on a URI that begins with the keyword or URI listed in the Exclude Paths field. For example, if the Exclude Paths field is set to best-seo, a 404 error encountered at https://yourdomain.com/best-seo-tools will not be logged.

d End With

When the End With option is selected, a 404 error will not be logged when a visitor encounters a 404 error on a URI that ends with the keyword or URI listed in the Exact Paths field. For example, if the Exclude Paths field is set to seo-tools, a 404 error encountered at https://yourdomain.com/best-seo-tools will not be logged.

e Regex

The Regex option allows you to create short, powerful rules that will exclude the 404 Monitor from logging many different kinds of 404 errors. A regex rule requires advanced knowledge of WordPress to create, and doing it wrongly may cause you to miss important 404 errors. You should contact a WordPress expert to create your regex rules.

5.4 Ignore Query Parameters

A query parameter is the part of a URL that helps web admins and visitors to identify specific content on a webpage.

For example, a product URL at https://yourdomain.com/t-shirt may lead to a t-shirt that is available in two colors: white and black. When the visitor selects the white shirt, the URL will change to https://yourdomain.com/t-shirt?color=white. When they choose the black one, the URL will change to https://yourdomain.com/t-shirt?color=black.

If you enable the Ignore Query Parameters option, the 404 Monitor will log the 404 URI without the query parameter. So a 404 error encountered at /t-shirt?color=white will be logged as /t-shirt.

You can enable the Ignore Query Parameters option by clicking the toggle button, as shown below.

5.5 Save Changes

When you make any changes to the 404 Monitor settings, scroll down and click Save Changes to ensure your changes are saved.

6 Fixing 404 Errors

Once you have the list of 404 errors, the next thing to do is to fix the errors. There are many actions you can take to fix the errors, and we’ve created a dedicated tutorial for it. Be sure to check it out to understand how to solve 404 errors.

7 Exporting 404 Errors PRO

The 404 errors that you’ve fixed are only for now. But new errors may be introduced in the future, and you may need to review and fix them periodically.

Though you can check the access time when the broken link was last accessed, the export option in Rank Math can help you generate a 404 error report easily for the chosen timeframe. This feature is available only in the Rank Math PRO version.

To export your 404 errors, return to the 404 Monitor at Rank Math SEO → 404 Monitor and click the Export option at the top of the page.

Once done, you will be presented with the option to specify the timeline of the logs to export.

If you want to export the entire log, you can ignore this option and click Export. Optionally, you can select a From date and To date to specify a timeline and then click Export, as shown below.

Now, the CSV file containing your 404 errors will be downloaded to your computer. You can open it with the spreadsheet editor of your choice and decide which errors need to be fixed.

8 Clearing the 404 Monitor

You may want to clear the 404 Monitor after you have fixed or exported the current errors. There are different ways to do this, depending on how many logs you want to delete.

If you want to delete the entire log, click Clear Log, as shown below.

Optionally, you can delete individual logs by hovering over the URI and clicking Delete, as shown below.

You can delete multiple logs by selecting the checkboxes of the individual URIs. Once done, scroll to the top of the page, select Delete in the dropdown menu, and click Apply, as shown below.

That’s all on monitoring 404 errors on your site. If you have any questions about monitoring 404 errors with Rank Math, feel free to reach our support team, and we’re always here to help.