How to Index a Website on Google? [Expert Guide for Businesses]

Are you having trouble indexing your website on Google? Perhaps a specific set of pages is not indexed?

Most of the time when you find out that your website is not showing on Google, it actually does. The problem is that pages are not ranking for the terms you are searching for. The best way to check this is by using a simple command in Google:

Submitting this command to Google shows if your website is indexed or not. You will also be able to see what set of pages is indexed, such as product categories, blog posts, pages, and others. This way you can take care of the pages you don’t want to be indexed and block them from indexing on Google. Do this also if you want to find any leaking pages that can affect your website security, such as admin login pages and other website files, which may get indexed against your will.

But, what happens when your website is not showing when you try this command?

Well, there are multiple reasons for this. In this article, we will explain the reasons and give solutions to make sure your website is indexed completely in the best possible way.

1. Your Website is in the Development Mode

Most of the time when you are working on your website, the maintenance mode will be set. This mode will discourage your website from indexing by Google. It will block all access to your pages and display only one page which is the homepage. The maintenance mode will contain part of the code in head tags which will tell Google to skip your website and come back after some time. Now, this is a good practice and you should always make sure to block your website while it’s in the developing phase. If you skip this step, you may end up with a big number of spam pages indexed on Google which will affect your rankings later on.

If you are using WordPress for your website, make sure that you go to Settings > Reading >  Enable box for Search Engine Visibility.

Sometimes, during your website development stage, you may forget to uncheck this box. This will block Google from indexing your website. If you are using other platforms such as Shopify, make sure to remove the no-index tag from your pages.

2. Start With Google Search Console

The best way to test your website performance and crawling ability is to see it first-hand on the Google Search Console. Simply register at Google Webmaster Tools and verify your domain by following simple steps from Google. Once you claim your ownership, you will be able to see the indexed pages on your website, as well as if there are any issues with crawling or any reports from Google regarding your website.

A quick way to get your pages indexed is to create a sitemap file, configure robots.txt file, and then fetch your entire website.

Sitemap.xml

This is a file that can be found by adding /sitemap.xml after your domain name. Example sitemap can be seen here: Persist Digital Sitemap 
This file will help Google better understand your website and what pages should be crawled. For example, you can create settings for Google to crawl your posts every few days. This way, whenever you write a new blog post, it will be indexed very quickly. Sitemaps are a great way to organize your pages. They are also very useful if you have a website with thousands of pages. Google may face difficulties trying to index thousands of your pages. Sometimes, pages are hard to reach, more than 3 clicks away, which means Google will probably skip on indexing them. By adding these pages to a well-organized sitemap, you will make sure that Google sees them and shows them in results.

Robots.txt

Similar to sitemap, this file can be found by adding /robots.txt after your domain name. Example robots file can be seen here: Persist Digital Robots
This file is a great way to block pages you don’t want to see in Google. Also, sometimes, this file is configured to block all pages on your website from the very start. If you see the command line in robots.txt file which starts with: Disallow: /  — this means that robots are not allowing any bot to crawl your website, and can’t index any pages. To learn more about robots and what commands you can use, visit Robots Website.

Google Fetching

This is a feature inside of Google Search Console. Recently, it was migrated into the new tool and it is not as it used to be. Still, you can use it to index pages of your website in just a few hours. Simply add all pages you want to be indexed here and after a few hours, they will be up. If there is something that is blocking pages to get indexed, you will get a notification about the issue and instruction on how to fix it.

3. Google Removed Your Website From Its Search Results

Depending on the activities on your website, in terms of SEO (Search Engine Optimization) and website performance, Google may penalize your website. Make sure to follow their guidelines. To make sure you’re doing everything by the book,  you will need to log into your search console. Simply navigate to the old version of the Google Webmasters Tool and navigate to Manual Actions.

If you believe your actions on the website did not follow Google webmasters guidelines, you can visit this link and find out if there were any manual actions taken against your website. In case Google believes your website does not meet its standards, it can remove your website from search results. Your website can be penalized when a website domain still exists but pages cannot be found. The website can be deindexed when a domain is completely removed from Google. If your website is sandboxed, it can come to a significant drop in traffic.

How to fix it

If Google penalized, de-indexed, or sandboxed your website for security reasons or simply because it violates its quality guidelines, modify your website and submit it again so Google can reconsider it. This will take some time, so make sure you do everything following their guidelines. Remember the goal is to provide the best user experience you can for visitors.

4. Check for 301 & 302 Redirects

This is a rare case scenario, but it can happen if you go through your pages and see some of the pages are not indexed on Google. Google will miss these pages when indexing and will value the page you applied redirection to as it will be pointed to that particular page. You should always be careful when doing redirects. Too many redirects can cause major issues with indexing your website. The more redirects you have, the more loops will be active. These loops may confuse Google when crawling your website. You should never have more than 2 to 3 consistent redirects. To put this into practice, let’s say we did 301 redirects like in the example that follows.

• persistdigital.com/1 | 301 redirect to
• persistdigital.com/2 | 301 redirect to
• persistdigital.com/3 | 301 redirect to
• persistdigital.com/4 | end of loop

When Google tries to crawl your first page, due to many redirects (to be exact, 3 redirects), Google may actually skip them and don’t go into the entire loop, which will result with not indexing your pages. To fix this, make sure you never have more than 2 redirects in the loop.

5. Your Website is Hacked

If you don’t have a secured website and use weak passwords, your website can be compromised. Hackers can play with your redirects, pages, indexing, and cause serious damage to your website. To check if your website is hacked, use this free scanning tool called Virus Total. Make sure your website is always green during scans. Also, regularly update plugins and perform website backup procedures. Having a website hacked will show to Google that it is not safe for use. This will result in removing your website from search results until the issue is resolved.

6. Google Could be Mistaken

Very rarely, Google can by accident remove your entire website. This happened close to a year ago with Search Engine Land. If you took all the steps and did everything we mentioned, but Google still removed your website, it may be best to give it some time and simply wait for a few days to see if there are any changes.

7. Your Website is Using a No-Index Tag

Even if you removed the no-index attribute from the Yoast plugin on WordPress or other platforms, you may still end up with custom no-index tags. Development work on the website may be the reason why a specific set of pages is not indexed. The best way to check this is by applying a free scan from SEMRush.com. The scan will tell you how many no-index pages there are on your website at the moment and it will be easier for you to fix the issue.

Conclusion and Takeaways

• While your website is in the development mode, make sure you don’t allow Google to crawl your website. The best way to do this is to use free plugins and apps that will block search engines from crawling your website. Do this to avoid indexing spam pages in the very beginning.
• Test your website by using a simple command site:websitename.com on Google to see what pages are indexed.
• Once the development mode is finished, start with Google Search Console. Configure sitemap.xml, robots.txt files and fetch your entire website for indexing.
• Check if there were any manual actions taken against your website by checking Google Search Console.
• If there is no progress and you still don’t see your pages, scan your website for viruses and run SEO audit using SEMRush. Make sure your website is both safe and clean in terms of technical SEO.
• Give it a few days or weeks after you applied everything from the list. By then you should see your website indexed on Google.