SEO is a very big, deep sea. To fully understand SEO, we should know some basic SEO terms.
Crawling and indexing are two such terms.
If you have been in the web world for a while, you’ve at least heard the words:
- Google Crawling and Indexing.
These are the two terms upon which the entire web world depends.
Let’s define, understand, and get some in-depth information about crawling and indexing.
- When Google visits your website for tracking purposes. This process is done by Google’s Spider crawler.
- After crawling has been done, the results get put onto Google’s index (i.e. web search).
What is Google Crawling?
Crawling basically means following a path.
In the SEO world, crawling means following your links and “crawling” around your website. When bots come to your website (any page), they follow other linked pages also on your website.
This is one reason why we create site maps, as they contain all of the links in our blog and Google’s bots can use them to look deeply into a website.
- The way we stop crawling certain parts of our site is by using the Robots.txt file.