How Search Engines Works: Crawling, Indexing and Ranking
The basic functions of search engines for producing a meaningful search result is the following.
What is crawling in SEO?
Crawling is the process where Google discovers a page in your website. Google use web crawlers often called spiders for discovering web pages and updating their index on a regular basis.
Google have allotted every website a crawl budget, which essentially determines how often and how much pages should spiders crawl for that website. Google also find new websites from the external links pointing to those websites in the web pages which Google crawls.
The crawlers does not render pages which they crawl, instead it analyses the source code of the pages using a parsing module. Crawlers are able to validate html code and hyperlinks.
When you are browsing the Internet using Google, you are not actually able to search the entire Internet. You are only able to search the Google’s index of the web.
Search engine spiders or Googlebot (crawler of Google) conduct the crawling process. Whereas, the indexing is done by Google’s search engine algorithm named as caffeine.
What is indexing?
Indexing refers to the process of analyzing the contents and relevance of each crawled web page and adding that page into Google’s index of web pages which are eligible to feature in search results. In simple terms, web pages which are not indexed by Google will not appear in search results.
Unlike crawlers the indexation process involves rendering the page using web rendering service (WRS). Google’s Webmaster tool (currently known as search console) show you how Google renders your web page, using the Fetch and Render function.
The crawling and indexing process often works hand in hand.
Crawlers find pages and send it to the indexer. After analyzing the pages sent by crawler, the indexer feeds new urls found on that page back to the crawler, so that crawlers can prioritize the URLs found on a page based on their value.
Only pages that are crawled and indexed are eligible to be ranked and appear in search results.
How Google Ranks Web Pages?
Ranking process begins when a user makes a search. To serve the most useful and relevant search results, search engines must perform some calculations and retrieve the best possible results from their index of web pages.
The ranking process involves the below three critical steps.
- Analyzing the intent and context of the search query
- Identifying the web pages that are eligible to appear for the search result
- Ordering the eligible web pages based on their importance and relevance to the search query.
In 2009, Google had publicly stated (in this official article) that AJAX applications are difficult to process by the search engine as AJAX content is dynamically created by the browser and hence not visible to crawlers.
Serving a pre-rendered page from server
- Angular JS SEO
- Backbone JS SEO
Contributed by https://www.globalmediainsight.com/