What Search Engines Do and Why

What Search Engines Do

Written by Jeremy Earle, JD

September 16, 2022

How do search engines show web pages? What motivates them to do this, and how often do they do it? This article provides answers to all of these concerns and more.

There’s a new twist when it comes to indexing – and that’s rendering.

Generally, when we think about page ranking, we think of indexing.

We tend to conceive of the moment when a search engine discovers a page via sitemaps or crawling and then visits the page for indexing as the point in time when the search engine has indexed the page.

  • The page source was used to collect all of the material.
  • Started giving the page a search engine ranking.
  • Rankings have historically been based on this series of steps, making it the most critical.

On the other hand, is ‘t indexing always the end of the research process.

As time goes on, the index’s weight will decrease, while the last step, rendering, will likely take its place.

What’s the Difference Between Indexing and Rendering?

Indexing vs. rendering may be best explained by comparing the following two images:

Indexing is taking place here:

To put it another way,

What we’re looking at here is essentially the same stuff that would be indexed (HTML) and rendered (J.S.) (Chrome).

What’s the Point of All of This?

You may be surprised to learn just how crucial rendering is.

Even before the current effort to incorporate JavaScript on websites, search engines were rendering pages.

In a nutshell, rendering is important that it delivers the truth.

Thanks to the coding, a search engine can get a sense of what a page is about and what’s going on.

They may better understand the user experience and the importance of different types of information by rendering it.

Questions like:

  • Is there content buried behind a click? It may be answered using rendering.
  • Does an ad take up the whole page?
  • As a developer, do you have any questions about how the code is organized?
  • How long does it take to get to a page’s content?

Rendering provides the answers to all of them and many more.

The answers to these questions are critical to understanding a page and its ranking.

Do Renderings Occur at All?

During the year 2018, the rendering process took many weeks.

It’s no surprise that the process has sped up.

Seconds.

You’ll hear Google’s Martin Splitt address this same topic at about the 18:20 point in the audio below.

Exactly How Long Is It Before Google Displays a Pages

Within minutes, 90% of the indexed pages will have gone through the rendering queue at the middle speed of 5 seconds.

There has to be a distinction between this and rendering.

You may expect your website to begin rendering within 5 seconds if you fall into a medium set that starts within that time, but it may not finish within that time.

We’d consider it part of the “good side of the medium set” if rendering can begin in 4 seconds but take 30 seconds before it is finished.

In only two years, we’ve gone from weeks to seconds.

Bing has a unique way of working.

This is the same statement I got from their Web Ranking & Quality Project Manager, Frédéric Debut when I asked: I can confirm we’re striving to prioritize rendering for the URLs supplied via the API.

—Frederic Debut (@CoperniX) In the year 2020,

My tweet from September of last year was the “before.”

There are occasions when it might take days, weeks, or even months to get a response. Our decision to render a page or not depends on what we value most in doing so.

—Frederic Debut (@CoperniX) The date of September 3, 2019

I’m assuming they’ve also sped things up, but I’m unable to confirm this at this moment.

To sum up: “After indexing,” rendering takes place. This means that although search engines will grasp the content and context of the page before they fully understand how it should be prioritized, in most situations, this is a moot point since the lag is small.

Search Engine’s Evergreen Googlebot

Googlebot’s Web Rendering Service (W.R.S.) component was modified in May of this year, making a significant jump ahead.

Since then, the Web Rendering Service has been running Chrome version 41 on its servers.

In terms of interoperability, this worked wonderfully. In terms of current JavaScript capabilities, this was a disaster.

When the Web Rendering Service was updated to evergreen in May 2019, Chrome was used for rendering (within a couple of weeks at any rate).

Google now renders your website similar to how you would view it if you were to open it in a browser.

A Web Rendering Service’s Role in the Process

Before realizing it, I thought about something incorrectly and was trying to solve an issue that had stumped me for some time.

If you find my mental glitch noticeable, you’re allowed to mock me.

Here are some basic questions to start with: How does a web rendering server obtain the information it needs?

Here’s a general breakdown of the rendering process:

Sitemaps, crawlers, and other tools are used to find a page.

  • It will be included in a site’s crawl budget when it becomes accessible.
  • When a user visits a page, the content gets indexed.
  • The page is put in the queue for rendering.
  • In this example, the page has been rendered.
  • The rendering queue is a key yet unrecognized part of the process.

The engine will send a “headless browser” to the page whenever a page is ready to be rendered.

I struggled with this stage the most.

A browser without a graphical user interface is known as a headless browser.

For whatever reason, I was having trouble visualizing how it worked.

What good is it if the information isn’t shown visually for Google?

“The bot doesn’t have eyes either, so… hmm… yep,” is the obvious response to this question.

This “browser light” produces a page for search engines to comprehend what shows where and how on a page – even if they don’t have eyes for it.

For the most part, Googlebot will see the displayed version as it appears in graphical browsers. If this is not the case, the website may be using an unsupported functionality, such as a user permission request or an error in one of the scripts or other resources on the page.

Do You Know About Pre-Rendering Methodologies?

Pre-rendering is the process of sending cached HTML versions of JavaScript sites to search engines like Google, using a library like Puppeteer (which is the one I’m most acquainted with and hence given as an example).

If you’re going to employ cloaking, you will do it ethically. You’re going to provide search engines a DOM-based duplicate of the page to see the same content that a user sees when they stop by to index the page.

Since indexing and rendering now take much less time, it begs the question: do I still have to pre-render my JavaScript site?

To get an answer from Google, “Ask, and ye shall get.”

We’ve all been there, and it was here again.

To which the reply was: You don’t normally need it today.

To put it another way: As of August 2, 2020, John Mu (@JohnMu)

Puppeteer and other pre-rendering libraries will be pleased to hear this.

Pre-rendering systems have crashed without error indication in the past, which has caused a slew of issues for me (read: pages dropping from the index).

To avoid these issues, we don’t need to pre-render at all.

It’s obvious that “usually” was the keyword in this sentence.

The pre-rendering mechanism should be disabled on a few pages to examine what happens when they’re reached rather than completely disabling it.

Is Google able to read the material as it appears on the screen?

If this is the case, pre-rendering may be eliminated.

Is there a good reason to use rendering?

The question you may be asking yourself is, “Why do I need to render?” Why would Google need to render pages if a site doesn’t use JavaScript or other rendering technology?

An engine’s rendering may prioritize content according to human behavior on a page.

It provides information to the engine about how the material is shown in a browser and the visibility of various aspects, allowing the engine to evaluate and prioritize content and assess usability using the same product that a visitor would.

Rendered the Future

As a result of the shortened time lag and Mueller’s pre-rendering pronouncement, it’s clear that this investigation is doomed.

From an SEO standpoint, indexing, as we now understand it, will become functionally obsolete, with rendering assuming center stage in terms of online content discovery.

We’ve attempted to provide a general overview of rendering on this page. You may have a lot of questions after reading this. It should, too. I’ll point you in the direction of some useful sites to help you out.

You May Also Like…