Skip to content

Cumulative Layout Shift (CLS) and A/B Testing

You’d be forgiven if you are not aware of Core Web Vitals or why they are important if the trend graph above has anything to say about it. Seldom do I hear it spoken about outside of SEO circles, and never in the optimization context, which is something I wish to address with this post.

What are Core Web Vitals?

Nearly an year ago now Google released details on a new set of measurements it termed Core Web Vitals. These measurements sought to quantify the behavior of web pages so that it could result in improved user experience. Specifically, they sought to add insight around the Largest Concertful Paint (LCP), First Input Delay (FID) and Cumulative Layout Shift (CLS), which would report on loading performance, interactivity, and visual stability respectively.

The following month Google released a second blog post, detailing that it was investigating using the Core Web Vital measurements in it’s search ranking results. Due to the COVID pandemic, they deferred the rollout to 2021, and would offer at least a six month’s advance notice. They followed up with a post in November announcing that in May of 2021, the search ranking changes would finally include Core Web Vitals in the ranking position that sites could obtain.

At the time of writing, this means we’re at most 7 weeks out from the changes going in, and that means we’re running out of time to adjust our thinking for how our optimization programs should be modified. If we ignore the announcements, it becomes very possible this could ultimately result in a loss of SEO ranking for Google Search, which could translate into lost revenue.

What is Cumulative Layout Shift?

Of the three Core Web Vitals, – why did I decide to focus on this one for an Optimization discussion? Most client side testing platforms cause layout shift and designing the experience served in specific ways could result in increasing the CLS score (lower is better) which could result in a shift in rankings after the update rolls out this coming May. Simply – doing this right could result in better SEO placement.

Let’s look at how Google defines CLS:

Cumulative Layout Shift (CLS) – a Core Web Vitals metric, measures the instability of content by summing shift scores across layout shifts that don’t occur within 500ms of user input. It looks at how much visible content shifted in the viewport as well as the distance the elements impacted were shifted.

https://web.dev/optimize-cls/

So in an ideal world, developers would seek to improve the visual stability (and by extension the user experience) by optimizing for layout shift. Doing so can be very time consuming and involves an understanding of how the browser engine works as you are redesigning the page to avoid layout shift.

Depending on what is causing the shift, one or more other external systems may need to be modified. Google lists 5 conditions are common causes of poor CLS some of which are easier to address than others.

  1. Images without Dimensions
  2. Ads, or iFrames without dimensions
  3. Dynamically Injected Content (which is common for all client side testing platforms).
  4. Web Fonts
  5. Actions waiting on a network response before updating the DOM (page).

What does this mean for Optimization Programs?

Client Side testing tools are particularly subject to CLS scenarios by virtue of how they work. Let’s take a look at this with an example:

Let’s say you have Google Optimize installed, and you decide to load it via Google Tag Manager. You have a test which replaces the hero image with an taller image in effort to improve conversion.

Here’s what happens with Dynamic Injection:

  • The page loads, and starts to layout the DOM.
  • The page finds Google Tag Manager, and loads it.
  • GTM loads Optimize.
  • Optimize fires a Network call to make a decision.
  • Optimize gets the response and changes the page. It Dynamically injects the new hero image.
  • All elements below the hero image visually shift downward to accommodate the deeper image.

Here we have a non-user interaction (the script did it) moving many elements after the initial placement of DOM structure had been rendered resulting in layout shift.

Depending on the experiment and how much work was done to minimize layout shift – the results could end up being quite dramatic. In optimization circles you sometimes hear about ‘Flicker’ (the visual observation of content changing) which sometimes also causes the page reflow dramatically once the variant is inserted.

Now for if this will be sizable enough to cause a change in a given site’s specific rankings, it’s hard to say. The testing platform is just one aspect of the entire page and the page very well could have other CLS issues not at all related to the optimization program. With that said – there is no reason to tempt fate, and so I recommend being proactive in fighting CLS, even in a client side testing scenario.

Three things to consider when developing an experience

On today’s highly dynamic websites working around CLS can be difficult, particularly if your testing platform the system causing the shift. Some planning however can go into the test design to help minimize this being a factor.

Consider how you load the testing platform

Loading a client side platform from a tag manager is not advised if you care about page performance. You ideally should take the hit of loading it synchronously from the HTML because you want the decision (and variant html) as soon as possible in effort to minimize layout shifts.

Tag Managers often defer the test platform loading until DOM ready – which means the majority of the page is already laid out before it even begins to fire off the network calls for how it’ll modify the page. It’s akin to building a house – then remodeling the kitchen before you move in rather than changing the plains during construction. Both get you the kitchen you want, but one is way more sensible and doesn’t cause you to move in a month later than you expected.

Consider the Variant

Client Side testing platforms either dynamically inject, remove or change content. Ideally, you want to minimize variant experiences which cause dramatic reflow. These would be better ran at server side tests.

For what remains, the best choice is to replace existing content with something the same size so that reflow is minimal to non-existent. Injecting a banner where one wasn’t before, or removing a panel will result in larger shifts (depending on the DOM, sometimes notably so) and so should be done sparingly and with an eye on if the page is already near the CLS threshold.

Beware of Fonts

Testing web fonts is tricky because if the font isn’t a Websafe font, the browser has to download a copy of it before it can use it for rendering. Once the font is available – it may repaint the entire page (causing reflow if the characters are different sizes) resulting in layout shifting. This is commonly referred to as a flash of unstyled text.

Test fonts sparingly and ideally only swap between fonts of similar character width / height to minimize shift.

As pretty as a picture

Images done with dated responsive practices can result in sizable layout shift as the layout isn’t determined until after the image is downloaded by the browser. It’s recommended to use the newer aspect ratio CSS properties in order to have responsive images without the layout shift. You can read more about this practice on this excellent blog post by Craig Buckler.

The Road Ahead

Core Web Vitals are here to stay. Regardless of testing platform, you will need to consider them going forward lest your search ranking ends up suffering as a result. The layout shifting can be mitigated to a degree for client side platforms (particularly if you can also modify the site code to enable the platform to work better) with a bit of effort and a understanding of what needs to be done to minimize shift.

You can also consider swapping the client platform for a server side one and eliminate or reduce aspects that may work against you (like dynamic content injection). Such a undertaking is not for the faint of heart but may be a solution for for those seeking to lower the CLS scores as much as possible.

Published inA/B Testing