Hiding in Plain Sight: How to affect what you can't see in your analytics

What if I told you there is a way to improve your Google rank (placement) and significantly increase conversion and that you would not be able to discover how to do it by looking at your analytics? Would you do it? Would it matter what the delta was or would it matter what work was involved?

Several years ago, I worked on a project at PayPal with the sole purpose of improving conversion during checkout. A few "features" were added, but the majority of the project was running A/B tests for (minor) design and content changes for the existing product, so the amount of actual development work was minimal aside from refactoring existing code.

One might think such a project would yield as much as one hundred basis points, but realistically, since these were "minor" changes it should have yielded about half that...but the thing is, it was twice that amount. The most reasonable explanation we might have for this remarkable difference is that we were affecting users that were not even on our radar.

Years before that project, as a side project I took over the website for a non-profit. Before running any A/B tests or modifying design or content in any way, I refactored the code to use semantic HTML and CSS and their rank went from fifth to first overnight. Nearly all the research up to that point indicated that selection of keywords and inbound and outbound links were the primary source for SEO, but here again, something hidden had a significant impact.

In both of these projects, it was the use of solid, time-tested best practices of web development that yielded these results, in particular, Semantic HTML and CSS, unobtrusive JavaScript, and Progressive Enhancement. It was what is ordinarily hidden from users and even those in decision-making roles in technology that had the greatest impact.

Frankly, it seems like an outlandish claim, and had I not seen them firsthand, I would suspect such claims were the usual puffery in which engineers engage - sort of like boosting performance by switching turbochargers. After all, we're often told that the best tools are the newest tools and that time-to-market is the most important factor. Abundant anecdotal evidence demonstrates that neither of those beliefs are based in reality.

Again, here are the four principles behind these transformations.

1. All HTML was Semantic HTML.
2. All CSS was Semantic CSS.
3. All JavaScript was unobtrusive.
4. Sites were built using Progressive Enhancement.

I should point out that it's possible to use any of these principles alone, and using any of them will give you some of the benefits. The goal when using these practices were to create pages that are

• fast,
• flexible, enabling a user to complete the task in any user agent,
• easily maintained

The primary goal in these projects was to get a page loaded in less than 4 seconds anywhere in the world. Because network reliability and speed are always unknowns outside the US, and even some places within the US, that benchmark meant the page had to load on an internal network in considerably less time. (It also meant I spent time writing a module that would allow me to simulate network rates and reliability that exist in the real world outside the US to more accurately gauge speed...but that's another story.)

The secondary goal was to build pages that would run in any user agent well enough to allow the user to accomplish the task in the shortest amount of time, in part because we know that the amount of time a process takes is inversely proportional to the likelihood of the user completing that process.

The simple truths that govern this secondary goal are that semantic HTML is easier for user agents other than browsers, like the bot(s) that rank your site, to understand; semantic CSS enables you to write fewer, better-performing rules; and unobtrusive JavaScript and Progressive Enhancement expand your audience into non-traditional user agents with the same codebase.

The Problem with Analytics

Examining the effect of these four principles is difficult, in part because of the way data collection for web analytics changed in the first decade of the 21st century. To simplify it, I'll use a mostly hypothetical example.

Let's say you're tracking user agents and see that seventy-three percent of your traffic is coming from Chrome, twenty-one percent is coming from Safari, and five percent is coming from Firefox - like the stats for this site report. Already, one percent is not reported, and if, as is the case for most analytics, your analytics package is using JavaScript to track that rather than the actual web server logs, there are even more users not tracked at all. It's unlikely that you will ever get insight into these users, which is why it has become so easy to dismiss the effect of certain principles, Progressive Enhancement being principle among them.

Progressive Enhancement

One percent is a small number, is it really enough to justify the amount of work Progressive Enhancement requires? On one hand, one percent likely is a small number - unless you're a company like PayPal, with more than 110 million active users - and on the other hand, that one percent represents a much larger number because it's the one percent we know exists. How many users fall into that not-the-popular-browsers group that typically hovers around "one percent", but aren't even measured because their browser isn't identified?

A government agency in the UK tracked visitors to government websites using actual web server logs to determine what percentage of visitors weren't being tracked by their JS analytics package and it was about two percent. Two percent. On government websites. Within a first-world country. For those government websites, that "one percent" is really three percent.

Retail websites are categorically different than government websites, so attempting to extrapolate user behavior in a government-assisted activity to private activity is a little like comparing apples and oranges. The user population in each group has different needs, desires, and expectations, and as a result, have different behavior. All we need consider is the desires and expectations around privacy between the two different categories to see there is a very clear difference in behavior. We can, however, easily support the case that the percentage of tracked users interacting with a government website is likely higher than those interacting with non-government websites. As a result, there is no way to determine what percentage of your actual traffic is untracked unless you are analyzing server logs rather than relying directly on analytics served by JavaScript. The principle of Progressive Enhancement, however, means even these untracked users are served, regardless of the user agent or its features.

Performance Matters

Since we are unlikely to gain insight into the untracked users, what does the data we collect about the tracked group tell us? The primary message we get from the data is that, performance is the most significant issue in web development - it's typically even more important than content. This is not something we've just learned - we've known it for decades. It's also not something that's unmeasured. Organizations frequently calculate the cost of each page or step in a process by calculating the abandonment rate and the drop off rate based on page load time or time-to-interactive, what Twitter used to call "time to first tweet". In a recent, rather finely-tuned test by etsy.com, we see a high correlation between the addition of a mere 160 kilobytes of page weight and a twelve percent drop off.

Semantic Code

Semantic CSS, writing CSS in a way that actually describes something about the code, and semantic HTML, writing HTML in a way that describes the content it contains, expands the robustness of the code (one of the measures of accessibility), is more maintainable, and reduces the amount of code delivered. Semantic code also performs better - there are fewer class and ID selectors to process and a cleaner, more compact Accessibility Tree is built.

Also, lighter pages load faster, and that's important because one of the lessons we've learned from gathering data about tracked users shows that nearly forty percent of users will abandon your site if it takes your page longer than three seconds to "load". If your HTML suffers from divitis and other non-semantic features, or if your CSS is bloated with non-semantic rules, it will take longer to load and parse, pushing the load time further out.

Even worse is the effect non-semantic HTML has on the Accessibility Tree. If your HTML is not semantic, you will have to add code to "fix" the Accessibility Tree, to make your code accessible. Unfortunately, as you add code the risk of harming accessibility even further raises more than the chance you will resolve issues. Here the problems become even more pronounced, because we run into a sub-group of "untracked" users within the tracked users - users who combine accessibility tools with their user agent.

Generally, we only identify site visitors using accessibility tools when they report an issue. There is no method available to determine if a user has engaged an accessibility tool. Conservative estimates generalize that for every accessibility issue registered, at least ten users have been blocked and are unable to complete their task. All industry experts agree about this premise - use of semantic HTML is the best way to minimize the accessibility risks.

A Hypothetical Example

Let's assume that you've looked at your target market and found that seventy-three percent use Chrome, twenty-one percent use Safari, and five percent use Firefox. Almost all - ninety-nine percent - of your users are using agents that have JavaScript enabled by default and you're not aware of "untracked" users. Based on this fantastic news, you've developed your site using ReactJs or Angular and you've tested it locally and the render time is pretty good - it only takes one or two seconds from the request until it's interactive from your development server.

Here's how that plays out in the real world. Your (average) page is really about eighty kilobytes of HTML, about ninety kilobytes of CSS, and almost two megabytes (or more) of JavaScript. If you've not used React's styled elements, your CSS is entirely separate, but nearly all of your eighty kilobytes of HTML is still embedded in the JavaScript. That JavaScript, which has to be entirely loaded before it can be parsed (to make sure there are no syntax errors) and rendered (or hydrated if you've used React's "server-side rendering") is on a CDN, so there's very little latency in the network, but on a 4G connection, mobile devices have a peak of about one-tenth the speed of your local network connection, which means at 100 percent reliability and speed, it's going to take about ten times as long to download the file(s) on a mobile device.

If we assume that you've really optimized and compressed and tweaked your code in production all the ways it's not optimized, compressed, and tweaked in development and your development network is not really that great, the difference between time to interactive on your development site for the more than two megabytes of code is about half of what it will be in production.

So your development version loads in about two seconds, which is pretty fast, and you're pretty confident. In production, your JavaScript resources load HTML into your page in four seconds (versus the two seconds in development) instead of two, but you decide that's still good. Of course, we know you've still lost forty percent of users. Unfortunately, what we now know is that's forty percent of tracked users.

Your beautifully designed site, which only renders for user agents with JavaScript enabled (rendering a "white screen of death" for at least three percent of visitors) unfortunately hasn't undergone an accessibility or usability review and was not written using semantic code. As a result of this oversight, your site has several color contrast issues, navigation is not identified, and your data collection form is not properly labeled - and some unknown portion of the roughly fifty percent of visitors who have made it to your page unscathed cannot complete the task. Here is where we see the final indignity - our analytics cannot tell us why that percentage does not convert, even if it tells us that they do not convert.

An Alternative Example

As an alternative, if your site were written with semantic code using Progressive Enhancement, the code would be lighter, loading in less than four seconds. Your analytics may not have changed, but by using Progressive Enhancement, no one gets a white screen of death, even those in the untracked category, which simply become a hidden bonus rather than a missed opportunity, because the technology fails gracefully.

The semantic code is, by its nature, accessible, and eliminating accessibility issues and offering an estimated ten-fold benefit that taps into an otherwise untracked market - a market that Barclays estimated in 2017 as worth twelve billion pounds, which is not a small number for any organization.

Conclusion

We know sites written with semantic code and unobtrusive JavaScript using Progressive Enhancement

• perform faster
• have greater accessibility
• have greater stability
• have greater durability
• are more maintainable

The question is what are these qualities worth to you - do you continue on as usual or do you try to tap into what is hidden in your analytics. We tend to know a lot of information about what we know - it's what we don't know where the greatest promise rests.

Happy coding.