Ten features you can build for your server-side testing tool

Server-side experimentation is a topic that is getting bigger and bigger. When you decide to switch to server-side experimentation, you face an important choice: build or buy.

Building the basic features of your server-side experimentation platform is often easier than most people think. In most cases, it is also cheaper than buying an existing solution.

This article will briefly cover why you should run your experiments server-side, followed by ten features you could build in your server-side tool, in chronological order.

Before doing so, you need a basic understanding of the difference between client-side and server-side experimentation:

When you type an URL in your browser, it sends a request to a server. The server sends data back to your browser, which results in you seeing the website. With client-side experimentation, the changes you make are loaded in the browser. With server-side, the changes are loaded on the server before being sent to your visitor’s browser.

Why should you run experiments server-side?

There are several advantages when running your experiments server-side:

• It is possible to perform complicated A/B tests, i.e., product (features), algorithms, and the sequence of steps in the checkout.
• There is no flicker effect. The flicker effect occurs when the original page appears briefly before the variation appears — also called Flash of Original Content (FOOC).
• It is often the preferred method by development, as server-side A/B testing fits better with website code.
• Winning experiments can be implemented straight away.
• It uses server-side cookies. As an increasing number of browsers delete client-side cookies after a few days, your experiment data is more reliable.

Of course, there are also some downsides:

• It is a lot harder to set up experiments as, in general, you have to use the website’s coding language. That said, I have seen solutions with the possibility to set up experiments using Javascript and CSS (as in client-side tools).
• There should be sufficient development capacity to run experiments.
• When you use an existing solution (i.e., Optimizely and ABTasty have server-side solutions), it can become quite costly.
• When you build the tool, you need developers to maintain and update the tool.

In my experience, building the tool will save you money over buying the tool.

Ten features to build

When you decide to build your server-side tool, you should consider implementing the following features chronologically. It is essential to start small to reduce complexity.

1. The coinflip

The coin flip splits the website visitors, so the control and variation(s) are displayed to the right visitors. This is, in general, relatively easy to build. The targeting could be a bit more complicated.

Here the same principle applies; start small. For instance, begin with targeting for the device first and work your way from there.

2. Stop/start button

With a stop and start button, more people can start and stop an experiment. Stopping an experiment can be important when the variation is performing a lot worse or causes errors.

3. User Interface

With a User Interface, the tool is more user-friendly, so more people can adopt it. This helps democratize experimentation, helping more product teams to run their experiments and validate their ideas.

Somewhere between 1 and 3

Somewhere between features 1 and 3, you should build a preview or quality assurance link. This helps other people check if your variation displays and performs as it should before the experiment goes live.

At some stage, you also want to build a start and end date feature in the tool, making it easier to plan your experiments.

4. Server-side data: most important KPI

Server-side data is generally more reliable than the data in a platform such as Google Analytics. Setting up server-side measurement can be a challenge. A solution is, again, to start small. For example, start by measuring only the number of visitors and conversions of your most important KPI.

5. Server-side data: several KPIs

The next step would be to include additional metrics for more KPIs.

6. Guardrail metrics and safety stops

With data available in the tool, you can set guardrail metrics. Guardrail metrics are critical metrics for your website, for which you set a threshold before you start experimenting. When the numbers exceed the threshold, the experiment should be terminated or not declared a winner. You can use these as safety checks. Page speed is a common guardrail metric, as well as browser-specific performance.

7. Advanced bucketing

With a lot of data available in the tool, you can now set up advanced bucketing, allowing personalization experiments.

In many situations, this feature can be built in an earlier stage when you have a data management platform that can connect to your tool.

8. Automate calculations

The more time you save, the more experiments you can run. When you automate the calculations, including the significance calculation, your analysts need less time to analyze the experiments’ results.

9. Reporting and documentation

The next step in automation would be automated reporting and documentation.

10. Automated experimentation

And the final step is automated experimentation. In this situation, Scrum teams/developers can no longer release updates to 100% of the visitors. Every release will be an automated experiment. This way, the organization is sure only winners will be fully released, resulting in rapid growth.

Server-side experimentation is the future.

The future of online experimentation lies with server-side testing solutions. Not only can you set up more complex experiments, but it also results in more trustworthy experiments due to privacy regulations, server-side tracking, and not having a flicker effect.

When deciding to build or buy, take your long-term goals into account and find the best fit.

This article gave a brief overview of features to consider for your tool. When your company lacks the experience to build such tools, reach out to others in the market. As they will likely tell you, the coinflip is relatively easy to build.

Therefore, get started, start small, and work your way from there.