A/B testing with Google Optimize

The Flight framework provides an easy way to set up A/B testing using Google Optimize. This requires some setup both in Google Optimize, as well as in the repository.

Limitations

• The current implementation only allows one active experiment at a time, which is configured in shop.config.js .
• The configuration is mostly done in the repository. In Google Optimize, you actually only create the variants, all weighting and other config is done in the code.
• No support for multi-variant tests (instead, create one variant for every combination)

Setting up Google Optimize

1. Create an account and make sure to connect it to your Google Analytics account.
2. Create a new experiment.
3. Set URL to "SERVER_SIDE".
4. Create the amount of variants you want. Don't mind the weighting settings or anything else, all of that needs to be set in the shop config.

Setting up the shop config

Add the following to your shop.config.js:

{
  // other shop settings...
  optimize: {
    container: '', // the container id that can be found in Google Optimize
    experimentId: '', // the experiment id that can be found in Google Optimize
    variantWeights: [0.5, 0.25, 0.25] // an array containing the relative weight for every variant. The first entry is the original variant
  }
}

Note that the weights are relative to each other and work the same way as flexbox styling. If you provide [1, 1, 2] the weight will be 25% for the first two variants and 50% for the last one.

Setting up the experiment

In order to be able to render different code based on variant selection, first wrap your shop in the ExperimentProvider:

import ExperimentsProvider from "@jetshop/core/experiments/ExperimentsProvider";

function Shop() {
  return (
    <ExperimentsProvider>
      <GenericError>
        <ModalProvider>
          <Container>
            <LoadingBar />
            <CustomFont primaryFont={fonts.primary} injectCss={loadFontCss} />
            <Helmet
              titleTemplate="%s - Template Trend"
              defaultTitle="Template Trend"
            />
            <Header />
            <Content>{/* etc */}</Content>
            <Footer />
            <ModalRoot />
            <ScrollRestorationHandler
              ignoreForRouteTypes={["sortOrderChange", "filterChange"]}
            />
          </Container>
        </ModalProvider>
      </GenericError>
    </ExperimentsProvider>
  );
}

The ExperimentProvider reads the cookie that is set on the server side to let React know what experiment and variant it should render. It also tracks the selected variant to GA using gtag.

Next you can use the useExperiment hook to see what code path you should take:

import useExperiment from "@jetshop/core/experiments/useExperiment";

const MyComponent = () => {
  const variant1 = useExperiment("experimentId", 1);
  const variant2 = useExperiment("experimentId", 2);

  if (variant1) {
    return "Variant 1";
  } else if (variant2) {
    return "Variant 2";
  } else {
    return "Original";
  }
};

Note that we match on experiment id and variant, meaning that if you change experiment in shop.config.js, it will not match any of these. This means you can set up multiple tests upfront, but can only have one active experiment at a time.

Support for other analytics trackers

While this solution is built with Google Optimize in mind, it's possible to track the experiment to another provider by extending the tracker to listen to a trackExperiment event. It gets the experiment id and selected variant as props so you can track this to any service you'd like.