Optimized Bayesian A/B Testing Capabilities - Implementation and Execution in the New Version 15.x

Optimized Bayesian A/B Testing Capabilities - Implementation and Execution in the New Version 15.x

A/B Bayesian Testing is a new capability implemented in version 15.x of AroniSmartIntelligence™ available on Apple's App Store® (see the introduction and background on A/B Test in AroniSmartIntelligence™ here). A/B Testing is a statistical methodology that involves randomly splitting an experiment into two groups with the goal to find what works better. For example, in digital marketing where the objective is to test which version of a banner, an e-mail, or  a web page leads to a higher  conversion rate,  a user base is split into two groups. Each of the two groups is showed a different version of the  web page, an app, or an e-mail. The goal is to see if the modified version has a higher impact on user behavior, conversion rates, increased sales, etc. A/B testing has become a key analytical methodology in machine learning  and data science across industries, in the technology in general, given its effectiveness to help test hypotheses and  get reliable analytics to support better informed decision making and drawing conclusions about any numerical data driven hypotheses.  

A/B Test analysis allows to make data-driven decisions about which  Recipe or design is most effective or performs better.  Hence, in digital marketing, A/B Statistical test can be used to compare marketing metrics like visits, clicks, conversion rates, engagements, drop-off rate, time spent on website, and more. 

AroniSmartIntelligence™  A/B Test Capabilities Objectives. 

A/B Test Approach in AroniSmartIntelligence™.

Given the required advanced knowledge of statistics probability distributions,  A/B Bayesian Statistical Testing is included in AroniSmartIntelligence Module dedicated to Statistics Testing and Inference. However, it also requires understanding of Bayesian models, in Module 7 of AroniSmartIntelligence™.
A/B  Bayesian Test is a significant improvement of the Frequentist A/B Testing, which is close to the regular Hypothesis testing implemented in the Statistical Inference.  

Advertisement

GET ARONISMARTINTELLIGENCE  on App Store

apple store 

AroniSmartIntelligence, the leading tool for Advanced Analytics, Machine Learning & Data Science

Statisticians, Data Scientists, Business and Financial Analysts,  Savvy Investors, Engineers, Researchers,  Students, Teachers, Economists, Political Analysts,  and most of  the practitioners use Advanced Analytics to answer questions, to support informed  decision making or to learn.

AroniSmartIntelligence™ is a leading Advanced Analytics, Machine Learning and Data Science tool, with optimized cutting edge Statistics models, Econometrics, Big Data and Text Analytics.

AronismartIntelligence™ includes modules covering Machine Learning,  Big Data mining, Bayesian Statistics, Neural Network Models, Unstructured Text Analysis, Sentiment and Emotion Analytics,  and other advanced analytics.

 AroniSmartIntelligence™ A/B Testing Steps.

 The  new version 15.x of AroniSmartIntellligence™ app on Apple's AppStore®

The data requirements conditions for each model, approach, and module  are documented through messages, tool tips, and documentation. FAQs and other relevant documents can be found on the AroniSoft (http://www.aronisoft.com ) Website.

1. Formulate  and set the hypotheses

Before conducting an A/B testing, it is critical to formulate a null (H0) and an  alternative hypothesis (H1),
Under the null hypothesis (H0), the assumptions are that there is no difference between the control and  test group.
 The null hypothesis assumes   that there is no difference between the control and test  groups.
The  alternative hypothesis relies on that assumptions that the changes are real and  will be confirmed by A/B test and  that changes in the sample observations are not random. From an A/B test perspective, the alternative hypothesis assumes that there is a difference between the control and  the test groups.

 For example,  an alternative hypothesis may be that adding features to a web page would increase  visits or clicks.

 2. Create your control group and test group

Once you determine your null and alternative hypothesis, the next step is to create your control and test (variant) group. There are two important concepts to consider in this step, random samplings and sample size. 

3. Determine the sample size

The sample size will determine how long long the experiment will last in order to collect all the required data. 

4. Fix the p value, before any analysis.

As a recommendation, A/B Testing needs to follow what is known as PICOT process. 

  • Population: the group of people or other targets that participate in the experiment or study
  • Intervention:  the new variant (Test) in the study
  • Comparison:  a reference group (Control) to compare against your intervention (Test)
  • Outcome: the result to be  measured
  • Time: the duration of the experiment, that is when and how long the data is collected.

A/B Testing: Frequentist vs Bayesian Approach AroniSmartIntelligence™.

Frequentist A/B Testing 

The regular A/B testing, as in the Frequentist Hypothesis testing,  has the following approach:

a) Null hypothesis:
The null hypothesis, or H0, assumes that there is no difference between Recipe A  (Control) and Recipe B  (Test or Treatment). For example,  in digital marketing, a Control and Test marketing asset would have no impact on user conversions.  

b) Alternative hypothesis :
An alternative hypothesis suggests the opposite of the null hypothesis. It  assumes that there is a difference between Recipe A  (Control) and Recipe B  (Test) , and hence  moving to Recipe B (Test)  will impact user  conversions.  
c) Statistical significance
Statistical significance is calculated by measuring the p-value, or probability value. A low p-value points to the fact that  the results of the A/B test were not random. In general , a p-value of 5% or lower  means that the A/B test is statistically significant. 
d) Confidence level
The confidence level is  the inverse of the p-value. It indicates that the results of A/B  experiment  are most likely due to the changes in  variables or assets. 

A statistically significant  test leads to a p-value less or equal to  5%, hence to  the confidence level of  95%.

The Frequentist A/B Testing approach usually  faces challenges related to errors. The errors include Type 1 and Type 2 errors. A Type 1 error  leads to deciding that a worse Recipe  is better when it really isn’t. A Type 2 error is finding that a better Recipe is equal or worse than a bad Recipe.  The reason is that the Frequentist A/B Testing relies on just the available samples that may be very small or not  representative.

Bayesian  A/B Testing 

Bayesian A/B Testing relies on the approach used in the Bayesian models  as implemented in Module 7 of AroniSmartIntelligence™. With advances in Statistics and Machine Learning of Artificial Intellligence, Bayesian analytics  framework offers a simpler,  yet more open, reliable, and intuitive approach to A/B testing.
It takes three steps: 
 a) Selecting and Setting Inputs:  Recipe Sample A (Control)  and Recipe Sample B (Treatment/Test), Distributions and Priors
 b) Running the Bayesian A/B Test Models and getting the results  
 c) Getting and Analyzing outputs and results, including the statistics, Probabilities of A  is better that B or B is better than A, confidence interval and lift, expected loss (A-B)/B and plots

The input takes two vectors of data: Recipe A ( Control or Variation A )  and Recipe B  (Test, Treatment or Variation B). Which Recipe is the control or treatment matters only for  the analysis and interpretability of the final  results, plots,  and intervals/point estimates. 

The following Probability Distributions are implemented for A/B Bayesian Test in AroniSmartIntelligence™: Binomial/Bernoulli, Exponential, Normal, LogNormal, Poisson, Uniform, along with  their conjugates: Beta, Gamma, NormalInverseGamma, and Pareto.

For More, check the AroniSmartIntellligence™ app on Apple's AppStore®

The data requirements conditions for each model, approach, and module  are documented through messages, tool tips, and documentation. FAQs and other relevant documents can be found on the AroniSoft (http://www.aronisoft.com ) Website.

Below are a few  an example of A/B Bayesian Test and Regular Frequentist Hypothesis Testing, executed in AroniSmartIntelligence™.

Welcome -- Statistical Inference Module

Figure 1: Welcome -- Select the Statistical Inference Module

Descriptive Statistics

Figure 2: Statistical Inference Module: Descriptive Analytics 

Hypothesis Testing

Figure 3 - a : Statistical Inference Module: Frequentist Hypothesis Testing Capability - T-Tests 

Hypothesis Testing

Figure 3 - b : Statistical Inference Module: Frequentist Hypothesis Testing Capability - F Tests 

AB Bayesian Test Capability

Figure 4: Statistical Inference Module: AB Bayesian Test Capability

AB Bayesian Test Inputs Analysis

Figure 5: AB Bayesian Test Inputs Analysis

AB Bayesian Test Inputs Analysis - Posteriors Analysis Results

Figure 6: AB Bayesian Test Analysis - Posteriors Analysis Results

AB Bayesian Test Inputs Analysis - Posteriors Analysis Results

Figure 6: AB Bayesian Test Analysis - (A-B)/B Samples Posteriors Analysis Results

  For more visit here AroniSmartIntelligence™: Optimized Machine Learning, Advanced Analytics, and Data Science Capabilities Including Econometrics, Bayesian Models, Neural Network Models, Marketing Mix Models, and NLP Analytics  

FOR QUICK DEMOS ON HANDBOOK and USER'S MANUALS,  PLEASE CLICK ON THE LINKS BELOW TO DOWNLOAD THE PDF FILES:

Advertising:

GET ARONISMARTINTELLIGENCE on App Store

 

apple store 

 

What is next?

The new version of AroniSmartIntelligence™ that includes improved Advanced Analytics, Machine Learning and Data Science  is available on Apple's App Store®.

More detailed information on the capabilities of  AroniSmartIntelligence™ is available in the tool on Apple®'s App Store and on the AroniSoft web site(http://www.aronisoft.com).  

Check the Frequently Asked Questions (FAQ) here.

More detailed use cases, real life applications,  and AroniSmart Tools in actions  can be found on  on the AroniSmart web site (http://www.aronismart.com). Check the following tabs: ABOUT US, SOLUTIONS, SCITECH, MONEY. 

Advertisement

GET ARONISMARTINVEST, A LEADING INVESTMENT RESEARCH TOOL, BASED ON  Advanced Analytics, machine Learning and Data Science  on App Store -- click here    apple store

@2024 AroniSoft LLC

For More on AroniSoft LLC andAroniSmart products click here

  1. You are here:  
  2. Home
  3. Products & Solutions
  4. Advanced Capabilities
  5. Optimized Bayesian A/B Testing Capabilities - Implementation and Execution in the New Version 15.x