Hello! My name is Sasha Zimin, I work as an iOS developer in the London office of Badoo . In Badoo, there is a very close interaction with product managers, and I have adopted the habit of testing all the hypotheses that I have regarding the product. So, I started writing split tests for my projects.

The framework discussed in this article was written for two purposes. First, to avoid possible errors, because it is better that there is no data in the analytics system than the data is incorrect (or, in general, data that can be misinterpreted and broken wood). Secondly, to simplify the implementation of each subsequent test. But let's begin, perhaps, with what split tests are.

Nowadays there are millions of applications that solve most user needs, so every day it becomes more and more difficult to create new competitive products. This led to the fact that many companies and start-ups first conducted various studies and experiments to find out which functions make their product better, and which ones can be dispensed with.

One of the main tools for conducting such experiments is split testing (or A / B testing). In this article I will tell how it can be implemented on Swift.

All project demonstration materials are available here . If you already have an idea about A / B testing, you can go directly to the code .

Brief introduction to split testing

Split testing, or A / B testing (this term is not always correct, because you may have more than two groups of participants) is a way to check different versions of a product for different groups of users in order to understand which version is better. You can read about it in “ Wikipedia ” or, for example, in this article with real examples.

In Badoo, we perform many split tests at the same time. For example, once we decided that the user profile page in our application looks outdated, and also wanted to improve user interaction with some banners. Therefore, we launched a split test with three groups:

1. Old profile
   
2. New profile, version 1
   
3. New profile, version 2
   

As you can see, we had three options, more like A / B / C testing (and that is why we prefer to use the term “split testing”).

So different users saw their profiles:



In the Product Manager console, we had four user groups that were randomly generated and had the same size:



Perhaps you ask why we have control and control_check (if control_check is a copy of control logic)? The answer is very simple: any change affects many indicators, so we can never be absolutely sure that a change is the result of a split test, and not other actions.

If you think that some indicators have changed due to the split test, then you should double check that they are the same inside the control and control_check groups.

As you can see, user opinions may differ, but empirical data are clear evidence. The product management team analyzes the results and understands why one option is better than the other.

Split Testing and Swift

Goals:

1. Create a library for the client part (without using the server).
   
2. Save the selected user in the permanent storage after it has been randomly generated.
   
3. Send reports on the selected options for each split test to the analytics service.
   
4. Use Swift as much as possible.
   

PS The use of such a library for split testing of the client part has its advantages and disadvantages. The main advantage is that you do not need to have a server infrastructure or a dedicated server. And the disadvantage is that if something goes wrong during the experiment, you will not be able to roll back without downloading the new version in the App Store.

A few words about the implementation:

1. When conducting an experiment, the option for the user is chosen randomly according to an equally probable principle.
2. Split testing service can use:

• Any data store (for example, UserDefaults, Realm, SQLite or Core Data) as a dependency and save the value assigned to the user (the value of his variant) into it.
• Any analytics service (for example, Amplitude or Facebook Analytics) as a dependency and send the current version at the moment when the user encounters a split test.

Here is a diagram of the future classes:



All split tests will be presented using the SplitTestProtocol , and each of them will have several options (groups), which will be presented in the SplitTestGroupProtocol .

The split test should be able to inform the analyst of the current version, so it will have an AnalyticsProtocol as a dependency.

The SplitTestingService will save, generate variants and manage all the split tests. It is he who downloads the current version of the user from the storage, which is determined by the StorageProtocol , and also sends the Analytics Protocol to the SplitTestProtocol .

---

Let's start writing code with AnalyticsProtocol and StorageProtocol dependencies :

protocol AnalyticsServiceProtocol {    func setOnce(value: String, for key: String) } protocol StorageServiceProtocol {    func save(string: String?, for key: String)    func getString(for key: String) -> String? } 

The role of analytics is to record events once. For example, to fix that user A is in the blue group in the split-test process button_color , when he sees the screen with this button.

The storage role is to save a specific option for the current user (after the SplitTestingService generated this option) and its subsequent reading each time the program accesses this split test.

So let's look at the SplitTestGroupProtocol , which characterizes a set of options for a particular split test:

 protocol SplitTestGroupProtocol: RawRepresentable where RawValue == String {   static var testGroups: [Self] { get } } 

Since RawRepresentable where RawValue is a string, you can easily create a variant from a string or convert it back to a string, which is very convenient for working with analytics and storage. Also SplitTestGroupProtocol contains an array testGroups, in which the composition of the current options can be specified (this array will also be used to randomly generate from the available options).

This is the basis for the SplitTestProtocol split test itself :

 protocol SplitTestProtocol {   associatedtype GroupType: SplitTestGroupProtocol   static var identifier: String { get }   var currentGroup: GroupType { get }   var analytics: AnalyticsServiceProtocol { get }   init(currentGroup: GroupType, analytics: AnalyticsServiceProtocol) } extension SplitTestProtocol {   func hitSplitTest() {       self.analytics.setOnce(value: self.currentGroup.rawValue, for: Self.analyticsKey)   }   static var analyticsKey: String {       return "split_test-\(self.identifier)"   }   static var dataBaseKey: String {       return "split_test_database-\(self.identifier)"   } } 

SplitTestProtocol contains:

1. The GroupType type, which implements the SplitTestGroupProtocol protocol to represent the type that defines a set of options.
   
2. String identifier for analytics and storage keys.
   
3. The currentGroup variable to write to the specific instance of SplitTestProtocol .
   
4. The analytics dependency for the hitSplitTest method.
   
5. And the hitSplitTest method, which informs the analyst that the user has seen the result of the split test.
   

The hitSplitTest method allows you to make sure that users are not only in a certain variant, but also saw the test result. Marking a user who has not visited the shopping section as “saw_red_button_on_purcahse_screen” will distort the results.

Now we have everything ready for SplitTestingService :

 protocol SplitTestingServiceProtocol {   func fetchSplitTest<Value: SplitTestProtocol>(_ splitTestType: Value.Type) -> Value } class SplitTestingService: SplitTestingServiceProtocol {   private let analyticsService: AnalyticsServiceProtocol   private let storage: StorageServiceProtocol   init(analyticsService: AnalyticsServiceProtocol, storage: StorageServiceProtocol) {       self.analyticsService = analyticsService       self.storage = storage   }   func fetchSplitTest<Value: SplitTestProtocol>(_ splitTestType: Value.Type) -> Value {       if let value = self.getGroup(splitTestType) {           return Value(currentGroup: value, analytics: self.analyticsService)       }       let randomGroup = self.randomGroup(Value.self)       self.saveGroup(splitTestType, group: randomGroup)       return Value(currentGroup: randomGroup, analytics: self.analyticsService)   }   private func saveGroup<Value: SplitTestProtocol>(_ splitTestType: Value.Type, group: Value.GroupType) {       self.storage.save(string: group.rawValue, for: Value.dataBaseKey)   }   private func getGroup<Value: SplitTestProtocol>(_ splitTestType: Value.Type) -> Value.GroupType? {       guard let stringValue = self.storage.getString(for: Value.dataBaseKey) else {           return nil       }       return Value.GroupType(rawValue: stringValue)   }   private func randomGroup<Value: SplitTestProtocol>(_ splitTestType: Value.Type) -> Value.GroupType {       let count = Value.GroupType.testGroups.count       let random = Int.random(lower: 0, count - 1)       return Value.GroupType.testGroups[random]   } } 

PS In this class, we use the Int.random function, taken from
here , but in Swift 4.2 it is already built in by default.

This class contains one public method fetchSplitTest and three private methods: saveGroup , getGroup , randomGroup .

The randomGroup method generates a random variant for the selected split test, while getGroup and saveGroup allow you to save or load a variant for a specific split test for the current user.

The main and public function of this class is fetchSplitTest: it tries to return the current version from the persistent storage and, if it fails, generates and stores a random version before returning it.

---

Now we are ready to create our first split test:

 final class ButtonColorSplitTest: SplitTestProtocol {   static var identifier: String = "button_color"   var currentGroup: ButtonColorSplitTest.Group   var analytics: AnalyticsServiceProtocol   init(currentGroup: ButtonColorSplitTest.Group, analytics: AnalyticsServiceProtocol) {       self.currentGroup = currentGroup       self.analytics = analytics   }   typealias GroupType = Group   enum Group: String, SplitTestGroupProtocol {       case red = "red"       case blue = "blue"       case darkGray = "dark_gray"       static var testGroups: [ButtonColorSplitTest.Group] = [.red, .blue, .darkGray]   } } extension ButtonColorSplitTest.Group {   var color: UIColor {       switch self {       case .blue:           return .blue       case .red:           return .red       case .darkGray:           return .darkGray       }   } } 

It looks impressive, but don't worry: as soon as you implement the SplitTestProtocol as a separate class, the compiler will ask you to implement all the necessary properties.

The important part here is the enum Group type. You should put all your groups in it (in our example it is red, blue and darkGray), and define the string values ​​here to ensure the correct transfer to the analytics.

We also have the ButtonColorSplitTest.Group extension, which allows you to use the full potential of Swift. Now let's create objects for the AnalyticsProtocol and StorageProtocol :

 extension UserDefaults: StorageServiceProtocol {   func save(string: String?, for key: String) {       self.set(string, forKey: key)   }   func getString(for key: String) -> String? {       return self.object(forKey: key) as? String   } } 

For the StorageProtocol, we will use the UserDefaults class, because it is easy to implement, but in your projects you can work with any other persistent storage (for example, I chose Keychain for myself, since it keeps the group as a user even after deletion).

In this example, I’ll create a class of fictitious analytics, but you can use real analytics in your project. For example, you can use the service Amplitude .

 // Dummy class for example, use something real, like Amplitude class Analytics {   func logOnce(property: NSObject, for key: String) {       let storageKey = "example.\(key)"       if UserDefaults.standard.object(forKey: storageKey) == nil {           print("Log once value: \(property) for key: \(key)")           UserDefaults.standard.set("", forKey: storageKey) // String because of simulator bug       }   } } extension Analytics: AnalyticsServiceProtocol {   func setOnce(value: String, for key: String) {       self.logOnce(property: value as NSObject, for: key)   } } 

Now we are ready to use our split test:

 let splitTestingService = SplitTestingService(analyticsService: Analytics(),                                                      storage: UserDefaults.standard) let buttonSplitTest = splitTestingService.fetchSplitTest(ButtonColorSplitTest.self) self.button.backgroundColor = buttonSplitTest.currentGroup.color buttonSplitTest.hitSplitTest() 

Simply create your own copy, extract the split test and use it. Generalizations allow you to call buttonSplitTest.currentGroup.color.

During first use, you can see something like ( Log once value ) : split_test-button_color for key: dark_gray , and if you do not delete the application from the device, the button will be the same every time it starts up.

---

The process of implementing such a library takes some time, but after that, each new split test inside your project will be created in a couple of minutes.

Here is an example of using the engine in a real application: in analytics, we segmented users according to the complexity factor and probability of buying game currency.



People who have never come across this coefficient of difficulty (none) probably do not play at all and do not buy anything in games (which is logical), which is why it is important to send the result (generated version) of split testing to the server at the moment when users really face your test.

Without the complexity factor, only 2% of users bought game currency. With a small purchase rate already made 3%. And with a high coefficient of difficulty, 4% of players bought currency. This means that you can continue to increase the coefficient and watch the numbers. :)

If you are interested in analyzing the results with maximum accuracy, I advise you to use this tool .

Thanks to the wonderful team that helped me in the work on this article (especially Igor , Kelly and Jairo ).

The entire demo project is available at this link .