Recently, we, the Data Science Big Data Sbertech team, completed one interesting pilot in the field of machine learning: we tried using it to identify incidents in the operation of Sberbank Online mobile, based on the texts of user reviews in app stores. We took reviews from Google Play for the period from October 2014 to October 2017 (882,864 reviews), filtered 92,711 negative (1-2 stars) and started work. What - we tell under the cut.
Immediately answer a possible question. The ratio of positive to negative reviews turned out to be in no way connected with incidents, therefore more cunning methods were needed. In addition, we wanted to identify the topics of the incidents.
To identify anomalies, we chose the Arima autoregressive integrated moving average model.
ARIMA (p, d, q) for non-stationary time series
Xt has the form:
Where
εt is the stationary time series;
c, ai, bj - model parameters; △
d is the difference operator of a time series of order
d (sequential taking of
d times of differences of the first order — first from the time series, then from the obtained differences of the first order, then from the second order, etc.).
Trained model on the entire sample. To predict the normal level of negative reviews on the selected date, we used an interval of three months before it. The prediction was built a week ahead of the selected date, with discretization in one day.
Next formed a confidence level. To this end, the estimated value of the confidence interval was added to the value of the prediction function for each date. It was taken for the third quantile of the distribution of all deviations of the prediction function from the actual amount of negative feedback for the three-month interval on which the prediction of the normal level is built.
The anomaly was recorded when the actual amount of negative feedback was above the confidence level (predicted value + confidence interval). Here is how it looks on the chart:
Red is the actual number of reviews. Yellow is the predicted value of the normal level with a confidence interval.For further analysis, we selected 5 distinct peaks, which fell on February 27, 2017, 03/15/2017, 09/14/2017, 09/18/2017, 09/20/2017.
The identification of the incidents was carried out by clustering the texts of reviews with two models based on BIGARTM and Word2Vec with integrated Kmeans clustering.
Library BIGARTM (thematic model)
The thematic model is a representation of the observed conditional distribution
p (w | d) of terms (words or phrases)
w in documents
d of collection
D :
where T is a variety of topics;
- unknown distribution of terms in topic
t ;
- unknown distribution of topics in the document
d .
Thematic model parameters - matrix
and
- are found by solving the problem of maximizing the likelihood:
The main advantage over other probabilistic models is the presence of regularizers for smoothing, thinning, and de-correlating.
Model with Word2Vec
The model using Word2Vec is based on tensor algebra. Under it was compiled a dictionary of all the words in the reviews. The Word2Vec vector word representation has become the base space for clusters.
The algorithm is such that it seeks to minimize the total quadratic deviation of cluster points from the centers of these clusters:
k is the number of clusters,
S i are the resulting clusters,
i = 1,2 ...
k , and
μ i are the mass centers of all vectors
x from the cluster
S i . Now let's look at the results of the models with concrete examples.
Anomalous period with a peak 03/15/2017
BIGARTM :
Cluster number
| Tags
| Cluster Feedback Examples
(original vocabulary)
|
one
| update version operation account work error cannot install delete transfer return
| “The old version is not working. I do not need an update. There is no space on the phone! Return the possibility of using the old version !!! ”
“I cannot install the application at all, there is plenty of memory on the phone, more than 50%, but the application stubbornly writes that it is not possible to install, error 24! I fixed the bug, I used it before, I really liked it !!! "
"After the upgrade, it stopped working! A window appears with the message: "" An error occurred in the application “Sberbank” “and goodbye. I pulled the damn upgrade! Before the update, everything worked fine, completely satisfied. And in the course I do not have one such crap! ”
|
2
| connection go write work can't_ memory can not fly out constantly horrible delete problem
| “I can’t enter the Application, enter the code, write, I can’t restore the secure connection! what the heck!? Or do you force to be updated! I was satisfied with the old update! Solve the problem!"
“What did you do with a great program? Got it and regretted. Every 2 minutes, a window pops up with the text ““ the connection with the savings bank was interrupted ”.” I decided to remove and reinstall. Now it does not open at all. Immediately throws out of the program. Guys, return the old version »
|
3
| sms not_to come in enter the code card you need to thank you not_work out the input can not take off
| "Do not come to the phone SMS with a password"
"Does not register, writes that SMS will come with the code and nothing ...."
“After the update, an SMS comes in about the entrance, but the threshold flies out in 4 seconds. How to be?
|
Word2Vec + Kmeans:Cluster number
| Tags
| Cluster Feedback Examples
(original vocabulary)
|
one
| update connection log in use update
| “I can’t enter the Application, enter the code, write, I can’t restore the secure connection! what the heck!? Or do you force to be updated! I was satisfied with the old update! Solve the problem!"
“The connection is broken, connect to another network - that's what I see most often when I want to go to the application. Updates always put immediately when they come out. If you can go, everything works fine. ”
|
2
| sms to come in phone enter
| “After the update, an SMS comes in about the entrance, but the threshold flies out in 4 seconds. How to be?
"Do not come to the phone SMS with a password"
"When registering, SMS does not come with the code ..."
"It is not possible to enter. Code by SMS does not come. Correct everything for soon."
|
BIGARTM
| Word2Vec + Kmeans
|
| 
|
- Only 194 reviews.
- Identified 3 clusters.
Problems identified during clustering:
- The problem of the application associated with the upgrade version.
- The problem of the application associated with the connection with Sberbank.
- The problem of sending the user sms code.
| - Total 190 reviews.
- Identified 2 clusters.
Problems identified during clustering:
- The problem with the entrance to the application associated with the connection and update version.
- The problem of sending the user sms code.
|
As can be seen from the results, BIGARTM more clearly highlights the topics compared to Word2Vec + Kmeans. For the Word2Vec + Kmeans model, short non-informative reviews with words that did not have a vector representation (words with errors, profanity, etc.) were not taken into account.
Interestingly, during this period, incidents in the internal system ServiceDesk were not classified according to the version update problem and were processed separately. This means that in this case the model did the job better than the person.
Early Recognition
And here is an example demonstrating the ability of our model to work in advance. On September 18, 2017, a massive problem was registered in ServiceDesk related to the inability to log into the application and make payments for some users. On the Arima graph, we see a strong surge of negative reviews on this date, as well as a slight release four days earlier.
We analyzed both of these emissions.
September 14, 2017:BIGARTM
| Word2Vec + Kmeans
|
| 
|
- Only 243 reviews.
- Identified 3 clusters.
Problems identified during clustering:
- The problem with the transfer of funds. The problem with the interface.
- The problem with entering the application after the update.
- The problem with the entrance to the application. The user is not satisfied with the privacy policy.
| - Total 171 reviews.
- Identified 1 cluster.
Clustering Issue:
- The problem of working in the application after the update. The user is not satisfied with the privacy policy, there is also a problem with the transfer of funds.
|
September 18, 2017:BIGARTM
| Word2Vec + Kmeans
|
| 
|
- Total 935 reviews.
- Identified 3 clusters.
Problems identified during clustering:
- The problem with entering the application after updating the version. The application gives the user a message about the outdated version.
- The problem with the entrance to the application. The user enters a password several times, the application hangs.
- The problem of the application associated with the built-in antivirus.
| - Total 828 reviews.
- Identified 1 cluster.
Clustering Issue:
- The problem with entering a password and entering the application. Version update error.
|
But how do these results relate to the data from ServiceDesk.
Date of anomaly detection
| Registered SM issue
| Identifying the nature of the problem with using
BIGARTM
| Identifying the nature of the problem with using
Word2Vec + Kmeans
|
2017-09-14
00:00 - 24:00
| After updating the application, an increased background of calls from updated applications was formed.
| The problem with the transfer of funds. The problem with the interface.
The problem with entering the application after the update.
The problem with the entrance to the application.
The user is not satisfied with the privacy policy.
| The problem of working in the application after the update. The user is not satisfied with the privacy policy, there is also a problem with the transfer of funds.
|
2017-09-18
00:00 - 24:00
| Between 09:23
at 15:20 Moscow time, for a part of the Bank's customers, an increase was recorded in the time for entering and conducting operations with Sberbank Online.
The incidents had virtually no effect on the work of clients using IOS.
Increased number of customer calls to the Bank's contact center (up to 1369 additional calls in 15 minutes).
| The problem with entering the application after updating the version. The application gives the user a message about the outdated version.
The problem with the entrance to the application. The user enters a password several times, the application hangs.
The problem of the application associated with the built-in antivirus.
| The problem with entering a password and entering the application. Version update error.
|
It is obvious that on September 18, a large flow of negative customer reviews and calls to the contact center was caused by the lack of proper response to the incident of the increased background on September 14. This example clearly demonstrates that the model could help avoid a mass problem by recognizing it at an early stage.
It is also interesting that at the same time, the model discovered another problem (with the built-in antivirus), which was not highlighted against the background of the main problem. For her registration in ServiceDesk is missing.
Let's sum up
Machine learning is good at analyzing user feedback. Sometimes even more accurate manual processing. This will help reduce the time to eliminate incidents of a mobile application and make it better.
Perhaps such methods are suitable not only for identifying incidents, but also for analyzing neutral and positive reviews in order to highlight priority user cases. This will help to develop the functionality of the application based on customer preferences at no additional cost for their collection and analysis. But this idea must also be checked ...