A Brief History of the IBM SPSS Random Trees

Random Forest is one of the most popular algorithms in the machine learning family. It is a highly flexible machine learning method widely used in many areas, from marketing to healthcare insurance. It can be used to perform marketing simulation modeling, customer source analysis, retention and churn. It can also be applied to predict disease risk and patient susceptibility.

What is Random Forest? The Random Forest algorithm is a powerful new approach for strong (accurate) predictive models. It is comparable and sometimes better than other state-of-the-art methods in classification or regression problems.

Random Forest is an ensemble model consisting of multiple C&RT-like trees. Each tree grows on a bootstrap sample which is obtained by sampling the original data cases with replacement. Moreover, during tree growth, for each node the best split variable is selected from a specified smaller number of variables which are drawn randomly from the full set of variables. Ideally, each tree grows to the largest extent possible. There is no pruning. In scoring, random forest combines individual tree scores by majority voting (for classification) or average (for regression).  The tree model can be built independently, so Random Forest is very suitable to be applied in a distributed setting. 

The IBM SPSS Random Trees algorithm is designed to implement all of the necessary functions of Random Forest, It has many additional nice features, as well. It is designed to build Random Forest in a distributed environment, so it supports massive, and potentially wide data.  In addition, it provides model evaluation measures and post-processing features, such as predictor importance, interesting decision rules, and so on.

Where can you get IBM SPSS Random Trees?

Product Integration with UI

• IBM SPSS Modeler V18.0.0 or later

Harvested in the Product

• IBM Watson Studio, all IBM SPSS algorithms are available in the Watson Studio Cloud, you should follow the Watson Studio access policy.

API Documentation for Spark and Python

• You can get the API Documentation for Spark and Python Here, this is the link to the API doc;  it is online and should be free.

More Information & a Use Case

There is also a video available to introduce IBM SPSS Random Trees, in which more details are provided. We present here a customer churn use case to show what issues IBM SPSS Random Trees can resolve. Both Modeler and Watson Studio are used in the example. 

You can watch the video on YouTube below. Some scripts, corresponding to the content in the video, are given as well.

The Script of the Video

Hello everyone. Today I want to introduce an IBM SPSS Algorithm in Spark –IBM SPSS Random Trees. First, I will go through “What’s Random Trees” and the “Advantage of the Random Trees”. Then, I will  present a use case showing how to use Random Trees in IBM SPSS Modeler. Finally, I will give the complete solution for a use case to predict churn for a telecom company using IBM SPSS Random Trees in the Jupiter notebook on the Watson Studio platform.

What is Random Trees? Well, in other products, such as from open source or other providers, it is called Random Forest, but in IBM SPSS products we  provide a similar algorithm called Random Trees. The Introduction from a wiki says Random Forest or random decision forest are an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time and outputting the class that is the mode of classes for classification, or the mean prediction for regression of the individual trees.

It seems a little complex but we can use some key words to learn about this. First, it is a combination of decision trees. That means there are lots of trees here. Second, it is an ensemble method. Third, it supports both classification and regression. And Fourth, it provides prediction by vote or average --- voting for classification and using the average for regression.

Random Forests is a well-known and popular algorithm in data mining. It is included in almost all the popular statistic tools.

So again, what’s Random Trees? Before I introduce the Random trees further, I want to first introduce the idea of a tree. Organization charts and directory trees are tree charts. They have a root, many nodes, some leaves, and branches. It is a commonly seen structure, and we call it a tree.

In data mining, we introduce a specific type of tree called a “Decision Tree”. What can decision trees do? For example, when you are planning a journey, a decision tree can provide you with rules based on other people’s choices that it found from data. A decision tree may find out that for a journey of less than 1000km, people tend to travel by high speed train, while using an airplane is a more popular choice if the distance is more than that. A person preferring flexibility favors driving, so that stops can be made anywhere, any time.  If someone enjoys the scenery, then a normal daytime train will provide an enjoyable journey. That’s how a decision tree works.

In the decision tree we have conditions;  if you choose a condition you can follow the branches to the final decision, so the decision tree helps us to make the decision;  it is a very common data mining method.

Then what about the decision forest? If you have many decision trees and they work together for the same task, and it predicts by vote or mean, we can call it a decision forest. It combines many decision trees together.

Next, it is time to introduce Random Trees (or random forest). We have already introduced the decision forest that combined by many decision trees. As for Random Trees, we use some random methods to build many decision trees and combine them together for the same work.

Random Trees has two random dimensions. The first dimension consists of  each tree in the Random Trees using input data randomly. We use the bootstrap sample to select the data to build each tree. Using this sampling method, we create many trees and each tree uses different data and the data may have an overlap due to sampling with replacement. Using the first tree as an example, let’s say that it selects four cases for model building( learning) randomly and the second tree, selects another four cases randomly, because we use the bootstrap sample to select data, it has the possibility of selecting the same cases so there is an overlap when growing this second tree. Even though there may be data overlaps, the majority of data for each tree are different. That is the first random dimension.

The second dimension is that the variables (candidate predictors) are selected randomly in each branching node in the same tree. We will take the 1^st tree  as an example. When we build the first node of tree 1, we randomly select 5 variables, we do not use all of the variables available, we just select these 5 variables as the candidates to learn the node 1 tree, then we will grow the node 2 and node 3 trees. For the node 2 tree we also select 5 variables all randomly from all of variables. Similarly, we do the same for the node 3 tree. That’s the 2nd dimension of randomness in Random Trees.

After we use the above random methods, we will build lots of trees using different records, and in the same tree we will use different variables.  We call the decision trees built with such a methodology Random Trees.

There are some advantages for the IBM SPSS Random Trees (see below):

• Embedded Predictor Importance:

Once the Random Trees model is built, predictor importance can be computed directly according to the information contained in the model

• Handling Imbalanced Classification:

IBM SPSS Random Trees can handle imbalanced data. It uses a balanced bootstrap sampling method to build base trees. This operation can improve the predictions of Random Trees on minor class cases.

• Exporting Interesting Decision Rules:

In SPSS Random Trees, the interesting rules in the trees can be exported. Interesting rules are defined as those decision rules that have high accuracy and also whose predictions are consistent with the predictions of the predominance of trees as much as possible. These rules provide  useful insights into  the data. Since we have lots of trees and rules, it is not easy for  a user to understand all the rules. They can only be understood by the computer. We can select some interesting rules for the user to understand the whole model.

• Predictor Contribution in Scoring:

Random Trees can indicate how much each predictor contributes to the prediction of each case. We can specify the Number of Predictors with top contributions, then the top N most important predictors are reported for each case.

• PMML File Output:

IBM SPSS Random Trees can output PMML(Predictive Model Markup Language) files. PMML is consistent with the DMG standard. The model information is stored in  a PMML file. Using PMML can help to separate model building from  scoring . We can store the model in a PMML file and then do the scoring later. This is a common feature for IBM SPSS model building algorithms. In the market, many 3rd party companies provide the tools to parse IBM SPSS Random Trees’ PMML file, and then do the prediction.  The 3rd party JPMML is a good example.

There are some other characteristics in IBM SPSS Random Trees. Since it uses the bootstrap to sample data, every tree uses sampled data and not all of the data, thus it can run efficiently on large data sets.  It uses the variables selected randomly when growing each of the trees, thus it can handle wide data directly without variable dimension reduction (feature selection).  Also the trees can be built independently, so IBM Random Trees is very suitable  for a distributed setting; And finally IBM Random Trees is an ensemble method, so comparing it with a single model, the ensemble model will usually get a better accuracy.

Below, I will use a use case in IBM SPSS Modeler to show its benefits.

Use Case:

Jack is the customer manager in a telecom company. He wants to know the possibility of potential churn of his customers based on previous experience, and then he can take some actions accordingly to retain the customers.

Currently he maintains a customer information table as below: Here are the data: telco_Feb.csv telco.csv

The last column “churn” in the table shows the information as to whether the customer churned or not.  The value “0.0” means we still kept the customer; otherwise, the value is “1.0”. For sure Jack should pay attention to this column.

To help Jack in doing his analysis, we plan to develop a suitable algorithm for him. The data from last year are used as the training data and the data of February of this year are taken as the testing data to verify the model accuracy. Data of all the columns are used to set up the model. To meet the requirement, many algorithms can be selected. Since we are interested in whether a customer churns or not, which can be indicated by a binary number, the logistic model is employed at the beginning.

As mentioned, the stream in IBM SPSS Modeler to run the logistic model and do the analysis is shown below:

The analysis result is shown below:

We can see that the accuracy of the Logistic model is only 10%

Considering that many categorical and continuous variables are in the data and all of them are used to build the model, we want to use the decision tree (C&RT) model as it should work well in such a scenario. So, we retest by using C&RT with default settings and with or without boosting, respectively.

The Stream in IBM SPSS Modeler to use the C&RT model with default settings  is shown below:

The accuracy analysis result is shown below:

For the Boosted C&RT model in Modeler, only the boosting parameter is set to true, other parameters are keeping the default values, the analysis is performed once model is built.

 The stream in IBM SPSS Modeler that uses a Boosted C&RT model and do the analysis, is shown below:

The Parameters Setting page for the Boosted C&RT shown as below:

The accuracy analysis result is shown below:

We get better results for C&RT (78.2%) and Boosted C&RT (80.4%). Although it is improved, is there still  room for improvement?

Finally, we select IBM SPSS Random Trees with default settings because it also supports both categorical and continuous variables.

Below is the Stream in IBM SPSS Modeler using Random Trees as the model, with the analysis results:

The analysis results for the Random Trees are shown below:

We see that the accuracy result for Random Trees is 93%. It is much better than the others we’ve tried. So the Random Trees model shows the highest accuracy for this kind of data, and it should be used to do any further predictions.

Next, I will provide another complete example to illustrate this use case: To predict churn for a telecom company with IBM SPSS Random Trees in a Jupiter notebook on the Watson Studio platform.

Use Case: Predicting churn with IBM SPSS Random Trees algorithm

From the churn prediction in IBM SPSS Modeler above, the Random Trees algorithm outperforms the other algorithms in terms of prediction accuracy. Now we use the same use case and the Random Trees algorithm as an example to demonstrate how to use IBM SPSS algorithms on Apache Spark version 2.1. Through the use case, you will learn about:

• How to create an IBM SPSS Random Trees model through the IBM SPSS Machine Learning API.
• How to view the model using the IBM SPSS Model Viewer.
• Based on the model we built, how to predict if a customer will churn or not given his/her demographic and service usage information.

The use case will run on a Watson Studio notebook which uses the kernel Scala 2.11 with Spark 2.1. Below are snippets in the notebook for each step in the analysis.

• Load the Telco Churn data to the cloud data repository

From the above data we can see that the Telco file is a hypothetical data file that concerns a telecom company's efforts to reduce turnover in its customer base. Each case corresponds to a single customer and it records various demographic and service usage information.

Upload telco.csv and telco_Feb.csv to Watson Studio, where telco.csv is used for training model and telco_Feb.csv is used for testing.

Create a DataFrame for the telco.csv data, from the Notebook “Find and add data”, select the file telco.csv that was just uploaded,  then click  “Insert to code”, then  “Insert SparkSession DataFrame”;  we just need to rename the DataFrame variable name to dfTelco.

Follow similar steps to load the telco_Feb.csv file.

val dfTelco = spark.
   read.format("org.apache.spark.sql.execution.datasources.csv.CSVFileFormat").
    option("header", "true").
    option("inferSchema", "true").
    load(cos.url("newpro-donotdelete-pr-szswpvsm7exzyc", "telco.csv"))
dfTelco.show(1)

val dfTelcoFeb = spark.
    read.format("org.apache.spark.sql.execution.datasources.csv.CSVFileFormat").
    option("header", "true").
    option("inferSchema", "true").
    load(cos.url("newpro-donotdelete-pr-szswpvsm7exzyc", "telco_Feb.csv"))
dfTelcoFeb.show(1)

• Configure the Random Trees model

To configure the model, you need to import the libraries first. Then if there are some fields that are nominal or ordinal in the data, you need to set them clearly. Then RandomTrees() is used to set the target, predictors and model setting parameters. If there is no inputFieldList value to be set, then all of the fields are treated as input fields except the target, frequency, and analysis weight fields. In the example, the filed churn is the target, and all of the other fields are predictors. The number of trees is set to 10 which you can change to another number to balance the model accuracy and the model building time.

import com.ibm.spss.ml.classificationandregression.ensemble.RandomTrees
import com.ibm.spss.ml.utils.DataFrameImplicits._

val ordinal = "ed"
    val nominal = Array("region",
      "marital",
    "retire",
      "gender",
      "tollfree",
      "equip",
      "callcard",
      "wireless","multline",
      "voice","pager","internet","callid","callwait","forward","confer",
      "ebill",
      "custcat",
      "churn"
    )
val dfTelCo_M = dfTelco.setNominalMeasure(nominal,true).setOrdinalMeasure(Array(ordinal),true)
val srf = RandomTrees().setTargetField("churn").setNumTrees(10)
val srfModel = srf.fit(dfTelCo_M)

Predict the churn of each record in the testing data set:

val predResult = srfModel.transform(dfTelcoFeb)
val predResultNew = predResult.withColumn("prediction", predResult("prediction").cast("double")).
    withColumn("churn", predResult("churn").cast("double"))

Using the function MulticlassClassificationEvaluator to get the model accuracy in the testing data set:

import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
val evaluator = new MulticlassClassificationEvaluator().setLabelCol("churn").setMetricName("accuracy")
val acc_result = evaluator.evaluate(predResultNew)
println(s"acc_result:$acc_result")

acc_result:0.958

• View the model

The IBM SPSS Model Viewer is used to see the result, which you can use to explore different views of the model.

Before you can run the Model Viewer, you need to generate a project token:

• In the My Projects banner, click the More icon(the three dots ellipsis option from the top panel.) and then click Insert project token. The project token is inserted into the first cell of the notebook, before the title.  Copy and paste it to the next cell.
• Copy the text, which appears at the beginning of the notebook, into the following cell and run it.

Run the code in the following cell to start the SPSS Model Viewer, where you can see a visualization and model related information.

// @hidden_cell
// The project token is an authorization token that is used to access project resources like data sources, connections, and used by platform APIs.
import com.ibm.analytics.projectNotebookIntegration._
val pc = ProjectUtil.newProjectContext(sc, "*********", "*********")

import com.ibm.spss.scala.ModelViewer
kernel.magics.html(ModelViewer.toHTML(pc,srfModel))

• Export the XML files (PMML, StatsXML) for other detail statistics

By exporting your results to different formats, such as Predictive Model Markup Language (PMML) or statsXML format you can share your model and related statistics outside of IBM Watson Studio.

import java.io.{File, PrintWriter}

srfModel.toPMML("randomTrees_pmml.xml")
val statXML = srfModel.statXML()
new PrintWriter("StatXML.xml") {
      write(statXML)
      close
}

Now the example for predicting churn on the Watson Studio is finished, so the contents of the video have been completely shown. We hope that it is useful.

Finally, many Thanks to my colleagues Zhiyuan Wang, Steve Barbee, Sherry Zhang, Sier Han and Jing Xu, who helped me very much in the materials preparation and article reviewing and revising.