decodIRT

ATENÇÃO: Fork experimental para a integração com a interface! Muito pode ser diferente.

A IRT application using OpenML database. The objective is to perform the complexity analysis of binary datasets, automatically downloaded from the OpenML platform, and also to compare the performance of several supervised learning algorithms, relating their performances to the instances of the datasets. For this we used the calculation of the IRT item parameters as well as the calculation of the probability of correctness of each classifier for each test instance of a given dataset.

This tool was presented on the paper "Decoding machine learning benchmarks" published at the BRACIS'2020 conference. For more details on the results and supplementary paper material, access the BRACIS folder.

How to use

decodIRT is a ready-to-use tool, just download and run.

Some dependencies need to be installed for the tool to work. They are listed below.

Dependencies

• OpenML API
• Scikit-Learning
• Numpy
• Pandas
• Matplotlib
• Package rpy2 (It is necessary to have R installed)
• Catsim

Once all the necessary packages are installed, we can move on to running the decodIRT tool.

Running decodIRT

The decodIRT tool is made up of three scripts written in Python, made for sequential use. The image below shows how to use the scripts in sequence.

Tutorial decodIRT as tool

Here is a brief tutorial on how to use decodIRT. For details of existing parameters and generated results, see the documentation files.

This tutorial is divided into three parts that correspond to each of the scripts.

decodIRT_OtML

Although the tool can analyze multiple datasets, in this tutorial only one will be used. The chosen dataset is Credit-g, which can be accessed at https://www.openml.org/d/31.

The first script will be executed using the -data parameter to enter the dataset ID to be downloaded. If more than one dataset, IDs can be passed in list form.

$ python decodIRT_OtML -OpenID 31

Information about the chosen dataset and the execution of the classifiers is displayed on screen during program execution. After execution, all generated data will be saved in 6 different files in a directory called "output".

The output directory can be changed if the script is executed with the -output parameter.

decodIRT_MLtIRT

This script is in charge of calculating IRT parameters using the R language. But don't worry! The script takes care of installing the R packages.

To run this script, just enter the directory where the generated files were saved and the script does the rest.

$ python decodIRT_MLtIRT -dir dir_name

In this case the default name (output) was left for the output directory in decodIRT_OtML so it would not be necessary to pass the -dir parameter. The result of this script will be a file containing the TRI item parameters for 30% of the dataset that was used for testing after a 70/30 split. In case the dataset is very large, the split is adapted to leave 500 instances for testing, the rest is used in training.

decodIRT_analysis

The decodIRT_analysis script, as the name already suggests, aims to perform analyzes using TRI and provide the user with graphs that allow the behavior of different classifiers to be visualized on dataset instances. In addition to showing the percentage of instances that have high values of Discrimination, Difficulty and Guessing.

$ python decodIRT_analysis -dir dir_name -plotAllHist -plotAllCCC -save

This command will generate all histograms, curve graphs and save them into the directory where the dataset was downloaded. If the -save parameter is not passed, all requested graphics will be shown on the run screen.

There are more parameters that allow the user to control the value limit of each item parameter, the number of histogram bins and for which dataset and item parameter (Discrimination, Difficulty, Guessing) the graphs will be generated.

It is also possible to calculate a score for the classifiers. For this, the IRT True-Score concept was implemented. The following command will perform this calculation and generate a graph comparing the scores of the classifiers.

$ python decodIRT_analysis -dir dir_name -scoreAll -save

Tutorial decodIRT as package

The decodIRT can now be used as a library. Allowing the tool's scripts and functions to be used separately. In addition, you can now create a benchmark from the IRT assessment generated by the decodIRT and publish it directly to OpenML as a new study.

To do this, download the python code and import the create_benchmark.py module. The module will be able to execute all steps of the decodIRT tool and create a new benchmark from datasets downloaded from OpenML.

A list of OpenML datasets will be passed to the code to be analyzed by the IRT. The user can choose which item parameters and the percentage of the datasets he plans to filter.

decodIRT_create_benchmark

There are two ways to create a new benchmark. In the first, you can only use a study ID that contains the datasets of interest. For example:

from decodIRT_create_benchmark import analyseBenchmarkStudy

perc = 15 #This is the percentage of datasets that you want to filter the datasets

param = ['dis','dif','ges'] #This is the list of parameters that you want to use to filter the datasets

study_id = 253 #This is the study created with the 60 datasets of OpenML-CC18 used in the paper "Decoding machine learnig benchmarks"

benchmark,text = analyseBenchmarkStudy(study_id,perc,param) #This function will use the study datasets to create the new benchmark

In the second form, you can use the OpenML API function list_tasks to search for datasets on the platform. For example:

import openml
from decodIRT_create_benchmark import analyseBenchmarkDataId

perc = 15 #This is the percentage of datasets that you want to filter the datasets

param = ['dis','dif','ges'] #This is the list of parameters that you want to use to filter the datasets

#From the tasks found, you can list the datasets
tasks = openml.tasks.list_tasks(task_type_id = 1,number_instances='400..2000', number_features='1..15', size=60)

benchmark,text = analyseBenchmarkDataId(tasks,perc,param) #This function will use the task datasets to create the new benchmark

And finally, using your OpenML apikey. A study can be published with the new benchmark created.

from decodIRT_create_benchmark import publishStudy

openml_apikey = 'yourapikey' #Apikey can be found on your OpenML profile page

study_id = publishStudy(benchmark,text,perc,openml_apikey,'New benchmark name') #This function will publish the study and return the ID

Author's Note

I hope this tool can help you realize interesting insights and assist you in your research!