Code for the seminar "Computational Empirical Research" 2023 at the University of Leipzig.
This project deals with the task "Image Classification for Argumentation" for the Touché 2023 dataset. It attempts to classify argumentative images into a relevant topic using extracted annotation labels from the computer vision APIs Google Cloud Vision and Clarifai. For this purpose, the results of different classification models are compared with each other. Our approach follows the AVCA framework of Araujo et al.
Araujo, T., Lock, I., van de Velde, B.: Automated visual content analysis (avca) in communication research: A protocol
for large scale image classification with pre-trained computer vision models. Communication Methods and Measures 14(4),
239–265 (2020), https://doi.org/10.1080/19312458.2020.1810648
Please download the touche23-image-search-main.zip an topics.xml from https://zenodo.org/record/7628213
Save unzipped dataset in data/touche23-image-search-main and topics under data/topics.xml
To check the actual quality of the dataset, we adopted the Aramis ImArg annotation platform and modified it slightly.
Start annotation server with "python start_annotation.py -web". Please append "/evaluation" to your local IP address. The annotation results will be saved at "working/image_eval.csv".
For Clarifai API usage please save Personal Access Token (PAT) at working/pat.txt. The Clarifai model can be specified in computer_vision.computer_vision.py (Default: MODEL_ID = 'general-image-recognition').
run_clarifai(size_dataset: int = -1, set_seed: bool = True, topic_ids: List[int] = None)
We also provide exploratory data analysis methods that can be applied to the Touché, Clarifai and combined data set labels.
run_exploratory_data_analysis(size_dataset: int = -1, use_clarifai_data: bool = False,
use_combined_data: bool = False, topic_ids: List[int] = None)
In addition, a syntactic matching of the Touché and Clarifai labels can be calculated using an exact matching and Levenshtein similarity matching approach.
run_label_matching(size_dataset=900, topic_ids=[51, 55, 76, 81, 100])
The classifier class can be used to carry out the training and evaluation of classifiers, as well as performing and evaluating a grid search on a specified classifier. The class was specially designed to handle data sets of computer vision generated image labels, like they can be generated with the preprocessing module of this project. The evaluation has an option to save results to a JSON-file. Generated JSON documents contain all relevant information about the classifier, like name, parameters, etc. as well as relevant evaluation metrics.
An example for the use of the class can be seen below:
data = load_dataset(size_dataset=900, topic_ids=[55,76,81])
clf = Classifier(SVC(), data, topic_id=55, grid_params=grid_params)
clf.evaluate_grid_search()
Where SVC()
is the passed classifier, load_dataset
is from the preprocessing module and creates the dataset,
topic_id
is one of the topic ids present in data
and grid_params
is a valid parameter configuration of the
specified classifier (which in this case is the default Support Vector Classifier from Scikit-Learn). For further use,
refer to the class documentation.
In this project, the used configurations for the grid search are defined in themodels.json
file.
Different evaluation tasks can be performed within this project.
dataset_evaluation.py
evaluates the annotations of the Touché23 data set by the two authors and saves them as
CSV-files. The same evaluations are additionally available in a notebook format. exploratory_data_analysis.py
creates
a markdown file with an extensive exploratory analysis of the outputs of the used computer vision APIs.
evaluation_of_results.ipynb
contains the evaluations of the tested model outputs in a notebook format.