UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models

This repository is the official implementation for the paper titled "UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models".

Usage

Step 1: Install and set up environment

pip install -r requirements.txt
conda create -n unimel python==3.8.18
conda activate unimel

Step 2: Dataset

You may download WikiMEL and RichpediaMEL from https://github.com/seukgcode/MELBench and WikiDiverse from https://github.com/wangxw5/wikiDiverse.

Step 3: Train

We provide a trained checkpoint "checkpoint-2200.zip", you just need to unzip it to the current directory and record its path.

If you want to train a new checkpoint, please refer to peft (https://github.com/huggingface/peft) or swift (https://github.com/modelscope/swift).

Step 4: Run

cd UniMEL
bash run.sh 0 wikidiverse  # for wikidiverse

Code Structure

├─code
│  │  main.py
│  │  
│  └─untils
│      │  dataset.py
│      │  functions.py
│      │  
│      └─__pycache__
│              dataset.cpython-38.pyc
│              functions.cpython-38.pyc
│              
└─config
        wikidiverse.yaml
│  framework.png
│  README.md
│  requirements.txt
│  run.sh

Results

Main results

Model	Richpedia				WikiMEL				Wikidiverse
	Top-1	Top-5	Top-10	Top-20	Top-1	Top-5	Top-10	Top-20	Top-1	Top-5	Top-10	Top-20
BERT	31.6	42.0	47.6	57.3	31.7	48.8	57.8	70.3	22.2	53.8	69.8	82.8
BLINK	30.8	38.8	44.5	53.6	30.8	44.6	56.7	66.4	22.4	50.5	68.4	76.6
ARNN	31.2	39.3	45.9	54.5	32.0	45.8	56.6	65.0	-	-	-	-
DZMNED	29.5	41.6	45.8	55.2	30.9	50.7	56.9	65.1	-	39.1	-	-
JMEL	29.6	42.3	46.6	54.1	31.3	49.4	57.9	64.8	21.9	54.5	69.9	76.3
MEL-HI	34.9	43.1	50.6	58.4	38.7	55.1	65.2	75.7	27.1	60.7	78.7	89.2
HieCoAtt	37.2	46.8	54.2	62.4	40.5	57.6	69.6	78.6	28.4	63.5	84.0	92.6
GHMFC	38.7	50.9	58.5	66.7	43.6	64.0	74.4	85.8	-	-	-	-
MMEL	-	-	-	-	71.5	91.7	96.3	98.0	-	-	-	-
CLIP	60.4	96.1	98.3	99.2	36.1	81.3	92.8	98.3	42.4	80.5	91.7	96.6
DRIN	-	-	-	-	65.5	91.3	95.8	97.7	-	-	-	-
DWE	67.6	97.1	98.6	99.5	44.7	65.9	80.8	93.2	47.5	81.3	92.0	96.9
DWE+	72.5	97.3	98.8	99.6	72.8	97.5	98.9	99.7	51.2	91.0	96.3	98.9
UniMEL (ours)	94.8	97.9	98.3	98.8	94.1	97.2	98.4	98.9	92.9	97.0	99.5	99.8