Skip to content

Latest commit

 

History

History
26 lines (22 loc) · 1.27 KB

README.md

File metadata and controls

26 lines (22 loc) · 1.27 KB
Raw

Multi-modality Unified Image-sequence Classifier (MUIsC)

Paper: Automatic Generation of Product-Image Sequence in E-commerce
In KDD 2022 ADS

This repo includes the inference code for MUIsC, a main module of our AGPIS (Automatic Generation of Product-Image Sequence) framework for JD.com.

Environment setup

  • conda create -n agpis python=3.6
  • pip3 install torch torchvision torchaudio
  • Go to ./transformers/ and run pip install -e .

Description

  • The main code is at ./muisc_inference.py. You can find input data processing code in function build_data; model loading code in function load_model (instantiation of language tower and image tower and their interaction are also here); and inference code is in function inference.
  • muisc_model.py contains data iterator and image-tower modeling.
  • Main changes in Hugging Face transformers:
    • Make GPT2 support optional cross-attention.
    • Fixed a bug in GPT2DoubleHeadsModel's classification head
  • The model checkpoint is not available for some reason, sorry.

Update

  • Performance could be further improved if
    • Simultaneously predicting product title in the training stage
    • Adding product category to its title
    • Training lm_head (a linear layer) from scratch