git clone https://github.com/thecharm/BDoG.git
cd BDoG
pip install -e .

Download the model weights and set the model path in the BDoG/vlmeval/config.py file

torchrun --nproc_per_node=1 run.py --data ScienceQA_TEST \
                                   --stage BDebate \
                                   --debate 2

• --data
  • Dataset supported: ScienceQA_TEST and MMBench_DEV_EN.
• --stage
  • Prompt Type: BDebate(Blueprint Debate on Graph) or ODebate(Debate without Graph).
• --debate
  • Number of rounds for the debate.
• --kg_init
  • (optional) Use Gemini Graph as the initialization for multi-round debates.
• --nproc_per_node=2
  • (optional) Speed up the inference process if you have two GPUs.
• --openai
  • (optional) Use the Openai API key to perform the final result validation.

The results are saved in the BDoG/results/instructblip_13b folder.

During this process, the datasets will be automatically downloaded to the /root/LMUData/ directory. If you need to change the data storage path, please reset --lmudata.

• VLMEvalKit: An open-source evaluation toolkit of large vision-language models (LVLMs).
• LLaVA: Wounderful MLLM based on Large Language and Vision Assistant.
• LAVIS: The amazing open-sourced multimodality learning codebase.

If this repo is useful to you, please cite using this BibTeX.

@inproceedings{zheng2024picture,
  title={A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoning},
  author={Zheng, Changmeng and Liang, Dayong and Zhang, Wengyu and Wei, Xiao-Yong and Chua, Tat-Seng and Li, Qing},
  booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
  pages={419--428},
  year={2024}
}