Filter Images First, Generate Instructions Later:
Pre-Instruction Data Selection for Visual Instruction Tuning

CVPR 2025

Bardia Safaei, Faizan Siddiqui, Jiacong Xu, Vishal M. Patel, Shao-Yuan Lo

Johns Hopkins University, Honda Research Institute USA

📄 ArXiv 🔗 GitHub 📥 Dataset

Abstract

Visual instruction tuning (VIT) for large vision-language models (LVLMs) requires training on expansive datasets of image-instruction pairs, which can be costly. Recent efforts in VIT data selection aim to select a small subset of high-quality image-instruction pairs, reducing VIT runtime while maintaining performance comparable to full-scale training. However, a major challenge often overlooked is that generating instructions from unlabeled images for VIT is highly expensive. Most existing VIT datasets rely heavily on human annotations or paid services like the GPT API, which limits users with constrained resources from creating VIT datasets for custom applications. To address this, we introduce Pre-Instruction Data Selection (PreSel), a more practical data selection paradigm that directly selects the most beneficial unlabeled images and generates instructions only for the selected images. PreSel first estimates the relative importance of each vision task within VIT datasets to derive task-wise sampling budgets. It then clusters image features within each task, selecting the most representative images with the budget. This approach reduces computational overhead for both instruction generation during VIT data formation and LVLM fine-tuning. By generating instructions for only 15% of the images, PreSel achieves performance comparable to full-data VIT on the LLaVA-1.5 and Vision-Flan datasets.

Motivation

Your Figure

PreSel Framework

Method Illustration

Main Results

Results on LLaVA-1.5
Table 1: Performance comparison of PreSel on LLaVA-1.5. Our method only requires instruction generation for the selected images (15%).
Results on Vision-Flan
Table 2: Performance comparison of PreSel on Vision-Flan. Our method only requires instruction generation for the selected images (15%).
Results on Vision-Flan
Table 3: Results for LLaVA-Vicuna-13B and LLaVA-Llama-8B models.

Cost Analysis

Results on LLaVA-1.5
Table 4: Comparison of VIT costs for PreSel, other VIT data selection methods, and full-scale LVLM fine-tuning.

Dataset

The data selected by PreSel will be released soon!

© 2025 Bardia Safaei | Last updated: March 2025