lora微调Minicpm-o遇到错误ValueError: Target module MultiModalProjector is not supported. Currently, only the following modules are supported: torch.nn.Linear, torch.nn.Embedding, torch.nn.Conv2d, transformers.pytorch_utils.Conv1D.

[RocksyWhite]

Reminder

• I have read the above rules and searched the existing issues.

System Info

• llamafactory version: 0.9.2.dev0
• Platform: Linux-5.4.0-156-generic-x86_64-with-glibc2.35
• Python version: 3.11.10
• PyTorch version: 2.4.0+cu121 (GPU)
• Transformers version: 4.48.3
• Datasets version: 3.2.0
• Accelerate version: 1.2.1
• PEFT version: 0.12.0
• TRL version: 0.9.6
• GPU type: NVIDIA A100-PCIE-40GB
• GPU number: 1
• GPU memory: 39.39GB

Reproduction

使用的命令:

### model
model_name_or_path: /hy-tmp/OpenBMB/MiniCPM-o-2_6
trust_remote_code: True

### method
stage: sft
do_train: true
finetuning_type: lora
optim: paged_adamw_8bit
freeze_vision_tower: true

### dataset
dataset: handwriting
template: minicpm_o
cutoff_len: 30720
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 8

### output
output_dir: /hy-tmp/OpenBMB/handwriting
logging_steps: 1
save_steps: 500
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 2
learning_rate: 1.0e-5
num_train_epochs: 1.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000

### eval
do_eval: false

遇到问题:

Traceback (most recent call last):
  File "/usr/local/bin/llamafactory-cli", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/LLaMA-Factory/src/llamafactory/cli.py", line 112, in main
    run_exp()
  File "/root/LLaMA-Factory/src/llamafactory/train/tuner.py", line 93, in run_exp
    _training_function(config={"args": args, "callbacks": callbacks})
  File "/root/LLaMA-Factory/src/llamafactory/train/tuner.py", line 67, in _training_function
    run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
  File "/root/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 52, in run_sft
    model = load_model(tokenizer, model_args, finetuning_args, training_args.do_train)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/LLaMA-Factory/src/llamafactory/model/loader.py", line 169, in load_model
    model = init_adapter(config, model, model_args, finetuning_args, is_trainable)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/LLaMA-Factory/src/llamafactory/model/adapter.py", line 299, in init_adapter
    model = _setup_lora_tuning(
            ^^^^^^^^^^^^^^^^^^^
  File "/root/LLaMA-Factory/src/llamafactory/model/adapter.py", line 250, in _setup_lora_tuning
    model = get_peft_model(model, lora_config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/peft/mapping.py", line 183, in get_peft_model
    return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/peft/peft_model.py", line 1542, in __init__
    super().__init__(model, peft_config, adapter_name, **kwargs)
  File "/usr/local/lib/python3.11/dist-packages/peft/peft_model.py", line 155, in __init__
    self.base_model = cls(model, {adapter_name: peft_config}, adapter_name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/peft/tuners/lora/model.py", line 139, in __init__
    super().__init__(model, config, adapter_name)
  File "/usr/local/lib/python3.11/dist-packages/peft/tuners/tuners_utils.py", line 175, in __init__
    self.inject_adapter(self.model, adapter_name)
  File "/usr/local/lib/python3.11/dist-packages/peft/tuners/tuners_utils.py", line 431, in inject_adapter
    self._create_and_replace(peft_config, adapter_name, target, target_name, parent, current_key=key)
  File "/usr/local/lib/python3.11/dist-packages/peft/tuners/lora/model.py", line 224, in _create_and_replace
    new_module = self._create_new_module(lora_config, adapter_name, target, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/peft/tuners/lora/model.py", line 346, in _create_new_module
    raise ValueError(
ValueError: Target module MultiModalProjector(
  (linear1): Linear(in_features=1024, out_features=3584, bias=True)
  (relu): ReLU()
  (linear2): Linear(in_features=3584, out_features=3584, bias=True)
) is not supported. Currently, only the following modules are supported: `torch.nn.Linear`, `torch.nn.Embedding`, `torch.nn.Conv2d`, `transformers.pytorch_utils.Conv1D`.

Others

参考其他教程后设置lora_target: linear1,linear2后会遇到类似
Target modules {'linear1','linear2'} not found in the base model
的错误

[hiyouga]

fixed