Skip to content

updated phi4 template-update sync-unsloth-bugs #7413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ankitcoder123
Copy link

This pull request introduces the following updates for the Phi‑4 chat template to address issues during fine-tuning and inference:

  • User Prompt Formatting:
    Removed the extra assistant prompt token (<|im_start|>assistant<|im_sep|>) from the user formatter. Now, the user messages are formatted as: <|im_start|>user<|im_sep|>{{content}}<|im_end|>
    This change ensures that an assistant prompt is only added when explicitly needed (e.g., via add_generation_prompt=True).

  • EOS Token Replacement:
    Ensured that the EOS token is properly replaced with <|im_end|> by setting replace_eos=True in the template registration.

  • Dedicated PAD Token:
    Modified the fix_special_tokens method so that if no PAD token is defined, a dedicated pad token (<|dummy_85|>) is assigned rather than reusing the EOS token. This prevents infinite generations during fine-tuning.

  • Jinja Template Override:
    Enabled replace_jinja_template=True to guarantee that the custom template is used without default additions.

  • Motivation:
    These updates address observed issues in the inference outputs (unexpected tokens and out-of-context responses) when using VLLM for Phi‑4. By refining the chat template and tokenizer special token settings, the model's prompt structure now aligns with the intended design, leading to more accurate and consistent results.

  • Changes Made:
    Updated the register_template call in the template registration module.
    Revised the fix_special_tokens function to use <|dummy_85|> as the pad token.
    Tested changes with VLLM to ensure the prompt no longer includes the extra assistant prompt.

  • reference links

  • https://huggingface.co/microsoft/phi-4/commit/6fbb3d3bbe726c99b4188087b4deeec1bceac5ae#d2h-782009

  • https://www.reddit.com/r/MachineLearning/comments/1i23zbo/p_how_i_found_fixed_4_bugs_in_microsofts_phi4/?rdt=63478

  • https://unsloth.ai/blog/phi4

@hiyouga hiyouga added the pending This problem is yet to be addressed label Mar 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pending This problem is yet to be addressed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants