Update tokenizer.model_max_length if necessary (Fix #6415)

[xiaosu-zhu]

What does this PR do?

Fixes #6415

It modifies logic of loader.py -> load_tokenizer(...) to update tokenizer.model_max_length if model_args.model_max_length changes.

Before submitting

• Did you read the contributor guideline?
• Did you write any new necessary tests?

[hiyouga]

Hi Xiaosu, thanks for your concern about the handling of tokenizer in llama factory. We thought that this property does not affect both the training and inference, so we would like not to modify it in our framework.

[xiaosu-zhu]

Thank you for your comments! You are right there will be no defects to do training and/or inference, except one place:

The VLLM inference will read the model_max_length as a hard token limit for generation.

If we create an vllm engine by LLM(max_model_len=something_larger) which is longer than model_max_length, it will raise an error and we need to set VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 to bypass it.

Therefore, I think this pr makes less confusion for further use cases.

[hiyouga]

LGTM

[xiaosu-zhu]

Thank you for your quick response ❤️