有没有简单的办法不shuffle trainning数据集

[XuanRen4470]

我知道trainner会shuffle，但是我不希望training数据集被shuffle。

[histmeisah]

现在有参数配置可以不shuffle 数据集吗,想做一下课程学习

[JerryDaHeLian]

同问！

[hiyouga]

--streaming --buffer_size 1 不会 shuffle 数据集

[iaoxuesheng]

--streaming --buffer_size 1 不会 shuffle 数据集

你好，在设置 --streaming True
--buffer_size 1 \时会报错：Traceback (most recent call last):
File "src/train_bash.py", line 14, in
main()
File "src/train_bash.py", line 5, in main
run_exp()
File "/cephfs/renjinshan/work/LLaMA-Factory-0.7.0/src/llmtuner/train/tuner.py", line 33, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "/cephfs/renjinshan/work/LLaMA-Factory-0.7.0/src/llmtuner/train/sft/workflow.py", line 33, in run_sft
dataset = get_dataset(model_args, data_args, training_args, stage="sft", **tokenizer_module)
File "/cephfs/renjinshan/work/LLaMA-Factory-0.7.0/src/llmtuner/data/loader.py", line 176, in get_dataset
print_function(next(iter(dataset)))
File "/cephfs/renjinshan/work/miniconda3/envs/llama_factory/lib/python3.8/site-packages/datasets/iterable_dataset.py", line 1384, in iter
for key, example in ex_iterable:
File "/cephfs/renjinshan/work/miniconda3/envs/llama_factory/lib/python3.8/site-packages/datasets/iterable_dataset.py", line 679, in iter
yield from self._iter()
File "/cephfs/renjinshan/work/miniconda3/envs/llama_factory/lib/python3.8/site-packages/datasets/iterable_dataset.py", line 718, in _iter
transformed_batch.update(self.function(*function_args, **self.fn_kwargs))
File "/cephfs/renjinshan/work/LLaMA-Factory-0.7.0/src/llmtuner/data/preprocess.py", line 79, in preprocess_supervised_dataset
if len(examples["prompt"][i]) % 2 != 1 or len(examples["response"][i]) != 1:
TypeError: object of type 'NoneType' has no len()