The slicing method is invalid when setting the block identifier under general

[chaijl]

In the general mode of the slicing method, I uploaded a very simple txt file with 4 # characters between each paragraph. I also configured the '#######' segment identifier in the block setting. The text block size settings range from 500-1000 and are not segmented according to this identifier. If there is a # character in the text, it will be deleted by default after document parsing. Attached are my configuration screenshots and test files.
测试数据.txt

[dosubot[bot]]

I found a similar resolved issue that might be relevant to your situation. The issue discussed how to handle text block segmentation when special characters are used as delimiters in RAGFlow. The suggested solution was to:

1. Use the correct backtick symbol when specifying the delimiter to ensure proper recognition and removal. The RAGFlowTxtParser class processes delimiters by extracting them from within backticks and also considers individual characters in the delimiter string.
2. Adjust the chunk token number to a lower value (e.g., 50 instead of the default 128) to prevent unwanted paragraph combinations and ensure better separation of text blocks.
3. Consider pulling the 'nightly' version of the docker image to incorporate the recent update that resolves special delimiter parsing issues ^[1][2][3].

You might want to try these steps to see if they resolve the issue with the block identifier not working as expected in the general mode.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[ZhenhangTung]

@chaijl Thanks for the report. I will do a double check and get back to you.

[asiroliu]

A delimiter or separator can consist of one or multiple special characters. If it is multiple characters, ensure they are enclosed in backticks( ``)

[chaijl]

I confirm that I used ('') to enclose the 4 # s, which is invalid, and after parsing, it will automatically remove the # s that originally existed in my text. For example, in the text 1 # workshop, after parsing, it becomes 1 workshop, which is missing one#

[chaijl]

After testing multiple times, I found the problem. I don't know if the system has this issue. As long as there is a character in my text that is the same as the ('') delimiter, the delimiter will be invalid, and this character will also be cleared in normal text. Just now, because my text has the symbol #, but my delimiter is 4 #, thank you for your reply, teachers

[asiroliu]

Note: These should be backticks (`) not single quotes (')