The chunk_overlap parameter is not documented in the Knowledge API specification.

[yagiyuki]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Provide a description of requested docs changes

It is possible to specify "chunk_overlap" as a parameter for document creation in the Knowledge API.
However, this is not mentioned in the API specifications, so an addition is required.

ex: Create a Document from a Fileにおけるchunk_overlap

curl \
  --location \
  --request POST \
  "http://localhost/v1/datasets/${dataset_id}/document/create-by-file" \
  --header "Authorization: Bearer ${api_key}" \
  --form 'data={
    "indexing_technique": "high_quality",
    "process_rule": {
      "rules": {
        "pre_processing_rules": [
          { "id": "remove_extra_spaces", "enabled": true },
          { "id": "remove_urls_emails",  "enabled": true }
        ],
        "segmentation": {
          "separator": "###",
          "max_tokens": 500,
          "chunk_overlap": 100
        }
      },
      "mode": "custom"
    }
  };type=application/json' \
  --form 'file=@"test.txt"'

[yagiyuki]

target or modification

• Create a Document from Text
  • Modifying the sample request : unnecessary
• Create a Document from a File
  • Modifying the sample request : necessary
• Update a Document with Text
  • Modifying the sample request : unnecessary
• Update a Document with a File
  • Modifying the sample request : necessary

[yagiyuki]

• Create a Document from Text

curl \
  --location \
  --request POST \
  "http://localhost/v1/datasets/${dataset_id}/document/create-by-text" \
  --header "Authorization: Bearer ${api_key}" \
  --header "Content-Type: application/json" \
  --data '{
    "name": "create-by-text",
    "text": "create-by-text",
    "indexing_technique": "high_quality",
    "process_rule": {
      "rules": {
        "pre_processing_rules": [
          { "id": "remove_extra_spaces", "enabled": true },
          { "id": "remove_urls_emails",  "enabled": true }
        ],
        "segmentation": {
          "separator": "###",
          "max_tokens": 300,
          "chunk_overlap": 30
        }
      },
      "mode": "custom"
    }
  }'

• Create a Document from a File

curl \
  --location \
  --request POST \
  "http://localhost/v1/datasets/${dataset_id}/document/create-by-file" \
  --header "Authorization: Bearer ${api_key}" \
  --form 'data={
    "indexing_technique": "high_quality",
    "process_rule": {
      "rules": {
        "pre_processing_rules": [
          { "id": "remove_extra_spaces", "enabled": true },
          { "id": "remove_urls_emails",  "enabled": true }
        ],
        "segmentation": {
          "separator": "###",
          "max_tokens": 300,
          "chunk_overlap": 30
        }
      },
      "mode": "custom"
    }
  };type=application/json' \
  --form 'file=@"test.txt"'

• Update a Document with Text

curl \
  --location \
  --request POST \
  "http://localhost/v1/datasets/${dataset_id}/documents/${document_id}/update-by-text" \
  --header "Authorization: Bearer ${api_key}" \
  --header "Content-Type: application/json" \
  --data '{
    "name": "create-by-text",
    "text": "create-by-text",
    "indexing_technique": "high_quality",
    "process_rule": {
      "rules": {
        "pre_processing_rules": [
          { "id": "remove_extra_spaces", "enabled": true },
          { "id": "remove_urls_emails",  "enabled": true }
        ],
        "segmentation": {
          "separator": "###",
          "max_tokens": 300,
          "chunk_overlap": 50
        }
      },
      "mode": "custom"
    }
  }'

• Update a Document with a File

curl \
  --location \
  --request POST \
  "http://localhost/v1/datasets/${dataset_id}/documents/${document_id}/update-by-file" \
  --header "Authorization: Bearer ${api_key}" \
  --form 'data={
    "indexing_technique": "high_quality",
    "process_rule": {
      "rules": {
        "pre_processing_rules": [
          { "id": "remove_extra_spaces", "enabled": true },
          { "id": "remove_urls_emails",  "enabled": true }
        ],
        "segmentation": {
          "separator": "###",
          "max_tokens": 300,
          "chunk_overlap": 55
        }
      },
      "mode": "custom"
    }
  }' \
  --form "[email protected]"