Skip to content

Add TextChunkingProcessor event and info stats#2

Draft
q-andy wants to merge 1 commit intometrics-framework-cleanfrom
metrics-framework-chunker
Draft

Add TextChunkingProcessor event and info stats#2
q-andy wants to merge 1 commit intometrics-framework-cleanfrom
metrics-framework-chunker

Conversation

@q-andy
Copy link
Copy Markdown
Owner

@q-andy q-andy commented Feb 8, 2025

Description

Example PR to showcase adding a stats for TextChunkingProcessor

Event stats:

  • Total processor execution count
  • Delimiter algorithm execution count
  • Fixed length algorithm execution count

Info stats:

  • Number of processors in pipelines
  • Number of processors by algorithm in pipelines
  • Number of fixed length algorithm processors by tokenizer (standard, lowercase, whitespace, letter) in pipelines

Example response:

{
	"_nodes": {
		"total": 1,
		"successful": 1,
		"failed": 0
	},
	"cluster_name": "integTest",
	"all_nodes": {
		"ingest_processor": {
			"text_chunking": {
				"executions": 0,
				"algorithm": {
					"fixed_length": 0,
					"delimiter": 0
				}
			}
		}
	},
	"cluster_version": "3.0.0",
	"pipelines": {
		"ingest": {
			"processors": {
				"text_chunking": {
					"count": 0,
					"tokenizer": {
						"standard": 0,
						"lowercase": 0,
						"letter": 0,
						"whitespace": 0
					},
					"algorithm": {
						"fixed_length": 0,
						"delimiter": 0
					}
				}
			}
		}
	}
}

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant