.Net: Fix TextChunker orphan merge token count

[pragnyanramtha]

Summary

Fixes #13713. TextChunker.SplitPlainTextParagraphs now checks the configured token counter before merging a short final paragraph into the previous paragraph. This prevents the orphan-paragraph balancing step from creating a chunk that exceeds the requested token limit when a custom token counter is used.

Root Cause

The orphan merge logic compared the number of whitespace-delimited words in the last two paragraphs against adjustedMaxTokensPerParagraph. That word count can be lower than the actual token count reported by the provided TokenCounter, so the merge could produce an oversized chunk.

Change

• Build the candidate merged paragraph using the existing paragraph strings.
• Call GetTokenCount(mergedParagraph, tokenCounter) before merging.
• Add regression tests using a length-based token counter for both the oversized merge case and the still-fits merge case.

Validation

Using local .NET SDK 10.0.100 installed under /tmp/dotnet:

• PATH=/tmp/dotnet:$PATH DOTNET_ROOT=/tmp/dotnet DOTNET_CLI_TELEMETRY_OPTOUT=1 dotnet test dotnet/src/SemanticKernel.UnitTests/SemanticKernel.UnitTests.csproj --filter TextChunkerTests -> 41 passed
• PATH=/tmp/dotnet:$PATH DOTNET_ROOT=/tmp/dotnet DOTNET_CLI_TELEMETRY_OPTOUT=1 dotnet format dotnet/src/SemanticKernel.UnitTests/SemanticKernel.UnitTests.csproj --include dotnet/src/SemanticKernel.Core/Text/TextChunker.cs dotnet/src/SemanticKernel.UnitTests/Text/TextChunkerTests.cs --verify-no-changes --no-restore
• git diff --check

Full repository test suite was not run because the focused TextChunker unit tests cover the changed behavior.

[github-actions]

Automated Code Review

Reviewers: 4 | Confidence: 93% | Result: All clear

Reviewed: Correctness, Security Reliability, Test Coverage, Design Approach

---

Automated review by pragnyanramtha's agents

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.