Utf16.IsValid and Utf8/16.IndexOfInvalidSubsequence by lilinus · Pull Request #120326 · dotnet/runtime

lilinus · 2025-10-02T14:24:49Z

Copilot

Pull Request Overview

This PR adds new UTF-16 validation APIs to System.Text.Unicode, introducing Utf16.IsValid and Utf16.IndexOfInvalidSubsequence methods alongside similar functionality for UTF-8. It also refactors existing code to use these new standardized validation APIs instead of custom validation logic.

Adds new Utf16 class with validation methods IsValid and IndexOfInvalidSubsequence
Adds Utf8.IndexOfInvalidSubsequence method to complement existing Utf8.IsValid
Refactors multiple codebases to use the new standardized validation APIs

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
System.Private.CoreLib.Shared.projitems	Adds the new Utf16.cs file to the build
Utf16.cs	New class implementing UTF-16 validation methods
Utf8.cs	Adds IndexOfInvalidSubsequence method and updates class documentation
System.Runtime.cs	Adds public API surface for new UTF-16 and UTF-8 validation methods
HttpEncoder.cs	Refactors surrogate validation to use new Utf16.IndexOfInvalidSubsequence
Normalization.Icu.cs	Replaces custom validation logic with Utf16.IsValid
StringSearchValues.cs	Simplifies surrogate validation using Utf16.IsValid
Utf8UtilityTests.ValidateBytes.cs	Adds test coverage for new Utf8.IndexOfInvalidSubsequence method
Utf16UtilityTests.ValidateChars.cs	Adds test coverage for new Utf16 validation methods

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

dotnet-policy-service · 2025-10-27T08:17:40Z

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

jeffhandley · 2026-02-01T19:00:55Z

@lilinus Thanks for your patience on this; we are planning to review it this month. I've updated the branch and CI will re-run, but there was an ApiCompat error in the build previously indicating Utf8.IndexOfInvalidSubsequence wasn't available in the net8.0 target and it wasn't obvious to me why that was the case.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

Update XML doc comments and remove unneeded using statements. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

...tem.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8UtilityTests.ValidateBytes.cs

…/Text/Unicode/Utf8UtilityTests.ValidateBytes.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Remove unnecessary blank line at the beginning of Utf16.cs

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

jeffhandley · 2026-02-01T19:28:58Z

src/libraries/System.Private.CoreLib/src/System/SearchValues/Strings/StringSearchValues.cs

            }
        }

        private static bool ContainsIncompleteSurrogatePairs(ReadOnlySpan<string> values)


I recommend renaming this private method to ContainsInvalidValues to since the implementation no longer directly checks the surrogate pairs.

jeffhandley · 2026-02-01T19:36:11Z

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

+        public static unsafe int IndexOfInvalidSubsequence(ReadOnlySpan<char> value)
+        {
+            fixed (char* pValue = &MemoryMarshal.GetReference(value))
+            {
+                char* pFirstInvalidChar = Utf16Utility.GetPointerToFirstInvalidChar(pValue, value.Length, out _, out _);
+                int index = (int)(pFirstInvalidChar - pValue);
+
+                return (index < value.Length) ? index : -1;
+            }
+        }


@jozkee We're trying to minimize our unsafe code additions, but I suspect this one is appropriate. Please assess this yourself and consult with @tannergooding and @PranavSenthilnathan if you agree it seems to be an appropriate use case.

If we do conclude this is an appropriate unsafe addition, please add a comment or a remarks section justifying it.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Copilot · 2026-02-16T04:34:29Z

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

        /// </summary>
        /// <param name="value">The <see cref="ReadOnlySpan{T}"/> string.</param>
        /// <returns><c>true</c> if value is well-formed UTF-8, <c>false</c> otherwise.</returns>
        public static bool IsValid(ReadOnlySpan<byte> value) =>


The XML docs for Utf8.IsValid still describe the input as a "ReadOnlySpan string", but the parameter is a UTF-8 byte span. Please update the text to describe a UTF-8 byte sequence (similar wording to the new IndexOfInvalidSubsequence docs) so the public API docs are accurate.

lilinus added 5 commits September 29, 2025 14:57

Add methods

3b6fdfb

Add tests

878724a

Use Utf16 CoreLib

9705165

Use helper in HttpEncoder

e2439c5

Add removed line ending

5edcb5c

Copilot AI review requested due to automatic review settings October 2, 2025 14:24

github-actions bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Oct 2, 2025

dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Oct 2, 2025

Copilot AI reviewed Oct 2, 2025

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs Outdated Show resolved Hide resolved

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs Outdated Show resolved Hide resolved

build-analysis bot mentioned this pull request Oct 2, 2025

[android] Android.Device_Emulator.JIT.Test failing on emulators with CoreCLR #112633

Open

lilinus added 2 commits October 3, 2025 09:28

Revert Normalization.Icu.cs change

307404b

Address copilot feedback

e0868b9

This was referenced Oct 3, 2025

STATUS_UNSUCCESSFUL in RsaCryptRoundtrip_OaepSHA1 #29683

Open

RSA 384 failing in Windows #120353

Closed

MihuBot mentioned this pull request Oct 7, 2025

[✨ Triage] dotnet/runtime#120499 by ericstj - System.Net docs for .NET 10.0 APIs MihuBot/runtime-utils#1562

Closed

IDisposable approved these changes Oct 18, 2025

View reviewed changes

marek-safar added the area-System.Runtime label Oct 27, 2025

jeffhandley removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jan 14, 2026

jeffhandley assigned jozkee Jan 14, 2026

Merge branch 'main' into utf8-utf16-validation

a0bd5e9

Copilot AI review requested due to automatic review settings February 1, 2026 18:58

Copilot started reviewing on behalf of jeffhandley February 1, 2026 18:59 View session

Copilot AI reviewed Feb 1, 2026

View reviewed changes

Apply suggestions from code review

b6cd454

Update XML doc comments and remove unneeded using statements. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings February 1, 2026 19:32

Copilot started reviewing on behalf of jeffhandley February 1, 2026 19:32 View session

Copilot AI reviewed Feb 1, 2026

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs Show resolved Hide resolved

...tem.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8UtilityTests.ValidateBytes.cs Outdated Show resolved Hide resolved

build-analysis bot mentioned this pull request Feb 1, 2026

Cannot find 'arm64-v8a' device dotnet/dnceng#2284

Open

3 tasks

This was referenced Feb 1, 2026

iOS.Device test WorkItemExecutions #122874

Open

XHarness package install failure on iOS due to devicectl NSPOSIXErrorDomain error 49 #123796

Open

Update src/libraries/System.Runtime/tests/System.Runtime.Tests/System…

d4e15e6

…/Text/Unicode/Utf8UtilityTests.ValidateBytes.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings February 4, 2026 11:26

Remove blank line in Utf16.cs

36a2cc9

Remove unnecessary blank line at the beginning of Utf16.cs

Copilot AI reviewed Feb 4, 2026

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs Show resolved Hide resolved

This was referenced Feb 4, 2026

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

[android-arm64] The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#6408

Open

jeffhandley reviewed Feb 16, 2026

View reviewed changes

Merge branch 'main' into utf8-utf16-validation

d381f18

jeffhandley reviewed Feb 16, 2026

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs Outdated Show resolved Hide resolved

Apply XML doc comment suggestion

fb36007

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings February 16, 2026 04:29

Copilot started reviewing on behalf of jeffhandley February 16, 2026 04:30 View session

Copilot AI reviewed Feb 16, 2026

View reviewed changes

Conversation

lilinus commented Oct 2, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

dotnet-policy-service bot commented Oct 27, 2025

Uh oh!

jeffhandley commented Feb 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

jeffhandley Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

jeffhandley Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants