Skip to content

Commit 7171070

Browse files
jyavenardjyavenard
committed
WebCodec's VideoDecoders don't handle colorspace properly
https://bugs.webkit.org/show_bug.cgi?id=282741 rdar://139412312 Reviewed by Youenn Fablet. Pass colorspace information to the created CVPixelBuffer. We add utility methods to construct the colorspace data from the vpcC box and VPx bytestream should the information not be provided on construction. We prefer colorspace information from this source order given colorspace > description data (vpcC) > inband bytestream. Added tests verifying that black are pure black, and yellow are almost pure yellow with 601 videos and video range. Technically they should be exactly pure (255, 255, 0), however, compression artifacts with the source makes it not so. Fly-by #1: m_isClosed can be accessed concurrently on the decoder's or caller's workqueue. Make it atomic. Fly-by #2: Make relevant members const and add annotation about where some members can be accessed from. * LayoutTests/media/content/test-h264-601-videorange.mp4: Added. * LayoutTests/media/content/test-vp8-601-videorange.webm: Added. * LayoutTests/media/content/test-vp9-601-videorange.webm: Added. * LayoutTests/media/media-source/media-source-vp8-hiddenframes.html: We can reduce the fuzz range now that both the VT decoder (mac) will return the same colours as VideoDecoder (ios family) * LayoutTests/media/media-video-fullrange.html: Wait a maximum of 500ms for the promise to be resolved as the rVFC callback may not always be called. * LayoutTests/media/media-video-videorange-expected.txt: Added. * LayoutTests/media/media-video-videorange.html: Added. * LayoutTests/media/media-vp8-hiddenframes.html: We can reduce the fuzz range now that both the VT decoder (mac) will return the same colours as VideoDecoder (ios family) * LayoutTests/platform/mac-wk1/TestExpectations: * LayoutTests/platform/wpe/TestExpectations: * Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp: (WebCore::createVideoDecoderConfig): * Source/WebCore/platform/VideoDecoder.h: * Source/WebCore/platform/graphics/VP9Utilities.cpp: (WebCore::vPCodecConfigurationRecordFromVPXByteStream): (WebCore::convertToPlatformVideoColorPrimaries): (WebCore::convertToPlatformVideoTransferCharacteristics): (WebCore::convertToPlatformVideoMatrixCoefficients): (WebCore::colorSpaceFromVPCodecConfigurationRecord): (WebCore::vpcCFromVPXByteStream): Deleted. * Source/WebCore/platform/graphics/VP9Utilities.h: * Source/WebCore/platform/graphics/cocoa/CMUtilities.h: * Source/WebCore/platform/graphics/cocoa/CMUtilities.mm: (WebCore::convertToCMTransferFunction): Add transfer value for BT601 (smpte170m) which is the same as 709.2 transfer. (WebCore::attachColorSpaceToPixelBuffer): * Source/WebCore/platform/graphics/cocoa/VP9UtilitiesCocoa.mm: Move methods to VP9Utilities.cpp (WebCore::convertToMatrixCoefficients): (WebCore::createVideoInfoFromVPCodecConfigurationRecord): (WebCore::convertToPlatformVideoColorPrimaries): Deleted. (WebCore::convertToPlatformVideoTransferCharacteristics): Deleted. (WebCore::convertToPlatformVideoMatrixCoefficients): Deleted. * Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.h: * Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm: (WebCore::WebCoreDecompressionSession::decodeSampleInternal): Retrieve colorspace from CMSampleBuffer and give it to the decoder initialization. (WebCore::WebCoreDecompressionSession::enqueueDecodedSample): Fly-by: the last video frame in a webm doesn't have a duration. A logic error would have caused to never notify the listener that the frame at currentTime had been decoded, leading to the play() promise to never be resolved (nor rVFC callback to be called) (WebCore::WebCoreDecompressionSession::initializeVideoDecoder): * Source/WebCore/platform/libwebrtc/LibWebRTCVPXVideoDecoder.cpp: (WebCore::LibWebRTCVPXInternalVideoDecoder::decode): (WebCore::LibWebRTCVPXInternalVideoDecoder::LibWebRTCVPXInternalVideoDecoder): (WebCore::LibWebRTCVPXInternalVideoDecoder::createPixelBuffer): (WebCore::LibWebRTCVPXInternalVideoDecoder::Decoded): Canonical link: https://commits.webkit.org/286474@main
1 parent 4010026 commit 7171070

22 files changed

+316
-148
lines changed

‎LayoutTests/media/content/test-h264-601-videorange.mp4‎

10.6 KB
Binary file not shown.

‎LayoutTests/media/content/test-vp8-601-videorange.webm‎

7.1 KB
Binary file not shown.

‎LayoutTests/media/content/test-vp9-601-videorange.webm‎

14.4 KB
Binary file not shown.

‎LayoutTests/media/media-source/media-source-vp8-hiddenframes.html‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,15 @@
11
<html>
22
<head>
3-
<meta name="fuzzy" content="maxDifference=0-16; totalPixels=0-57600" />
3+
<meta name="fuzzy" content="maxDifference=3-16; totalPixels=27387-57600" />
44
<title>MSE webm file with alternate reference frame</title>
55
<script src="../../resources/testharness.js"></script>
66
<script>window.requirePixelDump = true</script>
77
<script src=../video-test.js></script>
88
<script src="../utilities.js"></script>
99
<script>
1010
// VP8 files were generated such that alternative reference frames were used:
11-
// $ fmpeg -i dragon.webm -c:v libvpx -vf scale=320:-1 -auto-alt-ref 1 -arnr-maxframes 5 -arnr-strength 3 -pass 1 test-vp8-hiddenframes.webm
12-
// $ fmpeg -i dragon.webm -c:v libvpx -vf scale=320:-1 -auto-alt-ref 1 -arnr-maxframes 5 -arnr-strength 3 -pass 2 test-vp8-hiddenframes.webm
11+
// $ ffmpeg -i dragon.webm -c:v libvpx -vf scale=320:-1 -auto-alt-ref 1 -arnr-maxframes 5 -arnr-strength 3 -pass 1 test-vp8-hiddenframes.webm
12+
// $ ffmpeg -i dragon.webm -c:v libvpx -vf scale=320:-1 -auto-alt-ref 1 -arnr-maxframes 5 -arnr-strength 3 -pass 2 test-vp8-hiddenframes.webm
1313
// The command used to extract the last frame in png format was:
1414
// $ ffmpeg -sseof -3 -i test-vp8-hiddenframes.webm -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int -update 1 -q:v 1 test-vp8-hiddenframes.png
1515
async function init()

‎LayoutTests/media/media-video-fullrange.html‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
<!-- webkit-test-runner [ SWVPDecodersAlwaysEnabled=true ] -->
22
<html>
33
<head>
4-
<title>videos with fullrange color</title>
4+
<title>videos with full range color</title>
55
<script src=video-test.js></script>
66
<script>
77
var pixelData;
@@ -13,14 +13,14 @@
1313

1414
let promise = new Promise(resolve => element.requestVideoFrameCallback(resolve));
1515
await waitFor(element, 'loadedmetadata');
16-
16+
1717
await Promise.all([ element.play(), promise ]);
1818

1919
const canvas = document.createElement('canvas');
2020
canvas.width = element.videoWidth;
2121
canvas.height = element.videoHeight;
2222
var ctx = canvas.getContext('2d');
23-
23+
2424
ctx.drawImage(element, 0, 0);
2525
pixelData = ctx.getImageData(20, 20, 1, 1).data;
2626

‎LayoutTests/media/media-video-videorange-expected.txt‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
2+
EVENT(loadedmetadata)
3+
EXPECTED (pixelData[0] == '0') OK
4+
EXPECTED (pixelData[1] == '0') OK
5+
EXPECTED (pixelData[2] == '0') OK
6+
EXPECTED (pixelData[0] >= '253') OK
7+
EXPECTED (pixelData[1] >= '253') OK
8+
EXPECTED (pixelData[2] <= '2') OK
9+
EVENT(loadedmetadata)
10+
EXPECTED (pixelData[0] == '0') OK
11+
EXPECTED (pixelData[1] == '0') OK
12+
EXPECTED (pixelData[2] == '0') OK
13+
EXPECTED (pixelData[0] >= '253') OK
14+
EXPECTED (pixelData[1] >= '253') OK
15+
EXPECTED (pixelData[2] <= '2') OK
16+
EVENT(loadedmetadata)
17+
EXPECTED (pixelData[0] == '0') OK
18+
EXPECTED (pixelData[1] == '0') OK
19+
EXPECTED (pixelData[2] == '0') OK
20+
EXPECTED (pixelData[0] >= '253') OK
21+
EXPECTED (pixelData[1] >= '253') OK
22+
EXPECTED (pixelData[2] <= '2') OK
23+
EVENT(loadedmetadata)
24+
EXPECTED (pixelData[0] == '0') OK
25+
EXPECTED (pixelData[1] == '0') OK
26+
EXPECTED (pixelData[2] == '0') OK
27+
EXPECTED (pixelData[0] >= '253') OK
28+
EXPECTED (pixelData[1] >= '253') OK
29+
EXPECTED (pixelData[2] <= '2') OK
30+
EVENT(loadedmetadata)
31+
EXPECTED (pixelData[0] == '0') OK
32+
EXPECTED (pixelData[1] == '0') OK
33+
EXPECTED (pixelData[2] == '0') OK
34+
EXPECTED (pixelData[0] >= '253') OK
35+
EXPECTED (pixelData[1] >= '253') OK
36+
EXPECTED (pixelData[2] <= '2') OK
37+
EVENT(loadedmetadata)
38+
EXPECTED (pixelData[0] == '0') OK
39+
EXPECTED (pixelData[1] == '0') OK
40+
EXPECTED (pixelData[2] == '0') OK
41+
EXPECTED (pixelData[0] >= '253') OK
42+
EXPECTED (pixelData[1] >= '253') OK
43+
EXPECTED (pixelData[2] <= '2') OK
44+

‎LayoutTests/media/media-video-videorange.html‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
<!-- webkit-test-runner [ SWVPDecodersAlwaysEnabled=true ] -->
2+
<html>
3+
<head>
4+
<title>videos with video range color</title>
5+
<script src=video-test.js></script>
6+
<script>
7+
var pixelData;
8+
9+
async function test(element, file) {
10+
element.src = "";
11+
element.load();
12+
element.src = file;
13+
14+
let promise = new Promise(resolve => element.requestVideoFrameCallback(resolve));
15+
await waitFor(element, 'loadedmetadata');
16+
17+
await Promise.all([ element.play(), promise ]);
18+
19+
const canvas = document.createElement('canvas');
20+
canvas.width = element.videoWidth;
21+
canvas.height = element.videoHeight;
22+
var ctx = canvas.getContext('2d');
23+
24+
ctx.drawImage(element, 0, 0);
25+
pixelData = ctx.getImageData(20, 20, 1, 1).data;
26+
testExpected("pixelData[0]", 0, "==");
27+
testExpected("pixelData[1]", 0, "==");
28+
testExpected("pixelData[2]", 0, "==");
29+
30+
pixelData = ctx.getImageData(canvas.width / 2 + 20, canvas.width / 2 + 20, 1, 1).data;
31+
testExpected("pixelData[0]", 253, ">=");
32+
testExpected("pixelData[1]", 253, ">=");
33+
testExpected("pixelData[2]", 2, "<=");
34+
};
35+
36+
async function init()
37+
{
38+
if (window.testRunner)
39+
testRunner.waitUntilDone();
40+
41+
const files = [
42+
'content/test-h264-601-videorange.mp4',
43+
'content/test-vp9-601-videorange.webm',
44+
'content/test-vp8-601-videorange.webm',
45+
]
46+
47+
for (const file of files) {
48+
await test(document.getElementsByTagName('video')[0], file);
49+
}
50+
51+
// Same test, but with video not in the DOM.
52+
for (const file of files) {
53+
await test(document.createElement('video'), file);
54+
}
55+
56+
if (window.testRunner)
57+
testRunner.notifyDone();
58+
}
59+
</script>
60+
</head>
61+
<body onload="init();">
62+
<video/>
63+
</body>
64+
</html>

‎LayoutTests/media/media-vp8-hiddenframes.html‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
11
<html>
22
<head>
3-
<meta name="fuzzy" content="maxDifference=0-16; totalPixels=0-57600" />
3+
<meta name="fuzzy" content="maxDifference=3-16; totalPixels=27387-57600" />
44
<title>webm file with alternate reference frame</title>
55
<script src="../resources/testharness.js"></script>
66
<script>window.requirePixelDump = true</script>
77
<script src=video-test.js></script>
88
<script>
99
// VP8 files were generated such that alternative reference frames were used:
10-
// $ fmpeg -i dragon.webm -c:v libvpx -vf scale=320:-1 -auto-alt-ref 1 -arnr-maxframes 5 -arnr-strength 3 -pass 1 test-vp8-hiddenframes.webm
11-
// $ fmpeg -i dragon.webm -c:v libvpx -vf scale=320:-1 -auto-alt-ref 1 -arnr-maxframes 5 -arnr-strength 3 -pass 2 test-vp8-hiddenframes.webm
10+
// $ ffmpeg -i dragon.webm -c:v libvpx -vf scale=320:-1 -auto-alt-ref 1 -arnr-maxframes 5 -arnr-strength 3 -pass 1 test-vp8-hiddenframes.webm
11+
// $ ffmpeg -i dragon.webm -c:v libvpx -vf scale=320:-1 -auto-alt-ref 1 -arnr-maxframes 5 -arnr-strength 3 -pass 2 test-vp8-hiddenframes.webm
1212
// The command used to extract the last frame in png format was:
1313
// $ ffmpeg -sseof -3 -i test-vp8-hiddenframes.webm -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int -update 1 -q:v 1 test-vp8-hiddenframes.png
1414
async function init()

‎LayoutTests/platform/glib/TestExpectations‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1174,6 +1174,7 @@ webkit.org/b/270622 media/media-source/media-managedmse-noresumeafterpause.html
11741174
webkit.org/b/271631 media/media-source/media-managedmse-stall-endofstream.html [ Failure ]
11751175
webkit.org/b/282791 media/media-source/media-source-vp8-hiddenframes.html [ ImageOnlyFailure ]
11761176
webkit.org/b/282791 media/media-vp8-hiddenframes.html [ ImageOnlyFailure ]
1177+
webkit.org/b/282827 media/media-video-videorange.html [ Failure ]
11771178

11781179
# GStreamer (< 1.24) doesn't set the track's id to the container's value.
11791180
webkit.org/b/265919 media/track/media-audio-track.html [ Failure ]

‎LayoutTests/platform/ios/TestExpectations‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,10 @@ media/media-source/media-managedmse-video-with-poster.html [ Pass ]
143143

144144
webkit.org/b/269897 media/media-source/media-managedmse-poster.html [ Skip ]
145145

146+
webkit.org/b/282969 media/media-video-fullrange.html [ Timeout Pass ]
147+
webkit.org/b/282969 media/media-video-videorange.html [ Timeout Pass ]
148+
webkit.org/b/282968 media/media-vp8-hiddenframes.html [ ImageOnlyFailure Pass ]
149+
146150
fast/images/text-recognition [ Pass ]
147151
fast/images/text-recognition/ios [ Pass ]
148152
fast/images/text-recognition/mac [ Skip ]

0 commit comments

Comments
 (0)