Skip to content

Clarify color space handling across decoding, encoding, and rendering #998

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Djuffin wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Clarify color space handling across decoding, encoding, and rendering #998

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 98 additions & 34 deletions index.src.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -963,23 +963,39 @@
{{VideoDecoderConfig/displayAspectHeight}} [=map/exist=] in the
{{VideoDecoder/[[active decoder config]]}}, assign their values to
|displayAspectWidth| and |displayAspectHeight| respectively.
4. Let |colorSpace| be the {{VideoColorSpace}} for |output| as detected
by the codec implementation. If no {{VideoColorSpace}} is detected,
let |colorSpace| be `undefined`.

NOTE: The codec implementation can detect a {{VideoColorSpace}} by
analyzing the bitstream. Detection is made on a best-effort
basis. The exact method of detection is implementer defined and
codec-specific. Authors can override the detected
{{VideoColorSpace}} by providing a
{{VideoDecoderConfig/colorSpace}} in the {{VideoDecoderConfig}}.

5. If {{VideoDecoderConfig/colorSpace}} [=map/exists=] in the
{{VideoDecoder/[[active decoder config]]}}, assign its value to
|colorSpace|. In that case, User Agents MAY replace `null` members
of |colorSpace| with the corresponding values detected by the codec
implementation.
FIXME: Properly specify the case of `null` members.
4. Determine the [=internal color space=] of |output|'s
[=media resource=] from {{VideoDecoder/[[active decoder config]]}}'s
{{VideoDecoderConfig/colorSpace}} (if it [=map/exists=]), color space
information in the bitstream, and {{VideoDecoder/[[codec
implementation]]}} (such as color space transformations or
platform-specific defaults applied during decoding). When
{{VideoDecoder/[[codec implementation]]}} does not transform the
color space during decoding, User Agents <em class="rfc2119">SHOULD</em>
prefer non-`null` members of {{VideoDecoderConfig/colorSpace}} over
in-band bitstream values.
5. Determine |colorSpace| as follows:
1. If {{VideoDecoderConfig/colorSpace}} [=map/exists=] in
{{VideoDecoder/[[active decoder config]]}} and at least one of
its {{VideoColorSpaceInit/primaries}},
{{VideoColorSpaceInit/transfer}},
{{VideoColorSpaceInit/matrix}}, or
{{VideoColorSpaceInit/fullRange}} members is not `null`, let
|colorSpace| be {{VideoDecoder/[[active decoder config]]}}'s
{{VideoDecoderConfig/colorSpace}}.
2. Otherwise, let |colorSpace| be a new {{VideoColorSpaceInit}}
populated from the [=internal color space=] of |output|'s
[=media resource=], with any unspecified or unrecognized
components set to `null`.

NOTE: Any {{VideoDecoderConfig/colorSpace}} with at least one
non-`null` member overrides bitstream and decoder-detected values
on the emitted {{VideoFrame}}'s {{VideoFrame/colorSpace}}, leaving
unset members as `null`. An empty `colorSpace` (such as `{}`) is
treated the same as omitting {{VideoDecoderConfig/colorSpace}}.
Rendering and color space conversion operate on the
[=media resource=]'s [=internal color space=], which can combine
information from {{VideoDecoderConfig/colorSpace}}, the bitstream,
and {{VideoDecoder/[[codec implementation]]}}.
6. Assign the values of {{VideoDecoderConfig/rotation}} and
{{VideoDecoderConfig/flip}} to |rotation| and |flip| respectively.
7. Let |frame| be the result of running the [=Create a VideoFrame=]
Expand Down Expand Up @@ -1720,7 +1736,27 @@
associated with |output| to `outputConfig.rotation`.
7. Assign {{VideoFrame/[[flip]]}} from the {{VideoFrame}}
associated with |output| to `outputConfig.flip`.
8. Assign the remaining keys of `outputConfig` as determined by
8. Initialize `outputConfig.colorSpace` from the {{VideoFrame}}
associated with |output| as follows:
1. Let |inputColorSpace| be a copy of the [=internal color space=]
of the [=media resource=] referenced by the {{VideoFrame}}
associated with |output|.
2. Encoding <em class="rfc2119">MAY</em> modify
|inputColorSpace|'s {{VideoColorSpace/matrix}} and
{{VideoColorSpace/fullRange}} to satisfy codec or platform
constraints, assuming full-range
{{VideoMatrixCoefficients/rgb}} when the source
[=media resource=] uses an [=RGB format=] and limited-range
{{VideoMatrixCoefficients/bt709}} otherwise when
unspecified.
3. Let |encodedColorSpace| be |inputColorSpace|, with any
specified components of the color space emitted by
{{VideoEncoder/[[codec implementation]]}} replacing the
corresponding components of |inputColorSpace|.
4. Assign a new {{VideoColorSpaceInit}} populated from
|encodedColorSpace|, with any unspecified or unrecognized
components set to `null`, to `outputConfig.colorSpace`.
9. Assign the remaining keys of `outputConfig` as determined by
{{VideoEncoder/[[codec implementation]]}}. The User Agent
<em class="rfc2119">MUST</em> ensure that the configuration is
completely described such that |outputConfig| could be used to
Expand Down Expand Up @@ -2124,8 +2160,9 @@
<dd>
Configures the {{VideoFrame}}.{{VideoFrame/colorSpace}} for {{VideoFrame}}s
associated with this {{VideoDecoderConfig}}. If
{{VideoDecoderConfig/colorSpace}} [=map/exists=], the provided values will
override any in-band values from the bitsream.
{{VideoDecoderConfig/colorSpace}} [=map/exists=] and has at least one
non-`null` member, it overrides any in-band values from the bitstream on
emitted {{VideoFrame}}s, as described in [=Output VideoFrames=].
</dd>

<dt><dfn dict-member for=VideoDecoderConfig>hardwareAcceleration</dfn></dt>
Expand Down Expand Up @@ -2849,7 +2886,12 @@
### Reference Counting ### {#raw-media-memory-model-reference-counting}

A <dfn>media resource</dfn> is storage for the actual pixel data or the audio
sample data described by a {{VideoFrame}} or {{AudioData}}.
sample data described by a {{VideoFrame}} or {{AudioData}}. For video, a
[=media resource=] has an associated underlying pixel format and an
<dfn>internal color space</dfn> (e.g., [[H.273]] code points) which can be more
expressive than {{VideoPixelFormat}} and {{VideoColorSpace}}, and can
incorporate color space information from decoder configuration, the bitstream,
and platform codec transformations.

The {{AudioData}} {{AudioData/[[resource reference]]}} and {{VideoFrame}}
{{VideoFrame/[[resource reference]]}} internal slots hold a reference to a
Expand Down Expand Up @@ -3564,7 +3606,12 @@
:: The [=presentation timestamp=], given in microseconds.

: <dfn attribute for=VideoFrame>[[color space]]</dfn>
:: The {{VideoColorSpace}} associated with this frame.
:: A {{VideoColorSpace}} describing the exposed color space of this
{{VideoFrame}}. Individual members will be `null` when unspecified (for
example, in {{VideoFrameBufferInit}} or {{VideoDecoderConfig}}) or when a
component of the [=media resource=]'s [=internal color space=] does not map
to a value in {{VideoColorPrimaries}}, {{VideoTransferCharacteristics}}, or
{{VideoMatrixCoefficients}}.

: <dfn attribute for=VideoFrame>\[[metadata]]</dfn>
:: The {{VideoFrameMetadata}} associated with this frame.
Expand Down Expand Up @@ -3763,7 +3810,9 @@
12. Assign the result of running the [=VideoFrame/Pick Color Space=]
algorithm, with |colorSpace| and {{VideoFrame/[[format]]}}, to
{{VideoFrame/[[color space]]}}.
13. Assign the result of calling [=Copy VideoFrame metadata=]
13. Initialize |resource|'s [=internal color space=] from
{{VideoFrame/[[color space]]}}, leaving any `null` members unspecified.
14. Assign the result of calling [=Copy VideoFrame metadata=]
with |init|'s {{VideoFrameBufferInit/metadata}} to
|frame|.{{VideoFrame/[[metadata]]}}.
22. Return |frame|.
Expand Down Expand Up @@ -3862,7 +3911,15 @@
{{VideoFrame/[[duration]]}}.

: <dfn attribute for=VideoFrame>colorSpace</dfn>
:: The {{VideoColorSpace}} associated with this frame.
:: The {{VideoColorSpace}} associated with this frame. Individual members will
be `null` when unspecified or when the corresponding component does not map
to a value in {{VideoColorPrimaries}}, {{VideoTransferCharacteristics}}, or
{{VideoMatrixCoefficients}}.

NOTE: For decoded frames, {{VideoFrame/colorSpace}} reflects a non-empty
{{VideoDecoderConfig/colorSpace}} as-is when provided (see
[=Output VideoFrames=]), whereas rendering and color space conversion
operate on the underlying [=media resource=]'s [=internal color space=].

The {{VideoFrame/colorSpace}} getter steps are to return
{{VideoFrame/[[color space]]}}.
Expand Down Expand Up @@ -4038,11 +4095,6 @@
: <dfn for=VideoFrame>Pick Color Space</dfn> (with |overrideColorSpace| and |format|)
:: 1. If |overrideColorSpace| is provided, return a new {{VideoColorSpace}}
constructed with |overrideColorSpace|.

User Agents <em class="rfc2119">MAY</em> replace `null` members of the
provided |overrideColorSpace| with guessed values as determined by implementer
defined heuristics.

2. Otherwise, if {{VideoFrame/[[format]]}} is an [=RGB format=] return a new
instance of the [=sRGB Color Space=]
3. Otherwise, return a new instance of the [=REC709 Color Space=].
Expand Down Expand Up @@ -4155,10 +4207,9 @@
|frame|'s {{VideoFrame/[[duration]]}}.
12. Assign `init`.{{VideoFrameInit/timestamp}} to
|frame|'s {{VideoFrame/[[timestamp]]}}.
13. If |resource| has a known {{VideoColorSpace}}, assign its value to
{{VideoFrame/[[color space]]}}.
14. Otherwise, assign a new {{VideoColorSpace}}, constructed with an empty
{{VideoColorSpaceInit}}, to {{VideoFrame/[[color space]]}}.
13. Assign a new {{VideoColorSpace}} populated from |resource|'s
[=internal color space=], with any unspecified or unrecognized
components set to `null`, to {{VideoFrame/[[color space]]}}.

: <dfn for=VideoFrame>Initialize Visible Rect, Orientation, and Display Size</dfn>
(with |init|, |frame|, |defaultVisibleRect|, |baseRotation|, |baseFlip|,
Expand Down Expand Up @@ -4444,7 +4495,9 @@
conversion of [=media resource=] referenced by |frame|'s
{{VideoFrame/[[resource reference]]}} into a color space and pixel
format specified by {{VideoFrame/[[color space]]}} and
{{VideoFrame/[[format]]}} respectively.
{{VideoFrame/[[format]]}} respectively, with |resource|'s
[=internal color space=] initialized from
{{VideoFrame/[[color space]]}}.
9. Assign the reference to |resource| to {{VideoFrame/[[resource reference]]}}
3. Return |convertedFrame|.

Expand Down Expand Up @@ -4498,6 +4551,17 @@
be converted to a color space compatible with the rendering target, unless color
conversion is explicitly disabled.

When applying color space conversion (for rendering or in
[=VideoFrame/Convert to RGB frame=]), User Agents <em class="rfc2119">MUST</em>
use the [=media resource=]'s [=internal color space=] if its
{{VideoColorSpace/primaries}}, {{VideoColorSpace/transfer}}, and
{{VideoColorSpace/matrix}} components are specified, treating an unspecified
{{VideoColorSpace/fullRange}} component as `true` when the [=media resource=]
uses an [=RGB format=] and `false` otherwise. Otherwise, User Agents
<em class="rfc2119">MUST</em> treat the source color space as the
[=sRGB Color Space=] when the [=media resource=] uses an [=RGB format=], and as
the [=REC709 Color Space=] otherwise.

Color space conversion during {{ImageBitmap}} construction is controlled by
{{ImageBitmapOptions}} {{ImageBitmapOptions/colorSpaceConversion}}. Setting this
value to {{ColorSpaceConversion/"none"}} disables color space conversion.
Expand Down
Loading