[HTMLMediaElement] Pitch Control

[GarrettS]

[HTMLMediaElement] pitch control
https://html.spec.whatwg.org/multipage/embedded-content.html#htmlmediaelement

USE CASE: User can independently and easily adjust the pitch for videos for music transcription and learning.

Many old recordings done on tape are off by 1/4 step or more. Learning difficult musical passages is made much easier by seeing and hearing the passage performed slowly. Most music is in video format nowadays. Many musicians' first stop for investigating any particular piece music is YouTube. This is in fact why I developed the speed/pitch bookmarklet. Others have developed similar things with browser extensions. However, those solutions are quite limited by the HTMLMediaElement API - https://www.youtube.com/watch?v=b8B0dWQRC9c.

Solution 1:— HTMLMediaElement has playbackRate (boolean) to control speed.
Mozilla's non-standard mozPreservesPitch, when set to false, allows the user to adjust the pitch by adjusting the playbackRate.

The problems with that solution are 1) it's nonstandard, 2) it currently only works in Mozilla browsers, 3) pitch and speed are not independent (user cannot independently adjust pitch and speed). See https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement.

This would need to be standardized and implemented in more browsers to be a solution. But even then, it fails to allow independent control over pitch and speed.

Proposed Solution 2:—
A new property for pitchAdjustment (short) to adjust the pitch in cents, up or down. Arbitrary, independent pitch adjustment is already possible in Gecko - https://lists.w3.org/Archives/Public/public-whatwg-archive/2016Mar/0003.html.

Reading the pitchAdjustment property results in cents at which the media resource has been adjusted. If no adjustment has been made, the default, 0 results.

Setting the pitchAdjustment property adjusts the assigned value to a number in cents. Equally tempered semitones span 100 cents.

The limitation is that this API is designed to work only in equal temperament.

Example:

<video src="mediasource:http://example.net/" id="vv""></video>

<!-- transpose video up or down a 4th -->
<input type="range" min="-500" max="500" value="0"
 onchange="updateMoviePitch()">

Proposed solution 3:—
Create an API for audio only. Audio tracks can then be extracted from the video and then reinserted into the video.

var video = document.querySelector(‘video’);
video.preservesPitch = false;
var context = new AudioContext();
var sourceNode = context.createMediaElementSource(video);
var pitchShiftNode = context.createPitchShift();
pitchShiftNode.shiftAmount = 1.023;
sourceNode.connect(pitchShiftNode);
pitchShiftNode.connect(context.destination);

This solution is more complicated than required by the use case. It will require boilerplate code and will give rise third-party javascript libraries and other such problematic code management strategies to address these difficulties.

[GarrettS]

Moved the code to repository https://github.com/GarrettS/Speed-and-Pitch

Added support for multiple sites including Twitter. Users like adjusting the playback rate. Pitch Adjustment is still wanted.

[annevk]

cc @whatwg/media

[jernoble]

Safari already supports preservesPitch (as webkitPreservesPitch). Seems like the simplest solution to your specific use case would be to standardize it.

[GarrettS]

Hi @jernoble — That can work in some cases. However one major issue there is that pitch and speed are coupled. The problem from the user's perspective is that the user cannot independently adjust pitch and speed.

A lot of music needs pitch adjustment to be in standard tuning. Old Van Halen, for example, is about 80 cents flat of standard tuning. The user should be able to bring it up to pitch and then slow it down. But that's not possible with preservesPitch because pitch is coupled to playbackRate.

If you try the bookmarklet i made, you'll see what I mean.

[jernoble]

For the simple use case of “this video was recorded at the wrong speed”, there’s preservesPitch. For anything more complicated, including arbitrary pitch shifting, there’s WebAudio and the MediaElementSourceNode.

[GarrettS]

Jernoble, "This video was recorded at the wrong speed" is not this issue. But before filing a new issue for that, do a search, and also ask on the whatwg list "how do i ………"

[JohnGlassmyer]

Is this a dupe of #262 ?

Oh, I see. It's not.