You’ve probably seen it in a fancy visual journalism piece from a well-known media organization: the scrolling video.
Unfortunately, video formats were never designed with this use case in mind, and will often take seconds or longer to load the frame under normal circumstances. Used in a scrolling video, this results in a horribly choppy experience. The reason this happens is videos are typically encoded using keyframes set every 30 frames or so. In videos, keyframes are frames that contain the pixel data for the entire frame, whereas frames that are not keyframes only hold the “difference” between this frame and the last frame. Having frames only encode frames that have changed allows the video to be compressed to a smaller size, as most videos are only expected to be played forwards.
Therefore, when it comes to exporting video for a “scrolly” use case, the recommendation is to export it with the setting keyframes=1, which tells the encoder that every single frame is a keyframe. While this solves our problem of allowing the video to dynamically load the right frame of the video much faster, it also causes the size of the video file to increase significantly, all other settings being equal. In my experience, going down this path will result in re-exporting the video multiple times while adjusting the quality setting to find a compromise between file size and video quality, which is not ideal.
After experimenting with this method for a while, I found a second approach: simply playing and pausing the video while dynamically adjusting the playback rate. If you’ve ever played around with video player settings, you’ll know that videos on the web often have the option of changing the playback speed of the video, allowing you to power through a lecture at two or three times speed. In fact, most web browsers support up to eight times speed, a speed that I have no honest idea when one would actually want to use in real life. Using playback rate, I can essentially mimic the effect of a user scrolling fast or slow, while relying on the video player to decode the frames in order, making the forward scrolling experience extremely smooth.
However, the catch with this method is that playback rate cannot be a negative number, so scrolling backwards must still be done with the first method above. Theoretically you could export an identical video in reverse and have two video elements that show or hide depending on the scroll direction, but scrollyVideo.js currently does not support this option. Additionally, Safari for some reason is less performant using this approach than the one above, so this library detects Safari and forces it to use the first method.
The final approach I stumbled across was using the WebCodecs API to convert a video into individual frames in the browser. Unfortunately, WebCodecs is only supported in Chrome at the moment, with no estimated release in any of the other browsers. And while I did find a polyfill for WebCodecs, I was unable to get it working with ScrollyVideo, so this method is limited to Chromium-based browsers only.
Essentially, by reading all the frames from a video ahead of time, this method is able to have any possible frame immediately ready for painting to a canvas. The downside? It takes a bit of time before the video is fully processed, so any immediate usage of this method will likely fall back to one of the earlier ones. Going this route also requires more memory and processing power, something that lower-end android devices may not handle well.
Additional Use Cases
For more creative use cases, scrolling may not be the only way that a project may want to control the playback and position of a video. Perhaps you want to control the position of the video based on mouse movement or something else. By exposing
setCurrentTimePercent from the library, you can also directly set the position the video.
That said, I’m looking forward to seeing this in the wild, and if you have any further questions, find any bugs, or want to contribute, feel free to reach out and I’m happy to talk!