A while back I asked on the ffmpeg mailing list how to pipe RGB data in to ffmpeg. I described it as follows:
in my code I am building video frames, 720x480x24bit. I have in mind generating a large number of these, as long as a full DVD worth at 30fps, then using ffmpeg (followed by dvdauthor) to encode them in to MPEG2 for DVD usage.
There were a few replies, but no definitive answer. With considerable experimentation, I got it to work. It turns out that (as far as I can tell) ffmpeg does not have the ability to accept piped in RGB frames. It will however accept piped in data in its “yuv4mpegpipe” format. With some searching and reading I found that this is roughly akin to the format of raw DV video; each frame consists of a header something like this:
YUV4MPEG2 W%d H%d F%d:%d Ip A0:0 C420mpeg2 XYSCSS=420MPEG2
… then an LF character, then data for the the Y, U, and V “planes”. The Y data is full resolution, while the U and Y are half-resolution (this is called “420” in the video world). These planes are uncompressed, one byte per pixel. All of my past work with computer video (going back to Commodore 64s and Apple IIs) has arranged all of the bits for each pixel within a few bytes of each other; this format (with all the Y data for the whole frame, then all the U data, then all the V data) is starkly different.
The essential problem remaining was how to convert RGB to YUV. Happily there are plenty of online references for this. Unhappily there are few fast implementations, and a naive implementation will be very slow. I solved this problem by finding and hiring an expert in low-level data processing with MMX, SSE2, etc. instructions. (I am not in a position to publish that code here.)
In retrospect, though, there are routines included in Intel’s “Integrated Performance Primitives” library which perform this transformation in a highly optimized way. IPP is a bargain: for only a few hundred dollars you get a wealth of high optimized ready-to-use library routines for signal processing.
The ffmpeg piping solution consists, therefore, of:
- A module which generated frames in RGB format, to contain whatever contents your application requires.
- A module to very quickly convert these to YUV in yuv4mpegpipe format (write your own, or use routines in IPP, for the RGB->YUV420 part).
- Pipe this data stream to ffmpeg with stdin; ffmpeg is invoked something like this: ffmpeg -y -f yuv4mpegpipe -i – -i audio.mp3 -target ntsc-dvd -aspect 4:3 foo.mpg
By using a multicore CPU and threads, this whole process can be made to happen in real time or better (i.e., one second of “wall clock” processing time, for one second of finished MPEG2 video). The resulting MPEG2 file can be used with a DVD authoring application to produce a ready-to-burn DVD ISO image.
Update: the data format above is published here as part of the mjpegtools man pages.
I also encountered the problem of piping Y Cb Cr Data into ffmpeg. I found a solution and maybe it is useful to anybody out there in the WWW 🙂
An application, let’s call it “a.out”, extracts YCbCr Data (4:2:2) with 10 bit resolution from a video stream. The order ist [Cb] [Y] [Cr] [Y] ….
I created a named_pipe called “output_pipe” which is constantly filled with the RGB Data. A.out constantly is filling up the “output_pipe” with YCbCr data.
Now, I can use ffmpeg like the following:
ffmpeg -pix_fmt uyvy422 -s 720×576 -f rawvideo -i output_pipe -target pal-dvd -aspect 4:3 -y myDVD.mpg
Notice, that the codec is “rawvideo” and the “pix_fmt” is chosen corresponsing to the input stream. You can get a list of all pix_formats of ffmpeg by typing “ffmpeg -pix_fmt list”.
I hope this is helpful!
Currently, I’m trying to pipe video and audio to ffmpeg within 1 step. I use the command:
ffmpeg -pix_fmt uyvy422 -s 720×576 -f rawvideo -i output_pipe -target pal-dvd -aspect 4:3 -y myDVD.mpg -f s16le -ar 48000 -ac 2 -i pcm0_pipe -acodec mp2 -ab 224k -newaudio
I’m having 48kHz / 16bit stereo audio raw PCM data which I want to directly add to the video file (muxing). The audio data is piped into the pcm0_pipe.
I ran into the problem that, of course, there is more audio than video data and so, if the video pipe is full, ffmpeg waits for the audio data and the whole process is blocked….
If anybody knows a solution, it would be nice to share it 🙂
Maurice
I’m interested in doing something similar. After some research, I wonder why you
weren’t able, or chose not to, use libavcodecs own img_convert() method to go from RGB to YUV420.
RGB to YUV conversion, including sample C code is on wikipedia. Its a fairly trivial conversion.
the conversion code is not your biggest concern when implementing on x86, streaming/cache issues will be.
Boris’s comment is a great example of the difference between theory and practice. It turns out that in practice, a trivial implementation of the trivial RGB/YUV conversion is quite slow. That code, and many other parts, need to be well tested and optimized to process video in real time.
What do you mean ‘quite slow’. For instance, on a single thread i can convert 720×576 in 3.8ms on a Pentium4, on todays cheap dual core machines it would be 1/2 that. I dont do ‘theory’ i always speak from practice.
Boris, in a single thread converting a 720×576 frame may be reasonably fast. Multicore machines are not going to nett you much gain unless you understand the limitations of threaded programming and how to avoid them. Remember that synchronization will lose you as much as you gain from parallelising your code if you do it badly.
Kyle,
thanks for the post. I’ve also been looking for a way to pipe rgb data into ffmpeg. Do you know if there’s been any new developments since the time of your post? Or if there’s another format (other than y’cbcr) that can be piped into ffmpeg?
You mentioned that:
“By using a multicore CPU and threads, this whole process can be made to happen in real time or better (i.e., one second of “wall clock” processing time, for one second of finished MPEG2 video). ”
Are you speaking from personal experience? If so, was it using IPP’s function or your own solution?
thanks in advance
As I described in the post I solved the challenges here, with some IPP stuff and some code from before I found IPP. Sorry, but I’m not at liberty to publish any more details.