Shared Media Graph: A new linux API for complex media systems
Have you ever wanted to connect multiple complex media processing blocks into one media graph? Maybe you have CSI receivers that can mux their output to either a shared interconnect or to multiple ISPs. Or maybe you have a capture interface that can mux the image data from multiple sensors to memory or directly to the ISPs, and the ISPs also have the capability to take frames from memory.
These kinds of runtime-configurable interconnections were nearly impossible to support… until now! The Shared Media Graph API that was posted recently to the linux-media mailing list paves the way for supporting these setups.
Before shared media graph
We are finding ourselves in a world where the same IP cores are integrated into different designs with different features, and it’s becoming near-impossible to determine a single component in the system that should be responsible for registering the entire media graph.
Let’s take a look at an example. Here is the media graph of the capture interface, also known as VICAP (driven by the rkcif driver), on the Rockchip RK3588.
For just this media graph, the rkcif is the main driver and registers its own subdevices and video devices, and also indirectly registers and links to the CSI receiver (dw-mipi-csi2rx) and the image sensor (imx219).
Now the RK3588 also has two ISP instances (each driven by the rkisp2 driver), and we want to connect these to the same media graph as the VICAP to support ISP inline mode (when they are in separate media graphs the ISP can only operate in memory-to-memory). Here is the media graph of the rkisp2. It is comparatively simpler and only has to register its output and capture video nodes.
If we were to join these two into one media graph, which driver would be the main driver? rkcif or rkisp2?
For the RK3588 we could simply decide that one driver will be the “main driver”. As an example, let’s just say that we have decided that the rkcif will be the main driver. It will then be responsible for initializing the rkisp2’s video devices and subdevices and linking it into the rkcif’s media graph. The rkisp2 will do almost nothing with regards to setting up the media graph.
The problem with this is that since IP cores in different SoCs are so similar, the rkcif and rkisp2 in the future will support other platforms in which the other driver is not present. Continuing the example, if the rkisp2 is running on a platform where there is no rkcif, then there is nobody around to initialize the media graph for the rkisp2.
We could make the rkisp2 detect if it should initialize itself and do so, but then we will have lots of duplicate code and drivers will be stepping on each other’s toes for initialization and it will become very messy.
This is why setups similar to this were simply not supported and they could not be configured at runtime. The only good-ish solution that we had is to switch device tree overlays to support the other hardware configuration. This is what is currently done on the i.MX8MP for example to switch between using the ISI and the ISP, and the configuration cannot be changed at runtime. Without a new solution, the same would happen for switching between memory-to-memory mode and inline ISP mode on the RK3588.
After shared media graph
This problem was discussed extensively during the linux-media meeting at Embedded Recipies in Nice, and it was concluded that the best solution would be to allow multiple drivers to share a media graph. Each driver would be able to join an existing media graph or create one if none were available. Patch 1 of the Shared Media Graph API series implements such a mechanism. It uses a create-or-join call so that drivers can join an existing shared media graph or create one if one does not exist. The API also provides a way to create inter-driver links asynchronously.
As a first example, the above series implements support for the RK3588 example that we went over earlier. Here is the shared media graph.
The current version of the series implements switching between ISP inline mode and memory-to-memory mode by manipulating the links between the rkcif and the rkisp2. If you try out the setup using my kernel branch and libcamera branch then libcamera will automatically configure the links for memory-to-memory mode. Eventually libcamera will also support ISP inline mode and will be able to configure it automatically as well. Until then, to test inline mode you will need to disable the links from rkisp2_rawrd{0,1,2} and the links to rkcif-mipi2-id{0,1,2,3} and enable the link between rkcif-mipi2 and rkisp2_isp. After you configure the media graph you can capture from rkisp2_mainpath via yavta or v4l2-ctl.
Future potential
While the API still has some rough edges that need to be worked out, and the example that I tested and sent only handles the rkcif and a single instance of rkisp2, this framework has the potential to unlock even more complex setups.
The first one is using dual ISP on the RK3588. Being able to use both ISPs with configurable mode settings can unlock many more features of the RK3588 media block. Here is an imagined media graph that shows this potential future.
Another setup that the shared media graph enables is the ISI and ISP on the i.MX8MP. Currently we cannot use the ISI and the ISPs simultaneously, and we have to switch device tree overlays every time we want to use the other one. If we use the shared media graph, then we can put them all in one graph. Here is an imagined media graph for this potential future.
I’m excited for all the new cool things that this Shared Media Graph API will enable us to do with all the cool hardware that we all have! (Reviews and testing are always appreciated and will accelerate these getting merged upstream)