Bdtjtk

I'm working on an OpenCV project to monitor a 1080p 60fps video feed and also apply custom graphics to this feed. I'm looking for general guidance about designing some of the higher-level operations in my system which compose multiple matrix operations. For example, in one of my features I am resizing a video frame and applying an overlay to that resized frame. The following diagram describes the process:

Here is the implementation of the process (currently done in C# opencvsharp, however, I can shift to any language at this point):

private void updateFrame(Mat currentFrame, Mat background, Mat mask, Mat invertedMask)

{

    int w = 400, h = 224;



    using (var resizedFrame = new Mat(

        new OpenCvSharp.Size(currentFrame.Size().Width - w, currentFrame.Size().Height - h), 

        currentFrame.Type()))

    using (var resizedBorderFrame = new Mat(currentFrame.Size(), currentFrame.Type()))

    using (var maskedFrame = new Mat(currentFrame.Size(), currentFrame.Type()))

    using (var maskedBackground = new Mat(currentFrame.Size(), currentFrame.Type()))

    using (var output = new Mat(currentFrame.Size(), currentFrame.Type()))

    {

        Cv2.Resize(currentFrame, resizedFrame, resizedFrame.Size());

        Cv2.CopyMakeBorder(resizedFrame, resizedBorderFrame, h/4, h*3/4, w/2, w/2, BorderTypes.Constant, new Scalar(0));

        Cv2.BitwiseAnd(resizedBorderFrame, mask, maskedFrame);

        Cv2.BitwiseAnd(background, invertedMask, maskedBackground);

        Cv2.BitwiseOr(maskedBackground, maskedFrame, output);

        pictureBox.Image = OpenCvSharp.Extensions.BitmapConverter.ToBitmap(output);

    }

}

This process (along with a few other operations) is beginning to take longer than the framerate of the video, creating a noticeable lag. Currently the process is being performed using CPU based operations, however, I read that applying GPU operations could speed up the runtime a considerable amount. Further, I read that creating a custom kernel to combine operations (or creating the whole series as a compound-kernel operation) could speed this up even more. I'm also trying to analyze which operations are not constrained by the CPU, which might make the GPU equivalent operation an overkill.

If you were to evaluate this problem from the start, how would you go about determining which operations to put on CPU vs GPU vs custom kernel? Or rather, what resources and tools could I be using for analyzing the performance differences? And, what other optimizations or processes should I be employing when considering these types of problems?