FuseVideo, the Course Project of Digital Image Processing

This idea comes from a practical need. I want to make an opening for my skating video, where two performers cosplay Wednesday and Enid, two characters of the Netflix series Wednesday. Fans of this series know that Wednesday has a faithful servant, the Thing, a human hand-like creature that can think independently. I want to bring Thing to the opening of my video.

In the beginning, I naively believed that only using Poisson confusion would be fine, as the demo in the original paper shows even though the backgrounds of the two images are slightly different, this method can still generate natural results. However, the source video for Thing is very noisy with a messy background. If I brutally Poisson confuse them, the brightness and balance of the output image will become wired. That's the time when image segmentation comes to my mind. Empowered by the recently released large model, SAM, Thing can be separated from the original frame in a scraped quality since the model cannot handle low resolution and blur that well.

In addition, the performance of using Poisson confusion plus image segmentation is also better than barely using segmentation. Because Poisson confusion calculates the gradient of images, it fuses the two inputs more naturally, making the output less like a sticker on the background.