Video Processing

Video Inpainting

>> See detail in MINELab

 

Publication List:

  • Shih, T.K., Nick C. Tang, Jenq-Neng Hwang, “Exemplar-based Video Inpainting without Ghost Shadow Artifacts by Maintaining Temporal Continuity,” IEEE Transactions on Circuits and Systems for Video Technology, volume 19, number 2, pages 347 – 360, March 2009.

ABSTRACT

Image inpainting or image completion is the tech- nique that automatically restores/completes removed areas in an image. When dealing with a similar problem in video, not only should a robust tracking algorithm be used, but the temporal con- tinuity among video frames also needs to be taken into account, especially when the video has camera motions such as zooming and tilting. In this paper, we extend an exemplar-based image in- painting algorithm by incorporating an improved patch matching strategy for video inpainting. In our proposed algorithm, different motion segments with different temporal continuity call for dif- ferent candidate patches, which are used to inpaint holes after a selected video object is tracked and removed. The proposed new video inpainting algorithm produces very few “ghost shadows,” which were produced by most image inpainting algorithms di- rectly applied on video. Our experiments use different types of videos, including cartoon, video from games, and video from digital camera with different camera motions. Our demonstration at http://member.mine.tku.edu.tw/www/T CSVT/web/shows the promising results.

 Index Terms—Digital inpainting, image completion, motion map segmentation, object removal, object tracking, video inpainting, video special effect.

Link to PDF
  • Nick C. Tang, Chiou-Ting Hsu, Chih-Wen Su, Shih, T.K., and Hong-Yuan Mark Liao, “Video Inpainting on Digitized Vintage Films via Maintaining Spatiotemporal Continuity,” IEEE Transactions on Multimedia, volume 13, pages 602-614, July 2011.

ABSTRACT

Video inpainting is an important video enhancement technique used to facilitate the repair or editing of digital videos. It has been employed worldwide to transform cultural artifacts such as vintage videos/films into digital formats. However, the quality of such videos is usually very poor and often contain unstable lu- minance and damaged content. In this paper, we propose a video inpainting algorithm for repairing damaged content in digitized vintage films, focusing on maintaining good spatiotemporal conti- nuity. The proposed algorithm utilizes two key techniques. Motion completion recovers missing motion information in damaged areas to maintain good temporal continuity. Frame completion repairs damaged frames to produce a visually pleasing video with good spatial continuity and stabilized luminance. We demonstrate the efficacy of the algorithm on different types of video clips.

 Index Terms—Digital inpainting, image completion, motion map segmentation, object removal, object tracking, video inpainting, video special effect.

Link to PDF
  • Shih, T.K., Nick C. Tang, Joseph C. Tsai and Jenq-Neng Hwang, “Video Motion Interpolation for Special Effect Applications,” IEEE Transactions on Systems, Man, and Cybernetics, Part C, volume 41, pages 720 – 732, August 2011.

ABSTRACT

Video forgery, also referred as video falsifying, is a technique for generating fake videos by altering, combining, or creating new video contents. For instance, the outcome of a 100 m race in the olympic game is forged as an example in this paper. We track objects and segment motions using a modified mean shift mechanism. The resulting video layers can be played in different speeds and from different reference points with re- spect to the original video. In order to obtain a smooth move- ment of target objects, a motion interpolation mechanism is pro- posed based on reference stick figures (i.e., a structure of human skeleton) and a video inpainting mechanism. The video inpaint- ing mechanism is performed in a quasi-3-D space via guided 3- D patch matching. Interpolated target objects and background layers are then fused. The objective is to create a forged video, which is almost indistinguishable from the original video. We demonstrate the original and the forged videos in our Web site at http://member.mine.tku.edu.tw/www/TSMC09/. Although video forgery may create moral or legal issues, which is beyond the scope of this paper, our intension is to create special effects in video editing applications.

Index Terms—Image completion, mean shift, motion interpola- tion, object tracking, video falsifying, video forgery, video inpaint- ing, video special effect.

Link to PDF
  • Shih, T.K. and Nick C. Tang, “Video Falsifying by Motion Interpolation and Inpainting,” in the 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Anchorage, Alaska, June 24 – June 26, 2008

ABSTRACT

We change the behavior of actors in a video. For instance, the outcome of a 100-meter race in the Olympic game can be falsified. We track objects and segment motions using a mod- ified mean shift mechanism. The resulting video layers can be played in different speeds and at different reference points with respect to the original video. In order to obtain a smooth movement of target objects, a motion interpolation mechanism is proposed based on continuous stick figures (i.e., a video of human skeleton) and video inpainting. The video inpainting mechanism is performed in a quasi-3D space via guided 3D patch matching for filling. Interpolated target objects and background layers are fused by using graph cut. It is hard to tell whether a falsified video is the original. We demonstrate the original and the falsified videos in our website at http://www.mine.tku.edu.tw/video_demo/). The proposed technique can be used to create special effects in movie in- dustry.

Index Terms—motion interpolation, video inpainting, image completion, graph cut, mean shift, video special effect

Link to PDF
  •  Rong-Chi Chang and Shih, T.K.(corresponding), “Multilayer Inpainting on Digitalized Artworks,” Journal of Information Science and Engineering, 2008.

ABSTRACT

Image inpainting automatically restores damaged images and partially removed pic- tures. Since most inpainting mechanisms inpaint damaged portions in a single layer, this study proposes a multilayer inpainting mechanism by examining how Chinese paintings are drawn in layers. The proposed multilayer inpainting mechanism employs a layer fu- sion strategy to detect the optimal inpaint combination among layers to restore paintings. Because this multilayer multi-resolution strategy considers damages in each layer from a multi-resolution perspective, it is superior to several existing techniques for restoring Chinese and Western paintings. In this study, the proposed algorithm is tested on more than 1,500 still images, with evaluations showing the effectiveness of image inpainting. The results indicate that the proposed algorithm achieves high PSNR values as well as high user satisfactions, including inpainting in some extreme cases where more than ninety percent of a painting are destroyed.

Index Terms—Image inpainting, image restoration, image completion, multi-resolution in- painting, multilayer inpainting, image processing

Link to PDF

 

Keynote Speech:

  1. Keynote Speech, Video Forgery and Motion Editing, the 2011 International Conference on Advances in Electrical Engineering (ICAEE 2011), Dhaka, Bangladesh, December 19 – 21, 2013.
  2. Keynote Speech, Video Forgery and Motion Editing, International Conference on Advances in ICT for Emerging Regions (ICTer2013), Colombo, Sri Lanka, December 11 – 15, 2013.
  3. Keynote Speech, Video Forgery and Motion Editing, the 5th International Conference on Web Information Systems and Mining (WISM’13) and the 5th International Conference on Artificial Intelligence and Computational Intelligence (AICI’13), Guilin, Guangxi, China, 13-15 August 2013.
  4. Keynote Speech, Video Forgery and Motion Editing, the 9th. HSSS National & International Conference, Volos, Greece, 11 – 13 July, 2013.
  5. Keynote Speech,Video Forgery, the Third International Conference on Intelligent Networking and Collaborative Systems (INCoS 2011), Fukuoka, Japan November 30 – December 2, 2011.
  6. Keynote Speech, Video Forgery, the 3rd International Conference on Mechanical and Electronics Engineering (ICMEE 2011), Hefei, China, September 23-25, 2011.
  7. Keynote Speech, Video Forgery, the 4th International Conference on Ubi-media Computing (U-Media 2011), Sao Paulo, Brazil, July 3-4, 2011.
  8. Keynote Speech, Video Forgery, the First International Conference on Multimedia, Computer Graphics and Broadcasting (MulGraB 2009), Jeju Island, Korea, December 10 ~ 12, 2009.
  9. Keynote Speech, Video Forgery, the First International Conference on Visual Informatics 2009 (ICVI’09), Kuala Lumpur, Malaysia, November 11 – 13, 2009.
  10. Keynote Speech, Video Forgery, the 2009 IEEE International Conference on Information Assurance and Security (IAS’2009), Xi’an, China, August 18-20, 2009.
  11. Keynote Speech, Video Forgery, Computer Application in Control, Instrument and Simulation, (CACIS 2009), Nanning, Guangxi, China, July 18-20, 2009.
  12. Keynote Speech, Video Falsifying, the International IEEE Information Technology Symposium 2008 (IEEE ITSIM’ 08), Kuala Lumpur, Malaysia, August 26-29, 2008.
  13. Keynote Speech, Video Inpainting, the Second International Conference on Complex, Intelligent and Software Intensive Systems (CISIS-2008), Barcelona, Spain, March 4th-7th, 2008.
  14. Keynote Speech, Video Inpainting, the Asia-Pacific Workshop on Visual Information Processing 2007 (VIP 2007), Tainan, Taiwan, December 15-17, 2007.
  15. Keynote Speech, Video Inpainting, the 2007 International Conference on Multimedia and Ubiquitous Engineering (MUE’2007), Seoul, Korea, April 26th – 28th, 2007.
  16. Keynote Speech, Digital Inpainting, the 2007 IASTED International Conference on Graphics and Visualization in Engineering (GVE 2007), Clearwater, Florida, USA, January 3-5, 2007.
  17. Keynote Speech, Digital Inpainting, the 2006 International Workshop on Advanced Multimedia Technology and Applications (AMTA2006), Cheju Island, Korea, November 9-11, 2006.
  18. Keynote Speech, Digital Inpainting, the Eleventh International Conference on Distributed Multimedia Systems (DMS 2005), Banff, Canada, September 5 – 7, 2005.
  19. Keynote Speech, Digital Inpainting, the 3rd International Conference on Information Technology & Applications (ICITA 2005), Sydney, Australia, July 4 – 7, 2005.
  20. Keynote Speech, Digital Inpainting, the 2004 International Conference on Computer and Information Technology (CIT2004), Wuhan, China, September 14 – 16, 2004.

Testing Data

  • Testing Data (only Source Video) Download 
  • [Video Forgery and Motion Editing] ppt File  Download
  • [Video Forgery and Motion Editing] Source Video+ Inpainting Result Download

Please cite our paper if you use these data!