SCIENTIA SINICA Informationis, Volume 48 , Issue 3 : 248-260(2018) https://doi.org/10.1360/N112017-00252

3D human face transplanting via depth camera

More info
  • ReceivedJan 4, 2018
  • AcceptedFeb 5, 2018
  • PublishedMar 19, 2018


This paper presents a 3D face transplanting system with a general applicability and high efficiency. The system is capable of transplanting faces between 3D mesh models with textures quickly and simply. The main characteristic of this system is that it gets good effects on most 3D characters. This system performs face transformation and deformation with the help of the Laplace operators of the face mesh to maintain the similarity between face details. By combining Poisson image editing method and color optimization method, this system can make the texture tunes consistent and obtain a smooth transition. To make it more complete, an automatic 3D face scanning system is created to obtain 3D faces with high-quality meshes and high-resolution textures efficiently. Compared to the previous practice, the main contribution of this paper is the capability to change face transplanting objects from 2D pictures to 3D models. Moreover, the seamless connection of the two systems forms a 3D face application with a high level of automation.

Funded by






[1] Blanz V, Scherbaum K, Vetter T. Exchanging Faces in Images. Comput Graphics Forum, 2004, 23: 669-676 CrossRef Google Scholar

[2] Bitouk D, Kumar N, Dhillon S, et al. Face swapping: automatically replacing faces in photographs. ACM Trans Graph, 2008, 27: 39. Google Scholar

[3] Kemelmacher-Shlizerman I. Transfiguring portraits. ACM Trans Graph, 2016, 35: 94. Google Scholar

[4] Jing Tong , Jin Zhou , Ligang Liu . Scanning 3D full human bodies using Kinects.. IEEE Trans Visual Comput Graphics, 2012, 18: 643-650 CrossRef PubMed Google Scholar

[5] Zhou J, Pan J J, Tong J, et al. Fast 3D human body reconstruction from Kinects. J Comput-Aid Des Comput Graph, 2013, 25: 873--879. Google Scholar

[6] Yang H Z, Lu Y, Fang Q, et al. Automatic 3D scanning system via depth camera. J Comput-Aid Des Comput Graph, 2015, 27: 2039--2045. Google Scholar

[7] Zhou W, Guan Z Q, Lu Y, et al. Easy 3D self-portraits. In: Proceedings of the 10th China Computer Graphics Conference and the 18th National CAGD Conference Papers. Hefei: USTC, 2014. 178. Google Scholar

[8] Chandawarkar A A, Diaz-Siso J R, Bueno E M. Facial appearance transfer and persistence after three-dimensional virtual face transplantation.. Plastic Reconstructive Surgery, 2013, 132: 957-966 CrossRef PubMed Google Scholar

[9] Pérez P, Gangnet M, Blake A. Poisson image editing. ACM Trans Graph, 2003, 22: 313-318 CrossRef Google Scholar

[10] Reinhard E, Ashikhmin M, Gooch B, et al. Color transfer between images. IEEE Comput Graph Appl, 2001, 21: 34--41. Google Scholar

[11] Lempitsky V, Ivanov D. Seamless mosaicing of image-based texture maps. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, 2007. Google Scholar

[12] Sorkine O, Cohen-Or D, Lipman Y, et al. Laplacian surface editing. In: Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing, Nice, 2004. 175--184. Google Scholar

[13] Belkin M, Sun J, Wang Y. Discrete Laplace operator on meshed surfaces. In: Proceedings of the 24th Annual Symposium on Computational Geometry, College Park, 2008. Google Scholar

[14] Newcombe R A, Izadi S, Hilliges O, et al. KinectFusion: real-time dense surface mapping and tracking. In: Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Basel, 2011. 127--136. Google Scholar

[15] Kazhdan M, Hoppe H. Screened Poisson surface reconstruction. ACM Trans Graph, 2013, 32: 29. Google Scholar

[16] Kazhdan M, Bolitho M, Hoppe H. Poisson surface reconstruction. In: Proceedings of the 4th Eurographics Symposium on Geometry Processing, Cagliari, 2006. 61--70. Google Scholar

[17] Floater M S. Parametrization and smooth approximation of surface triangulations. Comput Aided Geometric Des, 1997, 14: 231-250 CrossRef Google Scholar

[18] Field D A. Laplacian smoothing and Delaunay triangulations. Int J Numer Method Biomed Eng, 1988, 4: 709--712. Google Scholar

  • Figure 1

    (Color online) 3D face transplanting effects. (a) Testing subject; (b) transplanting based on an image; (c) result of this paper

  • Figure 2

    (Color online) Symbolic description

  • Figure 3

    (Color online) Face scanning

  • Figure 4

    System flow

  • Figure 5

    (Color online) Face extracting procedure. (a) Original point cloud; (b) crop via depth image; (c) crop via feature points; (d) cropping result; (e) geo-reconstruction; (f) texture reconstruction and feature points

  • Figure 6

    (Color online) Geo-detail comparisons between Poisson and Screen Poisson reconstructions (${\rm~Depth}=8$). Details of (a) Poisson and (b) Screen Poisson reconstruction method

  • Figure 7

    (Color online) Laplace vector

  • Figure 8

    (Color online) Geo-warping of scanned face. (a) Rigid transform; (b) correspondence of boundary; (c) result of warping

  • Figure 9

    (Color online) Rigid transform effects between 5 pairs and 78 pairs of feature points. (a) Orthographic view; (b) overhead view; (c) side view

  • Figure 10

    (Color online) Boundary smoothing. (a) Before smoothing; (b) after smoothing

  • Figure 11

    (Color online) Texturing effect comparisons. (a) Initial texture; (b) color optimization; (c) Poisson editing; (d) combine both methods

  • Figure 12

    (Color online) Female character with male face

  • Figure 13

    (Color online) Avatar with female face

  • Figure 14

    (Color online) Cartoon model with human face

  • Figure 15

    (Color online) Facial detail enhancement


    Algorithm 1 自动三维面部提取算法

    Require:人脸三维点云$C_s$, 采样$(I_c,I_d,P)$;


    借助采样数据进行点云的初步裁剪 (如图4(b)): 将深度图$I_d$转化为点云, 并借助相机位置$P$将点云注册到$C_s$位置处, 记为$C_d$, 定义$C_s$ 上一点$\boldsymbol{p}$到$C_d$的距离: $$d(\boldsymbol{p},C_d)={\rm min}_{\boldsymbol{q}\in{C_d}}{{\rm length}(\boldsymbol{p}-\boldsymbol{q})},$$ 给定阈值$d_{\rm~max}$, 若$d(\boldsymbol{p},C_d)>d_{\rm~max}$, 则此$\boldsymbol{p}$点舍弃.

    借助特征点位置对点云进一步裁剪 (如图5(c)): 从采样图$I_c$可自动检测人脸特征点, 利用$I_c$ 与$I_d$的对应关系及相机位置$P$, 可获取$C_d$ 上特征点位置, 特征点的最低高度与最高高度分别设为$h_{\rm~min}$与$h_{\rm~max}$, 由于人脸基本处于竖直状态, 预设offset为0.01 m, 若$p_y<h_{\rm~min}-{\rm~offset}$或$p_y>h_{\rm~max}+{\rm~offset}$, 则此$\boldsymbol{p}$点舍弃.

    使用Screen Poisson [15]表面重建算法提取三角网格 (如图5(e)).


    Algorithm 2 面部网格变形算法

    Require:$F_s$, $F_m$和$M_l$;


    利用特征点刚性变换$F_s$至$F_m$ (图8(a));

    寻找$F_s$与$M_l$人脸边缘的对应关系, 以确定人脸新的边界点位置 (图8(b));

    使用基于网格Laplace的变形算法, 构造sect. 5.1小节中的线性系统求解顶点坐标, 变形$F_s$以适应$M_l$的面部边界轮廓 (图8(c)).