• What is human body shape?
  • How does your shape change with body pose?
  • How is shape related to health?
  • How can it be measured accurately?
  • What would you look like if you lost weight or exercised more?

3D body shape capture is becoming more common with scanning devices ranging from high quality systems like our 3dMD system to low-cost range sensors like the Microsoft Kinect.  In all cases, the result of a "scan" is simply a static 3D representation of the body.  Our goal is to bring such scans to "life" fully automatically, resulting in digital avatars the look like the person, deform like them, and can serve as a proxy for clothing shopping or clothing try-on.

Our research addresses all aspects of this problem, starting with a raw scan.

1. We automatically fit the scan with a posable avatar without the use of markers or manual intervention.  Automatic fitting is challenging because the body model is high dimensional, the pose is unknown, and the scan may contain noise or holes.

2. Given many scans of a single person, we bring them all into alignment with a single, individulaized, model using a process we call co-registration.  Co-registration simultaneously aligns all the scans of the person to the model while building a statistical model of the person that is used, in turn, to constrain the registration.  This process results in a digital avatar that captures the detailed shape and pose variation of the subject.  This avatar can be reposed and animated with realistic soft-tissue deformations.  Unlike traditional blend-skinned models, our tissued deformations are fully natural and are learned from the individual.

3. Given the recovered model of the body we can extract measurements of the body.  Our model-based approach draws on the statistics of body shape learned from approximately 4000 body scans.  Unlike previous methods, we use a variety of local and global features to accurately extract measurements with accuracies that exceed current commericial systems.

4. The shape of the avatar is a parametric model, learned from the statistics of the population.  This allows us to edit body shape by constraining different measurements.  One can change, weight, height, bust size, waist circumference, etc.  In all cases, the body shape represents the most likely shape in the population with those measurements.

5. We have also learned a statistical clothing model called DRAPE that allows our avatars to be automatically dressed.  We start with a standard clothing pattern and dress training bodies using physical simulation.  We effectively "compile" the information about how clothing shape changes with body shape and pose into a statistical model that can be efficiently and automatically applied to any new body shape in any pose. The result is appropriate for on-line virtual try-on.

Various research topics in this theme:
  • Body shape from images - Estimating body shape from images and video.
  • Learning the statistical variation in body shape.
  • Anthropometry - Measuring the moving body.
  • Capturing shape change and animating the body.
  • Recognizing mood, expression, and action from shape and pose.
  • Modeling the dynamics of fat and muscle.
  • Capturing hair and clothing.
  • "X-ray vision" - Estimating what is under the skin.
This suite of technologies represents a major, multi-year, multi-institution project.  Our aim has been to understand body shape and how it varies with pose.  The result is a highly realsitic model of the body that can be used in a wide range of applications:
  • Health - predicting disease from shape.
  • Fitness - how diet and exercise effects shape.
  • Ergonomics - how bodies fit products.
  • Graphics - animation for games and films.



Dyna: A Model of Dynamic Human Shape in Motion
Pons-Moll, G., Romero, J., Mahmood, N. and Black, M.J.
ACM Transactions on Graphics, (Proc. SIGGRAPH), 34(4):120:1-120:14, August 2015.
The Stitched Puppet: A Graphical Model of 3D Human Shape and Pose
Zuffi, S. and Black, M.J.
In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) 2015, June 2015.
Can I recognize my body’s weight? The influence of shape and texture on the perception of self
Piryankova, I., Stefanucci, J., Romero, J., de la Rosa, S., Black, M. and Mohler, B.
ACM Transactions on Applied Perception for the Symposium on Applied Perception, 11(3):13:1-13:18, 2014.
MoSh: Motion and Shape Capture from Sparse Markers
Loper, M.M., Mahmood, N. and Black, M.J.
ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 33(6):220:1-220:13, November 2014.
Breathing Life into Shape: Capturing, Modeling and Animating 3D Human Breathing
Tsoli, A., Mahmood, N. and Black, M.J.
ACM Transactions on Graphics, (Proc. SIGGRAPH), 33(4):52:1-52:11, July 2014.
FAUST: Dataset and evaluation for 3D mesh registration
Bogo, F., Romero, J., Loper, M. and Black, M.J.
In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3794 -3801, Columbus, Ohio, USA. June 2014.
Model-based Anthropometry: Predicting Measurements from 3D Human Scans in Multiple Poses
Tsoli, A., Loper, M. and Black, M.J.
In Proceedings Winter Conference on Applications of Computer Vision, IEEE , pages 83-90, March 2014.
Viewpoint and pose in body-form adaptation
Sekunova, A., Black, M., Parkinson, L. and Barton, J.J.S.
Perception, 42(2):176-186, 2013.
Coregistration: Simultaneous alignment and modeling of articulated 3D shape
Hirshberg, D., Loper, M., Rachlin, E. and Black, M.J.
In European Conf. on Computer Vision (ECCV), Springer-Verlag, LNCS 7577, Part IV, pages 242-255, October 2012.
From Deformations to Parts: Motion-based Segmentation of 3D Objects
Ghosh, S., Sudderth, E., Loper, M. and Black, M.
In Advances in Neural Information Processing Systems 25 (NIPS), MIT Press, pages 2006-2014, 2012.
Virtual Human Bodies with Clothing and Hair: From Images to Animation
Guan, P.
PhD thesis. Brown University, Department of Computer Science, December 2012.
Lie Bodies: A Manifold Representation of 3D Human Shape
Freifeld, O. and Black, M.J.
In European Conf. on Computer Vision (ECCV), Springer-Verlag, Part I, LNCS 7572, pages 1-14, October 2012.
DRAPE: DRessing Any PErson
Guan, P., Reiss, L., Hirshberg, D., Weiss, A. and Black, M.J.
ACM Trans. on Graphics (Proc. SIGGRAPH), 31(4):35:1-35:10, July 2012.
From pictorial structures to deformable structures
Zuffi, S., Freifeld, O. and Black, M.J.
In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), IEEE, pages 3546-3553, June 2012.
Shape and pose-invariant correspondences using probabilistic geodesic surface embedding
Tsoli, A. and Black, M.J.
In 33rd Annual Symposium of the German Association for Pattern Recognition (DAGM), Springer, volume 6835, Lecture Notes in Computer Science, pages 256-265, 2011.
Parameterized Model of 2D Articulated Human Shape
Black, M.J., Freifeld, O., Weiss, A., Loper, M. and Guan, P.
Patent application: PCT/US2011/039605, June 2011.
Evaluating the Automated Alignment of 3D Human Body Scans
Hirshberg, D.A., Loper, M., Rachlin, E., Tsoli, A., Weiss, A., Corner, B. and Black, M.J.
In 2nd International Conference on 3D Body Scanning Technologies, Hometrica Consulting, pages 76-86, Lugano, Switzerland. October 2011.
Home 3D body scans from noisy image and range data
Weiss, A., Hirshberg, D. and Black, M.J.
In Int. Conf. on Computer Vision (ICCV), IEEE, pages 1951-1958, Barcelona. November 2011.
A 2D human body model dressed in eigen clothing
Guan, P., Freifeld, O. and Black, M.J.
In European Conf. on Computer Vision, (ECCV), Springer-Verlag, pages 285-298, September 2010.
Contour people: A parameterized model of 2D articulated human shape
Freifeld, O., Weiss, A., Zuffi, S. and Black, M.J.
In IEEE Conf. on Computer Vision and Pattern Recognition, (CVPR), IEEE, pages 639-646, June 2010.
Estimating human shape and pose from a single image
Guan, P., Weiss, A., Balan, A. and Black, M.J.
Int. Conf. on Computer Vision, ICCV, pages 1381-1388, September 2009.
Method and Apparatus for Estimating Body Shape
Black, M.J., Balan, A., Weiss, A., Sigal, L., Loper, M. and St Clair, T.
US (12/541,898) and PCT patent application, US (12/541,898) and PCT patent application, August 2009.
The naked truth: Estimating body shape under clothing,
Balan, A. and Black, M.J.
In European Conf. on Computer Vision, ECCV, Springer-Verlag, volume 5304, LNCS, pages 15-29, Marseilles, France. October 2008.
Combined discriminative and generative articulated pose and non-rigid shape estimation
Sigal, L., Balan, A. and Black, M.J.
In Advances in Neural Information Processing Systems 20, NIPS-2007, MIT Press, pages 1337–1344, 2008.
Shining a light on human pose: On shadows, shading and the estimation of pose and shape,
Balan, A., Black, M.J., Haussecker, H. and Sigal, L.
In Int. Conf. on Computer Vision, ICCV, pages 1-8, Rio de Janeiro, Brazil. October 2007.
Detailed human shape and pose from images
Balan, A., Sigal, L., Black, M.J., Davis, J. and Haussecker, H.
In IEEE Conf. on Computer Vision and Pattern Recognition, CVPR, pages 1-8, Minneapolis. June 2007.
An adaptive appearance model approach for model-based articulated object tracking
Balan, A. and Black, M.J.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, CVPR, volume 1, pages 758-765, New York, NY. June 2006.