Blog

iPhone X Facial Capture – Apple blendshapes

By on May 22, 2018

Project bebyface is a series of tests I’ve been doing to find a fast pipeline for performance capture (Face and body) that we can use for our up and coming game Bebylon: Battle Royale. http://bebylon.world

Check out the previous tests for more info regarding the iPhone X capture process.  (starting with the most recent)


 

Apple ARKit Blendshapes

Along this journey a super cool developer extracted and sent me Apples blendshapes used in ARKit which I used in Maya (using built-in deformers) to generate a fresh set of blendshapes for our beby character. They were not a drastic improvement over the make shift blendshapes I originally made but they definitely helped achieve a more natural look and act as a perfect starting point for creative augmentation.   I wish this was a more full blown tutorial on that process (Hopefully thats something i can do very soon) but in the meantime, I’m attaching a link to the apple blendshape set that you can use as a reference or to generate blendshapes for your own character.

 

the Apple blendshapes are OBJ
 Apple blendshapes51 OBJs.zip

TAGS
RELATED POSTS
9 Comments
  1. Reply

    James Mack

    May 26, 2018

    Hey, I loved your demo of Apples motion tracking tech. I have the original Faceshift and I was wondering how Apple was going to develop it. One quick question: Will you be releasing commercially the app you are using for the capture and transfer to Maya? I want to develop my own pipeline and this looks promising. Before I invest the time in programming, I would like to know what your future plans are. Perhaps I can wait on you to release what you have.

    • Reply

      Cory Strassburger

      May 28, 2018

      thanks James! I don’t have any plans to develop a commercial version, partially due to the lack of time and I mainly developed this for generating data for our game. I think its a worthy work flow to build a commercial app around, especially considering how fast, easy and flexible it is for the quality.

      • Reply

        amine

        July 3, 2018

        Hello Cory!
        Amazing progress!
        I just want to ask you how is the iPhoneX performance compared to the old faceshift software if you have tried the later?
        thank you

        • Reply

          Cory Strassburger

          July 12, 2018

          Hey amine, I have used faceshift in the past and from my experience i’d say the arkit implementation is on par with the desktop version in terms of overall quality. Faceshift of course had a lot of features that i dont believe are accessible (or even integrated) in Arkit but that hasn’t been an issue for me so far.

      • Reply

        Graham

        July 11, 2018

        Wow uhhhh you REALLY should release something only because it would blow the lid off of other motion tracking software. EVERYONE would be flooding your way to create things and you’d START A REVOLUTION!

        … Well maybe less dramatic than that, but wow seriously… I can’t believe all of this. It’s something I always wanted when the iphone x was announced and I think you’ve got the best bet on making something quality. Keep up the good work, it’s extremely fascinating and I can’t wait to see your insane battle royale game come to life with this!

        • Reply

          Cory Strassburger

          July 12, 2018

          Lol, thanks Graham!!

    • Reply

      Cory Strassburger

      August 10, 2018

      Hey james, im obviously a bit lacking in response time! Did you end up jumping into it? I think a few apps have surfaced that accomplish the general task but not sure how easy that data can get into maya. In unreal engine its a bit easier with 4.20 to just capture the data live into the engine but Im not sure you can export it yet into maya?

  2. Reply

    J.N

    August 9, 2018

    What a great job! Amaging😆👍💕 I have a question. At 2:28 in the part 4 video, I have a question about merge another head.
    how did you apply Blend Shape when you combine or merge modeling together? (Face area and back head)
    The value of vertex changes, doesn’t it?

    • Reply

      Cory Strassburger

      August 10, 2018

      thanks j.n! my blendshapes for the beby are actually of the whole head (not just the front of the face). Though what your seeing is the apple blendshapes which are just the front of the face but I only used them to deform my beby characters face. I basically ‘wrap deform’ the apple blendshape to my beby characters head and use ‘delta mush’ to blend it smoothly and maintain the general shape of my beby character. the combination of the 2 deformers allow the apple blendshape to deform my beby characters face with a smooth transition into the rest of its head, If that makes sense? Honestly there is probably a better way to do it, maybe a plugin that makes the whole process fast and a better transfer of the deltas.

LEAVE A COMMENT

This site uses Akismet to reduce spam. Learn how your comment data is processed.