Blog

iPhone X Facial Capture – Apple blendshapes

By on May 22, 2018

Project bebyface is a series of tests I’ve been doing to find a fast pipeline for performance capture (Face and body) that we can use for our up and coming game Bebylon: Battle Royale. http://bebylon.world

Check out the previous tests for more info regarding the iPhone X capture process.  (starting with the most recent)


 

Apple ARKit Blendshapes

Along this journey a super cool developer extracted and sent me Apples blendshapes used in ARKit which I used in Maya (using built-in deformers) to generate a fresh set of blendshapes for our beby character. They were not a drastic improvement over the make shift blendshapes I originally made but they definitely helped achieve a more natural look and act as a perfect starting point for creative augmentation.   I wish this was a more full blown tutorial on that process (Hopefully thats something i can do very soon) but in the meantime, I’m attaching a link to the apple blendshape set that you can use as a reference or to generate blendshapes for your own character.

 

the Apple blendshapes are OBJ
 Apple blendshapes51 OBJs.zip

TAGS
RELATED POSTS
16 Comments
  1. Reply

    James Mack

    May 26, 2018

    Hey, I loved your demo of Apples motion tracking tech. I have the original Faceshift and I was wondering how Apple was going to develop it. One quick question: Will you be releasing commercially the app you are using for the capture and transfer to Maya? I want to develop my own pipeline and this looks promising. Before I invest the time in programming, I would like to know what your future plans are. Perhaps I can wait on you to release what you have.

    • Reply

      Cory Strassburger

      May 28, 2018

      thanks James! I don’t have any plans to develop a commercial version, partially due to the lack of time and I mainly developed this for generating data for our game. I think its a worthy work flow to build a commercial app around, especially considering how fast, easy and flexible it is for the quality.

      • Reply

        amine

        July 3, 2018

        Hello Cory!
        Amazing progress!
        I just want to ask you how is the iPhoneX performance compared to the old faceshift software if you have tried the later?
        thank you

        • Reply

          Cory Strassburger

          July 12, 2018

          Hey amine, I have used faceshift in the past and from my experience i’d say the arkit implementation is on par with the desktop version in terms of overall quality. Faceshift of course had a lot of features that i dont believe are accessible (or even integrated) in Arkit but that hasn’t been an issue for me so far.

      • Reply

        Graham

        July 11, 2018

        Wow uhhhh you REALLY should release something only because it would blow the lid off of other motion tracking software. EVERYONE would be flooding your way to create things and you’d START A REVOLUTION!

        … Well maybe less dramatic than that, but wow seriously… I can’t believe all of this. It’s something I always wanted when the iphone x was announced and I think you’ve got the best bet on making something quality. Keep up the good work, it’s extremely fascinating and I can’t wait to see your insane battle royale game come to life with this!

        • Reply

          Cory Strassburger

          July 12, 2018

          Lol, thanks Graham!!

    • Reply

      Cory Strassburger

      August 10, 2018

      Hey james, im obviously a bit lacking in response time! Did you end up jumping into it? I think a few apps have surfaced that accomplish the general task but not sure how easy that data can get into maya. In unreal engine its a bit easier with 4.20 to just capture the data live into the engine but Im not sure you can export it yet into maya?

  2. Reply

    J.N

    August 9, 2018

    What a great job! Amaging😆👍💕 I have a question. At 2:28 in the part 4 video, I have a question about merge another head.
    how did you apply Blend Shape when you combine or merge modeling together? (Face area and back head)
    The value of vertex changes, doesn’t it?

    • Reply

      Cory Strassburger

      August 10, 2018

      thanks j.n! my blendshapes for the beby are actually of the whole head (not just the front of the face). Though what your seeing is the apple blendshapes which are just the front of the face but I only used them to deform my beby characters face. I basically ‘wrap deform’ the apple blendshape to my beby characters head and use ‘delta mush’ to blend it smoothly and maintain the general shape of my beby character. the combination of the 2 deformers allow the apple blendshape to deform my beby characters face with a smooth transition into the rest of its head, If that makes sense? Honestly there is probably a better way to do it, maybe a plugin that makes the whole process fast and a better transfer of the deltas.

  3. Reply

    LUFFY

    October 15, 2018

    Hey man, my 3D model has big eyes and I try to use “wrap deform” to make blend shape which accroding to 51 blenshapes by Arkit. But the question is my model cannot close eye completely and I already try my best still can’t figure out. Could you tell me how to fix this plz? Thx.

    • Reply

      Cory Strassburger

      October 16, 2018

      If its just the eyes closed blendshapes I would use the grab sculpt tool and just manually close them the reset of the way.

  4. Reply

    Nick

    October 16, 2018

    Hey Cory, Is this blendshape generation method what you used for your final product or did you go another route?

    • Reply

      Cory Strassburger

      October 16, 2018

      Hey Nick, that method was the starting point for the final results and then I sculpted on top of a bunch of key shapes after that.

  5. Reply

    pavel

    November 8, 2018

    Hello Cory, i have a question about closeMouth blendShape,
    it should be ugly same as for faceShift because it’s corrective pose to bring lips closed with jawOpen blendShape at 100% or not?, for instance in scene with Sloth there was closeMouth blendShape act like correct, sorry for bad english),

    • Reply

      Cory Strassburger

      November 9, 2018

      Hey Pavel thats right, the MouthClosed should be lips together when jawOpen is at 100%, which looks overblown by itself. Typically the captured value of mouthClosed is very very small.

  6. Reply

    Nick

    November 15, 2018

    Hey Cory,

    thanks for the response earlier. I’m using your wrap deformer technique, but I’m having a hard time when it comes to the lips because they are very close together. The top lip gets affected by the movement of the bottom lip and vice versa. I was wondering if you had a process for this? The best way I can get it working is by opening the mouth a little on my main character and aligning the inner lips of the apple mask to fit through the slot. I’d prefer if there was a cleaner way of doing this without having to open the mouth on my character.

    Thanks!

LEAVE A COMMENT

This site uses Akismet to reduce spam. Learn how your comment data is processed.