Blog

iPhone X Facial Capture – Apple blendshapes

By on May 22, 2018

Project bebyface is a series of tests I’ve been doing to find a fast pipeline for performance capture (Face and body) that we can use for our up and coming game Bebylon: Battle Royale. http://bebylon.world

Check out the previous tests for more info regarding the iPhone X capture process.  (starting with the most recent)


 

Apple ARKit Blendshapes

Along this journey a super cool developer extracted and sent me Apples blendshapes used in ARKit which I used in Maya (using built-in deformers) to generate a fresh set of blendshapes for our beby character. They were not a drastic improvement over the make shift blendshapes I originally made but they definitely helped achieve a more natural look and act as a perfect starting point for creative augmentation.   I wish this was a more full blown tutorial on that process (Hopefully thats something i can do very soon) but in the meantime, I’m attaching a link to the apple blendshape set that you can use as a reference or to generate blendshapes for your own character.

 

the Apple blendshapes are OBJ
 Apple blendshapes51 OBJs.zip

TAGS
RELATED POSTS
21 Comments
  1. Reply

    James Mack

    May 26, 2018

    Hey, I loved your demo of Apples motion tracking tech. I have the original Faceshift and I was wondering how Apple was going to develop it. One quick question: Will you be releasing commercially the app you are using for the capture and transfer to Maya? I want to develop my own pipeline and this looks promising. Before I invest the time in programming, I would like to know what your future plans are. Perhaps I can wait on you to release what you have.

    • Reply

      Cory Strassburger

      May 28, 2018

      thanks James! I don’t have any plans to develop a commercial version, partially due to the lack of time and I mainly developed this for generating data for our game. I think its a worthy work flow to build a commercial app around, especially considering how fast, easy and flexible it is for the quality.

      • Reply

        amine

        July 3, 2018

        Hello Cory!
        Amazing progress!
        I just want to ask you how is the iPhoneX performance compared to the old faceshift software if you have tried the later?
        thank you

        • Reply

          Cory Strassburger

          July 12, 2018

          Hey amine, I have used faceshift in the past and from my experience i’d say the arkit implementation is on par with the desktop version in terms of overall quality. Faceshift of course had a lot of features that i dont believe are accessible (or even integrated) in Arkit but that hasn’t been an issue for me so far.

      • Reply

        Graham

        July 11, 2018

        Wow uhhhh you REALLY should release something only because it would blow the lid off of other motion tracking software. EVERYONE would be flooding your way to create things and you’d START A REVOLUTION!

        … Well maybe less dramatic than that, but wow seriously… I can’t believe all of this. It’s something I always wanted when the iphone x was announced and I think you’ve got the best bet on making something quality. Keep up the good work, it’s extremely fascinating and I can’t wait to see your insane battle royale game come to life with this!

        • Reply

          Cory Strassburger

          July 12, 2018

          Lol, thanks Graham!!

    • Reply

      Cory Strassburger

      August 10, 2018

      Hey james, im obviously a bit lacking in response time! Did you end up jumping into it? I think a few apps have surfaced that accomplish the general task but not sure how easy that data can get into maya. In unreal engine its a bit easier with 4.20 to just capture the data live into the engine but Im not sure you can export it yet into maya?

  2. Reply

    J.N

    August 9, 2018

    What a great job! Amaging😆👍💕 I have a question. At 2:28 in the part 4 video, I have a question about merge another head.
    how did you apply Blend Shape when you combine or merge modeling together? (Face area and back head)
    The value of vertex changes, doesn’t it?

    • Reply

      Cory Strassburger

      August 10, 2018

      thanks j.n! my blendshapes for the beby are actually of the whole head (not just the front of the face). Though what your seeing is the apple blendshapes which are just the front of the face but I only used them to deform my beby characters face. I basically ‘wrap deform’ the apple blendshape to my beby characters head and use ‘delta mush’ to blend it smoothly and maintain the general shape of my beby character. the combination of the 2 deformers allow the apple blendshape to deform my beby characters face with a smooth transition into the rest of its head, If that makes sense? Honestly there is probably a better way to do it, maybe a plugin that makes the whole process fast and a better transfer of the deltas.

  3. Reply

    Luffy

    October 15, 2018

    Hey Cory,I have been paying attention to your development. Thank you for bringing us so many surprises, I have always had a question. My model is very cartoon with large round eyes,I basically ‘wrap deform’ the apple blendshape to my characters head,Unfortunately
    The eyes of my model always have problems, I can’t close them.,Is there any way to solve it? Looking forward to your reply.

  4. Reply

    LUFFY

    October 15, 2018

    Hey man, my 3D model has big eyes and I try to use “wrap deform” to make blend shape which accroding to 51 blenshapes by Arkit. But the question is my model cannot close eye completely and I already try my best still can’t figure out. Could you tell me how to fix this plz? Thx.

    • Reply

      Cory Strassburger

      October 16, 2018

      If its just the eyes closed blendshapes I would use the grab sculpt tool and just manually close them the reset of the way.

  5. Reply

    Nick

    October 16, 2018

    Hey Cory, Is this blendshape generation method what you used for your final product or did you go another route?

    • Reply

      Cory Strassburger

      October 16, 2018

      Hey Nick, that method was the starting point for the final results and then I sculpted on top of a bunch of key shapes after that.

  6. Reply

    pavel

    November 8, 2018

    Hello Cory, i have a question about closeMouth blendShape,
    it should be ugly same as for faceShift because it’s corrective pose to bring lips closed with jawOpen blendShape at 100% or not?, for instance in scene with Sloth there was closeMouth blendShape act like correct, sorry for bad english),

    • Reply

      Cory Strassburger

      November 9, 2018

      Hey Pavel thats right, the MouthClosed should be lips together when jawOpen is at 100%, which looks overblown by itself. Typically the captured value of mouthClosed is very very small.

  7. Reply

    Daniel

    November 14, 2018

    Wow, it is too amazing what you’ve done. I’m on the way to making an iOS app and concerning making the model. I was not able to import the face model made on 3d max exported as DAE file format. After importing to Xcode, all the blend shape keys are broken. Can you help me why it is happening?
    Is it problem using 3d max and should use Maya? Or any other reason?

    • Reply

      Cory Strassburger

      November 19, 2018

      Thanks Daniel! I’ve been using Unreal Engine for all the coding side and not having to mess with Xcode directly so i’m not sure what your running into. Exporting from Max should be fine. Have you tried exporting to FBX vs DAE?

  8. Reply

    Nick

    November 15, 2018

    Hey Cory,

    thanks for the response earlier. I’m using your wrap deformer technique, but I’m having a hard time when it comes to the lips because they are very close together. The top lip gets affected by the movement of the bottom lip and vice versa. I was wondering if you had a process for this? The best way I can get it working is by opening the mouth a little on my main character and aligning the inner lips of the apple mask to fit through the slot. I’d prefer if there was a cleaner way of doing this without having to open the mouth on my character.

    Thanks!

    • Reply

      Cory Strassburger

      November 19, 2018

      Sadly Nick im in the same boat. I’ve had to adjust the lips on a couple of the characters for the same reason. I might be re-approaching the process on some new characters coming up and if that pans out i’ll let you know.

  9. Reply

    Nick

    November 22, 2018

    Hey Cory, sounds awesome, thanks. So far I’ve been readjusting the lips after I’ve applied the blendshapes and it works out fine. I’ve finished one character using only the apple masks deformations on the model (without any extra work done) and it’s turned out pretty good. I’m going to be running the same setup you used for your Siggraph presentation, so I should be able to pump out some content pretty soon.

    Thanks!

LEAVE A COMMENT

This site uses Akismet to reduce spam. Learn how your comment data is processed.