GSOC Proposal for Blender auto facial animations

Contributing to Blender Summer of Code
,
1

Name

Shivankar Sharma

Contact

Email: [email protected]

Synopsis

Auto animator for dialogues. Using predefined shape keys for the animator.

Benefits

I have been using blender for more than eight years as of now, and I know the potential of the software, and at last, it has started achieving the recognition it deserves. Even with the addition of the Rigify feature, blender helped many studios a lot to work faster by my project, the process for animating dialogues will become more comfortable. As you can write the full sentence in the box and it will auto animate the model, and blender will gain more name in the community.

Deliverables

A new feature in blender where artists can load the shape keys for different expressions and finally can put the sentence they want to animate. After analyzing the words, the model’s expressions will get animated.

Project Details

So I was working in a company and using unity to develop a 2d game. I was having problems animating the mouth of the character, so I developed an in house software for the company to be able to enter the sentence they want to be animated, and the character’s mouth will animate itself. This software struck me with the idea of integrating this to blender .In my opinion, this is really good for blender. Now how I want this to work is suppose I have a character named “John” now the artist can create different shape keys to animate for different words like this for the mouth.

Now the artist will assign the shape keys to the predefined slot names that we will give them example: {A},{O},{E},{W, R},{T, S},{L, N},{U, Q},{M, B, P},{F, V} as shown in the image above. Now, if a person wants to animate a dialogue, a person can’t enter the dialogue in the text box, and our system will analyze this. It will iterate to the sentence catching these phrases and will animate through itself. Using the shape keys provided by the artist and, in the end, can also change the influence of them.

Project Schedule

The project can take from 1 month to prototyping it and another month polishing it and a few weeks to fix the bugs if there exist some.

Timeline:

  1. Deliver the early prototype till the end of May.
  2. Deliver the finished product till the end of June.
  3. Fix the bugs if they exist in early July.

I will be free for the whole period.

Bio

I am a student currently pursuing double majors in computer science and design from IIITD, and I am in 3rd year of my degree right now. I have been using blender for 8 years now (Link to my portfolio: https://www.behance.net/shivankar199 ). I have been developing software and different technologies since the first semester of my bachelor’s. One of my achievements is winning the second place worldwide for the Zoohackathon 2018. I am recognized by the US government and WWF ( https://bit.ly/32o2OwD ).

6 Likes
2

Could you please elaborate on what this system is?

  • Is this something you’d implement as part of the GSoC, or is it an external software that is going to be used? If external, open source? license?
  • What languages do you intend to support?
3

A phoneme based system for shape keys is usually a bad idea. People can deliver the same sentence whispering, shouting, happily or angrily.

Most facial solvers used in production are FACS based, trained on FACS and Harvard line captures.

2 Likes
5

It is a component as part of the GSOC, I would be using python mainly to develop it.

2 Likes
6

That’s why I wrote that the animator would have the ability to manipulate the animation sequence on the basis of emotion in it.

1 Like
7

Just my 2 cents. Phoneme based approaches were always too limiting, and there are too many variables when adding expressions on top. Might be used in some fields, not sure. I mainly worked with facial solvers in VFX (ILM-FEZ, Weta-PCA), phoneme lipsync was investigated and very quickly abandoned.

4 Likes
8

Let me research about FACS approach to this project if it works and is better in my project will reconsider shifting to FACS

2 Likes
9

Yes I can achieve that with FACS as well. Thats good news I guess…

1 Like
10

Yes, I think it’d be a more manageable approach since it could cover the expressions in the form of combination shapes.

Are there plans for handling combination corrective shapes and inbetweens? Out of the box Blender is pretty bare bones when it comes to shape authoring, and shapes being linear deltas it’s essential for a shape key based system to avoid double transforms.

Shape keys as a modifier or live shape keys would be pretty cool, but that’s another topic. :slight_smile:

11

did not look that much into this topic will further research and come back to you

12

Here’s a paper that sums up correctives and intermediate shapes pretty well -

1 Like
13

Thanks for the help, will surely look into it now.

14

My 2 cents:

I would never use python for developing any intensive data solver. You should plan to go with full C/Cpp implementation.

3 Likes
15

Ok I am fluent with CPP and C also

1 Like