Interview with Brian Drewes founder of ZERO VFX
Adam Svystun (AS)- Comixify.ai
All right. Let’s start off with a basic question. Tell me more about yourself. Who are you basically?
I have been in the visual effects and technology space since college, which means that I’ve been doing it for 25 years at this point. Big picture my motivation comes down to a general fascination in applying science while creating popular art. More recently, we’ve been focusing on various ways of recreating digital versions of our world. It was about 10 years ago that I started ZERO VFX. It felt like a natural progression for me – you start learning the industry and technologies, then you start being able to look forward and see where everything is headed (or so you think). That’s the transition point of becoming primarily an entrepreneur versus a visual effects producer. Plus, spinning off a tech company which I sold to Google was a helping hand in this direction.
What were the biggest projects you were working on? What can you brag about?
We’ve been doing pretty decent sized movies in the last five years. Things like Equalizer 2, which was a Denzel Washington and Antoine Fuqua film with around 800 or 900 visual effects shots. More recently we oversaw VFX on the Oscar winning “Little Women”, a number of originals for Netflix and X-Men “New Mutants”. Our short-form division is consistently delivering projects for advertisers that spreads across broadcast, digital and AR.
Maybe some of the more casual readers would like to know – after learning all of this about film production, do you see movies in different ways? Is it harder to enjoy it?
Yes, it is. I’ll be honest, only in rare instances will I get sucked into a story on first watch. First time through I evaluate on sequences and how those sequences are structured. And of course how everything looks or how it was achieved. I’ll get joys from seeing really inventive thinking on the film-makers part. On second viewings I’ll be more apt to forget the technical and just watch for story and character.
What was the biggest challenge in your career, in your VFX career?
Well, we’re kind of there. I mean, the COVID-19 certainly is a big challenge.
As the owner of the company, though sales are down substantially, I at least have a bit of agency and can use my experience to look downstream and pivot the business to speak to those new worlds in 9-10 months from now.
I have a lot of feelings for the entire world at this point, but for the more junior in this industry it’s really difficult to grow your skillsets – so much of it is via osmosis from your team-mates – it’s very difficult to achieve in a distributed format. Overall I believe if this persists for much longer, we’ll see a large sea-change take hold in the industry of distributed work becoming normalized, the push to the cloud will be dominant and VFX companies are going to have to re-invent themselves and their value proposition. I just hope that whatever the result, the members of our industry find stability, consistent work and reasonable terms. Film incentives will definitely be an interesting thing to watch as well.
Pivoting more to an AI discussion, do you use any AI-based tools in your current work on a daily basis?
I’m sure that I do but just don’t know it.
Since I’m not on the box, I’m really talking about normal, everyday life softwares and services…And that’s where VFX toolsets will eventually get – you don’t even know how it’s happening. You’ll just know that there’s some sort of output that then you’re able to manipulate. It’ll feel very powerful, but also very normal. I believe the Adobe suite has some ML enhanced features, but for visual effects softwares specifically aimed towards the pro- there are no off the shelf AI-based toolsets.
But do you see at least people, maybe bigger studios, looking around, trying to invest the resources and time into AI development?
Absolutely, large studios definitely have a lot of internal facing tools that utilize ML to help them with the very large projects. They do have that repeated need during the movie, for a specific kind of technology to perform a certain kind of function. One example of that, that I could think of, would be Thanos in Avengers “Infinity Wars”. His animation was pretty interestingly powered by some ML tools to help with targeting. You have this sort of squishy problem with CG characters. If you’re doing motion capture on a person’s face, and then translating that human sized face with all those human sized points to a computer generated CG face, the dimensions of that face are different than the human face, the shape is different, larger, eyes are further apart, the chin is lower, all of the musculature is different, all those targets have moved. So the question is – how do you apply the human movement in the data to something that’s of a different size, different scale and has different ratios?
This is something where generally humans will be doing that targeting and then you get into keyframe animation from there. From my understanding they utilized machine learning to help with that initial targeting so that then the artists would go in and manipulate that output. And then that revised output would then get fed back into their machine learning tool. So their algorithm would continually improve as the artists iterated the output, so the artist output was helping influence the AI going forward so that then as the movie progressed, and the work progressed, the output of that initial retargeting became better and better. That is what was published about that and that’s the kind of thing that you see getting rolled out now. But that’s generally the purview of very large multinational corporations with large engineering staffs and large budgets to throw around for those kinds of things. And they also have this repeated need. You have this one thing that’s really complex that might take many people many days. And you have to do that 10,000 times for a movie. Economically, it makes sense to engineer a solution at the beginning to not have to spend that much human time doing that. But for a smaller company you don’t own that Thanos character for 500 shots where you have to engineer a solution like that. That’s sort of the difference between how the tools are getting made.
Let me try to see if I understand this correctly, you can say that big studio who have big budgets and produce a lot of work, they do develop internal tools for specific tasks. But these tasks are pretty specific, like the tools for Thanos, they might not ever use that tool again, but they still developed it, right?
They will stand on its shoulders for the next time. They just keep refining it for the next version, the next iteration. And I’m sure that it’s written to work within their pipeline. So all the inputs and outputs of that have to be in a way that doesn’t mess up the rest of the pipeline. As far as immigrating or transitioning into a tool that other people or other studios can use, I don’t think that’s part of their roadmap. Their roadmap is to be able to create the things that other people just can’t do at the scale that they can do it. And they don’t really care about externalizing those toolsets in any way because why would they if they don’t have to?
Okay. Thinking more broadly, in what parts of VFX pipeline do you think AI has the biggest potential to bring the most drastic improvements?
I think on things like rotoscoping, things like understanding camera motion, those things can be aided dramatically. There are all sorts of repeated tasks that we do. But not every repeated task is simple. There are always little differences or eccentricities with even a “simple internal” task like rotoscoping. For example – to create splines. Functionally, they look pretty simple, but there is this humanly intuited sort of solution that needs to get figured out. Those are difficult things to achieve. But I do think things like extraction of these things, that are like Thanos, in that people are doing them over and over and over again, and actually represent a lot of human effort. Those will be the first targets – so things like rotoscoping, plate prep, things like painting out, other kinds of paint tasks where you’re repairing the background.
You mentioned one of the things is rotoscoping. How influential would it be if we could rotoscope any object in the scene with 99% accuracy?
Well, it would be pretty amazing. Both from a time, money and resource standpoint, it keeps artists from beginning to be able to work on shots. There is a long prep time that has to go into this. So that is a dependency that other artists can’t do. From a process standpoint, the acceleration of rotoscoping would be amazing. But it’s just such a huge task, every single frame is getting rotoscoped on all these movies. Even when you have a green screen or blue screen, many times you still rotoscoping stuff because when you’re pulling a key, you need an artist who is trained on pulling a key, whereas if all you’re doing is just rotoscoping that person off the background – you can offload that to somebody that is less sophisticated an artist and you know the output is going to be the same. So it would dramatically change the way we shoot things, and the process that we go through in post.
And like speaking more broadly, how do you think that workflow and the pipeline of VFX and post-production will change after we automate many things with AI? How do you think the VFX industry will adapt to this?
If machine learning tools are able to have a predictable output that is of high quality, that also has all the flexibilities that would allow you to take that raw output and adjust it to be usable in visual effects – I can’t even express how different things become. I shouldn’t say it’s a barrier for the industry to grow, but I think what you’ll see is that there will be further democratization of artistry. If these are things we don’t have to worry about anymore then artists can get better at other things. Like if we don’t have to worry about rotoscoping, if you had 10 compositors in the room, and you said – ‘Hey, if you didn’t have to worry about rotoscoping stuff anymore, do you think this is a great thing for you guys?’ I think 10 out of 10 people are raising their hands, I don’t think that they see it as a threat. And I would look at all AI in that way as far as visual effects, because there will always be a level of human interaction and human creativity involved with interpreting, a very obscure or amorphous direction from another human being. We get direction from a director or a visual effects supervisor, and we have to interpret what they’re seeing in their mind’s eye. They have an image in their mind that they’re using words to express. We then are trying to build that same image off those words, so there’s always this level of misunderstanding or clouds or fog in that discussion. That is the human endeavor in visual effects. That is something that’s not going to change. And that’s really where the magic ends up happening a lot of times. And AI allows us a little bit more time potentially to find those happy accidents and to be able to create something cool. Now that we have more iteration cycles we can feel free to be more creative. More generally, AI in visual effects, I think people get a little concerned about AI and visual effects, because they feel like it’s gonna come and take their job, but I don’t think that has ever been shown. That’s always a fear, since the 1920s, that machines were going to come and take our jobs. Our machines have gotten better, but people are still working. So I think that it’s all going to be okay. I think our life will become better.
I see what you mean. It’s like you spend less time rotoscoping and spend more time on your creativity. Well, another question you mentioned twice during this discussion – degree of control. You said that for tasks in visual effects you could never really make a template. Because each task might differ in one way or another. And it’s important to be able to change and adjust the output from the machine learning models. This contradicts a little bit with how ML usually works, at least current models. They’re very strict and rarely do they, as they are now, allow you to modify their output. More generally, the degree of control in machine learning models is usually low. So do you share this concern, that this might be a problem for AI to really help in VFX industry?
I think that there’s a difference between helping and solving. If what you’re looking for is ML to solve a problem, if you think of visual effects process as like a simple equation where A plus B equals C, then I would say you’re misunderstanding the process. It doesn’t work that way. It’s very rare to deliver and get approved shots as version one. There’s always a round of notes and a round of changes. So where AI can’t solve is to say, you do it once and you’re done, that’s just never going to happen. I think that it’s a misinterpretation of the way it really works. But what it can do, like in the Thanos example, is to say – ‘Hey, here’s an output that fits back into the pipeline that you already have’. It just allowed you to have a more consistent output to get to that point. Not that output is a solve, It’s a help to get to a solve. So if you can say solve is 100%. Right now, it takes 100 man days to get to that hundred percent. If what we’re saying instead is that actually AI gets it’s still on there’s you still have to get to 100%. But the machine learning process gets you that first 50% of the way there with like, with like two man hours. Then you still have 98 man hours left to spend. So at the end of the day, you’re still gonna have to have that human intervention. It’s just a matter of how consistent you are, how much easier is it for that person to find the solution. Because I don’t believe AI can be taught finding that solution.
So you see AI being more of an assistant, then a solver. So you want it to be able to decrease the amount of work you do, but not really do that work for you?
Like for a rotoscoping tool, if what we’re getting is rotoscoping for rough use, like for previews or for early versions of shots, if that can be done automatically – phenomenal. Then we can take that automatic output, and then make it more final using humans that already know how to make things final. Then you can start nipping away at where can that ML get improved. Then you start creating tools that solve certain recurring issues of the initial output. So that’ll be my sense, you’ll end up with an ML that evolves as well. And that output will get better and better and closer and closer to the human output. But to start with just getting 60% of the way there and letting a human get that last 40% that’s already a huge win. I think there’s many other tasks like that, that sort of have a similar dynamic.
I understand, and for the last question – if you could choose for anything to be solved in VFX pipeline what would you choose? What task would you automate?
Rotoscoping is really a huge one. And I think that’s because it’s probably the most achievable. There would also be the idea of lighting objects to fit within a scenes lighting. If there’s a way where you could analyze the quality of light in a room and be able to have ML help you to light a 3d object in that room – that would be huge. If you’re replacing a vase in that room, right now you are dependent upon either HDR’s from set or a really talented artist to understand how to get that light to work on that CG objects so that it fits nicely in the scene. Lighting is one of the main fundamental pieces of visual effects that is just very laborious. All the things associated with digital humans, those become the most successful sort of thing as well things like the animation of the human being, all of the sort of micro movements that their muscles have, all their subdermal musculature are all very, very hard to do. And humans are really bad at making randomness – that’s kind of the problem. I think finding that randomness is something that ML can help a lot with, strangely enough.
Yeah, I understand. And you know, we have to dream before we accomplish anything.
Yes, and I think digital humans definitely are a big thing for sure. I see really adventurous things like maybe somebody’s driving down the street and there’s no leaves on the trees – tell the AI to put leaves on the trees. At low resolution and without really getting in and looking at it pixel by pixel – It looks really good. But the problem of taking it from that looks good at 15 frames a second on my computer compressed verse, let’s look at it at 4k frame by frame and really analyze those edge details and those errors along the way, you realize that there’s a big gulf to get it to be that hundred percent, that’s sort of the challenge. I think that the challenge is people really want to solve the big problems right now, but I think that it’s going to be a much longer process of finding little things along the way to help with. But I feel like rotoscoping is certainly the best place to start.
Well, I don’t have any other questions. This was great, I think we covered a lot of bases for sure. A lot of interesting information.
More about Brian & Adam
BRIAN DREWES Visual effects producer and media/technology entrepreneur. CEO and Co-founder of ZERO VFX. Also co-founded ZYNC, which was acquired by Google in 2014.
Adam Svystun Comixify.ai Front–end, co-founder
Previous professional experience in Machine Learning and Front-end development in an AI consulting company. Academically, he has done research in Computer Vision Lab @ Warsaw University of Technology. He also has international academic experience with several exchange programmes done at Nanyang Technological University and Politecnico di Milano. Last but not least, Adam is the biggest fan of comics of us all!
LinkedIn profile