iPhone Feature Will Improve Your Relationships in 2020

iPhone Feature Will Improve Your Relationships in 2020

iPhone Feature Will Improve Your Relationships in 2020

we’re going tocover real-time facial capture with iPhone using ournew app, Live Link Face. To do that, I’m going to enlistthe help of a familiar face, someone you might recognizefrom the UE5 reveal. She’s going to act asmy avatar, and we’re going to do thepresentation together. http://apple

We’ll primarily be takinga tour of the iOS app, leaving other deepertopics, like the animation and rigging setup, as well astimecode and hardcore stage configuration for a future talk. We’ll touch on them a bit,but these are definitely areas we want to coverlater as folks get further into the facialcapture pipeline. All right, let’s get started. I’ll go ahead andlaunch Live Link Face. https://worldgraphics20.com/2020/02/22/7-apple-event-2020-iphone-giant-reveals-apple-watch-6/

And you can see that she comesalive right away with my face driving hers through the app. Check it out. Nailed it. This may go without saying,but I’m going to say it anyway. One thing to call outis that the app uses the iPhone front-facing camera. So, my video is mirrored, whichin turn means that our head movements will look reversedonscreen since she’s actually looking at it. iPhone Feature Will Improve Your Relationships in 2020.

If I turn my head tothe right, she’s also turning her head to her right. And I think you get the idea. Next, I’ll put my marketing haton and make sure to be clear. The app is out now. And hopefully, everyone’salready downloaded it. If you haven’t, grab it fromthe Apple App Store ASAP. It’s free.

The only real requirementis that your iPhone has a True Depth camera, whichis needed for the ARKit face tracking. The basic rule of thumb isthat if your iPhone doesn’t have a Home button,you should be all good. Our big goal for the app wasto take the Face AR Sample that tons of people havealready had good success with, including us at Epic, andproductize the iOS portion so teams don’t have to buildand deploy the app themselves. iPhone Feature Will Improve Your Relationships in 2020.

This isn’t necessarilythat straightforward to do. And we found that alot of our clients don’t have thatexpertise in-house. You know, whereverpossible, we want to empower peopleto focus on creating the characters, performances,and content in general. And that’s what ledus down this path. The Face AR Sample is stillvery much with us, though.

And our old friend,the kite boy, is still a great example assetto start with even though he’s sitting this presentation out. Let’s get into the app. But first, a bitof housekeeping. In order to use the featureswe’re showing today, there are few plugins to enable. First up is Live Link.it might be obvious. Live Link is in thename of the app itself. But the facial capturestream is coming in over the Live Link protocol. So you’re, of course, goingto need that plugin on. iPhone Feature Will Improve Your Relationships in 2020.

Next up is the LiveLink Curve Debug UI. So technically this isn’trequired for the system to work. But this plugin issuper useful, especially if you’re doing the animationand character setup. It really lets you diagnosethe data coming in, figure out how it’s beingapplied to your character, look at the blendshapes,that sort of thing. So it’s super useful, and wehighly recommend having it on.

ARKit and ARKit FaceSupport– again, pretty self-explanatory sincewe are using the ARKit face tracking. But you’re going toneed those on for sure. Take Recorder iswhat we’re going to do to record the LiveLink stream in Unreal Engine and get it into Sequencer orwork with it as animation. And last but not least isthe Timed Data Monitor. iPhone Feature Will Improve Your Relationships in 2020.

iPhone Feature Will Improve Your Relationships in 2020

So, this is a new plugin youmay not be familiar with. It’s in 4.25. It’s super cool andit’s kind of something I was alluding to earlier, wherethis is the timecode workflow is its own demo. But in this case, we’re goingto use this new plugin just to visualize the timecodecoming in from the iPhone into on Unreal Engine. All right, let’s jumpinto the main screen.

So, we have kind ofa social-style UX, very much inspired by otherapps that use face tracking. We felt like this wouldmake it both familiar and fun to people as they werestarting out even though we don’t have any filters. So you can do them in Unreal. In terms of the main layout,the main thing to notice first is the big green”LIVE” at the top. iPhone Feature Will Improve Your Relationships in 2020.

So, green means you’re onair or streaming data, right? So my performance is beingsent over to her in the Engine. If we tap this, we can pause thestream, so now, no matter what I do, it’s not going to carryover into Unreal Engine. This can be a handything to do if you need to go offline for whateverreason, take a break, whatever. It’s just a tap away.

Let’s go back to live forpresentation purposes. Next up in the top right– it’skind of some related features, but a little bit moreoriented around battery life. So right now, we haveboth video on, right? You can see me. We have the face tracking on. It’s being sent over to her. If I tap this once, we’regoing to toggle the video off but leave the tracking on. iPhone Feature Will Improve Your Relationships in 2020.

iPhone Feature Will Improve Your Relationships in 2020

iPhone Feature Will Improve Your Relationships in 2020

So the video tends to use upa little bit more battery. It’s something you may wantto turn off during a shoot even though you keepthe capture going. In addition, we foundthat some actors maybe don’t like to see their facewhen they’re performing, right? It can be distracting.

So if that’s the case, youcan toggle the video off. And then they canjust focus easier on their lines, et cetera. The last option here is boththe video off and face tracking off, right? So this is superbattery saver mode. Might be something you wantto toggle in between takes, in between setups, or ifyou’re taking a break, but keeping the actors incostume or keeping them all rigged up– that’s an option thereto help prolong the life of the phone for the shoot. Let’s turn everything backon for our presentation. iPhone Feature Will Improve Your Relationships in 2020.

Next thing we’llmention is timecode. So no matter what,you’re always going to see a readout of thecurrent timecode right there underneath the LIVE button. We’re going to getinto this in more depth shortly as we diveinto the settings. But at minimum,you can always see you’re gettingtimecode, what it is, and a little icon ofwhere it’s coming from. Down in the bottomright is face detection. So this is green when theapp is detecting a face.

Obviously, it’sdetecting mine right now. But if it loses– if it losesthe face for whatever reason, then that’s going to go gray. We can demonstrate this by beingsafe, kind of masking up here. So if I put my maskon, the app no longer recognizes my face or a face. And now we’re offline. iPhone Feature Will Improve Your Relationships in 2020.

So even though we aretechnically still streaming, the app isn’t finding aface to actually stream. So if I take this off,we’ll be back again– picks up very quickly. But just note, maybe notsuper quarantine friendly. Another thing to mentionis down at the bottom are slate and take,bottom middle. So we’re going to comeback to this later when we do some recording.

But I just wantedto call out quickly you can tap thisand modify the slate and take by handdirectly in the app. Obviously, in general,it’s the best practice to keep this alignedand matched up to what you have in Take Recorder. But there is the option hereto set it yourself in the app directly. Also down here is theLive Link Subject Name. iPhone Feature Will Improve Your Relationships in 2020.

And that’s kind of a good segueway into the full settings menu, which is upat the top left. So let’s jump into there next. So, we have a lot of settings. We’re trying to keep this asuser friendly as possible. But we also wantto make sure we’re offering a robustset of features that help people ranged fromentry-level folks who may not touch these very much toprofessional stages that are going to really getinto the nitty-gritty.

First thing we’ll hit is LiveLink and the Subject Name, which you just saw displayed atthe bottom of the main screen. So by default, the subjectname is the name of the device or the name of the phone. You can see my phone isPandaaaaaaaa with eight As. You can override it if you want. And then now we’ll– let’squickly jump out and just show what this lookslike in the Engine. iPhone Feature Will Improve Your Relationships in 2020.

So I’ll go out offull screen, pop over to the Live Link panel. So we have a source, whichis the ARKit face tracking. Down here is our subjectname, which is Pandaaaaaaaa, matching exactly what we have uphere in the app for the subject name. And that’s how youmake that link. Pop back into the full screen. Next up is network targets.

So our main workflowidea for the app is to multicast thedata stream of the faces to all machines in apotential multi-user session. And the goal here isto minimize latency. We’ve really designedthe app to excel in a sort of collaborativevirtual production with multiple machines. iPhone Feature Will Improve Your Relationships in 2020.

Maybe you have onethat’s recording, one that’s doingVCam, one that’s focused on stage operation, andmaybe you’re scouting, right? So each of those machines isgetting the facial tracking data as fast as possible. And then we synchronizewith timecode. The network targets are basedon an IP/port combination.

iPhone Feature Will Improve Your Relationships in 2020

iPhone Feature Will Improve Your Relationships in 2020

So in this case, I’m asingle-user, single-machine setup just withmy home box here. You can find yourIP really easily. Just pop open a commandprompt, hit ipconfig, right? So I’m transmitting over Wi-Fi. I’m going to look atmy wireless adapter. And my IP addresshere is 192.168.1.17 on the home network. iPhone Feature Will Improve Your Relationships in 2020.

And that’s what I’veentered in as the target. The default port is 1.1.1.1. You probably onlyneed to change it if your IT team tells you to. And if you want to enter intoa multi-user session– you have multiple machines– you’rejust going to add a target, punch in anotherIP address here. Cancel out of that.

Close the thing here. Right, so next up isthe Live Link protocol. In Unreal 4.25, we madesome improvements here and added support forfractional frames which improved accuracy andhelps avoid the possibility of duplicate frames. But this is alsosomething that you need to be aware of forbackwards compatibility purposes. iPhone Feature Will Improve Your Relationships in 2020.

You can use the app with olderversions of Unreal, right? So right now, I’m using 4.25. But if I wanted to use anearlier project, like maybe a 4.24 project, then I can goin here and change the protocol. The main thing to point outhere is that it’s very specific. So you need an exactmatch between the versions for the protocol. Otherwise, thestream won’t work.

So because I’m ina 4.25 project, if I change it to 4.24 hereand let that setting take hold, I’m losing the stream. Everything looks likeit’s set up correctly, but I’m not getting anyfacial tracking data. So that’s somethingto keep an eye out on. If you are using differentversions of Unreal, this is somethingthat you’re going to need to make surehas an exact match. iPhone Feature Will Improve Your Relationships in 2020.

iPhone Feature Will Improve Your Relationships in 2020

So let’s switch ourselves backto 4.25, get the stream back– here we go– and then close outour Live Link settings talk with the last item,which is Live at Startup. This one’s prettystraightforward. You notice that whenwe launched the app, we started streamingright away, and that’s based on this toggle.

If you don’t wantthat to happen– you want it to be, like, atwo-step process where you open the app and thenopt in to streaming, then just toggle that guy off. Next setting to gothrough is timecode. So this is a big featurefrom the pro standpoint. And we have three differentoptions for timecode. They kind of build incomplexity in specification. iPhone Feature Will Improve Your Relationships in 2020.

So first up is theSystem Timer. So this comes from theclock of the phone itself. So it may look like anon-standard, weird timecode. In this case, there’slike a 94 in it. That’s not something thatyou usually see on a stage. And this is becausethe system timer is based on how longthe phone has been on. So I haven’t rebooted myphone for a long time.

That’s why it says 94 hours. The other two optionsare a little bit more of that kind of classic,expected timecode format. And those NTP and Tentacle Sync. So we’ll go throughNTP first, right? So NTP lets you synchronizetimecode with a time server. The default we provide is theApple one, the standard one, the same thing thatyour phone uses to figure out whattime it is based on what Apple thinks it is. We’ve seen some productionsrely on using their own NTP server on stage. iPhone Feature Will Improve Your Relationships in 2020.

And they use this tosynchronize all the devices. So this can be a really easyway to get time-of-day timecode. So note that right nowin LA, it’s about 8:20. And that’s what you see here. Last up is Tentacle Sync. So this is definitelythe most pro option. And this enables the app tosynchronize with a master clock hardware device on stage. And the way we do thatis with this device called a Tentacle Sync thatwe’ve done integration with.

So this is what aTentacle Sync looks like. It’s a little lightweightBluetooth guy. And it connectsto a master clock. So I’ve got one here,using the UltraSync. You can kind of seethis thing here, right? So, I know it’s mirrored,but the timecode on this guy should match whatyou see there, right? 4:21:27. iPhone Feature Will Improve Your Relationships in 2020.

And the way it works isthat the master clock drives the Tentacle Sync. And then the TentacleSync, through Bluetooth, tells the iPhone whatthe timecode should be. So, the Tentacledevice is really cool. We actually use thison our own shoots. And so one thing tonote is that if you have a shoot with multipleperformers, multiple iPhones, you should be able to use thesame Tentacle Sync to drive the timecode on all of them.

iPhone Feature Will Improve Your Relationships in 2020

iPhone Feature Will Improve Your Relationships in 2020

You don’t need a one-to-onerelationship between Tentacle Sync and phones. So that’s pretty cool. Last thing I want toshow here on timecode is– popping backout into the editor. And I mentioned it before,but the Timed Data Monitor here– so, this is the newplugin we were talking about. And this is kind of just toshow that the timecode is indeed coming through, and thisis what we’re getting. iPhone Feature Will Improve Your Relationships in 2020.

So nothing up our sleeves–timecode coming through. What you see on the phone iswhat you get in Unreal Engine. So I’m going to actuallyswitch this back to NTP. I always feel alittle bit weird when it’s not time-of-day timecode. It just looks odd to me. All right, back to the settings. So OSC is up next. We’re actually going to parkthat and come back to it.

OSC is for remotecontrol set of features that let you administerthe app externally. But that’s going tobe our grand finale. So we’ll look outfor that at the end. Next up is StreamHead Rotation. So, this is somethingthat you guys want to turn off when you’rein the mocap view. So if I turn this off here– so now she’s very still. iPhone Feature Will Improve Your Relationships in 2020.

Like, none of the headrotation I had before is coming through. When you’re in a mocap suitor you have a body mocap solution availableto you, you’re going to want to get the headrotation from the actual body mocap. If you have them both on, thenthe data you get from ARKit is going to fight with the datafrom the body mocap solution. iPhone Feature Will Improve Your Relationships in 2020.

And you’re probablygoing to get bad results. So if you’re combining theapp maybe with like a head rig or something likethat, you’re definitely going to want toturn this guy off. But since I’m at my desk, I’mkind of more streamer style. I’m going to leave that on. You know, it kind of givesa little bit more life in this scenario. iPhone Feature Will Improve Your Relationships in 2020.

You can also see how muchI bob my head when I talk, which is kind of embarrassing. But I will leave that in. Next up are somedisplay options. So first up here isthe Preview Mesh. So if you turn thison, you’re going to get a little bitmore direct feedback on what ARKit is seeing, right? So it sees my face. You can see the exactplacement of it.

And it’s more anobvious sign that we have data coming through. It also looks alittle bit creepy, so I didn’t want to do thewhole presentation this way. The Record Button lets youtoggle the record button on and off. This is something thatwe’ll come back to later when we hit recording. But the idea here is thatif you were administering the app externally,then you don’t want it so that the actors orwhoever accidentally bump the record button. iPhone Feature Will Improve Your Relationships in 2020.

So you can just takeit off entirely. Let’s leave it on for now. The BlendshapeData is more for debug. So if I turn this on, I’m goingto see a bunch of data here. And it’s actually a goodway to show this is, like– I don’t know. Hopefully, it’s goingto come through. But down in the bottom rightis the head yaw, head pitch, and head roll.

So if I move my head around, youcan see those numbers moving. But if I go back and Iturn head rotation off– right, so now you can seethose are all zeroed out. No matter what I do,they’re zeroed out. That means the app isnot sending that data over to Unreal Engine. So let’s turn this guy backon, turn blendshape off. iPhone Feature Will Improve Your Relationships in 2020.

Yeah, so Take Recorder– thisis the kind of format that’s displayed. Like, there’s two options here– you know, Slate and– there’s Slate & Takeand Filename. So the standard hereis Slate & Take. It’s the kind ofmore editorial style. You know, in this case,we’re 49D, take 1. But you can also changeit to the filename. So this is more of thepipeline-centric view. iPhone Feature Will Improve Your Relationships in 2020.

You know, some peoplereally don’t want spaces, so there’s anunderscore for you. But this reflectsthe name of the file that’s going to be laterextracted from the phone. But I prefer the editorial file. And we’ll leave it as that. And then the overlayis that we’ve had it on the entire time. You know, so it’s allthe stuff around here– “LIVE,” the different toggles. iPhone Feature Will Improve Your Relationships in 2020.

But you can alsoset it to fade out. So if you don’twant to see that– so if you wait acouple of seconds, then that overlayis going to go away. Kind of a betterexample of this is when you have video off, right? So this is maybe themost likely scenario that the performeris going to see. So we have the video off. We have the overlay off.

But we still letthem see what time it is and also the battery. You know, oftentimesthe performer is the one that will catch thatthe battery is almost dead, kind of like my phone now. But I’m going to leave thaton because it’s kind of easier or better for the demo. And the last thing to mentionis the reference video. iPhone Feature Will Improve Your Relationships in 2020.

So we do record referencevideo every time you record. And there is an option to adjustthe quality and file size. We are going to show that stuffright now because actually we’re going to record a take. So with that, let’srecord a take. Let’s turn the video back on. All right. So we’re going to recorda take in Take Recorder and from the iPhone manually.

So first thing we’regoing to do here is switch over toTake Recorder. So we’ve already set our slateand take to be the same thing. We always want theconsistency between the two. And in this case,we’re just going to record a LiveLink for us here. So now I’m going to hitRecord in Take Recorder. iPhone Feature Will Improve Your Relationships in 2020.

I’m going to hitRecord on the phone. And this will be our take. OK, this is a demo recordingthat I’ve initiated manually in both Take Recorder andon the iPhone, and yeah. We’ll cut. And now let’s take alook at it in Sequencer. So I didn’t end up recordingaudio in Sequencer. But it is recorded inthe reference video. It’s not really set up torecord audio on my computer, but let’s just play this back. You can see theaction going here. iPhone Feature Will Improve Your Relationships in 2020.

iPhone Feature Will Improve Your Relationships in 2020

Maybe I should haverecorded a shorter take, but my expressions at theend also came through. And now we will pop into thephone to take a look at it over there. So, Live Link Face has thistake browser feature as well. So in the bottom left,you can pop into here. So this is showing all thetakes I’ve already recorded. Not surprisingly,they’re all of me. You can browse them by alltakes, by slate, or by day, right? So you can also do a search. iPhone Feature Will Improve Your Relationships in 2020.

So we know we were 49D. So if I type “D,” Ican jump over to mine. And here’s the take we just did. The video, as I justmentioned, has audio in it. But it also hastimecode embedded. And the goal here is totake the reference video and be able to line it upto the recorded capture. iPhone Feature Will Improve Your Relationships in 2020.

So let’s play itjust as an example. OK, this is a demo recordingthat I’ve initiated manually in both Take Recorder andon the iPhone, and yeah. Another thing tonote here is that you can transfer the takes offof the phone using the iOS Activity Views. So if I hit theblue arrow up here, it’s going to bring up abunch of different options that will zip up the dataand potentially send it over. iPhone Feature Will Improve Your Relationships in 2020.

You know, AirDrop is generallythe best and the fastest if you have a Mac. But we also havesome other options. So you can doGoogle Drive, Box– you know, there’s a bunchof different options that let you upload the dataas well as just transfer it over AirDrop. But AirDrop isgenerally the fastest. Next up, we’re going totake a look at an example take on disk. iPhone Feature Will Improve Your Relationships in 2020.

So we’ll kind of cheat,but just looking at a take that I’ve alreadymoved over, right? So for each take,you’re going to end up with a series of files. Like, there’s twomain ones, really. So first is a video, andthen the second is a CSV. First thing we’ll showis that video, right? So, I actually havethat open already. So this is an example ofthe reference video you get. I’ll play it through now. Here’s an example recording toshow that it has both timecode and audio. iPhone Feature Will Improve Your Relationships in 2020.

So I’mgoing to disable the– let’s mute me. I don’t think we needto hear that again. But it’s also meant to show herethat we are getting timecode in the media that come through. Yeah, the QuickTimeitself uses a JPEG codec, so there’s notemporal compression. It’s frame-accurate. And again, goal here is totake the reference video, let you line it up to therecorded capture for reference. So it’s somethingthat your animators may want to use,or just something to check that things are working the way you expected. The other thing that you getout of the take data is a CSV. thankyou.

Leave a Reply

Your email address will not be published. Required fields are marked *