Create Sets from iPhone Panoramas and Virtual Set Editor
October 05, 2015 by Jeff Foster
While you can edit existing virtual sets with the NewTek Virtual Set Editor (VSE) or create a new one from 3D rendered sets you can purchase and download online, did you know that you can also create your own panoramic “sets” from your iPhone5 or 6? Wouldn’t it be great to use your company’s lobby or the quad at your school’s campus to insert your green screen keyed talent into and make it actually look GOOD?
By using the Panorama function on your iPhone’s built-in camera app, you can create up to about 240º of panoramic image area inside or out, that you can then use to create masks and map your keyed subjects into places they could actually be standing or sitting.
Planning the Shot
It’s helpful to understand how the VSE works, first. Knowing what restrictions you may have before you start shooting and building a set will help you make the right choices in where and how you shoot your panoramas. Since you only get to create one mask layer, make sure you don’t have any planned overlapping talent positions, such as a desk behind a wall that you might have someone walk from behind or sitting at the desk at the same time.
Think about the angle that you’d want your production “camera” to be shooting from. If you’re not creating a TV studio, but rather an office lobby or interior, then think about where the talent may be standing and how they’ll be looking at the camera. Don’t put them up too high above your level or too far down below.
Using the iPhone to shoot a panorama, you’ll start from the far left side of the “scene” you’re shooting and steadily pan to the right side and stop when you’ve reached the end. The camera app will automatically stitch together the entire panorama image for you. It’s best to always shoot at least 3 different shots to chose from - and also best to not have anything moving in the shot, such as cars or people.
You can download this image and use it directly in VSE if you wish, or you can use a photo editing software on your desktop, such as Adobe Photoshop. I use Photoshop expressly because of the image and brush controls. In this panorama below, the light from the morning sun was coming in from the left side of the scene and creating a warm glow on all the wood surfaces, while the cooler ambient light from the windows on the right side of the lobby was much cooler and threw off the entire image. I needed to create a mask layer and do color and exposure correction to bring both sides of this 200º panorama image together and look like one wide shot.
Creating a Mask
I used another shot from across the same lobby that had some dimension, different levels and some elements we could mask to integrate our subject into more. This was a narrower panorama at about 140º but it will still work great in VSE to create separate “zones” for our subjects.
I start by opening up the image in Photoshop and then zooming into an area I want to isolate, such as the reception desk and the brass railing leading down the stairs.
- NOTE: It’s very important to not alter the size, resolution or cropping of the original image or you will be off with your mask alignment. By creating a new layer above the photo, I can then paint with a 100% opaque white brush all the areas I want to mask out and place my subjects behind. This process takes a bit of skill and practice to perfect, especially in areas where there might be softer edges or transparency, like the detailed beveling in the glass.
After you’ve satisfactorily painted all the white mask areas, then create a new layer beneath the painted mask layer and fill it with 100% black. Then save the file as a JPG and name it differently from your original image but something you can find easily in your directory inside VSE. Your completed mask should look something like this, next to your original image file.
Creating Your Virtual Set in VSE
When you first open VSE it will prompt you to either open an existing virtual set or create a new one. Select the Panorama option and locate your original panorama image file on your hard drive. It will most likely open up with a drastically zoomed in view by default and your first Input shown on screen.
Before making any other adjustments, alter the Camera FOV so that the panorama image is scaled appropriately when you click and drag the cursor around in the image window to change the view. In this example, I reduced it to about 150º as the pano I shot wasn’t that wide. By default, the vertical FOV is set to Auto and I’ve found that as long as you keep the Projection Method set to Spherical, then it’s properly scaled and doesn’t need further manual adjustment.
Now you can select the Mask file from your hard drive which will automatically scale and match your panorama file properly. You may need to select the Invert checkbox next to the file selection to make the white areas opaque in the mask.
Select Input 1 and twirl-down to reveal the settings. Select the Behind Mask checkbox if you want this element to appear behind the mask that you created. Otherwise, your input video will show up in front of the mask it’s placed over.
Adjust the scale and rotation of your input window to properly fit the placement - taking care that if your keyed subject is going to be in motion that they will be moving in the right direction/angle to the placement window you’re creating - including behind masked objects.
Continue with other inputs you wish to add for your second video channel if you’re using a prerecorded element or a second live camera - perhaps on a second subject or closer view of your main subject. Take into account the size your subject will be shot in frame so you won’t have them “floating” in the air or out of scale with your set placements.
Also be sure to note if they will be positioned behind or in front of the masked objects you will put them into, then save your set and Export it to your current TriCaster so it can be accessed immediately.
Using Your Virtual Set in the TriCaster
Once you’ve started a new LiveSet production, open a new background in M/E1 and select the User LiveSets. Your created and saved virtual sets will appear and select the one you’ve just created.
Select a live camera that you have set up with green screen and key out/mask the background out. In this example, I used a couple default sample subject that NewTek provides for training, just to get a feel for the production workflow and placement. I’ve selected two different sources for my Input 1 (A) and Input 2 (B) locations. I can now zoom between presets to reveal one or the other or both, and zoom in to get a closer shot of either subject.
With your own virtual sets and talent, you can create customized productions that realistically reflect the institution you are representing or the location you’re talking about. And since most people have their iPhones with them all the time, it’s easy to be not only a “location scout” but also the set designer/creator, wherever you go!
More Articles by Jeff Foster:
Speak with a Solutions Expert
Where to Buy
NewTek has a network of resellers located around the world. All of them are experts in our product and in the video production and streaming industry.
Use this tool to find a NewTek reseller near you.
Speak with a Solutions Expert
Get expert advice on a live production workflow around NewTek systems and NDI or let us connect you with one of our local partners, our Solutions Experts are here to help.