To listen to this article, click Play below.
At Four Winds Creative, one of our company values is Fun, so we enjoy the unusual projects the most. It was the third quarter of 2024. A Fortune 100 customer of ours came to us with a project that had firm dates in December, and a budget of fifty thousand dollars. “Sure, we’d love to help,” so we set up a discovery meeting to learn more about what the clients had in mind. But, before we’d have that call, we learned that the full-time internal video and streaming group refused to support the project in 2024, although they fully supported the group and this exact meeting in 2023. Hummm that didn’t sound like Fun.
Concerning?
Maybe.
But we were going to learn the “why” of it, in our initial discovery call and then, after we had the details, we’d decide if this “impossible mission” was something we wanted to support.
The virtual Microsoft Teams meeting started like so many others and we did a round of introductions. The clients were nice folks and looking for help to bring their group together to help team build and discuss some restructuring and new company initiatives. All the normal topics you’d expect from a big company like theirs.
Todd Hallinan, one of the founders of Four Winds Creative, was to be the Executive Producer if FWC decided to support the client, so he led most of the questions and discussion topics throughout the discovery call. We went through the usual what, why, how, when, questions and we dug into the prickly details. We learned that the show would consist of in room audiences of varying size in Arizona, California, Oregon, Munich, Beijing, and Kuala Lumpur at a minimum.
Additionally, we’d have audience members watching the stream from home and some presenters and panel discussion members would need to join from their home offices or company facilities in Washington DC and Bangalore, India. Some sessions would have slides and visuals to support the discussion topics, some would not, and yes there would be videos that would need to play at different times. OK, no problem we’ve done all this and more before. The presenters would share their material in front of the live audience in their room and this would also be streamed to the other sites and to the “at home” audience.
Interestingly, the clients wanted to do a single show, that would allow their entire organization, in all its various geographies, to be connected through a single stream over the course of an entire day. The “ask” meant the show would be a total of 26.5 hours long, span multiple continents and time zones and the clients wanted to be able to present from anywhere. That is, have mixed panels of speakers that could be “virtually joined” and viewed by anyone in the group anywhere in the world, at home or in the office, at any time within that “around the clock” show schedule. “Oh, and one last thing, in 2023 the presenters had to leave their individual auditoriums and meeting rooms to participate in the live discussion because of audio feedback issues. This year, can we make it, so they don’t need to isolate from the rest of the in-room audience when they present, please?”
This led to a discussion about webcasting and latency and all the normal “streaming is not like satellite broadcast that you’d see on the news” talking points. And then, the client asked a very good question: “Can it be interactive like a news broadcast?” That brought us to another company value of Four Winds Creative, Customer Satisfaction. We strongly wanted to please the customer, and that question is so innocent: “Can it be interactive like a news broadcast?”
Spoiler, the answer to that question turned out to be “yes, yes it can be” but we weren’t so sure in that very moment, so we asked for a little time so we could investigate some possible technical solutions. As I mentioned before, this group of clients were very nice, and they were all too happy to give us a little time in order to come up with a solution to their “Can it be?” question.
Up to that point our team had a long history of doing all sorts of different kinds of events and streams, but we’d never run into this request in such a way before, so AJ VonWolfe, our Live Events manager dove in to start a little research on how we could answer with that “yes, yes it can be” answer.
AJ was a longtime proponent of ECamm Live, a visual mixer software created by fellow Massachusetts natives, Ken and Glen Aspeslagh, and AJ was very versed in the capabilities of Zoom. Now, at this time, the capabilities of Zoom had just been integrated into the ECamm Live software. Since AJ was already following these tools so closely, he suspected that the solution to our client’s problem was already in hand, we just had to “think different” to realize it. Then we’d need to convince their Microsoft centric IT team that we would need to use Zoom not MS Teams. No problem, right?
This led to some testing across the Four Winds Creative team from multiple sites to verify whether we could technically eliminate the latency and audio issues for in room audiences where live presenters would be. Through our testing, we became confident that by utilizing Zoom’s very robust noise cancelation and by configuring each room around the world to provide an audio mix minus solution, we’d easily solve the previous year’s audio challenges. Additionally, by instructing our local A/V teams to work in, what they each told us was, an unorthodox way, we were able to create an experience through ECamm Live, that was renamed a “visual mix minus” by those same A/V teams.
AJ had devised a way to have each presentation room around the world, create an in-room visual sub mix with ECamm Live, that had all the remote presenters, available as ISO sources within Zoom, displayed as projected panel members, with the live, in-room panel member seated on stage while interacting with their virtual colleagues, with no latency, and no feedback, for these multi-geo panel sessions. This helped meet the client’s team building and collaboration goals because they could all stay physically together in the same room at a given location and join virtually to each of the other geographically disperse locations that were participating in a given session.
Functionally, we outfitted our remote A/V Teams in each room, to have their own instance of ECamm Live. None of those teams had used ECamm Live so some training and familiarity time would need to be baked into the timeline. With ECamm Live’s new Zoom integration, the A/V teams were each able to create a custom mix for that local room audience. In addition, our “master control,” using the same Zoom Webinar sources would pull and mix a different show for what became known as the World Stream.
We knew this was a lot to ask from our remote crews, so we also needed to ensure that we’d have ISO recordings from each camera in each room in case we had to fix the show in Post for the archives. That meant that the equipment list for each room, as well as the necessary personal was going to grow. We knew that the original budget of fifty-thousand dollars would be a stretch. How badly did the client want this functionality and would that value be worth, what, was becoming, a significant increase to the operational budget? We’d need to book another meeting with the clients to propose our technical solution, as well as convince them that Zoom offered better options over their IT Department’s “blessed” Microsoft Teams tool, not to mention the increased budget that would be needed to make this a reality for them.
Todd, reached back out to the clients and we confirmed a date to meet to discuss the solution and what it would cost to make it a reality. The meeting went well, and we learned a lot more about what technology and equipment would be available in each room at some of the partner sites. The hope was to keep cost down so it was our plan to leverage any existing in room equipment, was critical. In the end, we needed to furnish 90% of the needed equipment since none of the event space owners were thrilled with the idea of us unwiring their custom meeting rooms to fit our single use hybrid show.
Through that client meeting, we determined that it would be best if Four Winds Creative had our producer in person at each location to handle the client needs, drive a “run of show” and to field any “day of event” issue that came up. In addition, we’d need to source the right A/V teams for each geography locally, since there would not be travel budget to put a technical lead from Four Winds Creative on site everywhere.
So, the clients dug into the schedule and tried to finalize who would be saying what, from where, and when; while we started documenting the technical setups that would be needed in each room, securing and re-training some tried and true, as well as new to Four Winds Creative A/V teams from around the world. This brought some fun language challenges, as well as acknowledging and blowing through several established norms that each of these partner/vendors had relied on event after event after event. Essentially, we needed them to risk the success of the event on our say so, when they would be the ones standing in the room facing the angry customer if something went sideways. Fun! 😊
One of Four Winds Creative’s company values is Quality. By using Zoom Webinar, we ensured the ability to support 1080HD sources and that we’d be able to use the “on stage” and “backstage” area for our remote presenters to join early, be briefed and have a moment to do one last quality check of their camera framing and audio before they’d “go live” on stage in the webinar. Once they go “on stage” they become a source for, not only our Master Control Technical Director (AJ in this case), but also our six other Technical Directors scattered around the world.
We wanted to ensure that the remote presenters would look and sound great as well as be comfortable with the technology. Not to mention, get them all to install Zoom well before the event. To help drive this, we knew that we would want to do tech tests with each of them to familiarize them with the Zoom Webinar platform as well as fine tune the audio and the visual look and feel of the locations they expected to present from. That meant finesse the lighting, tune up the background and ensure they had good camera shots and clean audio.
To deliver additional consistency, we built custom branded graphics and distributed them to our various AV teams as ECamm Live ProfileScene files. This ensured that each of the Picture In Picture (PIPs) were identical at each of the sites and we maintained a consistent look and feel for the in room screens as well as the World Stream. All these factors would help address our desire to deliver a quality product for our customer.
We knew that with a show that was nearly 27 hours long, we’d be pushing the limits of both Zoom Webinar and our ability to record the session. Would ECamm Live support a stream and show recording 27 hours long?
To hedge our bets, AJ reached out to the ECamm Live team as well as the Zoom Webinar team to verify the limits since the documentation lists 24 hours as the maximum event time available. We learned that Zoom Webinar could go as long as 30 hours. For ECamm Live’s event recording, there was no theoretical limit except drive space, but it would be one single recording, so we really hoped that nothing went wrong with the host machine, network, power or hard drive during the 26.5-hour show.
As I mentioned, we wanted to ensure we had ISO recordings of all the sources, from all the rooms. Each room averaged 3 cameras plus slides (see diagram 1a), and we had 4 remote presenters that would not be in room so that was a minimum of 22 ISO Recordings and one program recording.
Our contract said we’d edit and deliver all the individual “on demand” sessions within a week so the individual recordings, from all the geographies would need to be available for editing immediately after the live event wrapped. In the end, we generated just under a TB of h.264 files across all the different client sites and delivered all 15 sessions from all the presenters on time and on budget.
Speaking of budget, we were approved during that second meeting to deliver all the functionality, send our producers to each geography and to schedule and execute all the tech tests with the various presenters from across the globe. To help keep costs down, we used two technical teams from the customer’s internal group who had previously refused to support the event. We trained them to use our technical setup and provided ECamm Live licenses and orientation. The remaining sites were all 3rd party AV teams who we also trained within that same budget that had increased approximately 35%. In the end, we had successfully won over each of the AV teams to this new workflow.
As we locked in our producers for each location, we had them begin meetings with the local clients in each of those geographies. Where feasible we completed site surveys and walked through all the local details and our producers created a “Run of Show” for their individual room and geography. Those local documents helped inform the master schedule that would govern the almost 27-hour marathon event that we’d kick off. In just a few weeks.
The event inched closer, and closer and we finally reached the week where we sent our producers out to each location around the world. Meanwhile, at home in our Campbell, CA office we set up the “master control” space, gathered snacks and coffee and prepped for our sleepless show. As our individual AV teams got set up at each of their locations, we began a test stream about thirty hours before we’d have to start the real show stream.
We used this test stream to run through all the various combinations that each of the different geographies would be required to complete, over the course of the event. We did our final remote presenter tests from the individual client office locations. Everything we could check, we checked.
We found that everything for acquisition was working as designed so far. As each of the sites came online, we fine-tuned the shots and setups as we helped our local teams dial in their sources and prebuilds using the ECamm Live scene files. The teams in Kuala Lumpur and Beijing would begin the show and so their presenters would be up first. Through the magic of Zoom and ECamm Live we had great looking 1080 streams with perfect audio from each and every location we’d tested.
The final delivery stream, the “World Stream” would be provided to the general audience through Microsoft Team’s Town Hall, that meant that our RTMP stream from Master Control would feed that platform for the final “viewers.” We started our test Town Hall and fed our setup and testing signal into that platform. At this point, most of Asia was setup and Europe was just waking up. We had time for a short nap before the show started. We were going to need it.
We got back to Master Control at about 5AM Local time and found that our stream was solid and we had no lost packets or problems. The test streams had been operational for over a day. We had Europe up and ready to go and it was time to move from our test Zoom Meeting and test Town Hall over to the actual show links.
All our remote locations dropped their Zoom connections and joined the Program Zoom for the “real” show. At Master Control,we did the same and took about 30 minutes to reset all the locations, all the scenes and all the audio. Everyone was up and working except our Arizona site who couldn’t get into the venue yet to get setup, because that venue was in use by a different client. We’d be helping setup and test their sources while the show was live. No pressure.
We ran the countdown and began the show “live” from Kuala Lumpur with guests from Beijing. Everything worked like a charm. The in-room audiences in each location were engaged and able to ask questions live from their room and the multi-geo panels were flawless. We had our remote single presenters join from India and deliver their presentations with live feedback and collaboration from both Beijing and Kuala Lumpur.
As we readied ourselves for the live hand off between Asia and Europe, we had our Washington DC participant join early to test their setup one last time. They’d leave it setup and join for their presentation in about 7 hours.
Munich took over the bulk of the content and the day wore on. Lots of slides, videos and team interaction across the world as we got closer to the Europe to US hand off. Just to add a little drama, we had our Washington DC presenter go offline as a mandatory IT patch was pushed to his system less than 10 minutes before his presentation started. It’d had been about 16 hours of content at that point, so we needed a little adrenaline to wake us all up. Thanks IT, we’re awake now!!
The US teams in Oregon and California were up and online now and our Arizona team had just gotten access to the venue and begun setting up their cameras and ECamm Live system. They had time, we wouldn’t need their shot for another 4 hours. The Panels between Munich and the US worked great and our planning and testing had paid off so far.
At about 24 hours into the show, Beijing was coming back online since they would have an interaction with Oregon and Arizona before taking back the helm and closing out the show. The team in Kuala Lumpur had a series of questions for the Arizona presenters and everything was going well. The World Stream was solid, and in Master Control ECamm said it was still recording. AJ and the team at Master Control had managed a few cat naps but were mostly running on coffee and stress.
It was time for the US to Asia hand off and that meant we were almost done. Great! Then, the Beijing presenter jumped their cue and began to present in room before they were “live” on the World Stream. Yikes! We panicked appropriately but AJ hadn’t taken any of their presentation live so, no one knew accept Master Control and the in room audience in Beijing. To cover the problem, we got the Beijing presenter to start over and then took them live to the World Stream, so luckily, the global audience had no idea that we’d had an issue.
The final presentation went well, and Beijing signed off some 26 and a half hours after they’d signed on for the beginning of this ride. The technology worked, the streams held, and each room had their own unique experience while the World Stream delivered all the content live, for more than a day.
All around the world, our AV teams broke down and loaded out. Meanwhile, at Master Control, we hit stop and ECamm Live started to save our show file. We held our breath hoping we wouldn’t need to reconstruct the show from the ISOs that our remote teams had already begun uploading. AJ started making a list of ECamm feature requests for Ken and Glenn. The first one being the ability to segment recording files without stopping the stream. You’ll find they took his feedback and implemented it for everyone.
All in all, the show was a huge success. We tested the boundaries of both Zoom and ECamm Live and even MS Team’s Town Hall’s ability to stream for more than a day. The testing we did before the show and the client’s trust in us to find and deliver a solution brings us to another company value of Four Winds Creative; and that’s Freedom. The tools and technology we selected for this “impossible mission” allowed the client, our AV teams, and the presenters themselves to be free to share their information in the most impactful way. We had fun, taking this customer “Can it be?” question and making it a reality for them. The testing and final delivery pushed the technology, our teams and our endurance. We learned and improved ourselves and the tools along the way and we’re already looking forward to improving on the experience next year.