Watchout with Video Matrix (Discussion)

[Yousefjuma]

Hello everyone,

 

Hope you are having a good day. We use watchout for live events, corporate events and festivals. We use projectors, large LED screens and LCD monitors. Usually we have in each of our racks 1 production PC and 2 display PC. Display PCs are equipped with 2 DVI inputs and 1 SDI input at least and 6 miniDP 4k outputs (Firepro W9100).

 

For backup, we usually feed 2 identical inputs into LED sending card with each input coming from different display PC. And we use LED sending card software to switch between the inputs in case that 1 of the display PCs crashed or something.

 

When we work with projectors, we stack 2 projectors to project an identical image. Each projector is getting identical feed but being fed from different display PC. Therefore, if any of the projectors failed, we have another one ready. And if one display PC failed, we have another display PC running in parallel. 

 

We have seen people using video matrix along with watchout (An example: http://www.analogway.com/en/products/mixers-seamless-switchers/premium-av-switchers/ascender-16-4k-livecore-/).And right now we are studying weather or not this method is any advantageous over our current method.

 

I would like to discuss the pros and cons of each of the methods and weather or not you suggest other methods that might be better.

 

[orfischer]

We are using Barco E2 to solve that problem.

 

We feed main and backup feeds from separate display computer to the E2, and record several stil stores (logos usually) going to all displays in case anything crushes.

 

Basically, main display goes to all screens. If main display computer fail, the E2 allows a seamless switch between the machines (main and backup)

 

The problem is that display computer 1 and 2 share the same show, and a single component of the show may crush both machines.

 

This is where the still store gets handy, you can just switch seamlessly to a logo or something similar on all screens, while fixing the problem.

 

In the past year, only once (out of 30 shows) one of the machines crushed, we switched to stillstore on E2 while we solved the problem on Watchout and than went back live in about 30 seconds without affecting the show.

 

The last resort for us in terms of backup, is to feed a stilstore on all secondary displays, and a MacBook Pro pre loaded with all the content to main display. THis way, if the problem persist, can can stil playback the content on the main screen.

 

While We are not familiar with the Ascender family, we believe it serves the same purpose as the E2.

[Steve Farris]

I have worked with systems configured similar to both scenarios presented here.  There are pros and cons for each.  The critical question is to analyze what are the potential failure points and what will happen if those points fail.  Then you need to determine if the solution around the failure is more cumbersome than it's worth, or if it introduces new failure points.

 

The Ascender is a similar concept to the e2, just a simpler box that is easier to learn (although the e2 is quite a bit more versatile).  I had an Ascender crash on me once (due to a system bug that I believe they have fixed).  I've heard of failures on e2s as well.  If your entire show goes through a switcher and it fails, the entire show fails.  However, failures of those boxes are rare and quite frequently due to operator error.

 

The system you describe is pretty much a true fail safe since you have two identical systems running in parallel.  You could remove any component from one system, and the other could just take over.  The downside is if they get out of sync you might see a mess on the screen (unless the "backup" projectors are shuttered).  Another problem would be if the show has a problem, both machines will hit the same problem at the same time.  I commonly keep my backup machine synced by manually jumping ahead as the primary computer runs.  That way if running over a cue causes the failure, the backup has missed it.  I have switched shows this way with very minimal down time to the screen (although it seems like years back stage!).

 

There is no right or wrong way to do it, just different approaches.  Good luck, and I hope you never have to use your backup!

[NNinja]

I've tested both E2 and AnalogWay (unfortunately none of them currently in our rental).

 

Any solution that gives you fast and seamless switching between the main and the backup and minimum delay will be fine according to me.

[JJ Myers]

This topic - by far - is the most challenging part of our group's (Fuse Creative LLC - Chicago) service offerings to our live events clients - both from an implementation standpoint, and a decision standpoint. We have been dealing with this topic for many years and have tremendous amounts of experience worth sharing.

 

First of all - there are advantages and disadvantages to all the solution options out there. There is no one superior solution. And different scenarios can prove one solution better than the other, and in some cases comp[letely eliminate certain solution options.

 

Certainly, from a show risk and simplified implementation standpoint, using a multi-source switching system (in which incorporates double hardware scalers), such as Barco e2 is the way to go. However, there is one major disadvantage to this solution: the processing involved with using a hardware scalar comes at the cost of latency. An e2 will add system frames of latency to the signal. Our experience is 1-2, depending on variables. For running media cues, the issue of latency can be moot or circumventable. However, for live sources (mainly live camera), the latency that e2 adds can create a problem in a live event environment that is unfixable as your show begins to look like an erroneous slip edit.

 

If you do the math, and determine your capture card adds ~4 frames of delay and your e2 adds ~2 frames of delay. Right there, you are already at 6 frames of delay. LED processor in line? Add another 1-2 frames of delay. Projectors are using warp engine features (perhaps unbeknownst to you)? Add another 1-2 frames of latency. If your camera feed is of the interlaced variety? Add additional frames for the need to de-interlace in WO. You may say - no problem, we have a multi-format camera switcher and we can simply set our system output to a progressive format. Well guess what? the camera switcher needs to do some processing in order to accommodate format conversion which then adds latency. So... OK, let's set our cameras to 720p and have everything on the camera switcher run natively. Oh wait, the client would like their native show record format to be at least 1080i. Ok... let's just use 1080p cameras! Oh wait... that has now killed the budget to do WO for the show. This is seriously the conundrum we encounter frequently and at the end of the day we end up explaining to the client that some form of compromise needs to be made... or they need to increase the budget.

 

If you are playing to a large room and the audience is pushed way back, the latency can be a moot point as the natural audio delay (by the time it reaches the audience) catches up with video processing delay and the show no longer looks like our erroneous slip edit. But if you are playing to an audience that is anywhere near the screens (and we all know the ones cutting the checks like to sit in front), latency beyond 4 frames can easily start to look like that bad slip edit.

 

But wait? Why would I need to bring any live cameras into WO if we have an e2 downstream? Well, we don't.... that is, unless you have a client who would like a fancy DVE look for their live camera. Perhaps the producer has watched too much CCN or Sportscenter and insists on seeing a live camera feed swing around on it's Y-axis, or follow and become shaped by a moving track matte in coordination with some motion GFX. In such cases, where the client is insistent on a fancy representation of live camera, the only option is to integrate live camera into WO (or whatever media server you choose to use).

 

Also, e2 is not cheap and requires another skilled position/operator. If you are a company like ours that aims our focus on media server playback, speaker presentation support, and content creation - adding e2 into the list of services is a whole new world that now requires a whole new level of investment into resources and training. Now we are having to sacrifice some of our time we would otherwise spend learning a new plugin or inventing a new trick in WO.

 

I personally am not a fan of the approach that sends different hardware channels of the same redundant signal to each projector of a stack (in the case of projection). It means all your WO channels are live at all times, doubling the amount of channels you need to be actively managing and monitoring. It also means a single production machine managing all channels in order to keep the 2 channels frame locked. Lose your production machine for whatever reason? You are now scrambling to bring another production machine online or using cluster control to run your cues. It also means ANY form of failure will have an impact to the audience. If any "B" WO channel crashes in this scenario, the result is a screen that is now have as bright (as a result of shuttering the projector with the bad signal). And if that projector is part of a blend, the projectionists will likely be forced to shutter ALL "B" projectors for the sake of a lesser of evils in the presentation to the audience. So... you have a scenario where the client just lost 50% of their screen brightness due to an error on a single WO "B" channel. If a "B" channel crashes in other approaches, no one in front of the big black curtain has any clue.

 

Also, this approach adds more complexity in the implentation on the projection side of things. When a projectionist has a stack of projectors, they typically expect the same redundant signal to both projectors. That simplifies things in a case where signal troubleshooting of some sort is necessary. If the distribution is via a simple hardware splitter or matrix router, the projectionist at least knows the origin of both signals is identical (in terms of timing, RGB values, etc...). If the signals are of 2 different origins, that can add to the complexity when troubleshooting. 

 

More often, we use a matrix router with zero scaling/processing to manage primary/backup switching. We used to use Geffen Pro, but we have since started using Lightware. Yeah, they are expensive, but there are some really good cost-effective rental options out there - where if you budget and quote your show right, you can phase the cost of the rental into project with a good return of value and piece of mind. Lightware also makes some nice multi-card routers where you can have a system that supports a variety of computer signals. Geffen used to as well, but they have drastically changed their target in recent years. If you spec a large enough router, the router can also be your means of distribution. In many cases, this router ends up essentially serving 3 purposes and otherwise reducing a lot of extra peripherals like Fiber transmitters and/or signal splitters. If you use top of the line routers (I suggest stay away from stuff from companies like Kramer), you can get other great options like redundant power supplies and support for genlock. Having a modular card based router with a redundant power supply, redundant WO system, and a strategic power plan can eliminate (or at least mitigate) single points of failure. Using WO systems with S400 cards and properly gen locking everything to a black burst generator (including the router) will also provide you a clean switch between your primary system and backup system. The disadvantages of this approach are not having another failsafe to go to (other than the backup WO system) like a still store, and just generally being live at all time. That can make things challenging - for instance - if your projectionists need grid, but you need to work. There are ways to work around that (other than the obvious going offline to work), but just require some abstract thinking when planning your WO project. We have found ways to put ourselves in positions where projectionists have grid, and our stage manager and ourselves can still see what we are programming and thus continue to work on the show. We have also built some great custom software applications that allows us to run multiple matrix switches from the push of a button, so that main/backup switching is quick and easy to get to.

 

In the end - my suggestion is to always choose something that delivers to the client's expectations, while posing the least amount of show failure risk possible. Remember - show failure risk management is not always just what your hardware could potentially do, but what your operator could potentially do! And take a guess as to where most show failures occur - human or computer error? I don't think I even need to answer that. Operators must juggle the running of show cues, keep their head in the show, while monitoring and managing various system resources. If your operator is spending more time managing resources than keeping their head in the show... that poses a risk! Reduce distractions to the show operator and you will reduce your overall risk.

[Yousefjuma]

We are using Barco E2 to solve that problem.

 

We feed main and backup feeds from separate display computer to the E2, and record several stil stores (logos usually) going to all displays in case anything crushes.

 

Basically, main display goes to all screens. If main display computer fail, the E2 allows a seamless switch between the machines (main and backup)

 

The problem is that display computer 1 and 2 share the same show, and a single component of the show may crush both machines.

 

This is where the still store gets handy, you can just switch seamlessly to a logo or something similar on all screens, while fixing the problem.

 

In the past year, only once (out of 30 shows) one of the machines crushed, we switched to stillstore on E2 while we solved the problem on Watchout and than went back live in about 30 seconds without affecting the show.

 

The last resort for us in terms of backup, is to feed a stilstore on all secondary displays, and a MacBook Pro pre loaded with all the content to main display. THis way, if the problem persist, can can stil playback the content on the main screen.

 

While We are not familiar with the Ascender family, we believe it serves the same purpose as the E2.

 

So basically the only function of the E2 is switching between the Main and Backup systems.

I have a some question regarding your setup. Where does the Camera feed and laptop (For PPT) go into?

And Does the E2 add any extra delay to camera feed or not?

Thanks for your reply. 

[Yousefjuma]

The critical question is to analyze what are the potential failure points and what will happen if those points fail.  Then you need to determine if the solution around the failure is more cumbersome than it's worth, or if it introduces new failure points.

I totally agree with your point whenever we talk about backup system.

 

The system you describe is pretty much a true fail safe since you have two identical systems running in parallel.  You could remove any component from one system, and the other could just take over.  The downside is if they get out of sync you might see a mess on the screen (unless the "backup" projectors are shuttered).

 

 

They are usually not shuttered and I have faced the out of sync issue only once. It was about 1 second off sync but I personally think it was because I was jumping in the timeline.

 

 

I commonly keep my backup machine synced by manually jumping ahead as the primary computer runs.  That way if running over a cue causes the failure, the backup has missed it.  

 

 

 

Can you explain how you do this please?