Best Practice for Redundant Show Playback? (aka Backup)

[Nate]

I just recently finished a show where we ran a 3-output edge-blend from a single display computer. I also had an additional triple-output display computer on-site as a backup.  I ran both of them together on the same network switch with the production computer. All outputs ran into a HDSDI switcher so we could send any output we wanted up to the projectors.

 

For this event, I would push the final show to the primary display computer, save a "backup" version of the show, change the display IP addresses in the backup version, and then push the show to the backup. It worked well enough but, I would like a more seamless solution since, in the event of a display crash, the backup would be waiting for cues from my production machine and I would have to close the original show and open the backup version before being able to resume playback (losing precious seconds of recovery time during a show).

 

My question is: Is the solution as simple as stacking the backup displays on-top of (or underneath) the primary displays on the watchout stage so it fires all of them in unison? So, if layered, it would be layered something like this:

 

Primary Displays: [ Display 1a ][Display 2a][Display 3a]

Backup Displays: [ Display 4b ][Display 5b][Display 6b]

 

In theory it seems like it would be an ideal solution but, I was wondering if anyone else has a better live backup solution. Thanks!

[Jonas Dannert]

One simple way:

 

1) Create a Tiered Layer in the Stage Window. (Stage/Tier/Add/Name it/Save)

 

2) Return to the Base layer (Ctrl-Shift-1)

 

3) Select all your Displays in the Stage Window, copy.

 

4) Select your newly created Layer - Paste in your copied Displays.

 

5) Change IP-address/es to your Backup Systems, don't change name on the Displays.

 

Done!

 

This will not cover every situation, but everything you do on Primary will happen on Backup, too.

Of course you will not safeguard the Production computer, but your Backup Display system will run hot now.

 

/jonas

[Nate]

Awesome Jonas. I will test that configuration now. Thanks for the reply!

[avdavesound]

stage tiers is good for running hot display machines, but want about if i need a backup production machine running aswell? whats the best solution to have full redundancy on displays machines and production pc?

[Jonas Dannert]

Two completely separate systems and external control of both, maybe?

 

/jonas

[Thomas Leong]

...whats the best solution to have full redundancy on displays machines and production pc?

 

Here's my 2-cents -

 

Full redundancy would mean having 2 sets of Prodn and Displays as Jonas has pointed out. Control of both Prodn machines can be from a 3rd WO Prodn machine (eg. an Atom-based laptop since it need not run visuals on the Stage, or rather, since there is no Stage media, only output cues or Aux timelines without visual media). I would envisage the same Ethernet hub (or a backup - one set to each hub), and the same subnet mask.

 

Depending on the depth of redundancy one wants/needs, a Matrix Switcher (eg. Kramer VP88, or a seamless Matrix Switcher) would take all outputs from the two sets of Display machines to feed one set of projectors.

 

If there are 2 sets of projectors, then these would have to be stacked, and on a controllable dowser or shutter on/off command (from an Aux Timeline). Additionally, the projectors' source input switching commands can be in the same Aux Timeline as the dowser/shutter on/off commands such that they all switch at the same time.

 

A Matrix Switcher with a 'Take' feature is desirable as a manual backup.

 

Thomas Leong

[maccalder]

This is where I think a lot of peoples redundancy arguments fall over - the addition of a matrix switcher does 2 things - it increases the number of failure points by (2xoutputs)+1 as there are now an additional 2 cables for every output as well as a single point of failure in the matrix - there is also no redundancy from the second half of the signal chain. I would rather put my money on WO straight to projector over redundant WO to a matrix to projector.

 

From my point of view after several hundred one-off watchout builds I have found most issues come from bad cables causing displays to drop and watchpoint to crack a spaz or bad media. Redundant PC's will obviously see you through the spaz out if a display drops (But you also have >2 times the chance of it happening on ONE of your machines as the Matrix can also drop out) - what really makes a hardened system is tweaking the OS as per the WO guide then testing your media thoroughly. Re-image your display and production machines regularly and if you have any concerns about your primary setup - from cables to projectors - replace it. If it is so critical that everything goes smoothly, the only redundancy worth considering is a completely separate system running in parallel. Double the projectors, double the cabling, double the WO Displays and double the WO Production PC's.

 

My 2c anyway.

[Curtis Hutchins]

This is where I think a lot of peoples redundancy arguments fall over - the addition of a matrix switcher does 2 things - it increases the number of failure points by (2xoutputs)+1 as there are now an additional 2 cables for every output as well as a single point of failure in the matrix - there is also no redundancy from the second half of the signal chain. I would rather put my money on WO straight to projector over redundant WO to a matrix to projector.

 

From my point of view after several hundred one-off watchout builds I have found most issues come from bad cables causing displays to drop and watchpoint to crack a spaz or bad media. Redundant PC's will obviously see you through the spaz out if a display drops (But you also have >2 times the chance of it happening on ONE of your machines as the Matrix can also drop out) - what really makes a hardened system is tweaking the OS as per the WO guide then testing your media thoroughly. Re-image your display and production machines regularly and if you have any concerns about your primary setup - from cables to projectors - replace it. If it is so critical that everything goes smoothly, the only redundancy worth considering is a completely separate system running in parallel. Double the projectors, double the cabling, double the WO Displays and double the WO Production PC's.

 

My 2c anyway.

 

You hit upon an interested subject,  Is there a way to protect the system from the cabling side? as you say WO sometimes spaz's out from bad cabling or to be fair its the hardware. I recently had a flaky power to an HDMI extender that was drawing power from the card.  It caused a ton of problems before we figured it out.

 

My thinking is that some videos cards (once that don't draw as much power and perhaps have better internal circuitry, and a more powerful power supply may help in this regard but I'm not really sure what else can.

[jfk]

You hit upon an interested subject,  Is there a way to protect the system from the cabling side? as you say WO sometimes spaz's out from bad cabling or to be fair its the hardware. I recently had a flaky power to an HDMI extender that was drawing power from the card.  It caused a ton of problems before we figured it out.

 

My thinking is that some videos cards (once that don't draw as much power and perhaps have better internal circuitry, and a more powerful power supply may help in this regard but I'm not really sure what else can.

 

I would never trust a device that relies on power from the graphics card connection.

Look for extenders that have their own power supply to avoid such issues.

[Curtis Hutchins]

I would never trust a device that relies on power from the graphics card connection.

Look for extenders that have their own power supply to avoid such issues.

Yup.  this was bad power supply to the extender.  But as Macadler was saying,  a lot of trouble that initially gets blamed on WO can actually be external cabling.

[Jonas Dannert]

"a lot of trouble that initially gets blamed on WO can actually be external cabling."

 

You can usually count on that. One do need to do the homework here.

Always try with short, direct cabling first, circumventing Cat5 extenders etc.

Then, I always use EDID Managers, like DVI Parrot's, on all outputs...

 

/jonas

[Mike Fahl]

An alternative to external EDID Manager boxes (which add their own failure points) is the EDID management built into the drivers of some professional grade graphics cards (e.g., AMD FirePro). Doing it in the driver settings is a rather attractive solution.

 

Mike

[Curtis Hutchins]

I've never understood how/why EDID can actually crash a computer which I have seen a several occasions. It would be good to understand this...

 

I have definitely had issues with Extenders as mentioned above but we do what we must. Never used a Parrot but I know a DVI-Detective can sometimes make all the difference...

 

Thanks again.

[Jonas Dannert]

It's not EDID that crashes WATCHOUT and/or the Display PC,

but the loss of EDID or connection to the display device,

that causes Windows to re-arrange the number of graphic card outputs.

 

This in turn cause a WATCHOUT Display software crash.

Even an intermittent glitch, can set this off.

 

There's no good way for us to stop this from happening.

 

Using software or hardware EDID Management will remedy this, however.

 

This is distribution 101, in my opinion.

 

/jonas

[Benoit]

Hello,

I’d like to suggest something.

When you remove a media, it tells you if the media is used. If you work with a huge project, it’s not so easy to know where the media is used.

Maybe you can update the message box to tell the user where the media he tried to delete is used:

“Selected media is in use and cannot be removed. Used in the following tasks:

• Task 1 (00:00:01, layer 23)
• Task 2 (00:12:48, layer 3)

…”

Thanks,

Benoit.

[Bluetonesblue]

I agree EDID management is essential for a smooth project. It's one of the reasons that I always convert to SDI right out of the rack, or go direct into an E2 or Spyder switcher, so that the signal doesn't drop based on a long run. Once converted to SDI, the signal is robust and easy to manage and troubleshoot. With only 6' (1 M) cables Displayport/HDMI cables, I have little to no problems.

 

My 2 cents, (or .02 EUR)

Your mileage may vary...