latency Latency on running cues

[JJ Myers]

So, we believe we have discovered more information relating to this phenomenon. We - at the very least - have something somewhat reproducible. The WO project for the reproducible case is at this link:

https://www.dropbox.com/s/i6f12ixyzwglfde/Latency Bug.zip?dl=0

We also recorded a video that presents how to reproduce:

https://www.dropbox.com/s/krhh85m1e9z8b3c/IMG_2142.mov?dl=0

Here are our specs and setup for this test:

Display machines:
    - Windows 7 64b Professional
    - ASRock X99 WS-E LGA 2011-v3 Intel X99 SATA 6Gb/s USB 3.0 Extended ATX Intel Motherboard
    - Intel Core i7-5930K Haswell-E 6-Core 3.5 GHz LGA 2011-v3 140W BX80648I75930K Desktop Processor
    - SanDisk SSD PLUS 2.5" 120GB SATA III Internal Solid State Drive (SSD) SDSSDA-120G-G25 (OS)
    - SanDisk Ultra II 2.5" 480GB SATA III Internal Solid State Drive (SSD) SDSSDHII-480G-G25 x 4 (Media, Hardware RAID 0)
    - MD FirePro W7100 100-505724 8GB 256-bit GDDR5 PCI Express 3.0 x16 Full height/full length single-slot Workstation Video Card
    - ATI S400 sync card

Production machine:
    - Windows 10 Professional
    - Latest 8-core Mac Pro

In the video, we had 2 displays running 2 feeds each, all genlock synchronized via an NTSC composite signal. One machine is supplying Feeds 01 and 04, and the other was supplying Feeds 02 and 03. We set up both machines to sync from genlock (house sync) in this particular video.

After recording this video, I remembered a discussion where the official recommended approach was to set the assigned master to sync from genlock and then use a cat5 connections on the master s400 card to serve as the synchronization master to the slaved machine. Just for clarification, here are screen shots of the settings for that synchronization method:

https://www.dropbox.com/sh/vvvww8eos3trnlq/AABdhXbxj6v1SbiGVfFTvppda?dl=0

We ran this same test with that setup and experienced better results (an increase in stability). We had a much more difficult time reproducing what you see in the video, but none-the-less we were able to get it to occur under these conditions as well. We currently have a system running 24/7 with automated cueing in this set up so we can monitor prolonged usage under various types of synchronization and on various versions of WO.

We also ran this test on a system not using genlock, rather using Feed 01's display as master timing source in Firepro, and then a cat5 connections on the master s400 card to serve as the synchronization master to the slaved machine. Same results of instability and latency as you see in the video. Here are the firepro settings for this synchronization method:

https://www.dropbox.com/sh/uxrynsjauhdpbb2/AAAFdbS4OYRI6w6xrkDuPYQUa?dl=0

I want to point out that we not only experienced cases where both machines were equally latent (as seen in the video), but cases where the latency between the 2 machines was unequal... which of course is a far worse predicament to be in on a show. What we have never encountered is a scenario where individual feeds from a given machine to be unequally latent in relation to each other.

We also - at this point - can rule out 2nd nic and/or virtual displays being a culprit in these particular instances, as both were factored out of these tests. I am going to supply this test project to dataton support and hopefully get some answers to what we are encountering. I would love to have other members of the WO community run this same test and get feedback as to whether they experience the same results. Perhaps something can ultimately be traced down to a piece of hardware, tweak setting, or the like.

Our current plan is to run these same tests on rollbacked versions until we get back to a level of stabliity we can trust. Once we get to that point, we are going to set that as our version to use for future shows until a new release remedies the problems we are encountering. We otherwise do not feel comfortable with stability in v6.2.2 (regardless of method for synchronization). And we have encountered this latency bug in v6.1.6, so it is possible we will roll back all the way to v6.0.2. We will of course keep users informed of our results every step of the way.

[Walter Soyka]

On 10/2/2018 at 3:32 PM, JJ Myers said:

So, we believe we have discovered more information relating to this phenomenon. We - at the very least - have something somewhat reproducible....
We also ran this test on a system not using genlock, rather using Feed 01's display as master timing source in Firepro, and then a cat5 connections on the master s400 card to serve as the synchronization master to the slaved machine. Same results of instability and latency as you see in the video.

Nice, JJ! We were also syncing via S400 without genlock in the original post, and we were running WATCHOUT 6.1.6.

w.

[JJ Myers]

Ring the bells! We believe we may have found the culprit! Or at least, we have found something that prevents the test project from reproducing the bug. The test project in my post on October 2 uses IP address definition for 2D displays. I revised my project in the following way:

1. Set computer names and a cluster name for the 2 displays I have online

2. Set the cluster name in preferences

3. Define all my 2D displays by computer name instead of IP address

I would have never thought such a revision would have resolved this issue, but alas. I would - by no means - suggest that our results are conclusive, as they only have solved the reproducible case. For us to put our rubber stamp on this, we are going to run further prolonged testing to see if the latency issue is completely absolved. For us, that means several days of typical operator cueing.

I would love to hear back from other users as to their findings between IP address defined displays versus computer name defined displays as it relates to this bug. And let's please spread the word! If our efforts can prevent even just one WO show from having a catastrophe, the effort will have been worth it. So please by all means - pass it on!!

[jfk]

Sure explains why some people see and some people do not.

I have been an ardent proponent of removing IP addresses from Display dialog since v5.5.1. I have posted here about it before.

I understand the IP address support is retained for backward compatibility reasons, but the software should throw a warning when old shows are brought in with IP addresses in Display dialog, or anytime an IP address is found in a Display dialog for that matter.

I wish Dataton would revise the User Guide accordingly.

22 minutes ago, JJ Myers said:

Ring the bells! We believe we may have found the culprit! Or at least, we have found something that prevents the test project from reproducing the bug. The test project in my post on October 2 uses IP address definition for 2D displays. I revised my project in the following way:

1. Set computer names and a cluster name for the 2 displays I have online

2. Set the cluster name in preferences

3. Define all my 2D displays by computer name instead of IP address

I would have never thought such a revision would have resolved this issue, but alas. I would - by no means - suggest that our results are conclusive, as they only have solved the reproducible case. For us to put our rubber stamp on this, we are going to run further prolonged testing to see if the latency issue is completely absolved. For us, that means several days of typical operator cueing.

I would love to hear back from other users as to their findings between IP address defined displays versus computer name defined displays as it relates to this bug. And let's please spread the word! If our efforts can prevent even just one WO show from having a catastrophe, the effort will have been worth it. So please by all means - pass it on!!

BTW  I have found only one reason where it may be necessary to put an IP address in a display dialog. And even that is just a temporary solution. I have encountered v5.5.x WATCHPAX that would not respond to v6 software when the name/cluster name are used. In that instance, putting an IP address in long enough to update the WATCHPAX software got the job done gets around it. But as soon as the update is complete, switch back to computer name. (When upgrading WATCHPAX WATCHOUT software from v5 to v6, be certain the WATCHPAX license supports v6 using WATCHOUT v5.5.x software BEFORE updating software to v6 - updating a WATCHPAX with v5 only license to WATCHOUT 6 software will brick the WATCHPAX).

 

 

[WatchDog]

I remember avoiding the use of cluster names over IP addresses when I found bugs related to proxy and audio issues while using cluster names. A few years have passed so I'm hopeful I do not encounter those bugs now that I am forced to use cluster names to maintain sync between displays. Fingers crossed!

[jfk]

46 minutes ago, WatchDog said:

I remember avoiding the use of cluster names over IP addresses when I found bugs related to proxy and audio issues while using cluster names. A few years have passed so I'm hopeful I do not encounter those bugs now that I am forced to use cluster names to maintain sync between displays. Fingers crossed!

Most likely those issues are corrected.

Keep in mind that Dataton internally tests using computer name / cluster name.

[dchristo]

Please do not remove the ability to use IP address... We occasionally have Production machines on different subnets from the Display Cluster, and computer names don't work in this scenaio.

Thanks,

--D

[jfk]

12 minutes ago, dchristo said:

Please do not remove the ability to use IP address... We occasionally have Production machines on different subnets from the Display Cluster, and computer names don't work in this scenaio.

Thanks,

--D

One of WATCHOUT's requirements is all WATCHOUT computers be on the same subnet, always has been.

[zackboyd]

13 hours ago, jfk said:

One of WATCHOUT's requirements is all WATCHOUT computers be on the same subnet, always has been.

That seems awfully short-sighted... AV integration on largescale network infrastructures is becoming more and more commonplace everyday. You're going to begin forcing people's hands.

[jfk]

2 hours ago, zackboyd said:

That seems awfully short-sighted... AV integration on largescale network infrastructures is becoming more and more commonplace everyday. You're going to begin forcing people's hands.

While I see your point, the reason traces back to the synchronization over the network. Synch is maintained by UDP transmissions, for obvious reasons. And the UDP typically stays within the subnet.

[dchristo]

Sorry, don't mean to turn this thread into a network design discussion, but I'd like to add a couple of points:

a) we do not run shows from remote subnets; in this scenario we use shared production resources to push shows to remote clusters, which requires the displays be defined with IP addresses instead of names.  The remote clusters are each on their own subnet.  This setup has been working well for us for over 8 years.

b) UDP is a fully-routable protocol, and is not "typically" restricted to a single subnet (unless it's a broadcast packet, in which case that is true).  Both uni-cast and multi-cast UDP packets can traverse subnets just as TCP does.  There may be some consumer-level routers that come with a default firewall that blocks UDP, but that's not the default on enterprise-level gear.

c) In a properly designed and managed modern network (this being the key), there is little to no performance impact of crossing a subnet with either TCP or UDP.  We currently have facilities with a single network (tens of thousands of end devices), which supports hundreds of uncompressed, synchronous audio channels (both uni- and multi-cast), IPTV, Watchout, streaming ACN lighting control, and Show Control, along with all the typical office support services (email, file share, database, etc.).  These networks are properly segmented to restrict traffic, provide access control, and to provide redundancy.

d) What Zack said.

 

--D

[Walter Soyka]

On 10/12/2018 at 2:42 PM, JJ Myers said:

I would love to hear back from other users as to their findings between IP address defined displays versus computer name defined displays as it relates to this bug. And let's please spread the word! If our efforts can prevent even just one WO show from having a catastrophe, the effort will have been worth it. So please by all means - pass it on!!

Our thread-starting setup defined displays by direct IP address. I'll use computer names on the next one and see if it makes a difference. Thanks so much, JJ!

 

w.

[Klangreich]

Intresting outcome! Can anyone explain the difference between adressing the displaycomputers by ip or name?

[jfk]

6 minutes ago, Klangreich said:

Intresting outcome! Can anyone explain the difference between adressing the displaycomputers by ip or name?

Not sure what you re asking, but does this help? 
v5.5+ computer name - fundamental change in display dialog address assignment

[stiegzinator]

2 hours ago, Klangreich said:

Intresting outcome! Can anyone explain the difference between adressing the displaycomputers by ip or name?

I'm also wondering about this, myself. I understand how to go about doing it, but I don't really understand what Watchout is doing with one versus the other and why one causes problems. Isn't the IP address just one way of uniquely differentiating between computers? Why would one method of identification be so different from another method?

[jfk]

2 hours ago, Klangreich said:

Intresting outcome! Can anyone explain the difference between adressing the displaycomputers by ip or name?

 

2 minutes ago, stiegzinator said:

I'm also wondering about this, myself. I understand how to go about doing it, but I don't really understand what Watchout is doing with one versus the other and why one causes problems. Isn't the IP address just one way of uniquely differentiating between computers? Why would one method of identification be so different from another method?

While I do not know the answer to your question, why does it matter? Nothing is preventing you from using fixed IP addresses in the Windows NIC settings, just do not use them in WATCHOUT. 

Not every media producer is a network maven. My guess is Dataton wanted to remove the need to understand basic networking practices to get a basic WATCHOUT system up and running. Assuming all the WATCHOUT computers are set to DHCP (and a clean Windows install would set them that way ), connect all the WATCHOUT stations to a dumb switch and it worked without any NIC configuration. When WATCHPAX was introduced in conjunction with v5.5, Dataton eliminated the need to know IP addresses and replaced it with the WATCHOUT computer name. This also added the new feature where WATCHOUT Production could automatically discover the Display servers active on the network via the Production software's Network window.

BTW At the time WATCHPAX was introduced, with WATCHOUT v5.5.x and without a DHCP server, a WATCHPAX could not be given a fixed IP address, the primary IP address was always DHCP assigned. The WATCHOUT v5 setIP script command set an alternate alias IP,  but did not impact the primary IP. In WATCHOUT v6 the ability to set a true fixed IP was added. 

[Or Israel]

Hi Guys,

 

The issue is that the clock sync between the production machine and is off. 

 

Make sure that your time / date matches exactly between the production and display machine.

 

Just to make this clear

1. On the production machine -> right click on the time/ date in windows -> Adjust date and time -> Sync Now

2. On the Display / Runner machine -> right click on the time/ date in windows -> Adjust date and time -> Sync Now (if you are not connected to the internet, it'll take sync from the production machine over network).