artnet Art-net problems on backup machine

[GCLights]

We've been having issues with Art-net coming from Watchout going to our projectors. It started when we had two projectors on different universes. We were advised that we should consolidate them to a single universe due to some issues with how Watchout handles Art-net output. To make sure any potential issues were minimized, we addressed both projectors to Art-net Universe 1, Address 1.

Later today, we had a failure on the main machine, so we switched to the backup. I tested DMX as soon as we switched, and it worked. A few minutes later, it did not. Once we got the main back up and running, we tested DMX, and got no response. We tried opening another showfile with a known working DMX output, and got no response from the projectors. We closed that show file and tried DMX again from our real showfile, and got no response from the projectors. I changed the output to Universe 1 Address 11, just in case I'd lost my mind and changed the addresses without realizing it. The projectors obviously didn't respond.

I changed the address back to 1-1. It started working again.

After that, we came up with a theory. We'd been pushing the showfile from main to backup, opening it, editing the display IPs, then taking it online. What we tried was saving it on the new machine before taking it online. We suspected that some data about the machine was being stored in the save file, and until it is saved on the new machine the output is corrupted.

That didn't work, but we got some good data. DMX didn't work on the backup after the failover, but it worked after we edited the DMX address and set it back. Then I saved, closed, and re-opened the file on the backup and it didn't work. Editing and resetting the IP address made it work again.

I have two questions:

A: Is this a known bug?

B: Is there a way we can solve this problem? I'd prefer not to have to have our operator edit the show file every day.

[RBeddig]

Are you using two completely different sets for main and backup, i.e. two production computers and two display computers? Are you using Art-Net or DMX, i.e. an Art-Net to DMX converter between WATCHOUT and the projectors?

Usually Art-Net and DMX work in a way that the last command takes precedence over the commands before but to my knowledge and experience WATCHOUT keeps sending the current Art-Net data even if there is no change. If the first computer sends 0 because it is at the beginning of the timeline where the output should still be 0 and the other one send a different value it can happen easily that the value remains at 0 or does strange value jumps negotiating between the two values sent from two controllers.

 

[GCLights]

We have two production and two display computers.

Art-Net is running from the computers through a network switch to both projectors.

During the tests, we would take one computer offline before bringing the other one online, to simulate failover conditions. There was one time during testing that we had both systems able to broadcast art-net, when we brought both machines online at the same time to see what would happen. We got flickering output from the conflicting signals.

[GCLights]

I did some tests this morning. When we opened the show files, DMX worked on neither the primary nor the backup until I manually  edited the DMX addresses in Output on them.

After that, I could smoothly fade Art-Net on whichever machine was online, as long as the other one was offline, no cable pulling required.

Further monitoring with ArtNetominator shows that the offline system is not sending Art-net packets.

[GCLights]

I've done a packet capture with Wireshark. There's a difference in packets before and after we edit the DMX address in the show file. The UDP packet length increases from 30 to 38.

Before:

0000   ff ff ff ff ff ff 9c 5c 8e 97 c5 f8 08 00 45 00   ÿÿÿÿÿÿ.\..Åø..E.
0010   00 32 03 b4 00 00 80 11 db 0b ac 10 01 dc ac 10   .2.´....Û.¬..ܬ.
0020   01 ff c0 11 19 36 00 1e 51 07 41 72 74 2d 4e 65   .ÿÀ..6..Q.Art-Ne
0030   74 00 00 50 00 0e 00 00 01 00 00 04 00 00 00 00   t..P............
 

After:

0000   ff ff ff ff ff ff 9c 5c 8e 97 c5 f8 08 00 45 00   ÿÿÿÿÿÿ.\..Åø..E.
0010   00 3a 03 b6 00 00 80 11 db 01 ac 10 01 dc ac 10   .:.¶....Û.¬..ܬ.
0020   01 ff c0 11 19 36 00 26 50 ef 41 72 74 2d 4e 65   .ÿÀ..6.&PïArt-Ne
0030   74 00 00 50 00 0e 00 00 01 00 00 0c 00 00 00 00   t..P............
0040   00 00 00 00 00 00 00 00                           ........

[GCLights]

I've solved the problem. It's multi-layered, but has a simple solution.

Watchout only natively broadcasts as much of the DMX packet as it has to. If you're just broadcasting CH1, it will only include the first 2 channels of DMX. If, at any time, you broadcast something above that range, it will broadcast more of the packet. For instance, if you send Channel 7, it will send additional octets that will send control up to Channel 8. Sending Channel 8 will not extend the broadcast to Channel 9, but 9 will extend to 10, 11 to 12, etc.
This makes sense in many cases, since it optimizes network traffic.
 

On the other hand, there are the Panasonic projectors (PT-RX110). They have a customizable profile that runs between 1 and 11 addresses. As far as I can tell, they want to listen to packets that are the entire length of their maximum DMX range (11 channels). If it is shorter than that, they will not respond to DMX control. There is no indication that they are getting any data until they get a large enough packet to cover their largest possible personality.
 

To solve that, I set up a dummy channel to broadcast address 512 throughout the show, which guarantees that the packets will always be long enough that the projectors will always listen to them.

[Guest DavidA]

Thank you very much for your efforts on solving this problem! Your feedback and discovery is highly appreciated! 

Best regards, 
David / Dataton

[callum]

I second David's thanks! This is a very interesting software 'feature' to know about. I would have always thought that for art net to be valid it must output channels 1 to 512