PJ Link Projector Command Issues

[RobBergenstock]

Hello,

 

I am having difficulty maintaining reliability on sending shutter and power commands to our projectors via PJ Link. The codes are successfully sent about 60-70% of the time, so I am trying to figure out what could be causing these intermittent failures. Our equipment/setup is as follows...

Production Computer: Dell Optiplex 960

Display Computer: Show Sage Shuttle SH97R - using two thunderbolt outputs

Netgear ProSAFE Wireless-N 8-port Gigabit VPN Firewall (FVS318N)
(2) 150' cat5e cables
(2) PJ-Net Organizers (POA-PN03C)
(2) Sanyo PLC XP200L Projectors

 

 

Networking/Codes...

 

Router IP: 192.168.1.200

Production IP: 192.168.1.201

Display IP: 192.168.1.209

Projector 1 IP: 192.168.1.211

Projector 2 IP: 192.168.1.212

All have a subnet mask of 255.255.255.0

 

I am using a TDP connection to both projectors with the port number 4352.

 

My aux timelines send the following codes to both projectors simultaneously:

Power On: %1POWR 1$0D

Power Off: %1POWR 0$0D

Shutter Close: %1AVMT 11$0D

Shutter Open: %1AVMT 10$0D

 

 

As I said before, every so often one or more of these codes will fail to execute. This is sometimes accompanied by an error message in Watchout. Example...

 

From String Output "#ProjectorName" of Output List, 2015-10-02 08:53:16

Warning: Failed delivering data: %1AVMT 11$0D

 

Additionally, the shutter commands will occasionally send, but fail to execute at the same time (one will close or open its shutter a split second after the other). I'm not sure if these are two separate issues caused by two different things, or if there is one fix for both.

 

Any help is greatly appreciated. Thanks!

 

 - Rob

[Alex Ramos]

Hey,

 

Don't know if this is the case, but I had problems with TCP and UDP connection when sending more than one string simultaneously.

Placing the cues in different places on the time line helped.

Ex: Cue 1 @ 0.010 Cue 2 @ 0.050

[Mike Fahl]

As far as I understand, you're using TCP to send commands. That should guarantee that no commands are "missed", which can happen in the case of UDP (which is why it's sometimes referred to as "unreliable").

 

The message you're seeing "Failed delivering data" indicates a failure to connect to the device. Sending over TCP entails connecting, sending the data, and eventually disconnecting. WATCHOUT will disconnect automatically after about a minute of inactivity (assuming memory serves me correctly here). If it has disconnected, it will re-connect again if a new command is subsequently given. So it seems sometimes the device doesn't handle the connection attempt properly.

[cowboyclint]

I often send 2 commands in my aux timeline. 1 at .1sec, 1 at .5 sec or so. If one command wakes the connection, but doesn't execute, the other one should execute. Half a second late is better than not executing at all.

[Mike Fahl]

That spounds like a buggy implementation in the other end. They way WATCHOUT works here is:

 

1. If the connection is not established, initiate the connection.

2. Wait for the connection to succeed, getting a positive confirmation to that effect.

3. Send the command.

 

If the connection is already open, WATCHOUT goes straight to 3 and sends the command through the already open connection. If you need to repeat the command, it sounds like the device in the other end ignores the initial command. You may want to snoop the line to see if the first command returns an error or something like that (WATCHOUT won't deal with any data coming back, so you can't really tell from within WATCHOUT.

 

Hope this helps.

[matkeane]

I had similar experiences in a fixed installation sending TCP cues to a Crestron system a few years ago. The Watchout 4 setup used a Producer machine, rather than a Display cluster and we occasionally saw problems with missing cues. After some testing with Wireshark scanning for the commands sent to the Crestron we found two things:

 

Firstly, as Alex Ramos says, if two cues were sent at the exact same time from a timeline, sometimes only one would get triggered, so we staggered all commands by 150ms and that sorted that problem.

 

Secondly, after testing the show several times, it turned out that the Producer PC would sometimes skip cues in the timeline if it was under heavy load. This was a project with around 30 HD outputs and, when the Producer machine was set to 'Best' preview quality, cues would get skipped. It's not that the Crestron was ignoring cues; they never showed up in the Wireshark logs. With preview quality set to 'wireframe', or 'video as thumbnails', all cues were sent successfully. For some commands, like cowboyclint suggested, we doubled up cues with around 250ms delay to be sure they got through.

 

There was at least 10 minutes between the start/end of each show when commands were being fired, so I assume Watchout disconnected in between. Once a show ended though, a series of commands would be sent for lighting controls, door commands and machinery within a minute or so, but these were susceptible to the same problems.

[RobBergenstock]

Thank you all for your quick responses. I apologize for the delay in mine - had to handle some other projects.

 

As far as offsetting the commands by .1 second, I should have mentioned we are edge blending our two projectors for one full stage image. Offsetting the commands even by that small a margin looks too noticeable in the space. 

 

Mike, thank you for your info on Watchout's order of events. Out of curiosity, do the TCP commands ever execute from the display computer? In the past, we have run RS232 commands out of our display computers, but even if we disconnect our display computer from our router, we can send the TCP commands just from production. I assume that because the nature of networking is that everything is talking to each other, it would naturally work as long as any sending/receiving devices are connected. But I'm curious to know if our aux timeline cues placed in our main timeline send those commands out of the display computers. I hope that makes sense...

 

Since my original post, we went from our Netgear wireless router to a 10/100/1000 Mbps Gigabit switch. That improved the success rate up to an estimated 90%. This of course still leaves us with concerns over the other 10%.

 

On another point of note, we have connected a separate computer to our switch to interface with our PJ Link devices directly. When we send shutter commands through that, it works every time. It seems like the core difference there is that the PJ Link interface isn't ever closing that TCP connection to the devices. What is the reasoning behind Watchout closing and reestablishing the TCP connections after a given period of inactivity?

[Mike Fahl]

The TCP commands (String output) will be sent from the cluster master, which is the production computer when in use, or the primary display computer when youäre not using the production computer. The connection is closed after a period of inactivity. Closing it and then re-opening it on demand was deemed more robust than keeping a port (possibly with zero traffic) open indefinitely.

[bgarrett]

Hello,

I have been working with Rob on this issue. We have seemed to figure out a work around with the projectors not shuttering on a consistent basis through TCP/IP and PJ Link. We figured out that if we send a command moments or seconds after the port has closed the projector doesn't seem to receive the command. Watchout will give us an error. We figured this out with Wireshark. So we set up an Aux timeline that sends a carriage return every 35 seconds to the projectors. That keeps the port open and we seemed to have good results. The issue that we are still having is that sometimes the projectors don't always shutter at the same time and sometimes they do. In the theater we need them to shutter at the same time during blackouts. I have been looking at serial over IP devices. Do you think this will be more reliable than what we have going with PJ Link? If I have one quad output server can I assign four projectors their own com port and be able to control them individually? I thought Watchout can only be used on Com port 1 and it looks like the serial over IP devices take up additional comm ports. If someone has a good suggestion on what brand and type of serial to IP converters to purchase for controlling projectors, I would appreciate it.

Thanks,

Brent

[RobBergenstock]

Update:

 

 

At the recommendation of some, we installed Wireshark to capture/analyze the network traffic between our Watchout cluster and our PJ-Link projectors. We found that the failure to send commands occurs when those commands are attempted around the time that the Watchout/PJ-Link connection is severed (seems to be a 3-10 second window). For clarification, here is an example of a successful attempt, followed by a failed attempt...

 

Success:

Shutter Close command sent

Wait either less than 50 seconds, or more than 1 minute 10 seconds

Shutter Open command sent

 

Failure:

Shutter Close command sent

Wait between 50 seconds and 1 minute 10 seconds

Shutter Open command failed

 

 

The above is a general approximation, but based on those times, we were able to determine when to expect those commands to fail. On another note, we have tried the same concept with a Panasonic PT-EX600 (with internal PJLink protocol), and it closes its connection after only 30 seconds. 

 

Currently, our solution to either case is to loop an auxiliary timeline with an empty carriage return to each projector. Our loop runs about every 15 seconds. This seems to keep all connections (including our Panasonic projector) open instead of risking that a command be sent during the previously mentioned time. We feel this is currently our best option, though perhaps there is something we haven't thought of. If there is anyone that sees any potential issues with this (or as always, if there are any questions), please let us know.

 

Thank you.