JJ Myers

Hi there. I think our firm, Pixel Mosaic, may be right up your alley: www.pixelmosaic.com We are located in Chicago, so Toronto is within fair proximity. Feel free to contact me off forum. You can reach myself and our general inquiry staff at info@pixelmosaic.com.

Glad to see more attention and information to this issue. Has anyone in this discussion had this happen on a version earlier than v6.2.2? That is the only version we have seen this happen on, so my suspicion is v6.2.2 is when this became an issue. My other suspicion after reading this is that 2 different variables can potentially factor in the invocation of this bug: 1. Environmental factors, like what it appears to have been the case for asmundboe 2. Naming your displays via ip address instead of name, which appears to be the culprit in our case There are already several noted bugs involved with defining displays by ip address, so that is an easy one to just avoid from now on. Obviously, we may ultimately find that our experience was also environmentally influenced, but we had a pretty clear cut test case/project that we would run defined by ip address and hit the bug at least once per day. There were even events in the Windows Events log that correlated with the time of the bug invocation. We would then run the same exact test project, except defined by computer names - and find it to run for several days straight... and never invoke that bug. Ever since we had discovered issues surrounding ip address display definition, I have been a pretty strong advocate of getting the word out there. It takes a minute or 2 to re-define your displays by computer name and can avoid so many potential issues. It really is a no-brainer.

Hi, I believe we have encountered what you are describing as well. Although, it appears to me less like Watchpoint is minimized and more like it simply no longer displays anything at all. It looks more like Windows has assigned focus to something else. Our findings have been that from a task manager standpoint, Watchpoint thinks it is running completely as normal. Yet we see the desktop and no evidence that Watchpoint is running (other than task manager). If we go to Task Manager and click on "switch to" it restores back to displaying content. We also do not receive any notifications in Watchmaker, but Windows Event Log does report some information about Watchpoint no longer being responsive (or something along those lines). Does that match your experience? I would think if Watchpoint is truly "minimizing", you would indeed get some sort of feedback to your production machine, because that would mean Watchpoint has switched to "windowed mode". If our issues are indeed the same, I can tell you we have witnessed your issue in a project void of NDI, virtual displays, and projection on 3D objects. Recently, we have found issues of ours to resolve when we define our displays using computer names instead of IP address. Here is the post that led to this discovery: Ever since we redefined our displays via computer name, we have not encountered the "disappearance" issue again. We are running tests 24/7 at our office in an effort to find more absolution to our findings. I would say I am 99.999% certain to our findings as it relates to the "latency" issue. We had a pretty decently reproducible test case for the "disappearance" issue as well that we could no longer get to invoke after making the display definition edit. But our the reproducibility of that test case was still much less than what we had for the "latency" issue, so I cannot say I am as certain. We, of course, will continue to run tests for as long as we can. So my final question to you is - are you defining your displays via IP address or computer name? If so, perhaps make that edit and see if the issue resolves? Doing so will certainly help the knowledge base and I can help make sure the word on this gets out to users.

Ring the bells! We believe we may have found the culprit! Or at least, we have found something that prevents the test project from reproducing the bug. The test project in my post on October 2 uses IP address definition for 2D displays. I revised my project in the following way: 1. Set computer names and a cluster name for the 2 displays I have online 2. Set the cluster name in preferences 3. Define all my 2D displays by computer name instead of IP address I would have never thought such a revision would have resolved this issue, but alas. I would - by no means - suggest that our results are conclusive, as they only have solved the reproducible case. For us to put our rubber stamp on this, we are going to run further prolonged testing to see if the latency issue is completely absolved. For us, that means several days of typical operator cueing. I would love to hear back from other users as to their findings between IP address defined displays versus computer name defined displays as it relates to this bug. And let's please spread the word! If our efforts can prevent even just one WO show from having a catastrophe, the effort will have been worth it. So please by all means - pass it on!!

It looks like spaces in show file names no longer work when trying to launch from script (file-based control). I swear it used to once upon a time. Is there an escape sequence one can use in the script to force a space character (ASCI 32, I believe)? Or does file-based control require a file name without spaces? Sorry if there is already a topic on this - I couldn't find one.

Hey Jim - you probably never were alerted to this because I don't believe I properly quoted you in my above response. Just looking for some absolute clarification in the syntax for the command line argument. If you could post answers to my questions regarding the use of spaces and quotes that would be great. Thx!

Hey jfk, Can you clarify the syntax for that command line argument? You wrote "- WmAudioRegFix off". That exact string has a space between the dash and "W"... there shouldn't be a space, correct? Also, does the word "off" need to be encapsulated in quotes, or is just space then off? Thx!

So, we believe we have discovered more information relating to this phenomenon. We - at the very least - have something somewhat reproducible. The WO project for the reproducible case is at this link: https://www.dropbox.com/s/i6f12ixyzwglfde/Latency Bug.zip?dl=0 We also recorded a video that presents how to reproduce: https://www.dropbox.com/s/krhh85m1e9z8b3c/IMG_2142.mov?dl=0 Here are our specs and setup for this test: Display machines: - Windows 7 64b Professional - ASRock X99 WS-E LGA 2011-v3 Intel X99 SATA 6Gb/s USB 3.0 Extended ATX Intel Motherboard - Intel Core i7-5930K Haswell-E 6-Core 3.5 GHz LGA 2011-v3 140W BX80648I75930K Desktop Processor - SanDisk SSD PLUS 2.5" 120GB SATA III Internal Solid State Drive (SSD) SDSSDA-120G-G25 (OS) - SanDisk Ultra II 2.5" 480GB SATA III Internal Solid State Drive (SSD) SDSSDHII-480G-G25 x 4 (Media, Hardware RAID 0) - MD FirePro W7100 100-505724 8GB 256-bit GDDR5 PCI Express 3.0 x16 Full height/full length single-slot Workstation Video Card - ATI S400 sync card Production machine: - Windows 10 Professional - Latest 8-core Mac Pro In the video, we had 2 displays running 2 feeds each, all genlock synchronized via an NTSC composite signal. One machine is supplying Feeds 01 and 04, and the other was supplying Feeds 02 and 03. We set up both machines to sync from genlock (house sync) in this particular video. After recording this video, I remembered a discussion where the official recommended approach was to set the assigned master to sync from genlock and then use a cat5 connections on the master s400 card to serve as the synchronization master to the slaved machine. Just for clarification, here are screen shots of the settings for that synchronization method: https://www.dropbox.com/sh/vvvww8eos3trnlq/AABdhXbxj6v1SbiGVfFTvppda?dl=0 We ran this same test with that setup and experienced better results (an increase in stability). We had a much more difficult time reproducing what you see in the video, but none-the-less we were able to get it to occur under these conditions as well. We currently have a system running 24/7 with automated cueing in this set up so we can monitor prolonged usage under various types of synchronization and on various versions of WO. We also ran this test on a system not using genlock, rather using Feed 01's display as master timing source in Firepro, and then a cat5 connections on the master s400 card to serve as the synchronization master to the slaved machine. Same results of instability and latency as you see in the video. Here are the firepro settings for this synchronization method: https://www.dropbox.com/sh/uxrynsjauhdpbb2/AAAFdbS4OYRI6w6xrkDuPYQUa?dl=0 I want to point out that we not only experienced cases where both machines were equally latent (as seen in the video), but cases where the latency between the 2 machines was unequal... which of course is a far worse predicament to be in on a show. What we have never encountered is a scenario where individual feeds from a given machine to be unequally latent in relation to each other. We also - at this point - can rule out 2nd nic and/or virtual displays being a culprit in these particular instances, as both were factored out of these tests. I am going to supply this test project to dataton support and hopefully get some answers to what we are encountering. I would love to have other members of the WO community run this same test and get feedback as to whether they experience the same results. Perhaps something can ultimately be traced down to a piece of hardware, tweak setting, or the like. Our current plan is to run these same tests on rollbacked versions until we get back to a level of stabliity we can trust. Once we get to that point, we are going to set that as our version to use for future shows until a new release remedies the problems we are encountering. We otherwise do not feel comfortable with stability in v6.2.2 (regardless of method for synchronization). And we have encountered this latency bug in v6.1.6, so it is possible we will roll back all the way to v6.0.2. We will of course keep users informed of our results every step of the way.

We have experienced similar problems. Some things we have encountered, I have only seen since switching to v6.2.2. We are currently running tests to try to determine what behavior is causing some new, odd behavior. The most perplexing thing I recently experienced is I had a display machine, in which had been sitting completely idle in a static image state (JPEG and PNG images) for 15 minutes, suddenly gave away focus to the desktop. No report to production at all, because it seemed as if WP was completely unaware that focus had been taken away. AFAICT - from reviewing things like task manager, WP still thought it was in full screen, full system focus mode. I essentially had to click "Switch To" from task manager to restore full screen WP. We are now back at our shop re-evaluating the show file and are experiencing behavior similar to what you are describing... which I would not necessarily say is limited to v6.2.2, but we certainly are finding easier ways to reproduce. I am hoping to package a reproducible case to send to support today.

Just a recommendation, coming from someone who recently experienced a great deal of confusion on why I was not able to sign in... Allow 2 options for sign in - screen name or email address. I wasn't even aware of what my current screen name was, because - well, I had not been using that to login and I had logged in for a long time (I had something set to remain logged in). And my keychain of course had nothing set for the site with my screen name. I had to click "forgot my password" and follow the email trail in order to sign in, and had to do that several times before I came across this discussion. And if you are giving a user the option to reset their password by the traditional "send notification to email on file" method anyways, why not just allow a user to log in with their email in the first place? For reasons... A - You are in essence setting the security bar at one's email anyways B - Login using email is about as standard as it comes these days... people are used to it

Hey Keith - we have recently been doing some testing with ffmpeg and have found that our output files display significant frame loss when decoded in conventional players (Quicktime) or even no decoding whatsoever (VLC), but however look great in WO... or at least on par with files that we otherwise produce out of a traditional After Effects render queue. Is that consistent with your findings? Or are your remarks in this thread regarding quality more along the lines of things like color reproduction? Just curious. If the quality concerns with ffmpeg is really just a challenge in terms of accurate preview outside of WO, it certainly seems like more of a solvable issue. As in, someone just needs to build a decent preview application for HAP files that we can provide clients and vendors.

Hey Walter. Many thanks for the information on your set up. Certainly helps on our journey regarding unveiling answers to this bug. Our next efforts are going to be full show reconstruction that completely eliminates the use of VDs. Our thought is if we go a week of continual usage in a non-VD show-like environment and do not invoke the bug, it gives us great plausibility that VDs are a factor. And then we perhaps take it a step further - single mapping instances of VDs. And so on.... Hey - not much more we can do when you are dealing with a highly consequential, hard-to-reproduce type of bug without having the benefit of extensive knowledge of what is occurring under the hood.

A simple means of taking a slice anywhere on the stage, and remapping it to another part of the stage, but without the computational cost and latency that virtual displays incur. By using the term "slicing", you would be delineating the 2 features while introducing a feature that uses a term that is familiar to media server experts. It would be fair to then restrict 'slicing' from the extended capabilities of virtual displays, like mapping to planes of a 3D mesh. Just looking for something that can take an area of the stage and duplicate to multiple other areas of the stage with the ability to scale. That is about 90% of what we end up using VDs for, but it currently sometimes comes at an unnecessary computational cost.

Another thought - perhaps WO should ultimately add a feature that subscribes to the popular industry term "slicing"? That is essentially what I end up using VDs for quite a bit. I totally understand there was a particular usage Dataton had in mind for VDs, and that users like myself creating alternate usages for them may not be what was intended. Many media servers have a slicing feature where latency is negligible. I will go ahead and make an official request in the requests topic.

So, we just discovered this today: https://www.dropbox.com/s/fksxu98ls771lni/Test 1 Capture.mp4?dl=0 What you are looking at is a WO project that contains 2x 2D Displays (-y area of stage) and 16x virtual displays (+y area of stage). The VDs are named "VD A" - "VD P". There is a single mov file in the project that we use to measure latency. It is a 60fps file that displays current frame. There is one cue of the file mapped directly to the rightmost 2D display (Display "2"). The other is mapped to "VD A". "VD A" then maps to "VD B", which maps to "VD C", and so on and so on, until "VD P" finally maps to Display "1". As you can see, there is about 7-8 frames of latency between Displays "1" and "2". That would suggest that each VD is adding around 1/2 frame of latency in this particular arrangement. What I find incredibly fascinating is that by simply changing the arrangement of 2D Displays and VDs on the stage, I get completely different results - both in the way Watchmaker behaves and the amount of latency between Displays "1" and "2" (2-3 frames), as seen in the 2nd test here: https://www.dropbox.com/s/kfsimgefz76nil4/Test 2 Capture.mp4?dl=0 To me, that would suggest the order in which the stage data is sampled in the application code (eg. L>R, T>B) impacts the result of latency. But I am just grasping at straws here not knowing exactly how everything works "under the hood". Suffice to say, this discovery has me re-thinking our current strategies and usage of VDs. Until now, we had found tremendously powerful and unique uses for them. Usage that provides very dynamic and randomly accessible manners to cue a show ; usage that allowed us to build convenient "virtual multi viewers" for other client and crew resources ; usage that allowed us to be incredibly efficient and effective in creating mappings that could be used like "presets", which also mitigated risk of programmatic errors. It would be great to get a response from Dataton here, so we can gain an understanding of how cascading VD mappings can impact things like latency. Sort of a vague open-ended invitation I know, but if we understood more of what is happening "under the hood", that would help influence our thinking when inventing creative ways to use these tools. Anyways - circling this back to this topic... The creation of the above 2 show files resulted in my inability to get the latency bug to rear its ugly head again over the last 2 days. And the reason I started exploring VD mappings as a potential contributor, I realized at one point that all the shows we have encountered the latency issue of which this topic was founded upon, were shows in which we were using VDs in similar fashion. Don't get me wrong - we would never cascade a cue through 16 VDs before reaching an ultimate display destination as its end point. However, we have had cases where we would map a cue through 2 VDs before its end point. And there is a common thread here: latency. So without knowing more "under the hood" information, I think it is entirely plausible that what is witnessed in the projects above is potentially related to the latency bug in which this topic addresses. My research will continue. I will post more findings as I come across. For anyone interested in exploring the projects above, here is a link to the entire package (including the captures of the tests): https://www.dropbox.com/sh/ktb3rk6f8q9029p/AABHjpQWh2UGz4D_TRwobIiWa?dl=0