Watchpoint startup issue Windows 10

[DVS]

Can anybody advise on the below please:

Motherboard: Asus WS X299 Pro
CPU: Intel Core i7 7800X, S 2066, Skylake-X, 6 Core, 12 Thread
Memory: Corsair Vengeance LPX Black 16GB (4x4GB) DDR4 PC4-19200 (2400)
O/S Drive: Samsung SM961 Polaris M.2, 256GB NVMe, MLC V-NAND, SSD PCIe
Watchout Drive: Samsung 850 Pro 512GB 2.5" SSD, SATA III
GPU: AMD 16GB WX9100 Radeon Pro, PCIe 3.0, GDDR5
DATAPATH VISION SC-SDI4 VIDEO CAPTURE CARD
MS WINDOWS 10 PRO 64 Bit

Windows has been stripped back to recommended setup following Dataton instructions.
MS Windows Updated to 1709 (Build 16299.309) as of 27^th March 2018.
Latest AMD Driver as of 27^th March 2018 (V.18.Q1.1).  Still current.
Running Watchout 6.2.2.  Previously running 6.2.1
6x mDP to HDMI adaptors (Startech) to 1080P monitors running at 60Hz.
Forced EDID has been used.

Issue:  When attempting to start using Watchpoint.exe on display systems, it will attempt to load and then fail, which will then restart via the watchdog and continue to go through the cycle but fail to start.
When running WP.exe, this will take around 18 seconds to start but will actually start. 

Adding GPU scaling to each screen via Radeon Pro settings cuts load time to around 12 seconds and after 4 cycles through the watchdog, will then start the show when run via Watchpoint.

When running 4 outputs, the screens seem to initialise quick enough under Watchpoint without causing the watchdog to kick in and restart the programme, but as soon as we go to 5 or 6 outputs, the time taken to initialise is too long and therefore, can cause the watchdog to restart the programme.

 This is happening on multiple systems of identical specification. The start-up is consistent and repeatable, as well as being identical between the two systems making a hardware fault very unlikely.

Additionally, using different screen edids appear to impact the initialisation time, one edid with very flexible settings ( lots of colour depths, resolutions and refresh rates available ) took almost 45 seconds to initialise running WP.exe, subsequently putting watchpoint.exe into a restart loop.

We also tried the gaming driver variant which is available to the WX9100 and this made little to no difference to the initialisation times for WP.exe.

We found a forum post indicating that the TdrDelay and TdrDdiDelay registry entries be increased to prevent Windows from killing the graphics drivers too soon. However these registry entries were not present to edit, and adding them had no effect.

This has also been seen on multiple other systems running more than 4 outputs at any one time and also on systems running W600 GPU, both at 1080 and 4K.

Is there any way to resolve this and if not, is there something that can be done to extend the timeout of the watchdog so that everything has time to initialise?

Edited May 3, 2018 by DVS
Change of description of GPU

[zackboyd]

Just checking - do you have anything in the auto-start script?

[jfk]

11 hours ago, DVS said:

... GPU: AMD 16GB WX9100 Radeon Pro, Pro WX 7100, PCIe 3.0, GDDR5 ...

Are there really two GPUs in the WATCHOUT Display computer? 

[DVS]

9 hours ago, jfk said:

Are there really two GPUs in the WATCHOUT Display computer? 

Our apologies, no, it's just the one.  Not sure how the WX7100 got in there.  To confirm, it is just the WX9100 in the system.  Original post edited to reflect.

[DVS]

10 hours ago, zackboyd said:

Just checking - do you have anything in the auto-start script?

Nothing in the auto-start script.

[cowboyclint]

I can second this. I had recent show with a very similar, but not identical piece of hardware utilizing 5 outputs. We would see this daily. About 4-6 restarts by Watchdog before the Watchout loaded properly. We didn't have time to trouble shoot this issue, but I can confirm this is not just your rigs.

[MISTERK]

same problem for me

too much issues with win 10

go back to win 7 x 64, i did and it's a loooooooooooot better

[DVS]

5 hours ago, MISTERK said:

same problem for me

too much issues with win 10

go back to win 7 x 64, i did and it's a loooooooooooot better

We would agree 100% if new Windows 7 licenses were not so difficult to purchase and the drivers for newer hardware was available for 7.  Trouble is with Windows 7 was at the time of Win 8.1, Win 7 was having performance issues in playing back 1200 Projection media and while building systems for Expo 2015 in Milan, we worked with some of the guys from Dataton to achieve a stable playback across 10 outputs through 3 systems on what is now version 6.

Unfortunately, time moves on and so does technology.

[DVS]

11 hours ago, cowboyclint said:

I can second this. I had recent show with a very similar, but not identical piece of hardware utilizing 5 outputs. We would see this daily. About 4-6 restarts by Watchdog before the Watchout loaded properly. We didn't have time to trouble shoot this issue, but I can confirm this is not just your rigs.

Glad we are not the only ones.  For people who have not used Watchout before they may look at this and think it is normal and this is how it works, but for experienced users, we know this is an issue.

[Miro]

Hi, we have been running Windows 10 servers with WX 9100 for a long time and I haven't experienced this kind of issues.

There are known issues with AMD's 2018 drivers and we are running the 2017 Q4.1 drivers. I wouldn't recommend using Windows 10 Pro. It's also important to setup and configure the server in a offline mode in order to get control over which updates and drivers that will get installed. Use Microsoft Update Catalog to download necessary updates prior to installation. Here are the combinations of OS + updates that we use.

• Windows x64 Enterprise 1607 LTSB + update KB4057142
• Windows x64 Enterprise 1709 + update KB4090913 

Microsoft has both solved and introduced DirectX bugs and these updates are important to get correct functionality.

//Miro

[DVS]

1 hour ago, Miro said:

Hi, we have been running Windows 10 servers with WX 9100 for a long time and I haven't experienced this kind of issues.

There are known issues with AMD's 2018 drivers and we are running the 2017 Q4.1 drivers. I wouldn't recommend using Windows 10 Pro. It's also important to setup and configure the server in a offline mode in order to get control over which updates and drivers that will get installed. Use Microsoft Update Catalog to download necessary updates prior to installation. Here are the combinations of OS + updates that we use.

• Windows x64 Enterprise 1607 LTSB + update KB4057142
• Windows x64 Enterprise 1709 + update KB4090913 

Microsoft has both solved and introduced DirectX bugs and these updates are important to get correct functionality.

//Miro

Hello Miro,

Thank you for the information.  Do AMD's driver issues relate to DirectX bugs again?

We will attempt to move to the stated version and updates based on the information you have provided.  Can you guarantee this will rectify the issues we are seeing?

DVS

[cowboyclint]

Hi Miro,

That information about Win 10 is very useful. Do you think we could get a Win10  pinned post at the top of the forum that includes all the useful bits and pieces from other posts until we finally get a tweaking list?

[Miro]

DVS, some DirectX issues comes from Microsoft where some versions of Windows have issues to initiate multiple DirectX fullscreen windows. However this problem is solved by using an appropriate windows update. Installing the latest updates it's not always a good idea because sometimes new DirectX bugs are introduced. We verify a base configuration that works and are keeping that constant until we discover issues that needs to be addressed.

There are other fixes that can be applied to make the system more stable and reliable. For example some motherboards like variants of X99 are causing interference in audio when the GPU changes it's link speed. This is most noticeable for AMD cards since they constantly changes their link speed and frequency. It's possible to force the link speed to maximum by using the following registry entry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\4D36E968-E325-11CE-BFC1-08002BE10318\0000 There you can create a DWORD key named ForcePcieLinkSpeed and set it to 3. If your system had a different GPU previously then you will have several sub-trees like 0001, 0002 etc.. Then you can create and set the ForcePcieLinkSpeed in all those sub-trees. I had issues with early 2018 drivers from AMD where the driver would cause a blue screen when starting windows. The driver from late 2017 mentioned above works fine. You can get it here: https://support.amd.com/en-us/download/workstation/previous/detail?os=Windows 10 - 64&rev=17.Q4.1#pro-driver

If using the synchronization provided by a S400 card (or using a nvidia card in combinations with a sync card) you might run into issues where windows will kill the GPU driver when all displays are initialized and synchronized. Especially if you have many displays. This is not really related to windows 10 but is good to know about. By default if a driver doesn't respond within 2 seconds windows will interfere and in this case it will end up with a bluescreen. This behavior can be changed in the registry and you can set a different timeout.

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers TdrDelay should be increased to 15 seconds (default is 2 seconds, decimal value DWORD)
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers TdrDdiDelay should be increased to 60 seconds (default is 5 seconds, decimal value DWORD)

Setting up hi-end systems usually requires hi-end skills and isn't really related to WATCHOUT but any other visualization software.

Other issues that may occur in Windows is that if an application steels focus from WATCHOUT it might interfere with the fullscreen state. Make sure to disable all notifications and any conflicting applications. Best advice is to kill explorer.exe which is handling the desktop, taskbar and a lot of other things that you don't need when running a Media Server.

//Miro 

 

[DVS]

On 5/8/2018 at 2:08 PM, Miro said:

There are known issues with AMD's 2018 drivers and we are running the 2017 Q4.1 drivers. I wouldn't recommend using Windows 10 Pro. It's also important to setup and configure the server in a offline mode in order to get control over which updates and drivers that will get installed. Use Microsoft Update Catalog to download necessary updates prior to installation. Here are the combinations of OS + updates that we use.

• Windows x64 Enterprise 1607 LTSB + update KB4057142

Hello Miro,

Based on the information above, we have loaded a system as detailed and there does appear to be a dramatic improvement in the startup time, something more akin to how it run under Windows 7.  So all good under Enterprise LTSB

As an after thought, we are attempting to change certain aspects of Windows 10 Pro installation to see if the startup time can be improved.
It would appear that the latest AMD driver is allowing the system to start quicker than the old Q4.1 driver under the Pro installation.

We shall post our findings if we hit on a particular service on Pro that might improve startup time for 5+ outputs.

Although it would potentially mean stripping back most of the code, do Dataton plan to re-write Watchout to support latest DirectX versions and video card drivers rather than relying on Microsoft/AMD/NVidia to bring things into line so that Watchout works reliably under current mainstream Windows versions?  Basically, we will see an overhaul of Watchout or will it just be the usual additions and updates we normally see?

[DVS]

Has anybody performed testing using Windows 10 Pro for Workstations?

[Thomas Leong]

On 6/11/2018 at 7:59 PM, DVS said:

Has anybody performed testing using Windows 10 Pro for Workstations?

With W10 Pro OEM version, stick with update 1607 (Anniversary Update). Watchpoint has startup problems with Update 1709. IMO, Update 1709 does not seem to have any additional advantage for multi-display use. Seems to be more for gaming, VR, netcentric apps, etc....things we do not use.

Only anomaly I had with an ASUS X99WS i7-6850K workstation build with Win 10 Update 1607 and a Radeon Pro W9100 was when 5 or more displays are connected. No problems with 4 displays. The Display numbering with both Windows and Watchpoint would jump from 1,2,3,4,5,6 as connected top-down, becoming 1,2,5,6,3,4. Nothing that could not be solved with re-assigning Stage Displays in Watchpoint, but a hassle nonetheless. Can't recall the Radeon Pro driver numbers, but they were both the first Radon Pro driver and the first Radeon Pro Enterprise driver versions released after the WX9100 came  to market.

Thomas Leong

[Ali Serdar Gulseren]

On 5/10/2018 at 8:52 AM, Miro said:

DVS, some DirectX issues comes from Microsoft where some versions of Windows have issues to initiate multiple DirectX fullscreen windows. However this problem is solved by using an appropriate windows update. Installing the latest updates it's not always a good idea because sometimes new DirectX bugs are introduced. We verify a base configuration that works and are keeping that constant until we discover issues that needs to be addressed.

There are other fixes that can be applied to make the system more stable and reliable. For example some motherboards like variants of X99 are causing interference in audio when the GPU changes it's link speed. This is most noticeable for AMD cards since they constantly changes their link speed and frequency. It's possible to force the link speed to maximum by using the following registry entry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\4D36E968-E325-11CE-BFC1-08002BE10318\0000 There you can create a DWORD key named ForcePcieLinkSpeed and set it to 3. If your system had a different GPU previously then you will have several sub-trees like 0001, 0002 etc.. Then you can create and set the ForcePcieLinkSpeed in all those sub-trees. I had issues with early 2018 drivers from AMD where the driver would cause a blue screen when starting windows. The driver from late 2017 mentioned above works fine. You can get it here: https://support.amd.com/en-us/download/workstation/previous/detail?os=Windows 10 - 64&rev=17.Q4.1#pro-driver

If using the synchronization provided by a S400 card (or using a nvidia card in combinations with a sync card) you might run into issues where windows will kill the GPU driver when all displays are initialized and synchronized. Especially if you have many displays. This is not really related to windows 10 but is good to know about. By default if a driver doesn't respond within 2 seconds windows will interfere and in this case it will end up with a bluescreen. This behavior can be changed in the registry and you can set a different timeout.

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers TdrDelay should be increased to 15 seconds (default is 2 seconds, decimal value DWORD)
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers TdrDdiDelay should be increased to 60 seconds (default is 5 seconds, decimal value DWORD)

Setting up hi-end systems usually requires hi-end skills and isn't really related to WATCHOUT but any other visualization software.

Other issues that may occur in Windows is that if an application steels focus from WATCHOUT it might interfere with the fullscreen state. Make sure to disable all notifications and any conflicting applications. Best advice is to kill explorer.exe which is handling the desktop, taskbar and a lot of other things that you don't need when running a Media Server.

//Miro 

 

Hello Miro

How exactly do you kill the explorer.exe?

[DVS]

22 hours ago, Ali Serdar Gulseren said:

Hello Miro

How exactly do you kill the explorer.exe?

Hello Ali,

We normally use a batch file in startup using a taskkill command for explorer.  That would be our preference.

In that batch file, we would then have the Watchout startup on a delay, giving time for Windows to kill explorer and for all drivers etc. to initialise.

Hope this helps.