Help with Windows 10 Watchout

[actseattle]

Ok, I need some advice.  We are not getting consistent video out of two of our machines.  Here is the build:

Three identical water-cooled rigs:

Asus Prime X299-A

6-core i7

16GB DDR4 RAM

FirePro W7100 quad-output card

Dual Samsung SSDs, one 850EVO boot drive, 960Pro M.2 Watchout data.

Monoprice SDI-HDMI and HDMI-SDI converters

Dedicated gigabit network.

Windows 10 professional 64-bit

The cards are not putting out video consistently.  I have one machine up and running by forcing EDID, but the other two are just bonkers.  The Epson laser phosphor projectors are flashing and Watchpoint won't go online.  We've nuked and reinstalled one of the machine from scratch.  All drivers up to date.  Nothing seems to work.  

 

We have started tech rehearsals with 4 of 11 outputs working, and it's a disaster.  This is the same rig I asked advice about re: VNC.

 

 

[Thomas Leong]

Check with Settings > Update & Security > Windows Update > the Update version installed on the PCs.

AFAIK, I have not had a problem with Watchpoint 6.1.6 and 6.2.1 starting and working under Update v1607 (Anniversary Update).

I have not tried / dare not try the two Creators' Updates - v1709 and one other. The last I tried one of those versions, Watchpoint would not start up, flashing Watchout logo non-stop, etc.

So if you can uninstall the updates, and revert to v1607, I am quite confident you will be fine with that version. For a quick up-and-go with v1607, there are not many tweaks necessary. The usual Firewall, Power options and manual IP address should suffice until you have more time to go through the Settings individually and disable/uninstall any that may interfere with the network communication between Prodn and Display.

good luck!

Thomas Leong

 

Edited March 7, 2018 by Thomas Leong

[actseattle]

11 hours ago, Thomas Leong said:

Check with Settings > Update & Security > Windows Update > the Update version installed on the PCs.

AFAIK, I have not had a problem with Watchpoint 6.1.6 and 6.2.1 starting and working under Update v1607 (Anniversary Update).

I have not tried / dare not try the two Creators' Updates - v1709 and one other. The last I tried one of those versions, Watchpoint would not start up, flashing Watchout logo non-stop, etc.

So if you can uninstall the updates, and revert to v1607, I am quite confident you will be fine with that version. For a quick up-and-go with v1607, there are not many tweaks necessary. The usual Firewall, manual IP address should suffice until you have more time to go through the Settings individually and disable/uninstall any that may interfere with the network communication between Prodn and Display.

good luck!

Thomas Leong

 

Thanks.  Since Windows 10 does not allow full uninstallation of those updates, we are in the process of erasing and reinstalling AGAIN.  We won't do any updates this time and see what happens.  We got the video card to see the SDI converters finally and now have hit the wall of Watchpoint refusing to go online at all.  No communication with the production machine at all.

 

Will keep you updated.

[Thomas Leong]

2 hours ago, actseattle said:

"...have hit the wall of Watchpoint refusing to go online at all.  No communication with the production machine at all."

Mmm...

1. Disabling Firewall on both Production and Display is an obvious step

2. Control Panel > Network and Internet > Network and Sharing Center > Advanced Sharing Settings > All Networks > Default is 'Turn on Password Protected Sharing'.

Change this to 'Turn off...' and Save

3. Not sure if this will help, but I usually add 'Everyone' to my Network, and Security settings and give 'Everyone' full read and write access.

4. Try a direct one-to-one connection between Production and Display.

[actseattle]

22 hours ago, Thomas Leong said:

Mmm...

1. Disabling Firewall on both Production and Display is an obvious step

2. Control Panel > Network and Internet > Network and Sharing Center > Advanced Sharing Settings > All Networks > Default is 'Turn on Password Protected Sharing'.

Change this to 'Turn off...' and Save

3. Not sure if this will help, but I usually add 'Everyone' to my Network, and Security settings and give 'Everyone' full read and write access.

4. Try a direct one-to-one connection between Production and Display.

We did a full nuke and reinstall, and then did not install any updates.  We are back in business.  Still no built-in VNC, and we've done all your steps.  But the machines are displaying video.

 

We are getting a lot of errors when going online, saying:

From Display "TV 3" of Stage Tier "Base" of Stagelist of Stage, Warning: Changed Frequency from 60z to 59hz.  

Any ideas on that one?  I'll poke at AMD FirePro settings and see if I can just lock it to 60hz somehow.  Everything still functions but going on/offline is very annoying with 15 errors cropping up every time.

 

Thanks for the response.

[Thomas Leong]

Good to hear you are up and running!

The message re frequency changing from 60 to 59Hz is, IMO, quite common. I get it too with a particular monitor I use. It is not a show stopper message, and Watchout will happily continue functioning. However, if the freq change occurs during a run, it can disrupt momentarily the playing of a video.

To solve, you could put a EDID Manager between the graphics card's output(s) and the monitor/projector, locked to 60Hz, with, say a Gefen DVI Doctor, or HDMI Doctor, one of the cheapest on the market. You will need one DVI/HDMI Doctor for each output that is giving you the message.

Or use the software EDID built into FirePro / Radeon Pro Control Centre. Copy and save one of the other monitors/projectors that do not change frequency, and use its EDID as the Emulator.

[actseattle]

23 hours ago, Thomas Leong said:

Good to hear you are up and running!

The message re frequency changing from 60 to 59Hz is, IMO, quite common. I get it too with a particular monitor I use. It is not a show stopper message, and Watchout will happily continue functioning. However, if the freq change occurs during a run, it can disrupt momentarily the playing of a video.

To solve, you could put a EDID Manager between the graphics card's output(s) and the monitor/projector, locked to 60Hz, with, say a Gefen DVI Doctor, or HDMI Doctor, one of the cheapest on the market. You will need one DVI/HDMI Doctor for each output that is giving you the message.

Or use the software EDID built into FirePro / Radeon Pro Control Centre. Copy and save one of the other monitors/projectors that do not change frequency, and use its EDID as the Emulator.

Yes, we are indeed using the FirePro control center to force EDID.

 

We had a trouble-free day yesterday and now machines are dropping offline randomly, Watchpoint flashing on and off for 5-10 minutes before it comes back connected.  It seems to be triggered by live updating files across the network..   Which we of course do often during tech.  I'm going to reboot the Production machine on the next break and see if that helps.

 

 

[wiesemann]

Hi, don't use live-update when in show-mode.

[Thomas Leong]

"Machines dropping offline randomly, Watchpoint flashing on and off for 5-10 minutes before it comes back connected..." is worrying.

Look back to the 'trouble-free day'. Was Live Update on in Watchmaker? Any settings changed in Watchmaker since then?

Is it possible to try with Live Update off for a day to see if the machines would behave in a more stable manner?

Check Task Scheduler for anything that Windows 10 has auto set to kick in at certain times.

Any ASUS Utilities for the motherboard installed? Uninstall them.

Check through My Computer > Manage > Services to Stop and Disable Windows Update, and anything that may look suspicious in disrupting network connection between Production and Display.

For remote vnc, you could use any laptop with Windows 7 as Production PC, even a cheap netbook. As long as it DOES NOT use the Windows Standard VGA driver, and uses the Intel graphics driver instead, Watchmaker will work, and allow remote access into a Display Windows 10.

Edited March 10, 2018 by Thomas Leong

[actseattle]

3 hours ago, Thomas Leong said:

"Machines dropping offline randomly, Watchpoint flashing on and off for 5-10 minutes before it comes back connected..." is worrying.

Look back to the 'trouble-free day'. Was Live Update on in Watchmaker? Any settings changed in Watchmaker since then?

Is it possible to try with Live Update off for a day to see if the machines would behave in a more stable manner?

Check Task Scheduler for anything that Windows 10 has auto set to kick in at certain times.

Check through My Computer > Manage > Services to Stop and Disable Windows Update, and anything that may look suspicious in disrupting network connection between Production and Display.

For remote vnc, you could use any laptop with Windows 7 as Production PC, even a cheap netbook. As long as it DOES NOT use the Windows Standard VGA driver, and uses the Intel graphics driver instead, Watchmaker will work, and allow remote access into a Display Windows 10.

Live Update was enabled on the 'trouble-free' day, yes.  

We are starting the day with Live Update disabled.  It took two hours of reboots and such to get the system to lock and display all outputs, and it seems stable.

I will do a full reinstall of the other two Display machines and the Production machine on Monday. I really wish there was some kind of tuning guide for Windows 10, even just from the community.  There is so much garbage in Windows 10 that one has to disable, or can't be disabled, that I simply don't trust it at all.  I wish I had bought older i7s so I could have used 7.  Does Dataton claim Windows 10 compatibility?

 

Thanks everyone!  I may just publish my own Windows 10 tuning guide if I can ever get this system show-ready.

[actseattle]

Is there any kind of useful log file that will shed some light on the Display machines dropping offline randomly?  They all came back eventually but it was ugly for a while.  Without some kind of record of what's going on it it like stabbing in the dark.

 

There is a log file in the application folder I hear, and will pull and inspect as soon as the rehearsal ends.  How detailed are Watchout logs?

 

 

[actseattle]

Digging into the logs show a lot of errors saying that the registry key for the audio fix cannot be enabled, but we aren't doing audio.  I'll ask the designer to make sure his files are stripped of audio.

Something interesting was:                      "WATCHPOINT" exited abnormally (code 3221225477), restarting

Running this code through an Excel file to get the Windows error (why does Dataton convert it to decimal??) give me C0000005.   This has something  to do with DEP,  or the following:

 

It is an Exception Error that occurs when a process (application, driver, etc.) tries to execute a processor command that references memory outside the space allocated to that process by the Operating System. 

 

I can attempt to totally disable DEP for Watchpoint and all that security stuff.   Community, anyone have experience with that error?

[actseattle]

Further update:  I have asked a board member as well as our IT department to get me four Windows 10 LTSB licenses.  If i can get them before mid-day tomorrow I will nuke all display and production machines and see what this does.

 

 

[screenshaper]

Why don't you downgrade to win7? (it should be free of charge if you own win10 licenses)

It is the safe and certain way!

[jochri]

From my experience I can unfortunately say notging but bad things about these Asus boards!

To make things better I recommend you the following Bios tweaks.

Turn off:

- Hardware Prefetcher

- Adjecent Line Prefetcher

- Intel Adaptive Thermal Monitor

- Advanced Intel Speed Step ( This is just for the older CPU Generation! With the new Xeon W and Xeon Scalable you need to turn it on)

- CPU C States

Turn on: CPU Turbo Mode

 

General things about Asus X299 Boards. Have a look at the PCI lanes. What is the exact configuration of your cards? How are the SSDs connected?

Do you watercool the m.2 card? This one will get into thermal throtteling very fast! I can run enterprise SSDs at 70 degrees for 24/7 but an m.2 is just rated at 40 degrees until it goes into thermal throtteling. Please stay away from m.2 disks. They are not meant to be used in such a scenario! Verified and tested from me: Intel P4500, Micron 9100 and for max speed Samsung PM 1725a (PCIe slot model for full 8x Lanes and 6200 MB/s) You can get consistend data rates for hours out of them but not from m.2 disks...

Read this:

https://www.pugetsystems.com/labs/articles/Samsung-950-Pro-M-2-Throttling-Analysis-776/

 

 

[actseattle]

6 hours ago, screenshaper said:

Why don't you downgrade to win7? (it should be free of charge if you own win10 licenses)

It is the safe and certain way!

With the newest Intel Corei7 chips you MUST run Windows 10.  

[actseattle]

5 hours ago, jochri said:

From my experience I can unfortunately say notging but bad things about these Asus boards!

To make things better I recommend you the following Bios tweaks.

Turn off:

- Hardware Prefetcher

- Adjecent Line Prefetcher

- Intel Adaptive Thermal Monitor

- Advanced Intel Speed Step ( This is just for the older CPU Generation! With the new Xeon W and Xeon Scalable you need to turn it on)

- CPU C States

Turn on: CPU Turbo Mode

 

General things about Asus X299 Boards. Have a look at the PCI lanes. What is the exact configuration of your cards? How are the SSDs connected?

Do you watercool the m.2 card? This one will get into thermal throtteling very fast! I can run enterprise SSDs at 70 degrees for 24/7 but an m.2 is just rated at 40 degrees until it goes into thermal throtteling. Please stay away from m.2 disks. They are not meant to be used in such a scenario! Verified and tested from me: Intel P4500, Micron 9100 and for max speed Samsung PM 1725a (PCIe slot model for full 8x Lanes and 6200 MB/s) You can get consistend data rates for hours out of them but not from m.2 disks...

Read this:

https://www.pugetsystems.com/labs/articles/Samsung-950-Pro-M-2-Throttling-Analysis-776/

 

 

Thanks for the detailed response.  I will poke at the BIOS settings.  I also have acquired Windows 10 N LTSB and will be installing on separate SSDs and installing to see if they are more stable.

 

The m.2 card has a massive heatsink and has not exhibited any throttling, the issues I am seeing on this system are Watchpoint crashing and rebooting constantly during edits in tech.  It seems a tiny bit more stable without Live Update enabled, but still goes haywire when making changes.

[Thomas Leong]

5 hours ago, actseattle said:

With the newest Intel Corei7 chips you MUST run Windows 10.  

You can install Win 7 with the newest i7's. Basically, the problem is that Win 7 does not have the USB3.0 drivers auto-included in the installation process, so when the install asks for Drivers/harddisks, early in the process, insert the motherboard CD and negotiate to Drivers looking for USB3.0xHC or something like that (memory like my height!). It is one of those low down on a long list. Try each one. If after OK'ing, the install process gives you the next step, you are good to go. Else, it kicks you back to the list so you can try another.

I've done it with a i7-7700 in  Z270 motherboard before.

Edited March 12, 2018 by Thomas Leong

[screenshaper]

16 hours ago, actseattle said:

With the newest Intel Corei7 chips you MUST run Windows 10.  

Are you sure about that?

We have successfully installed win7 on that motherboard.

[actseattle]

On 3/12/2018 at 2:13 PM, screenshaper said:

Are you sure about that?

We have successfully installed win7 on that motherboard.

Well, I'll pull one of the machines and give it a shot.  Windows 10 is such a pile of garbage.

[actseattle]

That's the last time I believe the system requirements listed for motherboards and processors.

Thanks to Screenshaper and Thomas Leong I attempted the install.  It worked, after about 4 hours of hammering at the USB 3.0 driver issue, then it finally installed.  I have a working machine that I can clone tomorrow and do a full test of all three.  The machine running Windows 7 works with remote access, connected and displayed first time, and generally ran as expected.

 

I don't think Dataton should claim Windows 10 compatibility until they release a tweak list.  I wasted 80 hours of load-in, tech and previews trying to get Watchout on Windows 10 to work effectively and it never did.  If anything LTSB fell over faster and never came back.  

[Guest DavidA]

Hi!

I am glad that you have solved your problems and are running now. And I am also not happy to see how much time you have spent getting it done. 

We have a lot of users successfully running windows 10 and we are about to release our own server running windows 10 - the challenge with Windows 10 is that it has required both from us and others to invest a LOT of time and resources on it to make it bullet proof.

And we are NOT ever going to recommend any kind of hardware to build media servers - as the pace that new components keep coming to market is so high, that any kind of recommendation would not be valid more than a mere 15 minutes... AND, we have our own pre-built WATCHOUT servers that would be our recommendation.

But there are lots of people in this forum who wants to share their experiences with different components. 

Have a great weekend. 

 

[Simone]

On 3/12/2018 at 10:54 AM, jochri said:

- Advanced Intel Speed Step ( This is just for the older CPU Generation! With the new Xeon W and Xeon Scalable you need to turn it on)

Hi, is it valid also with Windows 7? I cannot found information about Windows 7 and Advanced Intel Speed Step compatibility on Xeon-W processor..

[jochri]

Didnt tested with Win 7 so far but its just a Win 10 feature I guess.

What is your mainboard / CPU? You should be able to hand over the task to the cpu.

https://pureinfotech.com/windows-10-fall-update-enable-intel-speed-shift-skylake/

Never been happier than with the latest Win 10 LTSB update and AMD 2018Q2 drivers!

[Simone]

On 5/12/2018 at 1:39 PM, jochri said:

What is your mainboard / CPU? You should be able to hand over the task to the cpu.

Hi jochri, mainboard is Asus WS C422 PRO/SE and Xeon W-2145. I need to do more test but for now I prefer to keep turbo mode and Speed shift Technology disabled on Windows 7.

About Speed shift Technology I see a strange behavior on Win 7 with following combination in bios and High performance power plan in windows:
Enhanced SpeedStep :Enabled
Turbo mode: Enabled
Speed Shift: Disabled 

with these settings processor is always overclocked at 4300Mhz, this not happen on Windows 10 1607 LTSB with same setting where processor dynamically change freq starting form 3700Mhz.

I'm going to better test Windows 10 LTSB 1607, what kind of tweaks are required in your exprerience?