Try building a task timeline with your live input. Have the timeline paused on the live input. Then go to Layer settings, General: Perform: In Standby.
Leave that timeline running all the time. You will be able to fade in your live inputs imediately with no glitch, delayed start.