Doug Arnold, Principal Technologist at Meinberg and IEEE 1588 Working Group Chair, has recently written some blog posts about IEEE 1588 PTP changes that bring the standard closer to TimeKeeper’s fault tolerance approach.
One of the earliest design decisions we made for TimeKeeper was to escape the limitations of both NTP and PTP by allowing the core system to operate outside the protocols while retaining complete interoperability at the protocol level. Released in 2011, TimeKeeper 5.0 implemented a “Multi-Source” time engine that monitors time from any number of clock sources (PTP or NTP or otherwise) in parallel and fails over from the primary down the list of sources. Later in 2011, TimeKeeper 5.2 was the first production version that added “Source-Check”, which dynamically scored multiple sources to detect compromise or failure and force failover. You can see a more limited version of the same idea in the diagram below that Doug has for the revised standard approach to robustness – with the box for “Robust clock” under the domains.
Over the last 8 years, the failover and error detection component in TimeKeeper has been proved out and improved in production systems – ranging from high performance, dedicated timing networks to clouds and hybrid cloud.
When we began telling people about our enhanced “Multi-Source/Source-Check” work, there was a lot of skepticism from the PTP community – where “Best Master Clock” (or BMC) algorithm was considered a solution to fault tolerance rather than a dangerous single-point-of-failure, which is how it operates in enterprise networks. It is good to see our TimeKeeper’s fault tolerance approach endorsed in the 1588 Standard, although implementing it is going to be difficult for competitors (we are willing to license our IP though, so let us know).
As Doug points out, this basic multi-source technology (“multi-domain” is what it is called in the PTP revision) also enables a number of other solutions. One common use in TimeKeeper is to make grandmasters (or boundary clocks or a mix) fault tolerant by cross connecting them. Suppose you have a TimeKeeper Grandmaster at two locations in London with a low latency cross connect (or any symmetric cross connect). Then each one can independently derive time from GNSS (Galileo or GPS or both) and also send time over NTP or PTP-telecom profile to the other one. If the GNSS reception fails on one, it can failover to the time from the other one. And each will independently monitor the other and alert if the other appears to be drifting off the real time. This advanced feature was very useful for our customers in London LD4, who were able to compensate for rooftop GPS/GNSS interference (and diagnose it using our SkyMap technology).
Another use, also noted by Doug, is to provide profile translation – although Timekeeper can also translate between NTP and PTP in either direction. For example, a clock source using PTP-telecom profile or NTP (point-to-point) can be translated to multicast for an isolated network. A firm that is incrementally changing from NTP to PTP can also use this feature to switch protocols dynamically. TimeKeeper Server instances (stand alone or within TimeKeeper GMs) can serve time over multiple network ports in any combination of protocols.
Many of our customers use this same feature to compensate for slow network speeds on legacy network clocks. A 1Gbps PTP feed from a slower network clock can be provided to a server computer running TimeKeeper Server and this system can serve PTP and NTP at, say, 25Gbps, over its own network ports.
The IEEE 1588 committee is making some advances. TimeKeeper customers have been able to see the benefits of this approach in production for 8 years – and are reaping the benefits of further advances that we have produced since then.
For comparison, see the TimeKeeper 5.0 press release from 2011.
TimeKeeper 5.0 offers the ability to monitor multiple time distribution channels, even those operating on different time distribution standards or of different quality due to distance or network issues. As an example, a TimeKeeper client may monitor two different Precision Time Protocol (PTP) “master clocks” and three different Network Time Protocol (NTP) servers. In addition, if the time quality of TimeKeeper’s primary sources becomes questionable, TimeKeeper can now switch from tracking one time source to another, according to a fail-over list provided at configuration time.
TimeKeeper produces a detailed log for each monitored time source including data on the dynamics of the internal time synchronization model. This data-rich log can be used both for documenting system state and for multi-source analysis that can expose failing time sources. Under development now, the coming TimeKeeper 5.1 will include software for automatically switching from compromised time sources based on security analysis.