FreeBSD driver for the OCP TAP Time Card

[John Hay]

Hi,

I have been working on a FreeBSD driver for the Open Compute Project (OCP)
Time Appliance Project (TAP) Time Card. The card consists of three main
parts, a clock module, a GNSS module and a FPGA module. The firmware for
the FPGA implements a counter that is synchronized to TAI using the GNSS
module and the clock module. The counter is implemented as two 32-bit
registers, seconds and nanoseconds, like struct timespec, and make that
available on the pci-e bus.

More about the Time Card is available at:
https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card

The driver / software consists of two main parts:

The kernel module timecard(4):
- Provides access to the card’s functionality,
- Implements a timecounter, “TimeCard”, using the timecounters(4) API,
- Make the PPS signal and timestamp available with the RFC 2783 Pulse Per
Second (PPS) API,
- Provides a timecard bus to which uart(4), iicbus(4) and spibus(4) drivers
can attach.

A daemon timecard(8):
- Provides a shm(28) driver for ntpd(8),
- Optionally synchronize the kernel time to the timecard(4) time,
- Discipline or train the clock on board of the TimeCard through the iic(4)
Clock interface.

With the above software and ntpd running and disciplining the kernel time,
the kernel time can stay within +-2ns according to ntpd. If neither ntpd
nor timecard(8) is set to discipline the kernel time, it will slowly drift
away because there is a small rounding error when converting from
timecounter nanoseconds read to bintime and then later to timespec. When
ntpd is set to discipline the kernel time and it has settled, ntpq’s
kerninfo command will report a pll frequency of 1.52588e-05 (ppm).

The card is Precision Time Protocol (PTP) capable, but the driver does not
support that.

The driver is available at:
https://github.com/JohnHay/TimeCard/

Regards

John

[Poul-Henning Kamp]

--------

Post by John Hay
I have been working on a FreeBSD driver for the Open Compute Project (OCP)
Time Appliance Project (TAP) Time Card. The card consists of three main
parts, a clock module, a GNSS module and a FPGA module. The firmware for
the FPGA implements a counter that is synchronized to TAI using the GNSS
module and the clock module. The counter is implemented as two 32-bit
registers, seconds and nanoseconds, like struct timespec, and make that
available on the pci-e bus.

That is /precisely/ the kind of hardware timecounters were designed for :-)
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
***@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[John Hay]

Hi Poul-Henning,

Post by Poul-Henning Kamp
--------

Post by John Hay
I have been working on a FreeBSD driver for the Open Compute Project

(OCP)

Post by John Hay
Time Appliance Project (TAP) Time Card. The card consists of three main
parts, a clock module, a GNSS module and a FPGA module. The firmware for
the FPGA implements a counter that is synchronized to TAI using the GNSS
module and the clock module. The counter is implemented as two 32-bit
registers, seconds and nanoseconds, like struct timespec, and make that
available on the pci-e bus.

That is /precisely/ the kind of hardware timecounters were designed for :-)

True, it did make it a lot easier. :-) Working close to the nanosecond does
seem to push things close to the limits though. :-)

There are a few things I wish could be handled differently. Maybe there are
ways, and I just did not find them. :-)

One is that you cannot just feed timecounters with a timespec value. It
assumes the hardware counter is in binary, while in this case, the
nanoseconds rolls over at 1 second, so for every tc_get_timecount(), you
have to multiply the seconds register value by 10^9 and then add the
nanosecond register value. Not a big deal and reading the registers over
the pci-e bus is the slowest part by far.

The other is that the conversion from the above value to bintime and later
back to what is used elsewhere, seems to lose a little precision. The
comments in sys/kern/kern_tc.c also note that.

In the pps code, I wished that one could provide a timestamp with
pps_capture(). It uses the time at which it handled the interrupt. The card
latch the counter values when the pps happens, so it knows the correct
time. Currently I hack around it by twiddling sc->sc_pps_state.capcount
directly.

Regards

John

[John Hay]

Hi Poul-Henning,

Post by Poul-Henning Kamp
--------

Post by John Hay
The other is that the conversion from the above value to bintime and

later

Post by John Hay
back to what is used elsewhere, seems to lose a little precision. The
comments in sys/kern/kern_tc.c also note that.

Yes once the /resolution/ of the hardware gets into the nanosecond
domain, some of that resolution is lost, because a 64x64=>128
multiplication would have been both surplus to requirements and
beyond the pale.

Yes, resolution is what I meant.

Post by Poul-Henning Kamp
Getting rid of 32bit platforms moves the needle, but it may still
not be warranted.
I will also note that most people who think they are approaching
nanosecond /precision/ are wrong about that: Only a few of the
institutions listed in Circular T manage it.
The important thing is to make sure the added noise is bias free.

I have 3 machines with Time Cards installed. If I use the TimeCard as
timecounter and do not discipline the kernel time, it will slowly drift
away. If I use ntpd to discipline it, all three machines settle on a pll
frequency of 1.52588e-05 (ppm) according to ntpq -c kerninfo:

<snip>

Post by Poul-Henning Kamp
ntpq -c kerninfo

associd=0 status=0415 leap_none, sync_uhf_radio, 1 event, clock_sync,
pll offset: 0
pll frequency: 1.52588e-05
</snip>

So just from that perspective, it feels that there is some bias.

Post by Poul-Henning Kamp
You should capture some millions of the the difference between the
HW counter and the hw->timecounter->bintime->timespec result, and
run that through the usual battery (Histogram, MVAR, FFT etc.)

I'll have a look at that.

Post by Poul-Henning Kamp

Post by John Hay
In the pps code, I wished that one could provide a timestamp with

pps_capture

Post by John Hay
(). It uses the time at which it handled the interrupt. The card latch

the

Post by John Hay
counter values when the pps happens, so it knows the correct time.

Currently I

Post by John Hay
hack around it by twiddling sc->sc_pps_state.capcount directly.

That is not hacking around, that is how it is done :-)
See for instance i386/i386/elan-mmcr.c

Ah, I forgot about that one and did not look there. :-) Thanks for the
reminder.

Regards

John

[Poul-Henning Kamp]

--------

Post by John Hay
The other is that the conversion from the above value to bintime and later
back to what is used elsewhere, seems to lose a little precision. The
comments in sys/kern/kern_tc.c also note that.

Yes once the /resolution/ of the hardware gets into the nanosecond
domain, some of that resolution is lost, because a 64x64=>128
multiplication would have been both surplus to requirements and
beyond the pale.

Getting rid of 32bit platforms moves the needle, but it may still
not be warranted.

I will also note that most people who think they are approaching
nanosecond /precision/ are wrong about that: Only a few of the
institutions listed in Circular T manage it.

The important thing is to make sure the added noise is bias free.

You should capture some millions of the the difference between the
HW counter and the hw->timecounter->bintime->timespec result, and
run that through the usual battery (Histogram, MVAR, FFT etc.)

Post by John Hay
In the pps code, I wished that one could provide a timestamp with pps_capture
(). It uses the time at which it handled the interrupt. The card latch the
counter values when the pps happens, so it knows the correct time. Currently I
hack around it by twiddling sc->sc_pps_state.capcount directly.

That is not hacking around, that is how it is done :-)

See for instance i386/i386/elan-mmcr.c
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
***@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de