Updating the Logshield 1307 RTC with the NTP protocol.

Last Modified: November 30, 2013, at 05:03 AM
By: robtillaart
Platform: UNO

Please read - http://arduino.cc/forum/index.php/topic,52018.0.html - about not bugging the NTP-server admin's.

Comments and remarks please use the forum - http://arduino.cc/forum/index.php/topic,85764.0.html -

Related UDP overrun bug: - http://code.google.com/p/arduino/issues/detail?id=669 -

Intro

One of the main applications for the Arduino board is logging of sensor data. One of the finest shields recently in the market is the http://www.ladyada.net/make/logshield/ as it combines a Real Time Clock RTC and a SD storage. The RTC enables labeling the sensordata with an accurate timestamp so they make far more sense than when logged with a simple uptime counter. NB one can relate the sensor readings to external events, under the condition that the RTC is set correctly! This story is about setting the RTC within a one second precision by means of the NTP protocol.

Note: the sketch discussed is mainly merging existing code with some small additions to enhance the precision.

Note: works also with "standalone" DS1307 as long as the backup battery stays connected.

RTC ds1307

The logshield comes with a nice library, which contains three classes:

  • DateTime
  • RTC_1307
  • RTC_Millis - not used, (see warning in .h file)

The dateTime class has several methods for converting between date and time formats, especially UTC and human readable form.

The RTC_1307 has two important methods:

  • void RTC_DS1307::adjust(const DateTime &dt); // set time
  • DateTime RTC_DS1307::now(); // get time

The documentation states that one can intialize the RTC with two consts from the compiler. This can be done with the following code:

RTC_1307 RTC;
RTC.adjust(DateTime now (__DATE__, __TIME__));

Clever but the accuracy depends on the size of the sketch, the speed of your compiler and the baudrate of the upload. Although the difference is not big in absolute terms we want the RTC as precise as possible and there is where NTP comes in.

NTP

NTP stands for Network Time Protocol. This protocol provides superb accuracy from atomic clocks for setting time in computers. When requesting an NTP timestamp one gets a (theoretical) precision in the order of 10^-10 second. The NTP answer might include information to calculate the network latency and more. It is one of the fundaments of the internet. NTP uses UDP, a "brother" of TCP, as transport protocol. The main difference with TCP is lack of handshake so it is faster and in theorie less reliable. In practice this gained speed makes the protocol more accurate.

For details of NTP I refer to:

Explanation of the Code

This sketch assumes you have a logshield and an ethershield attached to your Arduino.

The code is quite straightforward. In SETUP it initializes the ethernetboard and opens an UPD socket for the NTP handshake. Also an RTC object is started for setting the clock.

In LOOP the main actions are sending a NTP packet to the timeserver and wait for its reply. If there isn't an answer available retry every 10 seconds.

When the reply is received, the code extracts the timestamp from the timeserver. This timestamp is in UTC format, meaning seconds after Jan 1st 1970. the timestamp also has comes with a 32 bit fractional part.

As the UTC timestamp refers to wintertime in Greenwhich, we need to Add/subtract the right amount of hours for our own TimeZone (TZ) and DaylightSavingsTime (DST). Furthermore the code uses the fractional part to round off the time ==> add 1 second if fraction is larger than 0.4. This value takes 100msec network latency (etc) into account.

Note that there is also 1 second added to compensate for the explicit delay after the NTP request has been send. Finally the RTC is set with the adjusted timestamp and the programs stops with an idle loop.

The code contains some Serial.print() statements to see what happens, but in fact they form a delay ... Furthermore the code also contains some lines to extract all other timestamps of the NTP packet that are not used. See the spec for their meaning. Also note that the code only extract 2 of the 4 bytes of the fractional parts as this is more than enough to be precise within 1 second. A final version all this unneeded code can be stripped.

Results

In the first version I did not have the one second adjustment and the round off code with the fractional part. After adding both adjustments the code does its job very well, this can be seen by running the sketch twice and compare the RTC with the incoming NTP-timestamp.

As the logshield is battery backupped one can run this sketch before uploading your own sketch, saving some precious memory for setting the clock.

Todo

  • Clean up the code
  • add comments with sense,
  • create a stripped version (without float math)
  • create a proper interface
  • write an Arduino based NTP server :)

Enjoy tinkering,

rob.tillaart@removethisgmail.com

Update

  • 16/10/2010 : A bug is fixed in the adjustment of the TimeZone in the following line of code. Thanx Anthony.
t4 += (2 * 3600L);  // added the L preventing overflow

  • 19/10/2010 : Corrected the divider 65535.0 => 65536.0 in the floating part.
  • 07/01/2011 : http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1291637975#10 shows DST function, to be integrated.
  • 01/04/2012 : Added Arduino 1.0 compatibility by Rodrigo Castro & Rob Tillaart.
  • 19/02/2012 : refactored all #if ARDUINO lines + updated todo list

CODE

/*
 * NAME: NTP2RTC
 * DATE: 2012-02-19
 *  URL: http://playground.arduino.cc/Main/DS1307OfTheLogshieldByMeansOfNTP
 *
 * PURPOSE:
 * Get the time from a Network Time Protocol (NTP) time server
 * and store it to the RTC of the adafruit logshield
 *
 * NTP is described in:
 * http://www.ietf.org/rfc/rfc958.txt (obsolete)
 * http://www.ietf.org/rfc/rfc5905.txt 
 *
 * based upon Udp NTP Client, by Michael Margolis, mod by Tom Igoe
 * uses the RTClib from adafruit (based upon Jeelabs)
 * Thanx!
 * mod by Rob Tillaart, 10-10-2010
 * 
 * This code is in the public domain.
 * 
 */


// libraries for ethershield
#include <SPI.h>         
#include <Ethernet.h>

#if ARDUINO >= 100
#include <EthernetUdp.h>	// New from IDE 1.0
#else
#include <Udp.h>  
#endif	

// libraries for realtime clock
#include <Wire.h>
#include <RTClib.h>

RTC_DS1307 RTC;

// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = { 0x00, 0x11, 0x22, 0x33, 0xFB, 0x11 }; // Use your MAC address
byte ip[] = { 192, 168, 0, 1 };                      // no DHCP so we set our own IP address
byte subnet[] = { 255, 255, 255, 0 };                // subnet mask
byte gateway[] = { 192, 168, 0, 2 };                 // internet access via router

unsigned int localPort = 8888;             // local port to listen for UDP packets

// find your local ntp server http://www.pool.ntp.org/zone/europe or 
// http://support.ntp.org/bin/view/Servers/StratumTwoTimeServers
// byte timeServer[] = {192, 43, 244, 18}; // time.nist.gov NTP server
byte timeServer[] = {193, 79, 237, 14};    // ntp1.nl.net NTP server  

const int NTP_PACKET_SIZE= 48;             // NTP time stamp is in the first 48 bytes of the message

byte pb[NTP_PACKET_SIZE];                  // buffer to hold incoming and outgoing packets 

#if ARDUINO >= 100
// An EthernetUDP instance to let us send and receive packets over UDP
EthernetUDP Udp;		// New from IDE 1.0
#endif	


///////////////////////////////////////////
//
// SETUP
// 
void setup() 
{
  Serial.begin(19200);
  Serial.println("NTP2RTC 0.5");

  // start Ethernet and UDP

  Ethernet.begin(mac, ip);	   // For when you are directly connected to the Internet.
  Udp.begin(localPort);
  Serial.println("network ...");

  // init RTC
  Wire.begin();
  RTC.begin();
  Serial.println("rtc ...");
  Serial.println();
}

///////////////////////////////////////////
//
// LOOP
// 
void loop()
{
  Serial.print("RTC before: ");
  PrintDateTime(RTC.now());
  Serial.println();

  // send an NTP packet to a time server
  sendNTPpacket(timeServer);

  // wait to see if a reply is available
  delay(1000);

  if ( Udp.available() ) {
    // read the packet into the buffer
#if ARDUINO >= 100
    Udp.read(pb, NTP_PACKET_SIZE);      // New from IDE 1.0,
#else
    Udp.readPacket(pb, NTP_PACKET_SIZE);
#endif	

    // NTP contains four timestamps with an integer part and a fraction part
    // we only use the integer part here
    unsigned long t1, t2, t3, t4;
    t1 = t2 = t3 = t4 = 0;
    for (int i=0; i< 4; i++)
    {
      t1 = t1 << 8 | pb[16+i];      
      t2 = t2 << 8 | pb[24+i];      
      t3 = t3 << 8 | pb[32+i];      
      t4 = t4 << 8 | pb[40+i];
    }

    // part of the fractional part
    // could be 4 bytes but this is more precise than the 1307 RTC 
    // which has a precision of ONE second
    // in fact one byte is sufficient for 1307 
    float f1,f2,f3,f4;
    f1 = ((long)pb[20] * 256 + pb[21]) / 65536.0;      
    f2 = ((long)pb[28] * 256 + pb[29]) / 65536.0;      
    f3 = ((long)pb[36] * 256 + pb[37]) / 65536.0;      
    f4 = ((long)pb[44] * 256 + pb[45]) / 65536.0;

    // NOTE:
    // one could use the fractional part to set the RTC more precise
    // 1) at the right (calculated) moment to the NEXT second! 
    //    t4++;
    //    delay(1000 - f4*1000);
    //    RTC.adjust(DateTime(t4));
    //    keep in mind that the time in the packet was the time at
    //    the NTP server at sending time so one should take into account
    //    the network latency (try ping!) and the processing of the data
    //    ==> delay (850 - f4*1000);
    // 2) simply use it to round up the second
    //    f > 0.5 => add 1 to the second before adjusting the RTC
    //   (or lower threshold eg 0.4 if one keeps network latency etc in mind)
    // 3) a SW RTC might be more precise, => ardomic clock :)


    // convert NTP to UNIX time, differs seventy years = 2208988800 seconds
    // NTP starts Jan 1, 1900
    // Unix time starts on Jan 1 1970.
    const unsigned long seventyYears = 2208988800UL;
    t1 -= seventyYears;
    t2 -= seventyYears;
    t3 -= seventyYears;
    t4 -= seventyYears;

    /*
    Serial.println("T1 .. T4 && fractional parts");
    PrintDateTime(DateTime(t1)); Serial.println(f1,4);
    PrintDateTime(DateTime(t2)); Serial.println(f2,4);
    PrintDateTime(DateTime(t3)); Serial.println(f3,4);
    */
    PrintDateTime(DateTime(t4)); Serial.println(f4,4);
    Serial.println();

    // Adjust timezone and DST... in my case substract 4 hours for Chile Time
    // or work in UTC?
    t4 -= (3 * 3600L);     // Notice the L for long calculations!!
    t4 += 1;               // adjust the delay(1000) at begin of loop!
    if (f4 > 0.4) t4++;    // adjust fractional part, see above
    RTC.adjust(DateTime(t4));

    Serial.print("RTC after : ");
    PrintDateTime(RTC.now());
    Serial.println();

    Serial.println("done ...");
    // endless loop 
    while(1);
  }
  else
  {
    Serial.println("No UDP available ...");
  }
  // wait 1 minute before asking for the time again
  // you don't want to annoy NTP server admin's
  delay(60000L); 
}

///////////////////////////////////////////
//
// MISC
// 
void PrintDateTime(DateTime t)
{
    char datestr[24];
    sprintf(datestr, "%04d-%02d-%02d  %02d:%02d:%02d  ", t.year(), t.month(), t.day(), t.hour(), t.minute(), t.second());
    Serial.print(datestr);  
}


// send an NTP request to the time server at the given address 
unsigned long sendNTPpacket(byte *address)
{
  // set all bytes in the buffer to 0
  memset(pb, 0, NTP_PACKET_SIZE); 
  // Initialize values needed to form NTP request
  // (see URL above for details on the packets)
  pb[0] = 0b11100011;   // LI, Version, Mode
  pb[1] = 0;     // Stratum, or type of clock
  pb[2] = 6;     // Polling Interval
  pb[3] = 0xEC;  // Peer Clock Precision
  // 8 bytes of zero for Root Delay & Root Dispersion
  pb[12]  = 49; 
  pb[13]  = 0x4E;
  pb[14]  = 49;
  pb[15]  = 52;

  // all NTP fields have been given values, now
  // you can send a packet requesting a timestamp: 
#if ARDUINO >= 100
  // IDE 1.0 compatible:
  Udp.beginPacket(address, 123); //NTP requests are to port 123
  Udp.write(pb,NTP_PACKET_SIZE);
  Udp.endPacket(); 
#else
  Udp.sendPacket( pb,NTP_PACKET_SIZE,  address, 123); //NTP requests are to port 123
#endif	  

}
///////////////////////////////////////////
//
// End of program
//

Share