<< Summary

Connect and read an external GPS, part II Raspberry PI
Where is the Sun?

Reading Serial Data, parsing NMEA

We've seen in this other document how to read data out of a serial port.
What you have in this case is a stream of characters, which now needs to be parsed to be used any further.
The data emitted by the GPS follow the requirement of the NMEA specification.

The sample application, what it does

We present in the GPSandSun folder an application doing the following:
  1. It reads the data stream emitted by the GPS
  2. It identifies the different NMEA Sentences in the data stream
  3. It takes the sentences we are interested in to obtain our position and the GPS time (RMC)
  4. When position and time can be obtained from the GPS, the actual direction and altitude of the Sun are calculated and displayed
To simplify the code of the steps 2 and beyond, I'll be using different Java open source projects I own: The corresponding jar-files are already included in the lib directory under GPSandSun.

The output

Here is what we expect. The Altitude and Azimuth of the Sun, for a given position and time, updated no more than every 1 second, or when the position changes.

Parsing GPS Data

The challenge here is to put together all the chunks of data spitted out by the GPS, split the stream into NMEA Sentences, and make sense out of them.
Bulk data, from the serial port Reformatted, before parsing

$GPGGA,233900.000,3744.9279,N,12230.4340,W,1,5,1.47,145.2,M,-25.5,M,,*6F
$GPGSA,A,3,16,
20,23,13,07,,,,,,,,1.77,1.47,0.99*00
$GPGSV,4,1,13,23,72,021,14,16,53,094,31,13,50,314,24,20,48,181,22*
73
$GPGSV,4,2,13,07,31,248,16,10,20,316,16,32,20,153,23,04,14,283,*7C
$GPGSV,4,3,13,06,07,109,,03,06,1
35,,08,04,237,,27,03,118,*73
$GPGSV,4,4,13,43,,,*7C
$GPRMC,233900.000,A,3744.9279,N,12230.4340,W,0.31,
3.88,190214,,,A*79
$GPVTG,3.88,T,,M,0.31,N,0.58,K,A*31

$GPGGA,233901.000,3744.9279,N,12230.4340,W,1,5,1
.47,145.2,M,-25.5,M,,*6E
$GPGSA,A,3,16,20,23,13,07,,,,,,,,1.77,1.47,0.99*00
$GPRMC,233901.000,
A,3744.9279,N,12230.4340,W,0.35,3.88,190214,,,A*7C
$GPVTG,3.88,T,,M,0.35,N,0.64,K,A*3A

$GPGGA,233902.000,3744.9278,N,12230.4339,W,1,5,1.47,145.
2,M,-25.5,M,,*62
$GPGSA,A,3,16,20,23,13,07,,,,,,,,1.77,1.47,0.99*00
$GPRMC,233902.000,A,3744.9278,N,12
230.4339,W,0.54,3.88,190214,,,A*77
$GPVTG,3.88,T,,M,0.54,N,1.01,K,A*3F

$GPGGA,233903.000,3744.9276,N,12230.4337,W,1,6,1.29,145.2,M,-25.5,M,,*68
$GPGSA,A,3,16,20,23,13
,07,32,,,,,,,1.61,1.29,0.95*02
$GPRMC,233903.000,A,3744.9276,N,12230.4337,W,0.74,161.99,190214,,,A*71

$GPVTG,161.99,T,,M,0.74,N,1.37,K,A*3D

$GPGGA,233904.000,3744.9
168,N,12230.4334,W,1,6,1.29,144.3,M,-25.5,M,,*60
$GPGSA,A,3,16,20,23,13,07,32,,,,,,,1.61,1.29,0.95*02

$GPRMC,233904.000,A,3744.9168,N,12230.4334,W,0.69,177.30,190214,,,A*71
$GPVTG,177.30,T,,M,0.69,N,1.28,K
,A*3B

$GPGGA,233905.000,3744.9171,N,12230.4344,W,1,6,1.30,144.
0,M,-25.5,M,,*65
$GPGSA,A,3,16,20,23,13,07,32,,,,,,,1.61,1.30,0.96*09
$GPGSV,4,1,13,23,72,021,15,16,53
,094,32,13,50,314,25,20,48,181,22*70
$GPGSV,4,2,13,07,31,248,16,32,20,153,24,10,20,316,16,04,14,283,*7B

$GPGSV,4,3,13,06,07,109,,03,07,135,,08,04,237,,27,03,118,*72
$GPGSV,4,4,13,43,,,*7C
$GPRMC,233905.00
0,A,3744.9171,N,12230.4344,W,0.69,174.44,190214,,,A*7F
$GPVTG,174.44,T,,M,0.69,N,1.28,K,A*3B

$GPGGA,233906.000,3744.9140,N,12230.4337,W,1,6,1.29,143.3,M,-25.5,M,,*6C
$GPGSA,A,3,16,
20,23,13,07,32,,,,,,,1.61,1.29,0.95*02
$GPRMC,233906.000,A,3744.9140,N,12230.4337,W,1.16,169.31
,190214,,,A*7D
$GPVTG,169.31,T,,M,1.16,N,2.14,K,A*30

$GPGGA,233907.000,3744.9108,N,12230.4327,W,1,6,1.29,142.3,M,-25.5,M,,*61
$GPGSA,A,3,16,20,23,13
,07,32,,,,,,,1.61,1.29,0.96*01
$GPRMC,233907.000,A,3744.9108,N,12230.4327,W,1.58,114.60,190214,,,A*75

$GPVTG,114.60,T,,M,1.58,N,2.93,K,A*3B

$GPGGA,2
33908.000,3744.9090,N,12230.4322,W,1,6,1.29,141.5,M,-25.5,M,,*6E
$GPGSA,A,3,16,20,23,13,07,32,,,,,,,1.6
1,1.29,0.95*02
$GPRMC,233908.000,A,3744.9090,N,12230.4322,W,2.07,109.78,190214,,,A*73
$GPVTG,1
09.78,T,,M,2.07,N,3.84,K,A*30
              

$GPGGA,233900.000,3744.9279,N,12230.4340,W,1,5,1.47,145.2,M,-25.5,M,,*6F
$GPGSA,A,3,16,20,23,13,07,,,,,,,,1.77,1.47,0.99*00
$GPGSV,4,1,13,23,72,021,14,16,53,094,31,13,50,314,24,20,48,181,22*73
$GPGSV,4,2,13,07,31,248,16,10,20,316,16,32,20,153,23,04,14,283,*7C
$GPGSV,4,3,13,06,07,109,,03,06,135,,08,04,237,,27,03,118,*73
$GPGSV,4,4,13,43,,,*7C
$GPRMC,233900.000,A,3744.9279,N,12230.4340,W,0.31,3.88,190214,,,A*79
$GPVTG,3.88,T,,M,0.31,N,0.58,K,A*31
$GPGGA,233901.000,3744.9279,N,12230.4340,W,1,5,1.47,145.2,M,-25.5,M,,*6E
$GPGSA,A,3,16,20,23,13,07,,,,,,,,1.77,1.47,0.99*00
$GPRMC,233901.000,A,3744.9279,N,12230.4340,W,0.35,3.88,190214,,,A*7C
$GPVTG,3.88,T,,M,0.35,N,0.64,K,A*3A
$GPGGA,233902.000,3744.9278,N,12230.4339,W,1,5,1.47,145.2,M,-25.5,M,,*62
$GPGSA,A,3,16,20,23,13,07,,,,,,,,1.77,1.47,0.99*00
$GPRMC,233902.000,A,3744.9278,N,12230.4339,W,0.54,3.88,190214,,,A*77
$GPVTG,3.88,T,,M,0.54,N,1.01,K,A*3F
$GPGGA,233903.000,3744.9276,N,12230.4337,W,1,6,1.29,145.2,M,-25.5,M,,*68
$GPGSA,A,3,16,20,23,13,07,32,,,,,,,1.61,1.29,0.95*02
$GPRMC,233903.000,A,3744.9276,N,12230.4337,W,0.74,161.99,190214,,,A*71
$GPVTG,161.99,T,,M,0.74,N,1.37,K,A*3D
$GPGGA,233904.000,3744.9168,N,12230.4334,W,1,6,1.29,144.3,M,-25.5,M,,*60
$GPGSA,A,3,16,20,23,13,07,32,,,,,,,1.61,1.29,0.95*02
$GPRMC,233904.000,A,3744.9168,N,12230.4334,W,0.69,177.30,190214,,,A*71
$GPVTG,177.30,T,,M,0.69,N,1.28,K,A*3B
$GPGGA,233905.000,3744.9171,N,12230.4344,W,1,6,1.30,144.0,M,-25.5,M,,*65
$GPGSA,A,3,16,20,23,13,07,32,,,,,,,1.61,1.30,0.96*09
$GPGSV,4,1,13,23,72,021,15,16,53,094,32,13,50,314,25,20,48,181,22*70
$GPGSV,4,2,13,07,31,248,16,32,20,153,24,10,20,316,16,04,14,283,*7B
$GPGSV,4,3,13,06,07,109,,03,07,135,,08,04,237,,27,03,118,*72
$GPGSV,4,4,13,43,,,*7C
$GPRMC,233905.000,A,3744.9171,N,12230.4344,W,0.69,174.44,190214,,,A*7F
$GPVTG,174.44,T,,M,0.69,N,1.28,K,A*3B
$GPGGA,233906.000,3744.9140,N,12230.4337,W,1,6,1.29,143.3,M,-25.5,M,,*6C
$GPGSA,A,3,16,20,23,13,07,32,,,,,,,1.61,1.29,0.95*02
$GPRMC,233906.000,A,3744.9140,N,12230.4337,W,1.16,169.31,190214,,,A*7D
$GPVTG,169.31,T,,M,1.16,N,2.14,K,A*30
$GPGGA,233907.000,3744.9108,N,12230.4327,W,1,6,1.29,142.3,M,-25.5,M,,*61
$GPGSA,A,3,16,20,23,13,07,32,,,,,,,1.61,1.29,0.96*01
$GPRMC,233907.000,A,3744.9108,N,12230.4327,W,1.58,114.60,190214,,,A*75
$GPVTG,114.60,T,,M,1.58,N,2.93,K,A*3B
$GPGGA,233908.000,3744.9090,N,12230.4322,W,1,6,1.29,141.5,M,-25.5,M,,*6E
$GPGSA,A,3,16,20,23,13,07,32,,,,,,,1.61,1.29,0.95*02
$GPRMC,233908.000,A,3744.9090,N,12230.4322,W,2.07,109.78,190214,,,A*73
$GPVTG,109.78,T,,M,2.07,N,3.84,K,A*30
              
The program reading the serial port does not know anything about NMEA. It does not care about carriage return (CR) or line feed (LF). It continuously reads a stream of data, period.
An NMEA String (aka Sentence) begins with $, followed by 2 characters describing the device (GP here), then a three-character sentence identifier (VTG, RMC, GLL, etc). The last 3 characters are a checksum, made of a star (*) followed by the hexadecimal value of the bitwise XOR of all the preceeding characters in the sentence. Like *30 in the last sentence above.
We need a tool to analyze the serial port data stream, present the program with well-formatted and distinct NMEA sentences, and turn their content into Objects and scalars usable from Java.
For example, the structure of an RMC String is this one:

              1      2 3        4 5         6 7     8     9      10    11
       $xxRMC,123519,A,4807.038,N,01131.000,E,022.4,084.4,230394,003.1,W*6A
              |      | |        | |         | |     |     |      |     |
              |      | |        | |         | |     |     |      |     Variation sign
              |      | |        | |         | |     |     |      Variation value
              |      | |        | |         | |     |     Date DDMMYY
              |      | |        | |         | |     COG
              |      | |        | |         | SOG
              |      | |        | |         Longitude Sign
              |      | |        | Longitude Value
              |      | |        Latitude Sign
              |      | Latitude value
              |      Active or Void
              UTC HHMMSS
      
The position can be contained in a GeoPos object, dates and times in a Date object, Speed Over Ground (SOG) in a double, Course Over Ground (COG) as an int, etc.
This is what the NMEA Parser is doing.
It is designed after the Model-View-Controller (MVC) pattern.
In short, you will The hardware setting is the exact same one as in the first project.

A quick look at the code

The application is made out of two small files: The main method is in nmea.CustomNMEAReader.

The Model: nmea.CustomNMEASerialReader

      

     1  package nmea;
     2  
     3  import com.pi4j.io.serial.Serial;
     4  import com.pi4j.io.serial.SerialDataEvent;
     5  import com.pi4j.io.serial.SerialDataListener;
     6  import com.pi4j.io.serial.SerialFactory;
     7  
     8  import java.util.List;
     9  
    10  import ocss.nmea.api.NMEAEvent;
    11  import ocss.nmea.api.NMEAListener;
    12  import ocss.nmea.api.NMEAReader;
    13  
    14  public class CustomNMEASerialReader 
    15       extends NMEAReader 
    16  {
    17    private int baudRate = 4800;
    18    private CustomNMEASerialReader instance = this;
    19    
    20    public CustomNMEASerialReader(List<NMEAListener> al, int br)
    21    {
    22      super(al);
    23      baudRate = br;
    24    }
    25  
    26    @Override
    27    public void read()
    28    {
    29      if (System.getProperty("verbose", "false").equals("true")) 
    30        System.out.println("From " + this.getClass().getName() + " Reading Serial Port.");
    31      
    32      super.enableReading();
    33      
    34      // Opening Serial port
    35      try
    36      {
    37        final Serial serial = SerialFactory.createInstance();
    38  
    39        // create and register the serial data listener
    40        serial.addListener(new SerialDataListener()
    41        {
    42          @Override
    43          public void dataReceived(SerialDataEvent event)
    44          {
    45        //  System.out.print(/*"Read:\n" + */ event.getData());
    46            instance.fireDataRead(new NMEAEvent(this, event.getData()));
    47          }
    48        });
    49        System.out.println("Opening port [" + Serial.DEFAULT_COM_PORT + "]");
    50        serial.open(Serial.DEFAULT_COM_PORT, baudRate);
    51      }
    52      catch (Exception ex)
    53      {
    54        ex.printStackTrace();
    55      }
    56      // Reading on Serial Port
    57      System.out.println("Port is open...");
    58    }
    59  }
      
    
Notice the way the serial port is initiated - just like before - and the way the instance.fireDataRead(new NMEAEvent(this, event.getData())); is invoked, line 46.

The View: nmea.CustomNMEAReader

      

     1  package nmea;
     2  
     3  import calculation.AstroComputer;
     4  import calculation.SightReductionUtil;
     5  
     6  import java.text.DecimalFormat;
     7  
     8  import java.util.Calendar;
     9  import java.util.TimeZone;
    10  
    11  import ocss.nmea.api.NMEAClient;
    12  import ocss.nmea.api.NMEAEvent;
    13  import ocss.nmea.api.NMEAListener;
    14  import ocss.nmea.parser.GeoPos;
    15  import ocss.nmea.parser.RMC;
    16  import ocss.nmea.parser.StringParsers;
    17  
    18  public class CustomNMEAReader extends NMEAClient
    19  {
    20    private final static DecimalFormat DFH = new DecimalFormat("#0.00'\272'");
    21    private final static DecimalFormat DFZ = new DecimalFormat("##0.00'\272'");
    22    
    23    private static GeoPos prevPosition = null;
    24    private static long   prevDateTime = -1L;
    25  
    26    public CustomNMEAReader()
    27    {
    28      super();
    29    }
    30    
    31    @Override
    32    public void dataDetectedEvent(NMEAEvent e)
    33    {
    34  //  System.out.println("Received:" + e.getContent());
    35      manageData(e.getContent().trim());
    36    }
    37  
    38    private static CustomNMEAReader customClient = null;  
    39    
    40    private static void manageData(String sentence)
    41    {
    42      boolean valid = StringParsers.validCheckSum(sentence);
    43      if (valid)
    44      {
    45        String id = sentence.substring(3, 6);
    46        if ("RMC".equals(id))
    47        {
    48       // System.out.println(line);
    49          RMC rmc = StringParsers.parseRMC(sentence);
    50       // System.out.println(rmc.toString());
    51          if (rmc != null && rmc.getRmcDate() != null && rmc.getGp() != null)
    52          {
    53            if ((prevDateTime == -1L || prevPosition == null) ||
    54                (prevDateTime != (rmc.getRmcDate().getTime() / 1000) || !rmc.getGp().equals(prevPosition)))
    55            {
    56              Calendar current = Calendar.getInstance(TimeZone.getTimeZone("etc/UTC"));
    57              current.setTime(rmc.getRmcDate());
    58              AstroComputer.setDateTime(current.get(Calendar.YEAR), 
    59                                        current.get(Calendar.MONTH) + 1, 
    60                                        current.get(Calendar.DAY_OF_MONTH), 
    61                                        current.get(Calendar.HOUR_OF_DAY), 
    62                                        current.get(Calendar.MINUTE), 
    63                                        current.get(Calendar.SECOND));
    64              AstroComputer.calculate();
    65              SightReductionUtil sru = new SightReductionUtil(AstroComputer.getSunGHA(),
    66                                                              AstroComputer.getSunDecl(),
    67                                                              rmc.getGp().lat,
    68                                                              rmc.getGp().lng);
    69              sru.calculate();
    70              Double he = sru.getHe();
    71              Double  z = sru.getZ();
    72              System.out.println(current.getTime().toString() + ", He:" + DFH.format(he)+ ", Z:" + DFZ.format(z) + " (" + rmc.getGp().toString() + ")");
    73            }
    74            prevPosition = rmc.getGp();
    75            prevDateTime = (rmc.getRmcDate().getTime() / 1000);
    76          }
    77          else
    78          {
    79            if (rmc == null)
    80              System.out.println("... no RMC data in [" + sentence + "]");
    81            else
    82            {  
    83              String errMess = "";
    84              if (rmc.getRmcDate() == null)
    85                errMess += ("no Date ");
    86              if (rmc.getGp() == null)
    87                errMess += ("no Pos ");
    88              System.out.println(errMess + "in [" + sentence + "]");
    89            }
    90          }
    91        }
    92      }    
    93      else
    94        System.out.println("Invalid data [" + sentence + "]");
    95    }
    96  
    97    public static void main(String[] args)
    98    {
    99      System.setProperty("deltaT", System.getProperty("deltaT", "67.2810")); // 2014-Jan-01
   100  
   101      int br = 4800;
   102      System.out.println("CustomNMEAReader invoked with " + args.length + " Parameter(s).");
   103      for (String s : args)
   104      {
   105        System.out.println("CustomNMEAReader prm:" + s);
   106        try { br = Integer.parseInt(s); } catch (NumberFormatException nfe) {}
   107      }
   108      
   109      customClient = new CustomNMEAReader();
   110        
   111      Runtime.getRuntime().addShutdownHook(new Thread() 
   112        {
   113          public void run() 
   114          {
   115            System.out.println ("Shutting down nicely.");
   116            customClient.stopDataRead();
   117          }
   118        });    
   119      customClient.initClient();
   120      customClient.setReader(new CustomNMEASerialReader(customClient.getListeners(), br));
   121      customClient.startWorking(); // Feignasse!
   122    }
   123  
   124    private void stopDataRead()
   125    {
   126      if (customClient != null)
   127      {
   128        for (NMEAListener l : customClient.getListeners())
   129          l.stopReading(new NMEAEvent(this));
   130      }
   131    }
   132  }
      
    
Notice:
NB: Even if some code is featured here in this document, the point of truth remains the code checked in the repository. Should any disparity show up, the repository would be the one to trust.

Compile and Run

The code can be compiled on the Raspberry PI.
As I use an IDE running on another machine to develop the code, I usually compile it on this machine with a similar directory structure, and then FTP the classes to the Raspberry PI. But compiling on the RasPI works just as well.
A compilation script on the Raspberry PI would look as follow:

 #!/bin/bash
 JAVAC_OPTIONS="-sourcepath ./src"
 JAVAC_OPTIONS="$JAVAC_OPTIONS -d ./classes"
 CP=./classes
 PI4J_HOME=/home/pi/pi4j/pi4j-distribution/target/distro-contents
 CP=$CP:$PI4J_HOME/lib/pi4j-core.jar
 CP=$CP:./lib/almanactools.jar
 CP=$CP:./lib/geomutil.jar
 CP=$CP:./lib/nauticalalmanac.jar
 CP=$CP:./lib/nmeaparser.jar
 # -verbose can be removed
 JAVAC_OPTIONS="-verbose $JAVAC_OPTIONS -cp $CP"
 COMMAND="javac $JAVAC_OPTIONS ./src/nmea/*.java"
 echo Compiling: $COMMAND
 $COMMAND
 echo Done
     
A run script would be like:

 #!/bin/bash
 CP=./classes
 PI4J_HOME=/home/pi/pi4j/pi4j-distribution/target/distro-contents
 CP=$CP:$PI4J_HOME/lib/pi4j-core.jar
 CP=$CP:./lib/almanactools.jar
 CP=$CP:./lib/geomutil.jar
 CP=$CP:./lib/nauticalalmanac.jar
 CP=$CP:./lib/nmeaparser.jar
 sudo java -cp $CP nmea.CustomNMEAReader $*
      
If the script is named run, you would start it like this (after a chmod +x run)
 
 ./run 9600
      
You should end up with the output we've seen before:

And what next?

Now we know where we are, and where the Sun is.
We want to use those data to orient a solar panel so it faces the Sun.
This means that we have a solar panel mounted on some orientable bracket, driven by some servos.
The servos will be triggered by the Raspberry PI.
More to come...


Oliv did it