Author Topic: DIY "GREEN DOT" Data Logger  (Read 94314 times)

0 Members and 1 Guest are viewing this topic.

Offline dmurphydrtc

  • Member
  • *
  • Posts: 19
Re: DIY "GREEN DOT" Data Logger
« Reply #50 on: June 21, 2013, 11:00:27 AM »
Great news.  The information you provided is specific enough ie a total solution is close at hand. 
My technical skills are limited, not much to add other than thanks and admiration.

Offline torkelmj

  • Contributor
  • ***
  • Posts: 145
    • My weather station, etc.
Re: DIY "GREEN DOT" Data Logger
« Reply #51 on: July 02, 2013, 06:07:53 PM »
Building on the work already presented by others here, my own pre-Christmas tinkering, handwritten notes and experiences from running 15+ VP2 stations at different locations utilizing non-Davis equipment to gather, forward and store data (including the troubles we started running into when we found ourselves in need of a replacement console which happened to have FW v. 3.00 installed) - I've gathered what I believe is the essential stuff into a single document. For anyone with an interest in gaining access to the console's serial line using simple equipment and techniques, http://meteo.annoyingdesigns.com/DavisSPI.pdf may be just what you need to get started. Having run the "ATtiny-variant" for half a year now, concurrently at 15+ different sites, I'm confident that it works as it should.

As I at no time have been in need of the "data logger" functionality, I haven't cared to document the work required to assemble a "data logger" with functionality equal to the original Davis equipment. The solution already discussed in this thread works fine, though. I may get around to include a section with the required details, but that won't happen this summer.

Wishing you all a wonderful summer season.

Offline DeKay

  • Forecaster
  • *****
  • Posts: 399
    • Mad Scientist Labs
Re: DIY "GREEN DOT" Data Logger
« Reply #52 on: July 02, 2013, 10:31:13 PM »
Building on the work already presented by others here, my own pre-Christmas tinkering, handwritten notes and experiences from running 15+ VP2 stations at different locations utilizing non-Davis equipment to gather, forward and store data (including the troubles we started running into when we found ourselves in need of a replacement console which happened to have FW v. 3.00 installed) - I've gathered what I believe is the essential stuff into a single document. For anyone with an interest in gaining access to the console's serial line using simple equipment and techniques, http://meteo.annoyingdesigns.com/DavisSPI.pdf may be just what you need to get started. Having run the "ATtiny-variant" for half a year now, concurrently at 15+ different sites, I'm confident that it works as it should.

I'm not sure what impresses me the most about this writeup:
  • the depth and breadth of your work
  • your fantastic attention to detail
  • the extensive list of references
  • the tip to keep SPI wire lengths short (this might have been screwing up some stuff I was doing)
  • your willingness to share this freely with everyone
  • the shout-out to myself (always good for the old ego)
  • the absurdly excessive yet well-placed vitriol dripping from every page   \:D/
Outstanding!!!   =D>

Offline docbee

  • Forecaster
  • *****
  • Posts: 853
    • smartbedded
Re: DIY "GREEN DOT" Data Logger
« Reply #53 on: July 03, 2013, 03:37:00 AM »
@torkelmj: thanks for sharing this in such brilliant quality! a true masterpiece!
founder of smartbedded.com - home of meteohub, meteoplug, meteobridge, meteostick

Offline torkelmj

  • Contributor
  • ***
  • Posts: 145
    • My weather station, etc.
Re: DIY "GREEN DOT" Data Logger
« Reply #54 on: July 03, 2013, 03:39:08 AM »
  • the absurdly excessive yet well-placed vitriol dripping from every page   \:D/

There may be an unintended and excessive leak from the vitriol line, caused by English not being my primary language.
Having said that, I was truly disappointed by the limitations imposed in FW v. 3 just as we had started rolling out our monitoring system.
 ](*,)

Offline dmurphydrtc

  • Member
  • *
  • Posts: 19
Re: DIY "GREEN DOT" Data Logger
« Reply #55 on: July 03, 2013, 09:13:58 AM »
Torkel,

To say this was worth the wait would be a total understatement. A brilliant piece of technical writing and hats off to you for
sharing this wonderful work. Thanks once again. Now off to build me a unit........

Offline rdsman

  • Senior Contributor
  • ****
  • Posts: 249
Re: DIY "GREEN DOT" Data Logger
« Reply #56 on: July 03, 2013, 05:39:00 PM »
torkelmj:

Excellent!  The WXForum should award you the PhDD!  (Doctor of Documentation)


Ray

Offline torkelmj

  • Contributor
  • ***
  • Posts: 145
    • My weather station, etc.
Re: DIY "GREEN DOT" Data Logger
« Reply #57 on: July 09, 2013, 03:25:49 AM »
Thanks. Good to know that a number of users have already started out with their own ATtiny projects. Best of luck to you all.


Offline Haku

  • Member
  • *
  • Posts: 5
Re: DIY "GREEN DOT" Data Logger
« Reply #58 on: July 28, 2013, 03:24:58 PM »
Hi,
First big thanks to all who have contributed to this! I see that many of you have invested a large amount of time to figure this out.  I received a Davis Vantage Vue with the dreaded firmware V3.0, which (given the incredibly ugly looking console) has no other function than to stream the data to some place where it can be processed. I currently use a Kindle PW and an old Oregon Scientific 200A to display the readings in my living room and I upload it to a web-page. I’m happy to share this project if someone is interested.

I’m an open source guy and I would never have purchased any Davis products had I known that it cannot be used with any third-party (i.e. useful) software – not to speak of the locking down an otherwise perfectly good serial port for no other reason than to sell a useless piece of software with a $10 serial dongle.
Davis: You really don’t have to sell your products if you don’t want anyone to use them! (My 2cents).

Anyway, I tried to port Kashima’s code to a TI MSP430 Launchpad (I picked it up for $4.30 about a year ago from MOUSER. It came with two 16 bit MCUs and a USB to serial converter. How handy!). But still no success to date. I would assume that all that has been discussed here should also work with a Vantage Vue console as these seem to use the same dataloggers. Is that assumption correct?

What puzzles me is that the Vue again and again spits out something like 0xF5 (or 0x75, I cannot quite see the first bit on the analog scope) followed by a zero, which should essentially pump the response from the MCU to the console. (Please see the MOSI picture on my trusty analog scope). The code then essentially replies the same byte, as 0xF5 is not a recognized as a command. Hence the case in ‘default’ is taken, which does not change the serial register but just reloads the SPI counter. This also seems to tell me that the MCU is actually receiving and replying the data correctly. (Nothing seems to change if I enable pullups on the inputs, so I assume the outputs on the console are not open collector.)

Other things I have observed is that the slave select (SS, pin 1 on the console) goes to high when I power up the console, but then it stays high even during the SPI transmission. That’s unexpected! The SPI clock line initially idles high, but after the first byte it seems to idle low (again, please see the pictures: top trace is SPI clock (pin 2 on the console) and the bottom is either MISO or MOSI (as defined on the console side). All signals are measured directly at the pins of the MCU to make sure that the signals arrive undistorted, which seems to be the case. I use 2-3 inch wires between the console and the MCU.

I’m at a loss :oops: Does anyone have some suggestions?

Many thanks,
Haku

Offline rdsman

  • Senior Contributor
  • ****
  • Posts: 249
Re: DIY "GREEN DOT" Data Logger
« Reply #59 on: July 28, 2013, 07:01:32 PM »
At some point during the power-up sequence, the Vue reads the Status Register of the logger.  It then reads the Security Register.  If any of this fails, it never tries again unless you remove power.  You will have to sync on Slave Select or none of the commands will make any sense.



Ray

Offline DeKay

  • Forecaster
  • *****
  • Posts: 399
    • Mad Scientist Labs
Re: DIY "GREEN DOT" Data Logger
« Reply #60 on: July 28, 2013, 08:02:29 PM »
Hi,
First big thanks to all who have contributed to this! I see that many of you have invested a large amount of time to figure this out.  I received a Davis Vantage Vue with the dreaded firmware V3.0, which (given the incredibly ugly looking console) has no other function than to stream the data to some place where it can be processed. I currently use a Kindle PW and an old Oregon Scientific 200A to display the readings in my living room and I upload it to a web-page. I’m happy to share this project if someone is interested.

Please do!

Other things I have observed is that the slave select (SS, pin 1 on the console) goes to high when I power up the console, but then it stays high even during the SPI transmission. That’s unexpected! The SPI clock line initially idles high, but after the first byte it seems to idle low (again, please see the pictures: top trace is SPI clock (pin 2 on the console) and the bottom is either MISO or MOSI (as defined on the console side). All signals are measured directly at the pins of the MCU to make sure that the signals arrive undistorted, which seems to be the case. I use 2-3 inch wires between the console and the MCU.

The console also uses SPI to talk to the radio chip, but it uses a different pin off the MCU for SS that is not brought out to the rear connector.  The unexpected is actually very expected   :-)

Offline Haku

  • Member
  • *
  • Posts: 5
Re: DIY "GREEN DOT" Data Logger
« Reply #61 on: July 28, 2013, 09:33:25 PM »
Thank you all for the very fast replies! I'll try harder.

Please do!
I will as soon as I figure this one out  ](*,)

The console also uses SPI to talk to the radio chip, but it uses a different pin off the MCU for SS that is not brought out to the rear connector.  The unexpected is actually very expected   :-)
Does that mean that the Vue would need another datalogger than the VP? I would think that the flash chip would not respond without the /CS pulled low. Or maybe there are redundant lines on the data logger board. Would you happen to know which pin on the ATMega would be used for the SS in the Vue? I could solder a line directly on the chip inside the console. I am planning to put the MCU inside anyway so I get an unlocked console without the extra board hanging off the back.

Or should I try to sync on the first falling edge of the SPI clock? It is peculiar that it first idles high and later it seems to idle low. But again, I don't have a digital scope that would let me trigger on the very first events so I'm flying pretty much blind here. Maybe I should use another MCU as a logic analyzer.

Again, many thanks to all your help!

Offline DeKay

  • Forecaster
  • *****
  • Posts: 399
    • Mad Scientist Labs
Re: DIY "GREEN DOT" Data Logger
« Reply #62 on: July 29, 2013, 12:04:52 AM »
The console also uses SPI to talk to the radio chip, but it uses a different pin off the MCU for SS that is not brought out to the rear connector.  The unexpected is actually very expected   :-)
Does that mean that the Vue would need another datalogger than the VP? I would think that the flash chip would not respond without the /CS pulled low. Or maybe there are redundant lines on the data logger board. Would you happen to know which pin on the ATMega would be used for the SS in the Vue? I could solder a line directly on the chip inside the console. I am planning to put the MCU inside anyway so I get an unlocked console without the extra board hanging off the back.
[/quote]

I think I've confused you.  My mistake.  When I said MCU, I meant the processor in the console (an Atmega 128L).  The radio chip and the expansion port are connected to the same SPI MOSI and MISO lines.  The "native" SS pin on the Atmega is what comes out on the expansion port and this is the /CS to the flash chip.  The other activity you see is the Atmega talking to the radio.  Because it uses a different pin off the Atmega as the radio's /CS, the flash does not respond when the Atmega is talking to the radio.

When a compatible datalogger is connected, the console recognizes this and will now and then write data to the flash on the logger.  You would then see SS go low at these times.

To make a long story short, just use the pins brought out to the expansion connector and you'll be golden.


Offline Haku

  • Member
  • *
  • Posts: 5
Re: DIY "GREEN DOT" Data Logger
« Reply #63 on: July 29, 2013, 09:53:14 AM »
Thank you very much for clarifying this. It makes perfect sense!

I see now that the SS goes low occasionally and I got the MCU to detect both, the status register and the security register command. However, sending the response has to date never unlocked the serial port. At best I occasionally get the 'incompatible datalogger' message - but since I only get it occasionally (after power-cycling the console), I think the timing of my MCU is still off by a bit. I'll try some more and report back.

Offline rdsman

  • Senior Contributor
  • ****
  • Posts: 249
Re: DIY "GREEN DOT" Data Logger
« Reply #64 on: July 29, 2013, 12:17:59 PM »
Quote
Maybe I should use another MCU as a logic analyzer.

Brilliant idea!  Use the MSP430 to capture the SPI data and then print it to the serial port.  This way you can find out if your timing is off.

Ray

Offline Haku

  • Member
  • *
  • Posts: 5
Re: DIY "GREEN DOT" Data Logger
« Reply #65 on: August 04, 2013, 01:10:07 PM »
Hi everyone.

Unfortunately, still no success with the MSP430G2231. Below one of the many versions the code segment I used. I tried all possible combinations for the SPI timing and for some I see that the console requests the status byte but nothing happens after that. Once in a few dozen tries I get an 'incompatible datalogger'. There are some extra variables in the code for debugging but else it should do about the same as Kashima's code. I checked the signals on the MCU with a scope and all looks clean. Does anyone see where I go wrong here?

Is it neccessary to switch to a MCU that allows true 4-whire (plus ground) instead of this 'emulated' 4-wire using the GPIO 'SS' pin below? (The smallest MCU I have here with true 4 wire SPI is a STM32F407, which seems a huge overkill for this simple task, and it might skew the temperature readings on the console due to its pretty hefty power consumption.)

Best,
Haku

Code: [Select]
//***************************************************************************************
//  MSP430 Unlock Davis
//
//  Description: Unlock the Davis Vantage Vue/Pro(TM) Serial Port
// This code was based on the PIC version from kashima
//
//                Slave                      Master
//               MSP430G2231              Davis Weather Station Console
//             -----------------          -----------------
//            |                 |        |                 |
//            |                 |        |                 |
//            |                 |        |                 |
//            |                 | /|\    |                 |
//            |          RST/NMI|---     |                 |
//      LED <-|P1.0             |        |                 |
//            |                 |        |                 |
//            |         SDI/P1.7|<-------|3/MOSI           |
//            |         SDO/P1.6|------->|4/MISO           |
//            |        SCLK/P1.5|<-------|2/SCLK           |
//            |          SS/P1.1|<-------|1/SS             |
//            |                |         |                 |
//             -----------------          -----------------
//
//
//  Special thanks to (in no particular order): DeKay, rdsman, kashima, iBangkok24, franzz, belfryboy, and all the other nice folks over at WXForum.net
//
//  Built with Code Composer Studio v5
//***************************************************************************************

#include <msp430.h>

#define STATUS_REGISTER 0xD7  //215
#define STATUS_WORD 0x8C  //140
#define SECURITY_REGISTER 0x77  //119

#define SS (P1IN&0x02) //Slave Select pin: Slave is addressed if SS reads zero

const unsigned char SECURITY_RESPONSE[128] =
{
  0x80, 0x2D, 0x22, 0x6F, 0x52, 0x6F, 0x98, 0xA9, 0x21, 0x25, 0x5E, 0x2D, 0x2D, 0x31, 0xD2, 0x39,
  0x18, 0x1C, 0x63, 0x0C, 0x31, 0x21, 0x2D, 0x39, 0x90, 0xDE, 0x94, 0x6F, 0x6B, 0x77, 0x73, 0x7F,
  0x63, 0x8C, 0x88, 0x84, 0x80, 0x9C, 0x98, 0x94, 0x90, 0xAD, 0xA9, 0xA5, 0xA1, 0xBD, 0xB9, 0xB5,
  0xB1, 0xCA, 0xCE, 0xC2, 0xC6, 0xDA, 0xDE, 0xD2, 0xD6, 0xEB, 0xEF, 0xE3, 0xE7, 0xFB, 0xFF, 0xF3,
  0x0B, 0x02, 0x16, 0x17, 0x11, 0x15, 0x1F, 0x22, 0x00, 0x00, 0x46, 0x00, 0xFF, 0xFF, 0xAC, 0xFF,
  0x30, 0x30, 0x4D, 0x32, 0x36, 0x39, 0x37, 0x31, 0x0F, 0x1C, 0x0C, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
  0x3F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF
};

volatile unsigned char i;
volatile unsigned char dbg_cnt,statcnt,seccnt; //for debugging
volatile unsigned char cmd;


int main(void)
{
  WDTCTL = WDTPW + WDTHOLD;             // Stop watchdog timer

  BCSCTL1 = 0x8f; //16 MHz
  DCOCTL = 0x60;

  P1OUT =  0x10;  //0x10                // P1.4 set
  P1REN |= 0x10; //0x30;                // pull-up. P1.4 must be pulled up
  P1DIR = 0x01; //0x01;                 // P1.0 output, else input. MISO will be controlled below
//just to save power:
  P2OUT =  0x00; //all low
  P2REN |= 0xff; //pull-downs
  P2DIR = 0x00;  //all in

  dbg_cnt=statcnt=seccnt=0;  //just for debugging

  P1OUT &= ~0x01;


  USICTL0 |=  USIPE7 + USIPE6 + USIPE5 + USIOE + USIGE; // Port, SPI slave
//  USICTL1 |= USICKPH; //clockphase
//  USICKCTL |= USICKPL; //clock inactive state is high
// (I tried all the combinations of USICKPH, USICKPL, and USIGE a few dozen times.)

  USICTL0 &= ~USISWRST;                 // USI released for operation

  P1OUT |= 0x01;
  while(SS==0); //wait until ss goes high (idle). This ensures that we wait until the console is powered up

  USISRL = 0x00;                       // init-load data
  USICNT = 8; // ready SPI for 8 bit reception
  P1OUT &= ~0x01;

while(1){
reset:
USICTL0 &= ~USIOE; //release MISO line
USICNT = 8;
USISRL = 0;
while(USICNT){if(SS)USICNT = 8;} // wait for a command
cmd = USISRL;
dbg_cnt++;

if(cmd==215)  //status byte
{
    USICTL0 |= USIOE; //turn on MISO (output)
USISRL = STATUS_WORD; // response word
USICNT = 8;
statcnt++;  //for debugging
while(USICNT); // wait a sending
if(SS) goto reset;
statcnt++;
}
else
if(cmd==119)  //status register
{
  USICTL0 |= USIOE;
for(i = 0; i < 3; i++) // put a dummy word
{
USISRL = 0;
USICNT = 8;
seccnt++;  //for debugging
while(USICNT);
if(SS) goto reset;
}
for(i = 0; i < 128; i++) // put a security response
{
USISRL = SECURITY_RESPONSE[i];
USICNT = 8;
seccnt++;  //for debugging
while(USICNT);
  if(SS) goto reset;
}
}
  }
}

Offline travisc

  • Member
  • *
  • Posts: 13
Re: DIY "GREEN DOT" Data Logger
« Reply #66 on: August 04, 2013, 08:06:44 PM »
Hi Everyone,

I'm new to the weather station community, but not so new to electronics.  I'm hoping I can help out a little with this green dot logger problem.  I just finished the layout of a new PCB.  I'm hoping it will be a tidy solution that combines a lot of progress the other members have already achieved.  I've combined an FTDI USB-Serial chip with a PIC and 1Mb Flash chip.  The board is the same size as the one belfryboy used for his logger, so it should fit nicely inside the console.

The idea is the PIC and the Flash chip sit on the SPI bus together.  The PIC holds the Flash chip in reset while it chats it up with the console to spoof the verification process.  Once that's done, the PIC will effectively remove itself from the bus and release the flash chip from reset.  From that point on the logger should have all the features of a standard 1Mb memory Davis logger.  Except with a much more reliable USB interface.

I haven't had a chance to test this yet as I just finished the design, but I'll be surprised if it doesn't work.  I should have a few proto boards in hand later this week.  About the same time my first weather station arrives.  :)

The schematic and a picture of the board is attached.  I intended to use Eagle for the board layout so it could be easily modified by others, but my patience got the best of me, sorry.  I'm much more familiar with other software.  Once the design is proven I'll be more than happy to share the gerbers and source code.

Special thanks goes out to torkelmj, DeKay, belfryboy, and rdsman, as this is mostly their work.  :-)
« Last Edit: August 04, 2013, 08:09:18 PM by travisc »

Offline SlowModem

  • Weather at the speed of dialup!
  • Forecaster
  • *****
  • Posts: 6641
  • WX @ 26.4 kbs
    • Watts Bar Weather
Re: DIY "GREEN DOT" Data Logger
« Reply #67 on: August 04, 2013, 08:41:50 PM »
Welcome and enjoy your stay!  :D  I'm sure your input will be well received.
Greg Whitehead
Ten Mile, TN USA

Offline torkelmj

  • Contributor
  • ***
  • Posts: 145
    • My weather station, etc.
Re: DIY "GREEN DOT" Data Logger
« Reply #68 on: August 04, 2013, 11:07:14 PM »
Well done, travisc!

Offline schaffer970

  • Member
  • *
  • Posts: 38
Re: DIY "GREEN DOT" Data Logger
« Reply #69 on: August 05, 2013, 10:43:25 AM »
travisc, if you are looking for guinea pig (tester) let me know.  I just got my brand new 6153 up and running this weekend.  Would love to try out your solution.  PM me.
« Last Edit: August 05, 2013, 11:40:54 AM by schaffer970 »

Offline rdsman

  • Senior Contributor
  • ****
  • Posts: 249
Re: DIY "GREEN DOT" Data Logger
« Reply #70 on: August 05, 2013, 11:02:04 AM »
Quote
Well done, travisc!

I'll second that!

Ray

Offline travisc

  • Member
  • *
  • Posts: 13
Re: DIY "GREEN DOT" Data Logger
« Reply #71 on: August 08, 2013, 04:05:29 PM »
I received the proto board today.  The VP2 arrived the other day so I should be all set to test this tonight.  I've already made the changes to the source code and tested it on a little console simulator I made.  Everything looks great on the logic analyzer.  I'll post again tonight with some more pics and results.


Offline DeKay

  • Forecaster
  • *****
  • Posts: 399
    • Mad Scientist Labs
Re: DIY "GREEN DOT" Data Logger
« Reply #72 on: August 08, 2013, 08:24:06 PM »
I received the proto board today.  The VP2 arrived the other day so I should be all set to test this tonight.  I've already made the changes to the source code and tested it on a little console simulator I made.  Everything looks great on the logic analyzer.  I'll post again tonight with some more pics and results.


Epic!

Offline travisc

  • Member
  • *
  • Posts: 13
Re: DIY "GREEN DOT" Data Logger
« Reply #73 on: August 08, 2013, 09:52:13 PM »
Looks like we're good to go!  I haven't hooked up the USB cable to test the serial port, but I know that'll work as that's already been proven by DeKay.  What I can confirm for certain is that the console is spoofed by the logger and then starts talking to the flash chip.

Here's how it works:
1. The console asks the flash chip for its status.
2. The flash chip (actual flash chip, not PIC) responds.
3. The console asks for the security register.  This is where the PIC barges in.  It immediately puts the flash into reset and takes control of the SPI MISO line.
4. The PIC sends the spoofed security register contents.  Then shuts down it's MISO line and releases the flash from reset.
5. A little while later the console checks the flash's status again, then sends a 0x53 command (Main memory page to buffer transfer) to transfer page 0 to the buffer.

That's as far as I've gotten.  I'm assuming next the console would read the buffer and determine the contents of the flash so it knows where to start logging.  But I have to setup my console and weather station to get any further.   :-|

Though this does look like it's been a success, I may make one small change.  After the PIC releases the flash from reset, it appears the flash needs some pulses of the CLK line before it's status is ready.  This can be seen on the logic analyzer.  So what you see before the 0x53 command is about 22 reads of the status register.  After that the flash responds as ready and the console sends the 0x53 command.  While this it totally valid, I'm sure it's different behaviour than a standard logger, which may give Davis a method of detecting this logger in a future version of firmware.  I may hack this board to control the flash's CS line instead of the RESET line to prevent this.

@DeKay:
You don't happen to have a Logic capture session saved from a valid green dot logger do you?  I'd love to see the timing on a valid logger so I could make sure this setup emulates it as closely as possible.  

Here's a few pics for those interested ...

The PCB after SMT components installed.


Now from the side with the connectors and test points.


In the console while also connected to the logic analyzer.


If anyone wants to see the saleae logic capture session you can download below.  The action starts at 4536ms, then the 0x53 command is sent at 4808ms.
Saleae Logic File.zip
« Last Edit: August 08, 2013, 10:00:05 PM by travisc »

Offline dmurphydrtc

  • Member
  • *
  • Posts: 19
Re: DIY "GREEN DOT" Data Logger
« Reply #74 on: August 08, 2013, 10:22:43 PM »
Travisc...great work. Are u a son of Torkel ???