Topic: DIY "GREEN DOT" Data Logger

[DeKay]

@DeKay:
You don't happen to have a Logic capture session saved from a valid green dot logger do you?  I'd love to see the timing on a valid logger so I could make sure this setup emulates it as closely as possible.  

Awesome work here, travisc.  I'm happy to try and help out.  However, I don't have a green dot logger.  My console predates all this stuff, and I've never owned a real Davis logger.  So what I'm about to provide here probably isn't useful.

I'm shooting a bit in the dark here.  My openbench logic sniffer program is crashing with a weird Java exception right now so I'm just going by filename.  Grab the OLS software from here and see if you can't import one of the files in the attached zip.  As the name implies, I think this capture is a data write to the flash chip.

[torkelmj]

Travisc...great work. Are u a son of Torkel ???

I'd rather say like-minded. FWIW, apart from the PIC used here - I used the same approach when I assembled a data logger. Didn't turn out as appealing as this one, though - just wanted a proof-of-concept. Travisc has done a great job with this project! Now, what will be the next move from Hayward?

[travisc]

Awesome work here, travisc.  I'm happy to try and help out.  However, I don't have a green dot logger.  My console predates all this stuff, and I've never owned a real Davis logger.  So what I'm about to provide here probably isn't useful.

Thanks DeKay!  I think I confused myself there.  I remembered seeing a green dot logger in a screen shot, but that must have been in torkelmj's PDF.  Sorry, still getting familiar with the users here.    In any case, the data you sent is useful to compare timing.  Thank you.

I'd rather say like-minded. FWIW, apart from the PIC used here - I used the same approach when I assembled a data logger. Didn't turn out as appealing as this one, though - just wanted a proof-of-concept. Travisc has done a great job with this project! Now, what will be the next move from Hayward?

Thanks torkelmj!  Your excellent research on this topic made this possible.   

Now for a progress update.  In an attempt to eliminate the busy state on the flash chip after the reset, I spent the morning re-jiggering the prototype to control the CS line of the flash chip instead of the reset line.  After comparing the new captures with last night's, I realized that last night the flash wasn't responding as busy at all.  When I saw a command starting with D being sent to the chip, I assumed it was the status check command D7.   I should have paid closer attention, the console was actually sending the D2 command, which is a main memory page read.   So everything was working just fine as it was! 

That said, I'm glad I made this mistake.  It forced me to look into the flash chip's reset timing a little closer.  After being released from reset, the flash needs 1ms to wake up.  If I stuck with that technique, it could be a way for Davis to detect this was an invalid logger in future firmware.  If they did a read within 1ms of reading the security register it would likely return busy or nothing at all.  That'd be a tell for sure.

Next I'll get the USB cable attached so I can confirm the console is indeed logging data to the flash.

[travisc]

Mission Accomplished.   

I tested the serial port using belfryboy's serial utility, then enabled logging...


Launched Cumulus for a second time after letting it sit for a while so the logger would have some data waiting.  The tiny text in the corner shows it downloaded 5 records from the flash ...


I'll make the final changes to the gerbers, clean up the code, then share it here tomorrow.

[DeKay]

Mission Accomplished.   

<snip>

I'll make the final changes to the gerbers, clean up the code, then share it here tomorrow.

Very impressive!  You must have been pretty psyched the first time you saw Cumulus was able to grab archive data.  I know I was.  And thanks for sharing your work for others to learn from, use, and build upon.

Just curious: what additional power draw does the PIC add?  Does it remain feasible to power the console by battery?

[Weather Display]

good work!
kudos!

[travisc]

Thanks guys.  This was only possible because of all the research and progress everyone here already made.  What I did was pretty easy in comparison.  That said, it did indeed feel good to see Cumulus pull in that data.  I just started it up again this morning and it downloaded another 500 or so records. 

As for power usage, I just tested the PIC.  It takes 2.36mA.  I've got it topped out right now since it has to control the flash's SS line in response to the console.  I wanted the timing to be as close as possible.  My original approach would have taken way less power as I could have put the processor to sleep in between bytes.  I guess I could have put it to sleep for good after the spoof, but I was trying to program it in such a way that it would still work if the console authorized more than once.

I've got some ideas for a version 2 if I can get some time.  I want to bump up to the next PIC chip, one of the PIC16LF1823 series.  It's only 50 cents or so more.  It has 6 more pins available that would allow us to connect directly to the serial lines of the USB chip.  Then we could implement a boot loader so the code could be easily updated in the future should Davis try to thwart our efforts again.  Users would just need to run a small program on their PC to update it, no PIC programmers or crazy stuff.  I could also design it in such a way that we have control over both the SS and reset line of the flash.  Then we could choose between either of my methods as needed.

I can definitely lay out the board.  The programming on that one would be more involved.  Anyone else here capable of implementing a boot loader on a PIC?  I've done it before a few times but I'm much more proficient with the CCS compiler.  If I use that then there's no free option for others to play.  MicroChip has some boot loader examples, we'd probably just have to modify one of those.

[The pen is blue]

Awesome. I feel a bit more comfortable looking at actually buying a Vantage Pro now I know that I have a chance at not having to fork out for an expensive logger too.

[travisc]

Awesome. I feel a bit more comfortable looking at actually buying a Vantage Pro now I know that I have a chance at not having to fork out for an expensive logger too.

I agree.  I wouldn't have just bought one if I had to shell out $150 for a logger either.

I'm just about finished the second rev of the board.  I bumped up to the slightly larger chip so it can be firmware upgraded over USB.  It's cheap insurance if Davis blacklists the key or changes their approach.

[Haku]

After the mild disappointment with my experiments with the MSP430 I decided to give it another try with a more familiar MCU. But be warned, this is by far not as elegant as any of the solutions shown here (especially travisc's nice approach) but it might be OK if you don't want to solder much. I got a STM32F4Discovery board for $15. It comes with all you need except the USB cord and, of course, the 20pin connector for the console and a bunch of hook-up wires. The programmer is on the same circuit board but you might have to download the IDE. (I used Keil uVision4, which you can get for free with a 32kB code size limit. I guess you should not make commercial products with the evaluation version.)

OK. First the bad: the board is too big to fit on the back of the console. It likely consumes a bit too much power to be powered from batteries alone - I measured 21mA including the serial to USB converter - but that should be OK as you would be able to use the USB power anyway.

Now the not so bad: You don't need anything else as the board can be programmed to act as an UART to USB converter. The chip offers 1MB (i.e. 8Mbit) of flash and 192kB (1.5Mbit) of ram. So you could easily use this as your logger storage as well. I currently have the SPI unlock part running, which has been the main topics in this thread (again many thanks to all who have contributed to this!!) and the UART to USB converter, which allows you to connect the console to the PC. Latter comes from ST and other folks who have implemented a working VCP (see the Readme.txt for more info). The attached code should compile in Keil uVision 4 without modifications. It would likely be easy to add the logging part as well (look in the app.c file for starters), but I currently have no need for the logging part, as I have a linux box to fetch the data.

In any case, please note that this is only derivative work from what others have done – none of which is my achievement.
-Haku

btw: The attached files are split to comply with the 128kB attachment limit. They should be extracted into the same directory.

[samueltc]

Code: [Select]

/*
  Arduino program to spoof the Green Dot Logger
  SPI interface.  (Causes the serial interface
  to be enabled?)
    
  by rdsman
    
  20 FEB 2013
  [...]

It needs to be tried out by someone!

Hi,

I tried the code with an Envoy without success. I tried a couple of things (I'm putting a lot of detail because I'm new to the microcontroller/electronic world and maybe I'm missing something obvious):

- Tested the spoofer with a Bus Pirate, I get the Status and Security register right when I query them (so it seem to works!).
- I thought that the delay in the Arduino bootloader could cause a problem, so I disabled the part in the Arduino bootloader code that check for an UART connection. It boot straight to the spoofer. (uploaded the bootloader and application via the ISP header using the buspirate as a programmer)
- The cable between the Arduino and the Envoy is 5cm long.
- I got 3.04v from the VCC of the Envoy.
- I tried to sniff the SPI bus without success, I see some data, but wrong value. (first time I sniff an spi bus, so maybe my settings/wiring are wrong?, I'm wiring MOSI/MISO/CLK/SS/GND in parallel, between the Envoy and Arduino)
- Using an FTDI Basic, I wired the TX to RX and RX to TX and GND to GND. I only get the "NO" whatever I type.
- No SIM is connected at the moment to my Envoy.
- The Arduino is a MEGA328P, 8hmz, 3.3v.

Thanks for your help!

Sam

[travisc]

I tried the code with an Envoy without success.

You may have the SPI mode set wrong on the Arduino's SPI hardware.  I'd try sniffing the Envoy by itself using the Bus Pirate to confirm the settings on the Bus Pirate are setup correctly.  You'll see the status request (0xD7), followed by the security register request (0x77).  On a console, those commands aren't sent until about 4 seconds after powers up.

Once you confirm that you see those commands correctly on the Bus Pirate, test your Arduino spoofer again using those same settings.  Only when you have both working should you connect them together.

Also make sure when the Bus Pirate is connected to the Envoy that it is a slave mode and not controlling the SDO line.

[rdsman]

I'd try sniffing the Envoy by itself using the Bus Pirate to confirm the settings on the Bus Pirate are setup correctly.

I second that opinion!  Let us know when you have this working.  DeKay can most likely help you with this...........

[DeKay]

I'd try sniffing the Envoy by itself using the Bus Pirate to confirm the settings on the Bus Pirate are setup correctly.

I second that opinion!  Let us know when you have this working.  DeKay can most likely help you with this...........

Bad advice!  I've had issues with my Bus Pirate and SPI that I've never been able to sort out.  All the SPI stuff I've done has been with my logic analyzer.  I have heard that one of the keys to success though with the Bus Pirate and SPI is keeping the wires as short as you can.  Good luck.

[rdsman]

Bad advice!  I've had issues with my Bus Pirate and SPI that I've never been able to sort out.

DeKay:

You did help - Don't use the Bus Pirate!

[travisc]

I finished the latest design for a green dot USB logger.  This design includes a slightly larger PIC chip.

This design has the following advantages over my previous design:
- Bootloading new PIC code over USB.
- Overriding flash chip control using either the CS or reset line.
- Debugging PIC code while the board is running in the logger.

I've ordered prototype boards from http://oshpark.com.  The cost is $6.75 for 3 boards, free shipping.  At that price there's no sense trying to make your own.

The schematic and gerbers are attached.  I'll have source code available once I prove this design actually works.

Top:


Bottom:

[TomOmniSmith]

Hey sorry this might be a daft question but what's the specific components you are using? As there are a few versions of each?

Am planning to purchase from digikey.

Thanks

[travisc]

I updated the schematic above with part numbers.  Keep in mind I have not tested this design yet and I won't have any source code for a few weeks.

[JohnA]

Travisc,

Two questions...

- Is it possible that you also publish the Gerbers of the first version of the board that was working?
- Is it possible that you can provide me with a programmed PIC for that version 1 board?

John

[travisc]

Two questions...

- Is it possible that you also publish the Gerbers of the first version of the board that was working?
- Is it possible that you can provide me with a programmed PIC for that version 1 board?

The gerbers are in this thread somewhere, but I don't have code for you.  I modified the code when I hacked the board to use the SS line instead of the reset line. I don't have a version of the code before I made all those changes.  Even if I did, I can't program a PIC for you.  I don't have sockets to program surface mount PICs out of circuit.  All my PIC programming is done using the in-circuit programming header, after the PIC is installed on the PCB.

The best I can do is sell you one of the new ones.  I'm not looking to start a side-business or anything, but I may sell a few of the extra prototype boards to users if they are interested.

[JohnA]

I'm absolutely interested if I can buy a board from you.

John

[schaffer970]

I would also be interested in purchasing one.  Just got my 6153 up and running a couple of weeks ago and would like to communicate with it so I can put data online.

Thanks so much for the work you have done on getting a logger to work with the green dot stations!

[frozen]

Hi all !

Just a few words first to thank all of you guys' who did a fantastic job helping screwed customers circumvent the Davis "green dot" data logger problem/scandal ! I read this forum since several weeks, in fact since I got my Vantage Vue.

I'm new in the weather stations hobby, except for some basic Oregon stuff, and recently decided that I wanted a "real" weather station capable of sharing it's data to wunderground.com (etc).

A good friend of mine, a real pro tech head, also purchased a wired Vantage pro ISS without even buying the console (on purpose)  ...

He needs weather data for a heating system management software that he's currently developing/writing (from scratch). He may join the discussion here soon, so I'll let him tell you more about his system if he wants/can/has time left... So, for now consider me as his authorized "spokesman"

The new approach
Now, about the "green dot" problem : He decided to take another approach : I told you that he purchased a wired Vantage pro "ISS", and now he has successfully managed to "listen" to the wired ISS data "stream", without having a console, and "decode" and log (etc) the not so raw data that's transmitted by it !

The idea was to "simply" hack the wired raw data (wireless is next step), and feed it to a Raspberry PI that will easily be able to make good use of it, probably much better than any crazy expensive so called logger sold by Davis (log, publish, analyze, graph etc).

He, and one of his techs employees, are now at work to get their hands on the same data from the wireless radio output from another ISS (868 - 868.6 MHz), a wireless one, apparently this "stream" is not encrypted in any way (they may also try with my Vue ISS). He will gladly accept any suggestions from you guys' !

Why bother to hack the "green dot" console if we can get, log and use the weather data from the ISS ?

Even better, if the radio feed is successfully hacked and fed to a Raspberry, those who own a wireless console would still be able to use both the Raspberry and the console at the same time. Isn't it a good approach that could solve the green dot problem for many of us ?

If successful he will use the system for his commercial heating management software but he's also absolutely 100% positive about sharing/posting the code, methods and hardware components he used to hack the ISS to the open source community.

Any thoughts ? Anyone interested ? Should I open an new thread about his approach ?

[dmurphydrtc]

Frozen,

Wireless hack of ISS already done and dusted...check this out, it works I've implemented the solution.

http://www.wxforum.net/index.php?topic=18718.0

[samueltc]

Frozen,

IIRC communication between ISS and Console/Envoy are over RS-422. Hope it help.

All the best,

Sam