Topic: Implementing a Si1000 based wireless receiver for Davis ISS data

[rdsman]

tridge:

I have the Vue (US version) with the same console/ISS firmware that you have.  The picture below shows the capture from the CC1101 for a type 80 message.  I'm not sure what's going on here......

[tridge]

I have the Vue (US version) with the same console/ISS firmware that you have.  The picture below shows the capture from the CC1101 for a type 80 message.  I'm not sure what's going on here......

That is a puzzle. Your capture shows
 0x0c 0x01 0x20 0xca 0x34 0x56 0x50 0xa0 0xb3 0xff 0xff 0x1d 0xae
swapping msb to lsb gives:
 0x30 0x80 0x04 0x53 0x2c 0x6a 0x0a 0x05 0xcd 0xff 0xff 0xb8 0x75

The 12 is because of the addition of the RSSI and LQI bytes. That matches the 10 data bytes I see (RSSI is read from a separate register, not put into the FIFO on the Si1000). Presumably RSSI was 0x1d and LQI was 0xae.

Next puzzle is the data itself. The speed is 0x04, which is presumably wind speed. On mine I see 0xe^speed. How fast was the wind when you captured this?

Also interesting that you see 0xff 0xff at the end. I always see 0xc2 0x16. See this log example:
http://uav.tridgell.net/DavisISS/davis.log

If we have the same ISS fw version then I guess the differences would have to be radio differences. Could it be data whitening? If so, then I'd expect there to be a fixed xor pattern which will make my data look like yours. The cc1101 register settings I see in the im-me have PKTCTRL0 set to 0x00, which implies data whitening off. Hmm .....

Ahhh! I'm an idiot. I had data whitening turned on in my firmware for bit rates below 32kbaud, a hang over from the SiK telemetry firmware. So I was xoring the data with a pseudo-random stream. Doh!!

I've now turned it off and things make a lot more sense. Here is a type 0x80 message without data whitening screwing it up:

82 0 8C 27 C8 0 51 C1 FF FF

Much better!

I'm in at uni all day today, but I'll fixup my repo soon and remove the insane bit flipping code.

Cheers, Tridge

[DeKay]

In other news I've worked out another protocol type. Message type 8 seems to be light level. In a fairly dark room I get a value of around 200. In bright (winter) sunlight I get over 3000. I'm not sure what the units are. Message 8 is sent every 54 seconds.

Tridge, glad you found that bug!  Now that that is sorted, could you clarify which message and bytes within that message are the light level?

Also, I haven't updated my protocol document with this information yet, but another user here figured out the messages for UV and Solar.  This could be added to your decoder code for compatibility with the ISS version that supports these sensors, even though your Vue ISS will never send these values.

Finally, in case you are interested, the ISS can also send a Battery Low message.  That same user in this post got close to figuring out which message that was but didn't quite nail it down.

[tridge]

Tridge, glad you found that bug!  Now that that is sorted, could you clarify which message and bytes within that message are the light level?

I haven't had time to re-run the tests again, but based on the PR9 whitening sequence starting with 0xF0, this is what I have:
8 == temperature
E == rain spoons
7 == light level

those are the ones I'm confident of. The rest are a bit uncertain so far:
2 == humidity (as percent*40)
5 == something related to rapid rain. Maybe rain rate when spoon counter is overloaded?

The other two I see are message types 3 and 9. I thought 3 might be related to barometric pressure until you told me it doesn't have a baro. 9 is usually zero, but sometimes has a value - perhaps wind gust related? I need to experiment more.

Also, I haven't updated my protocol document with this information yet, but another user here figured out the messages for UV and Solar.  This could be added to your decoder code for compatibility with the ISS version that supports these sensors, even though your Vue ISS will never send these values.

yep, I'll take a look

Finally, in case you are interested, the ISS can also send a Battery Low message.  That same user in this post got close to figuring out which message that was but didn't quite nail it down.

Interesting, thanks!
I'm not at home at the moment, when I get back I'll do some more hacking.
Cheers, Tridge

[DeKay]

Tridge, glad you found that bug!  Now that that is sorted, could you clarify which message and bytes within that message are the light level?

I haven't had time to re-run the tests again, but based on the PR9 whitening sequence starting with 0xF0, this is what I have:
8 == temperature
E == rain spoons
7 == light level

those are the ones I'm confident of. The rest are a bit uncertain so far:
2 == humidity (as percent*40)
5 == something related to rapid rain. Maybe rain rate when spoon counter is overloaded?

The other two I see are message types 3 and 9. I thought 3 might be related to barometric pressure until you told me it doesn't have a baro. 9 is usually zero, but sometimes has a value - perhaps wind gust related? I need to experiment more.

According to what I had figured out, humidity is message "A".  I don't get any messages starting with 7 so this could be Vue specific (unfortunately for me).  I do get messages "5" and "9" that I haven't sorted, but I suspect one of those must be battery level.

[rdsman]

Ahhh! I'm an idiot.

Tridge

I sincerely doubt it.  I'm also glad that you figured it out!  I was going to recommend that you did all the bit twiddling on the packet itself.  Then use the standard conversion formulas worked out by Dekay and others.  Doesn't matter now, your on the right track!  Good job!

[BCJKiwi]

Vantage Vue DOES have a barometer in the ISS.

Inside sensors in the console are additional temp and humidity to those in the ISS

See the table on page 50 in the Vue Console Manual for a summary of the updates times for the sensors.

[dalecoy]

Vantage Vue DOES have a barometer in the ISS.

Where do you find that?

[BCJKiwi]

OK - My mistake - misread the table on page 50!
Thought I had checked my info prior to posting but still got it wrong.

The barometer is in the console, not the ISS

[tridge]

I'm now trying to decide what output formats to support on the 3DR radio firmware.
One obvious choice would be to support the LOOP and STRMON commands of a Davis console, but I'm not sure that is actually the most useful format.
What do others think about using JSON? Perhaps like this:

Code: [Select]

{ "temperature": 21.6, "humidity": 73, "windspeed": 7, "winddirection": 180,
  "lightlevel": 2700, "raw": "82008C27C80051C1FFFF", "version": "1.0" }
I'd setup the radio to output the data as each DavisTalk packet comes in (so every 2.7s).
It would run at 19200. Note that the "raw" field would give the raw DavisTalk packets, so as new fields are discovered post processing could be used to add historical data. It would only show fields that the unit has data for.
The advantage of this is it is easily extensible, and we always get all the data. It should be pretty easy to extend weewx, wview and other weather software to support this data format. The 'version' field would allow the display software to know what version of the 3DR radio firmware produced the values. The JSON format is supported by all modern languages, so should be very easy to parse. We could also add fields for numbers of lost packets, RSSI etc.
Sound OK?
Cheers, Tridge

[torkelmj]

Sounds great, but if using JSON - how about adding a little hint regarding what units the data is reported in? Like temperature_c, windspeed_msec, etc. --- just to avoid guessing and having to ask the question later? Surely, those few extra bytes can be afforded?

[tridge]

Sounds great, but if using JSON - how about adding a little hint regarding what units the data is reported in? Like temperature_c, windspeed_msec, etc. --- just to avoid guessing and having to ask the question later? Surely, those few extra bytes can be afforded?

yep, good idea!

[rfb]

I just saw this thread by googling & joined up to learn stuff & see if I can contribute. I have a Pi and a Davis Vantage Pro2 console + USB data logger, and I would really like to get a second "console" working, using the Pi.  I assume the actual radio module I need is the "3DR Radio USB-915 Mhz "Ground" module (US)" from diydrones.com (I'm located in Canada).

The ISS I have is the basic one (plus the shield/fan for the outside temperature sensor), i.e. wind speed & direction, temperature & humidity, rainfall. I haven't checked the firmware version, but it's about 3 years old, so I assume doesn't have 3.xx.

I have downloaded the source code from github, but have no idea what tools I need to build and install it (once I have the radio module).  Can you point me to the appropriate info somewhere?

Thanks again for a great project!

[tridge]

I've now added JSON output support. It produces messages like this:

Code: [Select]

{ "transmitter_id": 2, "RSSI": 144, "recv_packets": 52, "wind_speed_mph": 0, "wind_direction_degrees": 215, "temperature_F": 68.5, "humidity_pct": 63.5, "light": 1528, "rain_spoons": 12, "raw": "E2 00 98 0C 03 00 63 16 FF FF ", "version": "1.0" }
This should be pretty easy to parse and display. I'd like to add weewx support soon.
I've also released precompiled firmware here:
http://uav.tridgell.net/DavisSi1000/firmware/
and updated the README to give instructions here:
https://github.com/tridge/DavisSi1000
That should be enough for some other people to give it a try. If you do try it, please comment about how it went in this thread.

Cheers, Tridge

[tridge]

I just saw this thread by googling & joined up to learn stuff & see if I can contribute. I have a Pi and a Davis Vantage Pro2 console + USB data logger, and I would really like to get a second "console" working, using the Pi.  I assume the actual radio module I need is the "3DR Radio USB-915 Mhz "Ground" module (US)" from diydrones.com (I'm located in Canada).

That right. That is the one I test with. There are other variants of this radio that should work too.

The ISS I have is the basic one (plus the shield/fan for the outside temperature sensor), i.e. wind speed & direction, temperature & humidity, rainfall. I haven't checked the firmware version, but it's about 3 years old, so I assume doesn't have 3.xx.

Once you try it please send some logs of it running, and we can see if we need to adjust the firmware for your ISS

I have downloaded the source code from github, but have no idea what tools I need to build and install it (once I have the radio module).  Can you point me to the appropriate info somewhere?

see the new README info I uploaded today.

Thanks again for a great project!

Thanks!

[tridge]

I've now implemented a driver for the current svn version of weewx:
https://github.com/tridge/DavisSi1000/blob/origin/weewx/Si1000.py
and submitted it for inclusion in weewx:
https://groups.google.com/forum/#!topic/weewx-development/Hw_XMjThldA
I installed it on a RaspberryPi at the CMAC flying field (http://weather.cmac.org.au) this morning.
Cheers, Tridge

[dmurphydrtc]

I've now implemented a driver for the current svn version of weewx:
https://github.com/tridge/DavisSi1000/blob/origin/weewx/Si1000.py
and submitted it for inclusion in weewx:
https://groups.google.com/forum/#!topic/weewx-development/Hw_XMjThldA
I installed it on a RaspberryPi at the CMAC flying field (http://weather.cmac.org.au) this morning.
Cheers, Tridge

Great work Andrew...installed and running for 4 hours data looking good. Now I need to decide if I want to sell my Vantage Pro 2 Console. This has been a Chris Hadfield internet experience for me. A person willing to share their intellect, graft and all done with a smile. Thanks once again. If ever I can assist just drop us a line.

 

[tridge]

great, I'm glad it's working for you David!

[hmluk]

Hi,
I am working on creating a remote weather station using the Davis IIS and the normal Vantage Pro weather console and a mobile phone to transmit the data back to an http server. I am going to link to the phone from the console through a bluetooth link. A solution using the Si1000 would seem to be likely to be less power hungry and also gets round the Green dot logger and link issue. The only thing missing from the 3DR radio is the local pressure and temperature. The Si1000 has I2C port. Would it be possible to extend the firmware to read a BMP085 barometric sensor so that the radio is effectively equivalent an Envoy. 

[franzz]

This is a really interesting thread. It seems that it is necessary to use ISS without console with US-versions of stations. My problem is that I have the european version of ISS which works within frequency band of 868,0-868,6MHz which is much more smaller than the US band (902-928MHz). I think the count of used hopping frequecies in EU version is smaller than in US because of the smaller band. In console diagnostic mode I can see that frequency index loops from 51 to 55. If I'm correct only 5 frequencies will be used? Is that right?
My 2nd big question to the radio experts refers to usage of 2 transmitter. In my configuration I have the ISS + 1 additional Temp/Hum sensor. In the console diagnostic mode I can see that current used frequencies of ISS and add. Temp/Hum. sensor are not synchronized because the current frequency index of both devices is not the same. Now to my question: Is the firmware from tridge able to handle different IDs from different transmitters at the same time? Another big question is: How could the console handle this different data streams which are not synchronized? It's absolute unclear for me how the frequency hopping could be done in this case? I hope someone could explain a few of my questions.

BR
franzz

[DeKay]

This is a really interesting thread. It seems that it is necessary to use ISS without console with US-versions of stations. My problem is that I have the european version of ISS which works within frequency band of 868,0-868,6MHz which is much more smaller than the US band (902-928MHz). I think the count of used hopping frequecies in EU version is smaller than in US because of the smaller band. In console diagnostic mode I can see that frequency index loops from 51 to 55. If I'm correct only 5 frequencies will be used? Is that right?

Sounds reasonable.  But nobody to my knowledge has sniffed the SPI bus on an EU unit to figure out what those frequencies actually are.

My 2nd big question to the radio experts refers to usage of 2 transmitter. In my configuration I have the ISS + 1 additional Temp/Hum sensor. In the console diagnostic mode I can see that current used frequencies of ISS and add. Temp/Hum. sensor are not synchronized because the current frequency index of both devices is not the same. Now to my question: Is the firmware from tridge able to handle different IDs from different transmitters at the same time? Another big question is: How could the console handle this different data streams which are not synchronized? It's absolute unclear for me how the frequency hopping could be done in this case? I hope someone could explain a few of my questions.

BR
franzz

I don't believe tridge has implemented reception from multiple devices.  He has discovered that the hop sequence is the same regardless of the channel.  What is also known is that the spacing between transmissions is different based on the channel ID.

I believe the console just sits on a frequency in setup mode and listens for the different transmitters that will eventually hop through that frequency.  Knowing the time of reception, the channel ID, and the hop sequence, it can predict the next transmission time and frequency from all the stations it knows about.  The transmissions are very short, so it has time to do so.  It is probably tricky to do this well (accounting for missed transmissions, deciding what to do when transmissions will collide, etc) but certainly doable.

[franzz]

I've tried to sniff the setting of frequency registers (FREQ_2A:FREQ_1A:FREQ_0A) that shows the hops between 5 different frequencies. (EU version frequeny band 868,0 - 868,6MHz) The count of 5 different frequencies match with the info from the console diagnostic mode which also loops within 5 different index (51-55). There is only one firmware version for different bands, so the US version index loops between 0..50 and the EU version between 51..55.
Now my question to the radio expert (DeKay) :
How can I calculate the frequency from the information of my register to check if the recorded register info is correct?

Index: 00, FREQ2: 3A, FREQ1: 19, FREQ0: 45
Index: 01, FREQ2: 3A, FREQ1: 1D, FREQ0: 45
Index: 02, FREQ2: 3A, FREQ1: 21, FREQ0: 45
Index: 03, FREQ2: 3A, FREQ1: 1B, FREQ0: 45
Index: 04, FREQ2: 3A, FREQ1: 1F, FREQ0: 45

BR
franzz

[DeKay]

I've tried to sniff the setting of frequency registers (FREQ_2A:FREQ_1A:FREQ_0A) that shows the hops between 5 different frequencies. (EU version frequeny band 868,0 - 868,6MHz)

Awesome!

The count of 5 different frequencies match with the info from the console diagnostic mode which also loops within 5 different index (51-55). There is only one firmware version for different bands, so the US version index loops between 0..50 and the EU version between 51..55.
Now my question to the radio expert (DeKay) :
How can I calculate the frequency from the information of my register to check if the recorded register info is correct?

With great difficulty.

Or do it the easy way.  Read this blog post of mine that should tell you what you need to know.  Specifically, look over the datasheet and check out the embedded spreadsheet that shows you where to find all of the formulas.

[franzz]

Read this blog post of mine that should tell you what you need to know.

 
Thanks for the link to the useful summary of frequency calculation!

I've calculated the frequecies to make a plausibility check. It seems that I've sniffed the registers correct. All the calculated frequencies are within the specified band for the EU version (868,0-868,6Mhz) and spread over the full bandwith.

Index: 00, FREQ2: 3A, FREQ1: 19, FREQ0: 45 --> 868066725Hz
Index: 01, FREQ2: 3A, FREQ1: 1D, FREQ0: 45 --> 868297125Hz
Index: 02, FREQ2: 3A, FREQ1: 21, FREQ0: 45 --> 868527525Hz
Index: 03, FREQ2: 3A, FREQ1: 1B, FREQ0: 45 --> 868181925Hz
Index: 04, FREQ2: 3A, FREQ1: 1F, FREQ0: 45 --> 868412325Hz

BR
franzz

[DeKay]

Thanks for the link to the useful summary of frequency calculation!

I've calculated the frequecies to make a plausibility check. It seems that I've sniffed the registers correct. All the calculated frequencies are within the specified band for the EU version (868,0-868,6Mhz) and spread over the full bandwith.

Index: 00, FREQ2: 3A, FREQ1: 19, FREQ0: 45 --> 868066725Hz
Index: 01, FREQ2: 3A, FREQ1: 1D, FREQ0: 45 --> 868297125Hz
Index: 02, FREQ2: 3A, FREQ1: 21, FREQ0: 45 --> 868527525Hz
Index: 03, FREQ2: 3A, FREQ1: 1B, FREQ0: 45 --> 868181925Hz
Index: 04, FREQ2: 3A, FREQ1: 1F, FREQ0: 45 --> 868412325Hz

BR
franzz

Cool.   

People playing with this stuff should keep in mind that these frequencies might vary a tiny bit from one console to another (I think).  I believe the console keeps track of the frequency errors it measures between itself and the ISS and applies that as an offset as it does its thing, so different combinations of consoles and ISS's will give slightly different frequencies.  Because of this, it is very likely that you would get slightly different numbers if you re-sniffed the register settings.