Author Topic: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions  (Read 63604 times)

0 Members and 1 Guest are viewing this topic.

Offline cmatteri

  • Member
  • *
  • Posts: 11
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #125 on: February 25, 2015, 10:05:53 PM »
Kobuki, I didn't know about the Envoy8X until fairly recently.  If I had, I would have used one and tried to discover the serial protocol it uses.  That said, my goal with the weather system I'm setting up is as much to learn and have fun as it is to monitor weather.

Here's a text figure that demonstrates how message 9 works.  Each vertical bar represents a message 9 packet.  The number between the vertical bars is the maximum wind speed that occurred between the two packets represented by the two adjacent vertical bars (this is not real data).
packet      0 1 2 3 4 5 6 7 8 9 ...
max wind    |1|1|2|3|2|3|1|2|1|1|1|1|1|1|1|1|1|1|1|1|1|1|
byte3         1 1 2 3 3 3 3 3 3 3 3 3 3 3 3 2 2 1 1 1 1 1
byte5[7:4]    0 0 0 0 1 0 1 2 3 4 5 6 7 8 9 8 9 0 0 0 0 0
              X X X X   M M X                   X X X X X


That should help explain how it works, but the question remains whether byte5[7:4] is actually useful for anything.  I think it could be used to only record gust readings that are new.  Gust readings would be recorded if byte5[7:4] is 0, or if packets were missed, if it was 0 for one of the missed packets.  We would record the gust readings for all the packets marked with X in the above figure (where M represents a missed packet).  There are two advantages to doing this:
1. It increases the likelihood that the gust reading in an archive record actually represents a max wind from that archive interval, rather than from the previous one.
2. The wind direction for the gust needs to come from byte 2 of some packet, and by using byte5[7:4] we can get a wind direction from within 30 seconds of the gust, whereas ignoring byte5[7:4], the gust direction could be as much as 10 minutes out of date.

Dekay, we were indeed missing something.  I found this note on page 72 of the CC1021 datasheet:
"Note: The RX frequency deviation should be close to half the TX frequency deviation for GFSK at 100 kBaud data rate and below. The RX frequency deviation should be close to the TX frequency deviation for FSK and for GFSK at 100 kBaud data rate and above."

OTOH, the CC1101 datasheet specifies:
"the DEVIATN register specifies the expected frequency deviation of incoming signals in RX and should be the same as the TX deviation for demodulation to be performed reliably and robustly."

I believe you miscalculated the TX deviation for the CC1021.  The formula for the 915 MHz band is fdev = FREF * TXDEV_M * 2^(TXDEV_X - 15).  Plugging in FREF = 7.3728 MHz, TXDEV_M = 11 and TXDEV_X = 2 from your table we get
deviation = 7.3728 MHz * 11 * 2^(2 - 15) = 9.9 kHz.  You must have used the formula for the 402 - 470 MHz band.  Your calculation for the RX deviation looks right.

Thus for the CC1101 I believe 9.5 kHz is the correct deviation.

As for the channel filter bandwidth, DEC_DIV[2:0] is set to 5, so ChBW = 307.2 / (5 + 1) kHz = 51.2 kHz.  That would require a roughly 4 ppm or lower crystal, which is possible.

I can't find the tolerance of the crystal used on the RFM69W, but given the possibility that its tolerance is higher than 4 ppm, you may want to increase the channel bandwidth in the DavisRFM69 code.

Offline DeKay

  • Forecaster
  • *****
  • Posts: 399
    • Mad Scientist Labs
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #126 on: February 25, 2015, 11:19:37 PM »
Dekay, we were indeed missing something.  I found this note on page 72 of the CC1021 datasheet:
"Note: The RX frequency deviation should be close to half the TX frequency deviation for GFSK at 100 kBaud data rate and below. The RX frequency deviation should be close to the TX frequency deviation for FSK and for GFSK at 100 kBaud data rate and above."

OTOH, the CC1101 datasheet specifies:
"the DEVIATN register specifies the expected frequency deviation of incoming signals in RX and should be the same as the TX deviation for demodulation to be performed reliably and robustly."

I believe you miscalculated the TX deviation for the CC1021.  The formula for the 915 MHz band is fdev = FREF * TXDEV_M * 2^(TXDEV_X - 15).  Plugging in FREF = 7.3728 MHz, TXDEV_M = 11 and TXDEV_X = 2 from your table we get
deviation = 7.3728 MHz * 11 * 2^(2 - 15) = 9.9 kHz.  You must have used the formula for the 402 - 470 MHz band.  Your calculation for the RX deviation looks right.

I was indeed using the calculation for 915 MHz, but I totally botched the formula in the spreadsheet.  It was a fluke that the number came out looking reasonable.  Sigh.  Your number of 9.9 kHz is correct and my spreadsheet is now updated.

BUT, I think this explains why earlier attempts by rdsman, kobuki, and me to get our own transmitters going had failed - our transmit deviation was too low!  However, it does look like that got working in this thread.  I did a quick search here but couldn't find the critical incantation that got this working where others had failed.  Can somebody point me to this?  Was it the transmit deviation or something else???

As for the channel filter bandwidth, DEC_DIV[2:0] is set to 5, so ChBW = 307.2 / (5 + 1) kHz = 51.2 kHz.  That would require a roughly 4 ppm or lower crystal, which is possible.

I can't find the tolerance of the crystal used on the RFM69W, but given the possibility that its tolerance is higher than 4 ppm, you may want to increase the channel bandwidth in the DavisRFM69 code.

What would be preferable is to acquire the carrier in each channel and determine the offset in each using a wide bandwidth, then narrow the bandwidth and periodically recalculate the offset.  This would give a better signal to noise ratio and prove more reliable over longer distances.

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #127 on: February 26, 2015, 04:19:28 AM »
@cmatteri: thanks, it confirmed what I've understood about that index. I think the most importand bit is to be able to more accurately determine the direction of a gust. Losing packets happens all the time, but it can help reconstructing some missing ones as well as you noted. OTOH, for "direct" reception it has little importance unless archiving and aggregations are calculated for the archive perionds, like the original console does, for example.

Also you discovered the Envoy 8x, which is AFAIK not very popular since its software support is rather poor. I'm trying not to look as an advocate for them but it's often left unnoticed, so have a look at the Meteostick and the Weatherbridge. These are 3rd party solutions. They offer a lot more than the factory devices, including arbitrary station reception configurations.

@DeKay, cmatteri: DeKay spent so much time and so many blog posts were devoted to the subject that I never thought that such an important parameter as the deviation frequency might be wrong. But it seems to be off for transmissions only. I'll definitely have a look very soon and try to find out if it helps with my RFM69 transmitter.

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #128 on: February 27, 2015, 12:29:50 PM »
Gentlemen, I have good news to report. I've set the deviation to 9.9 kHz on my anemometer transmitter sketch and now the Envoy is receiving it quite stably. A big thanks goes to cmattery for unknowingly solving this mistery with my transmitter. My code is still emulating the Davis packet format using data bytes only but I'll probably be able to switch it back to standard automatic packet assembly mode. I'll leave it on for a few hours to test the transmission quality. I've set my Envoy to handle it as an extra temp sensor.
« Last Edit: February 27, 2015, 04:46:15 PM by kobuki »

Offline DeKay

  • Forecaster
  • *****
  • Posts: 399
    • Mad Scientist Labs
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #129 on: February 27, 2015, 02:09:07 PM »
Gentlemen, I have good news to report. I've set the deviation to 9.9 kHz on my anemometer transmitter sketch and now the Envoy is receiving it quite stably. A big thangs goes to cmattery for unknowingly solving this mistery with my transmitter. My code is still emulating the Davis packet format using data bytes only but I'll probably be able to switch it back to standard automatic packet assembly mode. I'll leave it on for a few hours to test the transmission quality. I've set my Envoy to handle it as an extra temp sensor.

 \:D/

This is great!  Another mystery solved.  Homebrew Temperature / Humidity sensors have just become pretty trivial.  Next we'll need to work on the low power aspects to see if we can get a remote sensor working off a solar cell and some kind of supercap / battery backup.  This should be reasonable to expect given that T/H are transmitted less often than wind, etc from the ISS, so the sensor can spend more time sleeping.

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #130 on: February 27, 2015, 02:30:11 PM »
Yeah, I'm already thinking of making a Davis-compatible T/H transmitter with one of my Moteinos, for the fun of it. Sadly the extra temp sensors are only represented as integer digits, no fractional part. The show-stopper however can be the requirement for precise transmission timing. To be able to make the 328p sleep as much as possible, the WD timer can be used for periodic wake-ups and subsequent sample/transmit procedures, but it's very imprecise. An external clock source would probably be needed, such as an RTC clock chip or maybe a low-power 32k crystal. The problem could indeed be alleviated by using solar power and a chargeable battery or similar.

Regarding the radio parameters, I think I've missed something, since cmattery said: "Thus for the CC1101 I believe 9.5 kHz is the correct deviation." - though the calculated value is 9.9k. Is it a CC1101 limitation?

Also, the mentioned bandwidth values and exact hopping frequency values will still need to be tuned, although I'm quite satisfied with the capabilities of the RFM69 receiver as it is now. But I guess reception can still be improved as it has also been mentioned. The RFM69 being weaker than the TI at reception could also be caused by these slightly off parameters and the wrong Fdev value.

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #131 on: February 28, 2015, 01:50:44 PM »
Since a few people asked me in PMs for code and I wanted to upload it anyway, I thought it's better to put it all on GitHub for easier access. I removed my fork of DeKay's old lib, which is my radio code is based on, since we're going in different directions as far as the rest of the code goes. I still want to contribute to existing stuff, of course. My lib is more like an attempt at a general, bare-bones solution to jump start other projects.

Have fun: https://github.com/kobuki/VPTools

Offline DeKay

  • Forecaster
  • *****
  • Posts: 399
    • Mad Scientist Labs
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #132 on: February 28, 2015, 02:05:06 PM »
Since a few people asked me in PMs for code and I wanted to upload it anyway, I thought it's better to put it all on GitHub for easier access. I removed my fork of DeKay's old lib, which is my radio code is based on, since we're going in different directions as far as the rest of the code goes. I still want to contribute to existing stuff, of course. My lib is more like an attempt at a general, bare-bones solution to jump start other projects.

Have fun: https://github.com/kobuki/VPTools

Makes sense.  I'm totally cool with this  :-)

Offline cmatteri

  • Member
  • *
  • Posts: 11
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #133 on: March 03, 2015, 10:50:49 PM »
Is it a CC1101 limitation?

Yes.  The next highest possible deviation on the CC1101 (with a 26 MHz crystal, which is what the Vue receiver uses, based on the FREQ register values sniffed by rdsman) after 9.521 kHz is 10.315 kHz.  9.521 kHz is what the Vue receiver uses.


Offline docbee

  • Forecaster
  • *****
  • Posts: 853
    • smartbedded
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #134 on: March 14, 2015, 05:39:29 PM »
Sniffing the air for Vantage packets, I tried to make use of re-transmitted packets a Vantage console does send out. When I get it correctly these packets have "FF 00" following the two byte CRC. What stumbles me is, that I can see these packets only rather sporadic. I would have thought the retransmit follows the same frequency hopping as the other transmitting sensors.

Was anyone successful in reading retransmitted data periodically?
founder of smartbedded.com - home of meteohub, meteoplug, meteobridge, meteostick

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #135 on: March 14, 2015, 06:02:58 PM »
Yeah. See this thread.

The console has a simple repeater function that's not the equivalent of the stand alone repeater functionality. It simply retransmits the received packets of a single channel on another channel, using the appropriate timing and schedule. You cannot just repeat the packets with the same timing on the same channel. This will confuse the receiver. In essence retransmitted packets must follow the standard hopping pattern, sequence and timing wise too. Naturally the CRC needs to be recalculated when changing the channel ID.

You can ignore those 0xff00 bytes for this case.

EDIT: I realised you only talk about the reception. But yes, it follows the same pattern. You should be able to receive it per usual, on the retransmitted channel.
« Last Edit: March 14, 2015, 06:06:02 PM by kobuki »

Offline docbee

  • Forecaster
  • *****
  • Posts: 853
    • smartbedded
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #136 on: March 14, 2015, 06:21:36 PM »
Dekay has written near the beginning of this thread that re-transmitted packets have 3x 0x55 instead of 2x 0x55 as preamble which might explain why my RFM69 code is not receiving retransmitted data accordingly. Do you really succeed reading retransmitted data on a reliable basis?
founder of smartbedded.com - home of meteohub, meteoplug, meteobridge, meteostick

Offline DeKay

  • Forecaster
  • *****
  • Posts: 399
    • Mad Scientist Labs
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #137 on: March 14, 2015, 06:37:34 PM »
Dekay has written near the beginning of this thread that re-transmitted packets have 3x 0x55 instead of 2x 0x55 as preamble which might explain why my RFM69 code is not receiving retransmitted data accordingly. Do you really succeed reading retransmitted data on a reliable basis?

Actually, it was C5250 that said it was 3x 0x55.  I disagreed and was able to back it up.   :-)

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #138 on: March 14, 2015, 06:37:58 PM »
I didn't remember that. Also, there are many small corrections he made or I discovered since then. It's several years old info. In essence, a retransmitted packet is indistinguishable from a non-retransmitted packet. There is no special setting needed on the receiver. Something else must be off. It has recently been discovered that the Fdev is wrong in his original code. It must be doubled for correctness on the RFM69. I'm not sure you've introduced that fix in your receiver, but it might help with the Vue.

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #139 on: March 14, 2015, 06:39:43 PM »
Actually, it was C5250 that said it was 3x 0x55.  I disagreed and was able to back it up.   :-)

Right. Though it shouldn't make a difference in reception. I'm thinking the Fdev correction might help.

Offline rdsman

  • Senior Contributor
  • ****
  • Posts: 249
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #140 on: March 15, 2015, 10:59:57 AM »
Quote
Was anyone successful in reading retransmitted data periodically?

docbee:

The retransmitted data from the Vue follows the same hop sequence, but is behind the ISS hop index numerically.  (At least it starts out that way.)  This is shown in the attached picture.  On the left is the ISS data, on the right is the repeated data.

The difference in the hop index doesn't stay the same amount because they roll over past 51 differently.





 
« Last Edit: March 15, 2015, 11:08:14 AM by rdsman »
Ray

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #141 on: March 15, 2015, 11:06:18 AM »
Well, rather, they are interleaved and after some time they interleave differently, in such a way that one that is behind the other now, will be ahead of the other later. And if you think it a little further, in the long run there isn't something you can call first or second or whatever. The point is, use the known jumping algo, always.

Offline cmatteri

  • Member
  • *
  • Posts: 11
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #142 on: June 29, 2015, 02:30:50 AM »
Hi,

My Davis receivers have always needed to jump to a channel significantly earlier than I would have expected based on the timings used by the Davis protocol in order to not miss packets and lose track of the transmitter.  Several months ago I figured out why (I thought I posted it here but apparently I didn't).  The problem is that many Arduinos, including the Moteino, use ceramic resonators to clock the microcontroller.  These have tolerances of up to 0.5 %, plus variance from temperature.  For the 2.5 - 3 s period of Davis transmitters, this translates into up to 12 - 15 ms error.  A typical crystal could have a tolerance of 25 ppm, which would give an error of 75 us for the 3 s period.  The Davis devices probably use crystals, and probably assume crystal accuracy in their designs.  Thus for those who are building transmitters or repeaters it may help to use a microcontroller with a crystal.

edit: You might be able to ask Felix Rusu to build a Moteino with a crystal.  The connections are the same and I believe you can find a crystal with the same dimensions as the resonator he uses.

« Last Edit: June 29, 2015, 02:36:07 AM by cmatteri »

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #143 on: June 29, 2015, 05:27:00 AM »
I never had this problem. As soon as I receive a packet on the current channel, I skip to the next right away and listen on it until a packet arrives or times out with a configurable interval. IIRC the accuracy of the resonators used in the Moteionos are better than that. Not by paper, but by experience.

Offline cmatteri

  • Member
  • *
  • Posts: 11
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #144 on: June 29, 2015, 02:01:27 PM »
Resonators are fine for receivers if you use them like you are.  There's a small chance of intercepting someone else's transmitter with that scheme though.  Resonators may even be fine for transmitters or repeaters, I haven't been paying too close attention to projects that use those, but if a resonator board isn't working as a transmitter that's a possible reason why.

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #145 on: June 29, 2015, 02:19:15 PM »
Well, yes, it's an inavitable drawback and can happen with a more accurate crystal and guarding timeframe, albeit with a lot less probability. For the purposes and intents I've made my receiver it's perfectly fine. It's been running it 24x7 for more than a year without much problems. Sadly I don't have much time to play around with these projects since winter :(

I think there are a few folks with working DIY transmitters/repeaters here, maybe they'll say hi.

Offline ct

  • Contributor
  • ***
  • Posts: 101
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #146 on: July 02, 2015, 12:22:13 AM »
I have been unable to get the VP2 console to reliably sync to a RFM69 transmitter, but my microcontroller has a crystal.

I have posted more information in this thread http://www.wxforum.net/index.php?topic=24567


Offline blacklistedcard

  • Member
  • *
  • Posts: 44
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #147 on: August 18, 2015, 09:12:13 PM »
I have my RFM69HW working picking up my ISS unit data.   I have a couple of problems.   The barometric readings are not working.   The inside temperature is also not working.

What are the defines for both barometer and inside temperature parameters?

// Davis packet types, also defined by kobuki
#define VP2P_UV             0x4 // UV index
#define VP2P_RAINSECS       0x5 // seconds between rain bucket tips
#define VP2P_SOLAR          0x6 // solar irradiation
#define VP2P_TEMP           0x8 // outside temperature
#define VP2P_WINDGUST       0x9 // 10-minute wind gust
#define VP2P_HUMIDITY       0xA // outside humidity
#define VP2P_RAIN           0xE // rain bucket tips counter

Offline C5250

  • Forecaster
  • *****
  • Posts: 840
    • Local weather
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #148 on: August 18, 2015, 10:19:38 PM »
Barometric and indoor temp are not transmitted by the ISS, they are local parameters to a Console/Envoy.
Precious little in your life is yours by right and won without a fight.

Offline blacklistedcard

  • Member
  • *
  • Posts: 44
Re: A Walk on the Wireless Side: Deciphering ISS to Console Transmissions
« Reply #149 on: August 18, 2015, 10:54:10 PM »
Barometric and indoor temp are not transmitted by the ISS, they are local parameters to a Console/Envoy.

ahhhh...  Yeah, I guess that's why everyone is using a BMP085.   Feeling stupid right now.  Thanks for point that out...