Author Topic: Transmitting data to a Vantage Pro2 console with RFM69  (Read 21532 times)

0 Members and 1 Guest are viewing this topic.

Offline ct

  • Contributor
  • ***
  • Posts: 101
Transmitting data to a Vantage Pro2 console with RFM69
« on: December 09, 2014, 06:39:39 PM »
I have been wanting to send weather data to a Vantage Pro2 console from a DIY weather station.

Forum member kobuki has informed me that he has successfully done this.

Kobuki's AnemometerTX is nicely structured and well documented.  I does most of what I need and can certainly be the basis of my own project.

Reading DeKay's blog posts and analysing his DavisRFM69 source has helped me learn a lot about the Davis protocol.

I bought 2 915 MHz RFM69HW modules and connected them each to an Arduino.  On one I run the ISSRx sketch to monitor received data.  On the other I run a slightly modified version of AnemometerTX.  As this Arduino doesn't actually have an anemometer attached, I am using a fixed number for the wind degree and incrementing a number for the wind speed.

Starting up each Arduino, I can see the packets being successfully sent and received.  Occasionally there in a CRC error on the receiver but most packets have the correct CRC.

I've defined DAVIS_FREQS_US in DavisRFM69.h and can see the sender and receiver cycling through the 50 frequencies and the various packet types - 80, e0, 50, 40 ... etc
Each packet contains wind degree and speed data in byte 1 and 2.  Bytes 3,4 and 5 are zero.

Having successfully completed this initial test, I bought a Vantage Pro2 console (US version).  I set the console to station 1 and set TX_ID to 0 in AnemometerTX source - one less than what the console displays.

However, I am not receiving any data whatsoever on the VP2 console.  The status displays 'R' for a while then 'L' a few minutes later.  It never displays 'X' to indicate a received packet.  When I enter the console diagnostics (TEMP+HUM) it usually shows zero for each field.  Occasionally it may show a packet with failed CRC, but it does this even when the Arduino transmitter is off.

Any ideas on what I can try?

I can provide sample output and source code if anyone is interested.

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #1 on: December 09, 2014, 07:18:17 PM »
I'm glad you've been able to use my TX sketch. I've also been successful with a 2-Moteino setup and it ran for weeks quite stable, but unfortunately haven't been able to test it on my Envoy yet (currently that's all I have, I have no use for a console).

There's a small difference in the structure of the radio packets between the Davis/TI transmitter and the RFM69. The TI chip transmits about 4 ms of the logic "1" carrier before the preamble and after the end of the payload. The RFM69 starts right with the preamble. This small difference might be enough to cause problems.

I currently have no idea how to couter this (if this is what causes the problems at all), but changing the preamble might help. IIRC I had to make it a byte shorter/longer for the Moteino to reliably pick up the signal, it's possible that using a longer or even shorter one could help with the Davis receiver. I'll also try to play around with this a bit when time permits, possibly at the weekend.

My other idea is that the sketch might not send the correct "no sensor" data in the packets and the FW of the console somehow verifies that instead of ignoring everything but the wind data bytes. The transmit sequence itself is definitely good. It might worth trying to apply the correct "no sensor" values in all packets, observing the ISS where all cables are unplugged except for the wind.

Offline linuxfreak

  • Hamilton Central Mountain Weather
  • Senior Contributor
  • ****
  • Posts: 188
  • Wink, wink!
    • Hamilton Central Mountain Weather
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #2 on: December 10, 2014, 01:02:11 AM »
How did you get past the
Code: [Select]
AnemometerTX:103: error: 'class DavisRFM69' has no member named 'setTxMode'

error???  #-o ](*,)

The code won't compile with that.

Code: [Select]
void setup() {
Serial.begin(19200);
pinMode(POT_VCC, INPUT); // hi-Z
pinMode(POT_BLIND_VCC, INPUT); // hi-Z
lastSwitchSensed = -WIND_DETECT_THRESH;
rotationPeriod = WIND_DETECT_THRESH;
lastTx = micros();
attachInterrupt(WIND_INTERRUPT, windInterrupt, FALLING);
radio.initialize();
radio.setTxMode(true); // enable tx params, default is false, call this before the first setChannel()
radio.setChannel(0); // Frequency / Channel is *not* set in the initialization. Do it right after.
#ifdef IS_RFM69HW
radio.setHighPower(); // uncomment only for RFM69HW!
#endif
seqIndex = 0;
randomSeed(analogRead(A1));
}

It's the line radio.setTxMode(true); // enable tx params, default is false, call this before the first setChannel()

George
George
Davis VP2/FARS, VVP, WD, WL, WSWin, Cumulus, NexStorm, StrikeStar, NSLog, XPort(GPS), WASP2, DigitalAtmosphere, ScannerCast(WUradio), Intel Atom N330 dual-core, 2Gig ram, Windows XP Home SP3  #-o
CWOP - DW3112, PWS & WU - IONHAMIL2, AWEKAS - 5112, COWN, WML - WD01901

Offline ct

  • Contributor
  • ***
  • Posts: 101
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #3 on: December 10, 2014, 01:32:50 AM »

It's the line radio.setTxMode(true); // enable tx params, default is false, call this before the first setChannel()


You need to use kobuki's DavisRFM69.cpp, as it contains setTxMode.

I also needed to get RFM69registers.h from LowPowerLab github repository and add back some of the missing defines that were taken out in a recent version of that file.

Offline ct

  • Contributor
  • ***
  • Posts: 101
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #4 on: December 12, 2014, 01:07:34 AM »
it's possible that using a longer or even shorter one could help with the Davis receiver.

I've had no success when changing REG_PREAMBLELSB to different values.

I tried also using proper values for bytes 3, 4 and 5 in the various packet types.

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #5 on: December 12, 2014, 05:09:10 AM »
And what if you set it longer? Like 6..10 or more. It could be possible to set the preamble length zero and emulate the starting carrier, preamble and sync word using a long sync byte sequence, IIRC the chip can use 16..32 bytes or so. It's a hack though and might or might not work, if that's the problem at all.

Offline ct

  • Contributor
  • ***
  • Posts: 101
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #6 on: December 15, 2014, 10:16:20 PM »
I've tried REG_PREAMBLELSB up to 12, but haven't been able to get the console to see any packets.


Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #7 on: January 02, 2015, 06:20:59 PM »
I've modified DavisRFM69.cpp to emulate the original transmitter as closely I can remember the packet format of it. The important bits are in sendFrame() and setTxMode(). It turns off the sync word and the preamble generation of the packet engine of the RFM69, and emulates those in the data packet instead, adding a "carrier on" signal before the preamble.

I can't try it ATM, if you'd like to try it, you can find it as an attachment to this post. I think I'll get around finishing/testing it this weekend.

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #8 on: January 03, 2015, 12:49:21 PM »
I got around testing it a bit and it works, sorta. Probably is in need of some more tweaking but seems to be stable enough to give it a go using real equipment. I was setting it to act as a standalone T/H transmitter transmitting fake data, using internal station ID 2. I've set my envoy accordingly and it was seeing it as a station with the proper ID, and shown up in the output of the STRMON command.

A couple notes:
 - more preamble bytes made reception stabler so I'm using way more than the necessary 4
 - it does need the leading/trailing "carrier emulation" (see in the code)
 - too strong a signal can make reception flaky; I had to cover it from one side with a metal sheet to make it weaker

Unfortunately it has a drawback: the Moteino/RFM69 receiver is unable to pick up the signal. With less preamble bytes it was. So I might have succeeded in making it work for an original receiver, but it appears that fixing the packet format is not the end of it. Something more or else is needed. Maybe the radio parameters need to be tweaked more. DeKay might have some idea, I'll try to poke him. The fact that a lot longer preamble is needed is hinting that something is not not quite right with the radio parameters.

I'm attaching the current code.
« Last Edit: January 03, 2015, 01:11:06 PM by kobuki »

Offline DeKay

  • Forecaster
  • *****
  • Posts: 399
    • Mad Scientist Labs
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #9 on: January 03, 2015, 07:26:31 PM »
I got around testing it a bit and it works, sorta. Probably is in need of some more tweaking but seems to be stable enough to give it a go using real equipment. I was setting it to act as a standalone T/H transmitter transmitting fake data, using internal station ID 2. I've set my envoy accordingly and it was seeing it as a station with the proper ID, and shown up in the output of the STRMON command.

A couple notes:
 - more preamble bytes made reception stabler so I'm using way more than the necessary 4
 - it does need the leading/trailing "carrier emulation" (see in the code)
 - too strong a signal can make reception flaky; I had to cover it from one side with a metal sheet to make it weaker

Unfortunately it has a drawback: the Moteino/RFM69 receiver is unable to pick up the signal. With less preamble bytes it was. So I might have succeeded in making it work for an original receiver, but it appears that fixing the packet format is not the end of it. Something more or else is needed. Maybe the radio parameters need to be tweaked more. DeKay might have some idea, I'll try to poke him. The fact that a lot longer preamble is needed is hinting that something is not not quite right with the radio parameters.

I'm attaching the current code.

I meant to look at this myself over the Christmas break but got sidetracked on to other things.  Glad you have taken a look at this.

The CC1021 used in the console doesn't have the automatic packet detection stuff that the RFM69 / CC1110 have.  There is a whole-lot of bit-banging going on.    My initial investigation on captures I did a while back shows that the carrier is up for around seven bytes of data before the preamble and sync bytes start.  My guess is there might be some manual carrier detection or AFC going on in the console while the carrier is first up.  My investigation also showed that this data was different for each packet that was sent.  My suspicion is that this data is not important and could be garbled.

Here is a way (I think) that the transmitter can be better emulated while still remaining compatible with the Moteino.
- Use the RFM69 Automodes as discussed in Section 4.4 of the datasheet.  Set the registers (RegDataModul 0x02 and RegFifoThresh 0x3c) so the Moteino transmits when all of the bytes listed below have been dumped into the FIFO.  I'm sure there are other registers involved too, but this is as far as I've gotten so far.
- Send around seven bytes to the FIFO.  These should NOT look like the preamble.  I think all 0xFF might work.  If not, throw in an odd zero bit here and there.
- Send the documented number of preamble bytes to the FIFO.
- Send the documented number of sync bytes to the FIFO
- Send the payload to the FIFO
- Send the manually calculated CRC to the FIFO.  At this point, the number of bytes in the FIFO will be what was programmed in the first bullet and the data will be sent.
- Hop to the next frequency.  Wait the appropriate amount of time before transmitting again.
- Rinse, lather, and repeat.

This way the Moteino shouldn't be confused because it doesn't see a valid preamble until you send the real thing.  Everything should just work.  I think.  Shouldn't be too tricky to implement, either.

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #10 on: January 03, 2015, 07:54:48 PM »
Thanks for looking into this. However, what I'm doing is essentially the same you just described, see my #7 post earlier. I'm turning off preamble and sync generation and emulate them using data bytes, including the starting/ending carrier stuff (see attached code). I also set the FIFO threshold as you suggested, and all is working, except the RFM69 can't pick up the packets when the Envoy can, so something's still off. I've started with the documented amount of bytes for the packet sections but it just doesn't work out of the box so I started changing byte counts until the Envoy started seeing my packets.

The transmiter is nicely working when the receiving end is another Moteino, even with the standard packet automagic of the RFM69. My moteino receiver was able to simultaneously receive all 3 transmitters I have (ISS, SAT and a "T/H sensor emulator" I set up for testing). And it has been working since last Spring when I wrote the AnemometerTX code, so no problems on that part, really.

So I think that either the radio parameters aren't spot on for TX, or some bit shuffling is in order in the packet... I think we should concentrate on these unless someone has a better idea.
« Last Edit: January 03, 2015, 07:57:27 PM by kobuki »

Offline ct

  • Contributor
  • ***
  • Posts: 101
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #11 on: January 03, 2015, 10:16:32 PM »
Thank you very much for this updated DavisRFM69.cpp.  I have been able to get some values to appear on the VP2 console, although for some reason the console is reporting a large number of 'bad' packets.  Only a small percentage of packets, < 10%, are listed as being good packets.

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #12 on: January 04, 2015, 06:43:16 AM »
Yeah, as I mentioned, it's far from perfect yet. Try moving the transmitter farther or use some metallic object to shield the signal so it gets a little weaker (I think output power level setting is not yet available in this code base), that has helped me a lot. This shows that the radio transmission is probably not shaped perfectly for the TI chip, but this is just guesswork so far. I'll test it some more sometime next week, but unfortunately while I test I have no weather uploads so I need to make my testing sessions short...

Offline rdsman

  • Senior Contributor
  • ****
  • Posts: 249
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #13 on: January 04, 2015, 12:00:12 PM »
DeKay, ct & kobuki:

I thought that I would share some of my experimentation results with you.  I don't have
a VP2, Moteino or a RFM69.  So you can ignore this post if you like!

I have a Vue, several RFBees and two of these:

https://www.anaren.com/sites/default/files/Part-Datasheets/A1101R09C.pdf

Preamble

The recommended as well as the default value for the CC1101 is 4 bytes.  I achieved around 50% reliability using this value (module to module).  Using 8 bytes, the modules would run for over 30 days without missing a message.  I don't recall ever receiving a single message on my Vue using 4 bytes of preamble.  I can transmit to the Vue for months using 8 bytes.

Message Length

The message length is 10, not 8.  The two trailing 0xFF's that Kobuki has added are in fact transmitted by the ISS and clocked out of the CC1101 along with the rest of the data on the receiving end.  If I put the Vue console in retransmit, it sends 0xFF and 0x00 as the two trailing bytes.  These two bytes may help solve the CRC mystery when using repeaters.

Message Sequence

The Vue appears to keep track of the message sequence.  If I send only a few messages, the Vue will acquire sync, but gives up pretty quickly.  If I send all of the messages, in the correct order, the Vue will track for months.

The Anaren modules are typically 6 to 8 dBm more sensitive than the Vue.  On some channels they are a whopping 15 dBm greater than the RFBee!

Maybe some of this will help you guys out............


Ray

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #14 on: January 04, 2015, 12:10:38 PM »
Thanks rdsman, it was very helpful. Your own findings confirm mine regarding the preamble length, in that the default 4 bytes are not sensed by the Davis/TI receivers. Mentioning the other transmitted values at the beginning and at the end of the packet are also indeed useful. I'll try to play around with those a bit sometime later. The comment on the repeater is an interesting one, but I think we should only consider payload that's included in the CRC calculation, at least for the original ISS/VUE packets that are essentially identical, even the consoles are interchangeable IIRC (as far as reception goes - not in functionality).

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #15 on: January 04, 2015, 04:52:43 PM »
Using rdsman's findings I played with the various packet sections a little more but I haven't had much success. It's possible that the RFM69 modules are more forgiving as far as reception goes and I still think that a few parameters might be a little off for transmitting. I find it a little weird that while reception can be tailored to be practically perfect, transmission on the other hand gives us a headache...

Offline rdsman

  • Senior Contributor
  • ****
  • Posts: 249
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #16 on: January 06, 2015, 09:26:33 AM »
Kobuki:

A suggestion:

Do away with the leading 0xFF's and extend the preamble only.  See if this gets the Moteino back to receiving as it should.

Ray

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #17 on: January 06, 2015, 09:40:29 AM »
I did that too. The leading 0xff part needs to be there or else the Envoy doesn't pick up a single packet... It would be really nice if someone with the proper equipment could analyse the radio parameters somehow for both transmitters.

Offline rdsman

  • Senior Contributor
  • ****
  • Posts: 249
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #18 on: January 10, 2015, 10:48:59 AM »
Kobuki:

I have some pretty good test equipment.  Unfortunately, the only thing that I have that can receive and decode the message from the ISS is the CC1101.  There are two examples attached. 

Vue_Capture.gif is the data being clocked out of the CC1101 by the Vue (taken from the expansion connector).  You can see the eight data bytes and two FF's are the actual packet.  RSSI (1D) and LQI(AE) are appended to the end.  This is why that I use 10 bytes for my packet length.

Raw_Data.gif was captured from the Anaren module by having it programmed to receive just the raw data from the ISS.  You can see that it only gets two bytes of Preamble and then the Sync Word.

If I can capture any thing that helps you, I would be glad to attempt it.  (Time permitting)

Ray

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #19 on: January 10, 2015, 11:36:34 AM »
Rdsman, thanks very much for looking into this.

The raw packet shows pretty much the well-known packet structure at the beginning, with 2 preamble bytes received instead of the documented 4 bytes. Most likely all 4 ones are transmitted but the Anaren module only picks up the last 2. But it seems just fine for perfect reception. Were ever some leading/trailing 0xFF bytes or not, it's not shown, unfortunately.

The bytes seen on the first capture are from the "expansion connector" which I'm not familiar with, but I assume it has some extra bytes added by the Vue at the beginning and at the end of the packet: 0x0c and 0xFFFF1DAE, respectively. 0x0c might be a length byte, since exactly 12 bytes follow it.

Would it be possible to see some "even more raw" data? Something like the demodulated logic-level or likewise interpretable output from the receiver, if it has a possibility to do that at all. So we might have a chance to see what's exactly transmitted on-air from your own transmitter, the Davis transmitter and we can compare the two. DeKay has dissected it already and I've been through his related blog posts several times but emulating the behaviour I see there doesn't work reliably at the moment.

Offline rdsman

  • Senior Contributor
  • ****
  • Posts: 249
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #20 on: January 10, 2015, 03:28:54 PM »
Quote
Would it be possible to see some "even more raw" data? Something like the demodulated logic-level or likewise interpretable output from the receiver

I'll work on this.

Maybe I should better explain the Vue Capture for now.  The expansion connector is where the data logger would plug in.  There you have direct access to MOSI, MISO and SCK.  You don't have SS for the CC1101, so you just have to capture data and then scroll through it until you find commands that make sense.

The first 0xFF that you see on the top is actually a command to the CC1101.  It is the RX_FIFO register address (0x3F) or'ed with READ_BURST (0xC0), which just happens to equal 0xFF.

The corresponding 0x0C is the returned status byte from the CC1101.  Which is in fact the number of data bytes in the RX_FIFO.

The remaining 0x00's are the SPI actually clocking the data out of the CC1101.  All of the data through the two 0xFF's was actually received by the CC1101 from the ISS.  This was a total of 10 bytes.  In the Vue, the CC1101 is programmed to automatically append the Received Signal Strength Indicator (0x1D) and the Link Quality Indicator (0xAE) bytes to the end of the packet.  That is why the reported packet length was 12 and not 10.

Hope this makes more sense.......
« Last Edit: January 10, 2015, 03:59:00 PM by rdsman »
Ray

Offline DeKay

  • Forecaster
  • *****
  • Posts: 399
    • Mad Scientist Labs
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #21 on: January 10, 2015, 11:32:16 PM »
This might help as well...

Attached is the Openbench Logic Sniffer project file featured in this blog post where I sniffed the DIO data on the CC1021.  The software to load and display it can be downloaded here (you need Java installed but don't actually need any logic sniffing hardware hooked up).  Once the software is installed and the project file is loaded, you should be able to recreate the screen captures in my blog post, with the markers in place and everything.  There are two receive bursts captured in this file.

What sucked about the VP2 is that the commands to the CC1021 were on MISO/MOSI expansion bus but the data and clock bits were just wired to the micro.  I had to probe the AVR with a couple sewing needles with my logic analyzer probe attached!

Offline kobuki

  • Forecaster
  • *****
  • Posts: 838
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #22 on: January 11, 2015, 06:25:53 AM »
Thanks, DeKay. I can clearly see all the packet sections like the preamble, sync word, and the 8 data bytes. It appears though that the VP2 is programmed a little differently than the Vue since I can't recognise the RSSI+LQI bytes at the end. However, there are some extra bits at the end, for 13 clocks if I counted right, which is weird (for me at least).

At the beginning, before the preamble there are about 5 bytes worth of data, including 2 bytes of 0xff right before the preamble. These might be the 0xff bytes on air, but I wonder about the rest. These could be coming out programmatically, or it's just the chip adjusting to the carrier and decoding/outputting garbage until it locks on the preamble (I admit, putting garbage on the bus makes little sense), I personally have no idea.

I'll have a thorough look at the RFM69 datasheet. IIRC it's able to put out the demodulated raw signal on one of its I/O pins (DIO2), this can mainly be used for OOK signal decoding but it's put in "data output" mode in packet mode too, so there's a worthy chance I'd be able to see the either raw demodulator logic-level output or the cleaned/reclocked result bits. I'll need to hack one of my moteino receivers by jumpering DIO2 to an Atmega data pin.

Offline DeKay

  • Forecaster
  • *****
  • Posts: 399
    • Mad Scientist Labs
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #23 on: January 11, 2015, 11:01:36 AM »
You'll want to dig into that data while referring to the spreadsheet in this blog post which should give some context into what the console is doing and when.  They do some strange and interesting things, like writing to three "test only" registers just after setting the frequency.

Offline rdsman

  • Senior Contributor
  • ****
  • Posts: 249
Re: Transmitting data to a Vantage Pro2 console with RFM69
« Reply #24 on: January 12, 2015, 10:14:52 AM »
Haven't found a way to just capture raw bits so far.  But I did put my Vue
in retransmit mode just to see what it is up to.  There are two more attachments.

Tx_Fifo.gif shows the following:

The Vue sets the packet length to 10 (06, 0A).
Flushes the Tx Fifo (3B).
Issues the Tx Fifo Write Burst(7F).
Clocks in the 10 data bytes (81 - 00).

Tx_Strobe.gif shows the following:

The Vue reads the number of bytes in the Tx Fifo(FA).
Receives 10 (0A).  (Just a check??)
Issues the Tx Strobe Command (35).

I didn't find it change the length of the Preamble anywhere.

Ray