Topic: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100

[iplay1515]

I'm interested in a discussion about the tcp/ip and http messages/packets that the LaCrosse Gateway uses in conjunction with it's communication with the host website: box.weatherdirect.com and decoding the protocol used to send the temperature and humidity data to the website.

My objective is to determine a method to capture the data from the LaCrosse sensor units such as the TX60U-IT at the local network level rather than having to resort to page scraping and similar methods to monitor the data.

Since I am new on this forum, I would first like to verify that this area is the appropriate place for the topic, and if not, I would ask a moderator to advise and to relocate this post.

[thobas]

I would be interested in what you find out. In fact, I signed up here because your post came up in my looking for these same answers. I am still doing my research and am currently on the fence as to whether to actually buy this device, because of the circumstances you describe. I do not want to have to rely solely on an external site, to get the measurements of local devices.

I have read some reviews that describe the reporting method used as a 'push' type data stream, instead of being able to poll the device directly, with say a HTTP GET request. Although, requests to port 80 of the gateway device IP do seem to return configuration information about the gateway device, I am not yet clear that any sensor information can be gathered this way.* Capturing the network traffic may be the only option, perhaps with a tool like tcpdump or WireShark.

I'm almost tempted to just bite the bullet and buy it, just to see what rube-goldberg type system I end up hacking together...

* I have only read this as reported by others, and not personally verified it is actually true.

[thobas]

This is just to followup with this discussion in case anyone else comes across the need to do this;

I finally broke down and purchased the gateway+temp/hum sensors, and have some bad news to report for those wishing to keep the data local. After installation, the gateway took some cajoling to finally connect to the correct server(since the IP changed since it was made it seems). I was able to get it to register successfully, and then started to capture TCP/IP packets coming from the device. Here is what happens;

1)device connects out with a TCP handshake
2) once complete, the device sends out an HTTP PUT request to the weather directservers;
http://box.weatherdirect.com/request.breql
3)Inside this request, is an HTTP_IDENTIFY header, which I assume is to uniquely identify your router/device
4)The put request is an unspecificed type application/octet stream
---The part that is making me return this device and look elsewhere for this
5)The octet stream is not in any way human readable, so I tried other possible encodings.
It is not UTF-8
It is not ASCII
It is not JSON compatible
It is not MIME

A request to the technical support email for the device and asking for clarification or documentation on this octet stream, was returned many days later. It was short and to the point;

"you are requesting proprietary information that we will not disclose.
The information will only go to the weather direct servers."

Which is unfortunate, because the connection to the servers from the device is marginal, at best. I have a SLA for my internet connection, and packets are rarely dropped, so I know it is not my end with the problem. I do not understand how they expect to maintain a business with a randomly available webserver to view this data on, and no way for the user to directly see their own measurements locally.

The temperature inside my house is not their proprietary information. This is BY FAR the worst device of this type I have ever encountered from a users perspective. I'm halfway torn between sending this back for a return, or spending a few more hours to crack this ridiculous hiding of my own data and post it for all others to use. It's your(and my) data after all, not someone else's property to be hidden at all costs from the person making the measurement.

[asquaredancer]

I've got one of these as part of the C84612 weather station. I'm still playing with it. the system works fine stand alone but won't connect to the server. My browser can see it fine, just not getting out.

[asquaredancer]

Since I posted the previous note I've been using Wireshark to sniff my network and can in fact see traffic between my gateway and box.weatherdirect.com, the server at 192.151.160.14.  See topic=16683.0 for more info.  The server is still not registering my gateway though. The saga continues...

[skydvrz]

---The part that is making me return this device and look elsewhere for this
5)The octet stream is not in any way human readable, so I tried other possible encodings.
It is not UTF-8
It is not ASCII
It is not JSON compatible
It is not MIME

Ah!  A puzzle!  I like puzzles and am a senior software engineer.  I am sniffing packets now to see if I can figure out the data format.

Did you use a dumb hub or did you sniff your switch/router in promiscuous mode?

[asquaredancer]

Just saw the reply from skydvrz.  I finally got around to trying my ERF-100 registration on a DSL connection and it worked just fine.  Apparently my problem was coming from my wimax internet provider in Missouri.  Now I think I'll go buy another C84612 (Costco has them in stock again) and start playing with it again.  @skydvrz, I used a dumb hub when I was sniffing packets.  I haven't looked at the packets on this network but probably will once I get a new 84612 to play with.  Right now I just have the gateway.  I left the rest of the stuff installed back home in Missouri.  Have you had any luck?

[skydvrz]

>Had any luck?

A little 

I have analyzed several hours of data from the IG module.  I used a dumb hub to tee the connection between the IG and my router. 

The IG sends the server 3 different commands, denoted by the two bytes before and after the 64-bit hex number in the HTTP header. I believe the 64 bit number is cryptographically related to your "Activation Code", so that they can ignore your IG for "revenue control" purposes.  Someone earlier in the thread found the IG MAC address in the hex string before the commands - I believe they are correct.

Commands in no particular order:

Command 01:/:00 is some sort of ack/status command - the packet sent with it contains 5 bytes that are mostly the same in each packet, but change occasionally.  The trailing two bytes are probably a 16-bit checksum.  The entire packet could also be the IG probing the server to check if it is valid, but that is a long shot.

Command 00:/:70 is some sort of ping / RTS command.  The server usually responds with 16 null bytes, but sometimes not.  Sometimes the server fails to respond at all.  I believe the trailing two bytes of the server's response is a 16-bit checksum, but I have not verified this.  This command may be a throttling command to lower bandwidth on the server, or to verify that the server is up, working and ready to receive data.  I will have to look at the timing data to see if this is a throttling mechanism for the server or something else.

Command 01:/:01 is the sensor data packet.  It *looks* like unencrypted data fields.  But seriously!  197 bytes to send this teensie amount of data?!  I could send full-precision data for all the sensors in one 8-9 byte packet.  I suspect the data fields are sent in some sort of record or structure, probably full of double precision numbers in binary format.  They may or may not be packed, but I doubt they are encrypted.  Unpacked data would have "padding bytes" stuffed in between fields with odd-numbered field-width byte counts.  Most of the payload records have only minor changes when compared to previous records;  They should look completely random if the fields are encrypted.  I believe the trailing two bytes of the sensor data payload is a 16-bit checksum - probably CRC16.

The only thing to figure out is the data type and field locations within the record for each sensor reading, and how to simulate a server - you would need to "keep the IG talking".  It may clam up if it cannot contact the mother ship.   I doubt a server simulator would take much rocket science.  It is possible to set the DNS server IP in the IG to an address of your choice, so it would be trivial to hijack it and have it chat with your PC instead of its normal server.

I noticed occasional data packets of 0x25 length.  These appear to be alarm packets produced if you set a min/max alarm value on your LCD controller.  They do not appear to be encrypted.  More research needed here.

Note: encryption may be in use, but if so it is probably simple XOR encryption with a fixed key.  Clueless newbie stuff.

I have no idea why they used checksums on all the data - the TCP/IP hardware layer insures there are no data errors, so it was a waste of time and bandwidth.

I am designing a program to do further analysis on the sensor data payload packets.  More to follow...

[skydvrz]

When the server responds with 38 bytes of data in response to an 01/00 command from the gateway, the contents sent by the server is the time of day, DMY info.  Used to keep the LCD controller clock accurate I would assume.  I am not sure what the leading 22 bytes do, and some of the trailing bytes remain a mystery.

[asquaredancer]

great! I sprung for another weather station and have it up and running with lacrossealerts.com. Their server seems a bit flakey imho. I'll have to get my sniffer back up and running to see if I can replicate what you found and add to it.
Good work so far.

[skydvrz]

I shot close-up video of my LCD display all night long and sniffed packets at the same time.  I got 7 gigabytes (18 hours!) of some of the most boring video ever shot.    

I created a spread sheet of (some of) the values on-screen when the big data packet went out. I noted packet numbers in the spread sheet, local time, Etc.  I will use it to correlate measured values with hex values in the packets. 

The LCD display blinks the "INTERNET" indicator when the big data packet goes out.  This indicator shuts off if the server does not ACK the packet. It turns back on after the next successful transmission.  The interval between large data packets is anywhere from 1-5 minutes or so, but averages about 4 minutes.

I also blasted the LCD display (location of the inside temp/humidity sensors) with a blow dryer to make a large change in readings during the recording period.  I will probably have to do something similar with the outside sensors.

Analyzing packets now, but it looks like they are shifting XOR keys or whatever encryption they use periodically. There is some suspicious hex in the big data packet:

AA AA AA, BB BB BB, EE EE EE found at offset 0x95 and 0xA4.  They are duplicated in the two offset locations.  The 3-byte hex values seem to perturb the other hex values as they change.

The repeated hex above may be the XOR key they used, but it would be silly to send it each time.  It might also be that they accidentally encrypted some fixed 0x00 or 0xFF bytes in the data record with the XOR key they are using.  If this is the case, then they are using a short key (1 byte) and it should be trivial to decrypt the fields.  Or not...

I am hoping that the GW is light on CPU power and memory space, and that the designers did not go to heroic lengths to encrypt the data.

Here is a good Wireshark _capture filter_ (not a display filter) for sniffing just the important data traffic to/from the GW:

host 192.168.xxx.xxx and tcp port http

where 192.168.xxx.xxx is the actual address of the GW. 

This keeps you from "drinking out of the firehose" during long sniffs.

The GW also chats with the DNS server to get the current server IP, and does the usual TCP/IP muttering, but this filter blocks that.

[asquaredancer]

Wow, I'm impressed.
You might just try taking the external sensors out of range. or just take the batteries out of the hum/temp sensor. Its the gateway for the other two.
I guess I'd be surprised if they encrypt weather data.
Have you peeked inside the gateway to see if you can tell what uP they're using?

[skydvrz]

I suspect that the GW would probably just repeat the last valid values if I somehow disabled a sensor.  I have seen the LCD display show ___.__ for missing values, so maybe there would be some value to remove batteries, Etc.  The weather outside is really crappy, so I'd rather not climb out on my roof right now 

As for the uP used - it really doesn't matter, since the task at hand is interpreting the TCP/IP data and then writing a server simulator - plus I don't want to break my GW module.  The case looks pretty cheesie.

If you haven't registered your unit yet, I'd be interested in seeing the entire process in Wireshark PCAP format.  It took me 2-3 tries to get my unit registered, so I don't want to go through that again! 

[asquaredancer]

I registered my old GW. Thats why I bought another system for use in TX. I've got the new GW that came with the new system and will register that before I go back to MO. It wouldnt register but it does communicate with the server so perhaps if I register it here it'll work with the system back in MO. Ive got part of the registration captured with wireshark from back in MO if you think thatll help.

[skydvrz]

I've got the new GW that came with the new system and will register that before I go back to MO. It wouldnt register but it does communicate with the server so perhaps if I register it here it'll work with the system back in MO. Ive got part of the registration captured with wireshark from back in MO if you think thatll help.

We may need a complete registration "conversation" to see if there is a way to simulate it with a replacement server.   For folks that have an existing registration with the actual server, I don't think there will be a problem.  Hijack the DNS (for the GW only), point box.weatherserver.com your own server as a replacement and it should continue to work as long as the GW thinks everything is normal.

There is much work to do 

[asquaredancer]

Now THAT'S an understatement.
Just got wireshark installed down here and ready to start sniffing again. I may try registering the new GW later. I'm reluctant to use my new activation code since I think there is a year limit on it. Maybe not. I also think I saw that it just reverts to display of current data with no history if you dont extend the subscription.
Later.

[skydvrz]

I made a lot of headway today, figuring out many of the data fields in what I call the SDP (Sensor Data Packet).  There are a lot of unknown/guessed-but-unconfirmed fields, but I have verified a few by comparing the hex data capture against the video I shot of my LCD panel over about 18 hours time.  The packet contains a lot of min/max values, timestamped with date/time.  I verified the following fields:

Code: [Select]

0x46 for Inside Humidity
0x53 for Outside Humidity
0xA7 Barometric Pressure MSD
0xA8 Barometric Pressure Center Digits
0xA9 Barometric Pressure LSD (only the upper nibble is used)
0xAA Barometric Pressure Delta since last reading MSD
0xAB Barometric Pressure Delta since last reading LSD

Typically, but not always, the byte value in hex is the actual digit displayed on the LCD panel.  Example LCD says 30.04 for barometric pressure - the hex would be 0x03 0x00 0x41;  If humidity is 92%, then the hex is 0x92.  I have to admit that using display values in hex to transmit telemetry to the server threw me for a loop.  You typically compress data like this to a smaller numeric format.

I suspect the wind direction vane output is some sort of 3-sensor angle detector.  There are some really weird values in the SDP and I suspect it is this sensor.  My wind readings are all over the place, with the light and variable winds we have going on right now.  Maybe someone that does not have to climb out on a roof can tie theirs down and capture all the outputs at the various points of the compass rose.

I am still looking for temperatures - they may be in centigrade and then converted at the LCD - I don't know.  They may also be offset by a fixed value.  More research required.  I did notice that the LaCrosse web site shows 32F if you disconnect your outside sensors.  Maybe this means that temps are passed as Degrees C, zero (disconnected) being freezing (32F).

I made up a spread sheet of many other suspected data columns in the SDP.  If you are interested in obtaining a copy or contributing, PM me here.

[asquaredancer]

sounds like data may be BCD.
I'm still playing with wire shark. my wind sensor is still on the porch so it would be easy to get angle data when I get to that point.
I'll PM you.

[skydvrz]

I think I just figured out the wind direction bytes (0x95->0x97). 

Each hex byte is divided into two 4-bit numbers.  There are 16 points to the compass rose, so you can represent all points in 4 bits.  There are 6 samples, right shifted (I think) across the three bytes. Each time a new sample comes in the bytes are shifted right by 4 bits and new sample goes into the upper 4 bits of offset 0x95.

So:

Code: [Select]

0 = N
1 = NNE
2 = NE
3 = ENE
4 = E
5 = ESE
6 = SE
7 = SSE
8 = S
9 = SSW
A = SW
B = WSW
C = W
D = WNW
E = NW
F = NNW

So if the bytes were

BC CC CC

Then the wind is currently blowing WSW.  A while ago (and for 5 sample periods) it was steadily blowing W.

Now... I have very little data to correlate, so I could be 180 degrees out of phase.  I believe the "big blip" on the LCD wind direction compass is the number found in the upper part of the 0x95 byte.  The "minor blips" are found in the other bytes.

[skydvrz]

So, to roll up all the findings so far...

The Sensor Data Packet (SDP) is not encrypted, and contains "display values" in most of the fields.  This means they are not the usual integers, long words, double precision numbers one usually finds in telemetry packets.  The values are (usually) the actual digits in hexadecimal format that appear on your LCD screen.  So if you screen says "98" then the corresponding value in the SDP data will be 0x98.  Decimal points are not found in the SDP, since all screen values use a fixed DP.  Dates and times are stored as "display values" too:  13 12 29 02 06 (in hex) is 2013 December 29th at 02:06 in the morning.  Note the Y3K problem with this format  

The SDP contains a lot of historical min/max values, paired with the date/time of the event.  That is the major reason the packet is so large (192 bytes).

What do you need to do to read/store your own weather data?  Here are two different methodologies:

For legal reasons, I strongly advocate the second approach. 

Simulate the Lacrosse server:

Create a DNS server to fake out the GW into using your own local server instead of theirs.  The GW can be reprogrammed to use a static IP address on your LAN, and look for the DNS server at an address of your choice (your own PC).  Redirect the GW to use your own HTML server instead of theirs.

Create an HTML server to handshake with the GW, so that it thinks it is talking to the real one.  You may need to capture the magic numbers that are exchanged during the registration process or simply mimic the normal chit-chat the GW does with the real server.  This new server should parse the SDP packets and store the data to a database.

Next, you need some sort of viewer program to watch the DB in real-time.  This is optional and probably only useful if you only want to watch your historical data at your house.  I suppose you could expose your DB server on a public IP and watch your data with a client/server app remotely.  A web server/browser would be simpler and platform agnostic...

So, to put you info on the web, you need some way to put sensor readings, graphs, Etc to something a web browser can read - and access over the Internet.  There are a many ways to do that - create an ISAPI DLL that hooks into an Apache or IIS web server.  The DLL reads the DB and creates HTML and JavaScript to render the data in a web browser.  Another way would be to create a small, stand alone web server that only does weather data display.  If you are a .NET weenie, you could do it that way and use IIS.

Use the Lacrosse server (and pay):

A second way to read/manage your own data, but only if you pay Lacrosse to show it on their server would be to snoop the SDP packets going to their server, parse them and store them locally in a database.  If you are a good programmer, there are libraries available to create your own WinPCap-compatible app to snoop the SDP packets I mentioned.  Parsing the SDP contents is relatively simple.  Next you would have to write some sort of viewer program to watch your historical data (see above).  Using Wireshark works for snooping, but it is a royal PITA to get usable data out of it.  I strongly recommend using an integrated snooper/parser/database updater solution.  Oh yeah - you need to acquire an old-school "hub", not to be confused with a "switch" or "router".  You need one of the now-obsolete CAT-5 hubs to snoop on the GW traffic.  There are lots of them on eBay (where I just bought mine).  They run $20-$30 USD.  Warning: many do not come with a power supply. Make sure the hub ships with one if you are not technically savvy enough to cobble one together.  Mine arrived sans power cube.  Luckily, I had one in my junk box that worked.

[skydvrz]

I wrote my own Internet Gateway packet sniffer today!    It listens in on the Internet Gateway chitchat with the Lacrosse server and displays sensor readings on my computer screen in real time (well, as fast as the Lacrosse server gets updated, which is about every 4 minutes or so).

The app is complete with a partially working SDP parser and data viewer. It runs on Windows - probably XP or higher.  Tested on Win7-64 Pro.  Written in Embarcadero Delphi XE, using the Magenta PCAP libs (free) and ICS (free).

It is a stand-alone app that does not need Wireshark or any other sniffer utility.  It does use WinPCAP, which is a free utility that is typically installed along with WireShark.  You fire up my app, select the Ethernet port you want to use to sniff (I have two ports on my machine) and set the IP address of your GW.  Click the start button and off it goes.  It can run without WinPCAP, but it might not work on certain network cards.  I plan to support both modes.

You do need a special (read obsolete) Hub, but the hardware is cheap. 

Lots of work to do (it was a really quick & dirty hack), but it does correctly display Barometric pressure and wind direction in real time.  I hope to get the rest of the sensors reading correctly tomorrow, if I have time.

[skydvrz]

Inside Temp seem to be encoded thusly:

Fahrenheit = (0.18033 * N) - 40.0

Where N is the BCD value found at offsets 0x13  and 0x14

Example:

0x06 0x27 (hex)  = 627 (base 10)

F = (627 * 0.18033) - 40.0

F = 73.1

0x03 0x06 = 306

F = (306 * 0.18033)

F = 14.8

I have confirmed the formula, but it may need minor tweaking to make it track the display value.  It seems pretty close though. 

I hit the LCD controller with a blow dryer, producing an 0x08 0x45 at around 112F

The 14.8F reading was arrived at by putting my LCD controller in a chest freezer for about 30 minutes.  The funny part was that it continued to chat with my outside sensors and the Internet while chillin' with the pork chops.  I guess my freezer makes a lousy Faraday Cage at 900 MHz

I imagine Outside Temp is encoded similarly.

Conversion to degrees C will require a different formula - sorry metric heads

[skydvrz]

Here is the alpha release of my La Crosse Gateway app.  It listens in on the Gateway's communications with the La Crosse server and displays the current sensor data on your Windows PC screen.  It also saves sensor readings to a CSV file.  Run the EXE and it should decompress into the app itself plus much of the source code used to create it.

Enjoy!

https://www.dropbox.com/s/zzlbbwrrvu0iwql/lcazSourceEXEInstall.exe

Open the included ReadMe.HTML file with your web browser to see required hardware, settings, screen shots, Etc.

Note:  The above link is obsolete.  Please look later in this thread for updates to this program.

Please comment here.

[10ACTony]

Nice job for an alpha, thanks.  I have the program working and I am getting data.  The only problem I have is that it has quit gathering data twice.  Once after ~2 hrs and the second after ~5 hrs.  Nothing unusual that I can see occurred.  No error messages and the computer did not go to sleep.  Getting ready to start it again. 

[skydvrz]

Yes, unfortunately there is a problem with the packet capture library I used.  It gives the rest of the app heartburn when it updates the screen. If some rare timing conditions occur, the program will freeze while trying to update the sensor readings fields.

I think the important part of this release is the conversion algorithms and SDP field mapping appears to work and the sniffing technique is sound. 

I'd like some feedback on Wind Speed.  Does the LCAZ wind speed reading track the speed showing on the LCD display?  In my neck of the woods, we don't see much higher than 1-2 MPH this time of year, and most of the time it is dead calm.  It is a bit difficult to test this sensor reading when it is zero most of the time   My anemometer is mounted on my roof at about 40' up, so I am not too inclined to climb up there to play with it.

I am aware of the 0.1-0.2F error on both temperatures.

I am already working on the next generation of this app.  It will probably use a database to store raw data, HTML headers and sensor readings.  This will be harder to deploy, because in addition to the other hardware, you'd need to install a (free) database server.  I will be using MySQL, as I already use it for a number of different software development projects.