Author Topic: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100  (Read 87249 times)

0 Members and 1 Guest are viewing this topic.

Offline mikemac99

  • Member
  • *
  • Posts: 6
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #25 on: January 07, 2014, 06:53:45 PM »
I am already working on the next generation of this app.  It will probably use a database to store raw data, HTML headers and sensor readings.  

Can I suggest that it have the ability to choose where the output goes?  My preference would be to simply append the data in CSV format to a user-specified file.  This would allow the user to post-process as desired.

I have a Lacrosse C84612 I got for Christmas.  I just ran across this topic and am interested in what you are doing since I want to upload the data from my weather station to the Weather Underground.  

Not having as much of a hardware/software background as you, I wrote a Perl script that goes to the www.lacrossealerts.com website and scrapes my data.  I then upload it to Weather Underground.  I trigger the script every 15 minutes on my Windows 7 machine via the Task Scheduler.  The script works ok, the problem is that the lacrossealerts website does not show updates from every 4 minutes.  In fact I regularly see delays of an hour or more between readings available on their website.  I don't know if the packets are being sent up from my device every 4 minutes and being dropped, or if the lag has some other cause.
« Last Edit: January 07, 2014, 06:55:16 PM by mikemac99 »

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 221
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #26 on: January 07, 2014, 07:00:04 PM »
The GW attempts to send data to the server every 2-4 minutes.  It doesn't always get there, but you can sniff each outbound SDP  transmission.  The server sometimes fails to ACK, or physically tells the GW to come back later.  After La Crosse gets the data, who knows? 

That said, do you really need to have that granularity of sensor information?  The SDP actually contains averages, as well as the current readings.  I am not sure which ones get displayed on the La Crosse server.


Offline mikemac99

  • Member
  • *
  • Posts: 6
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #27 on: January 07, 2014, 07:11:54 PM »
That said, do you really need to have that granularity of sensor information?  The SDP actually contains averages, as well as the current readings.  I am not sure which ones get displayed on the La Crosse server.

No, I don't need every 4 minutes.  But I would like to be able to post a reading every 15 minutes, which I think is the Weather Underground suggested interval.  I keep a log of the update timestamp LaCrosse reports. In about 7 days of running (I just wrote the script recently) I see 47 delays of 30 minutes or more between updates posted on their server, 13 of more than 45 minutes, and 6 of more than an hour.  So their server isn't the best source of data for reposting, which is what makes the snooping that you are doing sound intriguing.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 221
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #28 on: January 07, 2014, 07:59:57 PM »
Feel free to give my app a try.  Download, run the "installer" and open the ReadMe.HTML file found within.  It lists the hardware and software you need (reasonably cheap - way less than the weather station) and diagrams for hooking everything up. 

There is source code for the packet data structure and algorithms to convert the raw binary into sensor readings.  Maybe you could write your own custom sniffer/submission tool that takes the 2-4 minute GW samples, averages them and ships the results off to Wunderground?

You can run your spider and my app side by side to see if you get comparable results.  My app makes a CSV text file, so you can open it in a spread sheet.

Offline 10ACTony

  • Senior Member
  • **
  • Posts: 71
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #29 on: January 07, 2014, 11:31:00 PM »
From quick checks it seems like wind speed seems about right on.  I'll spend more time checking it closely in the next couple of days (tomorrows my wife's birthday and I may not have much time). I was going to play with the log file you have generated but I think I'll wait until I see what you have in the database first.  Like I said I like what you got so far.  It's been running since the last message I left and still collecting data.  Please let me know what else I can do for you.  We will be getting some wet windy weather beginning tomorrow night and Thursday so that may be a good time to match the WS data and the packets.

Thanks again. :grin:
« Last Edit: January 07, 2014, 11:32:58 PM by 10ACTony »
//
No trees were destroyed in the sending of this message though a significant number of electrons were terribly inconvenienced.
//

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 221
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #30 on: January 08, 2014, 12:26:33 AM »
Lurkers:  I am very interested in any sort of LCAZ inaccuracy.  If you can tell me the amount of error, I can correct it, but you too need to be precise.  I am especially interested in when it is 70F+ (toasty) inside your house and less than zero outside.  The wider apart the temperature between inside and outside, the greater the error will be - and the easier it will be to fix. I live in Western Washington state, and it rarely gets into the teens in winter. I need more gnarly weather! ;)

I put my LCD display in my freezer for a while, but it didn't get below about 14F.  Not a really good data point.

Please send the temperatures the LCD display says, versus what LCAZ says.  It is easy to adjust the calibration, but I need to recompile the entire app to change what LCAZ says on it's screens right now.

I am working on a version that can save to a real SQL database, Hex logs and CSV files - all at the same time. The full-blown database is necessary to gather enough raw data to make the next generation of the app. 

And... I absolutely have to come up with a better product name  :oops:

In case you were wondering - all versions will be free for the downloading.  This is way too much fun to charge money for  8-)

Just a question at this point, but how are you guys submitting weather data to Wunderground and other sites?  How do you identify your data as coming from "you" and how is it submitted? Do you use a user name and password and transmit data somehow?  I don't need user names and passwords - just a URL or FTP address + instructions for sending weather data.  I might be able to do this automatically - actually I am pretty sure it will be easy compared with figuring out the GW data format.   :lol:

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 221
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #31 on: January 08, 2014, 12:33:38 AM »
Can I suggest that it have the ability to choose where the output goes?  My preference would be to simply append the data in CSV format to a user-specified file.  This would allow the user to post-process as desired.

I am working on it.  I need full logs, but I have no problem with saving CSV files full of converted sensor readings for the folks that only need minimal logging.


Offline 10ACTony

  • Senior Member
  • **
  • Posts: 71
//
No trees were destroyed in the sending of this message though a significant number of electrons were terribly inconvenienced.
//

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 221
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #33 on: January 08, 2014, 10:20:53 AM »
Perfect.  Yes, I can do that - as soon as I figure out where in the SDP the wind gust speed is stored. 

Apparently dew point is calculated like so:

Td = T - ((100 - RH)/5.)

Offline Brandan

  • Member
  • *
  • Posts: 4
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #34 on: January 11, 2014, 03:42:21 PM »
Has anyone figured out a way to get it to think It has a subscription, can someone send me the packets? I can't get mind to send anything.

I've been capturing them though my laptops LAN port.

What LA Crosse is doing is kinda scaumful. Atleast it violates the ACM Code of ethics.

They should really make an API.

On port 80 it says there's a data server also that's on a different IP address than the box.blah.com addresses.

Also it has options disabled for the users about uploading to an FTP server. Maybe it's possible to break into that and have it upload data to my FTP server.

I've written similar programs that emulate servers in the past but. I need to know the packages on which the GW communicates with the server and itself.
« Last Edit: January 11, 2014, 03:52:32 PM by Brandan »

Offline rippleeffect

  • Member
  • *
  • Posts: 2
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #35 on: January 11, 2014, 07:36:27 PM »
I would love to be able to have this gateway send data to Weather Underground. I highly doubt my wife would let me drop any more money on the setup I already have, so I'm stuck with finding an option.

I'll be following this thread for hopes of an "easy" way of doing this in the future.

Offline 10ACTony

  • Senior Member
  • **
  • Posts: 71
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #36 on: January 11, 2014, 08:46:16 PM »
I don't see the gateway itself ever uploading to Wunderground, but it looks like skydvrz will provide us with the data that can be uploaded to WU.  If you download the package he uploaded a few messages ago, you'll see that some of the data is already available. It will have to be averaged out and formatted as listed above.  I personally am looking forward to the next version skydvrz releases.
//
No trees were destroyed in the sending of this message though a significant number of electrons were terribly inconvenienced.
//

Offline rippleeffect

  • Member
  • *
  • Posts: 2
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #37 on: January 11, 2014, 08:53:06 PM »
I've got a little Atom processor PC in the kitchen that doesn't use much power. If I was able to snatch the data and use the PC to send data to wunderground, that work probably work.

The PC only uses 40w.

Offline Brandan

  • Member
  • *
  • Posts: 4
LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #38 on: January 11, 2014, 08:53:13 PM »
The gateway won't but the software that intercepts it or simulates the server can.

I use a raspberry pi for those kinda of projects since it uses USB power

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 221
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #39 on: January 11, 2014, 09:39:56 PM »
I just completed a snooper program that is much more reliable than LCAZ - the program I posted here on the forum a few days ago.  It requires a full-up database to record the data it captures from the GW.  The good news is that the database server software (MySQL) is free and fairly easy to install.  LCAZ was a proof-of-concept program and a really fast-hack.  Unfortunately it hangs after a few hours, but it does successfully capture data sent from the GW to the LCX cloud server.

The hardware necessary to make LCAZ work is under $30 USD.  The same hardware will be necessary for my new second generation snooper: SkySpy Monitor.  Download the LCAZ package and look at ReadMe.HTML found within the archive to see how you need to hook things up.

I am now working on a database viewer program SkySpy Viewer that can look at weather station sensor readings in real-time.  Again, this requires the database mentioned above.

Once that is done, I may invite a few people to submit data to my database server so that I can look at a larger sample of GW data packets.  Using the Viewer, they will be able to see their own sensor readings in realtime.  I will not be leaving my database server up indefintely - this is a couple month experiment.  I don't have the disk space for everyone's data.  The software will be free.

I am not sure of the legality of completely replacing the LCX cloud server.  It can be done.... easily.  I invite advanced programmers that wish to take that step to look at our captured data and create a replacement solution.  I will do my own, but I probably won't publish that.

Sending data to Wunderground is very simple.  I will probably add that to SkySpy Monitor.  You do need a valid Wunderground account though.  The GW can only send data to the LCX Cloud Server - nowhere else.

All of my programs will be Windows-only, but I will post much of the source code.  The monitor program requires that your computer remain on 24/7/365 for obvious reasons.  I don't think it is possible to run any of this on a mobile device, even if you were able to rebuild the code for that device.  


Offline Brandan

  • Member
  • *
  • Posts: 4
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #40 on: January 12, 2014, 04:49:53 AM »
It's very possible and not to difficult to get it to run on a mobile platform. Does this require a subscription still? I'm not going to pay more than $3 to access something I can build myself.

I managed to capture the advanced setup packets, it sends it in plain text.

I should be pretty easy to change the servers address from there.

« Last Edit: January 12, 2014, 06:00:08 AM by Brandan »

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 221
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #41 on: January 12, 2014, 11:23:57 AM »
Brandan,

The packet you are looking at is a broadcast packet used by the Gateway Advanced Setup (GAS) utility to determine the GW's IP address on your LAN.  This is how GAS fills in all of its visible fields with IP addresses and hostnames.

The GW uses TCP/IP HTML headers with a binary payload to send/receive information from the La Crosse (LCX) Cloud Server.  There are 6 or 7 different HTML packet types that are exchanged to/from the server.

If you have not sniffed any HTML packets going to/from your GW, then you have not installed a "dumb hub" to allow interception of this traffic.  Without one, you will only be able to sniff broadcast packets that are sent to every device on your LAN.  Broadcast packets contain no sensor information.

If you are using a "switch" or a "router" on your LAN, then you can only sniff packets that are addressed to your computer's IP.  Since the GW is sending packets to an external server on the WAN, you will never see them.  There is a diagram in my LCAZ installer that shows how to connect a hub to remove this restriction.

Note:  You need a dumb hub to do any sort of serious sniffing on a switched LAN.  No exceptions.

The Sensor Data Packet (SDP) is 192 bytes of binary data attached to an HTML packet. None of the SDP is human-readable.  You need to capture this packet, strip off the HTML text, and then convert the binary information back into its sensor readings if you want to tap into the data being sent to the LCX Cloud server.  The SDP is sent every 2 to 4 minutes.

See my LCAZ code (posted earlier in this thread) for the methods I use to parse the SDP.  Most of the source code is included.

Faking out the GW into thinking it is talking to LCX when in fact it is talking to a server under your control is not simple.  Possible, but not simple.  You would not need a hub to do this, but you would need a computer running 24/7/365 to replace the LCX server.

I don't know if GW continues to send the full SDP to the LCX Cloud Server after your 3-month free Extended Service trial period expires.  If it does, then you would not need to pay for the extended service.  I also noticed that freebie Basic Service on the LCX server is limited to 3 years.  I don't know what happens after that.

I think the Extended Service costs around a buck a month.  Not too bad, price wise.  I think their server software sucks, and that is why I am interested in replacing it in some way.

As for changing the GW's server address - not so easy.  You would have to use GAS to set the GW's DNS IP to a fake DNS server on your LAN.  When the GW calls your fake DNS and asks for the IP of box.weatherdirect.com, it would reply that the new IP is an address on your LAN - the address of your fake cloud server.  A fake DNS server is pretty simple to write. 

Offline Brandan

  • Member
  • *
  • Posts: 4
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #42 on: January 12, 2014, 11:45:36 AM »
Most of that work was cut out when I plunged the GW into my laptop, bridged the connections between LAN and WLAN and then sniffed the packets that way. All gateway communications goes though my laptops wireless card, but its on its own IP so its easy to get the packets from them.

What you saw there was a UDP return from the GW after the GAS paged it with a binary information.

I've also seen how the GAS changes the settings on the GW by sending more binary. All you really need to do is make a UDP program in something like perl and rebroadcast what was just sent.

But of course you'll have to see how the GW talk to the server before changing the address. No DNS involved.

The problem is i'm not paying for a subscription and I would rather engineer a way to capture from the weather station directly. But thats a much harder task to complete.


Offline mycal

  • Member
  • *
  • Posts: 24
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #43 on: January 13, 2014, 03:23:42 PM »

This is a great thread, just got a costco station with this gateway.   I've disassembled the gateway and am trying to get  a serial console on it.  I will let you guys know what I find out.

Mike

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 221
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #44 on: January 13, 2014, 03:27:04 PM »
Serial console?

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 221
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #45 on: January 13, 2014, 03:53:23 PM »
Can I suggest that it have the ability to choose where the output goes?  My preference would be to simply append the data in CSV format to a user-specified file.  This would allow the user to post-process as desired.

I was just thinking about that and did some quick calculations.  Excel would bomb after around 6 months of data collection if you imported a CSV that large into it.  I think that the file name will have to be rotated periodically so that the CSV files do not grow too large.

The Viewer program I am working on will have Excel, PDF export out of all the data grids.  Data grids also allow user-designed printed reports.

I added all the date/time fields I found in the SDP - they are being logged to my SQL database, so I should be able to figure out which sensors they are attached to.  I believe that the LCD display keeps track of min/max values and dates.

I attached some screen shots of the SDP viewer.  The Current Readings grid updates in real-time, as new SDP records are added to the database by the GW.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 221
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #46 on: January 13, 2014, 04:07:45 PM »
Another screen shot, after an GW update.  The app allows you to pick from 45 different "skins" or themes.  This is the Xmas skin.  The previous skin was London Liquid Sky (my personal favorite) :-)

Note the popup on the current readings grid - you can export/print from any grid in the app.

Offline mycal

  • Member
  • *
  • Posts: 24
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #47 on: January 13, 2014, 04:09:38 PM »

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 221
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #48 on: January 13, 2014, 04:14:58 PM »
Serial console?


Yes so far no joy.

I would be surprised if you could get anything out of it via serial comm.  All the GW smarts are devoted to TCP/IP.  You probably found a firmware programming port or maybe a production-test I/O port.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 221
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #49 on: January 13, 2014, 04:19:48 PM »
The ssMonitor app, showing the debug screen.  You can see the HTTP packet header and binary data payload - this is the raw, unparsed data sent to or received from the LCX Cloud server. 

I will add a full-up hex viewer soon.

 

anything