Author Topic: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100  (Read 139104 times)

0 Members and 1 Guest are viewing this topic.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #75 on: February 03, 2014, 10:19:49 AM »
Quote
Yes I have a 10/100 hub

Great!  ssCapture uses WinPCap to do its magic, so you are all set up.  You can run ssMonitor on any PC with Internet access.

I will send you a SkySpy download link and login credentials via PM tonight.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #76 on: February 05, 2014, 09:51:15 PM »
Here is the latest SkySpy release:

https://www.dropbox.com/s/vkhkxmrgnbolarf/SkySpyInstall.exe

This one includes graphs of your data.

Contact me via PM or Email to request database server login credentials.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #77 on: February 09, 2014, 11:21:58 AM »
I got the Gateway to work through a Proxy/Sniffer (without a hub).  Unfortunately, I can't easily integrate the proxy software I used into ssCapture.  The proxy components are too old for my development environment and it would take weeks to rewrite them.

If anyone wants to follow in my footsteps using an HTTP proxy/sniffer then make sure you click the Reboot button on Gateway Advanced Setup utility.  Proxy settings do not "take" until you reboot - that drove me nuts for about an hour...

The good news is that the full HTTP and binary stream can be viewed in real-time using the output of the proxy server, and the Gateway maintains its connection to the LCX Cloud Server.  This would make a lower cost alternative to using an old-school hub, but you still need to parse the binary data somehow.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #78 on: February 09, 2014, 09:15:32 PM »
Interesting... The so called "proxy" feature in the GW simply redirects its normal HTTP traffic to the designated "proxy server" IP on whatever port you select.  It does not use SOCKS or do anything special.  The gateway continues to use HTTP PUT, not CONNECT.

I did get a proxy working that acts as an intermediary with the LCX Cloud Server, but I now think I will just abandon that in favor of writing a cloud server clone. 

It may be possible to use the clone as a personal web server, as well as a data bridge for the Gateway.  I think it could display real-time weather sensor readings on a mini-web site, store your weather data to a database, possibly send warning emails, and post updates to Wunderground.

This is getting interesting...


Offline mikemac99

  • Member
  • *
  • Posts: 6
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #79 on: February 10, 2014, 12:58:40 PM »
That would be awesome if you wrote a cloud server!  Right now I have a perl program that goes to the LaCrosse website and scrapes my data.  Which would be fine if the data upload was reliable.  However I see gaps on a daily basis of several hours between updates on their site. A cloud server hosted on my PC would be much more reliable, or so I hope.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #80 on: February 10, 2014, 01:34:11 PM »
Yes, I believe it is quite possible to combine the gateway "data getter" and a personal web site server.  You would be able to store/view sensor samples with a granularity of ~4 minutes.  Using a dynamic DNS service, you should be able to access your personal web server running on your home PC from remote locations or your smartphone.

I started designing the server yesterday - I have all the necessary libraries to complete it.

Offline mikemac99

  • Member
  • *
  • Posts: 6
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #81 on: February 10, 2014, 01:50:38 PM »
With the replacement cloud server will we still need the hub?  If I've understood correctly the GW will be told to use a proxy (not sure how that is done) and the "proxy" will be the replacement cloud server.  So no need to sniff packets. Also no communication with the LaCrosse webserver but given how spotty that communication is turning out to be that seems to be no loss.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #82 on: February 10, 2014, 02:04:23 PM »
With the replacement cloud server will we still need the hub? 

Nope.  Just use the GAS utility to set the "proxy" IP address to a PC on your LAN that is running the replacement "cloud server" (I hate that name BTW :-) )

Quote
If I've understood correctly the GW will be told to use a proxy (not sure how that is done) and the "proxy" will be the replacement cloud server. 

Exactly.  The replacement server will mimic the LCX server, so there is no need to sniff packets or have a valid LCX connection. 

You can download the Gateway Advanced Setup utility from the WeatherDirect web site.  You can configure the Gateway's so-called proxy setting by typing in a valid IP into one of the fields.  You click Set and then click the Reboot button. 

Earlier in this thread I though we might have to play DNS games to trick the gateway module, but LCX made it way too easy.  The proxy setting also allows you to use an alternate TCP/IP port, so you can avoid conflicts with existing web servers on the same PC.


Offline 10ACTony

  • Senior Member
  • **
  • Posts: 71
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #83 on: February 10, 2014, 03:00:49 PM »
//
No trees were destroyed in the sending of this message though a significant number of electrons were terribly inconvenienced.
//

Offline mycal

  • Member
  • *
  • Posts: 24
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #84 on: February 10, 2014, 03:32:08 PM »
With the replacement cloud server will we still need the hub? 

Nope.  Just use the GAS utility to set the "proxy" IP address to a PC on your LAN that is running the replacement "cloud server" (I hate that name BTW :-) )

Quote

Have you considered using the GAS utility and just set the Gatway to your software device?  In this case you will get all the packets bound for the service but on the link level they would be sent to you.   You could then just sniff the packets, replace the destination MAC address to your real gateway (your router) and send it on.   This way you see everything, and you don;t have to emulate a proxy.  The replies would bypass your device and go straight to the gateway.  All you need is the raw send function of winpcap to send the packets.

I will be back in my lab tomorrow, so I can look at this.




Offline mikemac99

  • Member
  • *
  • Posts: 6
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #85 on: February 10, 2014, 03:41:29 PM »
It may be possible to use the clone as a personal web server, as well as a data bridge for the Gateway.  I think it could display real-time weather sensor readings on a mini-web site, store your weather data to a database, possibly send warning emails, and post updates to Wunderground.
This is an cool set of features!  I hope that you make it selectable which ones a user enables.  For example I don't want to set up a mini-web site since I'd be happy getting the data from the Wunderground.  And I try to minimize the number of ports I open on my PC.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #86 on: February 10, 2014, 03:53:19 PM »
In my experiments, I found that pretty much any old proxy software can work as a middleman between the gateway and the LCX Cloud Server.  There are some headaches associated with that. The proxy server components I tried typically strip off the HTML header and output only the "document" itself.  I need the header, because it contains a lot of information needed to classify and parse the binary "document" - the sensor data. 

I think it is time to move beyond packet sniffing...

I think the consensus is that most people want an easy to install replacement server.  I also think the technical expertise required to set up a wired LAN and old school hub are a bit beyond the skill-set of the average backyard weatherman  :grin:

So, given those two design requirements, the best way forward is to create a complete web server that talks to the gateway as well as local and remote browsers.  Luckily, this is not that hard. 

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #87 on: February 10, 2014, 04:07:08 PM »
This is an cool set of features!  I hope that you make it selectable which ones a user enables.  For example I don't want to set up a mini-web site since I'd be happy getting the data from the Wunderground.  And I try to minimize the number of ports I open on my PC.

Your PC will have to have at least one open port, so that the gateway can talk to it.  If you don't open up a port on your router, then you can only look at your personal web site from inside your LAN.  The bad guys can't attack your PC.

You typically use the "port forwarding" feature of your firewall/router to run a public server on your LAN. Running a personal server like the one I have in mind should be pretty secure.  The server will be a "one trick pony" that can only display one web page and maybe some graphs.  About the worst an outsider could do is a DOS attack on your server, and that would be pointless and boring for the attacker.

The server will do Wunderground auto-updates at some point.

For the lurkers:

If you do open a port on your router, then you can look at your private web site from outside your home network.

Offline MtHighVs

  • Member
  • *
  • Posts: 12
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #88 on: February 11, 2014, 10:57:50 AM »
Just want to add my word of thank for all the time you guys have put into this.

I have the C84612 and have been very disappointed in the LCX Cloud server and it lag in updating. You guys are giving me hope for something better. This has been a very interesting thread. I also have some of the older Weather Direct TX-60u-it that use the same gateway and am wondering if anyone has worked with them.

Thanks again keep up the good work

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
« Last Edit: February 11, 2014, 11:43:46 PM by skydvrz »

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #90 on: February 12, 2014, 12:03:32 AM »
I made a little bit of headway tonight.  My server app can successfully respond to a Gateway ping packet.  That makes the Gateway happy enough to offer either a Sensor Data Packet (SDP) or a clock time reset request.  I hope to be able to parse the SDP with the new server software and save it to the database - probably tomorrow.

Hey guys - I could really use some more data feeds.  Those of you that can hook up the necessary hub and start sending me data using SkySpy can be a really big help.  I need more samples to make this thing work for other people besides me.  I am worried that some of the mysterious data fields the Gateway sends out are critical for operation.  I feel confident that I can get something working for my particular Gateway, but I am not so sure about *yours*.

Example:  There is some sort of Epoch field that increments every few minutes in steps of 12, until it reaches a magic value and then all sorts of data changes.  This is at offset 0x16 and 0x17 in the 01:xxxxxx:00 packet the LCX Cloud server sends back to the Gateway.  I am afraid that if my server responds in a way the Gateway does not expect, it may stop working. It may be nothing, but more data from other people would be a big help!

I have sent free log in credentials to a number of people.  Please consider running SkySpy at this critical time!

 




Offline maraford

  • Member
  • *
  • Posts: 1
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #91 on: February 13, 2014, 02:28:39 PM »
I joined the forum so I could reply to skydvrz.
First, thank you for all your work in helping to make the c84612 more useful. A few days ago, I found a method for stripping the data from the Lacrosse site and uploading it to Weather Underground using a php script. The link is http://www.ilasoft.net/professional-weather-station-100/. The script is here: http://www.ilasoft.net/wunderground-pws-la-crosse/. The author is Ilya Pupko who helped me get it to work  using a WAMPServer on a Windows 7 machine (I'm a novice in anything more involved then simple html).
Your approach appears far more sophisticated and complete. I post the above in case Ilya's method has any value (especially in the data conversion challenge).
Look forward to your posts. -- Ford

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #92 on: February 13, 2014, 03:22:15 PM »
Maraford,

Scraping or spidering a web site is cheating  :grin:

Some other problems with using that technique are:

  • If the LCX web site look and feel is updated, the spider software usually breaks
  • You probably have to pay for an LCX account after the trial period is up
  • The LCX weather alerts are STILL broken
  • LCX web site updates are hit or miss.  There are big gaps in the data.
  • There are many missing features, like automated Wunderground uploads

There really isn't much comparison between the scraping data from the LCX web page and what must be done to glean sensor data from the raw data packets sent to the server.  That said, we have figured out almost all of the raw sensor data and designed ways to convert it into a readable format. 

The current SkySpy software requires a valid LCX server connection but it does not depend on anything the LCX server displays on the web page.  It listens in on the server conversation and gathers sensor information before it ever gets to the server.

The part that remains is making a portable system that people can use to host their own "mini server" without the need for an LCX server account/connection. I am also getting rid of the requirement for any special hardware.  That is the state of the project at this point. 

I am making good progress, but need more data samples from folks like you.  Pitch in and help  :grin:

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #93 on: February 14, 2014, 04:31:48 PM »
There will be a brief outage of the SkySpy data server today at around 16:30 Pacific.  SkySpy Capture and Monitor should keep trying to reconnect and start logging again after the outage is over.  If you are unlucky, you may have to log back in.  You don't have to shut down SkySpy - Just keep an eye on it.

I need to update the software on my database server - It should take 20-30 minutes max.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #94 on: February 15, 2014, 08:39:36 PM »
Life got in the way of my database server software update yesterday.  A component library I updated last night broke my compiler so I spent most of yesterday and part of today fixing it.  I'll do the MySQL server update later tonight.

Some good news...  My SkySpy Server is coming along nicely.  It logged an hour or two of valid SDP packets today, without using a hub or help from the LCX cloud server.  \:D/  Woo hoo!!!!

The only problem so far is the LCD display loses its "INTERNET" connection indicator.  Something my server is doing gives the Gateway indigestion.  Fortunately, the Gateway continues to send all the usual chitchat packets to my server.  I will let my SkySpy server run over night and see if the Gateway gives up after too long of a "no internet" failure.  I am hopeful.

If anyone can figure out what the exact handshake is to turn the Internet indicator back on, let me know.  It may have something to do with the 5-byte packet the GW sends to the LCX server.  So far, I have seen 14 possible payload values to that packet.  Each Gateway sends out a different numbers, but always from the same group of 14.  My guess is that the server embeds a reply based on the last one the Gateway sent in one of its responses back to the Gateway.  If this response is correct, then the Gateway tells the LCD that everything is hunky dory; If not, then the LCD shuts off the Internet indicator.  What that response is and where it is hidden, I have no idea.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #95 on: February 16, 2014, 07:17:25 PM »
My server clone can now display the last 8 hours of weather sensor readings in a web browser while simultaneously accepting sensor readings from the Gateway.  Here is a screen shot of the quick and dirty web page it generates:

https://www.dropbox.com/s/wx1hyu9g0d6zsgv/Captured001.png

You can still look at graphs and grids using ssMonitor.

My server clone seems to work OK as a complete replacement for the LCX Cloud server.  However... It may only work for me, since I mimic the packets the LCX Cloud server sends to my gateway (not yours).

I have had my server clone up for about 14 hours and it has not skipped a beat, so I think I am on the right track.

I will run some more tests to see how loose the LCX Cloud server <-> Gateway conversation can get, before the Gateway stops sending data.  By loose, I mean conforming to the expected secret handshakes and magic packets they trade.  Everybody's data packets are somewhat different, based on your registration number.  I am hoping to come up with a one-size-fits-all server.  Time will tell.  You may have to train the server clone first, before switching from the LCX Cloud server to this one.  I am not sure yet.

« Last Edit: February 24, 2014, 04:32:45 PM by skydvrz »

Offline trellend

  • Member
  • *
  • Posts: 8
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #96 on: February 25, 2014, 03:28:44 PM »
http://www.ti.com/tool/cc1111emk868-915

USB Dongle, packet captures 915 Mghz.   

Analyze on computer, use this page for protocols: www.f6fbb.org/domo/

Little bit of coding and you wouldn't need the base station or gateway.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #97 on: February 25, 2014, 05:00:30 PM »
Trellend,

The C84612 weather station ships with an included Internet Gateway containing a built-in RF interface.  There is no need to listen in on the sensor RF signals.

My latest software works with the hardware that ships with the weather station - you don't need to buy anything else.


Offline trellend

  • Member
  • *
  • Posts: 8
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #98 on: February 25, 2014, 05:34:41 PM »
I have some 915 mghz junk here.  I could rig up it up on the scope and see if it picks up a signal.  Might be a nice alternate route as I don't have a dumb hub, my router won't fake DNS, and port 80 is occupied on my machine. 

I have a PIC32MX795 here with USB and an antenna.  Firmware for a HID device (for the chip) and Delphi code for HID USB devices.  I've done that before obviously.  I don't have to buy anything in reality.  I might try it as a fun way to gather the data.

I already have 3 of these: https://www.sparkfun.com/products/retired/8971   (similar) so just a few lines for receiver to power/data/ground and i'm set.  If I commit to doing this, I'll start a new thread.

I figured I'd just buffer all the incoming noise from each device and let the computer sort it out on a request basis.

Offline 10ACTony

  • Senior Member
  • **
  • Posts: 71
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #99 on: February 25, 2014, 05:46:48 PM »
Trellend,

The C84612 weather station ships with an included Internet Gateway containing a built-in RF interface.  There is no need to listen in on the sensor RF signals.

My latest software works with the hardware that ships with the weather station - you don't need to buy anything else.

I am currently testing it and it runs great.  No need to buy anything and you don't need to use port 80.
 =D&gt;
//
No trees were destroyed in the sending of this message though a significant number of electrons were terribly inconvenienced.
//

 

anything