Author Topic: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100  (Read 139309 times)

0 Members and 2 Guests are viewing this topic.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #125 on: March 02, 2014, 11:31:14 AM »
So I'd vote no change to these existing features that you've currently included.
Running the server as a service with an associated configuration app is a great idea!

Noted.

Quote
I'm with 10ACTony in preferring to host the DB locally on my own server, as well as hosting a web page to display the output dynamically.

SkySpy needs some sort of DB, to save rainfall data.  WU needs 1-hour and 24-hour Inches of Rain readings.  I save the total accumulated rain inches samples to the DB every couple minutes and the figure out how much it changed in the last hour/day using a SQL query.

What I might do is add a setting that selects a simple 1-user database (just raw tables actually) to store sensor readings for the folks that don't want to install MySQL.  This arrangement would NOT allow reading the tables or data within them with PHP or some other method.

An advanced DB setting would use full-blown MySQL, which as many people have pointed out is relatively easy to  install and is free for the downloading.

Quote
Retaining the ability to upload to WU would accommodate those that don't want to maintain a local instance would be the best of both worlds.

WU updates are my primary goal.  But as I note above, there has to be some sort of sensor data storage - for at least 24 hours.

Quote
Can the skyServer mySQL credentials be kept in an .ini or .conf to be passed at startup?  It's no more secure than the registry, but it might be less
work.  In this mode, you could do what you like with the data.

Its six of one, a half dozen of another (Registry vs INI).  Reading/writing to the registry is pretty simple.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #126 on: March 03, 2014, 09:53:58 PM »
I started working on the SkySpy Server-Service (that is a mouthful!) tonight.

The plan (although things may change):

1) The service runs in the background and does all the non-visual stuff that ssServer does now - GW comm, WU Update, Mini-web site, DB update.

2) A Small app will configure the service/show the latest sensor readings in real-time if you leave it running on your desktop.  This is optional - except for initial setup.  Might be good to let you know if your sensors are working OK and if communications are happening in a timely manner.

3) The database will work in one of two modes:

3a) The service will be able to use stand-alone database tables (no MySQL database install required).  This mode does not allow external programs to read the database tables:  Data is for the exclusive use of the service.

3b) The service will be able to use a local or remote MySQL database engine (but not mine).  Login credentials will be stored in the registry.  You can do what you like with the data in the tables.

4) All configuration information will be stored in the registry.  Update it with the small setup app.

5) Absolutely No Promises, but I am looking at a Cumulus interface.  If it turns out to be a pain, this will be the first thing to get tossed overboard.

Offline 10ACTony

  • Senior Member
  • **
  • Posts: 71
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #127 on: March 03, 2014, 10:17:59 PM »
I started working on the SkySpy Server-Service (that is a mouthful!) tonight.

The plan (although things may change):

1) The service runs in the background and does all the non-visual stuff that ssServer does now - GW comm, WU Update, Mini-web site, DB update.

2) A Small app will configure the service/show the latest sensor readings in real-time if you leave it running on your desktop.  This is optional - except for initial setup.  Might be good to let you know if your sensors are working OK and if communications are happening in a timely manner.

3) The database will work in one of two modes:

3a) The service will be able to use stand-alone database tables (no MySQL database install required).  This mode does not allow external programs to read the database tables:  Data is for the exclusive use of the service.

3b) The service will be able to use a local or remote MySQL database engine (but not mine).  Login credentials will be stored in the registry.  You can do what you like with the data in the tables.

4) All configuration information will be stored in the registry.  Update it with the small setup app.

5) Absolutely No Promises, but I am looking at a Cumulus interface.  If it turns out to be a pain, this will be the first thing to get tossed overboard.

Sounds good.

If someone uses the local DB, will the monitor program still work if it's directed to the local DB?
//
No trees were destroyed in the sending of this message though a significant number of electrons were terribly inconvenienced.
//

Offline keckec

  • Senior Member
  • **
  • Posts: 62
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #128 on: March 04, 2014, 02:28:49 AM »
I started working on the SkySpy Server-Service (that is a mouthful!) tonight.

The plan (although things may change):

1) The service runs in the background and does all the non-visual stuff that ssServer does now - GW comm, WU Update, Mini-web site, DB update.


I just discovered this thread. Your program looks great, and I appreciate all the hard work that you've put into it. I have a new LCX station and was getting ready to connect it to their server, but really would like to keep the data local so I can access it whenever I want.

I would use your program, but I only use Linux. I would like to write a server like you've done, but using python in linux and have it write data to a local MySQL server. Then write some PHP scripts to view data and graphs with a web browser. I've done this sort of thing before in python and in C. Is it possible to get the source for your program, or perhaps the part that communicates with the gateway device and keeps it happily sending data.

I'm happy to help this project however I can, and of course make available any code I write.

Thanks a lot for your help.
« Last Edit: March 04, 2014, 02:32:03 AM by keckec »

Offline redsbeach

  • Member
  • *
  • Posts: 3
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #129 on: March 04, 2014, 02:06:11 PM »
I started working on the SkySpy Server-Service (that is a mouthful!) tonight.

The plan (although things may change):

1) The service runs in the background and does all the non-visual stuff that ssServer does now - GW comm, WU Update, Mini-web site, DB update.

2) A Small app will configure the service/show the latest sensor readings in real-time if you leave it running on your desktop.  This is optional - except for initial setup.  Might be good to let you know if your sensors are working OK and if communications are happening in a timely manner.

3) The database will work in one of two modes:

3a) The service will be able to use stand-alone database tables (no MySQL database install required).  This mode does not allow external programs to read the database tables:  Data is for the exclusive use of the service.

3b) The service will be able to use a local or remote MySQL database engine (but not mine).  Login credentials will be stored in the registry.  You can do what you like with the data in the tables.

4) All configuration information will be stored in the registry.  Update it with the small setup app.

5) Absolutely No Promises, but I am looking at a Cumulus interface.  If it turns out to be a pain, this will be the first thing to get tossed overboard.

Wow! Very  8-)  =D>

Not a request for enhancements, just a question relating to potential SkySpy Server-Service configuration(s)...

If you have more than one LCX station/gateways (1 remote - 1 local)...
would it be possible to create/run separate instances of the SSS-S, on the same physical server - each SSS-S instance
having a unique (configurable) service name and associated registry configuration entry?

e.g.
LCX1 Gateway at remote location.
LCX2 Gateway at local location.

Physical Server (running two SkySpy Server-Service instances to handle two LCX Gateways):

SSS-S instance 1 (for LCX1):
registry config = Service name: "SkySpyServer1", SSS-S listening IP: "192.168.2.100:6005", output to mySQL DB on IP: "192.168.2.100:3306", DB username: "SSSS_user1", pw="xxxxxx" Database Name: "LCX1_SkySpyServer1")

SSS-S instance 2 (for LCX2):
registry config = Service name: "SkySpyServer2", SSS-S listening IP: "192.168.2.100:5080", output to mySQL DB on IP: "192.168.2.100:3306", DB username: "SSSS_user2", pw="xxxxxx", Database Name: "LCX2_SkySpyServer2")

If unique service names & configs are a problem, we could always run two servers, (maybe 1 physical, 1 VM) each with their own SSS-S installations.


« Last Edit: March 04, 2014, 02:08:07 PM by redsbeach »

Offline mycal

  • Member
  • *
  • Posts: 24
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #130 on: March 04, 2014, 02:18:18 PM »

FYI, I've written parts of the server in PHP already, that allows you to register a gateway and a weather station to the php server.

Currently it doesn't do anything with the data, but it handles the packets and responds keeping the gateway and weather station happy.

I have a few more things to document, but  I post the source code in a day or two.  Probably save you a lot of time for you to run on your
linux box.

Skydvrz has the .php server code, but I don't know what he thinks of it.

-M





I started working on the SkySpy Server-Service (that is a mouthful!) tonight.

The plan (although things may change):

1) The service runs in the background and does all the non-visual stuff that ssServer does now - GW comm, WU Update, Mini-web site, DB update.


I just discovered this thread. Your program looks great, and I appreciate all the hard work that you've put into it. I have a new LCX station and was getting ready to connect it to their server, but really would like to keep the data local so I can access it whenever I want.

I would use your program, but I only use Linux. I would like to write a server like you've done, but using python in linux and have it write data to a local MySQL server. Then write some PHP scripts to view data and graphs with a web browser. I've done this sort of thing before in python and in C. Is it possible to get the source for your program, or perhaps the part that communicates with the gateway device and keeps it happily sending data.

I'm happy to help this project however I can, and of course make available any code I write.

Thanks a lot for your help.

Offline keckec

  • Senior Member
  • **
  • Posts: 62
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #131 on: March 04, 2014, 03:10:40 PM »

FYI, I've written parts of the server in PHP already, that allows you to register a gateway and a weather station to the php server.
Currently it doesn't do anything with the data, but it handles the packets and responds keeping the gateway and weather station happy.
I have a few more things to document, but  I post the source code in a day or two.  Probably save you a lot of time for you to run on your
linux box.
Skydvrz has the .php server code, but I don't know what he thinks of it.

Thanks, I'm sure that will save me a bundle of time. I look forward to seeing your code.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #132 on: March 04, 2014, 07:30:07 PM »
If someone uses the local DB, will the monitor program still work if it's directed to the local DB?

No. A local database is usable by only one app at a time.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #133 on: March 04, 2014, 08:03:48 PM »
Is it possible to get the source for your program, or perhaps the part that communicates with the gateway device and keeps it happily sending data.

I'd like to see more development on this, and I "speak" several computer languages.  I would love to see more research done on the unknown SDP fields, and the meanings of some of the other packet that are currently ignored. Please pass along anything you learn and I'll update ssServer  - with attribution if you like.

I did this in Delphi (Object Pascal) because I use it daily for other projects. So, if you can read Delphi then here is the source for you or anyone else that is interested.  Note that a lot of 3rd Party component libraries are not included, so you probably can't compile it even if own Embarcadero Delphi XE or higher.  They are up to XE5 right now.

https://www.dropbox.com/s/b7x31l6uutjgznq/ServerCloneSrc.rar

ssMainForm.pas handles the TCP/IP, HTML, database login, and the packet handshake logic side of things;

ssDataModule.pas handles SDP packet parsing, writing to the database, mini-web page HTML generation, raw sensor data to American units conversion, spoofed GW packet creation, Wunderground PWS updates, and other housekeeping methods.

This is just the server code - the SkySpy Monitor program is composed of some really expensive components that I cannot publish, so there wouldn't be much point in releasing the remaining 10% of non-copyright code for that.  If you want to see SQL queries to read the data back and graph it, then write me privately.  Most of the SQL queries will be self-evident when I get around to publishing the MySQL schemas.  That won't come until I ween everyone off my personal MySQL server.  :grin:

Also... This is a work in progress: The server software is in the process of having a massive rewrite and I won't be updating the version found in the RAR.


Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #134 on: March 04, 2014, 08:19:14 PM »
If you have more than one LCX station/gateways (1 remote - 1 local)...
would it be possible to create/run separate instances of the SSS-S, on the same physical server - each SSS-S instance
having a unique (configurable) service name and associated registry configuration entry?

Define Local and Remote  :-)

If they are physically separated by a large distance, then you'd probably want to use separate PCs at each location to run the ssServer software and then have ssServer report readings to a common MySQL server - like we do now.

I could make the Gateway port configurable, so that you can run more than one instance of the ssServer, each listening to a different GW.  I'll have to think about how to get them to fly in formation though.  It shouldn't be a problem with the MySQL database to have multiple ssServer instances running. 

You would have to play port forwarding games on your router, and I am not sure how you could run a single server computer reliably if you don't have a static IP for it.  The remote GW needs a static IP to send its packets to.  I haven't tried putting a hostname into the GW Proxy IP field.  Might be interesting to see if it chokes.

Quote
If unique service names & configs are a problem, we could always run two servers, (maybe 1 physical, 1 VM) each with their own SSS-S installations.

At worst you would need a PC at each location to avoid DHCP issues.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #135 on: March 04, 2014, 08:23:20 PM »
Skydvrz has the .php server code, but I don't know what he thinks of it.

It looks good, especially the clock set code.  I noticed that without a valid Real Time Clock (RTC) packet, my display has drifted by a couple minutes in the last month.  My RTC packet is currently ignored by the display.

I will take your static code and translate it to a dynamically updating one.

Offline keckec

  • Senior Member
  • **
  • Posts: 62
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #136 on: March 05, 2014, 06:02:15 PM »

I'd like to see more development on this, and I "speak" several computer languages.  I would love to see more research done on the unknown SDP fields, and the meanings of some of the other packet that are currently ignored. Please pass along anything you learn and I'll update ssServer  - with attribution if you like.

I did this in Delphi (Object Pascal) because I use it daily for other projects. So, if you can read Delphi then here is the source for you or anyone else that is interested.  Note that a lot of 3rd Party component libraries are not included, so you probably can't compile it even if own Embarcadero Delphi XE or higher.  They are up to XE5 right now.

ssMainForm.pas handles the TCP/IP, HTML, database login, and the packet handshake logic side of things;

ssDataModule.pas handles SDP packet parsing, writing to the database, mini-web page HTML generation, raw sensor data to American units conversion, spoofed GW packet creation, Wunderground PWS updates, and other housekeeping methods.

This is just the server code - the SkySpy Monitor program is composed of some really expensive components that I cannot publish, so there wouldn't be much point in releasing the remaining 10% of non-copyright code for that.  If you want to see SQL queries to read the data back and graph it, then write me privately.  Most of the SQL queries will be self-evident when I get around to publishing the MySQL schemas.  That won't come until I ween everyone off my personal MySQL server.  :grin:

Also... This is a work in progress: The server software is in the process of having a massive rewrite and I won't be updating the version found in the RAR.

Thanks a lot skydvrz. I have used Pascal, so understanding this is no problem. And I completely understand about the proprietary libraries.

I already have some personal weather data in MySQL, so I'll probably just add to that database once I get more up to speed on this. It may take me a bit of time to have something working. Other projects are taking up time as well.

I see in the readme the GW must be registered with Lacrosse. I assume there may be hashes exchanged to authenticate, maybe in both directions. Is the registration still necessary or does your code handle that now? And do you know if the GW requires the registration to be renewed after some period of time?

Does the GW only use HTTP to send data, or does it open other TCP/UDP connections to the server as well?

Thanks again for the code and for your time working on this. Anything more I find out on SDP fields or other mysteries I'll let you know.

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #137 on: March 05, 2014, 09:37:38 PM »
I see in the readme the GW must be registered with Lacrosse. I assume there may be hashes exchanged to authenticate, maybe in both directions. Is the registration still necessary or does your code handle that now? And do you know if the GW requires the registration to be renewed after some period of time?

Mycal has done some work on the registration process, so maybe at some point it could be incorporated into either your server or mine.  He did a working registration spoof in PHP.

It probably won't matter for most folks as you get a free trial registration in the shipping box.  Just use that.

Quote
Does the GW only use HTTP to send data, or does it open other TCP/UDP connections to the server as well?

The GW uses strictly UDP to communicate with the GAS utility.  It uses HTTP over TCP/IP for everything else. 

All HTTP packets have a header that identifies the user's account and (somewhat) the packet type.  The rest of the packet is a mishmash of BCD, little-endian integers, repeated fields and other mysteries that keep me awake at night.

Study TSDPRec in the datamodule for notes on all known packets and how the binary junk maps into their various fields.  I show zero-based hex offsets for all the fields in the comments.  I may update it with MyCal's findings as I get further into his code.

Any help you can offer would be greatly appreciated.

Offline mycal

  • Member
  • *
  • Posts: 24
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #138 on: March 06, 2014, 04:39:36 PM »

Hi, its time to release the PHP code server.   Please read the file, as it is all text and explains what is going on.

One thing if someone could do that is sniffing the lacrosse alerts data is to set the packet update rate on the control panel and find out what
changes in the replys from the server.

https://www.dropbox.com/s/wmihsm2gh1d6obs/request.breq

Allthough this code will register a "new" never been registered weather station, I recommend you register it at lacrosse alerts first, since it
writes a serial number into the weather station, and so far I have not been able to reset this data once it has been written.

Let me know what you think

-M


Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #139 on: March 06, 2014, 10:50:45 PM »
Here is a link to the latest ssServer installer:

https://www.dropbox.com/s/r9d6b7axwe2ruwr/ssServerInstall.exe

Just install this over the top of your current installation.  If you have modified your mini-webpage HTML, then you probably want to back that up first.  See the enclosed Readme for installation instructions.

If you don't have a database login send me a private message and I will send you one.

Here is a list of bug fixes in this release:

Code: [Select]
ssServerSim - 1.1.0.14 (Released 2014-03-06) [ View Issues ]
============================================
- 0000123: [Bug] Web update fails to restart ssServer properly  - resolved.
- 0000121: [Bug] WU update is not sending software version  - resolved.
- 0000122: [Bug] WU Update is using 24 hour rainfall window  - resolved.
- 0000119: [General] Not receiving or sending data  - resolved.
- 0000117: [Bug] Setup form needs to deactivate server while it is visible  - resolved.

Back to work on the service version...



Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #140 on: March 07, 2014, 12:58:39 AM »
If you are having problems updating v1.1.0.7 to the latest version (v1.1.0.14) then use the link in the previous post and manually overwrite your existing installation directory.  You will have to shut your old ssServer software down during the update process.

v1.1.0.14 has some substantial bug fixes, including (I Hope!) a better automatic-update feature.

Offline keckec

  • Senior Member
  • **
  • Posts: 62
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #141 on: March 09, 2014, 01:33:36 PM »
I ran a network capture on the GW traffic for about 17 hours, starting before I powered up the GW for the first time. It should include all the registration traffic, and the transition to daylight savings time. It was run on my firewall/router, which is a pfsense machine and can do traffic captures, and included all traffic to/from box.weatherdirect.com. The capture should be compatible with Wireshark.

For security reasons I don't want to put this on a public site, but I'm happy to email. PM me if you want it.

Offline slim83301

  • Member
  • *
  • Posts: 3
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #142 on: March 10, 2014, 11:49:57 AM »
I just want to give a BIG THANKS to everyone out there working on this project. I have been AWOL for the past month but   last week I got back on track and just a few min ago I was able to see our data on wunderground!

Skydvrz, thanks so much for all the hrs you have put in!



Will

Offline 10ACTony

  • Senior Member
  • **
  • Posts: 71
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #143 on: March 12, 2014, 07:28:05 PM »
WOW!  It sure has been quiet here lately.
//
No trees were destroyed in the sending of this message though a significant number of electrons were terribly inconvenienced.
//

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #144 on: March 12, 2014, 10:45:28 PM »
It has been quiet since I have been up to my eyeballs in the next generation of SkySpy - the Windows Service version of ssServer.  I have about 1/3 of the infrastructure for that written, but even that needs testing. 

If you want to see the project development roadmap and things I am working on, check out my bug/feature-request web site for SkySpy:

http://idk.serveftp.net/bugs/roadmap_page.php For the project roadmap

and

http://idk.serveftp.net/bugs/view_all_bug_page.php For a list of all bugs and feature requests.

You can request a new login account on my bugs web site and I will grant it.

Back to work!  ](*,)

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #145 on: March 17, 2014, 09:13:39 PM »
Work progresses - I have the SkySpy Service app framework done, along with storage of settings in the registry.  The service will run in the background on your PC.  If it sees that you have SkySpy Monitor running, it will send new sensor readings to it and Monitor will display them in real-time.  I got that part working pretty well.  Service keeps running, reporting new readings to Wunderground and the database even if Monitor is offline.

I've been ping-ponging back and forth between working on Service and Monitor, since they both have to fly in tight formation.

Stay tuned to my Mantis web site for progress reports and roadmap changes!

Offline slim83301

  • Member
  • *
  • Posts: 3
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #146 on: March 20, 2014, 10:18:12 AM »
I wish there was more I could do other than send in data.

We have 2 very grateful teachers here that thank each and everyone who has made this work!

We are still waiting on maintenance to install the wind and rain collectors on the roof. Maybe next week over spring break.

Happy Spring Everyone!

Will

Offline Mitch W4OA

  • Member
  • *
  • Posts: 41
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #147 on: March 20, 2014, 03:12:34 PM »
Quick question, do the sensors communicate directly with the Gateway or communicate with the display unit then the display unit communicates with the Gateway?

I hope my question is clear.

Mitch
(sempra)

Offline skydvrz

  • Senior Contributor
  • ****
  • Posts: 224
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #148 on: March 20, 2014, 03:54:16 PM »
It is a little more complicated than that  :-)

The C84612 anemometer/weather vane and rain gauge connect to the outside temp/humidity sensor over a 915 MHz RF link.  The outside temp/humidity sensor then forwards all of it to the LCD console, also at 915 MHz.  The LCD console then processes all the readings and forwards them to the optional Gateway at... You guessed it... 915 MHz.  :grin:

The GW1000U ERF-100 communicates with the LaCrosse cloud server (or SkySpy) via TCP/IP.

Offline Mitch W4OA

  • Member
  • *
  • Posts: 41
Re: LaCrosse Wireless Internet Gateway Model GW1000U ERF-100
« Reply #149 on: March 20, 2014, 04:08:05 PM »

Thanks!

I can see why they would do it that way, to make sure what is displayed on the console is the same as displayed from their "cloud". However, seems very inefficient and results in spectrum pollution from duplicate rf transmissions of the same data.

Mitch