Author Topic: FOSHKplugin - brief introduction  (Read 15762 times)

0 Members and 1 Guest are viewing this topic.

Online olicat

  • Forecaster
  • *****
  • Posts: 887
  • GWxx00, HP2551C, HP3501C, WN1900C & FOSHKplugin
    • FOSHKplugin
Re: FOSHKplugin - brief introduction
« Reply #250 on: April 03, 2022, 06:07:08 PM »
Hello Jim,

Quote
Updated from beta yesterday and it's been problem-free.
thank you very much for the confirmation.
So far I haven't received any other (error) message or request. So it seems to be working.
That gives me some time for my other projects ...
;-)

Regards, Oliver

Offline joem

  • Member
  • *
  • Posts: 4
Re: FOSHKplugin - brief introduction
« Reply #251 on: April 17, 2022, 04:14:56 AM »
Hey olicat,

any chance to be able to connect it to InfluxDB 2?

Thanks

Online olicat

  • Forecaster
  • *****
  • Posts: 887
  • GWxx00, HP2551C, HP3501C, WN1900C & FOSHKplugin
    • FOSHKplugin
Re: FOSHKplugin - brief introduction
« Reply #252 on: April 17, 2022, 04:19:12 AM »
Hi!

Since InfluxDB2 should actually be backwards compatible with InfluxDB1, there are currently no concrete plans.
However, I always collect the wishes of the users and prioritise (also) according to them. In this respect, I can imagine that support may follow.

Oliver

Offline joem

  • Member
  • *
  • Posts: 4
Re: FOSHKplugin - brief introduction
« Reply #253 on: April 17, 2022, 04:22:47 AM »
I'm on a fresh influxdb2 install, and since this works with tokens, organizations and buckets, instead of username/pass/database, I'm out of ideas how could I feed the data to the database.

Offline zinz

  • Member
  • *
  • Posts: 6
Re: FOSHKplugin - brief introduction
« Reply #254 on: May 23, 2022, 04:53:55 PM »
Hi!

Since InfluxDB2 should actually be backwards compatible with InfluxDB1, there are currently no concrete plans.
However, I always collect the wishes of the users and prioritise (also) according to them. In this respect, I can imagine that support may follow.

Oliver

sadly influxdb is pretty incompatible between version one and 2.
I took a look at the code.
On the github page https://github.com/influxdata/influxdb-python it says that this library only supports Influxdb 1 and not 2.
It is suggested to use influxdb-client-python instead. For it to work The organization Token (password) and bucket would be needed. Maybe I will give it a try to implement it.

Offline zinz

  • Member
  • *
  • Posts: 6
Re: FOSHKplugin - brief introduction
« Reply #255 on: June 04, 2022, 06:31:41 PM »
So I added another forward for influxdb2 and the needed dependencies.

 [ You are not allowed to view attachments ]

@Oliver
I hope the support for influxdb2 makes into the next release.
Is there a way that FoshkPlugin resents data when my influxdb is back online after backup or maintenance?

Code: [Select]
[Forward-26]
FWD_CMT = InfluxDB 2 metric forward Testing
FWD_TYPE = INFLUXMET2
FWD_URL = http://server:port@myBucket
FWD_SID = myOrganization
FWD_PWD = "Token"
FWD_INTERVAL = 60
FWD_ENABLE = False

Online olicat

  • Forecaster
  • *****
  • Posts: 887
  • GWxx00, HP2551C, HP3501C, WN1900C & FOSHKplugin
    • FOSHKplugin
Re: FOSHKplugin - brief introduction
« Reply #256 on: June 05, 2022, 06:18:02 AM »
Hi!

That's exactly how I wanted to implement it. Thank you very much.
But I'll call the new forward types INFLUX2MET and INFLUX2IMP instead of INFLUXMET2 and INFLUXIMP2.

At the moment, however, I don't have the possibility to test it - I only have an InfluxDB1 server here.
Maybe I can set up an additional InfluxDB2 server for testing here before the next release. Then InfluxDB2 should also be implemented in FOSHKplugin.
But from your point of view, can this be used productively?

Otherwise, the new libraries for Influx2DB now explicitly require a Python version > 3.6.
Until now, I could also install FOSHKplugin on systems with e.g. Python 3.5.3. With integrated Influx2DB support, this would no longer work and I would exclude quite a few users.
I still have to think about whether I really want that or generally how to deal with this.

Oliver

Offline zinz

  • Member
  • *
  • Posts: 6
Re: FOSHKplugin - brief introduction
« Reply #257 on: June 06, 2022, 03:56:36 PM »
Hi!

That's exactly how I wanted to implement it. Thank you very much.
But I'll call the new forward types INFLUX2MET and INFLUX2IMP instead of INFLUXMET2 and INFLUXIMP2.

At the moment, however, I don't have the possibility to test it - I only have an InfluxDB1 server here.
Maybe I can set up an additional InfluxDB2 server for testing here before the next release. Then InfluxDB2 should also be implemented in FOSHKplugin.
But from your point of view, can this be used productively?

Otherwise, the new libraries for Influx2DB now explicitly require a Python version > 3.6.
Until now, I could also install FOSHKplugin on systems with e.g. Python 3.5.3. With integrated Influx2DB support, this would no longer work and I would exclude quite a few users.
I still have to think about whether I really want that or generally how to deal with this.

Oliver

Yes your naming scheme certainly makes more sense.
After some thought changing the location of the bucket and the orginization. As it is much more likely to have multiple buckets than organizations. So buckets would be more similar to the username (fwd_sid)
Code: [Select]
[Forward-26]
FWD_CMT = InfluxDB 2 metric forward Testing
FWD_TYPE = INFLUX2MET
FWD_URL = http://server:port@myOrganization
FWD_SID = myBucket
FWD_PWD = "Token"
FWD_INTERVAL = 60
FWD_ENABLE = False

For testing I would suggest to simply spin up one with docker Compose.
Here is the one I am using. change ports and volumes to the ones that fit your system. After you run the docker login in and under Data/ApiTokens generate a read write token for your bucket.
I am also not so sure why the weather community seems so averse to docker. Programs like CumulusMX, weewx etc. would be 100 times more user friendly for non programmers. No need to worry about dependencies, mono installs, data directories and so on. Everyone could be up and running in 5 minutes. Including secure public webpages. Well that is slightly off topic. For Foshkplugin this would also be helpful for updates and multiple instances.

Code: [Select]
version: "3.5"

services:
  influxdb2:
    image: influxdb:latest
    container_name: InfluxDBWeatherData
    ports:
      - "8086:8086"
    volumes:
      - "/mnt/user/appdata/InfluxDBWeatherData/data:/var/lib/influxdb2"
      - "/mnt/user/appdata/InfluxDBWeatherData/config:/etc/influxdb2"
    environment:
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_USERNAME=YourAdmin
      - DOCKER_INFLUXDB_INIT_PASSWORD=superSecretPassword
      - DOCKER_INFLUXDB_INIT_ORG=myOrg
      - DOCKER_INFLUXDB_INIT_BUCKET=WeatherData
    restart: always

So far this is running fine for me. (besides the data loss due to my server with the influxdb running performing a backup)
 [ You are not allowed to view attachments ]

From my understanding it should be possible to only provide this functionality for people with the appropriate python version. https://www.pythonpool.com/python-conditional-import/ Sadly python is not my primary programming language. So I would need to do some testing concerning what would be the best approach.

Some other points I have noticed about the code:
  • I am not a big fan of having only one file with 6k lines of code. This means that an import like influxdbclient has to be loaded in the main execution script. While it is only necessary for one specific function.
  • Your functions are doing too much and should be separated into several smaller functions that can be reused. For example in the forwardDictToInfluxDB function:
    • The first 40 lines are just for preparing the data from generic data to the specific influxdb one.
    • A further 20 lines are just for evaluating the URL. Which should ideally be done only once when ingesting the configuration file.
    • The real sending method is then another 30 lines.
  • Many variables are very short and therefore difficult to understand what they are supposed to do. In the function forwardDictToInfluxDB the variable d is not very clear what it should mean.
  • I haven't found the code on github or another Version Control software yet. If you haven't played around with that yet i highly recommend to do that as the sheer amount of code and effort you have put into this makes it really a lot easier to work with.

Offline Daali

  • weather n00b
  • Senior Contributor
  • ****
  • Posts: 178
    • The weather in Jefferson, GA
Re: FOSHKplugin - brief introduction
« Reply #258 on: June 21, 2022, 05:54:46 PM »
I'd love to get FOSHKplugin in a docker to ride as an addon in HomeAssistant.  I could repurpose a couple pi's =).  I need to dig in and learn a bit.

Offline thunder.ky

  • Member
  • *
  • Posts: 8
    • thunder.ky
Re: FOSHKplugin - brief introduction
« Reply #259 on: June 22, 2022, 11:27:29 PM »
Hi @olicat,

Thanks for sharing your work on FOSHKplugin. I'm trying to solve a problem that I believe I can with the REMAP functionality. So I have an Ecowitt Wittboy/GW2001 & WS90. It's configured via the Custom Server to forward to an instance of FOSHKplugin. Since it has a piezo rain gauge, the rain data comes in the EW format as the following variables:

&rrain_piezo=0.000
&erain_piezo=0.256
&hrain_piezo=0.000
&drain_piezo=0.256
&wrain_piezo=1.437
&mrain_piezo=8.736
&yrain_piezo=9.083


If I understand correctly I should be able to remap these with FOSHKplugin to the standard EW rain variables so that I can forward to "Personal Weather Tablet". Is this correct? However I cannot find a list of the FOSHKplugin internal variable names to successfully remap it.

For example with:
FWD_REMAP = @rainrate=@rrain_piezo

I get the error:
Remap: problem while remapping @rainrate with @rrain_piezo
Appreciate any assistance you can offer.

EDIT:
Got it working with the following:

FWD_REMAP = rainratein=@rrain_piezo,eventrainin=@erain_piezo,hourlyrainin=@hrain_piezo,dailyrainin=@drain_piezo,weeklyrainin=@wrain_piezo,monthlyrainin=@mrain_piezo,yearlyrainin=@yrain_piezo

Another question is about the "WeatherStation" section in the config file. Does this make FOSHKplugin use the Ecowitt API on the station instead of having to use the "Custom Server"?

Thanks,
Arthur
« Last Edit: June 23, 2022, 12:14:28 AM by thunder.ky »
Ecowitt GW2000B v2.1.5
Ecowitt WS90 v1.2.0
WeeWX v4.8.0
FOSHKplugin v0.09
https://wx.thunder.ky

Online olicat

  • Forecaster
  • *****
  • Posts: 887
  • GWxx00, HP2551C, HP3501C, WN1900C & FOSHKplugin
    • FOSHKplugin
Re: FOSHKplugin - brief introduction
« Reply #260 on: June 23, 2022, 12:55:14 AM »
Hi!

Quote
I'm trying to solve a problem that I believe I can with the REMAP functionality.
In fact there is a problem with FOSHKplugin in v0.09, which will be fixed in the next version:
Quote
Changelog:
...
fixed: WS90 rain-handling - replacing the new names with the old ones now works correctly (e.g. "rrain_piezo" --> "rainratein")
...
Actually, it was intended that FOSHKplugin would automatically convert to the conventional names if only the new key names were available. Precisely to ensure compatibility with software that has not (yet) been adapted, such as PWT.

The default active option for this is
Code: [Select]
Weatherstation\WS90_CONVERT = True
If WS90_CONVERT is set, the conversion to the conventional names takes place:

rrain_piezo --> rainratein
erain_piezo --> eventrainin
hrain_piezo --> hourlyrainin
drain_piezo --> dailyrainin
wrain_piezo --> wekklyrainin
mrain_piezo --> monthlyrainin
yrain_piezo --> yearlyrainin

However, due to a silly bug in v0.09, this conversion does not happen. Sorry.
Fortunately, though, you can actually solve this with the FWD_REMAP function, as you've already done:

Code: [Select]
FWD_REMAP = rainratein=@rrain_piezo,eventrainin=@erain_piezo,hourlyrainin=@hrain_piezo,dailyrainin=@drain_piezo,weeklyrainin=@wrain_piezo,monthlyrainin=@mrain_piezo,yearlyrainin=@yrain_piezo
However, with the upcoming v0.10 this remap will work automatically - as originally intended for v0.09.

Quote
However I cannot find a list of the FOSHKplugin internal variable names to successfully remap it.
The available variable names are - depending on the available sensors - user specific. Therefore I didn't want to generate a static list of all possible key names (very large and not very helpful) but a dynamic overview of what is possible for each user. You can reach this overview page with all variable names that can be reached by FOSHKplugin (for you) with http://ipaddress:port/FOSHKplugin/keyhelp. This page is very helpful, especially for individual queries (getvalue) or for the remap function.
I would also like to point out the help page of FOSHKplugin: http://ipaddress:port/FOSHKplugin/help - which documents all http control and query options.

Regards, Oliver

Offline thunder.ky

  • Member
  • *
  • Posts: 8
    • thunder.ky
Re: FOSHKplugin - brief introduction
« Reply #261 on: June 23, 2022, 10:56:03 AM »
Changelog:
...
fixed: WS90 rain-handling - replacing the new names with the old ones now works correctly (e.g. "rrain_piezo" --> "rainratein")
...

Actually, it was intended that FOSHKplugin would automatically convert to the conventional names if only the new key names were available. Precisely to ensure compatibility with software that has not (yet) been adapted, such as PWT.

The default active option for this is
Code: [Select]
Weatherstation\WS90_CONVERT = True
If WS90_CONVERT is set, the conversion to the conventional names takes place:

rrain_piezo --> rainratein
erain_piezo --> eventrainin
hrain_piezo --> hourlyrainin
drain_piezo --> dailyrainin
wrain_piezo --> wekklyrainin
mrain_piezo --> monthlyrainin
yrain_piezo --> yearlyrainin

However, due to a silly bug in v0.09, this conversion does not happen. Sorry.

Excellent, I look forward to the updated v0.10. Thanks for your work on such a useful tool! I'm about to make my install a bit more complicated  ](*,) by installing another instance of FOSHKplugin to get around some firewalls.

Quote
Fortunately, though, you can actually solve this with the FWD_REMAP function, as you've already done:

Code: [Select]
FWD_REMAP = rainratein=@rrain_piezo,eventrainin=@erain_piezo,hourlyrainin=@hrain_piezo,dailyrainin=@drain_piezo,weeklyrainin=@wrain_piezo,monthlyrainin=@mrain_piezo,yearlyrainin=@yrain_piezo
However, with the upcoming v0.10 this remap will work automatically - as originally intended for v0.09.

It took a bit of troubleshooting and testing, but was able to figure it out. The WS90_CONVERT will make it a lot easier on new users. I think the WS90 may create some new weather enthusiasts since it's so compact and easy to set up.

Quote
The available variable names are - depending on the available sensors - user specific. Therefore I didn't want to generate a static list of all possible key names (very large and not very helpful) but a dynamic overview of what is possible for each user. You can reach this overview page with all variable names that can be reached by FOSHKplugin (for you) with http://ipaddress:port/FOSHKplugin/keyhelp. This page is very helpful, especially for individual queries (getvalue) or for the remap function.

I would also like to point out the help page of FOSHKplugin: http://ipaddress:port/FOSHKplugin/help - which documents all http control and query options.

Perfect, I didn't realize how "self documenting" FOSHkplugin is. These URLs are tremendously helpful, and I've made note of them for future use. There's a lot of good information in there that I can use to extend my use case of it.

Quote
Another question is about the "WeatherStation" section in the config file. Does this make FOSHKplugin use the Ecowitt API on the station instead of having to use the "Custom Server"?

Referring to the above question, can FOSHKplugin query the GW2000/WS90 API directly or does it just work with the "Custom Server"?

Thanks again,
Arthur
Ecowitt GW2000B v2.1.5
Ecowitt WS90 v1.2.0
WeeWX v4.8.0
FOSHKplugin v0.09
https://wx.thunder.ky

Online olicat

  • Forecaster
  • *****
  • Posts: 887
  • GWxx00, HP2551C, HP3501C, WN1900C & FOSHKplugin
    • FOSHKplugin
Re: FOSHKplugin - brief introduction
« Reply #262 on: June 23, 2022, 11:17:56 AM »
Hi!

Quote
I'm about to make my install a bit more complicated  ](*,) by installing another instance of FOSHKplugin to get around some firewalls.
That should not be a problem.
I run about 8 or 9 instances in parallel on one host. As long as you don't get confused with the different directories, service names and ports, it works very well.

Quote
can FOSHKplugin query the GW2000/WS90 API directly or does it just work with the "Custom Server"?
FOSHKplugin only supports data reception via custom server.
The API functionality is not available for all devices and I wanted to have a solution that is as comprehensive as possible.

Oliver