Topic: FOSHKplugin - brief introduction

[olicat]

Hi!

An anecdote ...
Recently I was standing in the paddock and about every 30 seconds I received a push notification from one of my weather stations that the actual transmission interval of the station was significantly higher than the agreed transmission interval:

FOSHKplugin
From FOSHKplugin on 2022-02-21 at 17:25
<WARNING> real sending interval (41) mismatches the interval set to the weather station (30)
* 192.168.15.216; 21.02.2022 17:25:08 *

In principle, I like to be informed about such things promptly - but in the paddock I couldn't do anything with this knowledge!
After a while, of course, these constant messages got on my nerves and I wanted to turn them off. FOSHKplugin offers - also at runtime - the switching on and off of various functions. This also includes the interval warning or the push notifications.

The push notification clearly stated the reason for the message, the time and even which weather station this problem actually concerned.
But of course I couldn't remember which of my countless FOSHKplugin instances I would have to make the corresponding change to!
And it also took me a few tries until I had typed the correct URL into the mobile browser: http://ipaddress:port/FOSHKplugin/intvlwarning=disable.
Then finally there was peace!

But I was so annoyed that I quickly made a few changes to FOSHKplugin when I got home.
With next public beta there will be a clickable link in every push notification to the web interface of the triggering FOSHKplugin:

FOSHKplugin
From FOSHKplugin on 2022-02-23 at 16:43
missing data for mandatory sensor soilbatt1
* 192.168.15.236 for ws@192.168.15.215; 23.02.2022 16:43:52 *

Where the 192.168.15.236 is a link to the corresponding FOSHKplugin.
There, one then lands on a page on which all (!) URLs for controlling FOSHKplugin are listed as clickable links. So two clicks to disable something or to restart the weatherstation.
I think some of you will be happy about that too.

By the way, the cause of these constant error messages was a programming error - I fixed it afterwards.

Oliver

[olicat]

Hi!

Errors in the FOSHKplugin config file are serious. In the worst case, FOSHKplugin will no longer start.
But errors can also occur when configuring additional forwards. Each forward section must be uniquely named and only exist once.
If, for example, [Forward-1] is stored twice, only the first one is used and the second one with the same name is automatically deleted - with all its contents.

Whenever I want to set up a new forward (and in the meantime 100 forwards are possible per FOSHKplugin instance!) it is important to find the next free number - to prevent the forward designation from being assigned twice.
Up to now, the search was done manually for me via the command:

code]grep "\[Forward-" /opt/FOSHKplugin/foshkplugin.conf|sort -V[/code]

All forwards in the config file will then be displayed, sorted sensibly, so that it is easier to find the next free number.
In the next version of FOSHKplugin there will be a routine that reports duplicate sections in the config file (and two or three other common problems) and outputs the next free number and the first free gap.
This should make the search for a free number a little easier in the future:

Code: [Select]

# ./foshkplugin.py -checkconfig

checking config file /opt/FOSHKplugin/foshkplugin.conf

free section: [Forward-26]
next section: [Forward-89]

This function is also already included in the upcoming new public beta version.

Oliver

[olicat]

Hi!

There is once again a new public beta for FOSHKplugin v0.09.
Besides fixing some bugs that were found, there are also a few new functions that need some time for testing:

• the pushover notifications now additionally show the reporting host with a direct link to the FOSHKplugin web interface
• the general help page via http://ipaddress:port/FOSHKplugin/help showing all http commands to control FOSHKplugin
• the parameter -checkConfig for foshkplugin.py to output the next free [Forward-n]-section
  start with sudo -u username ./foshkplugin -checkConfig
• FWD_REMAP can now access all variables known to FOSHKplugin - see recipe Remapping keys/values (from v0.09)
  so you can now generate your own keys in most output formats and assign values to them
  Example:
  If you want to output the max. wind speed of the day as key windmaxday, you only have to enter FWD_REMAP = windmaxday = @windspeedmph_max under FWD_REMAP of the respective forward.
  several assignments must be separated by a comma: FWD_REMAP = tempf_max = @tempf_max,tempf_max_time = @tempf_max_time
  so if the output is in Ecowitt or WU format today, &tempf_max=45.3&tempf_max_time=1645788080 would be appended to the output line
• the page with all available variables that can be accessed in the FWD_REMAP function or via getvalue: http://ipaddress:port/FOSHKplugin/keyhelp

You are welcome to use this version as well and report any errors, problems or ambiguities.
Especially the -checkConfig option is used more often by me now. I think this is also helpful for you.
Once again I have decided to do a feature freeze. So now I just want to reduce bugs to finally release v0.09.

To update, please change to the installation directory of FOSHKplugin (/opt/FOSHKplugin/) on the console or via ssh and start there:

Code: [Select]

sudo -u username ./generic-FOSHKplugin-install.sh -upgrade generic-FOSHKplugin-0.0.9Beta.zip
Where username corresponds to the user name in whose context FOSHKplugin is running (e.g. owner of the log files).
The settings should remain the same.
Users updating from a version earlier than v0.09Beta may need to run

Code: [Select]

sudo -u username ./generic-FOSHKplugin-install.sh -repair
after installation to reload required libraries.

Have fun!

Oliver

[KC5JIM]

Beta running fine, no issues at all.

[olicat]

To the beta testers of the GW2000/WS90:
There is a problem with the current beta version v0.09 of FOSHKplugin and the processing of the data of the GW2000 with firmware v2.1.1.
I have fixed the problem here now and will release a new public beta shortly.
If you don't want to wait that long: please contact me!

Oliver

[olicat]

Hi!

The changes for GW2000/WS90 compatibility were more extensive than expected - I still had to make adjustments in many places.
Therefore, a new public beta with a slightly modified range of functions will be released.

• Bug fixed: UDP_MAXLEN for defining the max. length of a UDP datagram was not considered
• Problem fixed with GW2000/WS90 transmission with firmware v2.1.1 (inverted commas in rfdata)
• new config option Logging\BUT_PRINT = False respects IGNORE_LOG also for output on console (default: True)
  can also be changed at runtime with http://ipaddress:port/FOSHKplugin/printignored=disable/enable
• new config option Weatherstation\WS90_CONVERT = True to convert the special keys for rain values of a WS90 into the conventional ones if only these rain values are present (default=True)
  The rain values of a WS90 combination sensor are transmitted by the station with their own separate keys.
  If this option is activated (WS90_CONVERT = True), they are converted - provided the WS90 is the only rain sensor in the system:
  rrain_piezo --> rainratein
  erain_piezo --> eventrainin
  hrain_piezo --> hourlyrainin
  drain_piezo --> dailyrainin
  wrain_piezo --> wekklyrainin
  mrain_piezo --> monthlyrainin
  yrain_piezo --> yearlyrainin
  If another rain sensor is present, the values of both rain sensors are used for further processing, with those of the classic sensor being output as rain values.
  You may override this behaviour forward-specific with FWD_REPLACE
• for adjustments to support multiple FOSHKplugin instances in the LoxBerry version, changes were necessary that could also affect the generic-version

Therefore I ask again for active participation in the public beta test and for feedback. Thank you very much!

To update, please change to the installation directory of FOSHKplugin (/opt/FOSHKplugin/) on the console or via ssh and start there:

Code: [Select]

sudo -u username ./generic-FOSHKplugin-install.sh -upgrade generic-FOSHKplugin-0.0.9Beta.zip
Where username corresponds to the user name in whose context FOSHKplugin is running (e.g. owner of the log files).
The settings should remain the same but it is always good to save them beforehand.

Users updating from a version earlier than v0.09Beta may need to run

Code: [Select]

sudo -u username ./generic-FOSHKplugin-install.sh -repair
after installation to reload required libraries.

Have a nice weekend!

Oliver

[zoomx]

I get 403 forbidden 

[olicat]

Sorry, fixed!

Oliver

[olicat]

Hi!

A small adjustment for the next release ...

A frequently observed problem is that the "http://" in the FWD_URL is forgotten when creating a forward, although it is mandatory for http requests. Users are then surprised that the corresponding forwards do not work.
In the next public beta, FOSHKplugin will automatically repair these incorrect URLs and only report this once in the log:

<WARNING> FWD-52: URL must start with "http://" - adapted to http://192.168.15.100/gib/es/nicht/52/

By default, this functionality is activated - but can be deactivated with the switch Export\URL_REPAIR = False in the config file.

Regards, Oliver

[KC5JIM]

Weatherwizz is now working just fine. All I did was delete the forward and re-create the forward and it worked!

[alfmk]

Hi

Running FOSHKplugin v 0.09 beta, and has a couple of times seen that max gusts show two different values in PWS Dashboard.

When I run http://192.168.10.185:8080/minmax&status&units=m I get these values:

maxdailygust   29.52
windgustkmh_max   34.92
windgustkmh_max_time   1648595358

windgustkmh_max shows the correct value

any idea


Alf

[olicat]

Hi!

I am not sure if I have understood your problem correctly.

maxdailygust   29.52
windgustkmh_max   34.92
windgustkmh_max_time   1648595358

PWSDashboard accepts the key maxdailygust in the Ecowitt output from the weather station and displays its value as the daily maximum of the gusts. Wim is welcome to correct me if I am wrong.
This key is generated in the weather station and output via Ecowitt protocol - without any modification by FOSHKplugin.

During the min/max calculation by FOSHKplugin, it is checked in each received transmission whether the value windgustmph coming from the station is greater than the daily value known so far.
If this is the case, this value is saved together with the corresponding time as windgustkmh_max (or windgustmph_max) and output if necessary.

So if there are differences between maxdailygust and windgustmph_max already in FOSHKplugin, there is either a problem in the calculation of windgustkmh_max by FOSHKplugin or in the station when generating maxdailygust.
(Or I have not quite understood the generation of this value by the station - because I would assume the same procedure as with FOSHKplugin.)

I haven't observed this yet - but I haven't explicitly monitored it either. Whenever I check, both values are identical.

windgustkmh_max shows the correct value

How do you know that this value is correct and maxdailygust is wrong? Do you have an external reference?

Since this discrepancy is already obvious in the FOSHKplugin, PWSDashboard is ruled out as the cause. Because even here there would be a few more problem sources.

BTW: By default, PWSDashboard does not process the key windgustmph_max generated by FOSHKplugin. However, you could change this by changing the PWS_livedata.php.

Oliver

[wvdkuil]

. . .
PWSDashboard accepts the key maxdailygust in the Ecowitt output from the weather station and displays its value as the daily maximum of the gusts. Wim is welcome to correct me if I am wrong.
This key is generated in the weather station and output via Ecowitt protocol - without any modification by FOSHKplugin.
. . .

Hi Oliver,

Yes that is correct.
During the day the gust values "windgustmph" and  "maxdailygust" are compared with the previous vales to have the max-gust-today value for the small block on the top-row.
The daily/monthly/yearly values are saved in the history file.

Checking the site a few minutes ago:
Dashboard wind/gust block
= >  "Gust (Max)" 8.2 m/s  comes from    ecowitt file  [maxdailygust] => 18.34 mph => 8.2 m/s

Todays history (small block) shows 9.7 m/s at 01:10 and during the day that value was in some way uploaded
The history pop-up shows also 9.7 m/s at 01:10

The graphs with WU show also 9.7 m/s at 01:10 and that is a separate upload directly from the station, I assume.

The FOSHKplugin windgustkmh_max value of 34.92 is also correctly showing  9.7 m/s

The current station value maxdailygust seems to low, maybe the clock is running incorrectly?
Its UTC time is correct but what is the timezone it is using?

Wim

[alfmk]

How do you know that this value is correct and maxdailygust is wrong? Do you have an external reference?

My old laptop running weewx 4.7.0 and GW1000 API driver, and my HP2550 console show the same value as windgustkmh_max. (Time 1:09 local time)
This is not a big deal, just a little confusing


Alf

[wvdkuil]

How do you know that this value is correct and maxdailygust is wrong? Do you have an external reference?

My old laptop running weewx 4.7.0 and GW1000 API driver, and my HP2550 console show the same value as windgustkmh_max. (Time 1:09 local time)
This is not a big deal, just a little confusing


Alf

Weewx and the console have their own time-setting. Weewx and the Dashboard use correct time to calculate high-lows.

Could you please check your GW1x00 device time setting?
Attached a screenshot of a GW1000 which is running 1 hour 50 minutes behind.

Maybe your device is also running with incorrect time?

Wim

[alfmk]

Hi Wim

Could you please check your GW1x00 device time setting?

my GW1000 uses the correct time zone

see attached screenshot

Alf

[olicat]

Hi Alf,

my GW1000 uses the correct time zone

is the time as well as the time zone on the host running FOSHKplugin also correct?

Oliver

[alfmk]

Hi Oliver

The reason for this is not the FOSHKplugin, but rather my hardware
I found the answer in the .csv file. See attached screenshot


Sorry to bother you with this

Alf

[davidefa]

Would be interesting to understand why that happened.
The maxdailygust is generated by the ecowitt hardware ( so it is not a FOSHKplugin problem )
The only reason I can think ( for the maxdailygust to change at 8:03 ) is a power outage of your console/gateway ( which console/gateway are you using? ).
( the missed reception from the outdoor sensor should not cause this )

[olicat]

Hi!

is a power outage of your console/gateway

In fact, this would be a good explanation.
It's time for the key "runtime" also for the GW1000 (that's the station Alf uses).

The current firmware beta v1.7.0 seems to contain the key. This would make it very easy to determine whether the station has/was restarted or not.

Oliver

[davidefa]

Hi!

is a power outage of your console/gateway

In fact, this would be a good explanation.
It's time for the key "runtime" also for the GW1000 (that's the station Alf uses).

The current firmware beta v1.7.0 seems to contain the key. This would make it very easy to determine whether the station has/was restarted or not.

Oliver

Yes, another thing we add ( to our hardware  ) is a counter that increments each time the unit is power cycled ( it helps diagnose power problems when the counter increments too much quickly )

[Rover1822]

Hi Alf, is this occurring at the same time every day? if you answered that, I missed it.
Also I'm looking at the recording intervals, which to me, seem a little short, under 20 seconds, maybe that is OK, just curious

[alfmk]

Hi Alf, is this occurring at the same time every day?

No, this happens randomly. Two or three times in the last couple of weeks.

Also I'm looking at the recording intervals, which to me, seem a little short

Only for a few days to check uv reading, otherwise I use cvs_dailyfile for the weather statistics


Alf

[olicat]

Hi!

After an incredibly long beta phase, I have finally released version v0.09 of FOSHKplugin.

In addition to many bug fixes and optimisations as well as necessary adjustments for compatibility with new stations (GW1100, GW2000, WS1900) and sensors (WN34, WN35, WS90, AQIN), there is now also support for APRS/CWOP and, as a "killing" feature, the possibility of making any contents of fields available to other fields (FWD_REMAP).
The installation routine has also been significantly improved and FOSHKplugin has been made more error-tolerant overall - many "usual" problems during configuration are now automatically bypassed.
Furthermore, the improved sunshine hours calculation should now provide far more realistic values and it should now be even easier to run different instances of FOSHKplugin in parallel.
Have a look at the "Version History" here for a complete list of changes and improvements.

I recommend all users of FOSHKplugin (including beta testers) to update to this version. On the other hand, if your current version works without problems and you have no need for the new possibilities, you can of course stay with the previous version. It only makes it more difficult to get help if problems arise.

To upgrade:
Change via ssh to the directory in which FOSHKplugin is running - according to my recommendations, it could be /opt/FOSHKplugin.
If you run FOSHKplugin as root, you can do the following commands without sudo - if you run FOSHKplugin in a certain user context, the specification of that user would be important so that the permissions are correct:

Code: [Select]
sudo -u username ./generic-FOSHKplugin-install.sh -upgrade

Let me know if there're any problems or you need some explanations of the new functions. Most things should already be described on the website.

For first time (or parallel) installation just use this link and the common installation procedure.
LoxBerry users should download and install this version the usual LoxBerry-installation-way instead.

Have fun!

Regards, Oliver

[KC5JIM]

Updated from beta yesterday and it's been problem-free.