Author Topic: ws1400 ip (observer ip) scraper (linux shell code)  (Read 1504 times)

0 Members and 1 Guest are viewing this topic.

Offline gtoal

  • Member
  • *
  • Posts: 24
ws1400 ip (observer ip) scraper (linux shell code)
« on: December 15, 2016, 03:59:48 PM »
http://gtoal.com/wx/ws1400.sh.txt (http://gtoal.com/wx/ws1400.sh to download)

It's a quick hack but if you need something and can't find a simple stand-alone utility that just extracts the parameters from the livedata.htm webpage, this might do the trick for you.

It formats the data and sends it to CWOP.  Tweak it to do what you need for any other service.

(V1.07 corrects the barometer reading from the raw station reading to QNH, which is what CWOP expects)

Graham
« Last Edit: December 16, 2016, 08:44:35 PM by gtoal »

Offline gtoal

  • Member
  • *
  • Posts: 24
Re: ws1400 ip (observer ip) scraper (linux shell code)
« Reply #1 on: June 22, 2019, 05:02:43 PM »
http://gtoal.com/wx/ws1400.sh.txt (http://gtoal.com/wx/ws1400.sh to download)

It's a quick hack but if you need something and can't find a simple stand-alone utility that just extracts the parameters from the livedata.htm webpage, this might do the trick for you.

It formats the data and sends it to CWOP.  Tweak it to do what you need for any other service.

(V1.07 corrects the barometer reading from the raw station reading to QNH, which is what CWOP expects)

Graham
I've updated this script a little over the years and posted the most recent version just now.  In particular I've added some minor index corrections to the data for my own site, which you can use as a template if you want to tweak the data you're sending to CWOP.  Since I recently set up a 'man-in-the-middle' interception of the ObserverIP's web uploads to wunderground and ambient (to inject PM2.5/PM10 data into the payload), I'm going to use that mechanism in future for pushing new values to CWOP and therefore will not be updating this Observer IP scraper from now on.  I have noticed that the ObserverIP slows down after a few days, probably due to a memory leak, and requires once a day rebooting to stay on peak performance, and I suspect the memory leak is exacerbated by the 5-minute polling I do all day to scrape for CWOP, so we'll see if using the MITM mechanism instead of scraping reduces that problem.

In fact since this is most likely the final version I may as well post the code here for anyone searching for ObserverIP scraper code.

Code: [Select]
#!/bin/bash
if [ "${USER}" == "" ] ; then
  USER="`whoami`"
fi
Version="1.10m"
if [ "${USER}" == "gtoal" ] ; then

  # For Anne's WS1400-IP.  Actually now replaced with a WS2902 (WS1600)?
  if [ ! -f /tmp/last-weather-server-contact.txt ]; then
    echo 0 > /tmp/last-weather-server-contact.txt
  fi

  LASTHEARD="$(( `date +%s` - `cat /tmp/last-weather-server-contact.txt`))"
  if [[ $LASTHEARD -gt 3600 && $LASTHEARD -lt 4000 ]]; then
    echo Your weather station has not reported for an hour|mail -s "Weather station down" gtoal
  fi

  Callsign="FW0153"           # replace with yours
  StationIP="192.168.2.240"   # ditto.
  LatLon="2616.99N/09810.37W" # format is important - see APRS docs
  Elevation=38                # elevation of barometer in M above sea level
else
  echo "Error: ${USER}, you must edit this script and configure Callsign/StationIP/LatLon/Elevation to match your own site."
  exit 1
fi
#
# View results here: http://www.findu.com/cgi-bin/wxpage.cgi?call=FW0153
#
# My server's units:
#   Wind mph
#   Rain in
#   Pressure inhg
#   Temp F
#   Solar w/m^2
#
# My crontab looks like this:
#   SHELL=/bin/sh
#   PATH=/home/gtoal/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#   4,9,14,19,24,29,34,39,44,49,54,59 * * * * /home/gtoal/bin/ws1400 > /dev/null
#
# alternative scraper for weewx:
#    https://github.com/poblabs/weewx-ObserverIP/blob/master/observerip.py
#
# ecce is a text editor from ecce.sourceforge.net to replace sed/awk:
#    wget http://ecce.sourceforge.net/ecce.c; cc -o ecce ecce.c
#
# if you don't have the 'timeout' command, remove 'timeout nns' from below
#
timeout 100s wget -O /dev/stdout -o /dev/null -q http://${StationIP}/livedata.htm > /tmp/weather.$$
if (( $? != 0 )); then
  exit $?   # probably the house wifi acting up again. Notify or reboot using Kasa switch?
fi
fgrep \</html\> /tmp/weather.$$ > /dev/null
if (( $? != 0 )); then
  exit $?   # partial fetch - should send an SMS alert to let me know...
fi
/usr/local/bin/ecce /tmp/weather.$$ <<EOF > /dev/null 2>&1
(f1/value=/m,k)0
m-0(u.name=".s..f1/"/u.value=".s.=.f1/"/e0m)0
m-0f/Apply=/k0
m-0(rli.export .t/=/i/"/r0i/"/m)0
m-0g
#!/bin/sh
%c
EOF
# AbsPress is coming through as "" if there is a fetch error. Initialise these so they can be tested safely.
AbsPress=0
solarrad=0
source /tmp/weather.$$
# solar radiation not used by weather site, but I use it elsewhere for controlling domestic lighting
solarrad=`echo "scale=0;(${solarrad}+0.5)/1" | bc`
# we have to use bc for rounding because bash is integer only
avgwind=`echo "scale=0;(${avgwind}+0.5)/1" | bc`
gustspeed=`echo "scale=0;(${gustspeed}+0.5)/1" | bc`
outTemp=`echo "scale=0;(${outTemp}+0.5)/1" | bc`
rainofhourly=`echo "scale=0;((${rainofhourly}*100)+0.5)/1" | bc`
rainofdaily=`echo "scale=0;((${rainofdaily}*100)+0.5)/1" | bc`
outHumi=`echo "scale=0;(${outHumi}+0.5)/1" | bc`
#echo "Raw abs pressure in inches: ${AbsPress}"
AbsPress=`echo "(${AbsPress}*33.8637526)" | bc`  # convert to hPa
#echo "Raw abs pressure in hPa: ${AbsPress}"
# CurrTime is initialised in the data from the observerip
CurrTime="`date --date=\"${CurrTime}\"`"
CurrTime="`date -u --date=\"${CurrTime}\" +%d%H%Mz`"

if (( (${outTemp} != 0) && (${outHumi} != 0) )); then
  # (I know, that will fail if the temperature and humidity both really are zero...)

  # I also have a pm2.5/pm10 sensor. They are not handled here.

  SeaPress=`echo "(${AbsPress} - 0.3) * e(5.2553026 * l(1 + (0.0000845115 * (${Elevation} / e(0.190284 * l(${AbsPress} - 0.3))))));" | bc -l`
  #echo "Corrected pressure: ${SeaPress}"
  SeaPress=`echo "scale=0;((${SeaPress}*10)+0.5)/1" | bc`  # scale and round
  #echo "Scaled Corrected pressure: ${SeaPress}"

  # apply dubious index corrections to move data towards MADIS (optional, experimental 2019/5/25)
  if [ "${USER}" == "gtoal" ] ; then
    # Currently the report at https://weather.gladstonefamily.net/site/F0153 shows Average barometer error of 0.0
    # and average daily temperature error of 0.0!  The humidity is crazily off but that is due to problems at the
    # MADIS side... (their estimate oscillates wildly).  So the small tweaking below seems to be worthwhile.
 
    SeaPress=`echo "scale=0;(${SeaPress} - 10)/1" | bc`
    # went from - 11 to - 14 to fix average error "-0.3"
    # if that changes the error to -0.6 then use " - 8" instead!
    #echo "Scaled and tweaked Corrected pressure: ${SeaPress}"
    # To do: temp needs scaling as well as offset.

    # offset
    outTemp=`echo "(${outTemp} - 1.5);" | bc -l`

    # scale.  84 is about where the actual and madis lines cross.
    outTemp=`echo "(${outTemp} - 84);" | bc -l`
    outTemp=`echo "(${outTemp} * 0.75);" | bc -l`
    outTemp=`echo "(${outTemp} + 84);" | bc -l`

    outHumi=`echo "(${outHumi} - 1.2);" | bc -l`
    avgwind=`echo "(${avgwind} * 4.3);" | bc -l`
    # 4.3 = 3.9 * 1.1 (ie make already-tweaked speed 10% higher)
    gustspeed=`echo "(${guestspeed} * 4.3);" | bc -l`
  fi


  if [[ "$SeaPress" != "" ]]; then
    if [ "${USER}" == "gtoal" ] ; then  # The data for CWOP is also used in my home automation system:
      /usr/bin/logger -d -p user.notice -t Weather "solar: ${solarrad} rain: ${rainofhourly} temp: ${outTemp} humidity: ${outHumi} pressure: ${SeaPress}"
    fi
  fi
 
  # reformat values with leading zeroes immediately prior to sending.  Otherwise misinterpreted as hex constants in expressions :-(
  outTemp=`printf %03d ${outTemp}`
  outHumi=`printf %02d ${outHumi}`
  if ((${outHumi} == 100)); then
    outHumi="00"  # special case - I guess 100% humidity is more likely than 0%
  fi
  avgwind=`printf %03d ${avgwind}`
  windir=`printf %03d ${windir}`
  gustspeed=`printf %03d ${gustspeed}`
  rainofhourly=`printf %03d ${rainofhourly}`
  rainofdaily=`printf %03d ${rainofdaily}`
  if [[ "$SeaPress" != "" ]]; then
    ( echo "user ${Callsign} pass -1 vers gtoal.com/wx/ws1400.sh ${Version}"; sleep 3; echo "${Callsign}>APRS,TCPIP*:@${CurrTime}${LatLon}_${windir}/${avgwind}g${gustspeed}t${outTemp}r${rainofhourly}p${rainofdaily}h${outHumi}b${SeaPress}http://gtoal.com/wx/ws1400.sh;${Version}" ) | nc -C -q 3 cwop.aprs.net 14580 > /dev/null 2>&1
  fi
  date +%s > /tmp/last-weather-server-contact.txt
fi
rm /tmp/weather.$$
exit 0


Graham

Offline ipierre

  • Member
  • *
  • Posts: 2
    • IYONGINS21
Re: ws1400 ip (observer ip) scraper (linux shell code)
« Reply #2 on: October 27, 2019, 11:51:22 PM »
It is very helpful to me. Thank you.
I need to check WS-8478 for your observer IP scraper code compatibility.
« Last Edit: October 27, 2019, 11:56:32 PM by ipierre »

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: ws1400 ip (observer ip) scraper (linux shell code)
« Reply #3 on: November 05, 2019, 03:32:22 PM »
The WS-8478 has nothing in common with other Ambient branded stations. This is because the WS-8478 is manufactured by a different company than the other Ambient branded stuff. Ambient sells hardware from a few different manufacturers and slaps their name on it (they also tweak firmware slightly). Their biggest sellers are the stuff made by Fine Offset like WS-2000, WS-2902A. The ObserverIP is also made by Fine Offset.
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

 

anything