Author Topic: Python code to capture WS-1400-IP values from its web server...  (Read 4396 times)

0 Members and 1 Guest are viewing this topic.

Offline sswallace

  • Member
  • *
  • Posts: 1
I'm new to the forum, so this may have been covered before...

To capture WS-1400-IP data from its web server for logging locally, I wrote the following python code. It runs under OSX, probably Linux, not sure about windows (does windows have curl?).

It works for me, may not work for anyone else.



import subprocess
import re
import time

re.DOTALL # cause re matching for entire string, rather than just per line

IP_address_of_server = "192.168.1.21"
web_page_name_for_data = "http://"+IP_address_of_server+"/"+"livedata.htm"
delay_between_web_page_fetches = 15

vars = ('inTemp','inHumi','AbsPress','RelPress','outTemp','outHumi','windir',
      'avgwind','gustspeed','solarrad','uv','uvi','rainofhourly','rainofdaily',
      'rainofweekly','rainofmonthly','rainofyearly') # tags for data scraped from the web page

list = {} # contains key: values for web page data
old_list = {} # used to compair previous web page data values (only print when they differ)

def get_data_from_webpage(): # return dict of (data tags : values)
   readings = {}
   
   try:
      webpage =subprocess.check_output(["curl","-s","-m 3",web_page_name_for_data])
      # curl needs -m flag to prevent auto page refresh
   except subprocess.CalledProcessError, e:
      return "" # web page fetch failed

   for var in vars:
      match=re.search(var+r'.*value=\"([0123456789\.]+)',webpage)
      
      if match is None:
         return "" # on the tage or its value not found
      else:
         readings[var]=match.group(1)
         
   return readings

def print_file_header_columns():
   print "\"date\"",
   print ",\"time\"",
   for var in vars:
      print ",\""+var+"\"",
   print "\n"
   

#let's go

print_file_header_columns()

while (True):
   list = get_data_from_webpage()
   
   if cmp(old_list,list) and (list != ""): #only perform if readings have changed or webpage returned ""
      tmp = time.strftime("%Y/%m/%d,",time.gmtime())
      tmp = tmp+ time.strftime("%H:%M:%S",time.gmtime())
      #print tmp
      for var in vars:
         tmp = tmp+','+list[var]
      print tmp
   
   old_list = list
   time.sleep(delay_between_web_page_fetches)

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
Re: Python code to capture WS-1400-IP values from its web server...
« Reply #1 on: March 30, 2015, 02:36:22 PM »
Cool.  There is a Windows version of curl.
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline billybob

  • Member
  • *
  • Posts: 21
Re: Python code to capture WS-1400-IP values from its web server...
« Reply #2 on: March 30, 2015, 09:59:03 PM »
Thanks a lot for this. I will fire up a linux vm to see how it works.

Offline denemc

  • Member
  • *
  • Posts: 3
    • Tainui Weather
Re: Python code to capture WS-1400-IP values from its web server...
« Reply #3 on: April 26, 2015, 10:38:28 PM »
This works really well on a Raspberry Pi running Rasparian. Going to use the to get the data to my webserver so I can easily see indoor temp - the delivered view is really terrible!

Thanks for posting this.

Offline peter334

  • Member
  • *
  • Posts: 5
Re: Python code to capture WS-1400-IP values from its web server...
« Reply #4 on: November 26, 2015, 11:39:35 PM »
Hi, I'm a newbie and wanted to try this to capture data--especially the indoor data that cannot be obtained from WU. I'm using Windows, Python 2.7.10, and cURL_745_0. I get the column headers to print out, but cannot get the variables to output. The program keeps working every 15 seconds with no output. Do I need insert a delay so that the values are captured. It seems that when I directly access the ObserverIP unit through a browser it slowly displays the livedata page. Any suggestions would be appreciated.

Offline rwa2

  • Member
  • *
  • Posts: 3
Re: Python code to capture WS-1400-IP values from its web server...
« Reply #5 on: December 03, 2015, 02:53:54 AM »
Hello!  Thanks for this!
I made a somewhat simpler web scraper (well, the complexity is handled in the modules  ;) ).
This just spits out json once.  Or cut it down even further if you just want to spit out key/value pairs... hopefully someone will find this version useful!

Code: [Select]
#!/usr/bin/python

from lxml import html
import requests
import simplejson as json
from decimal import *

ws_ip = '192.168.1.36'
ws_id = 'ws1'

page = requests.get('http://' + ws_ip + '/livedata.htm')
tree = html.fromstring(page.content)
data = tree.xpath('//input[@name]')

def num_or_str(s):
    try:
        return Decimal(s)
    except (ValueError, InvalidOperation):
        return s

ws = {}
ws[ws_id] = {}
for node in data:
    # print("%s %s" % (node.name,node.value))
    ws[ws_id][node.name] = num_or_str(node.value)

print(json.dumps(ws, sort_keys=True))

Some sample usage / output:
Code: [Select]
./ws_parse_sj.py | python -mjson.tool
{
    "ws1": {
        "AbsPress": 29.94,
        "CurrTime": "13:44 12/02/2015",
        "IndoorID": "0x2d",
        "Outdoor1ID": "0xd9",
        "Outdoor2ID": "0x--",
        "RelPress": 30.04,
        "avgwind": 0.0,
        "gustspeed": 0.0,
        "inHumi": 60,
        "inTemp": 60.4,
        "null": "Stop Refresh",
        "outHumi": 70,
        "outTemp": 49.5,
        "rain_Default": "Rain Reset",
        "rainofdaily": 0.0,
        "rainofhourly": 0.0,
        "rainofmonthly": 0.0,
        "rainofweekly": 0.0,
        "rainofyearly": 4.21,
        "solarrad": 88.57,
        "uv": 308,
        "uvi": 1,
        "windir": 200
    }
}

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
Re: Python code to capture WS-1400-IP values from its web server...
« Reply #6 on: December 03, 2015, 09:19:31 AM »
Ever consider using Beautiful Soup?
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline bryzo469

  • Member
  • *
  • Posts: 1
Re: Python code to capture WS-1400-IP values from its web server...
« Reply #7 on: February 16, 2016, 10:37:15 AM »
I am trying to connect the Observer IP webpage directly to a rasbperry pi in order to utilize this code. I can't even figure out how to open the ObserverIP module's webpage. 

I do not have internet connection where I plan on installing this so i have been trying to setup a local network.

When I type the http://192.168.0.99/ into Epiphany Web Browser.  I get Could not connect: Connection Refused

Thanks!


Offline peter334

  • Member
  • *
  • Posts: 5
Re: Python code to capture WS-1400-IP values from its web server...
« Reply #8 on: February 16, 2016, 10:55:48 AM »
You may want to check this thread...I am hoping to use this solution, but am having to get up to speed with Linux and DD_WRT before I get into the scripts that DRSprite has written.

http://www.wxforum.net/index.php?topic=28262.0

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
Re: Python code to capture WS-1400-IP values from its web server...
« Reply #9 on: February 16, 2016, 11:03:53 AM »
I am trying to connect the Observer IP webpage directly to a rasbperry pi in order to utilize this code. I can't even figure out how to open the ObserverIP module's webpage. 

I do not have internet connection where I plan on installing this so i have been trying to setup a local network.

When I type the http://192.168.0.99/ into Epiphany Web Browser.  I get Could not connect: Connection Refused

Thanks!

Try another machine/browser.  Is port 5000 blocked?  Are you dyn parameters correct, etc.?
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline Mark55

  • Member
  • *
  • Posts: 3
Re: Python code to capture WS-1400-IP values from its web server...
« Reply #10 on: February 16, 2016, 06:55:50 PM »
Hello!  Thanks for this!
I made a somewhat simpler web scraper (well, the complexity is handled in the modules  ;) ).
This just spits out json once.  Or cut it down even further if you just want to spit out key/value pairs... hopefully someone will find this version useful!

Code: [Select]
#!/usr/bin/python

from lxml import html
import requests
import simplejson as json
from decimal import *

ws_ip = '192.168.1.36'
ws_id = 'ws1'

page = requests.get('http://' + ws_ip + '/livedata.htm')
tree = html.fromstring(page.content)
data = tree.xpath('//input[@name]')

def num_or_str(s):
    try:
        return Decimal(s)
    except (ValueError, InvalidOperation):
        return s

ws = {}
ws[ws_id] = {}
for node in data:
    # print("%s %s" % (node.name,node.value))
    ws[ws_id][node.name] = num_or_str(node.value)

print(json.dumps(ws, sort_keys=True))

Some sample usage / output:
Code: [Select]
./ws_parse_sj.py | python -mjson.tool
{
    "ws1": {
        "AbsPress": 29.94,
        "CurrTime": "13:44 12/02/2015",
        "IndoorID": "0x2d",
        "Outdoor1ID": "0xd9",
        "Outdoor2ID": "0x--",
        "RelPress": 30.04,
        "avgwind": 0.0,
        "gustspeed": 0.0,
        "inHumi": 60,
        "inTemp": 60.4,
        "null": "Stop Refresh",
        "outHumi": 70,
        "outTemp": 49.5,
        "rain_Default": "Rain Reset",
        "rainofdaily": 0.0,
        "rainofhourly": 0.0,
        "rainofmonthly": 0.0,
        "rainofweekly": 0.0,
        "rainofyearly": 4.21,
        "solarrad": 88.57,
        "uv": 308,
        "uvi": 1,
        "windir": 200
    }
}

Thanks,
Just tried this and works great.
Mark