Topic: How to setup custom HTTP export?

[fvanwijk]

Hi there!

I'm trying to get the custom export working, so that I can process the data myself.
I have a Firebase cloud function running, which accepts HTTP requests.
The WiFiLogger setup is posted as attachment.

Looking at the wifilogger frontpage I see "Post current weather data to major weather services" and "Custom export -> HTTP – JSON file format".
When I check the http://[WL-IP]//debug page, I see every 10 seconds:

Code: [Select]

[CustomLoggerHTTP]
PUT... failed, error: connection lost"
This suggests that a HTTP PUT is sent, not a POST.

The problem is that Firebase function doesn't log any traffic. When I do a PUT on the cloud function manually, everything just works.

What are the correct settings?

[WiFiLogger]

I will back to you with this problem, because I have to test it. Something is not working correctly with option WFLEXP.JSON.
PHP&MySQL is working fine.

Do you have WiFiLogger 1, or WiFiLogger 2?

[fvanwijk]

Thanks for the quick reply.
I remember that PHP/MySQL setting is also not working. Same error on debug page. Will check this again when back home.

From the UI is is unclear what the difference is, so you should consider to priorize to add documentation to your website (after this has been fixed of course )

I have WFL2.

[WiFiLogger]

Documentation is a very nice thing, but this project is not finish.
WFLEP.JSON is first version of JSON file which you can see when you open
http://datalogger_IP/wflexp.json
This file can be exported via MQTT, FTP and HTTP PUT.

PHP&MySQL was complete, up to the moment when one user explained to me, that it is not complete
PHP&MySQL is a set of JSON files and PHP script to receive JSONs, and MySQL database to save set of JSONS in separated tables.
PHP&MySQL is a copy of WeatherLink.com.
PHP&MySQL works perfectly
PHP and MySQL script will be delivered from me, I just need to rebuild some little things.

[fvanwijk]

So when I select "WFLEXP.json" I expect the WFL to send a PUT with application/json headers and the wflexp.json as body to the endpoint I filled in, every x seconds. Since this file contains all information, I can decide on the receiving side what information I use and what not. So no need for wflreg, wflhilow, etc

As I understood from a message by you in another topic, the main purpose of WiFiLogger is to read information from Davis weather station and to make it available to outside world via WiFi.
So I'm not fully understanding why you want to build a PHP+MySQL application in the WiFiLogger firmware and prioritize this over having a smooth setup and export experience. Or did I understand incorrectly what PHP+MySQL setting is doing?

(The installation went fine by the way and these docs are clear )

[WiFiLogger]

Ok I have checked issue. Issue was in libraries for WiFiLogger1.
WiFiLogger2 should not have any problem to send HTTP PUT.
Please note that WiFiLogger 1 and 2 is not using HTTPS yet.
WiFiLogger1 will not using SSL, WiFiLogger2 will in the future.

Please test HTTP PUT on this little PHP scipt.
It will capture stream from PUT and save it to wflexp.json file
mode: wflexp.json
server: http://wifilogger.net/station/php/bin.php
after that you will be able to check your data on this template:
https://wifilogger.net/station/php/


About PHP&MySQL
It is made for batch of WiFiLoggers. There are companies who are collecting data.
It is very easy to make a copy of console to the database
mode: PHP&MySQL
server: http://wifilogger.net/wflsql/wflsql.php
Then all data: settings, real time data, hilows1, hilows2, archive records. All will be stored in database.
Anyone will be able to install it on own hosting to store data.
Hosting cost to much for one station!!!!

[fvanwijk]

Thanks for helping out. It seems that setting to http:// is working. On /debug page I see:

Code: [Select]

[CustomLoggerHTTP]
Hello from Firebase!
JSON root false

I think I understand the differences between PHP/MySQL and wflexp.json, however that is from the receiving side. What is the difference from the sending side by setting the mode?
Is it possible to share some of the WiFiLogger code, as it is not exposed on /edit page?

I will host on Firebase which is free: https://firebase.google.com/pricing
You see Cloud Firestore rate limits are enough for sending all data for 1 WFL to the database. Maybe even more when you want to send more data.
I'm sure Azure or Amazon has similar free plan (Amplify, S3). Alternatives are now.sh, Netlify, Heroku, Surge.sh. All free for smaller personal projects such as WiFiLogger.

Here is my project code hosted: https://github.com/fvanwijk/wifilogger
The project can be used by everyone, but note that it is still work in progress and you need some programming skills (JavaScript).
It should be possible for everyone to host WiFiLogger data in the cloud (for free) instead of on WiFiLogger.

See src/server.js is similar to your bin.php.
PHP or JS or whatever doesn't matter for WiFiLogger so I suggest to not mention PHP/MySQL because it will send to any HTTP endpoint regardless of programming language used.

WFL has a web server, but I don't think it is a good idea. The closest alternative is to host on local network only for personal access, which is also free of course. Just run your own web server, for example on a Raspberry Pi or NAS.
The WiFiLogger just should send the data to an internal endpoint and not save and host all data + db + complex webapp on WiFiLogger.

I think it is better to spend time on improving debugging + UI on WFL and documentation. I can help you with that if you like.

[WiFiLogger]

Thanks for helping out. It seems that setting to http:// is working. On /debug page I see:

Wflexp.json was a first idea to make export to database over PHP.
After some work. I have decided to chunk it to different data, because HiLows data you don't need every 3 sec. Archive records are made every 1,5,10,15,30 minutes. Setup console data is needed once. etc. etc.
After new idea. I wanted to erase Wflexp.json, but some people started to use it. I had idea to rebuild it, because JSON standard has null, but in my JSON is Davis 1995 "---" as null. "---" is not treated by JSON parser as null, but string.
It was too late. Wflexp.json was used by others. It had to stay in this form.

PHP&MySQL
this is solution with couple of JSONs, depend on data.
- wflreg.json - registration - setup of console and WiFi (no passwords) / on boot
- wflrtd.json - real time data / interval can be set, should be 3-60 sec.
- wflhilow.json - hilows basic/ every 3 minutes
- wflhilowex.json - hilows extended VP2 additional modules like leafs, soil moisture etc. (this is exported for every VP2, even if additional module is not avaible) / every 3 minutes
- lastlog.json - short message asking for last exported to data base datalogger entry. / when new entry was made.
- wfllastlog.json - batches up to 7 data logger data. This is being send up to latest entry, but when connection is stable and all is working, then only last entry in data logger is send. So one json every 30 minutes.

The most important is logger data. WiFiLogger is a logger. It is device to register data. Only WeatherLink.com and local software like WeatherLink PC, Cumulus etc. can download and save this data.
WeatherUnderground is using current weather data and then they are making some statictics from it.
Other weather services the same.

WiFiLogger has FTP export option, but this is the worst technic. WiFiLogger every interval (15 minutes egzample) Is reading flash data from console. Reading 512 Pages takes 2min. 15 sec. In this time console is blocked and can't receive data from sensors.
15/30 minute interval is not very big problem, but for 1 or 5 min. interval it is just not working. Not only export is bad, but also data, because console can't receive it.
WiFiLogger can't save information to its flash memory every 1 minute, because flash can't be used so often.

WeatherLink.com export and PHP&MySQL both has same technic. Ask database what was the last datalogger entry and inform WiFiLogger about last log time.
After that WiFiLogger will send all avaible data, but this is only for the first time, because then every time new data logger entry was made, it is exported in same time when was made.

PHP&MySQL makes full export of data from console to database with RapidFire speed.
It is not finished, but works fine.
I want to add: to HiLows month and year.
Console has very nice graph data. It has 25 entries for some different data.
I don't know which data I would like to export at the moment. I just know I like to make it. There is no software making such thing.
What is nice in graph data (console display purpose). It has data like rain: 25 days back, 25 months back and 25 years back.
The same with wind, temp, etc.

I can give you access to my test database. (please contact on priv.).
When you put address of right php file to your WiFiLogger. Database will be populated with data.
I don't have any website to view weather data, but data base can be check via database manager.

Purpose of PHP&MySQL, why for who:
It can be used by companies to save data on own server (WeatherLink saves only 10000 entries). Maybe for club. Imagine club of 50 farmers with Davis Weather Station. Or 50 schools with some new social project.
I know companies who are gathering and recalculating data to some special charts and sells this info.
PHP&MySQL is closed technic, not by me, but by console firmware. How it works, how it is sharing data etc.
To make easy setup for end user I have made minimal options.
You have to only enable PHP&MySQL option and paste link to php file.
My is: http://wifilogger.net/wflsql/wflsql.php
Thats it.

WeatherLink IP was better. No http link paste, no WiFi password and SSID during setup.
WiFiLogger need these two steps, which are two steps too much for users.
I am using MAC address changed to decimal as ID in database. Called WID (WiFiLogger ID). This number you can have on SETUP-> Network Info.
MAC address is unique, so it can be used. User don't have to write it.
No KEY like it is in WeatherLink IP.
You have to know proper link to proper PHP before export. Treat this link knowledge as a KEY.
+ you can make white/black list in database/PHP.

My plan is to deliver PHP script, MySQL script, export scripts from database to JSONs, some simple tempate to view this JSONs imported from database.

Some raw info in attachement.

[fvanwijk]

Thanks for explaining more about your development work and ideas for the future. I continued working with WiFilogger after some months having no time for it.

If I understand correctly, the Flash memory of the console is not suitable for a lot of reading. Therefore the WifiLogger stored readed values in memory in the form of JSON files (wflexp.json) and a MySQL database is in development, to support more dynamic ways of exporting.

In my case I'm just interested in getting data out of WifiLogger (or in fact Vantage Pro2). I can take care of processing the data, creating a web application (UI) myself.

When I request (HTTP GET) http://[IP]/wflexp.json I get all the data in the body. I need to poll every 5 minutes to get all data. This works fine, but I don't want to actively request the data.
I want WiFiLogger to send the data to my HTTP endpoint every 5 minutes.

It looks like this functionality is in "HTTP -> PHP -> MySQL export settings" under Custom Export Setup (ftpexport.html).
However it is not documented and as mentioned I cannot see the differences between PHP&MySQL and wflexp.json Mode, because WifiLogger is a black box (I cannot see how it works).

To test this I have a local endpoint (cloud function) running on http://localhost:5001/wifilogger-api/europe-west1/storeObservation. I fill in this URL in the "Your Server" field.
The endpoint is called with PUT by WiFiLogger but the body is empty. See attached screenshot for logging empty body.
It doesn't matter if I set Mode to wflexp.json or PHP&MySQL.

What does WiFiLogger exactly do when Custom export is enabled? Can you help me with getting the data every 5 minutes automatically from WiFiLogger?

[WiFiLogger]

I will return to you today very late, or tomorrow. Please send me email support@wifilogger.net   
It will be easier to me share some info with you.

Before that
Please test PHP&MySQL on my site
Go to SETUP -> WiFi Setup and change DEVICE NAME for something more unique than "WiFiLogger"
Please set PHP&MySQL info:
SETUP| Custome Export Setup
Mode: PHP&MySQL
Your Server: http://wifilogger.net/wflsql/wflsql.php
Interval: 10sec.

Data you can view here. Please find your station on the list. On first page Real Time Data you can set units.
https://wifilogger.net/wflsql/
https://wifilogger.net/wflsql/arch.html

https://wifilogger.net/wflsql/test.php  -> Simple tool to extract data from database as JSON(RAW).

On priv I can give you access to view data in database and I can share PHP files.

[fvanwijk]

Ok it works!

The main function to process the request is:

Code: [Select]

$putdata = fopen("php://input", "r");
while ($data = fread($putdata, 1024)){
  $json_source = $json_source.$data;
}
You have posted that before.

In NodeJS / Express the equivalent is:

Code: [Select]

app.put('/', (req, res) => {
  const body = [];
  req
    .on('data', chunk => {
      body.push(chunk)
    })
    .on('end', () => {
      const contents = Buffer.concat(body).toString();
      if (contents) {
        const data = JSON.parse(contents);
        // Now do something with data
      }
   });
});

This is quite low level, and usually people use (JSON) bodyparser middleware to take care of the chunks and the JSON parsing. (I forgot this and therefore the body was {}).
So I added bodyparser middleware but it did not trigger automatically because the content-type of the request is not set to something json. So it took some time to debug.

Could you add the following header to the PUT request?

Code: [Select]

Content-Type: application/json


[WiFiLogger]

Could you add the following header to the PUT request?

Code: [Select]

Content-Type: application/json


Will do.