Topic: storing davis data in aws

[mczakk]

I'm a PhD student looking at forests and flooding and have a Vantage vue set up near my study area. I currently pull the data using NodeRed to a raspberry pi, but am looking for a long term alternative. Has anyone successfully integrated the weatherlink data download into aws?
Any help would be greatly appreciated!

[johnd]

Not really sure what you mean. AWS (Amazon Web Services, but also, confusingly, Automatic Weather Station) is 'just' a cloud platform, albeit with many web services running on top of it. The Davis weatherlink.com cloud platform already runs on AWS for instance. Think you need a more specific question.

[kobuki]

For storage you could use Amazon Timestream, for analysis and further processing, maybe AWS IOT. But you should be more specific as cloud providers give you a world of possibilities to store, process, query, analyze data, besides letting you run your own VM instances for anything you can think of. But first you need to outline your goals and to try anything, define your interfaces. I'd suggest looking at WeeWX for customized data upload.

[WiFiLogger]

I'm a PhD student looking at forests and flooding and have a Vantage vue set up near my study area. I currently pull the data using NodeRed to a raspberry pi, but am looking for a long term alternative. Has anyone successfully integrated the weatherlink data download into aws?
Any help would be greatly appreciated!

If you will use my devices WiFi/Ethernet then I have PHP and Database solution for people like you.
You need only hosting with PHP and MySQL.

Here you have prototype: http://wifilogger.net/wflsql/
 [ You are not allowed to view attachments ]

 [ You are not allowed to view attachments ]

 [ You are not allowed to view attachments ]

[mczakk]

OK, further to the answers posted so far, i will try and make the question more specific.
I have a network of rain gauges connected to an IoT network and reporting via the things network. I have a Davis vantage vue which reports to weatherlink live, and a Bresser weather station that is connected via a raspberry pi to the internet and it's data is stored on hive. I need to collect all the data from all three sensor sets in one place to replace the current situation where the data is stored on a raspberry pi on my desk via node-red.
Due to my organisation's policies the best place to do this will be Amazon Web Services, so that it can be integrated into a university data repository in the future.
I am almost there with the rain gauge data, and am going to use Amazon's Dynamodb to store that data.
My question is - how do i get the Davis station data into Dynamodb?

I hope this makes things clearer

[vinceskahan]

author=mczakk link=topic=43876.msg446726#msg446726 date=1652086542]
OK, further to the answers posted so far, i will try and make the question more specific.
I have a network of rain gauges connected to an IoT network and reporting via the things network. I have a Davis vantage vue which reports to weatherlink live, and a Bresser weather station that is connected via a raspberry pi to the internet and it's data is stored on hive. I need to collect all the data from all three sensor sets in one place to replace the current situation where the data is stored on a raspberry pi on my desk via node-red.
...
My question is - how do i get the Davis station data into Dynamodb?

I'd rethink the whole thing and simplify your data collection first.
You have multiple incompatible solutions feeding incompatible data storage locations.
I also need to toss in an obligatory 'weewx can do it' mention too.

This is the kind of thing MQTT is perfect for.   A couple small weewx instances could publish your weather station(s) data.  You'd need to switch your rain gauges to publishing to MQTT.  You'd need 'one' piece of software to subscribe to the MQTT topics and feed your db of choice (telegraf might solve that).

I have a similar LAN-only thing using influxdb as the db:

• docker container running mosquitto MQTT broker
• docker container running weewx (ecowitt sensors) - publishes to MQTT
• docker container running weewx (weatherflow station) - publishes to MQTT
• custom nodeMCU code for small breadboard ds18b20 sensor rigs - publishes to MQTT
• standalone small ARM box running weewx (Davis VP2 station) - publishes to MQTT
• docker container running influxdb
• docker container running telegraf - subscribes to MQTT and feeds influxdb

All the docker stuff runs on a 3 year old i3 Intel NUC under ubuntu and the box doesn't even breathe hard due to load.

[mczakk]

Thanks for your reply @vinceskahan, but I would respectfully suggest that if you can't answer the question, please don't bother.
I cannot 'rethink the whole thing' as the data collection is already up and running.
The raingauge data is already published as MQTT, which you would see if you actually read the question, but I'm struggling to write a rule to insert it into dynamodb.
I am currently storing all the data from all three sources on a raspberry pi using node-red, so your comment about incompatible sources is mystifying, all the data goes successfully into a pgadmin database, and can be extracted with ease using python. I do need to get away from this localised storage and need to use aws to comply with my institution's requirements, so if your answer does not help me to integrate with aws it is pointless.

[kobuki]

We're trying to help here. If someone attempts to help but fails, it's best to stay silent instead of replying using some rude tone. On the other hand, you still fail to spec what you mean by AWS integration. You won't get adequate help unless you specify exactly what you need. I fear if you fail to do at least that, you might have to re-think your thesis subject.

[WiFiLogger]

I am really confused about MQTT and popularity of this technology.
MQTT needs sustained communication all the time. Port is open 24/7 and needs confirmation that is a live. Add also multiples disconnections, and reconnections.
I am not telling that HTTP header not provides unnecessary data, but after all it is much less.

In my opinion for serving weather data sending HTTP request to PHP script is better solution, than making the same data transfer over MQTT.
WeatherUnderground use HTTP GET
WeatherLink IP to WeatherLink.com use HTTP PUT
In general all services are using HTTP. It just is the simplest solution, as it should.

But I am not an outstanding specialist in these techniques, so I will leave this topic to wiser people.

[WiFiLogger]

We're trying to help here. If someone attempts to help but fails, it's best to stay silent instead of replying using some rude tone. On the other hand, you still fail to spec what you mean by AWS integration. You won't get adequate help unless you specify exactly what you need. I fear if you fail to do at least that, you might have to re-think your thesis subject.

Thank you. I just didn't know how to write it 

[johnd]

I am really confused about MQTT and popularity of this technology.

Sorry, I know that this is really OT to this particular thread so I'll keep this short, but since the question has been raised:

The role I see for MQTT is for dissemination (especially to multiple clients) of real-time data, eg the 2.5sec updates for wind that Davis stations can provide. If you don't need rapid updates then HTTP is fine.

But the problem for MQTT adoption is that you need both ends of the link working for MQTT to be useful. Devices like WFL can obviously now provide the MQTT uploads, but AFAIAA we're still waiting for real-time dashboards that can consume the MQTT data. So we're halfway there, but we need new software to close the MQTT loop. It's only individuals that can write the necessary code for their own specific applications that can really take advantage of MQTT for now.

PS Wojtek: Latest package just arrived yesterday - stuck in UK Customs for 3 weeks!  Will get back to you in a day or two.

[mczakk]

We're trying to help here. If someone attempts to help but fails, it's best to stay silent instead of replying using some rude tone. On the other hand, you still fail to spec what you mean by AWS integration. You won't get adequate help unless you specify exactly what you need. I fear if you fail to do at least that, you might have to re-think your thesis subject.

I don't think I used a rude tone, I thanked the poster for replying and merely pointed out that his answer wasn't helpful. I didn't suggest, even though it crossed my mind, that  the poster hasn't understood the question as they went on to tell me that my raingauges should publish mqtt format. (they already do). Your comment that I should rethink my thesis title is if anything, also a little rude. Do you emden know my thesis title, or the subject of my study? I am not a data scientist or a programmer, just a PhD student who has to be a jack of all trades, but very rarely master of any!
As for your comment on having to spec what I need, it's quite simple. As far as I am aware, one function of AWS is as a data repository. I would like to get the data from my Davis vantage vue into dynamodb.
In not trying to be antagonistic, it's just very frustrating when you ask a question and someone tells you that everything you are already successfully doing is wrong!

[johnd]

I'm a PhD student looking at forests and flooding and have a Vantage vue set up near my study area. I currently pull the data using NodeRed to a raspberry pi, but am looking for a long term alternative. Has anyone successfully integrated the weatherlink data download into aws?

No offence intended,  but I still think you're expecting people here to be telepathic. It sounds like you have a Vue connected presumably (?) via a USB logger to your Pi. So to write the NR scripts then you must already have understood the legacy API that such loggers use, as described in the Davis Serial Tech Ref document? Or have you made use of someone else's NR flow to do this? (I think I remember seeing one online somewhere.)

So I'm guessing that what you want to do instead of using NR is still to use the Pi but have some Python code running there that extracts, what, just the rainfall data? Or other readings too? And then uploads them to the AWS service that you want to use?

[mcrossley]

I *think* that basically you want to set-up a AWS web service & app to receive the data, and then store it in the DynanoDB backend?

You would set-up scripts/apps on the station servers to to the upload to the AWS web service?

You will need a server "somewhere" to receive the data and forward it to the DB. It sounds like AWS is your orgs preferred cloud system and that is the logical place to create the web app too.

Something like this tutorial maybe? - https://aws.amazon.com/getting-started/hands-on/build-web-app-s3-lambda-api-gateway-dynamodb/

[mczakk]

I'm a PhD student looking at forests and flooding and have a Vantage vue set up near my study area. I currently pull the data using NodeRed to a raspberry pi, but am looking for a long term alternative. Has anyone successfully integrated the weatherlink data download into aws?

No offence intended,  but I still think you're expecting people here to be telepathic. It sounds like you have a Vue connected presumably (?) via a USB logger to your Pi. So to write the NR scripts then you must already have understood the legacy API that such loggers use, as described in the Davis Serial Tech Ref document? Or have you made use of someone else's NR flow to do this? (I think I remember seeing one online somewhere.)

So I'm guessing that what you want to do instead of using NR is still to use the Pi but have some Python code running there that extracts, what, just the rainfall data? Or other readings too? And then uploads them to the AWS service that you want to use?

Hehe 😊 now you're assuming!
I have a vantage  on a school in Wales which reports to weatherlink live in the usual manner. I do have a raspberry pi, but that's many miles away from the school, on my desk. The pi runs Node-red and part of the flow is an https request generator which requests data from the Davis api every five minutes, then writed that data to a .txt file. Each day the new data in that file is added to a pgadmin table on my laptop.
From there I can query the table using python and work with the data. I am aware that this is not the best way to do this, but it's the way it is.
In order for this data collection to continue it is better for the data to go into a data repository which is being set up by my university, which will be hosted on aws. I can see from the amazon docs that you can write a similar api request using amazon, but can't transfer the code from my pi to the laptop. The machanics of wiring this request is what I'm asking for help with.
@mcrosley has replied below, and I'm be looking at the document he has suggested in the morning 😊

[johnd]

Well, Weatherlink Live is the local upload device that receives data from the Vue ISS and then uploads to the weatherlink.com cloud platform. If you're using WLL to upload then you must be accessing data via the v2 API at weatherlink.com. All the details for that API can be found here:

https://weatherlink.github.io/v2-api/

which IIRC also includes some sample Python scripts. But maybe you know all that already? So is your real question whether anyone has turnkey Python code to read the v2 API and then upload data to a dynamodb (or whatever it is) script running on AWS? I dare say someone has done it, but I'm not aware of it in the public domain. (Actually, come to think of it, our Prowlar software does that (ie reads the v2 API and dumps data into an SQL database) but runs - currently, at least - on an DigitalOcean droplet, not AWS)

[vinceskahan]

I don't think I used a rude tone, I thanked the poster for replying and merely pointed out that his answer wasn't helpful. I didn't suggest, even though it crossed my mind, that  the poster hasn't understood the question as they went on to tell me that my raingauges should publish mqtt format. (they already do).
...
In not trying to be antagonistic, it's just very frustrating when you ask a question and someone tells you that everything you are already successfully doing is wrong!

No - you were quite rude.

I didn't say wrong.  I said you might want to rethink your approach to simplify it so you'd have a better chance of a supportable successful result.

But the heck with you.  Go wild.  Up your google-fu because I'm sure not going to try to help any more.

[kobuki]

Okay, so you have your data in a local PostgreSQL database. If you want to make that accessible on the internet (to your uni or wherever), you can use PGSQL in AWS, insert your data to that and allow access for remote queries. You mentioned DynamoDB, but that's a general noSQL database engine, not really a good fit for your project. What's more useful for telemetry data that changes over time at fixed (or dynamic) intervals is a kind of time-series database. PGSQL, DynamoDB, Timestream is a managed service you can activate and pay for it as you go. Timestream is a time-series database, I suggest taking a closer look. For data and graphs display, you could use AWS managed Grafana and share the dashboards with your colleagues. Or you can fire up an EC2 instance with Linux, install your own services, or even a Kubernetes cluster as managed service, but that's probably an overkill. Many things are ready to use, but need some research and RTFM. Make some time for yourself to get a bit familiar with your intended cloud service provider. It'll pay off on the longer term.

As you see, the possibilities are practically endless. I work daily with AWS and I can assure you it's a f**g maze for newbies. We're starting to get some input out of you, but you really should work on the questions you ask. The quality of the responses directly correlate to the quality of your questions and specified goals. Whether you're a meteorologist, data scientist or a human gene researcher.

[mczakk]

I dont want to make the pgsql database avaialable online, ultimately i will transfer the data in it to the university repository.
Your comments about the time series databases is very interesting, thankyou, I will look into them as they may, as you say, be more suitable for my project, as all the data is time series based.
Edit: i just looked at Timestream, and it's not available in my region - typical Amazon!!

[kobuki]

Edit: i just looked at Timestream, and it's not available in my region - typical Amazon!!

In that case, you might be interested in InfluxDB, Prometheus, VictoriaMetrics or TimescaleDB (a free Postgres extension). Maybe some of these is available in your region in managed cloud service or you can install any of these yourself on an EC2 instance. Run a few searches on the Marketplace for these.

[mczakk]

Edit: i just looked at Timestream, and it's not available in my region - typical Amazon!!

In that case, you might be interested in InfluxDB, Prometheus, VictoriaMetrics or TimescaleDB (a free Postgres extension). Maybe some of these is available in your region in managed cloud service or you can install any of these yourself on an EC2 instance. Run a few searches on the Marketplace for these.

It's got to be aws I'm afraid, as that is who the university are paying to host their overarching data repository. I managed to get halfway to timestream by changing my region to Ireland, back to square 1.5 for the rain gauges though, can see the mqtt messages, but can't put them into a timestream table!!

[kobuki]

It's got to be aws

Everything I mentioned is either an AWS native managed service or a managed service where the software itself is provided by a 3rd party, but runs in AWS. I didn't suggest you work out of AWS with anything. If you're unsure, make your uni spec the acceptable services - I presume that they will allow anything as long as it's running in AWS.

Regions mostly matter where you want to control where your data is stored (e. g. you need to store data in the EU to be covered by GDPR) or you need faster access - but that can be handled by several other techniques besides choosing an appropriate region.

[gnezdavrukavah]

Although the original post was made a year ago, I can still provide some insights for anyone else who might come across this discussion regarding storing Davis weather data in AWS. If you're looking for a long-term alternative to pulling data from your Vantage Vue weather station to a Raspberry Pi using NodeRed, integrating it with AWS is a great option. AWS provides comprehensive documentation and resources that can guide you through the integration process. While Trainocate is a microsoft learning partner and their resources primarily focus on Microsoft technologies, AWS's official documentation should be your go-to for information specific to integrating Davis weather data into AWS.