Author Topic: "Ecowitt.net outage on 09/24/2021"  (Read 1745 times)

0 Members and 1 Guest are viewing this topic.

Offline OldAlaskaGuy

  • Alaska, The Last American Frontier
  • Forecaster
  • *****
  • Posts: 399
  • Living in the "Last American Frontier"
Re: "Ecowitt.net outage on 09/24/2021"
« Reply #25 on: September 24, 2021, 10:00:47 PM »
Confirmed. No data loss. [tup] Altogether fairly quick.

Offline plunet

  • Member
  • *
  • Posts: 44
Re: "Ecowitt.net outage on 09/24/2021"
« Reply #26 on: September 25, 2021, 04:25:15 AM »
This really is an important service since WU is so unstable. For my friends roll out tomorrow, his neighbors are going to rely on that "share link" to see river temps.

I hope it comes back up tonight!

It does highlight that for free-to-use cloud based services such as these and even the likes of Gmail and O365 you have no contract and no SLA, and to be honest a fairly one sided set of terms and conditions that offers no liability. If you are truly dependant upon these services to make time critical decisions then you should really consider if you need to have a backup plan if the service were to be down, or the service were to disappear along with any historical data.

Even the likes of Google and Microsoft can get tripped up by expiring certificates, it's sorted in a few hours. But what would it mean to you if Ecowitt were to disappear overnight along with all the data?

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3325
Re: "Ecowitt.net outage on 09/24/2021"
« Reply #27 on: September 25, 2021, 05:22:13 AM »
This really is an important service since WU is so unstable. For my friends roll out tomorrow, his neighbors are going to rely on that "share link" to see river temps.

I hope it comes back up tonight!

It does highlight that for free-to-use cloud based services such as these and even the likes of Gmail and O365 you have no contract and no SLA, and to be honest a fairly one sided set of terms and conditions that offers no liability. If you are truly dependant upon these services to make time critical decisions then you should really consider if you need to have a backup plan if the service were to be down, or the service were to disappear along with any historical data.

Even the likes of Google and Microsoft can get tripped up by expiring certificates, it's sorted in a few hours. But what would it mean to you if Ecowitt were to disappear overnight along with all the data?
That's why IT professionals suggest you have a copy of your data locally or/and on your own web site (and if deemed necessary) with the respective SLAs (even though this gives you only a pseudo 100% safety - you cannot financially afford to have a 99.99 % SLA [100 % nobody would give you] for all components of a service involved [front-end, local network, WAN, local server LAN, server], only banks or some governmental institutions can [or do their own hosting]) etc.
99% SLA (it's made over one year usually) means 88 hours (3.67 days) of service downtime accepted !
Usual, affordable SLAs cover 95% = 18.25 days downtime without breaking the SLA !!!
 
This whole cloud stuff only works reasonably if you have small amounts of data to transfer or have 10 Gbit (and more) network (and internet) access. Anyway "cloud" for most people is just remote storage but it's much more, including the network and server infrastructure.

I keep on laughing when Amazon or Microsoft (and many computer magazines) want to tell me I should make a backup of my PC in the cloud. It's ridiculous. How long does it take to transfer 100 GB (or more) for a disk image, even with a 1 Gbit internet connection (which would be shared in the local access infrastructure) ? And how long does the restore take ? Forget about it.
Such things you organize locally with either classical (external) hard disks (offline, ransomware protected) or with a high(er) capacity NAS with at least 1 Gbit connections, better 10 Gbit.
We simply don't have the access infrastructure yet to use "cloud services" for such purposes.

Of course, the weather data databases are of just a few megabytes size - but still. Your servers can be RPis, your backup media can be old HDDs connected to the RPis. A UPS covering your RPis, your WLAN access point and your consoles [and even your internet access infrastructure] is easily affordable.

I personally am relying on my own local private cloud (in modern terms). My weather data retrieved and logged by Meteobridge, weewx and CumulusMX (potentially a logging "overkill"  ;)) are on different local servers, with a UPS protected local infrastructure, and with a distributed backup concept of the databases.

The weather networks are more for sharing even though I have my own public website using the templates and depiction of my choice. Relying on them for business critical applications is highly careless and irresponsible.

Also, @plunet, the certificate thing is a speculation on your end as this doesn't refer to the ecowitt.net web site but to the webapi.www.ecowitt.net which is responsible for the ecowitt.app, so Ecowitt say. These are different certificates.
The certificate error would not come up when connecting to https://www.ecowitt.net/home/index?id=your-console-id
your usual access to your Ecowitt dashboard.
What the issue finally was Ecowitt have not revealed and they might not do so at all.
« Last Edit: September 25, 2021, 05:29:17 AM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline plunet

  • Member
  • *
  • Posts: 44
Re: "Ecowitt.net outage on 09/24/2021"
« Reply #28 on: September 25, 2021, 06:30:26 AM »
Also, @plunet, the certificate thing is a speculation on your end as this doesn't refer to the ecowitt.net web site but to the webapi.www.ecowitt.net which is responsible for the ecowitt.app, so Ecowitt say. These are different certificates.
The certificate error would not come up when connecting to https://www.ecowitt.net/home/index?id=your-console-id
your usual access to your Ecowitt dashboard.
What the issue finally was Ecowitt have not revealed and they might not do so at all.

I agree that my suggestions that it's the certificate are speculation as I have no internal knowledge of their IT Infrastructure, but Iwould suggest there is very strong evidence that it was the certificate for webapi.www.ecowitt.net that was the issue.
After we discovered the impact was global and not just one region I suggested that it would be an API or middleware issue.

In that case it's probably the API or middleware that sits behind the ecowitt.net website that permits it to access the backend data that has fallen over. But it's fallen over in more than one global region as it doesn't matter where we get sent by the CDN, it's down for all of us.

I was out and about later in the afternoon yesterday so only had a mobile browser with me which limited my ability to do much more diagnosis, but following that posting several members noticed they were getting certificate errors. davidefa confirmed additional detail that the cert that had expired was for webapi.www.ecowitt.net

Yes, the certificate for webapi.www.ecowitt.net/user/site/login expired 24/09/2021
I could login correctly to ecowitt.net setting the computer date to 23/09/2021

You're right that the certificate for www.ecowitt.net was fine and had no issues, but the login process and logged in webpage both make several calls in the background to webapi.www.ecowitt.net.... calls such as

[https:]//webapi.www.ecowitt.net/index/get_device_info
[https:]//webapi.www.ecowitt.net/user/site/switch_language
[https:]//webapi.www.ecowitt.net/user/site/get_user_info
[wss:]//webapi.www.ecowitt.net/mqtt
[https:]//webapi.www.ecowitt.net/index/get_devices
[https:]//webapi.www.ecowitt.net/index/home


I have made these non-clickable as they are not meant to be accessed directly as a naked URL...

These are all clearly visible if you use the developer options in a web browser and look at the calls that the ecowitt.net website is making when you log in and refresh a logged in page.

The fact that the certificate expired yesterday, that it has been changed overnight to one that expires in 2022, and it's name webapi - being a API that supports web requests is all speculation but strong evidence that this was if not the root cause, almost certainly implicated in the issue.

The app which I didn't know existed and was happy to discover through this process probably uses a different API that has a different certificate that didn't expire at the same time. I could probably work that out if I had some more time....