Author Topic: SmartHUB data costs  (Read 2013 times)

0 Members and 1 Guest are viewing this topic.

Offline iszekeres

  • Member
  • *
  • Posts: 19
SmartHUB data costs
« on: February 16, 2018, 01:09:42 AM »
Since I have been reading through the hoopla surrounding the upcoming EOL for the hub, I have seen cost mentioned as the driving reason behind the change.....I further saw mention that AWS is being used for the back-end.  I was curious as to what the costs might actually be.  These are ALL ASSUMPTIONS.  I simply would like to start a discussion if possible.....

Hopefully someone who is more familiar with AWS can chime in, it looks as though inbound data to AWS is free?!?!?!?!? Granted they have to get the data to WU, but jeeze you would think WU already has their own AWS stuff, and it looks as though data between AWS instances are free as well?!

So through some hints it looks as though they are using AWS for their back end. I just did some quick calculations.....

Very roughly (from memory) the strings are around 200 bytes long. This would be sent every 18 seconds, per sensor.

This works out to just under 1 meg per day per sensor data transfer.

Assuming 1,000,000 sensors reporting, as I really have NO idea what their sales are.....this is about 1tb per day data transfer to their AWS servers.

I am not up on AWS products or pricing, but a quick look seems to indicate inbound data is not charged. They would have to pay to send data to WU, assuming WU does not also have some sort of AWS instance they can send the data to for free as well......worst case is 15.6c per gig for data. Round that up to 16 and ignore any discount as the amount of data transferred increases......

Roughly 160 bucks a day for 1TB data transfer costs.

Online nincehelser

  • Forecaster
  • *****
  • Posts: 2653
    • nincehelser.com
Re: SmartHUB data costs
« Reply #1 on: February 16, 2018, 05:24:03 AM »
Since I have been reading through the hoopla surrounding the upcoming EOL for the hub, I have seen cost mentioned as the driving reason behind the change.....I further saw mention that AWS is being used for the back-end.  I was curious as to what the costs might actually be.  These are ALL ASSUMPTIONS.  I simply would like to start a discussion if possible.....

Hopefully someone who is more familiar with AWS can chime in, it looks as though inbound data to AWS is free?!?!?!?!? Granted they have to get the data to WU, but jeeze you would think WU already has their own AWS stuff, and it looks as though data between AWS instances are free as well?!

So through some hints it looks as though they are using AWS for their back end. I just did some quick calculations.....

Very roughly (from memory) the strings are around 200 bytes long. This would be sent every 18 seconds, per sensor.

This works out to just under 1 meg per day per sensor data transfer.

Assuming 1,000,000 sensors reporting, as I really have NO idea what their sales are.....this is about 1tb per day data transfer to their AWS servers.

I am not up on AWS products or pricing, but a quick look seems to indicate inbound data is not charged. They would have to pay to send data to WU, assuming WU does not also have some sort of AWS instance they can send the data to for free as well......worst case is 15.6c per gig for data. Round that up to 16 and ignore any discount as the amount of data transferred increases......

Roughly 160 bucks a day for 1TB data transfer costs.

So what's the cost of processing that data, Ian?  How many transactions are going to be incurred sorting the data and putting it in a database?

There's a lot more to running a data ingestion system than pushing bits around.

You really need to start with a functional diagram of the AWS services you're going to need to accomplish the task.
« Last Edit: February 16, 2018, 05:35:05 AM by nincehelser »

Offline iszekeres

  • Member
  • *
  • Posts: 19
Re: SmartHUB data costs
« Reply #2 on: February 16, 2018, 05:59:40 AM »

So what's the cost of processing that data, Ian?  How many transactions are going to be incurred sorting the data and putting it in a database?

There's a lot more to running a data ingestion system than pushing bits around.


Of course there are more costs associated George, but we are talking about data that would be handed back to the server regardless of the device sending.  As I have mentioned on the company forum, you are the one that keeps bringing up cost as a reason for killing off the old hubs.  Is the data from the access not going to be processed and put into a database as well?  Sure, some of the data will be worked on at the hub.  With your connections at acurite, why don't you give us some insight into on the amount of processing the new hub will save.

Looking a bit more into AWS pricing, there should be NO cost send data to the back end. 

I was speaking with a friend about the issues and your "contributions".  She said you sounded like the lumbago brigade on 75m.  Sure enough!  You really should upgrade your ticket, you would fit right in down there.

Before you start crying about personal attacks, please take a look at your tone in your responses.  You have, and continue, to imply that anyone with a differing opinion is simply not as intelligent as you are.  Reminds me of the guy down the block who thinks that everyone is crazy but himself.....

Online nincehelser

  • Forecaster
  • *****
  • Posts: 2653
    • nincehelser.com
Re: SmartHUB data costs
« Reply #3 on: February 16, 2018, 06:21:00 AM »

So what's the cost of processing that data, Ian?  How many transactions are going to be incurred sorting the data and putting it in a database?

There's a lot more to running a data ingestion system than pushing bits around.


Of course there are more costs associated George, but we are talking about data that would be handed back to the server regardless of the device sending.  As I have mentioned on the company forum, you are the one that keeps bringing up cost as a reason for killing off the old hubs.  Is the data from the access not going to be processed and put into a database as well?  Sure, some of the data will be worked on at the hub.  With your connections at acurite, why don't you give us some insight into on the amount of processing the new hub will save.

Looking a bit more into AWS pricing, there should be NO cost send data to the back end. 

I was speaking with a friend about the issues and your "contributions".  She said you sounded like the lumbago brigade on 75m.  Sure enough!  You really should upgrade your ticket, you would fit right in down there.

Before you start crying about personal attacks, please take a look at your tone in your responses.  You have, and continue, to imply that anyone with a differing opinion is simply not as intelligent as you are.  Reminds me of the guy down the block who thinks that everyone is crazy but himself.....

With the SmartHUB, all data from all sensors are sent to myAcurite as they occur.  That means the myAcurite servers need to sort out all the data from all sensors from all users and get it into a database.  It's a large amount of traffic that incurs a lot of cost on AWS.

The Access, on the other hand, bundles all the local sensor data and sends it to myAcurite in 5-minute intervals.  That means the myAcurite servers don't have to handle nearly as much data, and that data is already pre-sorted by device.  That means far less data for AWS to process, and thus a lower cost.

Let's take the transmissions from a 5n1, for example.  On a SmartHUB, it will generate about 17 messages in a 5 minute period.  The Access, on the other hand, will just generate 1.  If you add a tower sensor to the mix, the SmartHUB will generate a total of 35 messages in that 5 minute period, while the Access will still be at 1 message, although a bit larger than before.




« Last Edit: February 16, 2018, 06:41:18 AM by nincehelser »

Offline iszekeres

  • Member
  • *
  • Posts: 19
Re: SmartHUB data costs
« Reply #4 on: February 16, 2018, 06:44:13 AM »

With the SmartHUB, all data from all sensors is sent to myAcurite as they occur.  That means the myAcurite servers need to sort out all the data from all sensors from all users and get it into a database.  It's a large amount of traffic that incurs a lot of cost on AWS.

The Access, on the other hand, bundles all the local sensor data and sends it to myAcurite in 5-minute intervals.  That means the myAcurite servers don't have to handle nearly as much data, and that data is already pre-sorted by device.  That means far less data for AWS to process, and thus a lower cost.

George, I was never arguing that the access did not save money for acurite.  I brought up the cost of the AWS services to try and deduce what the cost savings may be.  I want to deduce the cost savings so I can better decide how moronic the decision to EOL the hub is.  Are they saving 10 bucks a month by doing this? 100? 100,000?  From what I have researched so far, the AWS costs are not exorbitant. If they are saving a small amount of money, well I hope it was worth all the bad blood.  If they are saving a ton of money, I would say planning was poor enough to get themselves into a mess.

You make some broad statements.....alot, far less.....some informed numbers would be helpful.




Offline John Z

  • Member
  • *
  • Posts: 34
Re: SmartHUB data costs
« Reply #5 on: February 16, 2018, 07:23:08 AM »
Ian,

We don't know the size of the hub installed base, or the average number of sensors per hub. The best one can do is to SWAG some range estimates for that, and calculate up, based on additional assumptions as to AWS use.

I don't think that AcuRite would have done this just to save a little. Further, I don't think they would have done this if the status quo was sustainable. Just too painful.




Offline iszekeres

  • Member
  • *
  • Posts: 19
Re: SmartHUB data costs
« Reply #6 on: February 16, 2018, 07:33:58 AM »
Ian,

We don't know the size of the hub installed base, or the average number of sensors per hub. The best one can do is to SWAG some range estimates for that, and calculate up, based on additional assumptions as to AWS use.

I don't think that AcuRite would have done this just to save a little. Further, I don't think they would have done this if the status quo was sustainable. Just too painful.

I also didn't think they would brick hubs or manage the last roll out so poorly either ;)

I was hoping George would pull some strings to get some useful data to back up his proclamations.


Offline John Z

  • Member
  • *
  • Posts: 34
Re: SmartHUB data costs
« Reply #7 on: February 16, 2018, 07:45:06 AM »
I did a back-of-the envelope calculation based on a single user who claims the use of 18 towers and a 5n1. I assumed each hub transmission was an API call to AWS, and used their published per-million API calls rate as the only charge in this calculation. It came up to over $100 bucks annually to support that one user. If that is anything close to correct, that is stunning.

Offline iszekeres

  • Member
  • *
  • Posts: 19
Re: SmartHUB data costs
« Reply #8 on: February 16, 2018, 08:07:46 AM »
I did a back-of-the envelope calculation based on a single user who claims the use of 18 towers and a 5n1. I assumed each hub transmission was an API call to AWS, and used their published per-million API calls rate as the only charge in this calculation. It came up to over $100 bucks annually to support that one user. If that is anything close to correct, that is stunning.

Truly!

So the new hub would cache the data, and send every 5 min.  No matter how many sensors you have.  288 calls a day total per user then instead of ~4800 calls per sensor per day.

Soooooo yes, they will save a butt-load of money.  ~105k calls per year per user.  In your example it was something like 33mil calls per year for that user. A substantial savings.  I am bleary but does that work out to 0.36c for that particular user for the year? Almost enough to cover the cost of upgrading someone with a free access hub, just to get them off the old one :P

Offline John Z

  • Member
  • *
  • Posts: 34
Re: SmartHUB data costs
« Reply #9 on: February 16, 2018, 08:16:23 AM »
I think you've got the gist of it, Ian.

It may be that retiring hubs in the way you suggest is affordable over the long run, but it would be a cash-flow nightmare.

I plunked down my $40, installed my new device, and am ready to move on.

Offline iszekeres

  • Member
  • *
  • Posts: 19
Re: SmartHUB data costs
« Reply #10 on: February 16, 2018, 08:27:09 AM »
I think you've got the gist of it, Ian.

It may be that retiring hubs in the way you suggest is affordable over the long run, but it would be a cash-flow nightmare.

I plunked down my $40, installed my new device, and am ready to move on.

I am moving on as well.  I currently sniff traffic to report to CWOP in addition to WU.  Looks like that will no longer work.  In addition, the HF weather net I used to check into disbanded since the move to Cocorahs reporting.  A window and a wet finger will do for me once the old hub crashes and burns.

George, you are still a wiener ;)

Offline John Z

  • Member
  • *
  • Posts: 34
Re: SmartHUB data costs
« Reply #11 on: February 16, 2018, 08:34:07 AM »
Good luck, Ian & BTW, thanks for re-checking my math.
KJ4A

Offline iszekeres

  • Member
  • *
  • Posts: 19
Re: SmartHUB data costs
« Reply #12 on: February 16, 2018, 08:36:21 AM »
Good luck, Ian & BTW, thanks for re-checking my math.
KJ4A

You too,
73
ac9kw

Offline vreihen

  • El Niņo chaser
  • Forecaster
  • *****
  • Posts: 876
  • K2BIG
Re: SmartHUB data costs
« Reply #13 on: February 16, 2018, 06:41:59 PM »
I am moving on as well.  I currently sniff traffic to report to CWOP in addition to WU.  Looks like that will no longer work.  In addition, the HF weather net I used to check into disbanded since the move to Cocorahs reporting.

Sounds like it is time to re-purpose one of the gazillion SDR receiver dongles that every ham should have in their shack junk drawer to grab the Acu-Rite data straight off the air with rtl_433 and post it wherever using weewx.....
WU Gold Stars for everyone! :lol:

Offline John Z

  • Member
  • *
  • Posts: 34
Re: SmartHUB data costs
« Reply #14 on: February 19, 2018, 12:10:36 PM »
Thinking about this a bit more, I wouldn't be surprised if the Spectre/Meltdown security mess played a role here, too.

One of the ways to mitigate this CPU vulnerability involves turning off predictive instruction pre-fetching. A machine, or a virtualization, might suddenly lose as much as 40% of capacity as the patch is applied. More hardware is then needed to get the same job done, and so plenty of pain to spread around.
« Last Edit: February 19, 2018, 02:15:30 PM by John Z »

Offline gdicarlo

  • Member
  • *
  • Posts: 8
Re: SmartHUB data costs
« Reply #15 on: February 19, 2018, 05:49:25 PM »
The whole idea of the Access/ smarthub is illogical. It doesn't make sense to send my data to Myacurite to be stored, and then have to download it to my computer. It makes much more sense to have a data logger that connects to the sensor or display that you can download from. I know they sell the one display that saves data every 12 minutes, but they need one that saves it more often. That would also save Acurite a lot of money from not having to run their servers. They really screwed the pooch on this one.

Online nincehelser

  • Forecaster
  • *****
  • Posts: 2653
    • nincehelser.com
Re: SmartHUB data costs
« Reply #16 on: February 19, 2018, 07:18:35 PM »
The whole idea of the Access/ smarthub is illogical. It doesn't make sense to send my data to Myacurite to be stored, and then have to download it to my computer. It makes much more sense to have a data logger that connects to the sensor or display that you can download from. I know they sell the one display that saves data every 12 minutes, but they need one that saves it more often. That would also save Acurite a lot of money from not having to run their servers. They really screwed the pooch on this one.

I responded to your same comment on another thread.

myAcurite isn't geared toward long-term data storage or even downloading data to your PC.  It's more of a way of being able to access your weather and house monitoring data remotely via a website or mobile app.

The Atlas displays have a logger that writes to a removable SD card every 10 seconds.

Online nincehelser

  • Forecaster
  • *****
  • Posts: 2653
    • nincehelser.com
Re: SmartHUB data costs
« Reply #17 on: February 19, 2018, 07:21:33 PM »
Thinking about this a bit more, I wouldn't be surprised if the Spectre/Meltdown security mess played a role here, too.

One of the ways to mitigate this CPU vulnerability involves turning off predictive instruction pre-fetching. A machine, or a virtualization, might suddenly lose as much as 40% of capacity as the patch is applied. More hardware is then needed to get the same job done, and so plenty of pain to spread around.

Oh, yeah.  I'm sure that little problem is affecting the budget of many data centers.

And let's not forget the problems it recently caused for wunderground (also hosted on AWS).

Offline iszekeres

  • Member
  • *
  • Posts: 19
Re: SmartHUB data costs
« Reply #18 on: February 20, 2018, 04:51:05 PM »


Oh, yeah.  I'm sure that little problem is affecting the budget of many data centers.

And let's not forget the problems it recently caused for wunderground (also hosted on AWS).

Who could forget, as I am sure we all have been inundated with requests for subscription cost increases and depreciated hardware.  Why just yesterday I received an email from WU notifying me that the data costs were too high and I would need to pay to post my data.

Oh wait, it was only acurite....NM!

 :roll:


Offline John Z

  • Member
  • *
  • Posts: 34
Re: SmartHUB data costs
« Reply #19 on: February 20, 2018, 05:04:12 PM »
My two WU pages have been off-and-on for weeks now.
I am concerned that the pressure will cause them to fold, too.
I would not be surprised to see Rapid-Fire vanish.

Offline iszekeres

  • Member
  • *
  • Posts: 19
Re: SmartHUB data costs
« Reply #20 on: February 20, 2018, 06:03:26 PM »
My two WU pages have been off-and-on for weeks now.
I am concerned that the pressure will cause them to fold, too.
I would not be surprised to see Rapid-Fire vanish.

Huh, mine has been working flawlessly for the past few weeks.  Are you sure there are no configuration issues on your end?

Offline John Z

  • Member
  • *
  • Posts: 34
Re: SmartHUB data costs
« Reply #21 on: February 20, 2018, 06:08:08 PM »
Ian,

No configuration issues, just random data outages, with the added feature of irrelevant Disqus comments posted beneath.

Offline iszekeres

  • Member
  • *
  • Posts: 19
Re: SmartHUB data costs
« Reply #22 on: February 20, 2018, 06:22:00 PM »
Ian,

No configuration issues, just random data outages, with the added feature of irrelevant Disqus comments posted beneath.

Hrm, that seems odd as it has been working fine for me and everyone I know.  I would work with George to sort things out as there must be something wrong with your config to have this much trouble.


Online nincehelser

  • Forecaster
  • *****
  • Posts: 2653
    • nincehelser.com
Re: SmartHUB data costs
« Reply #23 on: February 20, 2018, 06:37:20 PM »
There have been some reports of trouble with wunderground the last week or so.  Typical yo-yo stuff.  Nothing widespread, though.


Offline Glenn

  • Forecaster
  • *****
  • Posts: 365
Re: SmartHUB data costs
« Reply #24 on: February 21, 2018, 09:10:12 AM »
There have been a few minor issues with my WU pages as well. Station reporting times, missing temps on the map. They seem to come and go as of late.
Acurite 5 in 1 Pro+
Southern VT
Connecticut