Topic: NOAA Reports not being populated

[PernicketyPete]

Hi All
I'm having an issuse with cumulusmx populating the NOAA reports. It would appear that the report is being updated as the time and date at the bottom of the window are changing on day rollover and I can force the update using Utils, but no new information is being displayed. The report has at sometime worked correctly as the 15th July is displayed, this is the day installed Cumulusmx, so not sure if I have changed something in settings and broken it.! I am running Cumulusmx on a raspberry pi, not sure if this will make a difference.? Link to website below. Hope someone can point me in the right direction.
Regards Pete

https://www.chilthorneweather.co.uk/Cumulus/index.htm

[PaulMy]

Hi,
Your reports files only have data for July 15/23 so likely a settings issue in the uploads  https://www.chilthorneweather.co.uk/Cumulus/Reports/NOAAMO0723.txt

The MXdiags log file that includes the day changeover period may give you some indication. 
Are you using FTP or PHP for uploads and what is your setting in NOAA settings > Upload Option > Upload folder: ?

Enjoy,
Paul

[PernicketyPete]

Hi Paul
Thanks for your prompt response.
I am using FTP.  NOAA settings > Upload Option > Upload folder: = /public_html/Cumulus/Reports/
I have set up the logging options and will report back.

Many thanks Pete

[PaulMy]

Hi,
Presumably your Internet settings > Remote FTP Directory is set correctly as your standard files are being updated.  Is that setting also /public_html/Cumulus (with a leading /)?
I have found that usually you would not have the trailing slash / after Reports so try it with /public_html/Cumulus/Reports or public_html/Cumulus/Reports

Unfortunately you will need to wait until a day rollover to find out as the reports are created and updated only at the end of day rollover.  You can also manually update the remote /Cumulus/Reports folder with missing files through FileZilla if you want to get the remote updated before the end of day.

Enjoy,
Paul

[PernicketyPete]

Hi Paul
I have changed the NOAA Upload Settings to: = public_html/Cumulus/Reports   

I will wait and see what happens.

Thanks

[PernicketyPete]

Hi
With the upload folder set as  “public_html/Cumulus/Reports”  it would seem that a new report was created and uploaded, but still no new data is being displayed?
Where does CumulusMX look for the information it needs to generate the NOAA report? Could it be that apart from the 15th July there is no other days of data to use? Is it looking in the wrong place? Is data not being stored? The day file seems to be complete and up to date.

Regards Pete

[PaulMy]

Hi Pete,

it would seem that a new report was created and uploaded, but still no new data is being displayed?

In your local Dashboard > Reports folder does the NOAA Month Report contain data for all the days that you have been using CumulusMX?

Where does CumulusMX look for the information it needs to generate the NOAA report?
Could it be that apart from the 15th July there is no other days of data to use? Is it looking in the wrong place?  The day file seems to be complete and up to date.

  It [/size]comes from the dayfile.txt.  You indicate that the dayfile is complete for all the days you have been using CumulusMX.  Do all the days appear in your local Dashboard > Data Logs > Dayfile?

[/size]Was there anything in your MXdiags folder for the day's rollover time like this?

Code: [Select]

2023-07-31 00:00:00.173 Creating NOAA monthly report for July 30, 2023
2023-07-31 00:00:00.173 Creating NOAA monthly report
2023-07-31 00:00:00.261 Saving monthly NOAA report as Reports\NOAAMO0723.txt
2023-07-31 00:00:00.262 Creating NOAA yearly report
2023-07-31 00:00:00.262 Creating NOAA yearly report
2023-07-31 00:00:00.748 Saving yearly NOAA report as Reports\NOAAYR2023.txt
2023-07-31 00:00:00.749 NOAA reports will be uploaded at next web update


Enjoy,
Paul

[PernicketyPete]

Hi Paul
On my local Dashboard the NOAA report is the same as that published on the web - (only the 15th showing). All days I have been using CumulusMX are in the day file on my local Dashboard (15th to 30th inclusive)
on closer inspection of the July NOAA report, I can see that the summary line that appears under the last day displays entries that are different from my single line of data - the 15th. It looks as if Cumulus is working out the averages at the end of every day but not showing that days data??

Below is what appeared in the MXdiags Log

cheers Pete

2023-07-31 00:00:03.684 FTP[Int]: Uploading NOAA reports

2023-07-31 00:00:03.685 >         FileExists("public_html/Cumulus/Reports/NOAAMO0723.txttmp")
2023-07-31 00:00:03.685 Command:  SIZE /public_html/Cumulus/Reports/NOAAMO0723.txttmp
2023-07-31 00:00:03.685 Status:   Waiting for response to: SIZE /public_html/Cumulus/Reports/NOAAMO0723.txttmp
2023-07-31 00:00:03.860 Response: 550 Can't check for file existence [175ms]
2023-07-31 00:00:03.861 FTP[Int]: Uploading public_html/Cumulus/Reports/NOAAMO0723.txttmp
2023-07-31 00:00:03.861 >         UploadStream("public_html/Cumulus/Reports/NOAAMO0723.txttmp", Overwrite, False)
2023-07-31 00:00:03.861 >         FileExists("public_html/Cumulus/Reports/NOAAMO0723.txttmp")
2023-07-31 00:00:03.861 Command:  SIZE /public_html/Cumulus/Reports/NOAAMO0723.txttmp
2023-07-31 00:00:03.861 Status:   Waiting for response to: SIZE /public_html/Cumulus/Reports/NOAAMO0723.txttmp
2023-07-31 00:00:04.014 Response: 550 Can't check for file existence [152ms]
2023-07-31 00:00:04.015 >         OpenWrite("public_html/Cumulus/Reports/NOAAMO0723.txttmp", Binary)
2023-07-31 00:00:04.015 Command:  TYPE I
2023-07-31 00:00:04.015 Status:   Waiting for response to: TYPE I
2023-07-31 00:00:04.165 Response: 200 TYPE is now 8-bit binary [149ms]
2023-07-31 00:00:04.165 >         OpenDataStream("STOR public_html/Cumulus/Reports/NOAAMO0723.txttmp", 0)
2023-07-31 00:00:04.166 >         OpenPassiveDataStream(AutoPassive, "STOR public_html/Cumulus/Reports/NOAAMO0723.txttmp", 0)
2023-07-31 00:00:04.166 Command:  EPSV
2023-07-31 00:00:04.166 Status:   Waiting for response to: EPSV
2023-07-31 00:00:04.321 Response: 229 Extended Passive mode OK (|||42651|) [154ms]
2023-07-31 00:00:04.321 Status:   Connecting to IP #1= ***:42651
2023-07-31 00:00:04.460 Command:  STOR public_html/Cumulus/Reports/NOAAMO0723.txttmp
2023-07-31 00:00:04.460 Status:   Waiting for response to: STOR public_html/Cumulus/Reports/NOAAMO0723.txttmp
2023-07-31 00:00:04.611 Response: 150 Accepted data connection [150ms]
2023-07-31 00:00:04.612 Status:   Uploaded 969 bytes
2023-07-31 00:00:04.612 Status:   Closing/Disposing FtpSocketStream(data connection)
2023-07-31 00:00:04.612 Status:   Waiting for response to: *UPLOAD*
2023-07-31 00:00:04.794 Response: 226-File successfully transferred
2023-07-31 00:00:04.794 Response: 226 0.153 seconds (measured here), 6.20 Kbytes per second [333ms]
2023-07-31 00:00:04.795 FTP[Int]: Renaming public_html/Cumulus/Reports/NOAAMO0723.txttmp to public_html/Cumulus/Reports/NOAAMO0723.txt
2023-07-31 00:00:04.795 >         Rename("public_html/Cumulus/Reports/NOAAMO0723.txttmp", "public_html/Cumulus/Reports/NOAAMO0723.txt")
2023-07-31 00:00:04.795 Command:  RNFR /public_html/Cumulus/Reports/NOAAMO0723.txttmp
2023-07-31 00:00:04.795 Status:   Waiting for response to: RNFR /public_html/Cumulus/Reports/NOAAMO0723.txttmp
2023-07-31 00:00:04.956 Response: 350 RNFR accepted - file exists, ready for destination [160ms]
2023-07-31 00:00:04.956 Command:  RNTO /public_html/Cumulus/Reports/NOAAMO0723.txt
2023-07-31 00:00:04.956 Status:   Waiting for response to: RNTO /public_html/Cumulus/Reports/NOAAMO0723.txt
2023-07-31 00:00:05.115 Response: 250 File successfully renamed or moved [158ms]
2023-07-31 00:00:05.116 FTP[Int]: Renamed public_html/Cumulus/Reports/NOAAMO0723.txttmp

[PaulMy]

Hi Pete,
From what you describe the error appears to be in the creation of the NOAA file.  Your MXdiags snipped covers the time that the NOAA files are being uploaded but not from a fraction earlier when they are being created at end of day.  Have you tried to (Re)Generate both the Monthly and Yearly Reports?  When I just did a (Re)Generate on my CMX I got this logged in my MXDiags

Code: [Select]

2023-07-31 15:20:26.038 Creating NOAA monthly report
2023-07-31 15:20:26.097 Saving monthly NOAA report as Reports\NOAAMO0723.txtAfter you do a (Re)Generate look in your MXDiags and see if you get the same.

Presumably your NOAA settings are ok as the first day was generated and uploaded.  A cause could be that there is bad data in the dayfile.txt.  Can you see any abnormality in the local Dashboard > Dayfile view - difference in date format and data items between July 15 and later?

Enjoy,
Paul

[PernicketyPete]

Hi Paul
I have now regenerated both the monthly and the yearly reports 3 times, I don't see anything that says "creating" or "saving" in my MXdiags. Just the same as before - snippet  posted earlier. I have inspected the day file all the data looks good. However, I have noticed that I do have in my local Reports folder on the pi, a NOAA report file for June, that has just one line of data that only has dashes and zeros in it and also an A with accent instead of a decimal point within the latitude and longitude coordinates. could this be the problem? I have tried to edit / delete this, but need to learn how to do this on the pi, as at the moment it tells me I don't have permission.

cheers Pete

[PaulMy]

Hi Pete,

I have noticed that I do have in my local Reports folder on the pi, a NOAA report file for June, that has just one line of data that only has dashes and zeros in it and also an A with accent instead of a decimal point within the latitude and longitude coordinates. could this be the problem?

On the A with accent, I have seen that before and may just be a UTF-8 encoding issue (of which I understand very little) and should have no issue in the file creation.  However having a June NOAA file which is before your July 15th start date could be an issue. One thing you can try is remove everything from your \CumulusMX\Reports folder so it is empty, or delete the whole folder and then add it in as a new folder (nothing to lose and if all is working properly you can (Re)Generate the July report files at any time as long as your dayfile.txt is good.

I would have a more detailed look if you zip-up and PM me your dayfile.txt and your Cumulus.ini file (remove your Host name, Username, and Password from the [FTP Site] section (there is no other confidential information in this settings file), and also the file in the MXdiags folder that includes the end of day rollover period - I presume it is midnight but if you use another time - i.e. 9:00 am then from around that time.

p.s. One other thought, what is your Station settings > Advanced Options for your records began date?

Code: [Select]

Records Began Date:


IMPORTANT: You MUST keep the date format YYYY-MM-DD.
This is the 'Date of birth' for your Cumulus install, it defaults to the first time you run the program.
 
You may need to alter this if for instance you import historic data in Cumulus from another source, in that case set it to the first date of your imported data.
Enjoy,
Paul

[PernicketyPete]

Hi Paul
Thanks for the advice. Once I can figure out how to delete etc on the pi I will delete the Report folder as you suggested. I’m off to work now, but will get the zipped up files to you later.
Thanks.

Cheers Pete

[mcrossley]

The snippet of log posted above is from the FTP log file, not the MXdiags log file.

[PernicketyPete]

Hi Mark
thanks for the that. when I looked in the right file I could see that the NOAA repots were/are being created.

Paul
I had a look in the "Jul23log.txt", it had a lot of white space at the beginning of the 16th, I have now edited that out and saved the file, (I have learnt how to do that as well) I regenerated the July report on the dashboard and all is looking good.! A  complete report is displayed.   I think we have got it sorted.?
Thanks so much for your help, I would never have got there on my own.

Cheers Pete

[PaulMy]

Hi Pete, good to hear and had seen the updated report on your website.
Yes, invalid characters in data files is often the cause.  And while I asked about the dayfile.txt I should have also mentioned the monthly log file.


Enjoy,
Paul

[mkutche]

mine aren't uploading either and idk why i haven't changed any settings to when I had my WS-2080 weather station.. i just recently got a Davis Vantage Vue.

http://gosportwx.com/wxtempsummary.php data should of stopped uploading Jun 29th 2023 because that was when a big tree landed on it and took it out.

[PaulMy]

Hi,

mine aren't uploading either and idk why i haven't changed any settings to when I had my WS-2080 weather station.. i just recently got a Davis Vantage Vue.

You may need to check your NOAA reports settings, and also check the MXdiags file that has the end of day rollover period and search for NOAA reports and see if there are any errors in doing the NOAA processing and upload.  You can (Re)Generate the reports as long as there is data for the period that wasn't initially generated.

Enjoy,
Paul

[mkutche]

Hi,

mine aren't uploading either and idk why i haven't changed any settings to when I had my WS-2080 weather station.. i just recently got a Davis Vantage Vue.

You may need to check your NOAA reports settings, and also check the MXdiags file that has the end of day rollover period and search for NOAA reports and see if there are any errors in doing the NOAA processing and upload.  You can (Re)Generate the reports as long as there is data for the period that wasn't initially generated.

Enjoy,
Paul

seems like it changed the directory to cumx/Reports instead of /Reports odd. thing had a mind of its own it seemed lol.. we'll see if it generates at 12am.