WXforum.net

Miscellaneous Debris => Tech Corner => Topic started by: W3DRM on August 06, 2016, 08:04:56 PM

Title: About to FLING FLING...
Post by: W3DRM on August 06, 2016, 08:04:56 PM
No, that's not a typo. I've about had it with FLING. Since upgrading to Win10 and 1and1's simultaneous move to force everyone to SFTP, I have not been able to update my website files with FLING as I had in the past.

The symptoms are that I can never get a connection to the server to even begin the upload. The connection just times out and the transfer aborts. Filezilla works just fine and I can access my website without any issues it's just that FLING won't connect. FLING works fine on all of my other websites that don't require SFTP connections. 1and1 Support has been totally useless and unable to help. To them, everything is working fine.

I have attempted to use every combination of options available on the FLING setup for SFTP (implied and explicit) to no avail.

Does anyone have any suggestions that I can try? I have the latest version of FLING (2.35). Is there some other software I could use that has similar functionality of FLING?
Title: Re: About to FLING FLING...
Post by: saratogaWX on August 06, 2016, 09:47:00 PM
I'm running 2.35 also.

I had to change the properties on 'fling.exe' to use 'Windows 7' compatibility mode to get it to connect successfully :)

However.. if you do that, it may also engage the memory-munching non-pageable memory ..  I have to run a small perl script with the task scheduler to watch for that, then use the pskill tool to pop the service (it's the service that is doing it).  I reported the bug to NCH, but no word ever came back. :(

Title: Re: About to FLING FLING...
Post by: W3DRM on August 06, 2016, 11:34:16 PM
Well, tried your suggestion but it didn't make any difference. I also used the Windows troubleshooter and it changed it to Win 8 compatibility mode without any positive results.

Now, I am realizing that VWS doesn't seem to be able to handle SFTP transfers either so this is becoming a more serious issue than I had first thought. I've searched through the latest VWS PDF Manual but it has no reference to SFTP at all.

WD seems to be working okay using SFTP. That may not be 100% correct but at least I do see some files getting through to the server which is better than it was earlier this afternoon.

As far as your memory-munching of non-pageable memory, does StartWatch have the capability to catch that particular parameter? I know it catches some kind of memory usage but not sure if is the non-pageable value. I have my VWS set to restart when StartWatch detects "Leak Protection - Close & restart if memory above 275 MB" but, I don't know what memory value that refers to. That particular setting, in the past, has been the only thing keeping my VWS running 24/7 for the past few years.

EDIT: Forgot to mention that I have sent an email to Ambient Support asking them if VWS is going to support SFTP in the near future now that many hosting companies are forcing customers to use SFTP. The answer will determine whether I stay with VWS or abandon it completely and go with Weather Display or some other software package.
Title: Re: About to FLING FLING...
Post by: W3DRM on August 07, 2016, 04:47:44 PM
Here is Ed's reply to my question regarding VWS using SFTP:

Quote
I am sorry but VWS does not support SFTP and there are no plans.
Regards,
Ed

Guess I'll have to find another way to get the job done. FLING is probably the best way, but I have to get it working first. I really hate to think that I have to manually set up all of the FTP transfers with FLING or some other software that VWS has routinely been handling for many years.
Title: Re: About to FLING FLING...
Post by: saratogaWX on August 07, 2016, 05:29:06 PM
Don,

I have Fling handle the VWS uploads with two entries:

VWS-main  local: c:\vws\root  every 5 minutes
VWS-noaa local: c:\vws\data\noaa  every 5 minutes

For WeatherFlash, I select the HTTP method to update the site.

Then, no VWS FTP is needed :)
Title: Re: About to FLING FLING...
Post by: waterside on August 07, 2016, 11:59:37 PM
Many providers are slowly switching to forcing secure communication, either via FTPs or SFTP.  The concern is not so much over the actual data, but moreso over the authentication.  Standard ftp sends your password in clear-text, visible to anyone able to snoop on the connection.  The risk and ability to do so is a hotly-debated topic for another time.

The best option is for the native client to support this, but for cases where this either is not supported or does not work with one's particular provider (more common than you'd think), a "bridge" may be an option.

For SFTP, I have used Bitvise (https://www.bitvise.com/ssh-client-download) client as an ftp->sftp bridge.  The client is configured to connect to a provider using sftp, and the local application connects to the local bitvise client acting as a bridge using ftp.  Definitely not ideal, but is a potential option.

Title: Re: About to FLING FLING...
Post by: W3DRM on August 18, 2016, 12:13:31 AM
Ken,

I have revised my VWS uploads so all of them are handled by Fling, as you suggested. However, my FTP uploads are still sporadic. Running a netstat grep, shows at most two simultaneous connections. I have checked this numerous times at various times of the day with the exact same results. Am I interpreting your posts to indicate that if I have an FTP problem from my one and only computer that uploads to my website that I should see those uploads in the netstat reports?

Is there any way to run something similar from the server side to see if I am getting FTP's from some other source than my laptop? 1and1 support says they don't see any indication that I am at or exceeding my FTP limit for my account. My only other thought is that there is something else uploading to my website that I don't know about. Filezilla from any of my computers fails constantly as do manual Fling uploads.

I do have FlightRadar24, Bloomsky and Blitzortung uploads taking place on my router but those uploads do not go to my server.

Still stumped and baffled...
Title: Re: About to FLING FLING...
Post by: ke9lz on October 20, 2016, 10:02:41 AM
Quote
However.. if you do that, it may also engage the memory-munching non-pageable memory ..  I have to run a small perl script with the task scheduler to watch for that, then use the pskill tool to pop the service (it's the service that is doing it).  I reported the bug to NCH, but no word ever came back. :(

Ken,  Would you be willing to share your perl script and how you set it up?  I to have been struggling to keep Fling up and running for longer than a couple of days and have also reported to NCH.

Thanks,  Steve
Title: Re: About to FLING FLING...
Post by: saratogaWX on October 20, 2016, 10:51:52 AM
Quote
However.. if you do that, it may also engage the memory-munching non-pageable memory ..  I have to run a small perl script with the task scheduler to watch for that, then use the pskill tool to pop the service (it's the service that is doing it).  I reported the bug to NCH, but no word ever came back. :(

Ken,  Would you be willing to share your perl script and how you set it up?  I to have been struggling to keep Fling up and running for longer than a couple of days and have also reported to NCH.

Thanks,  Steve
Sure, here's the Perl script (I called it zap-fling.pl)
Code: [Select]
#!/usr/bin/perl
#
# use the pstools to find the current NP memory for fling, and optionally
# use pskill to zap it to release the held memory
#
# K. True - 30-Oct-2011
#
# Run via Windows Scheduler or Linux cron .. all output is to the log file(s)
#
# ---- configurable settings ----
$maxMem = 65392;  # amount in KB that we'll tolerate
$lookFor = 'fling';

$logsDir = "./fling-logs/"; # place to store the YYYYMMDD.txt

# ---- end of configurable settings --
#
$|=1; # no buffering of output

 $cur_time = time();
 @months = ("Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec");

 $log_filename = ISO_datestamp($cur_time) . ".txt";
 $nice_date = nice_datestamp($cur_time);

 open (OUT,">>${logsDir}${log_filename}");

# old-- open(PS,"pslist -m fling |") || die "..unable to run pslist $!\n";

#class Win32_PerfRawData_PerfProc_Process : Win32_PerfRawData
#{
#  string Caption;
#  uint32 CreatingProcessID;
#  uint64 ElapsedTime;
#  uint64 Frequency_Object;
#  uint64 Frequency_PerfTime;
#  uint64 Frequency_Sys100NS;
#  uint32 HandleCount;
#  uint32 IDProcess;
#  uint64 IODataOperationsPerSec;
#  uint64 IOOtherOperationsPerSec;
#  uint64 IOReadBytesPerSec;
#  uint64 IOReadOperationsPerSec;
#  uint64 IOWriteBytesPerSec;
#  uint64 IOWriteOperationsPerSec;
#  uint64 IODataBytesPerSec;
#  uint64 IOOtherBytesPerSec;
#  string Name;
#  uint32 PageFaultsPerSec;
#  uint64 PageFileBytes;
#  uint64 PageFileBytesPeak;
#  uint64 PercentPrivilegedTime;
#  uint64 PercentProcessorTime;
#  uint64 PercentUserTime;
#  uint32 PoolNonpagedBytes;
#  uint32 PoolPagedBytes;
#  uint32 PriorityBase;
#  uint64 PrivateBytes;
#  uint32 ThreadCount;
#  uint64 Timestamp_Object;
#  uint64 Timestamp_PerfTime;
#  uint64 Timestamp_Sys100NS;
#  uint64 VirtualBytes;
#  uint64 VirtualBytesPeak;
#  uint64 WorkingSet;
#  uint64 WorkingSetPeak;
#};
$wmicArgs = "Name,IDProcess,VirtualBytes,WorkingSet,PrivateBytes,PageFaultsPerSec,PoolNonpagedBytes,PoolPagedBytes";

#open(PS,"wmic path Win32_PerfFormattedData_PerfProc_Process WHERE Name=\"$lookFor\" GET $wmicArgs |") || die "..unable to run wmic $!\n";
open(PS,"wmic path Win32_PerfFormattedData_PerfProc_Process GET $wmicArgs |") || die "..unable to run wmic $!\n";


while (<PS>)
{
  #print $_;
  $rec = $_;
# IDProcess  Name   PageFaultsPersec  PoolNonpagedBytes  PoolPagedBytes  PrivateBytes  VirtualBytes  WorkingSet 
  ($PID,$name,$Faults,$NonP,$Page,$Priv,$VM,$WS) = split(/\s+/);

#old  ($name,$PID,$VM,$WS,$Priv,$Faults,$NonP,$Page) = split(/\s+/);
  next unless $WS =~ m/\d+/;
  next unless $name =~ m/fling/;
 
  #print STDOUT "$name\t$PID\t$NonP\n";
  print OUT "$nice_date\t$rec";
  print STDOUT "$nice_date\t$rec";
  $NonP = int($NonP / 1024);
 
  if($NonP >= $maxMem) {
    print OUT "$nice_date\t$name PID=$PID pskill for $NonP > $maxMem KB\n";
    print STDOUT "$nice_date\t$name PID=$PID pskill for $NonP > $maxMem KB\n";
open(PK,"pskill $PID |") || die "..unable to run pskill $!\n";
while (<PK>) {
    print OUT "$nice_date\t$_";
    print STDOUT "$nice_date\t$_";
 
}
    print OUT "$nice_date\t-------------------------------------------\n";
    print STDOUT "$nice_date\t-------------------------------------------\n";

  }
}

# ------  end of main program -----

sub nice_datestamp {
    my $d = shift;
   
    my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($d);
    my $nicedate = sprintf(
        "%02d\-%s\-%04d %02d\:%02d\:%02d",
        $mday,$months[$mon],$year+1900,$hour,$min,$sec);
    return("$nicedate");

}

sub ISO_datestamp {
    my $d = shift;
   
    my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($d);
    my $nicedate = sprintf("%04d%02d%02d",$year+1900,$mon+1,$mday);
    return("$nicedate");

}
You'll have to have installed Perl from activestate.com (link here (http://www.activestate.com/activeperl)) and the pskill utility from Microsoft SystemTools here (https://technet.microsoft.com/en-us/sysinternals/bb842062.aspx?f=255&MSPPError=-2147217396).

The script uses the built-in wmic command to find/list the memory used by fling, and if it's larger than 64MB, issue a pskill to kill off the process and thereby release the non-pagable memory.  Fling will start a new process when needed.

I use the Windows Scheduler to run the zap-fling.pl program every 15 minutes round the clock.  It logs the results in a yyyymmdd.txt file in the ./fling-logs/ directory.

Hope this helps...
Best regards,
Ken
Title: Re: About to FLING FLING...
Post by: ke9lz on October 20, 2016, 07:34:02 PM
Hi Ken,  Yes, this helps a lot!  Seems to have done the trick!  Thank you very much!

Steve
Title: Re: About to FLING FLING...
Post by: zmarfak on February 04, 2018, 08:13:31 AM
Hi,
got the same memory issue (see attachment, monitored by free PRTG)
problems started after installing fling when I checked on the date I started to use it
When I searched on the net this tread came up.
Apparently no change on the software since 2016

I just implemented the script of Ken
I will keep an eye on it.
 
Fling version 2.35 is running here on Microsoft Windows 10 Home, OS Version:  10.0.16299 , 16MB memory
Title: Re: About to FLING FLING...
Post by: GregJ on February 05, 2018, 02:10:05 PM
I used Fling for years...  I kept 2.21 and it was rock solid.  Recently I had to make a change...  I had too many simultaneous FTP sessions and my webhost was blacklisting my IP because their firewall was interpreting my activity as an attack....  So, I consolidated all of my ftp activity into a single session that I run every 2 minutes.  I am using www.scriptftp.com (http://www.scriptftp.com).  It has been fantastic and very easy to customize over my network.  Great customer service during setup too....

Thought I'd Share,
Greg
Title: Re: About to FLING FLING...
Post by: zmarfak on February 05, 2018, 04:52:26 PM

  I am using www.scriptftp.com (http://www.scriptftp.com).  It has been fantastic and very easy to customize over my network.  Great customer service during setup too....

Thought I'd Share,
Greg


Thanks for the info
Title: Re: About to FLING FLING...
Post by: ConligWX on February 05, 2018, 05:27:51 PM
WinSCP also has command line utilities for ftp/ftps/sftp uploading/downloading and work very well.
Title: Re: About to FLING FLING...
Post by: carl357 on February 06, 2018, 06:04:52 AM
I used Fling for years...  I kept 2.21 and it was rock solid.  Recently I had to make a change...  I had too many simultaneous FTP sessions and my webhost was blacklisting my IP because their firewall was interpreting my activity as an attack....  So, I consolidated all of my ftp activity into a single session that I run every 2 minutes.  I am using www.scriptftp.com (http://www.scriptftp.com).  It has been fantastic and very easy to customize over my network.  Great customer service during setup too....

Thought I'd Share,
Greg

Thanks Greg. I am the guy behind ScriptFTP and have just realized of this conversation.

Yes, the plain old FTP is being deprecated by many providers such as 1and1, Amazon web services etc. From the point of view of the network administrator is a pain to manage, specially because it is not secure and has a strange way to handle ports to transfer files.

Hopefully SFTP is now very common and most FTP clients as ScriptFTP support it. This is a small script (for ScriptFTP) that synchronizes a remote folder from a folder in your computer (uploads new or modified files).

Code: [Select]
OPENHOST("sftp://ftp.myhost.com","myuser","mypassword")
SYNC("C:\my_folder","/my_remote_folder",UPLOAD,SUBDIRS)
CLOSEHOST

As you can see ScriptFTP is meant to be easy to understand and, unlike WinSCP which also supports scripting, it is built from ground up to support scripting

If any of you have any questions on how to configure ScriptFTP to work as Fling (or anything else) feel free to ask me anything on the website chat at ScriptFTP.com. I will also stay tuned on this forum thread  :grin: ;) ;)




Title: Re: About to FLING FLING...
Post by: zmarfak on December 20, 2018, 07:25:16 AM
Hi
I received an email on 4th december of NCH software with fling 3.00 link (beta), about memory leaks
As an answer to a message of me some times ago
Seems to me it's solved in this version.

I haven't checked on their website if this version is already available
 
Title: Re: About to FLING FLING...
Post by: ke9lz on December 20, 2018, 10:47:56 AM
I got the same email and did download it to find that my software key didn't work with the fixed version.  They require you to pay for a new key.  I decided to stay with Kens fix which has been working flawlessly.
Title: Re: About to FLING FLING...
Post by: saratogaWX on December 20, 2018, 11:47:00 AM
I'd done the $29 upgrade today from V2.35 (with the leaky nonpaged memory issue) to V3.00, and activated the new version.  Installed (after I did a backup of the old fling directory), and it is running fine.  It's not eating nonpaged memory any more on Win10-Pro !!

For insurance, I'm still leaving the zap-fling.pl script running via Scheduler at 15 minute intervals, just to be safe.

I think the major change of adding SFTP support is a reason to do the upgrade.. be sure to get the upgrade pricing ($29 for the pro version).
Title: Re: About to FLING FLING...
Post by: Cutty Sark Sailor on December 20, 2018, 08:12:45 PM
Yep, V3.0 cures the non-paged pool issue... smooth as silk...
as Ken says,
I'd done the $29 upgrade today from V2.35 (with the leaky nonpaged memory issue) to V3.00, and activated the new version.  Installed (after I did a backup of the old fling directory), and it is running fine.  It's not eating nonpaged memory any more on Win10-Pro !!

For insurance, I'm still leaving the zap-fling.pl script running via Scheduler at 15 minute intervals, just to be safe.

I think the major change of adding SFTP support is a reason to do the upgrade.. be sure to get the upgrade pricing ($29 for the pro version).
Especially if you ever think of SFTP...do the upgrade...  Good to Go
The new version also cured a small  glitch reading one folder that was a tad slower writing at times, for some reason,... no more error on that one... was no biggie, it'd catch it on the next scan, .. just irritating to see the 'error at times... Tip:   export your 'Folder List' just in case... so you can import it if need b
also a good idea to export it and save it somewhere.
Title: Re: About to FLING FLING...
Post by: ke9lz on December 21, 2018, 12:59:11 PM
Quote
I think the major change of adding SFTP support is a reason to do the upgrade..

Guess I didn't realize that this was also changed with the addition to the memory fix.  For that I agree it is worth the upgrade.  Now that you guys had a few days of use with it is the memory leak for sure fixed?
Title: Re: About to FLING FLING...
Post by: saratogaWX on December 21, 2018, 01:27:44 PM
Yep.. still fixed.  Nonpaged memory about 49KB instead of growing 4KB each second until I zap it at 16MB.

Yes.. do the upgrade!