6 Replies Latest reply on Mar 9, 2018 12:43 PM by Jay Xu

    Control-m File watcher thinks a file is completed while it is still being downloaded

    Jay Xu

      I have file watcher job.

      1. there is a wildcard * in the file name because there is random time stamp in the file name. The file size watch were defaulted to check every 10 second for 3 times.

      2. the wildcard support is set to yes

      3. The file is about 1G. It is download from a remote ftp, takes about 20 minutes to finish. But 30 seconds is more than enough to detect the file increase!

       

      The problem I have now is that the file watcher job thinks the file is completed while the file is still being downloaded.

      I checked the logging from the file downloading job, 30 seconds is more than enough to notice the file increase!

       

      I am really confused by this behavior. Have I set up something wrong? Please help. Thank you very much!!

        • 1. Re: Control-m File watcher thinks a file is completed while it is still being downloaded
          James Perrone

          The wildcard causes the size interval to be ignored:

          File size interval

          Interval between attempts to monitor the size of a file after it is detected (in seconds). This parameter is ignored when using wildcards in FILE or when using DELETE mode.

           

          Mode

          Determines whether to detect creation or deletion of a file as follows:

          • Create: Detects creation of a file. File size is ignored if the filename parameter contains wildcards (unless the monitor file size when wildcard is used parameter is set to Y).
          • Delete: Detects deletion of a file. When the ctmfw utility is run in this mode, it first checks for files that match the specified name. After a specified file is detected, the ctmfw utility checks at the specified interval for deletion of that file.
          • 3. Re: Control-m File watcher thinks a file is completed while it is still being downloaded
            Jay Xu

            Thanks, James.

            There are some other strange behaviors besides this one. I think they are bugs. BMC should probably take a look at file watcher.

            • 4. Re: Control-m File watcher thinks a file is completed while it is still being downloaded
              Jay Xu

              In case somebody had the same issue, I get around this by setting up a minimal age. File Watcher fails to notice the file size increase, but manages to detect the time stamp changes. Setting minimal age to a couple minutes solves the problem.

              • 5. Re: Control-m File watcher thinks a file is completed while it is still being downloaded
                MunKeong Lee

                Hi Jay

                Could it be possible that for 30 seconds, the file transfer was interrupted and then resumed subsequently? If so, this will cause ctmfw to accidentally detect that the file is  ready. ctmfw has been around for a very long time and it's not a complicated utility. Although it's possible, it's not likely that there's a bug in ctmfw which has not been noticed by BMC.

                Regards,

                MK

                • 6. Re: Control-m File watcher thinks a file is completed while it is still being downloaded
                  Jay Xu

                  MunKeong,

                  That was the first thing I thought about. Good thing the file is downloaded daily by a control-m ftp job set up by another team. So I checked the log/output right after I noticed the problem. The log/output showed significant file size increase every 4-5 seconds. Here is part of the ftp job log.

                   

                    Transferred: 493920256     Elapsed: 566sec   Percent: 138  status: In Progress

                    Transferred: 497500160     Elapsed: 569sec   Percent: 139  status: In Progress

                    Transferred: 501080064     Elapsed: 573sec   Percent: 140  status: In Progress

                    Transferred: 504659968     Elapsed: 577sec   Percent: 141  status: In Progress

                    Transferred: 508239872     Elapsed: 580sec   Percent: 142  status: In Progress

                    Transferred: 511815680     Elapsed: 583sec   Percent: 143  status: In Progress

                    Transferred: 515395584     Elapsed: 587sec   Percent: 144  status: In Progress

                    Transferred: 518975488     Elapsed: 590sec   Percent: 145  status: In Progress

                    Transferred: 522555392     Elapsed: 594sec   Percent: 146  status: In Progress

                    Transferred: 526135296     Elapsed: 597sec   Percent: 147  status: In Progress

                  .

                  .

                  .

                   

                    Transferred: 962785280     Elapsed:1155sec   Percent: 269  status: In Progress

                    Transferred: 966365184     Elapsed:1160sec   Percent: 270  status: In Progress

                    Transferred: 969945088     Elapsed:1164sec   Percent: 271  status: In Progress

                    Transferred: 973524992     Elapsed:1168sec   Percent: 272  status: In Progress

                    Transferred: 977104896     Elapsed:1171sec   Percent: 273  status: In Progress

                    Transferred: 980680704     Elapsed:1175sec   Percent: 274  status: In Progress

                    Transferred: 984260608     Elapsed:1178sec   Percent: 275  status: In Progress

                   

                   

                  Checking every 10 seconds for 3 times is more than enough to notice the file increase. I would have to think that wildcard and large file might not be so common? I would have not noticed the problem, if I were creating the ftp job myself, because I would have just built a dependency, instead of using a file watcher. The ftp job belongs to another team, and I did not want to build the dependency.