1 2 Previous Next 16 Replies Latest reply: Aug 21, 2008 2:38 AM by huang hao RSS

    PemnBPutFile problem

    Geert De Peuter

      Original email:

       

      Hi,
      It seems there is a problem with PemnBPutFile. There seems to be a limitation of 8k and it doesn't seem to work for binary files.

       

      Any workarounds available ?

       

      Reply:

       

      I have to warn you when sending large files !! (and you should warn your customers as well, if you will expose this feature)

       

      PemnBPutFile will send files using the PEMAPI.  That means that the file fragments will be handled by the same mainloop as events or other things in PATROL. So you should prevent sending big files over the line or you should understand the consequence ! As a result you may see the agent's CPU go high, or the agent might even disconnect !

       

      Anyway, that's not an excuse... so I produced this workaround.

       

      Cheers,
      -- Geert

       

      #include <stdio.h>
      #include <string.h>
      #include <pemapi.h>
      #include <malloc.h>
      #include <ctype.h>
      #include <stdlib.h>
      #include <sys/types.h>
      #include <sys/stat.h>

       

      #define STREAM_MAX    8096

       

      int g_timeout=2000;

       

      char *
      FileSend(PemnCommHandle commhandle,char *localfilename, char *remotefilename)
      {
          FILE*     fp;
          struct stat buf;
          int iStreamSize;
          char *pcStreamData;
          int i,n;
          int TransferOK=1;
          char *ret;
               
          if ( (fp= fopen(localfilename, "rb")) == NULL )
              return ("ERROR: cannot open source file\n");
       
          if (fstat (fileno(fp), &buf) != 0)
          {
              fclose(fp);
              return("ERROR : cannot obtain file stat on sourcefile\n");
          }
       
          iStreamSize=(int) buf.st_size;
       
          if ( iStreamSize < STREAM_MAX){
              pcStreamData= (char*)malloc(iStreamSize);
          } else {
              pcStreamData= (char*)malloc(STREAM_MAX);
          }
               
          n =  (iStreamSize/STREAM_MAX) +1;
               
          i=1;
          while ((i <= n) && TransferOK) {
              size_t iSize = fread(pcStreamData, 1, STREAM_MAX, fp);
              ret=PemnBPutStream(
                  commhandle,
                  iSize,
                  pcStreamData,
                  NULL,
                  NULL,
                  0,
                  remotefilename,
                  (i==1)?PEMN_WRITE:PEMN_APPEND,
                  "",
                  "511",    /*permissions (decimal !!)*/
                  g_timeout
                  );
                   
              TransferOK=(strcmp(ret,"OK") == 0) ? 1 : 0;
           
              i++;
          }
          free(pcStreamData);
          fclose (fp);

       

          if (TransferOK)
              return "OK";
          else
              return "ERROR : Error in filetransfer\n";
      }

        • 1. Re:
          Patrick NameToUpdate

          Hi Geert,  Thank you for producing this code.

          One question though.

          Should the line:   TransferOK=(strcmp(ret,"OK") == 0) ? 1 : 0;

          not read:  TransferOK=(strcmp(ret,PEMN_OK) == 0) ? 1 : 0; 

           

          Otherwise the strcmp seems to always fail, as ret never returns the string OK.  I think if it is evaluated this way, what is actually returned is:  OK  Am I understanding this correctly?

          • 2.
            Geert De Peuter

            Patrick,

            What you say sounds right. However, it used to work.
            Thanks for making the suggestions to make it work on current releases

            -- Geert

            • 3.
              Sandra Van Maren

              I came across those limitations with PemnPutFile too. I ended up using remote_file_send(). It required a Patrol agent on the source end, but the remote_file_send() appears to not have that tight file size limitation (I sent some pretty huge files) and I had success sending binaries too.

              • 4. RE: PemnBPutFile problem
                Peter Harrison

                Hi

                Thanks for the code snippet Gert. the file size would have caught me out later on.

                The code you sent works fine when I'm sending a text file from windows to windows but I get a ^M character at the end of each line when sending files to unix boxes.

                Is there a solution to this?

                Regards
                Pete

                • 5. RE: PemnBPutFile problem
                  Vladimir Tsichevski

                  "... when I'm sending a text file from windows to windows but I get a ^M character at the end of each line when sending files to unix boxes.
                  "

                  Files do not change while they are being transferred. You never see any ^M characters on Windows because  on Windows (aka MS-DOS, aka CP/M) that characters are part of the "end-of-line" sequence, and no windows text tool ever displays them.

                  • 6. Re: RE: PemnBPutFile problem
                    Geert De Peuter

                    Probably the easiest solution for you is to replace

                     

                    if ( (fp= fopen(localfilename, "rb")) == NULL )

                     

                    with

                     

                    if ( (fp= fopen(localfilename, "rt")) == NULL )

                     

                    Of course this will only work if you are transferring text files. Like mentioned before, with a "b" the file will not change (but probably you want it to change). See manpages on fopen() for more information.

                     

                    Hope this helps,

                    Geert

                    • 7. RE: PemnBPutFile problem
                      Peter Harrison

                      It all makes sense now!

                      Looks like I have a few options. Either send the file as text and lose the ability to send binary files (something I may need to do in the future). Try to find out if the source file is text or binary before opening with the apropriate parameter or using the "dos2unix" tool on the unix boxes.

                      Thanks both for the information and suggestions.

                      Pete

                      • 8. RE: PemnBPutFile problem
                        Peter Harrison

                        I've gone for the option of finding out if the file to send is binary or text. 

                        The only way I could find out the file type in windows is by reading through the file and looking for null strings.

                        Anyway here's my code for information and comment.

                        // Test to see if the file is binary or text #######
                        if ( (fp= fopen(localfilename, "rb")) == NULL ){
                        LogError("FileSend: cannot open source file","ERROR");
                        return (false);
                        }

                        fseek(fp,0,SEEK_END);


                        iStreamSize=(int) ftell(fp);
                        rewind(fp);


                            pcStreamData= (char)malloc(iStreamSize); <br />   <br />     fread(pcStreamData, 1, iStreamSize, fp);<br /><br />     for (i=0;i&lt;=iStreamSize;i+){<br />          if (pcStreamData==0){<br />               AmountOfNulls++;<br />          }<br />     }<br /><br />     free(pcStreamData);<br />   fclose (fp); <br />   if ( ((float)((float)AmountOfNulls/(float)iStreamSize))100 &gt;2.5 ) {
                        strcpy(OpenType,"rb");
                        }else{
                        strcpy(OpenType,"rt");
                        }
                           // ##################################################

                        • 9. RE: PemnBPutFile problem
                          Peter Harrison

                          One last question on this (I promise).

                          I'm trying to create a function to retrieve a file in the same way as this "FileSend" function does.

                          Do you have any examples of this as I'm having some difficulty creating the function.

                          Regards
                          Pete

                          • 10. RE: PemnBPutFile problem
                            Lynne Lawrence

                            Does anyone know if the 8K limit on PemnBPutFile still exists in the newer Patrol agents (v3.6)?  The original post on this topic is old and I thought I had read somewhere that this was no longer an issue.

                            Anyone know?

                            Thanks,

                            Lynne Lawrence

                            • 11. RE: PemnBPutFile problem
                              Vladimir Tsichevski

                              "Does anyone know if the 8K limit on PemnBPutFile still exists in the newer Patrol agents (v3.6)?  The original post on this topic is old and I thought I had read somewhere that this was no longer an issue.

                              Anyone know?

                              Thanks,

                              Lynne Lawrence"

                              No the maximum size of data which can be transferred in one packet is limited by PEMAPI, which uses statically pre-allocated buffers. Agents have no size limits AFAIK.

                              • 12. RE: PemnBPutFile problem
                                Peter Harrison

                                Hi, (been on holiday and just seen this reply).

                                Are you saying that I no longer need the function kindly shared by Geert?

                                That I can now use PemnBPutFile instead of PemnBPutStream and the supporting code?

                                Pete

                                • 13. RE: PemnBPutFile problem
                                  Vladimir Tsichevski

                                  "Hi, (been on holiday and just seen this reply).

                                  Are you saying that I no longer need the function kindly shared by Geert?

                                  That I can now use PemnBPutFile instead of PemnBPutStream and the supporting code?

                                  Pete"

                                  No, I am saying that the problem is on the client, not on the agent side.

                                  PEMAPI uses statically allocated fixed size buffers for all functions, so no matter which PEMAPI function will you use (PemnBPutFile or PemnBPutStream), the 8K limit will remain.

                                  • 14. RE: PemnBPutFile problem
                                    Peter Harrison

                                    Hi,

                                    I still can't get my head around trying to bring a file back using PemnBGetStream and iterating round until I have it all. I've tried to turn Geerts code round to do this but I can't figure out all the parameters that PemnBGetStream requires and it seems to be missing in the documentation.

                                    Any help or a code snippet to give me a leg up would be great.

                                    Pete

                                    1 2 Previous Next