8 Replies Latest reply on Jul 8, 2009 3:07 PM by S Crawford

    How are snapshot files cleaned up on the app servers?

    S Crawford

      We are running snapshot jobs against inventory components on all our servers (for BSARA). Having over 3000 servers means that a lot of snapshot files (.snp) are being created on our app servers. They are being stored in /usr/nsh/br/bnp, and I'm guessing they are then being copied over to the File Server in the bnp directory. When are they actually copied over?

       

      Are the snapshot files supposed to be removed from the app server file system once they are copied over? Or is there some cleanup job that is supposed to be run to clean these out? If they aren't cleaned out, we won't be able to run more than one snapshot job because the .snp files will fill the app server file system after one run.

        • 1. Re: How are snapshot files cleaned up on the app servers?
          Bill Robinson

          The snapshot data gets stored in 3 places:

          /usr/nsh/br/bnp - 'temp/immediate retrieval'

          file server - /usr/nsh/storage/bnp - 'near term storage - medium retrieval'

          database - 'long term/slow retrieval'

           

          pretty much everytime you run a snapshot or audit we're going to populate the /usr/nsh/br/bnp directory.

           

          I don't think this is a great place to store things - /usr isn't usually a huge file system. you can symlink it to somewhere that has more storage.

           

          I'm not sure when in the process they are copied over to the file server.

           

          if you run the blcli 'Delete cleanupAppServerCache' that should clean out this directory. but it will get repopulated on the next snapshot/audit.

           

          if we can't find the snapshot/audit data in the /usr/nsh/br/bnp dir, I think we try and pull it from the file server, then the db.

           

          that's how it's been explained to me. I'd rather we just store all that info on the file server.

          • 2. Re: How are snapshot files cleaned up on the app servers?
            S Crawford

            I agree, they should be stored on the file server. The large number of snapshots we need to take will in effect turn the app servers into additional file servers. But that's good that the cleanupAppServerCache job will clean that directory out. Since we don't know when the snapshots are copied, I'm thinking it will be best to create a batch job that will first run the cleanup job, followed by the snapshot job so there's never more than one set of snapshots in that directory.

             

            For the inventory snapshots, I tried to simplify things by only having one snapshot job that checks for all six inventory templates (AIX, ESX, HP-UX, Linux, Solaris, Windows) that runs on all our servers. This way we would only need one snapshot job. However, having this run over 3,000 servers is taking a very long time, I'm guessing because it has to check through all six templates on each server, even though only one will be used per server. It wouldn't even run until I set limited the # of targets to run in parallel (to 10). But even now, it is running very slow (already been running for 9 hours, and only about 20% completed). Should I be splitting these snapshot jobs up so that only one component is checked per job, and have each job run against the correct servers (by using OS smart group)?

             

            Thanks,

            Scott

            • 3. Re: How are snapshot files cleaned up on the app servers?
              Bill Robinson

              You could run that cleanup as a separate job daily, you can only clean out snapshots 3 days or more old.

               

              As for the snapshot job, you might want to split it up into the separate job per template, though runnings across 3000 servers at a time is going to take a while anyway. Another thing to look at there are problems w/ any of the data being snapshot - like a slow running EO.

               

              I like to run the snapshot (and audit and compliance for that matter) jobs against the components, not servers, though I'm not sure if it goes any faster.

              • 4. Re: How are snapshot files cleaned up on the app servers?
                S Crawford

                So I broke that single snapshot job into separate ones like we mentioned. I broke them out by OS, so each job now only looks for the single OS inventory template (instead of all of them) and the target is the server OS smart group. I would've targeted the components instead, but I had to select the specific templates I wanted to run against. So if there are new servers that are enrolled and the components are added onto those servers, I would need to add those components to the targe list manually, right? I was hoping I could just select the inventory component template (which had all the current components listed under it) but it doesn't let me. I really need a smart group for 'components' I guess.

                 

                I added these new jobs into one batch job that will run them in sequence, and so far they are running successfully. It looks like it will take around 4-6 hrs to complete all four jobs.

                • 5. Re: How are snapshot files cleaned up on the app servers?
                  Bill Robinson

                  For the Components - you can create Component smart groups w/ the condtions:

                   

                  TEMPLATE equals /Path/To/Template

                  TARGET.AGENT_STATUS = agent is alive

                   

                  which will give you all of the Components for that template where the server is online. Then you can target that SmartGroup in your Jobs. When you run a Discovery Job, newly discovered components on new servers will show up in the Component Smart Groups.

                   

                  You could try, in your batch job, to run all the jobs in parallel. That could be faster. If we need to having things run faster we'd need to look at your infrastructure and various config settings on the appservers.

                  • 6. Re: How are snapshot files cleaned up on the app servers?
                    S Crawford

                    Good idea! I forgot that component smart groups were available. I setup the smart groups like you specified and am now using them as the targets in the snapshot jobs.

                     

                    Everytime the snapshot job runs, it will overwrite the existing component, correct? I just want to make sure because our database needed to be expanded after the snapshot job was run. But if future snapshot jobs overwrite the data, we won't have to worry about the database growing out of control.

                    • 7. Re: How are snapshot files cleaned up on the app servers?
                      Bill Robinson

                      The snapshots save only the deltas from the last snapshot run - so the first snapshot could be huge based on the content of the snapshot, but the next snapshot will only be what the delta is between the first snapshot and the current data.

                       

                      So it doesn't really overwrite, it kind of appends.

                      • 8. Re: How are snapshot files cleaned up on the app servers?
                        S Crawford

                        Bill, earlier you mentioned the snapshots were stored in three places:

                         

                        /usr/nsh/br/bnp - 'temp/immediate retrieval'

                        file server - /usr/nsh/storage/bnp - 'near term storage - medium retrieval'

                        database - 'long term/slow retrieval'

                         

                        I also found that they are stored in another place on the file server:

                        /usr/nsh/storage/tmp/application_server/2003002/usr/nsh/br/bnp

                         

                        There are also a couple other folders named 2000002, 2000003, etc that are setup the same way as 2003002.

                         

                        Is this another temporary storage area for the snapshots? Where in the chain does this fit? Is it safe to clean out /usr/nsh/storage/tmp after a while if the cleanup job doesn't do it?