11 Replies Latest reply on Jan 16, 2008 5:55 PM by John van Ommen

    Why Would a Provisioning Job go Backwards?

      When you are provisioning a server, what would cause a job to go BACKWARDS?

       

      When provisioning servers via Gentoo, my provisioning job goes from step 3, then back to step 1 when BMI kicks off.

       

      What would cause this?

       

      Here's the log:

      + Anonymous:Anonymous:10.192.139.17 STATE received from remote machine:

      0

      Anonymous:Anonymous:10.192.139.17 received a set state to discovered from device: 00-0C-29-E7-C2-C3. This should not be happening. Will process it, but get next state will return disk cleanup instead of biosinfo in this instance

      Anonymous:Anonymous:10.192.139.17 Connection disconnecting+

       

      I watched the provisioning job in the console, and you can see the the job goes from step three to step one at the moment that BMI is kicked off.

       

      We're provisioning Linux via Gentoo using PM 7.4.1, with all the latest hotfixes, on Windows Server 2003.

        • 1. Re: Why Would a Provisioning Job go Backwards?





          Now I'm seeing another error, probably tied to the first.



          Here's what happened:



          - The managed server was powered down.

          - The device was deleted from Provisioning Manager.

          - Since DOS provisioning was working, I tried to add the device back into PM.

          - When I tried to do this, PM complained that the device already exists - BUT the device isn't visible in the "Imported" list or in "Provisioning Progress."





           

           

          • 2. Re: Why Would a Provisioning Job go Backwards?
            Bill Robinson

            look in the PM_DEVICE table, find the device and set the is_deleted column to 1

            • 3. Re: Why Would a Provisioning Job go Backwards?

              Yes, the following SQL query will do that:

               

              update pm_device set is_deleted = 1 where mac_address = '00-0C-29-E7-C2-C3'

               

              But that doesn't address the root problem -

               

              Why is the provisioning process going from step 3 to step 1 once BMI is called?

               

              BTW, my database is SQL Server, app server is Windows 2003.

              • 4. Re: Why Would a Provisioning Job go Backwards?
                Bill Robinson

                it's almost like it's confused about the device, or the device already exists in another state?

                 

                does it do this after you 'delete' the device from the pm_device table?

                 

                maybe run the blcli cleanup and try again ?

                • 5. Re: Why Would a Provisioning Job go Backwards?

                  It doesn't matter if I delete the device outright from the PM_DEVICE table outright, or set the "is_deleted" field to 1 - the same thing happens:

                   

                  The server is "discovered" again, but silently. So it doesn't appear in the PM gui, but it's in the database table. The only way I've found to prevent this is to POWER off the server, then MANUALLY add it to PM before powering it on again.

                   

                  If you don't follow that process, as soon as the server reboots it will begin provisioning as the default image. So for instance, if you're provisioning Linux, and your default image is WinPE, when you restart the server it will begin provisioning as WinPE. Even though the server is provisioning, it won't appear in the PM gui.

                  • 6. Re: Why Would a Provisioning Job go Backwards?
                    Bill Robinson

                    weird. is there another mac it could be registering as?

                     

                    when you say it starts provisioning - you mean it starts installing the OS, as if a provision job has already been run against it ?

                     

                    hmm... can you look in the job table, maybe there's a job floating around saying to provision that system? search for the provision jobs - type id 5029

                     

                    also, check in the /tftproot directories for hostid files that would contain something staged for that server's mac maybe...

                    • 7. Re: Why Would a Provisioning Job go Backwards?

                      weird. is there another mac it could be registering

                      as?

                       

                      +I checked the table - it's the only device in the table with that MAC

                       

                       

                      when you say it starts provisioning - you mean it

                      starts installing the OS, as if a provision job has

                      already been run against it ?

                       

                       

                      Yes! Isn't that crazy? A rather dangerous situation I'd say.

                       

                       

                      hmm... can you look in the job table, maybe there's

                      a job floating around saying to provision that

                      system? search for the provision jobs - type id

                      5029

                       

                      Hmmm I'm going to get a parking ticket if I don't move my car soon :)

                       

                      also, check in the /tftproot directories for hostid

                      files that would contain something staged for that

                      server's mac maybe...

                       

                      Yes - that directory is just filling up. Isn't something supposed to delete those files? If I'm not mistaken, it's these entries that are triggering the provisioning. I have files in that directory that date back months.

                      • 8. Re: Why Would a Provisioning Job go Backwards?
                        Bill Robinson

                        the files in that dir will either tell the box to local boot if it's already 'provisioned' or to boot off the boot image and start/continue the provisioning process

                        • 9. Re: Why Would a Provisioning Job go Backwards?

                          A couple more notes -

                           

                          When the server is "invisible", it's pm_state_id in the DB is "1".

                          When you set the "is_deleted" flag to 1, then add it in the GUI, it's pm_state_id changes to -2.

                          When you start a job with the device, pm_state_id goes up to 2. In the GUI it shows that progress is at step 3 of 16.

                          • 10. Re: Why Would a Provisioning Job go Backwards?

                            Made a bit of a breakthrough. I found that when I ran this bit of SQL against the database, the provisioning job would continue to run:

                             

                            update pm_device set pm_state_id = 3 where mac_address = '00-0C-29-E7-C2-C3'

                             

                            Which begs the question - why is the pm_state_id getting set incorrectly when BMI is kicked off?

                            • 11. Re: Why Would a Provisioning Job go Backwards?

                              I wanted to dump some notes here on what's going on in the database. This is mostly for the sake of investigation Gentoo's behavior.

                               

                              In a server provisioned with WinPE the last_known_name is 2k3_ent_x64

                              In a server provisioned with Gentoo the last_known_name is 00-0C-29-E7-C2-C3

                              With Gentoo, why is the former name a MAC?

                               

                              In a server provisioned with WinPE the no_of_cpu is 2

                              In a server provisioned with Gentoo the no_of_cpu is 0

                              Why does Gentoo think it has zero CPUs?