1 2 Previous Next 18 Replies Latest reply on Feb 26, 2015 6:39 AM by Linda K. Whitsitt

    Windows Patch Catalog update fails after upgrade to 8.6

    Charlie Sullivan

      Our Windows patch catalog update job runs every Wednesday and had been completing successfully for months. We upgraded the blapp server to 8.6 on Tuesday and now the job fails. Here are a couple of snippets from the log:

      ***********************************************************************************************************************************************************

      Run at 02/05/2015 09:58:07,Info,02/05/2015 09:58:29,Using patch signature file at location http://xml.shavlik.com/data/oem/BMC-Bladelogic/data/8.0/hf7b.cab

      Run at 02/05/2015 09:58:07,Info,02/05/2015 09:58:29,Using package information file at location http://xml.shavlik.com/data/oem/BMC-Bladelogic/data/8.0/pd5.cab

      Run at 02/05/2015 09:58:07,Info,02/05/2015 09:58:29,Using Oem file at location http://xml.shavlik.com/data/OemCatalog.zip

      Run at 02/05/2015 09:58:07,Info,02/05/2015 09:58:29,Initializing

      Run at 02/05/2015 09:58:07,Info,02/05/2015 09:58:50,Catalog update progress is 0% completed.

      Run at 02/05/2015 09:58:07,Info,02/05/2015 09:58:50,Catalog update progress is 0% completed. Downloading metadata : 1/4 completed.

      Run at 02/05/2015 09:58:07,Info,02/05/2015 09:58:50,Downloading Content metadata from Shavlik

      Run at 02/05/2015 09:58:07,Info,02/05/2015 09:58:59,Catalog update progress is 0% completed. Downloading metadata : 2/4 completed.

      Run at 02/05/2015 09:58:07,Info,02/05/2015 09:59:11,Catalog update progress is 1% completed. Downloading metadata : 3/4 completed.

      Run at 02/05/2015 09:58:07,Info,02/05/2015 09:59:21,Catalog update progress is 1% completed. Downloading metadata : 4/4 completed.

      Run at 02/05/2015 09:58:07,Info,02/05/2015 09:59:33,Building metadata

      ..........................

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Reading the bulletin metadata...

      Run at 02/05/2015 09:58:07,Error,02/05/2015 10:00:48,", Error: null"

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Patches newly added: 0

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Patches updated: 0

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Patches obsoleted: 0

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Patches failed to be process: 0

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Patches downloaded: 0

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Patches failed to be downloaded: 0

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Patches with invalid/deprecated urls: 0

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Referrers newly added: 0

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Referrers updated: 0

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Referrers obsoleted: 0

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Referrers failed to be process: 0

      Run at 02/05/2015 09:58:07,Warning,02/05/2015 10:00:48,"Catalog state is unchanged, Analysis can be performed using this catalog only when Use last successful catalog run is set to Yes under All Operating Systems tab in patch global configurations."

      Run at 02/05/2015 09:58:07,Warning,02/05/2015 10:00:48,Analysis will be performed on the basis of last successful catalog state and its parameters.

      Run at 02/05/2015 09:58:07,Info,02/05/2015 10:00:48,Catalog update progress is completed.

      Run at 02/05/2015 09:58:07,Error,02/05/2015 10:00:48,The job 'WindowsPatchCatalog' has failed

      ************************************************************************************************************************

      (I understand that this is new to version 8.6: "Oem file at location http://xml.shavlik.com/data/OemCatalog.zip", and I think it's needed.)

       

      The job seems to fail at "Reading the bulletin metadata...", but "Error: null" is not very helpful in explaining why.

       

      Is anyone else experiencing this problem?

        • 1. Re: Windows Patch Catalog update fails after upgrade to 8.6
          Bill Robinson

          Do you have ‘windows media player’ in the filter list?  if so, try taking it out.

          • 2. Re: Windows Patch Catalog update fails after upgrade to 8.6
            Charlie Sullivan

            Yes, it appears to be working now that I've removed Media Player. So, it sounds like this is a known issue that will be addressed?

             

            Thanks.

            • 3. Re: Windows Patch Catalog update fails after upgrade to 8.6
              Bill Robinson

              Yep – you might still want to open a ticket so you can track the defect.

              • 4. Re: Windows Patch Catalog update fails after upgrade to 8.6
                Charlie Sullivan

                Thanks.  The job still has some failures, but for much different reasons.  I'm getting a lot of this:

                 

                , Error: lazy loading error : change signature  (Caused By: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object (Caused By: Cannot get a connection, pool error Timeout waiting for idle object (Caused By: Timeout waiting for idle object)))

                 

                , Error: error on lazy load :  (Caused By: An error occurred while attempting to access the database:

                Message : Cannot get a connection, pool error Timeout waiting for idle object SQLState: null ErrorCode: 0

                (Caused By: Cannot get a connection, pool error Timeout waiting for idle object (Caused By: Timeout waiting for idle object)))

                 

                ....and this:

                 

                Error while add/update patch in depot: hotfix : WindowsXP-KB954600-x86-ENU.exe-MS08-076-en-WINDOWS XP HOME EDITION-SP2, , Error: An error occurred while attempting to access the database:

                Message : Cannot get a connection, pool error Timeout waiting for idle object SQLState: null ErrorCode: 0

                (Caused By: Cannot get a connection, pool error Timeout waiting for idle object (Caused By: Timeout waiting for idle object))

                 

                Error while add/update patch in depot: hotfix : Windows6.0-2008-KB2536275-x86.msu-MS11-048-en-WINDOWS SERVER 2008, STANDARD EDITION-SP1, , Error: An error occurred while attempting to access the database:

                Message : Cannot get a connection, pool error Timeout waiting for idle object SQLState: null ErrorCode: 0

                (Caused By: Cannot get a connection, pool error Timeout waiting for idle object (Caused By: Timeout waiting for idle object))

                 

                Is this something we need to pursue internally, or does it suggest a problem on the Shavlik/BMC side?

                Thanks.

                • 5. Re: Windows Patch Catalog update fails after upgrade to 8.6
                  Charlie Sullivan

                  So today we tried to run a patch job and we are now in a worse state then when the entire catalog update failed. The last catalog update was considered successful, even with the errors mentioned above, but it's in a state that is not usable as evidenced by this error:

                  Failed to load last successful catalog update job: , Error: Last successful catalog update may exist but it is not usable. The metadata might have been corrupted by some catalog job run..


                  Is anyone else at version 8.6 and using a Windows Patch Catalog?

                   

                  We have a general Windows patching case open with BMC and I'll see if we can get help from the technician assigned to that. If not, I'll open a separate case. However, it would be helpful to know if others are experiencing this problem.

                   

                  Thanks.

                  • 6. Re: Windows Patch Catalog update fails after upgrade to 8.6
                    Bill Robinson

                    i'd open a ticket w/ support if you haven't yet.?

                    • 7. Re: Windows Patch Catalog update fails after upgrade to 8.6
                      Charlie Sullivan

                      Thanks, I opened a ticket yesterday.

                      • 8. Re: Windows Patch Catalog update fails after upgrade to 8.6
                        Patrick O'Callaghan

                        We are having an issue as well after the 8.6 Upgrade, however it is a little different.
                        When running the catalog update job we get

                        Error    02/24/2015 10:09:43     Precheck Failed : url [https://xml.shavlik.com/data/hf7b.xml] not pingable, Please check url and proxy details

                         

                        Checked the appserver log and it shows

                        24 Feb 2015 10:09:43,219] [PSU-Thread-1] [ERROR] [patrick:BLAdmins:] [Patch-Metadata-Updator] com.bmc.sa.patchfeed.FeedException:  Precheck Failed : url [https://xml.shavlik.com/data/hf7b.xml] not pingable, Please check url and proxy details

                        [24 Feb 2015 10:09:43,220] [WorkItem-Thread-5] [ERROR] [patrick:BLAdmins:] [Patch-Metadata-Updator]  Precheck Failed : url [https://xml.shavlik.com/data/hf7b.xml] not pingable, Please check url and proxy details

                         

                        Ran in Debug mode and the only thing I'm seeing is

                        MESSAGE:  Precheck Failed : url [http://xml.shavlik.com/data/hf7b.xml] not pingable, Please check url and proxy details  (String)

                         

                        and

                         

                        [patrick:BLAdmins:] [Patch-Metadata-Updator] [] Using patch signature file at location http://xml.shavlik.com/data/hf7b.xml

                        [23 Feb 2015 12:52:11,541] [WorkItem-Thread-0] [INFO] [patrick:BLAdmins:] [Patch-Metadata-Updator] [] Using package information file at location http://xml.shavlik.com/data/pd5.xml

                        [23 Feb 2015 12:52:11,541] [WorkItem-Thread-0] [INFO] [patrick:BLAdmins:] [Patch-Metadata-Updator] [] Using Oem file at location http://xml.shavlik.com/data/OemCatalog.zip

                        [23 Feb 2015 12:52:11,545] [WorkItem-Thread-0] [INFO] [patrick:BLAdmins:] [Patch-Metadata-Updator] [] Initializing

                        [23 Feb 2015 12:52:32,706] [WorkItem-Thread-0] [ERROR] [patrick:BLAdmins:] [Patch-Metadata-Updator] []  Precheck Failed : url [http://xml.shavlik.com/data/hf7b.xml] not pingable, Please check url and proxy details

                         

                        and

                         

                        [23 Feb 2015 12:52:32,738] [Job-Execution-0] [DEBUG] [patrick:BLAdmins:] [CatalogUpdate] [] Requesting proxy for BLFileServer:4750

                        [23 Feb 2015 12:52:32,738] [Job-Execution-0] [DEBUG] [patrick:BLAdmins:] [CatalogUpdate] [] Couldn't find proxy server for servername= BLFileServer

                        [23 Feb 2015 12:52:32,744] [Job-Execution-0] [DEBUG] [patrick:BLAdmins:] [CatalogUpdate] [] [JNI] Connecting to host: BLfileServer

                         

                        There is a proxy server configured in Patch Global Configuration and the proxy was also configured on all App Servers using blasadmin -a set appserver httpproxy(name,port,user,type,password) and blasadmin -a set appserver ntlmproxydomain (as it is an NTLM proxy server).

                         

                        The pd5.xml, hf7b.xml, and oemcatalog.zip are all able to be downloaded manually from the Shavlik URL Configuration under Patch Global Configuration.  Ran Wireshark and saw that the App server was communicating to the proxy server while it was manually downloading the files.

                        Attempted to use many different URL's and tried using three different proxy servers and making sure the configurations were set across the board.

                         

                        The weird thing is when I run the patch catalog update, there are no packets getting sent back and forth between the proxy as it does when manually downloading the xml's.  There is no communication at all between app server and proxy server during the patch catalog update.

                         

                        Also, we have a ticket open for this issue, ISS04429178.

                        • 9. Re: Windows Patch Catalog update fails after upgrade to 8.6
                          Bill Robinson

                          Can you try this:

                           

                          Manually download the metadata files and put them on a path reachable by nsh.

                          Edit the PGC and change the url locations of the files to the nsh location

                          Run the cuj

                          • 10. Re: Windows Patch Catalog update fails after upgrade to 8.6
                            Charlie Sullivan

                            This may not be the same issue, but I ran into something quite similar with Shavlik URLs a couple of months ago. If you do lookups on xml.shavlik.com from different sources, you'll see that it's an apparent DNS round robin. I just tried three different machines and got three different IP addresses (they use several).

                             

                            So our BLAPP server got name resolution pointing to an IP address that was down, and it seemed to be that way for weeks, as far as I could tell. We had to use the IP address of one of the ones that was up.

                             

                            So you may want to first find out what IP address the name resolves to from the BLAPP server, then ping that IP address from the BLAPP server, if you haven't already done this.

                            • 11. Re: Windows Patch Catalog update fails after upgrade to 8.6
                              Patrick O'Callaghan

                              Will try your suggestion but it cannot be kept this was moving forward.

                              • 12. Re: Windows Patch Catalog update fails after upgrade to 8.6
                                Patrick O'Callaghan

                                The java security cache is set to the default of indefinite caching but that should reset when the appserver service gets restarted which has been done multiple times so it shouldn't be just pointing to the same ip address everytime.  Unfortunately due to security measures in place the client does not allow for ICMP to go out to the internet, but when discussing with BMC support the Shavlik not pingable error does not actually mean that it is trying to ping shavlik.  So we cannot test with a ping but again we are able to access the sites through the manual download process.

                                • 13. Re: Windows Patch Catalog update fails after upgrade to 8.6
                                  Charlie Sullivan

                                  Right, we recently went back to using the DNS name, but I'm not sure if it still resolves to the same IP address as before, thus I don't know if that "bad" IP address is still in the mix. I should have pursued this further and opened up a ticket, but I just didn't have time.

                                   

                                  I mentioned doing that for troubleshooting purposes only, since it's not a real solution, as you pointed out.

                                  • 14. Re: Windows Patch Catalog update fails after upgrade to 8.6
                                    Patrick O'Callaghan

                                    Bill,

                                    When running the catalog this way it works.  I pointed the Shavlik urls to BLFileserver/d/filestorage/templates/(pd5.xml, hf7b.xml, oemcatalog.zip) and it is updating.

                                     

                                    However the customer is going to need the xml files to automatically update.

                                    1 2 Previous Next