0 Replies Latest reply on Nov 6, 2015 9:26 PM by Mark Saunders

    BMC Portal Web Services Error

    Mark Saunders

      In the absence of the support portal at the moment can anyone help me with this frustrating problem.

       

      Our old portal server was rebooted today and the web services are not coming back up correctly. The portal web page is showing the site is currently unavailable message.

       

      In the mod_jk.log with debuging on it looks like tomcat is not starting up. This has happened before and a reboot fixed this but this time the error is persisting.

       

      Netstat shows that the connection is trying but there is nothing listening (I have been trying to fix the sockets error but I am not sure if that is related).

       

      C:\Users\saundersm-prv>netstat -anob | findstr :9375

      x: Windows Sockets initialization failed: 5

      x: Windows Sockets initialization failed: 5

      x: Windows Sockets initialization failed: 5

      x: Windows Sockets initialization failed: 5

      x: Windows Sockets initialization failed: 5

      x: Windows Sockets initialization failed: 5

      x: Windows Sockets initialization failed: 5

        TCP    10.210.40.20:49575     10.210.40.20:9375      SYN_SENT        2156

       

       

       

      This error just cycles again and again (see log below)

       

       

      [Sat Nov 07 00:38:24 2015][5968:2632] [info]  jk_open_socket::jk_connect.c (451): connect to 10.210.40.20:9375 failed with errno=61

      [Sat Nov 07 00:38:24 2015][5968:2632] [info]  ajp_connect_to_endpoint::jk_ajp_common.c (873): Failed opening socket to (10.210.40.20:9375) with (errno=61)

      [Sat Nov 07 00:38:24 2015][5968:2632] [info]  ajp_send_request::jk_ajp_common.c (1259): (node1) error connecting to the backend server (errno=61)

      [Sat Nov 07 00:38:24 2015][5968:2632] [info]  ajp_service::jk_ajp_common.c (1916): (node1) sending request to tomcat failed,  recoverable operation attempt=2

      [Sat Nov 07 00:38:24 2015][5968:2632] [error] ajp_service::jk_ajp_common.c (1928): (node1) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] ajp_done::jk_ajp_common.c (2258): recycling connection pool slot=0 for worker node1

      [Sat Nov 07 00:38:24 2015][5968:2632] [info]  service::jk_lb_worker.c (1012): service failed, worker node1 is in error state

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] service::jk_lb_worker.c (1023): recoverable error... will try to recover on other worker

      [Sat Nov 07 00:38:24 2015][5968:2632] [info]  service::jk_lb_worker.c (1106): All tomcat instances are busy or in error state

      [Sat Nov 07 00:38:24 2015][5968:2632] [info]  jk_handler::mod_jk.c (2142): Service error=0 for worker=loadbalancer

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (575): Attempting to map URI '/errors/error.html' from 35 maps

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/help_services/html/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/servlet/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/ServletRedirector=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/JspRedirector=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/executivedashboard/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/smsConsoleServer/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/agentinterface/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/simwebservice/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/jmx-console/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/web-console/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/drmop-ws25/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/*.nav=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/*.jsp=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/*.do=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/operator/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/drmop-ws/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/cmdbhelp/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/jbossws/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/moduleX/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/jkstatus/=status' source 'JkMount'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/smsIwc/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/smsSme/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/drmop/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/smsIX/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/arsys/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/pere/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/ace/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/ri/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/executivedashboard=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/simwebservice=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/web-console=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/jmx-console=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/cmdbhelp=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/arsys=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (575): Attempting to map URI '/errors/error.html' from 35 maps

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/help_services/html/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/servlet/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/ServletRedirector=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/JspRedirector=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/executivedashboard/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/smsConsoleServer/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/agentinterface/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/simwebservice/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/jmx-console/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/web-console/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/drmop-ws25/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/*.nav=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/*.jsp=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/*.do=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/operator/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/drmop-ws/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/cmdbhelp/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/jbossws/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/moduleX/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/jkstatus/=status' source 'JkMount'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/smsIwc/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/smsSme/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/portal/=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/drmop/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/smsIX/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/arsys/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/pere/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/ace/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/ri/*=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/executivedashboard=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/simwebservice=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/web-console=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/jmx-console=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/cmdbhelp=loadbalancer' source 'uriworkermap'

      [Sat Nov 07 00:38:24 2015][5968:2632] [debug] map_uri_to_worker::jk_uri_worker_map.c (587): Attempting to map context URI '/arsys=loadbalancer' source 'uriworkermap'