8 Replies Latest reply on Dec 11, 2015 11:54 AM by James Donohue

NSH Job - Job Execution timed out

James Donohue Dec 10, 2015 2:10 PM

Hello,

I have an NSH Script Job that I run against a Smart Group containing approx. 14K Servers. All it does is obtain about 6 properties from each Server using blcli calls and appends them to a csv file. When it completes, the Job emails the csv file to recipients. This works without issue with a smaller subset of Servers. Unfortunatley, after 48 hours (it was 60 Servers hort of being done) it failed with the message block below:

Error Dec 10, 2015 12:33:21 PM Job Execution timed out.

Error Dec 10, 2015 12:33:21 PM Work item for job (NSH Script Job:SJ-Create Server list from Group; ) was cancelled. Reason: Job Execution timed out.

Info Dec 10, 2015 12:33:23 PM Server: mlscbuttms801.mldad.ml.com - 14321

Error Dec 10, 2015 12:33:30 PM Job Work item cancelled. Stopping execution.

Why would I get this timeout message?

1. Re: NSH Job - Job Execution timed out

Bill Robinson Dec 10, 2015 2:25 PM (in response to James Donohue)

Is this a ‘type1’ (runscript) nsh script or ‘type2’ ?
Can you post the script ?
Like Show 0 Likes(0)

Actions
2. Re: NSH Job - Job Execution timed out

James Donohue Dec 10, 2015 2:48 PM (in response to Bill Robinson)
I think its a 'Type 2' (%h or %f) but the Depot property in the Console has it as a 'Type 1' which I never noticed. Anyway, its attached.

Note: I found a better way to run this job, removing a lot of the 'blcli_execute' commands but don't want to run it until I figure why I got a 'Timeout' on this job.

Create Server List from Group.nsh 2.9 K
Like Show 0 Likes(0)

Actions
3. Re: NSH Job - Job Execution timed out

Bill Robinson Dec 10, 2015 2:56 PM (in response to James Donohue)

so you are doing a loop across 14k servers and it's taking a long time and w/ less (how many less) servers it runs quickly... see the problem. ?

how long does one loop (server) take ? times 14k...

one thing that may be faster is Server.getBulkServerPropertyValuesAndLogResults. i'm not sure if you could do 14k in one go though - you might need to break it into a couple groups.
Like Show 0 Likes(0)

Actions
4. Re: NSH Job - Job Execution timed out

James Donohue Dec 10, 2015 3:07 PM (in response to Bill Robinson)
This Job averages 4.8 Server per minute (48 hours for the job to run) and the new Job that I created from some suggestions an Engineer gave me averages 15 Servers per minute(15 hours for the job to run) which is a significant improvement. I'm going to start that Job tonight 'as is' and see what I get and then adjust from there if needed. My main concern is why did the first job timeout? This new job should take 15 hours to run (33 hour improvement) but I still don't want to run this and on the 15th hour get this 'timeout' error again.

1-Create Server list from Group.nsh 2.1 K
Like Show 0 Likes(0)

Actions
5. Re: NSH Job - Job Execution timed out

Bill Robinson Dec 10, 2015 3:12 PM (in response to Bill Robinson)

what is JOB_TIMEOUT set to on the job ?
Like Show 1 Likes(1)

Actions
6. Re: NSH Job - Job Execution timed out

James Donohue Dec 10, 2015 3:18 PM (in response to Bill Robinson)

JOB_TIMEOUT 2,880

which I now know is in minutes because it equates to 48 hours which is the EXACT time the Job STOPPED! So obvious!!!!

Thanks Bill, as always. I'm going to leave it at the default becasue I'm going to kick off my new code (15 hour ETA) tonight. Thanks for the guidance.
Like Show 0 Likes(0)

Actions
7. Re: NSH Job - Job Execution timed out

Jim Campbell Dec 11, 2015 10:37 AM (in response to James Donohue)

We struggled with jobs of this kind as well and I found that while not technically 'supported' its much easier to just write something that directly queries the database. Between the devices table and the v_server_property view you can get server properties with a 10 second SQL query that take hours of an nsh script running ( and frequently failing or timing out ) to get.

It would be helpful to have a CLI command that pulls a specific server property value for all servers but I have never been able to find one.
Like Show 0 Likes(0)

Actions
8. Re: NSH Job - Job Execution timed out

James Donohue Dec 11, 2015 11:54 AM (in response to Jim Campbell)

Hi Jim, Yeah I tried going down that road with the Engineers and DBA's and they offered to run the Report but they will not get to it for weeks/months. So, In the mean time, I need to perform this operation. As you mentioned, I'm sure it would be dramatically improved doing it this way.
Like Show 0 Likes(0)

Actions