I had same experience for some of the NSH job, especially for ETL job.
I always end up restarting the job server from BSA console.
Just make sure you have nothing else running on that server ;-)
For NSH Script jobs, you may try to find the PID in the Application Server logs and kill it.
[10 Feb 2016 11:45:11,581] [WorkItem-Thread-55] [INFO] [XXXX@XYZ.COM:ROLE_ABC:] [NSHScript] Started pid 399
Or, If you have access to Database, mark it as Completed in the DB:
SELECT j.JOB_ID,j.NAME,jr.JOB_RUN_ID,jr.PROGRESS_STATUS_ID FROM JOB j, JOB_RUN jr WHERE j.JOB_ID=jr.JOB_ID and j.NAME like '%JOB_NAME%' and jr.PROGRESS_STATUS_ID=5;
update job_run set PROGRESS_STATUS_ID=7 where JOB_RUN_ID=123456;
If it is possible to restart the application server then you can be able to cancel all running jobs, I faced similar kind of issue in patch catalog update Job. I restarted the application server and the issue got resolved. if it is production environment then I think you can go with solution provided by Santosh .
As this is production and multiple users are connected i cannot restart any service. Though i found PID in appserver log but on OS it says invalid PID.
[11 Feb 2016 14:20:07,973] [WorkItem-Thread-3] [INFO] [user@domain:role_Linux:] [NSHScript] Started pid 1038: /opt/bmc/bladelogic/NSH/bin/nsh --norc -c /opt/bmc/bladelogic/NSH/tmp/application_server/scripts/job__aa23e044-42b7-4d7b-82e1-cd48e21d7734/master_b9fe7f39-beb1-4fb9-8c78-72dafd845ead
[root@lapclm10 ~]# kill -9 1038
-bash: kill: (1038) - No such process
Looks like the PID does not exist anymore. can you grep and see the PID? (ps -ef | grep 1038).
if no PID found then look at the job run log from console and see whether it is showing cancelled/aborted.
If it is showing aborted then the job will be cleared from task in progress by restarting job server only.
Then, it must be due to stale entry in the DB. Please check with DBA and get it marked as Completed.
I just restarted Job server and now its fine....