It hangs only when run from the console?
From spoon it works 100% of the time?
How much memory have you assigned to the Carte Server process?
Yes, from spoon it work everytime.
It hang only when running on the carte server ( only launchable from the Integrator console in Atrium 9.1)
Initially the memory assigned was 512, then whe have tested with an Xmx value of 2048. But the process is still hanging.
I also suspect a memory management issue But I can't find any clue about that. (nothing in the java logs, nor the Carte logs)
Can you monitor the Carte Server process using JvisualVM while you run the job?
If there are HeapSpace issues they should show in Arcarte.log though... unless there is something else really going on.
Do you have ARCarte.log from one of these hanging scenarios?
Here is the arcarte.log and also the arjavaplugin.log.
The Linux team has been called to investigate dthe system while the job is hanging.
I was expecting to see some momory allocation error messages in the arjavaplugin log. But there is nothing very usefull there.
1 of 1 people found this helpful
The issue is here:
TEST_BGL_LoadPeople - Transformation is about to allocate step [ExtractGenerique Input] of type [CsvInput]
ExtractGeneriqueInput step is of type CSV Input.
CSV Input is deprecated and should not be used (stopped support in Pentaho version n2). You need to remove that and replace with TextFileInput which is the current available and supported step for reading CSV and other text files.
Thank you for your support.
The client have restored a copy of his environment before the upgrade to 9.1.
So, on the same system (linux) in Remedy 8.1 (all components) , the job is working.
It seems it's a side-effect of the upgrade to Remedy 9.1
Replacing the CSV Input components by Text Inputs haven't solved the issue on 9.1. Even if it is a good things to do this correction.
Is it possible to have a log of the Carte engine? I mean the log of what the engine is doing internally, not what the jobs are doing.
1 of 1 people found this helpful
In 9.1 CSV Input started showing even more problems than in 8.1.
I have seen many users with this issue in this version while using CSV Input.
I'm surprised changing to Text File Input did not resolve the issue.
Can you share the new arcarte.log from that particular execution.
The client replaced the CSV inputs with Text inputs.
It helps but the job is still freezing. Now,sometime it completes, but often it freeze.
As the process is still working during the freeze, is it a place where we can get a log of what it is doing?
Have you assigned sufficient memory to the process?
It seems to be freezing during reading CSV. How many records are in the CSV?
According to the screenshot it reads 10000 records (first chunk) and then it stops. Can you try with smaller CSVs?
The freeze is not always at the same place.
Most of the time it is after the read step is completed.
Often, it is during an AROUTPUT.
The XMX has been set to 2048m without it change anything.
The csv input was replaced by text input everywhere. The freeze doesn't happen anymore during the input step. If the system freezes it is during a AR or CMDB output step. Previous steps are all finished at that time. It doesn't always stop and the number of records processed does seem to change between runs. The only means to stop the job is to kill kettle.
So during the hang do you see the memory going to the top?
Did you try monitoring the process using JVisualVM?
Can you attach a log from one of the latest hangs after you changes all CSVInputs to TextFileInputs?
The problem did not occur for some time. But recently it occurred again and the client asked me to have a look.
During the hang memory seems to be stable at first sight. I didn't have the chance to monitor the process using jvisualvm.
I did a quick and dirty check using top and strace. During the hang on top -h we see 3 threads using cpu. I did a strace on these. One was clearly the web interface. On the others I saw only futex calls that timed out very fast like in a loop.
I'm not at the client currently. But I will ask them to provide the requested information.
Thanks for the reaction.