AnsweredAssumed Answered

ETL Job, running with a cronjob, sometimes do not end

Question asked by Eloy Chang on Aug 14, 2018
Latest reply on Aug 22, 2018 by Joao Figueiredo



I have some cronjobs on a m5d.xlarge AWS EC2 instance with Ubuntu 16.04.4, running kjb files, the command I'm using is:

sh /home/ubuntu/Pentaho/data-integration/ -file="/home/ubuntu/Documents/pdi_etls/etl_job.kjb" --level=Minimal > /home/ubuntu/Documents/pdi_logs/etl_job.log


The jobs just copy some data from a Mongo DB or PostgreSQL and put it on a Amazon Redshift Cluster.

I have run the job localy and on the instance, and it works well, also it does with the cron, but sometimes the job never end, so I have to kill it to make it work again.


I have develop a automatic process to do this, but I thinks that is not the way the job should behave, can you help me with some ideas of what the job get stuck?


In the instance stats neither RAM nor CPU have a excesive use (both below 70%) when this happens, so do not looks like a hardware problem.


I hope you can help me.