When a node running a Retz job fails, then Retz automatically restarts that job on a different node with the same job ID. In our cluster, we have experienced that such a restarted job may at some point turn into KILLED with Retz, even though it is still running and visible from the Mesos web interface.
The log that Retz shows when the running job turns into KILLED looks as follows:
2017-12-18 19:08:41.104 INFO [Thread-366804] i.g.retz.scheduler.RetzScheduler - Status update of task retz-foobar-id-3218: TASK_FAILED / Abnormal executor termination: unknown container (REASON_EXECUTOR_TERMINATED)
2017-12-18 19:08:41.181 INFO [pool-3-thread-1] io.github.retz.db.Database - Job (id=3218) status updated to KILLED
2017-12-18 19:08:41.181 INFO [pool-3-thread-1] io.github.retz.scheduler.JobQueue - Job id=3218 has failed: Abnormal executor termination: unknown container
I am not sure this is a Retz issue, it may as well be a Mesos bug.
When a node running a Retz job fails, then Retz automatically restarts that job on a different node with the same job ID. In our cluster, we have experienced that such a restarted job may at some point turn into KILLED with Retz, even though it is still running and visible from the Mesos web interface.
The log that Retz shows when the running job turns into KILLED looks as follows:
I am not sure this is a Retz issue, it may as well be a Mesos bug.