How to get logs for a Spark job

  • To get the initial logs that contain information about the pod spinup, do curl -X GET -u admin:password123 http://localhost:8764/api/apollo/spark/driver/log/{jobId}

  • Get the pod ID by running k get pods -l spark-role=driver -l jobConfigId=<job-id>

  • Logs from failed jobs can be obtained by using kubectl logs [DRIVER-POD-NAME]

  • Logs from running containers can be tailed using the -f parameter: kubectl logs -f [POD-NAME]

Spark deletes failed and successful executor pods. Fusion provides a cleanup Kubenetes cron job that removes successfully completed driver pods every 15 minutes.