* cancel job instead of killing SparkContext
This PR changes the default behavior that kills SparkContext. Instead, This PR
cancels jobs when coming across task failed. That means the SparkContext is
still alive even some exceptions happen.
* add a parameter to control if killing SparkContext
* cancel the jobs the failed task belongs to
* remove the jobId from the map when one job failed.
* resolve comments
* add back train method but mark as deprecated
* add back train method but mark as deprecated
* fix scalastyle error
* fix scalastyle error
* consider spark.task.cpus when controlling parallelism
* fix bug
* fix conf setup
* calculate requestedCores within ParallelismController
* enforce spark.task.cpus = 1
* unify unit test case framework
* enable spark ui
* Add SparkParallelismTracker to prevent job from hanging
* Code review comments
* Code Review Comments
* Fix unit tests
* Changes and unit test to catch the corner case.
* Update documentations
* Small improvements
* cancalAllJobs is problematic with scalatest. Remove it
* Code Review Comments
* Check number of executor cores beforehand, and throw exeception if any core is lost.
* Address CR Comments
* Add missing class
* Fix flaky unit test
* Address CR comments
* Remove redundant param for TaskFailedListener
