377 Commits

Author SHA1 Message Date
Nan Zhu
785094db53
[jvm-packages] fix issue when spark job execution thread cannot return before we execute first() (#3758)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* fix scalastyle error

* fix scalastyle error

* sparjJobThread

* update

* fix issue when spark job execution thread cannot return before we execute first()
2018-10-05 22:20:50 -07:00
zengxy
9e73087324 [jvm-packages] support specified feature names when getModelDump and getFeatureScore (#3733)
* [jvm-packages] support specified feature names for jvm when get ModelDump and get FeatureScore (#3725)

* typo and style fix
2018-10-04 09:05:42 -07:00
weitian
efc4f85505 [jvm-packages] Fix #3489: Spark repartitionForData can potentially shuffle all data and lose ordering required for ranking objectives (#3654) 2018-10-03 08:43:55 -07:00
Sergei Lebedev
87aca8c244 [jvm-packages] Fixed the distributed updater check (#3739)
The updater used in distributed training is grow_histmaker and not 
grow_colmaker as the error message stated prior to this commit.
2018-10-01 11:22:01 -07:00
Nan Zhu
79d854c695
[jvm-packages] fix errors in example (#3719)
* add back train method but mark as deprecated

* fix scalastyle error

* add back train method but mark as deprecated

* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* fix scalastyle error

* instrumentation

* use log console

* better measurement

* fix erros in example

* update histmaker
2018-09-22 16:39:38 -07:00
Nan Zhu
aa53e9fc8d
[jvm-packages] bump spark version (#3709) 2018-09-19 11:18:01 -07:00
Michael Mui
20a9e716bd [jvm-packages] Fix "obj_type" error to enable custom objectives and evaluations (#3646)
credits to @mmui
2018-09-14 12:06:33 -07:00
Jerry Lin
9acd549dc7 [jvm-packages] Add rank:ndcg and rank:map to Spark supported objectives (#3697) 2018-09-13 09:51:24 -07:00
Chen Qin
42b108136f [jvm-packages] bump flink version number (#3686)
* bump flink version number

* bump flink version number

* add missing hadoop dependency
2018-09-13 09:33:09 -07:00
Nan Zhu
d1e75d615e
[jvm-packages] Remove copy paste error in test suite (#3692)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* remove copy paste error
2018-09-11 13:08:36 -07:00
Joseph Bradley
14a8b96476 [jvm-packages] xgboost-spark warning when Spark encryption is turned on (#3667)
* added test, commented out right now

* reinstated test

* added fix for checking encryption settings

* fix by using RDD conf

* fix compilation

* renamed conf

* use SparkSession if available

* fix message

* nop

* code review fixes
2018-09-10 14:21:01 -07:00
Matthew Tovbin
beab6e08dd Remove println in jsonDecode (#3665)
Following issue  #3578
2018-09-07 15:47:26 -07:00
Nan Zhu
3261002099
[jvm-packages] throw ControlThrowable instead of InterruptedException (#3632)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* interrupted exception is not rethrown
2018-08-25 20:30:21 -07:00
Nan Zhu
4912c1f9c6
[jvm-packages] fix checkpoint save/load (#3614)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* fix update checkpoint func
2018-08-21 12:34:24 -07:00
Matthew Tovbin
b53a5a262c [jvm-packages] getTreeLimit return type should be Int 2018-08-17 09:36:00 -07:00
Nan Zhu
73bd590a1d
[jvm-packages] add the missing scm urls (#3589)
for some reason this part was missing in master branch????
2018-08-14 15:05:23 -07:00
Matthew Tovbin
2b7a1c5780 [jvm-packages] Avoid loosing precision when computing probabilities by converting to Double early (#3576) 2018-08-13 14:05:07 -07:00
Matthew Tovbin
ce0f0568a6 Make sure 'thresholds' are considered when executing predict method (#3577) 2018-08-13 14:04:47 -07:00
Philip Hyunsu Cho
6288f6d563 Update JVM packages version to 0.81-SNAPSHOT (#3584) 2018-08-13 10:17:52 -07:00
Philip Hyunsu Cho
96826a3515
Release version 0.80 (#3541)
* Up versions

* Write release note for 0.80
2018-08-13 01:38:37 -07:00
Mathew
06ef4db4cc Fix Spark 2.2 Support (Amending #3062) (#3325)
This pull request amends the broken #3062 allow Spark 2.2 to work.

Please note this won't work in Spark <=2.1 as sc.removeSparkListener was implemented in Spark 2.2. (So perhaps a more general method is better, although that is what was attempted in #3062)

This PR fixes: #3208, #3151 and the discussion in #1927.

I do find it strange that #3062 dose not work in Spark 2.2, it's probably due to some sort of public/private issue in the org.apache.spark.scheduler.LiveListenerBus class inheritance (In Spark itself). The error is: `java.lang.NoSuchMethodError: org.apache.spark.scheduler.LiveListenerBus.removeListener(Ljava/lang/Object;)V`
2018-08-12 18:35:20 -07:00
Matthew Tovbin
7300002516 [jvm-packages] Use treeLimit param in getTreeLimit (#3575) 2018-08-10 09:38:58 -07:00
Philip Hyunsu Cho
aa4ee6a0e4
[BLOCKING] Adding JVM doc build to Jenkins CI (#3567)
* Adding Java/Scala doc build to Jenkins CI

* Deploy built doc to S3 bucket

* Build doc only for branches

* Build doc first, to get doc faster for branch updates

* Have ReadTheDocs download doc tarball from S3

* Update JVM doc links

* Put doc build commands in a script

* Specify Spark 2.3+ requirement for XGBoost4J-Spark

* Build GPU wheel without NCCL, to reduce binary size
2018-08-09 13:27:01 -07:00
Matthew Tovbin
bad76048d1 Eliminate use of System.out + proper error logging (#3572) 2018-08-09 10:06:17 -07:00
Nan Zhu
1c08b3b2ea
[jvm-packages] enable predictLeaf/predictContrib/treeLimit in 0.8 (#3532)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* partial finish

* no test

* add test cases

* add test cases

* address comments

* add test for regressor

* fix typo
2018-08-07 14:01:18 -07:00
Philip Hyunsu Cho
4a429a7c4f Add reg:tweedie to supported objectives in XGBoost4J-Spark (#3552) 2018-08-05 07:42:59 -07:00
Nan Zhu
31d1baba3d [jvm-packages] Tutorial of XGBoost4J-Spark (#3534)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* add new

* update doc

* finish Gang Scheduling

* more

* intro

* Add sections: Prediction, Model persistence and ML pipeline.

* Add XGBoost4j-Spark MLlib pipeline example

* partial finished version

* finish the doc

* adjust code

* fix the doc

* use rst

* Convert XGBoost4J-Spark tutorial to reST

* Bring XGBoost4J up to date

* add note about using hdfs

* remove duplicate file

* fix descriptions

* update doc

* Wrap HDFS/S3 export support as a note

* update

* wrap indexing_mode example in code block
2018-08-03 21:17:50 -07:00
Nan Zhu
6cf97b4eae
[jvm-packages] consider spark.task.cpus when controlling parallelism (#3530)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* consider spark.task.cpus when controlling parallelism

* fix bug

* fix conf setup

* calculate requestedCores within ParallelismController

* enforce spark.task.cpus = 1

* unify unit test case framework

* enable spark ui
2018-07-31 06:19:45 -07:00
Nan Zhu
b546321c83
[jvm-packages] the current version of xgboost does not consider missing value in prediction (#3529)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* consider missing value in prediction

* handle single prediction instance

* fix type conversion
2018-07-30 14:16:24 -07:00
Nan Zhu
b5178d3d99
[jvm-packages] a better explanation about the inconsistent issue (#3524) 2018-07-28 17:34:39 -07:00
Nan Zhu
e2f09db77a
[jvm-packages] minor fix for parameter name in example (#3507) 2018-07-25 19:57:40 -07:00
Yanbo Liang
c004cea788 Expose setCustomObj & setCustomEval for XGBoostClassifier & XGBoostRegressor. (#3486) 2018-07-17 21:16:51 -07:00
Yanbo Liang
2f8764955c [JVM-packages] Support single instance prediction. (#3464)
* Support single instance prediction.

* Address comments.
2018-07-12 14:17:53 -07:00
ngoyal2707
cb017d0c9a [jvm-packages] removed old group_data from spark api (#3451) 2018-07-07 22:21:01 -07:00
Nan Zhu
aa90e5c6ce
[jvm-packages] disable booster setup for xgboost4j-spark (#3456)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* disable booster setup in spark

* check in parameter conversion

* fix compilation issue

* update exception type
2018-07-07 21:57:24 -07:00
Yanbo Liang
aec6299c49 [jvm-packages] Expose nativeBooster for XGBoostClassificationModel and XGBoostRegressionModel. (#3428) 2018-07-01 15:06:16 -07:00
Yun Ni
30d10ab035 Convert handle == nullptr from SegFault to user-friendly error. (#3021)
* Convert SegFault to user-friendly error.

* Apply the change to DMatrix API as well
2018-06-29 06:30:26 +00:00
Adam Johnston
0988fb191f [jvm-packages] avoid use of Seq.apply in buildGroups (#3413) 2018-06-26 16:00:28 -07:00
Nan Zhu
d062c6f61b
[jvm-packages] Maven central release stuffs (#3401)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* maven central release
2018-06-22 06:41:28 -07:00
James
eecf341ea7 [jvm-packages] Added latest version number example (#3374)
* Added latest version number example

* Added latest version number example
2018-06-18 22:09:39 -07:00
Yanbo Liang
2c4359e914 [jvm-packages] XGBoost Spark integration refactor (#3387)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* [jvm-packages] XGBoost Spark integration refactor. (#3313)

* XGBoost Spark integration refactor.

* Make corresponding update for xgboost4j-example

* Address comments.

* [jvm-packages] Refactor XGBoost-Spark params to make it compatible with both XGBoost and Spark MLLib (#3326)

* Refactor XGBoost-Spark params to make it compatible with both XGBoost and Spark MLLib

* Fix extra space.

* [jvm-packages] XGBoost Spark supports ranking with group data. (#3369)

* XGBoost Spark supports ranking with group data.

* Use Iterator.duplicate to prevent OOM.

* Update CheckpointManagerSuite.scala

* Resolve conflicts
2018-06-18 15:39:18 -07:00
Bruce Qu
578a0c7ddb params confusion fixed (#3386) 2018-06-15 13:17:35 -07:00
Nan Zhu
f66731181f
Update 0.8 version num (#3358)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* update 0.80
2018-06-02 07:06:01 -07:00
Sergei Lebedev
8f6aadd4b7 [jvm-packages] Fixed CheckpointManagerSuite for Scala 2.10 (#3332)
As before, the compilation error is caused by mixing positional and
labelled arguments.
2018-05-19 18:28:11 -07:00
Nan Zhu
49b9f39818
[jvm-packages] update xgboost4j cross build script to be compatible with older glibc (#3307)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* static glibc glibc++

* update to build with glib 2.12

* remove unsupported flags

* update version number

* remove properties

* remove unnecessary command

* update poms
2018-05-10 06:39:44 -07:00
Nan Zhu
e1f57b4417
[jvm-packages] scripts to cross-build and deploy artifacts to github (#3276)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* cross building files

* update

* build with docker

* remove

* temp

* update build script

* update pom

* update

* update version

* upload build

* fix path

* update README.md

* fix compiler version to 4.8.5
2018-04-28 07:41:30 -07:00
Yanbo Liang
4850f67b85 Fix broken link for xgboost-spark example. (#3275) 2018-04-26 06:45:01 -07:00
Nan Zhu
25b2919c44
[jvm-packages] change version of jvm to keep consistent with other pkgs (#3253)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* change version of jvm to keep consistent with other pkgs
2018-04-19 20:48:50 -07:00
Nan Zhu
d9dd485313
[jvm-packages] upgrade spark version to 2.3 (#3254)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* update default spark version to 2.3
2018-04-19 20:15:19 -07:00
Nan Zhu
4109818b32
[jvm-packages] add back libsvm notes (#3232)
* add back train method but mark as deprecated

* add back train method but mark as deprecated

* fix scalastyle error

* fix scalastyle error

* add back libsvm notes
2018-04-10 09:00:58 -07:00