Hendrik/xgboost
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6,777 Commits 32 Branches 59 Tags
Commit Graph

134 Commits

Author SHA1 Message Date
Jiaming Yuan
c75a3bc0a9
[breaking] [jvm-packages] Remove rabit check point. (#9599) 
- Add `numBoostedRound` to jvm packages
- Remove rabit checkpoint version.
- Change the starting version of training continuation in JVM [breaking].
- Redefine the checkpoint version policy in jvm package. [breaking]
- Rename the Python check point callback parameter. [breaking]
- Unifies the checkpoint policy between Python and JVM.
 2023-09-26 18:06:34 +08:00
Bobby Wang
344f90b67b
[jvm-packages] throw exception when tree_method=approx and device=cuda (#9478) 
---------

Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>
 2023-08-14 17:52:14 +08:00
Jiaming Yuan
f4fb2be101
[jvm-packages] Add the new device parameter. (#9385) 2023-07-17 18:40:39 +08:00
jinmfeng001
a1367ea1f8
Set feature_names and feature_types in jvm-packages (#9364) 
* 1. Add parameters to set feature names and feature types
2. Save feature names and feature types to native json model

* Change serialization and deserialization format to ubj.
 2023-07-12 15:18:46 +08:00
Jiaming Yuan
1fcc26a6f8
Set ndcg to default for LTR. (#8822) 
- Add document.
- Add tests.
- Use `ndcg` with `topk` as default.
 2023-06-09 23:31:33 +08:00
Emil Ejbyfeldt
a84a1fde02
[jvm-packages] Update scalatest to 3.2.15 (#8925) 
---------

Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>
 2023-04-20 22:16:56 +08:00
Jiaming Yuan
564df59204
[breaking] [jvm-packages] Remove scala-implemented tracker. (#9045) 2023-04-20 16:29:35 +08:00
Jiaming Yuan
26209a42a5
Define git attributes for renormalization. (#8921) 2023-03-16 02:43:11 +08:00
Jiaming Yuan
badeff1d74
Init estimation for regression. (#8272) 2023-01-11 02:04:56 +08:00
Rong Ou
668b8a0ea4
[Breaking] Switch from rabit to the collective communicator (#8257) 
* Switch from rabit to the collective communicator

* fix size_t specialization

* really fix size_t

* try again

* add include

* more include

* fix lint errors

* remove rabit includes

* fix pylint error

* return dict from communicator context

* fix communicator shutdown

* fix dask test

* reset communicator mocklist

* fix distributed tests

* do not save device communicator

* fix jvm gpu tests

* add python test for federated communicator

* Update gputreeshap submodule

Co-authored-by: Hyunsu Philip Cho <chohyu01@cs.washington.edu>
 2022-10-05 14:39:01 -08:00
Rong Ou
7d43e74e71
JNI wrapper for the collective communicator (#8242) 2022-09-21 04:20:25 +08:00
Bobby Wang
545fd4548e
[jvm-packages] refactor xgboost read/write (#7956) 
1. Removed the duplicated Default XGBoost read/write which is copied from
  spark 2.3.x
2. Put some utils into util package
 2022-06-01 11:38:49 +08:00
Bobby Wang
6275cdc486
[jvm-packages] add format option when saving a model (#7940) 2022-05-30 15:49:59 +08:00
Bobby Wang
fbc3d861bb
[jvm-packages] remove default parameters (#7938) 2022-05-28 10:31:19 +08:00
Bobby Wang
5ef33adf68
[jvm-packges] set the correct objective if user doesn't explicitly set it (#7781) 2022-05-18 14:05:18 +08:00
Bobby Wang
b41cf92dc2
[jvm-packages] move dmatrix building into rabit context for cpu pipeline (#7908) 2022-05-17 14:52:25 +08:00
Bobby Wang
11e46e4bc0
[Breaking][jvm-packages] make classification model be xgboost-compatible (#7896) 2022-05-14 15:43:05 +08:00
Bobby Wang
9fa7ed1743
[Breaking][jvm-packages] remove timeoutRequestWorkers parameter (#7839) 2022-05-13 16:26:25 +08:00
Bobby Wang
dc2e699656
[Breaking][jvm-packages] Use barrier execution mode (#7836) 
With the introduction of the barrier execution mode. we don't need to kill SparkContext when some xgboost tasks failed. Instead, Spark will handle the errors for us. So in this PR, `killSparkContextOnWorkerFailure` parameter is deleted.
 2022-04-25 17:09:52 +08:00
Bobby Wang
c45665a55a
[jvm-packages] move the dmatrix building into rabit context (#7823) 
This fixes the QuantileDeviceDMatrix in distributed environment.
 2022-04-23 00:06:50 +08:00
Bobby Wang
2d83b2ad8f
[jvm-packages] add hostIp and python exec for rabit tracker (#7808) 2022-04-15 16:28:43 +08:00
Bobby Wang
3f536b5308
[jvm-packages] fix evaluation when featuresCols is used (#7798) 2022-04-13 12:52:50 +08:00
Bobby Wang
118192f116
[jvm-packages] xgboost4j-spark should work when featuresCols is specified (#7789) 2022-04-08 13:21:04 +08:00
Bobby Wang
89aa8ddf52
[jvm-packages] fix the prediction issue for multi:softmax (#7694) 2022-02-24 01:09:45 +08:00
Jiaming Yuan
001503186c
Rewrite approx (#7214) 
This PR rewrites the approx tree method to use codebase from hist for better performance and code sharing.

The rewrite has many benefits:
- Support for both `max_leaves` and `max_depth`.
- Support for `grow_policy`.
- Support for mono constraint.
- Support for feature weights.
- Support for easier bin configuration (`max_bin`).
- Support for categorical data.
- Faster performance for most of the datasets. (many times faster)
- Support for prediction cache.
- Significantly better performance for external memory.
- Unites the code base between approx and hist.
 2022-01-10 21:15:05 +08:00
Bobby Wang
e8c1eb99e4
[jvm-package] Clean up the legacy gpu support tests (#7523) 2021-12-21 09:15:51 +08:00
Bobby Wang
cb685607b2
[jvm-packages] Rework the train pipeline (#7401) 
1. Add PreXGBoost to build RDD[Watches] from Dataset
2. Feed RDD[Watches] built from PreXGBoost to XGBoost to train
 2021-11-10 17:51:38 +08:00
nicovdijk
31a307cf6b
[XGBoost4J-Spark] Serialization for custom objective and eval (#7274) 
* added type hints to custom_obj and custom_eval for Spark persistence


Co-authored-by: Bobby Wang <wbo4958@gmail.com>
 2021-10-21 16:22:23 +08:00
Bobby Wang
2c684ffd32
[jvm-packages] fix "key not found: train" issue (#6842) 
* [jvm-packages] fix "key not found: train" issue

* fix bug
 2021-04-18 23:28:39 -07:00
Bobby Wang
49c22c23b4
[jvm-packages] fix early stopping doesn't work even without custom_eval setting (#6738) 
* [jvm-packages] fix early stopping doesn't work even without custom_eval setting

* remove debug info

* resolve comment
 2021-03-06 20:19:40 -08:00
zhang_jf
cc581b3b6b
Misleading exception information: no such param of "allow_non_zero_missing" (#6418) 2020-11-20 19:33:34 +08:00
Nan Zhu
4d1d5d4010
[jvm-packages] fix potential unit test suites aborted issue (#6373) 
* fix race conditio

* code cleaning

rm pom.xml-e

* clean again

* fix compilation issue

* recover

* avoid using getOrCreate

* interrupt zombie threads

* safe guard

* fix deadlock

* Update SparkParallelismTracker.scala
 2020-11-17 10:59:26 -08:00
Jiaming Yuan
d61b628bf5
Remove RABIT CMake targets. (#6275) 
* Now it's built as part of libxgboost.
* Set correct C API error in RABIT initialization and finalization.
* Remove redundant message.
* Guard the tracker print C API.
 2020-10-27 01:30:20 +08:00
Bobby Wang
0e2d5669f6
[jvm-packages] cancel job instead of killing SparkContext (#6019) 
* cancel job instead of killing SparkContext

This PR changes the default behavior that kills SparkContext. Instead, This PR
cancels jobs when coming across task failed. That means the SparkContext is
still alive even some exceptions happen.

* add a parameter to control if killing SparkContext

* cancel the jobs the failed task belongs to

* remove the jobId from the map when one job failed.

* resolve comments
 2020-09-02 14:20:59 -07:00
Anthony D'Amato
f58e41bad8
Fix deterministic partitioning with dataset containing Double.NaN (#5996) 
The functions featureValueOfSparseVector or featureValueOfDenseVector could return a Float.NaN if the input vectore was containing any missing values. This would make fail the partition key computation and most of the vectors would end up in the same partition. We fix this by avoid returning a NaN and simply use the row HashCode in this case.
We added a test to ensure that the repartition is indeed now uniform on input dataset containing values by checking that the partitions size variance is below a certain threshold.

Signed-off-by: Anthony D'Amato <anthony.damato@hotmail.fr>
 2020-08-18 18:55:37 -07:00
Shaochen Shi
71197d1dfa
[jvm-packages] Fix wrong method name setAllowZeroForMissingValue. (#5740) 
* Allow non-zero for missing value when training.

* Fix wrong method names.

* Add a unit test

* Move the getter/setter unit test to MissingValueHandlingSuite

Co-authored-by: Hyunsu Cho <chohyu01@cs.washington.edu>
 2020-08-01 17:16:42 -07:00
Jiaming Yuan
75b8c22b0b
Fix prediction heuristic (#5955) 
* Relax check for prediction.
* Relax test in spark test.
* Add tests in C++.
 2020-07-29 19:24:07 +08:00
Bobby Wang
8943eb4314
[BLOCKING] [jvm-packages] add gpu_hist and enable gpu scheduling (#5171) 
* [jvm-packages] add gpu_hist tree method

* change updater hist to grow_quantile_histmaker

* add gpu scheduling

* pass correct parameters to xgboost library

* remove debug info

* add use.cuda for pom

* add CI for gpu_hist for jvm

* add gpu unit tests

* use gpu node to build jvm

* use nvidia-docker

* Add CLI interface to create_jni.py using argparse

Co-authored-by: Hyunsu Cho <chohyu01@cs.washington.edu>
 2020-07-26 21:53:24 -07:00
Philip Hyunsu Cho
487ab0ce73
[BLOCKING] Handle empty rows in data iterators correctly (#5929) 
* [jvm-packages] Handle empty rows in data iterators correctly

* Fix clang-tidy error

* last empty row

* Add comments [skip ci]

Co-authored-by: Nan Zhu <nanzhu@uber.com>
 2020-07-25 13:46:19 -07:00
Bobby Wang
9f85e92602
[jvm-packages] update spark dependency to 3.0.0 (#5836) 2020-07-12 20:58:30 -07:00
Bobby Wang
ad826e913f
[jvm-packages]add feature size for LabelPoint and DataBatch (#5303) 
* fix type error

* Validate number of features.

* resolve comments

* add feature size for LabelPoint and DataBatch

* pass the feature size to native

* move feature size validating tests into a separate suite

* resolve comments

Co-authored-by: fis <jm.yuan@outlook.com>
 2020-04-07 16:49:52 -07:00
Nan Zhu
d7b45fbcaf
[jvm-packages] do not use multiple jobs to make checkpoints (#5082) 
* temp

* temp

* tep

* address the comments

* fix stylistic issues

* fix

* external checkpoint
 2020-02-01 19:36:39 -08:00
Philip Hyunsu Cho
37fdfa03f8
[jvm-packages] Comply with scala style convention + fix broken unit test (#5134) 
* Fix scala style check

* fix messed unit test
 2019-12-18 17:26:58 -08:00
cpfarrell
bc9d88259f [jvm-packages] Allow for bypassing spark missing value check (#4805) 
* Allow for bypassing spark missing value check

* Update documentation for dealing with missing values in spark xgboost
 2019-12-18 10:48:20 -08:00
Chen Qin
b29b8c2f34 [jvm-packages] update rabit, surface new changes to spark, add parity and failure tests (#4966) 
* [phase 1] expose sets of rabit configurations to spark layer

* add back mutable import

* disable ring_mincount till https://github.com/dmlc/rabit/pull/106d

* Revert "disable ring_mincount till https://github.com/dmlc/rabit/pull/106d"

This reverts commit 65e95a98e24f5eb53c6ba9ef9b2379524258984d.

* apply latest rabit

* fix build error

* apply https://github.com/dmlc/xgboost/pull/4880

* downgrade cmake in rabit

* point to rabit with DMLC_ROOT fix

* relative path of rabit install prefix

* split rabit parameters to another trait

* misc

* misc

* Delete .classpath

* Delete .classpath

* Delete .classpath

* Update XGBoostClassifier.scala

* Update XGBoostRegressor.scala

* Update GeneralParams.scala

* Update GeneralParams.scala

* Update GeneralParams.scala

* Update GeneralParams.scala

* Delete .classpath

* Update RabitParams.scala

* Update .gitignore

* Update .gitignore

* apply rabitParams to training

* use string as rabit parameter value type

* cleanup

* add rabitEnv check

* point to dmlc/rabit

* per feedback

* update private scope

* misc

* update rabit

* add rabit_timtout, fix failing test.

* split tests

* allow build jvm with rabit mock

* pass mock failures to rabit with test

* add mock error and graceful handle rabit assertion error test

* split mvn test

* remove sign for test

* update rabit

* build jvm_packages with rabit mock

* point back to dmlc/rabit

* per feedback, update scala header

* cleanup pom

* per feedback

* try fix lint

* fix lint

* per feedback, remove bootstrap_cache

* per feedback 2

* try replace dev profile with passing mvn property

* fix build error

* remove mvn property and replace with env setting to build test jar

* per feedback

* revert copyright headlines, point to dmlc/rabit

* revert python lint

* remove multiple failure test case as retry is not enabled in spark

* Update core.py

* Update core.py

* per feedback, style fix
 2019-11-01 14:21:19 -07:00
Jiaming Yuan
010b8f1428 Revert "[jvm-packages] update rabit, surface new changes to spark, add parity and failure tests (#4876)" (#4965) 
This reverts commit 86ed01c4bbecef66e1bc4d02fb13116bd6130fae.
 2019-10-18 14:02:35 -07:00
Chen Qin
86ed01c4bb [jvm-packages] update rabit, surface new changes to spark, add parity and failure tests (#4876) 
* Expose sets of rabit configurations to spark layer
 2019-10-18 15:07:31 -04:00
Nan Zhu
fc8c9b0521
[jvm-packages] enable deterministic repartitioning when checkpoint is enabled (#4807) 
* do reparititoning in DataUtil

* keep previous behavior of partitioning without checkpoint

* deterministic repartitioning

* change
 2019-09-19 15:21:05 -07:00
Nan Zhu
7b5cbcc846
[jvm-packages] cleaning checkpoint file after a successful training (#4754) 
* cleaning checkpoint file after a successful file

* address comments
 2019-08-14 10:57:47 -07:00
Oleksandr Pryimak
b68de018b8 [jvm-packages] jvm test should clean up after themselfs (#4706) 2019-08-04 14:09:11 -07:00