Hendrik/xgboost
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6,940 Commits 32 Branches 59 Tags
a39fef2c67de3fef60afb6a3cae83983e7d1748f
Commit Graph
96 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Bobby Wang cf0c1d0888 [pyspark] Avoid repartition. (#10408) 2024-06-12 02:26:10 +08:00
Bobby Wang bc7643d35e [jvm-packages] Don't cast to float if it's already float (#10386) 2024-06-04 18:01:51 +08:00
Bobby WangandHyunsu Cho 9def441e9a [CI] add script to generate meta info and upload to s3 (#10295) 
* [CI] add script to generate meta info and upload to s3

* Write Python script to generate meta.json

* Update other pipelines

* Add wheel_name field

* Add description

---------

Co-authored-by: Hyunsu Cho <phcho@nvidia.com>
 2024-05-24 10:03:28 -07:00
Bobby WangandJiaming Yuan 932d7201f9 [jvm-packages] refine tracker (#10313) 
Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>
 2024-05-23 12:46:21 +08:00
Bobby Wang 9b465052ce [jvm-packages] fix group col for gpu packages (#10254) 2024-05-09 07:44:07 +08:00
Bobby Wang 8fb05c8c95 [pyspark] support stage-level for yarn/k8s (#10209) 2024-04-20 00:24:40 +08:00
Bobby Wang d24df52bb9 [pyspark] rework the log (#10077) 2024-02-29 16:47:31 +08:00
Bobby Wang f88c43801f [jvm-packages] update rapids dep to 23.12.1 (#9951) 
With this PR, XGBoost GPU can support scala 2.13
 2024-01-11 14:04:32 -08:00
Bobby Wang 3ff3a5f1ed [jvm-packages] support jdk 17 for test (#9959) 2024-01-08 17:30:49 +08:00
Bobby Wang 178cfe70a8 [pyspark][doc] Test and doc for stage-level scheduling. (#9786) 2023-11-16 18:15:59 +08:00
Bobby Wang 36a552ac98 [jvm-packages] support stage-level scheduling (#9775) 2023-11-14 08:59:45 +08:00
Bobby WangandJiaming Yuan fa65cf6646 [doc] How to configure regarding to stage-level (#9727) 
---------

Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>
 2023-10-31 01:28:34 +08:00
Bobby Wang 1323531323 [pyspark] unify the way for determining whether runs on the GPU. (#9724) 2023-10-27 11:21:30 +08:00
Bobby Wang 4d1607eefd [pyspark] Support stage-level scheduling for training (#9519) 2023-10-17 10:35:39 +08:00
Bobby WangandJiaming Yuan 6c791b5b47 [pyspark] support gpu transform (#9542) 
---------

Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>
 2023-09-07 12:15:50 +08:00
Bobby Wang 419e052314 [pyspark] rework transform to reuse same code (#9292) 2023-09-04 15:57:16 +08:00
Bobby WangandJiaming Yuan 68be454cfa [pyspark] hotfix for GPU setup validation (#9495) 
* [pyspark] fix a bug of validating gpu configuration

---------

Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>
 2023-08-17 16:01:39 +08:00
Bobby WangandJiaming Yuan 344f90b67b [jvm-packages] throw exception when tree_method=approx and device=cuda (#9478) 
---------

Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>
 2023-08-14 17:52:14 +08:00
Bobby Wang d495a180d8 [pyspark] add logs for training (#9449) 2023-08-09 18:32:23 +08:00
Bobby Wang 8f0efb4ab3 [jvm-packages] automatically set the max/min direction for best score (#9404) 2023-07-27 11:09:55 +08:00
Bobby Wang 1b657a5513 [jvm-packages] set device to cuda when tree method is "gpu_hist" (#9412) 2023-07-24 18:32:25 +08:00
Bobby WangandJiaming Yuan 320323f533 [pyspark] add parameters in the ctor of all estimators. (#9202) 
---------

Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>
 2023-05-29 05:58:16 +08:00
Bobby Wang 6274fba0a5 [pyspark] support tying (#9172) 2023-05-19 14:39:26 +08:00
Bobby Wang caf326d508 [pyspark] Refactor and typing support for models (#9156) 2023-05-17 16:38:51 +08:00
Bobby Wang cb370c4f7d [jvm] separate spark.version for cpu and gpu (#9166) 2023-05-17 07:12:20 +08:00
Bobby Wang e922004329 [doc] fix the cudf installation [skip ci] (#9106) 2023-04-28 19:43:58 +08:00
Bobby Wang 17add4776f [pyspark] Don't stack for non feature columns (#9088) 2023-04-25 23:09:12 +08:00
Bobby Wang 339f21e1bf [pyspark] fix a type hint with old pyspark release (#9079) 2023-04-24 20:04:14 +08:00
Bobby Wang d237378452 [jvm-packages] Clean up the dependencies after removing scala versioned tracker (#9078) 2023-04-24 17:49:08 +08:00
Bobby Wang 72ec0c5484 [pyspark] support pred_contribs (#8633) 2023-01-11 16:51:12 +08:00
Bobby Wang 4e12f3e1bc [Breaking][jvm-packages] Bump rapids version to 22.12.0 (#8648) 
* [jvm-packages] Bump rapids version to 22.12.0

This PR bumps spark version to 3.1.1 and the rapids version
to 22.12.0, which results in the latest xgboost can't run
with the old rapids packages.
 2023-01-07 18:59:17 +08:00
Bobby Wang d3ad0524e7 [pyspark] Re-work _fit function (#8630) 2023-01-04 18:21:57 +08:00
Bobby Wang 40a1a2ffa8 [pyspark] check use_qdm across all the workers (#8496) 2022-12-08 18:09:17 +08:00
Bobby Wang f1e9bbcee5 [breakinig] [jvm-packages] change DeviceQuantileDmatrix into QuantileDMatrix (#8461) 2022-12-05 12:23:21 +08:00
Bobby Wang 8e41ad24f5 [pyspark] sort qid for SparkRanker (#8497) 
* [pyspark] sort qid for SparkRandker

* resolve comments
 2022-12-01 16:40:35 -08:00
Bobby Wang 2dde65f807 [ci] reduce pyspark test time (#8324) 2022-11-21 16:58:00 +08:00
Bobby Wang 76f95a6667 [pyspark] Filter out the unsupported train parameters (#8355) 2022-10-18 23:26:02 +08:00
Bobby Wangandfis cbf3a5f918 [pyspark][doc] add more doc for pyspark (#8271) 
Co-authored-by: fis <jm.yuan@outlook.com>
 2022-09-29 11:58:18 +08:00
Bobby Wang c91fed083d [pyspark] disable repartition_random_shuffle by default (#8283) 2022-09-29 10:50:51 +08:00
Bobby Wang 8d247f0d64 [jvm-packages] fix spark-rapids compatibility issue (#8240) 
* [jvm-packages] fix spark-rapids compatibility issue

spark-rapids (from 22.10) has shimmed GpuColumnVector, which means
we can't call it directly. So this PR call the UnshimmedGpuColumnVector
 2022-09-22 23:31:29 +08:00
Bobby WangandHyunsu Cho 4f42aa5f12 [pyspark] make the model saved by pyspark compatible (#8219) 
Co-authored-by: Hyunsu Cho <chohyu01@cs.washington.edu>
 2022-09-20 16:43:49 +08:00
Bobby WangandHyunsu Philip Cho 520586ffa7 [pyspark] fix empty data issue when constructing DMatrix (#8245) 
Co-authored-by: Hyunsu Philip Cho <chohyu01@cs.washington.edu>
 2022-09-20 16:43:20 +08:00
Bobby Wang 7ee10e3dbd [pyspark] Cleanup the comments (#8217) 2022-09-05 16:20:12 +08:00
Bobby Wang 03cc3b359c [pyspark] support a list of feature column names (#8117) 2022-08-08 17:05:27 +08:00
Bobby Wang f801d3cf15 [PySpark] change the returning model type to string from binary (#8085) 
* [PySpark] change the returning model type to string from binary

XGBoost pyspark can be can be accelerated by RAPIDS Accelerator seamlessly by
changing the returning model type from binary to string.
 2022-07-19 18:39:20 +08:00
Bobby Wang a33f35eecf [PySpark] add gpu support for spark local mode (#8068) 2022-07-17 07:59:06 +08:00
Bobby Wang 91bb9e2cb3 [PySpark] fix raw_prediction_col parameter and minor cleanup (#8067) 2022-07-16 17:58:57 +08:00
Bobby Wang a68580e2a7 [jvm-packages] fix executor crashing issue when transforming on xgboost4j-spark-gpu (#8025) 
* [jvm-packages] fix executor crashing issue when transforming on xgboost4j-spark-gpu

the API XGBoosterSetParam is not thread-safe. Dring the phase of transforming,
XGBoost runs several transforming tasks at a time, and each of them will set
the "gpu_id" and "predictor" parameters, so if several tasks (multi-threads)
all XGBoosterSetParam simultaneously, it may cause the memory to be corrupted
and cause SIGSEGV.

This PR first get the booster from broadcast and set to the correct gpu_id
and predictor, and then all transforming taskes will use the same booster to
do the transforming.
 2022-06-24 01:18:41 +08:00
Bobby Wang e44a082620 [jvm-packages] update nccl version to 2.12.12-1 (#8015) 2022-06-21 17:34:09 +08:00
Bobby Wang 78694405a6 [jvm-packages] add jni for setting feature name and type (#7966) 2022-06-03 11:09:48 +08:00
Bobby Wang 545fd4548e [jvm-packages] refactor xgboost read/write (#7956) 
1. Removed the duplicated Default XGBoost read/write which is copied from
  spark 2.3.x
2. Put some utils into util package
 2022-06-01 11:38:49 +08:00
Bobby Wang 5a7dc41351 [doc] update doc for dumping model to be json or ubj for jvm packages (#7955) 2022-05-31 14:43:13 +08:00
Bobby Wang 6275cdc486 [jvm-packages] add format option when saving a model (#7940) 2022-05-30 15:49:59 +08:00
Bobby Wang fbc3d861bb [jvm-packages] remove default parameters (#7938) 2022-05-28 10:31:19 +08:00
Bobby Wang 5ef33adf68 [jvm-packges] set the correct objective if user doesn't explicitly set it (#7781) 2022-05-18 14:05:18 +08:00
Bobby Wang b41cf92dc2 [jvm-packages] move dmatrix building into rabit context for cpu pipeline (#7908) 2022-05-17 14:52:25 +08:00
Bobby Wang 1496789561 [doc] update the doc for jvm model compatibility (#7907) 2022-05-16 14:05:26 +08:00
Bobby Wang 11e46e4bc0 [Breaking][jvm-packages] make classification model be xgboost-compatible (#7896) 2022-05-14 15:43:05 +08:00
Bobby Wang 9fa7ed1743 [Breaking][jvm-packages] remove timeoutRequestWorkers parameter (#7839) 2022-05-13 16:26:25 +08:00
Bobby Wang 1b103e1f5f [CI] make container be able to re-attached (#7848) 
When re-starting the container, it will fail in entrypoint.sh which
will exit when adding an existing group or user
 2022-04-29 19:00:35 -07:00
Bobby Wang a94e1b172e [jvm-packages] Fix model compatibility (#7845) 2022-04-28 02:05:38 +08:00
Bobby Wang 686caad40c [jvm-package] remove the coalesce in barrier mode (#7846) 2022-04-27 23:34:22 +08:00
Bobby Wang bef1f939ce [doc] remove the doc about killing SparkContext [skip ci] (#7840) 2022-04-25 19:29:16 +08:00
Bobby Wang dc2e699656 [Breaking][jvm-packages] Use barrier execution mode (#7836) 
With the introduction of the barrier execution mode. we don't need to kill SparkContext when some xgboost tasks failed. Instead, Spark will handle the errors for us. So in this PR, `killSparkContextOnWorkerFailure` parameter is deleted.
 2022-04-25 17:09:52 +08:00
Bobby Wang 6ece549a90 [doc] update the jvm tutorial to 1.6.1 [skip ci] (#7834) 2022-04-24 14:25:22 +08:00
Bobby Wang c45665a55a [jvm-packages] move the dmatrix building into rabit context (#7823) 
This fixes the QuantileDeviceDMatrix in distributed environment.
 2022-04-23 00:06:50 +08:00
Bobby Wang 2d83b2ad8f [jvm-packages] add hostIp and python exec for rabit tracker (#7808) 2022-04-15 16:28:43 +08:00
Bobby Wang 6f032b7152 [doc] fix a typo in jvm/index.rst (#7806) 2022-04-13 17:02:42 -07:00
Bobby Wang 3f536b5308 [jvm-packages] fix evaluation when featuresCols is used (#7798) 2022-04-13 12:52:50 +08:00
Bobby WangandSameer Raheja 4b00c64d96 [doc] improve xgboost4j-spark-gpu doc [skip ci] (#7793) 
Co-authored-by: Sameer Raheja <sameerz@users.noreply.github.com>
 2022-04-12 12:02:16 +08:00
Bobby Wang 118192f116 [jvm-packages] xgboost4j-spark should work when featuresCols is specified (#7789) 2022-04-08 13:21:04 +08:00
Bobby Wang 729d227b89 [jvm-packages] remove the dep of com.fasterxml.jackson (#7791) 2022-04-08 13:04:34 +08:00
Bobby WangandJiaming Yuan 89d6419fd5 [jvm-packages] add doc for xgboost4j-spark-gpu (#7779) 
Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>
 2022-04-07 11:35:01 +08:00
Bobby Wang 2454407f3a [jvm-packages] unify setFeaturesCol API for XGBoostRegressor (#7784) 2022-04-05 13:35:33 +08:00
Bobby Wang 89aa8ddf52 [jvm-packages] fix the prediction issue for multi:softmax (#7694) 2022-02-24 01:09:45 +08:00
Bobby Wang e3e6de5ed9 [jvm-packages] unify the set features API (#7692) 
xgboost4j-spark provides 2 sets of API for setting features, one for CPU, another for GPU, which may cause confusion.

This PR removes the GPU API and adds an override CPU function setFeaturesCol to accept Array[String] parameters.
 2022-02-23 03:37:25 +08:00
Bobby Wang 131858e7cb [jvm-packages] Do not repartition when nWorker = 1 (#7676) 2022-02-19 21:45:54 +08:00
Bobby Wang e8c1eb99e4 [jvm-package] Clean up the legacy gpu support tests (#7523) 2021-12-21 09:15:51 +08:00
Bobby Wang 24e25802a7 [jvm-packages] Add Rapids plugin support (#7491) 
* Add GPU pre-processing pipeline.
 2021-12-17 13:11:12 +08:00
Bobby Wang 24be04e848 [jvm-packages] Add DeviceQuantileDMatrix to Scala binding (#7459) 2021-11-24 20:23:18 +08:00
Bobby Wang 7cfb310eb4 Rework transform (#7440) 
extract the common part of transform code from XGBoostClassifier
and XGBoostRegressor
 2021-11-18 15:48:57 +08:00
Bobby Wang cb685607b2 [jvm-packages] Rework the train pipeline (#7401) 
1. Add PreXGBoost to build RDD[Watches] from Dataset
2. Feed RDD[Watches] built from PreXGBoost to XGBoost to train
 2021-11-10 17:51:38 +08:00
Bobby Wang b81ebbef62 [jvm-packages] Fix json4s binary compatibility issue (#7376) 
Spark 3.2 depends on 3.7.0-M11 which has changed some implicited functions'
signatures. And it will result the xgboost4j built against spark 3.0/3.1
failed when saving the model.
 2021-10-30 03:20:57 +08:00
Bobby Wang 4fd149b3a2 [jvm-packages] update checkstyle (#7335) 
* [jvm-packages] update scalastyle

1. bump scalastyle-maven-plugin and maven-checkstyle-plugin to latest
2. remove unused imports

* fix code style check
 2021-10-18 18:42:01 +08:00
Bobby Wangandjiamingy 0ee11dac77 [jvm-packages][xgboost4j-gpu] Support GPU dataframe and DeviceQuantileDMatrix (#7195) 
Following classes are added to support dataframe in java binding:

- `Column` is an abstract type for a single column in tabular data.
- `ColumnBatch` is an abstract type for dataframe.

- `CuDFColumn` is an implementaiton of `Column` that consume cuDF column
- `CudfColumnBatch` is an implementation of `ColumnBatch` that consumes cuDF dataframe.

- `DeviceQuantileDMatrix` is the interface for quantized data.

The Java implementation mimics the Python interface and uses `__cuda_array_interface__` protocol for memory indexing.  One difference is on JVM package, the data batch is staged on the host as java iterators cannot be reset.

Co-authored-by: jiamingy <jm.yuan@outlook.com>
 2021-09-24 14:25:00 +08:00
Bobby Wangandfis 2828da3c4c [jvm-packages] Add XGBOOST_RABIT_TRACKER_IP_FOR_TEST to set rabit tracker IP. (#6869) 
* Add `XGBOOST_RABIT_TRACKER_IP_FOR_TEST` to set rabit tracker IP

* change spark and rabit tracker IP to 127.0.0.1on GitHub Action.

Co-authored-by: fis <jm.yuan@outlook.com>
 2021-04-22 02:00:22 +08:00
Bobby Wang 2c684ffd32 [jvm-packages] fix "key not found: train" issue (#6842) 
* [jvm-packages] fix "key not found: train" issue

* fix bug
 2021-04-18 23:28:39 -07:00
Bobby Wang 49c22c23b4 [jvm-packages] fix early stopping doesn't work even without custom_eval setting (#6738) 
* [jvm-packages] fix early stopping doesn't work even without custom_eval setting

* remove debug info

* resolve comment
 2021-03-06 20:19:40 -08:00
Bobby Wang 9d2832a3a3 fix potential TaskFailedListener's callback won't be called (#6612) 
there is possibility that onJobStart of TaskFailedListener won't be called, if
the job is submitted before the other thread adds addSparkListener.

detail can be found at https://github.com/dmlc/xgboost/pull/6019#issuecomment-760937628
 2021-01-21 14:20:32 +08:00
Bobby Wang 00b0ad1293 [Doc] add doc for kill_spark_context_on_worker_failure parameter (#6097) 
* [Doc] add doc for kill_spark_context_on_worker_failure parameter

* resolve comments
 2020-09-09 21:28:44 -07:00
Bobby Wang 0e2d5669f6 [jvm-packages] cancel job instead of killing SparkContext (#6019) 
* cancel job instead of killing SparkContext

This PR changes the default behavior that kills SparkContext. Instead, This PR
cancels jobs when coming across task failed. That means the SparkContext is
still alive even some exceptions happen.

* add a parameter to control if killing SparkContext

* cancel the jobs the failed task belongs to

* remove the jobId from the map when one job failed.

* resolve comments
 2020-09-02 14:20:59 -07:00
Bobby WangandHyunsu Cho 8943eb4314 [BLOCKING] [jvm-packages] add gpu_hist and enable gpu scheduling (#5171) 
* [jvm-packages] add gpu_hist tree method

* change updater hist to grow_quantile_histmaker

* add gpu scheduling

* pass correct parameters to xgboost library

* remove debug info

* add use.cuda for pom

* add CI for gpu_hist for jvm

* add gpu unit tests

* use gpu node to build jvm

* use nvidia-docker

* Add CLI interface to create_jni.py using argparse

Co-authored-by: Hyunsu Cho <chohyu01@cs.washington.edu>
 2020-07-26 21:53:24 -07:00
Bobby WangandPhilip Hyunsu Cho 730866a7bc [CI] update spark version to 3.0.0 (#5890) 
* [CI] update spark version to 3.0.0

* Update Dockerfile.jvm_cross

Co-authored-by: Philip Hyunsu Cho <chohyu01@cs.washington.edu>
 2020-07-16 00:23:44 -07:00
Bobby Wang 9f85e92602 [jvm-packages] update spark dependency to 3.0.0 (#5836) 2020-07-12 20:58:30 -07:00
Bobby Wangandfis ad826e913f [jvm-packages]add feature size for LabelPoint and DataBatch (#5303) 
* fix type error

* Validate number of features.

* resolve comments

* add feature size for LabelPoint and DataBatch

* pass the feature size to native

* move feature size validating tests into a separate suite

* resolve comments

Co-authored-by: fis <jm.yuan@outlook.com>
 2020-04-07 16:49:52 -07:00
Bobby 3e2c472944 Fix model parameter recovery (#4738) 2019-08-07 02:32:10 -04:00