7054 Commits

Author SHA1 Message Date
Jiaming Yuan
34a2616696
[jvm-packages] Update dependencies. (#9809)
- scalatest: 3.2.17
- maven-checkstyle-plugin: 3.3.1
- maven-surefire-plugin: 3.2.2
- maven-project-info-reports-plugin: 3.5.0
- maven-javadoc-plugin: 3.6.2
2023-11-27 20:09:25 +08:00
Jiaming Yuan
e9f149481e
[sklearn] Fix loading model attributes. (#9808) 2023-11-27 17:19:01 +08:00
Jiaming Yuan
3f4e22015a
Mark NCCL python test optional. (#9804)
Skip the tests if XGBoost is not compiled with dlopen.
2023-11-25 11:25:47 +08:00
Jiaming Yuan
8fe1a2213c
Cleanup code for distributed training. (#9805)
* Cleanup code for distributed training.

- Merge `GetNcclResult` into nccl stub.
- Split up utilities from the main dask module.
- Let Channel return `Result` to accommodate nccl channel.
- Remove old `use_label_encoder` parameter.
2023-11-25 09:10:56 +08:00
Jiaming Yuan
e9260de3f3
[breaking] Remove dense libsvm parser plugin. (#9799) 2023-11-23 00:12:39 +08:00
Jiaming Yuan
1877cb8e83
Change default metric for gamma regression to deviance. (#9757)
* Change default metric for gamma regression to deviance.

- Cleanup the gamma implementation.
- Use deviance instead since the objective is derived from deviance.
2023-11-22 21:17:48 +08:00
Jiaming Yuan
0715ab3c10
Use dlopen to load NCCL. (#9796)
This PR adds optional support for loading nccl with `dlopen` as an alternative of compile time linking. This is to address the size bloat issue with the PyPI binary release.
- Add CMake option to load `nccl` at runtime.
- Add an NCCL stub.

After this, `nccl` will be fetched from PyPI when using pip to install XGBoost, either by a user or by `pyproject.toml`. Others who want to link the nccl at compile time can continue to do so without any change.

At the moment, this is Linux only since we only support MNMG on Linux.
2023-11-22 19:27:31 +08:00
Jiaming Yuan
fedd9674c8
Implement column sampler in CUDA. (#9785)
- CUDA implementation.
- Extract the broadcasting logic, we will need the context parameter after revamping the collective implementation.
- Some changes to the event loop for fixing a deadlock in CI.
- Move argsort into algorithms.cuh, add support for cuda stream.
2023-11-17 04:29:08 +08:00
Bobby Wang
178cfe70a8
[pyspark][doc] Test and doc for stage-level scheduling. (#9786) 2023-11-16 18:15:59 +08:00
Jiaming Yuan
ada377c57e
[coll] Reduce the scope of lock in the event loop. (#9784) 2023-11-15 14:16:19 +08:00
Bobby Wang
36a552ac98
[jvm-packages] support stage-level scheduling (#9775) 2023-11-14 08:59:45 +08:00
Ken Geis
162da7b52b
fix typo in Parameters doc (#9781) 2023-11-13 03:09:06 +08:00
Jiaming Yuan
6fd4a30667
[coll] Increase timeout for allgather test. (#9777) 2023-11-09 05:26:40 +08:00
Jiaming Yuan
44099f585d
[coll] Add C API for the tracker. (#9773) 2023-11-08 18:17:14 +08:00
Jiaming Yuan
06bdc15e9b
[coll] Pass context to various functions. (#9772)
* [coll] Pass context to various functions.

In the future, the `Context` object would be required for collective operations, this PR
passes the context object to some required functions to prepare for swapping out the
implementation.
2023-11-08 09:54:05 +08:00
Jiaming Yuan
6c0a190f6d
[coll] Add comm group. (#9759)
- Implement `CommGroup` for double dispatching.
- Small cleanup to tracker for handling abort.
2023-11-07 11:12:31 +08:00
Jiaming Yuan
c3a0622b49
Fix using categorical data with the score function of ranker. (#9753) 2023-11-07 07:29:11 +08:00
Jiaming Yuan
82828621d0
[doc] Add doc for linters and simplify c++ lint script. (#9750) 2023-11-07 05:03:30 +08:00
Jiaming Yuan
98238d63fa
[dask] Change document to avoid using default import. (#9742)
This aligns dask with pyspark, users need to explicitly call:

```
from xgboost.dask import DaskXGBClassifier
from xgboost import dask as dxgb
```

In future releases, we might stop using the default import and remove the lazy loader.
2023-11-07 02:44:39 +08:00
Bobby Wang
093b675838
[Doc] update the tutorial of xgboost4j-spark-gpu (#9752)
---------

Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>
2023-11-03 18:19:28 +08:00
Hui Liu
c81731308c fix RCCL 2023-11-02 16:39:24 -07:00
Hui Liu
51efb7442e support HIP for half in coll 2023-11-02 10:53:12 -07:00
Hui Liu
3af5dfd546 Merge branch 'master' 2023-11-02 09:05:31 -07:00
david-cortes
be20df8c23
[Python] Accept numpy generators as random_state (#9743)
* accept numpy generators for random_state

* make linter happy

* fix tests
2023-11-01 16:20:44 -07:00
Jiaming Yuan
4da4e092b5
[coll] Improvements and fixes for tracker and allreduce. (#9745)
- Allow the tracker to wait.
- Fix allreduce type cast
- Return args from the federated tracker.
2023-11-02 04:06:46 +08:00
Philip Hyunsu Cho
0ff8572737
[CI] Build libxgboost4j.dylib with CMAKE_OSX_DEPLOYMENT_TARGET (#9749) 2023-11-01 11:20:28 -07:00
Philip Hyunsu Cho
1b9ed4a4a1
[CI] Improve CI for Mac M1 (#9748)
* [CI] Improve CI for Mac M1

* Add -v flag

* Disable OpenMP in libxgboost4j.dylib

* Target MacOS 10.15+ to use C++17
2023-11-01 10:03:56 -07:00
Hui Liu
129bb76941 enable federated 2023-10-31 16:31:56 -07:00
Hui Liu
123af45327 Merge branch 'master' 2023-10-31 15:59:31 -07:00
david-cortes
d3f0646779
[R] Avoid modifying importance dt in-place, fix aggregation (#9740) 2023-11-01 05:10:59 +08:00
Jiaming Yuan
bc995a4865
[coll] Add federated coll. (#9738)
- Define a new data type, the proto file is copied for now.
- Merge client and communicator into `FederatedColl`.
- Define CUDA variant.
- Migrate tests for CPU, add tests for CUDA.
2023-11-01 04:06:46 +08:00
Hui Liu
6ac806fefd Merge branch 'master' 2023-10-31 09:05:56 -07:00
Philip Hyunsu Cho
6b98305db4
[CI] Enable gmock in gtest (#9737) 2023-10-31 20:09:35 +08:00
Hui Liu
8fab17ae8f rm hip.h files 2023-10-30 21:20:28 -07:00
Hui Liu
9b7aa1a7cd unify cuda to hip 2023-10-30 17:12:06 -07:00
Hui Liu
4eb371b3f0 unify cuda to hip 2023-10-30 17:10:06 -07:00
Hui Liu
6df27eadc9 rm hip_category from source 2023-10-30 16:34:49 -07:00
Hui Liu
02f5464fa6 enable coll and comm 2023-10-30 15:15:05 -07:00
Hui Liu
b6b5218245 enable RCCL 2023-10-30 14:05:04 -07:00
Hui Liu
d7f1235b7d Merge branch 'master' into sync-condition-2023Oct11 2023-10-30 13:19:33 -07:00
Hui Liu
1bedd76e94 rm un-necessary code 2023-10-30 13:14:45 -07:00
Hui Liu
40dc263602 enable ROCm for jvm and R 2023-10-30 12:52:44 -07:00
Jiaming Yuan
80390e6cb6
[coll] Federated comm. (#9732) 2023-10-31 02:39:55 +08:00
Bobby Wang
fa65cf6646
[doc] How to configure regarding to stage-level (#9727)
---------

Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>
2023-10-31 01:28:34 +08:00
omahs
2cfc90e8db
Fix typos (#9731) 2023-10-30 16:52:12 +08:00
Jiaming Yuan
6755179e77
[coll] Add nccl. (#9726) 2023-10-28 16:33:58 +08:00
James Lamb
0c621094b3
[CI] enforce cmakelint checks (#9728) 2023-10-28 05:38:04 +08:00
Hui Liu
32ae49ab92 temp hack for multi GPUs 2023-10-27 13:00:49 -07:00
Hui Liu
6bbca9a8b7 restore learner 2023-10-27 11:15:06 -07:00
Hui Liu
6762230d9a namespace to reduce code 2023-10-27 10:51:32 -07:00