From 7354955cbb04f493d77c3fa72dad74302d3e9876 Mon Sep 17 00:00:00 2001 From: Jiaming Yuan Date: Wed, 29 May 2024 02:28:14 +0800 Subject: [PATCH] Test federated plugin using GitHub action. (#10336) Co-authored-by: Philip Hyunsu Cho --- .github/workflows/main.yml | 3 ++- doc/build.rst | 26 ++++++++++++++++++++------ plugin/federated/README.md | 23 +++-------------------- tests/ci_build/conda_env/cpp_test.yml | 2 ++ 4 files changed, 27 insertions(+), 27 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 4be35e9de..621c8b465 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -156,8 +156,9 @@ jobs: - name: Build and install XGBoost shared library run: | cd build - cmake .. -DBUILD_STATIC_LIB=OFF -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -GNinja + cmake .. -DBUILD_STATIC_LIB=OFF -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -GNinja -DPLUGIN_FEDERATED=ON -DGOOGLE_TEST=ON ninja -v install + ./testxgboost cd - - name: Build and run C API demo with shared run: | diff --git a/doc/build.rst b/doc/build.rst index cba75ff57..4a2d47c0f 100644 --- a/doc/build.rst +++ b/doc/build.rst @@ -134,7 +134,7 @@ From the command line on Linux starting from the XGBoost directory: .. note:: Specifying compute capability - To speed up compilation, the compute version specific to your GPU could be passed to cmake as, e.g., ``-DGPU_COMPUTE_VER=50``. A quick explanation and numbers for some architectures can be found `in this page `_. + To speed up compilation, the compute version specific to your GPU could be passed to cmake as, e.g., ``-DCMAKE_CUDA_ARCHITECTURES=75``. A quick explanation and numbers for some architectures can be found `in this page `_. .. note:: Faster distributed GPU training with NCCL @@ -147,6 +147,8 @@ From the command line on Linux starting from the XGBoost directory: cmake .. -DUSE_CUDA=ON -DUSE_NCCL=ON -DNCCL_ROOT=/path/to/nccl2 make -j4 +Some additional flags are available for NCCL, ``BUILD_WITH_SHARED_NCCL`` enables building XGBoost with NCCL as a shared library, while ``USE_DLOPEN_NCCL`` enables XGBoost to load NCCL at runtime using ``dlopen``. + On Windows, run CMake as follows: .. code-block:: bash @@ -165,6 +167,17 @@ The above cmake configuration run will create an ``xgboost.sln`` solution file i To speed up compilation, run multiple jobs in parallel by appending option ``-- /MP``. +Federated Learning +================== + +The federated learning plugin requires ``grpc`` and ``protobuf``. To install grpc, refer +to the `installation guide from the gRPC website +`_. Alternatively, one can use the +``libgrpc`` and the ``protobuf`` package from conda forge if conda is available. After +obtaining the required dependencies, enable the flag: `-DPLUGIN_FEDERATED=ON` when running +CMake. Please note that only Linux is supported for the federated plugin. + + .. _build_python: *********************************** @@ -228,11 +241,12 @@ There are several ways to build and install the package from source: 3. Editable installation - To further enable rapid development and iteration, we provide an **editable installation**. - In an editable installation, the installed package is simply a symbolic link to your - working copy of the XGBoost source code. So every changes you make to your source - directory will be immediately visible to the Python interpreter. Here is how to - install XGBoost as editable installation: + To further enable rapid development and iteration, we provide an **editable + installation**. In an editable installation, the installed package is simply a symbolic + link to your working copy of the XGBoost source code. So every changes you make to your + source directory will be immediately visible to the Python interpreter. To install + XGBoost as editable installation, first build the shared library as previously + described, then install the Python package: .. code-block:: bash diff --git a/plugin/federated/README.md b/plugin/federated/README.md index 631c44cee..1040c933e 100644 --- a/plugin/federated/README.md +++ b/plugin/federated/README.md @@ -1,33 +1,16 @@ XGBoost Plugin for Federated Learning ===================================== -This folder contains the plugin for federated learning. Follow these steps to build and test it. +This folder contains the plugin for federated learning. -Install gRPC ------------- -Refer to the [installation guide from the gRPC website](https://grpc.io/docs/languages/cpp/quickstart/). +See [build instruction](../../doc/build.rst) for how to build the plugin. -Build the Plugin ----------------- -```shell -# Under xgboost source tree. -mkdir build -cd build -cmake .. -GNinja \ - -DPLUGIN_FEDERATED=ON \ - -DUSE_CUDA=ON\ - -DUSE_NCCL=ON -ninja -cd ../python-package -pip install -e . -``` -If CMake fails to locate gRPC, you may need to pass `-DCMAKE_PREFIX_PATH=` to CMake. Test Federated XGBoost ---------------------- ```shell # Under xgboost source tree. -cd tests/distributed +cd tests/distributed/test_federated # This tests both CPU training (`hist`) and GPU training (`gpu_hist`). ./runtests-federated.sh ``` diff --git a/tests/ci_build/conda_env/cpp_test.yml b/tests/ci_build/conda_env/cpp_test.yml index d08b904d8..99a4caef1 100644 --- a/tests/ci_build/conda_env/cpp_test.yml +++ b/tests/ci_build/conda_env/cpp_test.yml @@ -8,3 +8,5 @@ dependencies: - c-compiler - cxx-compiler - gtest +- protobuf +- libgrpc