[CI] Set up pipeline to build manylinux2014 wheels (#10478)

doc/changes/v2.1.0.rst

@@ -34,6 +34,27 @@ NCCL is now fetched from PyPI
 
 In the previous version, XGBoost statically linked NCCL, which significantly increased the binary size and led to hitting the PyPI repository limit. With the new release, we have made a significant improvement. The new release can now dynamically load NCCL from an external source, reducing the binary size. For the PyPI package, the ``nvidia-nccl-cu12`` package will be fetched during installation. With more downstream packages reusing NCCL, we expect the user environments to be slimmer in the future as well. (#9796, #9804, #10447)
 
+***************************************************
+Parts of the Python package now require glibc 2.28+
+***************************************************
+Starting from 2.1.0, XGBoost Python package will be distributed in two variants:
+
+* ``manylinux_2_28``: for recent Linux distros with glibc 2.28 or newer. This variant comes with all features enabled.
+* ``manylinux2014``: for old Linux distros with glibc older than 2.28. This variant does not support GPU algorithms or federated learning.
+
+The ``pip`` package manager will automatically choose the correct variant depending on your system.
+
+Starting from **May 31, 2025**, we will stop distributing the ``manylinux2014`` variant and exclusively
+distribute the ``manylinux_2_28`` variant. We made this decision so that our CI/CD pipeline won't have
+depend on software components that reached end-of-life (such as CentOS 7). We strongly encourage
+everyone to migrate to recent Linux distros in order to use future versions of XGBoost.
+
+Note. If you want to use GPU algorithms or federated learning on an older Linux distro, you have
+two alternatives:
+
+1. Upgrade to a recent Linux distro with glibc 2.28+.  OR
+2. Build XGBoost from the source.
+
 ************
 Multi-output
 ************
@@ -98,7 +119,7 @@ This section lists some new features that are general to all language bindings.
 - Change default metric for gamma regression to ``deviance``. (#9757)
 - Normalization for learning to rank is now optional with the introduction of the new ``lambdarank_normalization`` parameter. (#10094)
 - Contribution prediction with ``QuantileDMatrix`` on CPU. (#10043)
-- XGBoost on macos no longer bundles OpenMP runtime. Users can install the latest runtime from their dependency manager of choice. (#10440). Along with which, JVM packages on MacoOS are not built with OpenMP support (#10449).
+- XGBoost on macos no longer bundles OpenMP runtime. Users can install the latest runtime from their dependency manager of choice. (#10440). Along with which, JVM packages on MacoOS are now built with OpenMP support (#10449).
 
 *********
 Bug fixes

doc/install.rst

@@ -27,6 +27,26 @@ Pre-built binary wheels are uploaded to PyPI (Python Package Index) for each rel
 You might need to run the command with ``--user`` flag or use ``virtualenv`` if you run
 into permission errors.
 
+.. note:: Parts of the Python package now require glibc 2.28+
+
+  Starting from 2.1.0, XGBoost Python package will be distributed in two variants:
+
+  * ``manylinux_2_28``: for recent Linux distros with glibc 2.28 or newer. This variant comes with all features enabled.
+  * ``manylinux2014``: for old Linux distros with glibc older than 2.28. This variant does not support GPU algorithms or federated learning.
+
+  The ``pip`` package manager will automatically choose the correct variant depending on your system.
+
+  Starting from **May 31, 2025**, we will stop distributing the ``manylinux2014`` variant and exclusively
+  distribute the ``manylinux_2_28`` variant. We made this decision so that our CI/CD pipeline won't have
+  depend on software components that reached end-of-life (such as CentOS 7). We strongly encourage
+  everyone to migrate to recent Linux distros in order to use future versions of XGBoost.
+
+  Note. If you want to use GPU algorithms or federated learning on an older Linux distro, you have
+  two alternatives:
+
+  1. Upgrade to a recent Linux distro with glibc 2.28+.  OR
+  2. Build XGBoost from the source.
+
 .. note:: Windows users need to install Visual C++ Redistributable
 
   XGBoost requires DLLs from `Visual C++ Redistributable

tests/buildkite/build-cuda-with-rmm.sh

@@ -42,7 +42,7 @@ $command_wrapper python tests/ci_build/rename_whl.py  \
   --platform-tag ${WHEEL_TAG}
 
 echo "--- Audit binary wheel to ensure it's compliant with ${WHEEL_TAG} standard"
-tests/ci_build/ci_build.sh auditwheel_x86_64 auditwheel repair \
+tests/ci_build/ci_build.sh manylinux_2_28_x86_64 auditwheel repair \
   --plat ${WHEEL_TAG} python-package/dist/*.whl
 $command_wrapper python tests/ci_build/rename_whl.py  \
   --wheel-path wheelhouse/*.whl  \
@@ -50,7 +50,7 @@ $command_wrapper python tests/ci_build/rename_whl.py  \
   --platform-tag ${WHEEL_TAG}
 mv -v wheelhouse/*.whl python-package/dist/
 # Make sure that libgomp.so is vendored in the wheel
-tests/ci_build/ci_build.sh auditwheel_x86_64 bash -c \
+tests/ci_build/ci_build.sh manylinux_2_28_x86_64 bash -c \
   "unzip -l python-package/dist/*.whl | grep libgomp  || exit -1"
 
 echo "--- Upload Python wheel"

tests/buildkite/build-cuda.sh

@@ -41,7 +41,7 @@ $command_wrapper python tests/ci_build/rename_whl.py  \
   --platform-tag ${WHEEL_TAG}
 
 echo "--- Audit binary wheel to ensure it's compliant with ${WHEEL_TAG} standard"
-tests/ci_build/ci_build.sh auditwheel_x86_64 auditwheel repair \
+tests/ci_build/ci_build.sh manylinux_2_28_x86_64 auditwheel repair \
   --plat ${WHEEL_TAG} python-package/dist/*.whl
 $command_wrapper python tests/ci_build/rename_whl.py  \
   --wheel-path wheelhouse/*.whl  \
@@ -49,7 +49,7 @@ $command_wrapper python tests/ci_build/rename_whl.py  \
   --platform-tag ${WHEEL_TAG}
 mv -v wheelhouse/*.whl python-package/dist/
 # Make sure that libgomp.so is vendored in the wheel
-tests/ci_build/ci_build.sh auditwheel_x86_64 bash -c \
+tests/ci_build/ci_build.sh manylinux_2_28_x86_64 bash -c \
   "unzip -l python-package/dist/*.whl | grep libgomp  || exit -1"
 
 echo "--- Upload Python wheel"

tests/buildkite/build-manylinux2014-aarch64.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+set -euo pipefail
+
+source tests/buildkite/conftest.sh
+
+WHEEL_TAG=manylinux2014_aarch64
+command_wrapper="tests/ci_build/ci_build.sh manylinux2014_aarch64"
+python_bin="/opt/python/cp310-cp310/bin/python"
+
+echo "--- Build binary wheel for ${WHEEL_TAG}"
+# Patch to add warning about manylinux2014 variant
+patch -p0 < tests/buildkite/manylinux2014_warning.patch
+$command_wrapper bash -c \
+  "cd python-package && ${python_bin} -m pip wheel --no-deps -vvv . --wheel-dir dist/"
+git checkout python-package/xgboost/core.py  # discard the patch
+
+$command_wrapper auditwheel repair --plat ${WHEEL_TAG} python-package/dist/*.whl
+$command_wrapper ${python_bin} tests/ci_build/rename_whl.py  \
+  --wheel-path wheelhouse/*.whl  \
+  --commit-hash ${BUILDKITE_COMMIT}  \
+  --platform-tag ${WHEEL_TAG}
+rm -rf python-package/dist/
+mkdir python-package/dist/
+mv -v wheelhouse/*.whl python-package/dist/
+
+echo "--- Upload Python wheel"
+buildkite-agent artifact upload python-package/dist/*.whl
+$command_wrapper bash -c "${python_bin} -m pip install awscli"
+if [[ ($is_pull_request == 0) && ($is_release_branch == 1) ]]
+then
+  ${python_bin} -m awscli s3 \
+    cp python-package/dist/*.whl s3://xgboost-nightly-builds/${BRANCH_NAME}/ \
+    --acl public-read --no-progress
+fi

tests/buildkite/build-manylinux2014-x86_64.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+set -euo pipefail
+
+source tests/buildkite/conftest.sh
+
+WHEEL_TAG=manylinux2014_x86_64
+command_wrapper="tests/ci_build/ci_build.sh manylinux2014_x86_64"
+python_bin="/opt/python/cp310-cp310/bin/python"
+
+echo "--- Build binary wheel for ${WHEEL_TAG}"
+# Patch to add warning about manylinux2014 variant
+patch -p0 < tests/buildkite/manylinux2014_warning.patch
+$command_wrapper bash -c \
+  "cd python-package && ${python_bin} -m pip wheel --no-deps -vvv . --wheel-dir dist/"
+git checkout python-package/xgboost/core.py  # discard the patch
+
+$command_wrapper auditwheel repair --plat ${WHEEL_TAG} python-package/dist/*.whl
+$command_wrapper ${python_bin} tests/ci_build/rename_whl.py  \
+  --wheel-path wheelhouse/*.whl  \
+  --commit-hash ${BUILDKITE_COMMIT}  \
+  --platform-tag ${WHEEL_TAG}
+rm -rf python-package/dist/
+mkdir python-package/dist/
+mv -v wheelhouse/*.whl python-package/dist/
+
+echo "--- Upload Python wheel"
+buildkite-agent artifact upload python-package/dist/*.whl
+$command_wrapper bash -c "${python_bin} -m pip install awscli"
+if [[ ($is_pull_request == 0) && ($is_release_branch == 1) ]]
+then
+  ${python_bin} -m awscli s3 \
+    cp python-package/dist/*.whl s3://xgboost-nightly-builds/${BRANCH_NAME}/ \
+    --acl public-read --no-progress
+fi

tests/buildkite/manylinux2014_warning.patch

@@ -0,0 +1,37 @@
+diff --git python-package/pyproject.toml python-package/pyproject.toml
+index a273d8c13..dee49686a 100644
+--- python-package/pyproject.toml
++++ python-package/pyproject.toml
+@@ -30,8 +30,7 @@ classifiers = [
+ ]
+ dependencies = [
+     "numpy",
+-    "scipy",
+-    "nvidia-nccl-cu12 ; platform_system == 'Linux' and platform_machine != 'aarch64'"
++    "scipy"
+ ]
+ 
+ [project.urls]
+diff --git python-package/xgboost/core.py python-package/xgboost/core.py
+index e8bc735e6..030972ef2 100644
+--- python-package/xgboost/core.py
++++ python-package/xgboost/core.py
+@@ -262,6 +262,18 @@ Likely cause:
+         )
+         raise ValueError(msg)
+ 
++    warnings.warn(
++        "Your system has an old version of glibc (< 2.28). We will stop supporting "
++        "Linux distros with glibc older than 2.28 after **May 31, 2025**. "
++        "Please upgrade to a recent Linux distro (with glibc 2.28+) to use "
++        "future versions of XGBoost.\n"
++        "Note: You have installed the 'manylinux2014' variant of XGBoost. Certain "
++        "features such as GPU algorithms or federated learning are not available. "
++        "To use these features, please upgrade to a recent Linux distro with glibc "
++        "2.28+, and install the 'manylinux_2_28' variant.",
++        FutureWarning
++    )
++
+     return lib
+ 
+ 

tests/buildkite/pipeline.yml

@@ -26,18 +26,17 @@ steps:
     key: run-clang-tidy
     agents:
       queue: linux-amd64-cpu
-  - wait
   - label: ":console: Build CPU"
     command: "tests/buildkite/build-cpu.sh"
     key: build-cpu
     agents:
       queue: linux-amd64-cpu
-  - label: ":console: Build CPU ARM64"
+  - label: ":console: Build CPU ARM64 + manylinux_2_28_aarch64 wheel"
     command: "tests/buildkite/build-cpu-arm64.sh"
     key: build-cpu-arm64
     agents:
       queue: linux-arm64-cpu
-  - label: ":console: Build CUDA"
+  - label: ":console: Build CUDA + manylinux_2_28_x86_64 wheel"
     command: "tests/buildkite/build-cuda.sh"
     key: build-cuda
     agents:
@@ -62,6 +61,16 @@ steps:
     key: build-jvm-doc
     agents:
       queue: linux-amd64-cpu
+  - label: ":console: Build manylinux2014_x86_64 wheel"
+    command: "tests/buildkite/build-manylinux2014-x86_64.sh"
+    key: build-manylinux2014-x86_64
+    agents:
+      queue: linux-amd64-cpu
+  - label: ":console: Build manylinux2014_aarch64 wheel"
+    command: "tests/buildkite/build-manylinux2014-aarch64.sh"
+    key: build-manylinux2014-aarch64
+    agents:
+      queue: linux-arm64-cpu
   - wait
   #### -------- TEST --------
   - label: ":console: Test Python package, CPU"

tests/ci_build/Dockerfile.manylinux2014_aarch64

@@ -0,0 +1,15 @@
+FROM quay.io/pypa/manylinux2014_aarch64
+
+# Install lightweight sudo (not bound to TTY)
+ENV GOSU_VERSION 1.10
+RUN set -ex; \
+    curl -o /usr/local/bin/gosu -L "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-arm64" && \
+    chmod +x /usr/local/bin/gosu && \
+    gosu nobody true
+
+# Default entry-point to use if running locally
+# It will preserve attributes of created files
+COPY entrypoint.sh /scripts/
+
+WORKDIR /workspace
+ENTRYPOINT ["/scripts/entrypoint.sh"]

tests/ci_build/Dockerfile.manylinux2014_x86_64

@@ -0,0 +1,15 @@
+FROM quay.io/pypa/manylinux2014_x86_64
+
+# Install lightweight sudo (not bound to TTY)
+ENV GOSU_VERSION 1.10
+RUN set -ex; \
+    curl -o /usr/local/bin/gosu -L "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-amd64" && \
+    chmod +x /usr/local/bin/gosu && \
+    gosu nobody true
+
+# Default entry-point to use if running locally
+# It will preserve attributes of created files
+COPY entrypoint.sh /scripts/
+
+WORKDIR /workspace
+ENTRYPOINT ["/scripts/entrypoint.sh"]