92b7577c62
* Initial commit to support multi-node multi-gpu xgboost using dask * Fixed NCCL initialization by not ignoring the opg parameter. - it now crashes on NCCL initialization, but at least we're attempting it properly * At the root node, perform a rabit::Allreduce to get initial sum_gradient across workers * Synchronizing in a couple of more places. - now the workers don't go down, but just hang - no more "wild" values of gradients - probably needs syncing in more places * Added another missing max-allreduce operation inside BuildHistLeftRight * Removed unnecessary collective operations. * Simplified rabit::Allreduce() sync of gradient sums. * Removed unnecessary rabit syncs around ncclAllReduce. - this improves performance _significantly_ (7x faster for overall training, 20x faster for xgboost proper) * pulling in latest xgboost * removing changes to updater_quantile_hist.cc * changing use_nccl_opg initialization, removing unnecessary if statements * added definition for opaque ncclUniqueId struct to properly encapsulate GetUniqueId * placing struct defintion in guard to avoid duplicate code errors * addressing linting errors * removing * removing additional arguments to AllReduer initialization * removing distributed flag * making comm init symmetric * removing distributed flag * changing ncclCommInit to support multiple modalities * fix indenting * updating ncclCommInitRank block with necessary group calls * fix indenting * adding print statement, and updating accessor in vector * improving print statement to end-line * generalizing nccl_rank construction using rabit * assume device_ordinals is the same for every node * test, assume device_ordinals is identical for all nodes * test, assume device_ordinals is unique for all nodes * changing names of offset variable to be more descriptive, editing indenting * wrapping ncclUniqueId GetUniqueId() and aesthetic changes * adding synchronization, and tests for distributed * adding to tests * fixing broken #endif * fixing initialization of gpu histograms, correcting errors in tests * adding to contributors list * adding distributed tests to jenkins * fixing bad path in distributed test * debugging * adding kubernetes for distributed tests * adding proper import for OrderedDict * adding urllib3==1.22 to address ordered_dict import error * added sleep to allow workers to save their models for comparison * adding name to GPU contributors under docs
54 lines
2.1 KiB
Docker
54 lines
2.1 KiB
Docker
ARG CUDA_VERSION
|
|
FROM nvidia/cuda:$CUDA_VERSION-devel-centos6
|
|
|
|
# Environment
|
|
ENV DEBIAN_FRONTEND noninteractive
|
|
|
|
# Install all basic requirements
|
|
RUN \
|
|
yum -y update && \
|
|
yum install -y tar unzip wget xz git && \
|
|
wget http://people.centos.org/tru/devtools-2/devtools-2.repo -O /etc/yum.repos.d/devtools-2.repo && \
|
|
yum install -y devtoolset-2-gcc devtoolset-2-binutils devtoolset-2-gcc-c++ && \
|
|
# Python
|
|
wget https://repo.continuum.io/miniconda/Miniconda2-4.3.27-Linux-x86_64.sh && \
|
|
bash Miniconda2-4.3.27-Linux-x86_64.sh -b -p /opt/python && \
|
|
# CMake
|
|
wget -nv -nc https://cmake.org/files/v3.5/cmake-3.5.2-Linux-x86_64.sh --no-check-certificate && \
|
|
bash cmake-3.5.2-Linux-x86_64.sh --skip-license --prefix=/usr
|
|
|
|
# NCCL2 (License: https://docs.nvidia.com/deeplearning/sdk/nccl-sla/index.html)
|
|
RUN \
|
|
export CUDA_SHORT=`echo $CUDA_VERSION | egrep -o '[0-9]+\.[0-9]'` && \
|
|
if [ "${CUDA_SHORT}" != "10.0" ]; then \
|
|
wget https://developer.download.nvidia.com/compute/redist/nccl/v2.2/nccl_2.2.13-1%2Bcuda${CUDA_SHORT}_x86_64.txz && \
|
|
tar xf "nccl_2.2.13-1+cuda${CUDA_SHORT}_x86_64.txz" && \
|
|
cp nccl_2.2.13-1+cuda${CUDA_SHORT}_x86_64/include/nccl.h /usr/include && \
|
|
cp nccl_2.2.13-1+cuda${CUDA_SHORT}_x86_64/lib/* /usr/lib && \
|
|
rm -f nccl_2.2.13-1+cuda${CUDA_SHORT}_x86_64.txz && \
|
|
rm -r nccl_2.2.13-1+cuda${CUDA_SHORT}_x86_64; fi
|
|
|
|
ENV PATH=/opt/python/bin:$PATH
|
|
ENV CC=/opt/rh/devtoolset-2/root/usr/bin/gcc
|
|
ENV CXX=/opt/rh/devtoolset-2/root/usr/bin/c++
|
|
ENV CPP=/opt/rh/devtoolset-2/root/usr/bin/cpp
|
|
|
|
# Install Python packages
|
|
RUN \
|
|
pip install numpy pytest scipy scikit-learn wheel kubernetes urllib3==1.22
|
|
|
|
ENV GOSU_VERSION 1.10
|
|
|
|
# Install lightweight sudo (not bound to TTY)
|
|
RUN set -ex; \
|
|
wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-amd64" && \
|
|
chmod +x /usr/local/bin/gosu && \
|
|
gosu nobody true
|
|
|
|
# Default entry-point to use if running locally
|
|
# It will preserve attributes of created files
|
|
COPY entrypoint.sh /scripts/
|
|
|
|
WORKDIR /workspace
|
|
ENTRYPOINT ["/scripts/entrypoint.sh"]
|