[Breaking] Remove rabit support for custom reductions and grow_local_histmaker updater (#7992)

amalgamation/xgboost-all0.cc

@@ -55,7 +55,6 @@
 #include "../src/tree/tree_updater.cc"
 #include "../src/tree/updater_approx.cc"
 #include "../src/tree/updater_colmaker.cc"
-#include "../src/tree/updater_histmaker.cc"
 #include "../src/tree/updater_prune.cc"
 #include "../src/tree/updater_quantile_hist.cc"
 #include "../src/tree/updater_refresh.cc"

doc/parameter.rst

@@ -151,15 +151,6 @@ Parameters for Tree Booster
     - ``hist``: Faster histogram optimized approximate greedy algorithm.
     - ``gpu_hist``: GPU implementation of ``hist`` algorithm.
 
-* ``sketch_eps`` [default=0.03]
-
-  - Only used for ``updater=grow_local_histmaker``.
-  - This roughly translates into ``O(1 / sketch_eps)`` number of bins.
-    Compared to directly select number of bins, this comes with theoretical guarantee with sketch accuracy.
-  - Usually user does not have to tune this.
-    But consider setting to a lower number for more accurate enumeration of split candidates.
-  - range: (0, 1)
-
 * ``scale_pos_weight`` [default=1]
 
   - Control the balance of positive and negative weights, useful for unbalanced classes. A typical value to consider: ``sum(negative instances) / sum(positive instances)``. See :doc:`Parameters Tuning </tutorials/param_tuning>` for more discussion. Also, see Higgs Kaggle competition demo for examples: `R <https://github.com/dmlc/xgboost/blob/master/demo/kaggle-higgs/higgs-train.R>`_, `py1 <https://github.com/dmlc/xgboost/blob/master/demo/kaggle-higgs/higgs-numpy.py>`_, `py2 <https://github.com/dmlc/xgboost/blob/master/demo/kaggle-higgs/higgs-cv.py>`_, `py3 <https://github.com/dmlc/xgboost/blob/master/demo/guide-python/cross_validation.py>`_.
@@ -170,7 +161,6 @@ Parameters for Tree Booster
 
     - ``grow_colmaker``: non-distributed column-based construction of trees.
     - ``grow_histmaker``: distributed tree construction with row-based data splitting based on global proposal of histogram counting.
-    - ``grow_local_histmaker``: based on local histogram counting.
     - ``grow_quantile_histmaker``: Grow tree using quantized histogram.
     - ``grow_gpu_hist``: Grow tree with GPU.
     - ``sync``: synchronizes trees in all distributed nodes.

doc/treemethod.rst

@@ -5,7 +5,7 @@ Tree Methods
 For training boosted tree models, there are 2 parameters used for choosing algorithms,
 namely ``updater`` and ``tree_method``.  XGBoost has 4 builtin tree methods, namely
 ``exact``, ``approx``, ``hist`` and ``gpu_hist``.  Along with these tree methods, there
-are also some free standing updaters including ``grow_local_histmaker``, ``refresh``,
+are also some free standing updaters including ``refresh``,
 ``prune`` and ``sync``.  The parameter ``updater`` is more primitive than ``tree_method``
 as the latter is just a pre-configuration of the former.  The difference is mostly due to
 historical reasons that each updater requires some specific configurations and might has
@@ -37,27 +37,18 @@ approximated training algorithms.  These algorithms build a gradient histogram f
 node and iterate through the histogram instead of real dataset.  Here we introduce the
 implementations in XGBoost below.
 
-1. ``grow_local_histmaker`` updater: An approximation tree method described in `reference
+1. ``approx`` tree method: An approximation tree method described in `reference paper
-   paper <http://arxiv.org/abs/1603.02754>`_.  This updater is rarely used in practice so
+   <http://arxiv.org/abs/1603.02754>`_.  It runs sketching before building each tree
-   it's still an updater rather than tree method.  During split finding, it first runs a
+   using all the rows (rows belonging to the root). Hessian is used as weights during
-   weighted GK sketching for data points belong to current node to find split candidates,
+   sketch.  The algorithm can be accessed by setting ``tree_method`` to ``approx``.
-   using hessian as weights.  The histogram is built upon this per-node sketch.  It's
-   faster than ``exact`` in some applications, but still slow in computation.
 
-2. ``approx`` tree method: An approximation tree method described in `reference paper
+2. ``hist`` tree method: An approximation tree method used in LightGBM with slight
-   <http://arxiv.org/abs/1603.02754>`_.  Different from ``grow_local_histmaker``, it runs
-   sketching before building each tree using all the rows (rows belonging to the root)
-   instead of per-node dataset.  Similar to ``grow_local_histmaker`` updater, hessian is
-   used as weights during sketch.  The algorithm can be accessed by setting
-   ``tree_method`` to ``approx``.
-
-3. ``hist`` tree method: An approximation tree method used in LightGBM with slight
    differences in implementation.  It runs sketching before training using only user
    provided weights instead of hessian.  The subsequent per-node histogram is built upon
    this global sketch.  This is the fastest algorithm as it runs sketching only once.  The
    algorithm can be accessed by setting ``tree_method`` to ``hist``.
 
-4. ``gpu_hist`` tree method: The ``gpu_hist`` tree method is a GPU implementation of
+3. ``gpu_hist`` tree method: The ``gpu_hist`` tree method is a GPU implementation of
    ``hist``, with additional support for gradient based sampling.  The algorithm can be
    accessed by setting ``tree_method`` to ``gpu_hist``.
 
@@ -102,19 +93,32 @@ Other Updaters
 Removed Updaters
 ****************
 
-2 Updaters were removed during development due to maintainability.  We describe them here
+3 Updaters were removed during development due to maintainability.  We describe them here
-solely for the interest of documentation.  First one is distributed colmaker, which was a
+solely for the interest of documentation.
-distributed version of exact tree method.  It required specialization for column based
-splitting strategy and a different prediction procedure.  As the exact tree method is slow
-by itself and scaling is even less efficient, we removed it entirely.  Second one is
-``skmaker``.  Per-node weighted sketching employed by ``grow_local_histmaker`` is slow,
-the ``skmaker`` was unmaintained and seems to be a workaround trying to eliminate the
-histogram creation step and uses sketching values directly during split evaluation.  It
-was never tested and contained some unknown bugs, we decided to remove it and focus our
-resources on more promising algorithms instead.  For accuracy, most of the time
-``approx``, ``hist`` and ``gpu_hist`` are enough with some parameters tuning, so removing
-them don't have any real practical impact.
 
+1. Distributed colmaker, which was a distributed version of exact tree method.  It
+   required specialization for column based splitting strategy and a different prediction
+   procedure.  As the exact tree method is slow by itself and scaling is even less
+   efficient, we removed it entirely.
+
+2. ``skmaker``.  Per-node weighted sketching employed by ``grow_local_histmaker`` is slow,
+   the ``skmaker`` was unmaintained and seems to be a workaround trying to eliminate the
+   histogram creation step and uses sketching values directly during split evaluation.  It
+   was never tested and contained some unknown bugs, we decided to remove it and focus our
+   resources on more promising algorithms instead.  For accuracy, most of the time
+   ``approx``, ``hist`` and ``gpu_hist`` are enough with some parameters tuning, so
+   removing them don't have any real practical impact.
+
+3. ``grow_local_histmaker`` updater: An approximation tree method described in `reference
+   paper <http://arxiv.org/abs/1603.02754>`_.  This updater was rarely used in practice so
+   it was still an updater rather than tree method.  During split finding, it first runs a
+   weighted GK sketching for data points belong to current node to find split candidates,
+   using hessian as weights.  The histogram is built upon this per-node sketch.  It was
+   faster than ``exact`` in some applications, but still slow in computation.  It was
+   removed because it depended on Rabit's customized reduction function that handles all
+   the data structure that can be serialized/deserialized into fixed size buffer, which is
+   not directly supported by NCCL or federated learning gRPC, making it hard to refactor
+   into a common allreducer interface.
 
 **************
 Feature Matrix

jvm-packages/xgboost4j-spark/src/main/scala/ml/dmlc/xgboost4j/scala/spark/XGBoostClassifier.scala

@@ -100,8 +100,6 @@ class XGBoostClassifier (
 
   def setMaxLeaves(value: Int): this.type = set(maxLeaves, value)
 
-  def setSketchEps(value: Double): this.type = set(sketchEps, value)
-
   def setScalePosWeight(value: Double): this.type = set(scalePosWeight, value)
 
   def setSampleType(value: String): this.type = set(sampleType, value)

jvm-packages/xgboost4j-spark/src/main/scala/ml/dmlc/xgboost4j/scala/spark/XGBoostRegressor.scala

@@ -102,8 +102,6 @@ class XGBoostRegressor (
 
   def setMaxLeaves(value: Int): this.type = set(maxLeaves, value)
 
-  def setSketchEps(value: Double): this.type = set(sketchEps, value)
-
   def setScalePosWeight(value: Double): this.type = set(scalePosWeight, value)
 
   def setSampleType(value: String): this.type = set(sampleType, value)

jvm-packages/xgboost4j-spark/src/main/scala/ml/dmlc/xgboost4j/scala/spark/params/BoosterParams.scala

@@ -182,20 +182,6 @@ private[spark] trait BoosterParams extends Params {
 
   final def getSinglePrecisionHistogram: Boolean = $(singlePrecisionHistogram)
 
-  /**
-   * This is only used for approximate greedy algorithm.
-   * This roughly translated into O(1 / sketch_eps) number of bins. Compared to directly select
-   * number of bins, this comes with theoretical guarantee with sketch accuracy.
-   * [default=0.03] range: (0, 1)
-   */
-  final val sketchEps = new DoubleParam(this, "sketchEps",
-    "This is only used for approximate greedy algorithm. This roughly translated into" +
-      " O(1 / sketch_eps) number of bins. Compared to directly select number of bins, this comes" +
-      " with theoretical guarantee with sketch accuracy.",
-    (value: Double) => value < 1 && value > 0)
-
-  final def getSketchEps: Double = $(sketchEps)
-
   /**
    * Control the balance of positive and negative weights, useful for unbalanced classes. A typical
    * value to consider: sum(negative cases) / sum(positive cases).   [default=1]

plugin/federated/engine_federated.cc

@@ -238,35 +238,5 @@ void Allreduce_(void *sendrecvbuf, size_t type_nbytes, size_t count, IEngine::Re
   if (engine.GetWorldSize() == 1) return;
   engine.Allreduce(sendrecvbuf, type_nbytes * count, dtype, op);
 }
-
-ReduceHandle::ReduceHandle() = default;
-ReduceHandle::~ReduceHandle() = default;
-
-int ReduceHandle::TypeSize(const MPI::Datatype &dtype) { return static_cast<int>(dtype.type_size); }
-
-void ReduceHandle::Init(IEngine::ReduceFunction redfunc,
-                        __attribute__((unused)) size_t type_nbytes) {
-  utils::Assert(redfunc_ == nullptr, "cannot initialize reduce handle twice");
-  redfunc_ = redfunc;
-}
-
-void ReduceHandle::Allreduce(void *sendrecvbuf, size_t type_nbytes, size_t count,
-                             IEngine::PreprocFunction prepare_fun, void *prepare_arg) {
-  utils::Assert(redfunc_ != nullptr, "must initialize handle to call AllReduce");
-  if (prepare_fun != nullptr) prepare_fun(prepare_arg);
-  if (engine.GetWorldSize() == 1) return;
-
-  // Gather all the buffers and call the reduce function locally.
-  auto const buffer_size = type_nbytes * count;
-  auto const gathered = engine.Allgather(sendrecvbuf, buffer_size);
-  auto const *data = gathered.data();
-  for (int i = 0; i < engine.GetWorldSize(); i++) {
-    if (i != engine.GetRank()) {
-      redfunc_(data + buffer_size * i, sendrecvbuf, static_cast<int>(count),
-               MPI::Datatype(type_nbytes));
-    }
-  }
-}
-
 }  // namespace engine
 }  // namespace rabit

rabit/include/rabit/internal/engine.h

@@ -245,51 +245,6 @@ void Allreduce_(void *sendrecvbuf,  // NOLINT
                 mpi::OpType op,
                 IEngine::PreprocFunction prepare_fun = nullptr,
                 void *prepare_arg = nullptr);
-/*!
- * \brief handle for customized reducer, used to handle customized reduce
- *  this class is mainly created for compatiblity issues with MPI's customized reduce
- */
-class ReduceHandle {
- public:
-  // constructor
-  ReduceHandle();
-  // destructor
-  ~ReduceHandle();
-  /*!
-   * \brief initialize the reduce function,
-   *   with the type the reduce function needs to deal with
-   *   the reduce function MUST be communicative
-   */
-  void Init(IEngine::ReduceFunction redfunc, size_t type_nbytes);
-  /*!
-   * \brief customized in-place all reduce operation
-   * \param sendrecvbuf the in place send-recv buffer
-   * \param type_n4bytes size of the type, in terms of 4bytes
-   * \param count number of elements to send
-   * \param prepare_func Lazy preprocessing function, lazy prepare_fun(prepare_arg)
-   *                     will be called by the function before performing Allreduce in order to initialize the data in sendrecvbuf_.
-   *                     If the result of Allreduce can be recovered directly, then prepare_func will NOT be called
-   * \param prepare_arg argument used to pass into the lazy preprocessing function
-   */
-  void Allreduce(void *sendrecvbuf,
-                 size_t type_nbytes,
-                 size_t count,
-                 IEngine::PreprocFunction prepare_fun = nullptr,
-                 void *prepare_arg = nullptr);
-
-  /*! \return the number of bytes occupied by the type */
-  static int TypeSize(const MPI::Datatype &dtype);
-
- protected:
-  // handle function field
-  void *handle_ {nullptr};
-  // reduce function of the reducer
-  IEngine::ReduceFunction *redfunc_{nullptr};
-  // handle to the type field
-  void *htype_{nullptr};
-  // the created type in 4 bytes
-  size_t created_type_nbytes_;
-};
 }  // namespace engine
 }  // namespace rabit
 #endif  // RABIT_INTERNAL_ENGINE_H_

rabit/include/rabit/internal/rabit-inl.h

@@ -225,122 +225,5 @@ inline void LazyCheckPoint(const Serializable *global_model) {
 inline int VersionNumber() {
   return engine::GetEngine()->VersionNumber();
 }
-// ---------------------------------
-// Code to handle customized Reduce
-// ---------------------------------
-// function to perform reduction for Reducer
-template<typename DType, void (*freduce)(DType &dst, const DType &src)>
-inline void ReducerSafeImpl(const void *src_, void *dst_, int len_, const MPI::Datatype &dtype) {
-  const size_t kUnit = sizeof(DType);
-  const char *psrc = reinterpret_cast<const char*>(src_);
-  char *pdst = reinterpret_cast<char*>(dst_);
-
-  for (int i = 0; i < len_; ++i) {
-    DType tdst, tsrc;
-    // use memcpy to avoid alignment issue
-    std::memcpy(&tdst, pdst + (i * kUnit), sizeof(tdst));
-    std::memcpy(&tsrc, psrc + (i * kUnit), sizeof(tsrc));
-    freduce(tdst, tsrc);
-    std::memcpy(pdst + i * kUnit, &tdst, sizeof(tdst));
-  }
-}
-// function to perform reduction for Reducer
-template<typename DType, void (*freduce)(DType &dst, const DType &src)> // NOLINT(*)
-inline void ReducerAlignImpl(const void *src_, void *dst_,
-                          int len_, const MPI::Datatype &dtype) {
-  const DType *psrc = reinterpret_cast<const DType*>(src_);
-  DType *pdst = reinterpret_cast<DType*>(dst_);
-  for (int i = 0; i < len_; ++i) {
-    freduce(pdst[i], psrc[i]);
-  }
-}
-template<typename DType, void (*freduce)(DType &dst, const DType &src)>  // NOLINT(*)
-inline Reducer<DType, freduce>::Reducer() {
-  // it is safe to directly use handle for aligned data types
-  if (sizeof(DType) == 8 || sizeof(DType) == 4 || sizeof(DType) == 1) {
-    this->handle_.Init(ReducerAlignImpl<DType, freduce>, sizeof(DType));
-  } else {
-    this->handle_.Init(ReducerSafeImpl<DType, freduce>, sizeof(DType));
-  }
-}
-template<typename DType, void (*freduce)(DType &dst, const DType &src)> // NOLINT(*)
-inline void Reducer<DType, freduce>::Allreduce(DType *sendrecvbuf, size_t count,
-                                               void (*prepare_fun)(void *arg),
-                                               void *prepare_arg) {
-  handle_.Allreduce(sendrecvbuf, sizeof(DType), count, prepare_fun,
-                    prepare_arg);
-}
-// function to perform reduction for SerializeReducer
-template<typename DType>
-inline void SerializeReducerFuncImpl(const void *src_, void *dst_,
-                                     int len_, const MPI::Datatype &dtype) {
-  int nbytes = engine::ReduceHandle::TypeSize(dtype);
-  // temp space
-  for (int i = 0; i < len_; ++i) {
-    DType tsrc, tdst;
-    utils::MemoryFixSizeBuffer fsrc((char*)(src_) + i * nbytes, nbytes); // NOLINT(*)
-    utils::MemoryFixSizeBuffer fdst((char*)(dst_) + i * nbytes, nbytes); // NOLINT(*)
-    tsrc.Load(fsrc);
-    tdst.Load(fdst);
-    // govern const check
-    tdst.Reduce(static_cast<const DType &>(tsrc), nbytes);
-    fdst.Seek(0);
-    tdst.Save(fdst);
-  }
-}
-template<typename DType>
-inline SerializeReducer<DType>::SerializeReducer() {
-  handle_.Init(SerializeReducerFuncImpl<DType>, sizeof(DType));
-}
-// closure to call Allreduce
-template<typename DType>
-struct SerializeReduceClosure {
-  DType *sendrecvobj;
-  size_t max_nbyte, count;
-  void (*prepare_fun)(void *arg);
-  void *prepare_arg;
-  std::string *p_buffer;
-  // invoke the closure
-  inline void Run() {
-    if (prepare_fun != nullptr) prepare_fun(prepare_arg);
-    for (size_t i = 0; i < count; ++i) {
-      utils::MemoryFixSizeBuffer fs(BeginPtr(*p_buffer) + i * max_nbyte, max_nbyte);
-      sendrecvobj[i].Save(fs);
-    }
-  }
-  inline static void Invoke(void *c) {
-    static_cast<SerializeReduceClosure<DType>*>(c)->Run();
-  }
-};
-template<typename DType>
-inline void SerializeReducer<DType>::Allreduce(DType *sendrecvobj,
-                                               size_t max_nbyte, size_t count,
-                                               void (*prepare_fun)(void *arg),
-                                               void *prepare_arg) {
-  buffer_.resize(max_nbyte * count);
-  // setup closure
-  SerializeReduceClosure<DType> c;
-  c.sendrecvobj = sendrecvobj; c.max_nbyte = max_nbyte; c.count = count;
-  c.prepare_fun = prepare_fun; c.prepare_arg = prepare_arg; c.p_buffer = &buffer_;
-  // invoke here
-  handle_.Allreduce(BeginPtr(buffer_), max_nbyte, count,
-                    SerializeReduceClosure<DType>::Invoke, &c);
-  for (size_t i = 0; i < count; ++i) {
-    utils::MemoryFixSizeBuffer fs(BeginPtr(buffer_) + i * max_nbyte, max_nbyte);
-    sendrecvobj[i].Load(fs);
-  }
-}
-
-template<typename DType, void (*freduce)(DType &dst, const DType &src)>  // NOLINT(*)g
-inline void Reducer<DType, freduce>::Allreduce(DType *sendrecvbuf, size_t count,
-                                               std::function<void()> prepare_fun) {
-  this->Allreduce(sendrecvbuf, count, InvokeLambda, &prepare_fun);
-}
-template<typename DType>
-inline void SerializeReducer<DType>::Allreduce(DType *sendrecvobj,
-                                               size_t max_nbytes, size_t count,
-                                               std::function<void()> prepare_fun) {
-  this->Allreduce(sendrecvobj, max_nbytes, count, InvokeLambda, &prepare_fun);
-}
 }  // namespace rabit
 #endif  // RABIT_INTERNAL_RABIT_INL_H_

rabit/include/rabit/rabit.h

@@ -274,100 +274,6 @@ inline void LazyCheckPoint(const Serializable *global_model);
  * \sa LoadCheckPoint, CheckPoint
  */
 inline int VersionNumber();
-// ----- extensions that allow customized reducer ------
-// helper class to do customized reduce, user do not need to know the type
-namespace engine {
-class ReduceHandle;
-}  // namespace engine
-/*!
- * \brief template class to make customized reduce and all reduce easy
- *  Do not use reducer directly in the function you call Finalize,
- *   because the destructor can execute after Finalize
- * \tparam DType data type that to be reduced
- * \tparam freduce the customized reduction function
- *  DType must be a struct, with no pointer
- */
-template<typename DType, void (*freduce)(DType &dst, const DType &src)>  // NOLINT(*)
-class Reducer {
- public:
-  Reducer();
-  /*!
-   * \brief customized in-place all reduce operation
-   * \param sendrecvbuf the in place send-recv buffer
-   * \param count number of elements to be reduced
-   * \param prepare_fun Lazy preprocessing function, if it is not NULL, prepare_fun(prepare_arg)
-   *                     will be called by the function before performing Allreduce, to initialize the data in sendrecvbuf.
-   *                     If the result of Allreduce can be recovered directly, then prepare_func will NOT be called
-   * \param prepare_arg argument used to pass into the lazy preprocessing function
-   */
-  inline void Allreduce(DType *sendrecvbuf, size_t count,
-                        void (*prepare_fun)(void *) = nullptr,
-                        void *prepare_arg = nullptr);
-#if DMLC_USE_CXX11
-  /*!
-   * \brief customized in-place all reduce operation, with lambda function as preprocessor
-   * \param sendrecvbuf pointer to the array of objects to be reduced
-   * \param count number of elements to be reduced
-   * \param prepare_fun lambda function executed to prepare the data, if necessary
-   */
-  inline void Allreduce(DType *sendrecvbuf, size_t count,
-                        std::function<void()> prepare_fun);
-#endif  // DMLC_USE_CXX11
-
- private:
-  /*! \brief function handle to do reduce */
-  engine::ReduceHandle handle_;
-};
-/*!
- * \brief template class to make customized reduce,
- *  this class defines complex reducer handles all the data structure that can be
- *  serialized/deserialized into fixed size buffer
- *  Do not use reducer directly in the function you call Finalize, because the destructor can execute after Finalize
- *
- * \tparam DType data type that to be reduced, DType must contain the following functions:
- * \tparam freduce the customized reduction function
- *   (1) Save(IStream &fs)  (2) Load(IStream &fs) (3) Reduce(const DType &src, size_t max_nbyte)
- */
-template<typename DType>
-class SerializeReducer {
- public:
-  SerializeReducer();
-  /*!
-   * \brief customized in-place all reduce operation
-   * \param sendrecvobj pointer to the array of objects to be reduced
-   * \param max_nbyte maximum amount of memory needed to serialize each object
-   *        this includes budget limit for intermediate and final result
-   * \param count number of elements to be reduced
-   * \param prepare_fun Lazy preprocessing function, if it is not NULL, prepare_fun(prepare_arg)
-   *                     will be called by the function before performing Allreduce, to initialize the data in sendrecvbuf.
-   *                     If the result of Allreduce can be recovered directly, then the prepare_func will NOT be called
-   * \param prepare_arg argument used to pass into the lazy preprocessing function
-   */
-  inline void Allreduce(DType *sendrecvobj,
-                        size_t max_nbyte, size_t count,
-                        void (*prepare_fun)(void *) = nullptr,
-                        void *prepare_arg = nullptr);
-// C++11 support for lambda prepare function
-#if DMLC_USE_CXX11
-  /*!
-   * \brief customized in-place all reduce operation, with lambda function as preprocessor
-   * \param sendrecvobj pointer to the array of objects to be reduced
-   * \param max_nbyte maximum amount of memory needed to serialize each object
-   *        this includes budget limit for intermediate and final result
-   * \param count number of elements to be reduced
-   * \param prepare_fun lambda function executed to prepare the data, if necessary
-   */
-  inline void Allreduce(DType *sendrecvobj,
-                        size_t max_nbyte, size_t count,
-                        std::function<void()> prepare_fun);
-#endif  // DMLC_USE_CXX11
-
- private:
-  /*! \brief function handle to do reduce */
-  engine::ReduceHandle handle_;
-  /*! \brief temporal buffer used to do reduce*/
-  std::string buffer_;
-};
 }  // namespace rabit
 // implementation of template functions
 #include "./internal/rabit-inl.h"

rabit/src/engine.cc

@@ -103,27 +103,5 @@ void Allreduce_(void *sendrecvbuf,  // NOLINT
   GetEngine()->Allreduce(sendrecvbuf, type_nbytes, count, red, prepare_fun,
     prepare_arg);
 }
-
-// code for reduce handle
-ReduceHandle::ReduceHandle() = default;
-ReduceHandle::~ReduceHandle() = default;
-
-int ReduceHandle::TypeSize(const MPI::Datatype &dtype) {
-  return static_cast<int>(dtype.type_size);
-}
-
-void ReduceHandle::Init(IEngine::ReduceFunction redfunc, size_t ) {
-  utils::Assert(redfunc_ == nullptr, "cannot initialize reduce handle twice");
-  redfunc_ = redfunc;
-}
-
-void ReduceHandle::Allreduce(void *sendrecvbuf,
-                             size_t type_nbytes, size_t count,
-                             IEngine::PreprocFunction prepare_fun,
-                             void *prepare_arg) {
-  utils::Assert(redfunc_ != nullptr, "must initialize handle to call AllReduce");
-  GetEngine()->Allreduce(sendrecvbuf, type_nbytes, count,
-                         redfunc_, prepare_fun, prepare_arg);
-}
 }  // namespace engine
 }  // namespace rabit

rabit/src/engine_mpi.cc

@@ -158,92 +158,5 @@ void Allreduce_(void *sendrecvbuf,
   MPI::COMM_WORLD.Allreduce(MPI_IN_PLACE, sendrecvbuf,
                             count, GetType(dtype), GetOp(op));
 }
-
-// code for reduce handle
-ReduceHandle::ReduceHandle(void)
-    : handle_(NULL), redfunc_(NULL), htype_(NULL) {
-}
-ReduceHandle::~ReduceHandle(void) {
-  /* !WARNING!
-
-     A handle can be held by a tree method/Learner from xgboost.  The booster might not be
-     freed until program exit, while (good) users call rabit.finalize() before reaching
-     the end of program.  So op->Free() might be called after finalization and results
-     into following error:
-
-      ```
-        Attempting to use an MPI routine after finalizing MPICH
-      ```
-
-     Here we skip calling Free if MPI has already been finalized to workaround the issue.
-     It can be a potential leak of memory.  The best way to resolve it is to eliminate all
-     use of long living handle.
-  */
-  int finalized = 0;
-  CHECK_EQ(MPI_Finalized(&finalized), MPI_SUCCESS);
-  if (handle_ != NULL) {
-    MPI::Op *op = reinterpret_cast<MPI::Op*>(handle_);
-    if (!finalized) {
-      op->Free();
-    }
-    delete op;
-  }
-  if (htype_ != NULL) {
-    MPI::Datatype *dtype = reinterpret_cast<MPI::Datatype*>(htype_);
-    if (!finalized) {
-      dtype->Free();
-    }
-    delete dtype;
-  }
-}
-int ReduceHandle::TypeSize(const MPI::Datatype &dtype) {
-  return dtype.Get_size();
-}
-void ReduceHandle::Init(IEngine::ReduceFunction redfunc, size_t type_nbytes) {
-  utils::Assert(handle_ == NULL, "cannot initialize reduce handle twice");
-  if (type_nbytes != 0) {
-    MPI::Datatype *dtype = new MPI::Datatype();
-    if (type_nbytes % 8 == 0) {
-      *dtype = MPI::LONG.Create_contiguous(type_nbytes / sizeof(long));  // NOLINT(*)
-    } else if (type_nbytes % 4 == 0) {
-      *dtype = MPI::INT.Create_contiguous(type_nbytes / sizeof(int));
-    } else {
-      *dtype = MPI::CHAR.Create_contiguous(type_nbytes);
-    }
-    dtype->Commit();
-    created_type_nbytes_ = type_nbytes;
-    htype_ = dtype;
-  }
-  MPI::Op *op = new MPI::Op();
-  MPI::User_function *pf = redfunc;
-  op->Init(pf, true);
-  handle_ = op;
-}
-void ReduceHandle::Allreduce(void *sendrecvbuf,
-                             size_t type_nbytes, size_t count,
-                             IEngine::PreprocFunction prepare_fun,
-                             void *prepare_arg) {
-  utils::Assert(handle_ != NULL, "must initialize handle to call AllReduce");
-  MPI::Op *op = reinterpret_cast<MPI::Op*>(handle_);
-  MPI::Datatype *dtype = reinterpret_cast<MPI::Datatype*>(htype_);
-  if (created_type_nbytes_ != type_nbytes || dtype == NULL) {
-    if (dtype == NULL) {
-      dtype = new MPI::Datatype();
-    } else {
-      dtype->Free();
-    }
-    if (type_nbytes % 8 == 0) {
-      *dtype = MPI::LONG.Create_contiguous(type_nbytes / sizeof(long));  // NOLINT(*)
-    } else if (type_nbytes % 4 == 0) {
-      *dtype = MPI::INT.Create_contiguous(type_nbytes / sizeof(int));
-    } else {
-      *dtype = MPI::CHAR.Create_contiguous(type_nbytes);
-    }
-    dtype->Commit();
-    created_type_nbytes_ = type_nbytes;
-  }
-  if (prepare_fun != NULL) prepare_fun(prepare_arg);
-  MPI::COMM_WORLD.Allreduce(MPI_IN_PLACE, sendrecvbuf, count, *dtype, *op);
-}
 }  // namespace engine
 }  // namespace rabit

src/tree/hist/histogram.h

@@ -24,7 +24,6 @@ class HistogramBuilder {
   common::HistCollection hist_local_worker_;
   common::GHistBuilder builder_;
   common::ParallelGHistBuilder buffer_;
-  rabit::Reducer<GradientPairPrecise, GradientPairPrecise::Reduce> reducer_;
   BatchParam param_;
   int32_t n_threads_{-1};
   size_t n_batches_{0};
@@ -199,8 +198,8 @@ class HistogramBuilder {
           }
         });
 
-    reducer_.Allreduce(this->hist_[starting_index].data(),
+    rabit::Allreduce<rabit::op::Sum>(reinterpret_cast<double*>(this->hist_[starting_index].data()),
-                       builder_.GetNumBins() * sync_count);
+                                     builder_.GetNumBins() * sync_count * 2);
 
     ParallelSubtractionHist(space, nodes_for_explicit_hist_build,
                             nodes_for_subtraction_trick, p_tree);

src/tree/param.h

@@ -65,8 +65,6 @@ struct TrainParam : public XGBoostParameter<TrainParam> {
   // whether to subsample columns during tree construction
   float colsample_bytree;
   // accuracy of sketch
-  float sketch_eps;
-  // accuracy of sketch
   float sketch_ratio;
   // option to open cacheline optimization
   bool cache_opt;
@@ -162,10 +160,6 @@ struct TrainParam : public XGBoostParameter<TrainParam> {
         .set_range(0.0f, 1.0f)
         .set_default(1.0f)
         .describe("Subsample ratio of columns, resample on each tree construction.");
-    DMLC_DECLARE_FIELD(sketch_eps)
-        .set_range(0.0f, 1.0f)
-        .set_default(0.03f)
-        .describe("EXP Param: Sketch accuracy of approximate algorithm.");
     DMLC_DECLARE_FIELD(sketch_ratio)
         .set_lower_bound(0.0f)
         .set_default(2.0f)
@@ -203,12 +197,6 @@ struct TrainParam : public XGBoostParameter<TrainParam> {
     return loss_chg < this->min_split_loss ||
            (this->max_depth != 0 && depth > this->max_depth);
   }
-  /*! \brief maximum sketch size */
-  inline unsigned MaxSketchSize() const {
-    auto ret = static_cast<unsigned>(sketch_ratio / sketch_eps);
-    CHECK_GT(ret, 0U);
-    return ret;
-  }
 
   bst_node_t MaxNodes() const {
     if (this->max_depth == 0 && this->max_leaves == 0) {

src/tree/tree_updater.cc

@@ -33,7 +33,6 @@ DMLC_REGISTRY_LINK_TAG(updater_colmaker);
 DMLC_REGISTRY_LINK_TAG(updater_refresh);
 DMLC_REGISTRY_LINK_TAG(updater_prune);
 DMLC_REGISTRY_LINK_TAG(updater_quantile_hist);
-DMLC_REGISTRY_LINK_TAG(updater_histmaker);
 DMLC_REGISTRY_LINK_TAG(updater_approx);
 DMLC_REGISTRY_LINK_TAG(updater_sync);
 #ifdef XGBOOST_USE_CUDA

src/tree/updater_histmaker.cc

@@ -1,629 +0,0 @@
-/*!
- * Copyright 2014-2022 by XGBoost Contributors
- * \file updater_histmaker.cc
- * \brief use histogram counting to construct a tree
- * \author Tianqi Chen
- */
-#include <rabit/rabit.h>
-#include <vector>
-#include <algorithm>
-
-#include "xgboost/tree_updater.h"
-#include "xgboost/base.h"
-#include "xgboost/logging.h"
-
-#include "../common/quantile.h"
-#include "../common/group_data.h"
-#include "./updater_basemaker-inl.h"
-#include "constraints.h"
-
-namespace xgboost {
-namespace tree {
-
-DMLC_REGISTRY_FILE_TAG(updater_histmaker);
-
-class HistMaker: public BaseMaker {
- public:
-  explicit HistMaker(GenericParameter const *ctx) : BaseMaker(ctx) {}
-  void Update(HostDeviceVector<GradientPair> *gpair, DMatrix *p_fmat,
-              common::Span<HostDeviceVector<bst_node_t>> out_position,
-              const std::vector<RegTree *> &trees) override {
-    interaction_constraints_.Configure(param_, p_fmat->Info().num_col_);
-    // rescale learning rate according to size of trees
-    float lr = param_.learning_rate;
-    param_.learning_rate = lr / trees.size();
-    // build tree
-    for (auto tree : trees) {
-      this->UpdateTree(gpair->ConstHostVector(), p_fmat, tree);
-    }
-    param_.learning_rate = lr;
-  }
-  char const* Name() const override {
-    return "grow_histmaker";
-  }
-
- protected:
-    /*! \brief a single column of histogram cuts */
-  struct HistUnit {
-    /*! \brief cutting point of histogram, contains maximum point */
-    const float *cut;
-    /*! \brief content of statistics data */
-    GradStats *data;
-    /*! \brief size of histogram */
-    uint32_t size;
-    // default constructor
-    HistUnit() = default;
-    // constructor
-    HistUnit(const float *cut, GradStats *data, uint32_t size)
-        : cut{cut}, data{data}, size{size} {}
-    /*! \brief add a histogram to data */
-  };
-  /*! \brief a set of histograms from different index */
-  struct HistSet {
-    /*! \brief the index pointer of each histunit */
-    const uint32_t *rptr;
-    /*! \brief cutting points in each histunit */
-    const bst_float *cut;
-    /*! \brief data in different hist unit */
-    std::vector<GradStats> data;
-    /*! \brief return a column of histogram cuts */
-    inline HistUnit operator[](size_t fid) {
-      return {cut + rptr[fid], &data[0] + rptr[fid], rptr[fid+1] - rptr[fid]};
-    }
-  };
-  // thread workspace
-  struct ThreadWSpace {
-    /*! \brief actual unit pointer */
-    std::vector<unsigned> rptr;
-    /*! \brief cut field */
-    std::vector<bst_float> cut;
-    // per thread histset
-    std::vector<HistSet> hset;
-    // initialize the hist set
-    inline void Configure(int nthread) {
-      hset.resize(nthread);
-      // cleanup statistics
-      for (int tid = 0; tid < nthread; ++tid) {
-        for (auto& d : hset[tid].data) { d = GradStats(); }
-        hset[tid].rptr = dmlc::BeginPtr(rptr);
-        hset[tid].cut = dmlc::BeginPtr(cut);
-        hset[tid].data.resize(cut.size(), GradStats());
-      }
-    }
-    /*! \brief clear the workspace */
-    inline void Clear() {
-      cut.clear(); rptr.resize(1); rptr[0] = 0;
-    }
-    /*! \brief total size */
-    inline size_t Size() const {
-      return rptr.size() - 1;
-    }
-  };
-  // workspace of thread
-  ThreadWSpace wspace_;
-  // reducer for histogram
-  rabit::Reducer<GradStats, GradStats::Reduce> histred_;
-  // set of working features
-  std::vector<bst_feature_t> selected_features_;
-  // update function implementation
-  virtual void UpdateTree(const std::vector<GradientPair> &gpair,
-                          DMatrix *p_fmat,
-                          RegTree *p_tree) {
-    CHECK(param_.max_depth > 0) << "max_depth must be larger than 0";
-    this->InitData(gpair, *p_fmat, *p_tree);
-    this->InitWorkSet(p_fmat, *p_tree, &selected_features_);
-    // mark root node as fresh.
-    (*p_tree)[0].SetLeaf(0.0f, 0);
-
-    for (int depth = 0; depth < param_.max_depth; ++depth) {
-      // reset and propose candidate split
-      this->ResetPosAndPropose(gpair, p_fmat, selected_features_, *p_tree);
-      // create histogram
-      this->CreateHist(gpair, p_fmat, selected_features_, *p_tree);
-      // find split based on histogram statistics
-      this->FindSplit(selected_features_, p_tree);
-      // reset position after split
-      this->ResetPositionAfterSplit(p_fmat, *p_tree);
-      this->UpdateQueueExpand(*p_tree);
-      // if nothing left to be expand, break
-      if (qexpand_.size() == 0) break;
-    }
-    for (int const nid : qexpand_) {
-      (*p_tree)[nid].SetLeaf(p_tree->Stat(nid).base_weight * param_.learning_rate);
-    }
-  }
-  // this function does two jobs
-  // (1) reset the position in array position, to be the latest leaf id
-  // (2) propose a set of candidate cuts and set wspace.rptr wspace.cut correctly
-  virtual void ResetPosAndPropose(const std::vector<GradientPair> &gpair,
-                                  DMatrix *p_fmat,
-                                  const std::vector <bst_feature_t> &fset,
-                                  const RegTree &tree) = 0;
-  // initialize the current working set of features in this round
-  virtual void InitWorkSet(DMatrix *,
-                           const RegTree &tree,
-                           std::vector<bst_feature_t> *p_fset) {
-    p_fset->resize(tree.param.num_feature);
-    for (size_t i = 0; i < p_fset->size(); ++i) {
-      (*p_fset)[i] = static_cast<unsigned>(i);
-    }
-  }
-  // reset position after split, this is not a must, depending on implementation
-  virtual void ResetPositionAfterSplit(DMatrix *p_fmat,
-                                       const RegTree &tree) {
-  }
-  virtual void CreateHist(const std::vector<GradientPair> &gpair,
-                          DMatrix *,
-                          const std::vector <bst_feature_t> &fset,
-                          const RegTree &)  = 0;
-
- private:
-  void EnumerateSplit(const HistUnit &hist,
-                      const GradStats &node_sum,
-                      bst_uint fid,
-                      SplitEntry *best,
-                      GradStats *left_sum) const {
-    if (hist.size == 0) return;
-
-    double root_gain = CalcGain(param_, node_sum.GetGrad(), node_sum.GetHess());
-    GradStats s, c;
-    for (bst_uint i = 0; i < hist.size; ++i) {
-      s.Add(hist.data[i]);
-      if (s.sum_hess >= param_.min_child_weight) {
-        c.SetSubstract(node_sum, s);
-        if (c.sum_hess >= param_.min_child_weight) {
-          double loss_chg = CalcGain(param_, s.GetGrad(), s.GetHess()) +
-                            CalcGain(param_, c.GetGrad(), c.GetHess()) - root_gain;
-          if (best->Update(static_cast<bst_float>(loss_chg), fid, hist.cut[i],
-                           false, false, s, c)) {
-            *left_sum = s;
-          }
-        }
-      }
-    }
-    s = GradStats();
-    for (bst_uint i = hist.size - 1; i != 0; --i) {
-      s.Add(hist.data[i]);
-      if (s.sum_hess >= param_.min_child_weight) {
-        c.SetSubstract(node_sum, s);
-        if (c.sum_hess >= param_.min_child_weight) {
-          double loss_chg = CalcGain(param_, s.GetGrad(), s.GetHess()) +
-                            CalcGain(param_, c.GetGrad(), c.GetHess()) - root_gain;
-          if (best->Update(static_cast<bst_float>(loss_chg), fid,
-                           hist.cut[i - 1], true, false, c, s)) {
-            *left_sum = c;
-          }
-        }
-      }
-    }
-  }
-
-  void FindSplit(const std::vector <bst_feature_t> &feature_set,
-                 RegTree *p_tree) {
-    const size_t num_feature = feature_set.size();
-    // get the best split condition for each node
-    std::vector<SplitEntry> sol(qexpand_.size());
-    std::vector<GradStats> left_sum(qexpand_.size());
-    auto nexpand = qexpand_.size();
-    common::ParallelFor(nexpand, ctx_->Threads(), common::Sched::Dyn(1), [&](auto wid) {
-      const int nid = qexpand_[wid];
-      CHECK_EQ(node2workindex_[nid], static_cast<int>(wid));
-      SplitEntry &best = sol[wid];
-      GradStats &node_sum = wspace_.hset[0][num_feature + wid * (num_feature + 1)].data[0];
-      for (size_t i = 0; i < feature_set.size(); ++i) {
-        // Query is thread safe as it's a const function.
-        if (!this->interaction_constraints_.Query(nid, feature_set[i])) {
-          continue;
-        }
-
-        EnumerateSplit(this->wspace_.hset[0][i + wid * (num_feature + 1)], node_sum, feature_set[i],
-                       &best, &left_sum[wid]);
-      }
-    });
-    // get the best result, we can synchronize the solution
-    for (bst_omp_uint wid = 0; wid < nexpand; ++wid) {
-      const bst_node_t nid = qexpand_[wid];
-      SplitEntry const& best = sol[wid];
-      const GradStats &node_sum = wspace_.hset[0][num_feature + wid * (num_feature + 1)].data[0];
-      this->SetStats(p_tree, nid, node_sum);
-      // set up the values
-      p_tree->Stat(nid).loss_chg = best.loss_chg;
-      // now we know the solution in snode[nid], set split
-      if (best.loss_chg > kRtEps) {
-        bst_float base_weight = CalcWeight(param_, node_sum);
-        bst_float left_leaf_weight =
-            CalcWeight(param_, best.left_sum.sum_grad, best.left_sum.sum_hess) *
-            param_.learning_rate;
-        bst_float right_leaf_weight =
-            CalcWeight(param_, best.right_sum.sum_grad,
-                       best.right_sum.sum_hess) *
-            param_.learning_rate;
-        p_tree->ExpandNode(nid, best.SplitIndex(), best.split_value,
-                           best.DefaultLeft(), base_weight, left_leaf_weight,
-                           right_leaf_weight, best.loss_chg,
-                           node_sum.sum_hess,
-                           best.left_sum.GetHess(), best.right_sum.GetHess());
-        GradStats right_sum;
-        right_sum.SetSubstract(node_sum, left_sum[wid]);
-        auto left_child = (*p_tree)[nid].LeftChild();
-        auto right_child = (*p_tree)[nid].RightChild();
-        this->SetStats(p_tree, left_child, left_sum[wid]);
-        this->SetStats(p_tree, right_child, right_sum);
-        this->interaction_constraints_.Split(nid, best.SplitIndex(), left_child, right_child);
-      } else {
-        (*p_tree)[nid].SetLeaf(p_tree->Stat(nid).base_weight * param_.learning_rate);
-      }
-    }
-  }
-
-  inline void SetStats(RegTree *p_tree, int nid, const GradStats &node_sum) {
-    p_tree->Stat(nid).base_weight =
-        static_cast<bst_float>(CalcWeight(param_, node_sum));
-    p_tree->Stat(nid).sum_hess = static_cast<bst_float>(node_sum.sum_hess);
-  }
-};
-
-class CQHistMaker : public HistMaker {
- public:
-  explicit CQHistMaker(GenericParameter const *ctx) : HistMaker(ctx) {}
-  char const *Name() const override { return "grow_local_histmaker"; }
-
- protected:
-  struct HistEntry {
-    HistMaker::HistUnit hist;
-    unsigned istart;
-    /*!
-     * \brief add a histogram to data,
-     * do linear scan, start from istart
-     */
-    inline void Add(bst_float fv,
-                    const std::vector<GradientPair> &gpair,
-                    const bst_uint ridx) {
-      while (istart < hist.size && !(fv < hist.cut[istart])) ++istart;
-      CHECK_NE(istart, hist.size);
-      hist.data[istart].Add(gpair[ridx]);
-    }
-    /*!
-     * \brief add a histogram to data,
-     * do linear scan, start from istart
-     */
-    inline void Add(bst_float fv,
-                    GradientPair gstats) {
-      if (fv < hist.cut[istart]) {
-        hist.data[istart].Add(gstats);
-      } else {
-        while (istart < hist.size && !(fv < hist.cut[istart])) ++istart;
-        if (istart != hist.size) {
-          hist.data[istart].Add(gstats);
-        } else {
-          LOG(INFO) << "fv=" << fv << ", hist.size=" << hist.size;
-          for (size_t i = 0; i < hist.size; ++i) {
-            LOG(INFO) << "hist[" << i << "]=" << hist.cut[i];
-          }
-          LOG(FATAL) << "fv=" << fv << ", hist.last=" << hist.cut[hist.size - 1];
-        }
-      }
-    }
-  };
-  // sketch type used for this
-  using WXQSketch = common::WXQuantileSketch<bst_float, bst_float>;
-  // initialize the work set of tree
-  void InitWorkSet(DMatrix *p_fmat,
-                   const RegTree &tree,
-                   std::vector<bst_feature_t> *p_fset) override {
-    if (p_fmat != cache_dmatrix_) {
-      feat_helper_.InitByCol(p_fmat, tree);
-      cache_dmatrix_ = p_fmat;
-    }
-    feat_helper_.SyncInfo();
-    feat_helper_.SampleCol(this->param_.colsample_bytree, p_fset);
-  }
-  // code to create histogram
-  void CreateHist(const std::vector<GradientPair> &gpair,
-                  DMatrix *p_fmat,
-                  const std::vector<bst_feature_t> &fset,
-                  const RegTree &tree) override {
-    const MetaInfo &info = p_fmat->Info();
-    // fill in reverse map
-    feat2workindex_.resize(tree.param.num_feature);
-    std::fill(feat2workindex_.begin(), feat2workindex_.end(), -1);
-    for (size_t i = 0; i < fset.size(); ++i) {
-      feat2workindex_[fset[i]] = static_cast<int>(i);
-    }
-    // start to work
-    this->wspace_.Configure(1);
-    // if it is C++11, use lazy evaluation for Allreduce,
-    // to gain speedup in recovery
-    auto lazy_get_hist = [&]() {
-      thread_hist_.resize(ctx_->Threads());
-      // start accumulating statistics
-      for (const auto &batch : p_fmat->GetBatches<SortedCSCPage>()) {
-        auto page = batch.GetView();
-        // start enumeration
-        common::ParallelFor(fset.size(), ctx_->Threads(), common::Sched::Dyn(1), [&](auto i) {
-          int fid = fset[i];
-          int offset = feat2workindex_[fid];
-          if (offset >= 0) {
-            this->UpdateHistCol(gpair, page[fid], info, tree, fset, offset,
-                                &thread_hist_[omp_get_thread_num()]);
-          }
-        });
-      }
-      // update node statistics.
-      this->GetNodeStats(gpair, *p_fmat, tree,
-                         &thread_stats_, &node_stats_);
-      for (int const nid : this->qexpand_) {
-        const int wid = this->node2workindex_[nid];
-        this->wspace_.hset[0][fset.size() + wid * (fset.size() + 1)]
-                .data[0] = node_stats_[nid];
-      }
-    };
-    // sync the histogram
-    this->histred_.Allreduce(dmlc::BeginPtr(this->wspace_.hset[0].data),
-                             this->wspace_.hset[0].data.size(), lazy_get_hist);
-  }
-
-  void ResetPositionAfterSplit(DMatrix *,
-                                 const RegTree &tree) override {
-    this->GetSplitSet(this->qexpand_, tree, &fsplit_set_);
-  }
-  void ResetPosAndPropose(const std::vector<GradientPair> &gpair,
-                          DMatrix *p_fmat,
-                          const std::vector<bst_feature_t> &fset,
-                          const RegTree &tree) override {
-    const MetaInfo &info = p_fmat->Info();
-    // fill in reverse map
-    feat2workindex_.resize(tree.param.num_feature);
-    std::fill(feat2workindex_.begin(), feat2workindex_.end(), -1);
-    work_set_.clear();
-    for (auto fidx : fset) {
-      if (feat_helper_.Type(fidx) == 2) {
-        feat2workindex_[fidx] = static_cast<int>(work_set_.size());
-        work_set_.push_back(fidx);
-      } else {
-        feat2workindex_[fidx] = -2;
-      }
-    }
-    const size_t work_set_size = work_set_.size();
-
-    sketchs_.resize(this->qexpand_.size() * work_set_size);
-    for (auto& sketch : sketchs_) {
-      sketch.Init(info.num_row_, this->param_.sketch_eps);
-    }
-    // initialize the summary array
-    summary_array_.resize(sketchs_.size());
-    // setup maximum size
-    unsigned max_size = this->param_.MaxSketchSize();
-    for (size_t i = 0; i < sketchs_.size(); ++i) {
-      summary_array_[i].Reserve(max_size);
-    }
-    {
-      // get summary
-      thread_sketch_.resize(ctx_->Threads());
-
-      // TWOPASS: use the real set + split set in the column iteration.
-      this->SetDefaultPostion(p_fmat, tree);
-      work_set_.insert(work_set_.end(), fsplit_set_.begin(), fsplit_set_.end());
-      std::sort(work_set_.begin(), work_set_.end());
-      work_set_.resize(std::unique(work_set_.begin(), work_set_.end()) - work_set_.begin());
-
-      // start accumulating statistics
-      for (const auto &batch : p_fmat->GetBatches<SortedCSCPage>()) {
-        // TWOPASS: use the real set + split set in the column iteration.
-        this->CorrectNonDefaultPositionByBatch(batch, fsplit_set_, tree);
-        auto page = batch.GetView();
-        // start enumeration
-        common::ParallelFor(work_set_.size(), ctx_->Threads(), common::Sched::Dyn(1),
-                            [&](auto i) {
-                              int fid = work_set_[i];
-                              int offset = feat2workindex_[fid];
-                              if (offset >= 0) {
-                                this->UpdateSketchCol(gpair, page[fid], tree, work_set_size, offset,
-                                                      &thread_sketch_[omp_get_thread_num()]);
-                              }
-                            });
-      }
-      for (size_t i = 0; i < sketchs_.size(); ++i) {
-        common::WXQuantileSketch<bst_float, bst_float>::SummaryContainer out;
-        sketchs_[i].GetSummary(&out);
-        summary_array_[i].SetPrune(out, max_size);
-      }
-      CHECK_EQ(summary_array_.size(), sketchs_.size());
-    }
-    if (summary_array_.size() != 0) {
-      size_t nbytes = WXQSketch::SummaryContainer::CalcMemCost(max_size);
-      sreducer_.Allreduce(dmlc::BeginPtr(summary_array_), nbytes, summary_array_.size());
-    }
-    // now we get the final result of sketch, setup the cut
-    this->wspace_.cut.clear();
-    this->wspace_.rptr.clear();
-    this->wspace_.rptr.push_back(0);
-    for (size_t wid = 0; wid < this->qexpand_.size(); ++wid) {
-      for (unsigned int i : fset) {
-        int offset = feat2workindex_[i];
-        if (offset >= 0) {
-          const WXQSketch::Summary &a = summary_array_[wid * work_set_size + offset];
-          for (size_t i = 1; i < a.size; ++i) {
-            bst_float cpt = a.data[i].value - kRtEps;
-            if (i == 1 || cpt > this->wspace_.cut.back()) {
-              this->wspace_.cut.push_back(cpt);
-            }
-          }
-          // push a value that is greater than anything
-          if (a.size != 0) {
-            bst_float cpt = a.data[a.size - 1].value;
-            // this must be bigger than last value in a scale
-            bst_float last = cpt + fabs(cpt) + kRtEps;
-            this->wspace_.cut.push_back(last);
-          }
-          this->wspace_.rptr.push_back(static_cast<unsigned>(this->wspace_.cut.size()));
-        } else {
-          CHECK_EQ(offset, -2);
-          bst_float cpt = feat_helper_.MaxValue(i);
-          this->wspace_.cut.push_back(cpt + fabs(cpt) + kRtEps);
-          this->wspace_.rptr.push_back(static_cast<unsigned>(this->wspace_.cut.size()));
-        }
-      }
-      // reserve last value for global statistics
-      this->wspace_.cut.push_back(0.0f);
-      this->wspace_.rptr.push_back(static_cast<unsigned>(this->wspace_.cut.size()));
-    }
-    CHECK_EQ(this->wspace_.rptr.size(),
-             (fset.size() + 1) * this->qexpand_.size() + 1);
-  }
-
-  inline void UpdateHistCol(const std::vector<GradientPair> &gpair,
-                            const SparsePage::Inst &col,
-                            const MetaInfo &info,
-                            const RegTree &tree,
-                            const std::vector<bst_feature_t> &fset,
-                            bst_uint fid_offset,
-                            std::vector<HistEntry> *p_temp) {
-    if (col.size() == 0) return;
-    // initialize sbuilder for use
-    std::vector<HistEntry> &hbuilder = *p_temp;
-    hbuilder.resize(tree.param.num_nodes);
-    for (int const nid : this->qexpand_) {
-      const unsigned wid = this->node2workindex_[nid];
-      hbuilder[nid].istart = 0;
-      hbuilder[nid].hist = this->wspace_.hset[0][fid_offset + wid * (fset.size()+1)];
-    }
-    if (this->param_.cache_opt != 0) {
-      constexpr bst_uint kBuffer = 32;
-      bst_uint align_length = col.size() / kBuffer * kBuffer;
-      int buf_position[kBuffer];
-      GradientPair buf_gpair[kBuffer];
-      for (bst_uint j = 0; j < align_length; j += kBuffer) {
-        for (bst_uint i = 0; i < kBuffer; ++i) {
-          bst_uint ridx = col[j + i].index;
-          buf_position[i] = this->position_[ridx];
-          buf_gpair[i] = gpair[ridx];
-        }
-        for (bst_uint i = 0; i < kBuffer; ++i) {
-          const int nid = buf_position[i];
-          if (nid >= 0) {
-            hbuilder[nid].Add(col[j + i].fvalue, buf_gpair[i]);
-          }
-        }
-      }
-      for (bst_uint j = align_length; j < col.size(); ++j) {
-        const bst_uint ridx = col[j].index;
-        const int nid = this->position_[ridx];
-        if (nid >= 0) {
-          hbuilder[nid].Add(col[j].fvalue, gpair[ridx]);
-        }
-      }
-    } else {
-      for (const auto& c : col) {
-        const bst_uint ridx = c.index;
-        const int nid = this->position_[ridx];
-        if (nid >= 0) {
-          hbuilder[nid].Add(c.fvalue, gpair, ridx);
-        }
-      }
-    }
-  }
-  inline void UpdateSketchCol(const std::vector<GradientPair> &gpair,
-                              const SparsePage::Inst &col,
-                              const RegTree &tree,
-                              size_t work_set_size,
-                              bst_uint offset,
-                              std::vector<BaseMaker::SketchEntry> *p_temp) {
-    if (col.size() == 0) return;
-    // initialize sbuilder for use
-    std::vector<BaseMaker::SketchEntry> &sbuilder = *p_temp;
-    sbuilder.resize(tree.param.num_nodes);
-    for (int const nid : this->qexpand_) {
-      const unsigned wid = this->node2workindex_[nid];
-      sbuilder[nid].sum_total = 0.0f;
-      sbuilder[nid].sketch = &sketchs_[wid * work_set_size + offset];
-    }
-    // first pass, get sum of weight, TODO, optimization to skip first pass
-    for (const auto& c : col) {
-        const bst_uint ridx = c.index;
-        const int nid = this->position_[ridx];
-        if (nid >= 0) {
-          sbuilder[nid].sum_total += gpair[ridx].GetHess();
-      }
-    }
-    // if only one value, no need to do second pass
-    if (col[0].fvalue  == col[col.size()-1].fvalue) {
-      for (int const nid : this->qexpand_) {
-        sbuilder[nid].sketch->Push(
-            col[0].fvalue, static_cast<bst_float>(sbuilder[nid].sum_total));
-      }
-      return;
-    }
-    // two pass scan
-    unsigned max_size = this->param_.MaxSketchSize();
-    for (int const nid : this->qexpand_) {
-      sbuilder[nid].Init(max_size);
-    }
-    // second pass, build the sketch
-    if (this->param_.cache_opt != 0) {
-      constexpr bst_uint kBuffer = 32;
-      bst_uint align_length = col.size() / kBuffer * kBuffer;
-      int buf_position[kBuffer];
-      bst_float buf_hess[kBuffer];
-      for (bst_uint j = 0; j < align_length; j += kBuffer) {
-        for (bst_uint i = 0; i < kBuffer; ++i) {
-          bst_uint ridx = col[j + i].index;
-          buf_position[i] = this->position_[ridx];
-          buf_hess[i] = gpair[ridx].GetHess();
-        }
-        for (bst_uint i = 0; i < kBuffer; ++i) {
-          const int nid = buf_position[i];
-          if (nid >= 0) {
-            sbuilder[nid].Push(col[j + i].fvalue, buf_hess[i], max_size);
-          }
-        }
-      }
-      for (bst_uint j = align_length; j < col.size(); ++j) {
-        const bst_uint ridx = col[j].index;
-        const int nid = this->position_[ridx];
-        if (nid >= 0) {
-          sbuilder[nid].Push(col[j].fvalue, gpair[ridx].GetHess(), max_size);
-        }
-      }
-    } else {
-      for (const auto& c : col) {
-        const bst_uint ridx = c.index;
-        const int nid = this->position_[ridx];
-        if (nid >= 0) {
-          sbuilder[nid].Push(c.fvalue, gpair[ridx].GetHess(), max_size);
-        }
-      }
-    }
-    for (int const nid : this->qexpand_) { sbuilder[nid].Finalize(max_size); }
-  }
-  // cached dmatrix where we initialized the feature on.
-  const DMatrix* cache_dmatrix_{nullptr};
-  // feature helper
-  BaseMaker::FMetaHelper feat_helper_;
-  // temp space to map feature id to working index
-  std::vector<int> feat2workindex_;
-  // set of index from fset that are current work set
-  std::vector<bst_feature_t> work_set_;
-  // set of index from that are split candidates.
-  std::vector<bst_uint> fsplit_set_;
-  // thread temp data
-  std::vector<std::vector<BaseMaker::SketchEntry> > thread_sketch_;
-  // used to hold statistics
-  std::vector<std::vector<GradStats> > thread_stats_;
-  // used to hold start pointer
-  std::vector<std::vector<HistEntry> > thread_hist_;
-  // node statistics
-  std::vector<GradStats> node_stats_;
-  // summary array
-  std::vector<WXQSketch::SummaryContainer> summary_array_;
-  // reducer for summary
-  rabit::SerializeReducer<WXQSketch::SummaryContainer> sreducer_;
-  // per node, per feature sketch
-  std::vector<common::WXQuantileSketch<bst_float, bst_float> > sketchs_;
-};
-
-XGBOOST_REGISTER_TREE_UPDATER(LocalHistMaker, "grow_local_histmaker")
-    .describe("Tree constructor that uses approximate histogram construction.")
-    .set_body([](GenericParameter const *ctx, ObjInfo) { return new CQHistMaker(ctx); });
-}  // namespace tree
-}  // namespace xgboost

src/tree/updater_refresh.cc

@@ -100,7 +100,8 @@ class TreeRefresher : public TreeUpdater {
         }
       });
     };
-    reducer_.Allreduce(dmlc::BeginPtr(stemp[0]), stemp[0].size(), lazy_get_stats);
+    rabit::Allreduce<rabit::op::Sum>(&dmlc::BeginPtr(stemp[0])->sum_grad, stemp[0].size() * 2,
+                                     lazy_get_stats);
     // rescale learning rate according to size of trees
     float lr = param_.learning_rate;
     param_.learning_rate = lr / trees.size();
@@ -154,8 +155,6 @@ class TreeRefresher : public TreeUpdater {
   }
   // training parameter
   TrainParam param_;
-  // reducer
-  rabit::Reducer<GradStats, GradStats::Reduce> reducer_;
 };
 
 XGBOOST_REGISTER_TREE_UPDATER(TreeRefresher, "refresh")