68 Commits

Author SHA1 Message Date
Jiaming Yuan
7968c0d051
Test on s390x. (#7038)
* Fix && remove unused parameter.
2021-06-18 14:55:08 +08:00
Jiaming Yuan
34df1f588b
Reduce Travis environment setup time. (#6912)
* Remove unused r from travis.
* Don't update homebrew.
* Don't install indirect/unused dependencies like libgit2, wget, openssl.
* Move graphviz installation to conda.
2021-04-30 09:02:40 +08:00
Philip Hyunsu Cho
84b726ef53
Vendor libgomp in the manylinux Python wheel (#6461)
* Vendor libgomp in the manylinux2014_aarch64 wheel

* Use vault repo, since CentOS 6 has reached End-of-Life on Nov 30

* Vendor libgomp in the manylinux2010_x86_64 wheel

* Run verification step inside the container
2020-12-03 19:55:32 -08:00
Philip Hyunsu Cho
c763b50dd0
[CI] Upgrade to MacOS Mojave image (#6406) 2020-11-18 20:29:10 -08:00
Jiaming Yuan
2686d32a36
Skip dask tests on ARM. (#6267)
Co-authored-by: Hyunsu Cho <chohyu01@cs.washington.edu>
2020-10-26 15:09:05 +08:00
Philip Hyunsu Cho
7f6ed5780c
[CI] Build a Python wheel for aarch64 platform (#6253) 2020-10-18 22:35:19 -07:00
odidev
a2fea33103
Added arm64 job in Travis-CI (#6200)
Signed-off-by: odidev <odidev@puresoftware.com>
2020-10-07 15:02:09 +08:00
Philip Hyunsu Cho
d0ccb13d09
Work around a compiler bug in MacOS AppleClang 11 (#6103)
* Workaround a compiler bug in MacOS AppleClang

* [CI] Run C++ test with MacOS Catalina + AppleClang 11.0.3

* [CI] Migrate cmake_test on MacOS from Travis CI to GitHub Actions

* Install OpenMP runtime

* [CI] Use CMake to locate lz4 lib
2020-09-09 21:21:55 -07:00
Qi Zhang
989ddd036f
Swap byte-order in binary serializer to support big-endian arch (#5813)
* fixed some endian issues

* Use dmlc::ByteSwap() to simplify code

* Fix lint check

* [CI] Add test for s390x

* Download latest CMake on s390x

* Fix a bug in my code

* Save magic number in dmatrix with byteswap on big-endian machine

* Save version in binary with byteswap on big-endian machine

* Load scalar with byteswap in MetaInfo

* Add a debugging message

* Handle arrays correctly when byteswapping

* EOF can also be 255

* Handle magic number in MetaInfo carefully

* Skip Tree.Load test for big-endian, since the test manually builds little-endian binary model

* Handle missing packages in Python tests

* Don't use boto3 in model compatibility tests

* Add s390 Docker file for local testing

* Add model compatibility tests

* Add R compatibility test

* Revert "Add R compatibility test"

This reverts commit c2d2bdcb7dbae133cbb927fcd20f7e83ee2b18a8.

Co-authored-by: Qi Zhang <q.zhang@ibm.com>
Co-authored-by: Hyunsu Cho <chohyu01@cs.washington.edu>
2020-08-18 14:47:17 -07:00
Rory Mitchell
b47b5ac771
Use hypothesis (#5759)
* Use hypothesis

* Allow int64 array interface for groups

* Add packages to Windows CI

* Add to travis

* Make sure device index is set correctly

* Fix dask-cudf test

* appveyor
2020-06-16 12:45:59 +12:00
Philip Hyunsu Cho
a23de1c108
[CI] Grant public read access to Mac OSX wheels (#5602) 2020-04-25 11:51:26 -07:00
Philip Hyunsu Cho
f68155de6c
Fix compilation on Mac OSX High Sierra (10.13) (#5597)
* Fix compilation on Mac OSX High Sierra

* [CI] Build Mac OSX binary wheel using Travis CI
2020-04-25 10:53:03 -07:00
Jiaming Yuan
780de49ddb
Resolve travis failure. (#5445)
* Install dependencies by pip.
2020-03-27 19:37:58 +08:00
Jiaming Yuan
595a00466d
Rewrite setup.py. (#5271)
The setup.py is rewritten.  This new script uses only Python code and provide customized
implementation of setuptools commands.  This way users can run most of setuptools commands
just like any other Python libraries.

* Remove setup_pip.py
* Remove soft links.
* Define customized commands.
* Remove shell script.
* Remove makefile script.
* Update the doc for building from source.
2020-02-04 13:35:42 +08:00
Philip Hyunsu Cho
4240daed4e
Make pip install xgboost*.tar.gz work by fixing build-python.sh (#5241)
* Make pip install xgboost*.tar.gz work by fixing build-python.sh

* Simplify install doc

* Add test

* Install Miniconda for Linux target too

* Build XGBoost only once in sdist

* Try importing xgboost after installation

* Don't set PYTHONPATH env var for sdist test
2020-01-28 23:18:23 -08:00
Rory Mitchell
9c56480c61
Support dmatrix construction from cupy array (#5206) 2020-01-22 13:15:27 +13:00
Philip Hyunsu Cho
9b0af6e882 Enable OpenMP with Apple Clang (Mac default compiler) (#5146)
* Add OpenMP as CMake target

* Require CMake 3.12, to allow linking OpenMP target to objxgboost

* Specify OpenMP compiler flag for CUDA host compiler

* Require CMake 3.16+ if the OS is Mac OSX

* Use AppleClang in Mac tests.

* Update dmlc-core
2019-12-26 16:53:12 +08:00
Jiaming Yuan
d667ea9335
[CI] Fix Travis tests. (#5062)
- Install wget explicitly to match openssl.
- Install CMake explicitly.
- Use newer miniconda link.
- Reenable unittests.
- gcc@9 + xcode@10 for osx due to missing <_stdio.h>.  Other versions of gcc should also work.  But as homebrew pour gcc@9 after update by default, so I just stick with latest version.
- Disabled one external memory test for OSX.  Not sure about the thread implementation in there and fixing external memory is beyond the scope of this PR.
- Use Python3 with conda in jvm package.
2019-11-25 03:32:10 +08:00
Jiaming Yuan
7663de956c
Run training with empty DMatrix. (#4990)
This makes GPU Hist robust in distributed environment as some workers might not
be associated with any data in either training or evaluation.

* Disable rabit mock test for now: See #5012 .

* Disable dask-cudf test at prediction for now: See #5003

* Launch dask job for all workers despite they might not have any data.
* Check 0 rows in elementwise evaluation metrics.

   Using AUC and AUC-PR still throws an error.  See #4663 for a robust fix.

* Add tests for edge cases.
* Add `LaunchKernel` wrapper handling zero sized grid.
* Move some parts of allreducer into a cu file.
* Don't validate feature names when the booster is empty.

* Sync number of columns in DMatrix.

  As num_feature is required to be the same across all workers in data split
  mode.

* Filtering in dask interface now by default syncs all booster that's not
empty, instead of using rank 0.

* Fix Jenkins' GPU tests.

* Install dask-cuda from source in Jenkins' test.

  Now all tests are actually running.

* Restore GPU Hist tree synchronization test.

* Check UUID of running devices.

  The check is only performed on CUDA version >= 10.x, as 9.x doesn't have UUID field.

* Fix CMake policy and project variables.

  Use xgboost_SOURCE_DIR uniformly, add policy for CMake >= 3.13.

* Fix copying data to CPU

* Fix race condition in cpu predictor.

* Fix duplicated DMatrix construction.

* Don't download extra nccl in CI script.
2019-11-06 16:13:13 +08:00
Chen Qin
b29b8c2f34 [jvm-packages] update rabit, surface new changes to spark, add parity and failure tests (#4966)
* [phase 1] expose sets of rabit configurations to spark layer

* add back mutable import

* disable ring_mincount till https://github.com/dmlc/rabit/pull/106d

* Revert "disable ring_mincount till https://github.com/dmlc/rabit/pull/106d"

This reverts commit 65e95a98e24f5eb53c6ba9ef9b2379524258984d.

* apply latest rabit

* fix build error

* apply https://github.com/dmlc/xgboost/pull/4880

* downgrade cmake in rabit

* point to rabit with DMLC_ROOT fix

* relative path of rabit install prefix

* split rabit parameters to another trait

* misc

* misc

* Delete .classpath

* Delete .classpath

* Delete .classpath

* Update XGBoostClassifier.scala

* Update XGBoostRegressor.scala

* Update GeneralParams.scala

* Update GeneralParams.scala

* Update GeneralParams.scala

* Update GeneralParams.scala

* Delete .classpath

* Update RabitParams.scala

* Update .gitignore

* Update .gitignore

* apply rabitParams to training

* use string as rabit parameter value type

* cleanup

* add rabitEnv check

* point to dmlc/rabit

* per feedback

* update private scope

* misc

* update rabit

* add rabit_timtout, fix failing test.

* split tests

* allow build jvm with rabit mock

* pass mock failures to rabit with test

* add mock error and graceful handle rabit assertion error test

* split mvn test

* remove sign for test

* update rabit

* build jvm_packages with rabit mock

* point back to dmlc/rabit

* per feedback, update scala header

* cleanup pom

* per feedback

* try fix lint

* fix lint

* per feedback, remove bootstrap_cache

* per feedback 2

* try replace dev profile with passing mvn property

* fix build error

* remove mvn property and replace with env setting to build test jar

* per feedback

* revert copyright headlines, point to dmlc/rabit

* revert python lint

* remove multiple failure test case as retry is not enabled in spark

* Update core.py

* Update core.py

* per feedback, style fix
2019-11-01 14:21:19 -07:00
Jiaming Yuan
010b8f1428 Revert "[jvm-packages] update rabit, surface new changes to spark, add parity and failure tests (#4876)" (#4965)
This reverts commit 86ed01c4bbecef66e1bc4d02fb13116bd6130fae.
2019-10-18 14:02:35 -07:00
Chen Qin
86ed01c4bb [jvm-packages] update rabit, surface new changes to spark, add parity and failure tests (#4876)
* Expose sets of rabit configurations to spark layer
2019-10-18 15:07:31 -04:00
Oleksandr Pryimak
80977182c5 Use bundled gtest (#4900)
* Suggest to use gtest bundled with dmlc

* Use dmlc bundled gtest in all CI scripts

* Make clang-tidy to use dmlc embedded gtest
2019-10-09 16:26:19 -07:00
Jiaming Yuan
b8433c455a
Rewrite Dask interface. (#4819) 2019-09-25 01:30:14 -04:00
Jiaming Yuan
ab357dd41c
Remove plugin, cuda related code in automake & autoconf files (#4789)
* Build plugin example with CMake.

* Remove plugin, cuda related code in automake & autoconf files.

* Fix typo in GPU doc.
2019-08-18 16:54:34 -04:00
Philip Hyunsu Cho
ea850ecd20
[CI] Refactor Jenkins CI pipeline + migrate all Linux tests to Jenkins (#4401)
* All Linux tests are now in Jenkins CI
* Tests are now de-coupled from builds. We can now build XGBoost with one version of CUDA/JDK and test it with another version of CUDA/JDK
* Builds (compilation) are significantly faster because 1) They use C5 instances with faster CPU cores; and 2) build environment setup is cached using Docker containers
2019-04-26 18:39:12 -07:00
Philip Hyunsu Cho
bbe0dbd7ec
Migrate pylint check to Python 3 (#4381)
* Migrate lint to Python 3

* Fix lint errors

* Use Miniconda3 to use Python 3.7

* Use latest pylint and astroid
2019-04-21 01:01:54 -07:00
Jiaming Yuan
207f058711 Refactor CMake scripts. (#4323)
* Refactor CMake scripts.

* Remove CMake CUDA wrapper.
* Bump CMake version for CUDA.
* Use CMake to handle Doxygen.
* Split up CMakeList.
* Export install target.
* Use modern CMake.
* Remove build.sh
* Workaround for gpu_hist test.
* Use cmake 3.12.

* Revert machine.conf.

* Move CLI test to gpu.

* Small cleanup.

* Support using XGBoost as submodule.

* Fix windows

* Fix cpp tests on Windows

* Remove duplicated find_package.
2019-04-15 10:08:12 -07:00
James Lamb
edae664afb [r-package] cut CI-time dependency on craigcitro/r-travis (fixes #4348) (#4353)
* [r-package] cut CI-time dependency on craigcitro/r-travis (fixes #4348)

* Install R

* Install R on OSX

* Remove gfortran symlink

* Specify CRAN repo

* added more R dependencies needed for testing

* removed heavy R dependencies in CI

* fixed bug in env var, removed unnecessary apt installs of R

* fix to R installs
2019-04-12 00:22:48 -07:00
sriramch
2f7087eba1 Improve HostDeviceVector exception safety (#4301)
* make the assignments of HostDeviceVector exception safe.
* storing a dummy GPUDistribution instance in HDV for CPU based code.
* change testxgboost binary location to build directory.
2019-03-31 22:48:58 +08:00
Philip Hyunsu Cho
7aed8f3d48
[CI] Upgrade to GCC 5.3.1, CMake 3.6.0 (#4306)
* Upgrade to GCC 5.3.1, CMake 3.6.0

* <regex> is now okay
2019-03-28 00:21:21 -07:00
Jiaming Yuan
cecbe0cf71 Fix test_gpu_coordinate. (#3974)
* Fix test_gpu_coordinate.

* Use `gpu_coord_descent` in test.
* Reduce number of running rounds.

* Remove nthread.

* Use githubusercontent for r-appveyor.

* Use githubusercontent in travis r tests.
2019-02-19 14:09:10 -08:00
Philip Hyunsu Cho
37ddfd7d6e
Fix broken R test: Install Homebrew GCC (#4142)
* Fix broken R test: Install Homebrew GCC

Missing GCC Fortran causes installation failure of a dependency package
(igraph)

* Register gfortran system-wide

* Use correct keg

* Set env vars to change compiler choice

* Do not break other Mac builds

* Nuclear option: symlink gfortran

* Use /usr/local/bin instead of /usr/bin

* Symlink library path too

* Update run_test.sh
2019-02-15 07:23:05 -08:00
Philip Hyunsu Cho
a1c35cadf0
Fix failing Travis CI on Mac (#4086)
* Fix failing Travis CI on Mac

Use Homebrew Addon + latest Mac image

* Use long command for pytest

* Downgrade OSX image to xcode9.3, to use Java 8

* Install pytest in Python 2 environment

* Remove clang-tidy from Travis
2019-01-30 09:43:57 -08:00
Jiaming Yuan
fe999bf968
Add back python2 tests for Travis light weight tests. (#3901) 2018-11-15 22:17:35 +13:00
Jiaming Yuan
2ea0f887c1
Refactor Python tests. (#3897)
* Deprecate nose tests.
* Format python tests.
2018-11-15 13:56:33 +13:00
Philip Hyunsu Cho
20d5abf919
Disallow std::regex since it's not supported by GCC 4.8.x (#3870) 2018-11-05 22:57:04 -08:00
Philip Hyunsu Cho
55ee9a92a1
Fix Python environment for distributed unit tests (#3806) 2018-10-18 00:12:02 -07:00
Rory Mitchell
34522d56f0
Allow plug-ins to be built by cmake (#3752)
* Remove references to AVX code.

* Allow plugins to be built by cmake
2018-10-04 22:03:52 +13:00
Philip Hyunsu Cho
51478a39c9
Fix #3730: scikit-learn 0.20 compatibility fix (#3731)
* Fix #3730: scikit-learn 0.20 compatibility fix

sklearn.cross_validation has been removed from scikit-learn 0.20,
so replace it with sklearn.model_selection

* Display test names for Python tests for clarity
2018-09-27 15:03:05 -07:00
trivialfis
cf2d86a4f6 Add travis sanitizers tests. (#3557)
* Add travis sanitizers tests.

* Add gcc-7 in Travis.
* Add SANITIZER_PATH for CMake.
* Enable sanitizer tests in Travis.

* Fix memory leaks in tests.

* Fix all memory leaks reported by Address Sanitizer.
* tests/cpp/helpers.h/CreateDMatrix now returns raw pointer.
2018-08-19 16:40:30 +12:00
trivialfis
55caad6e49 Remove redundant FindGTest.cmake. (#3533)
During removal of FindGTest.cmake, also

* Fix gtest include dirs.
* Remove some blanks and use PWD for gtest dir.
2018-08-07 10:08:08 +12:00
Philip Hyunsu Cho
7fefd6865d
Fix #3402: wrong fid crashes distributed algorithm (#3535)
* Fix #3402: wrong fid crashes distributed algorithm

The bug was introduced by the recent DMatrix refactor (#3301). It was partially
fixed by #3408 but the example in #3402 was still failing. The example in #3402
will succeed after this fix is applied.

* Explicitly specify "this" to prevent compile error

* Add regression test

* Add distributed test to Travis matrix

* Install kubernetes Python package as dependency of dmlc tracker

* Add Python dependencies

* Add compile step

* Reduce size of regression test case

* Further reduce size of test
2018-08-04 19:20:04 -07:00
Philip Hyunsu Cho
cafc621914
Do not unzip google test archive if exists (#3416) 2018-06-28 04:10:39 +00:00
Philip Hyunsu Cho
e2743548ed
Fix wget for google tests in tests (#3414)
CI tests were failing because wget prompts "the user" for a response
whenever the google test archive is already on the disk.

Fix: Use `-nc` option to skip download when the archive already
exists
2018-06-27 22:12:56 +00:00
PSEUDOTENSOR / Jonathan McKinney
9ac163d0bb Allow import via python datatable. (#3272)
* Allow import via python datatable.

* Write unit tests

* Refactor dt API functions

* Refactor python code

* Lint fixes

* Address review comments
2018-06-20 13:16:18 -07:00
Rory Mitchell
a96039141a
Dmatrix refactor stage 1 (#3301)
* Use sparse page as singular CSR matrix representation

* Simplify dmatrix methods

* Reduce statefullness of batch iterators

* BREAKING CHANGE: Remove prob_buffer_row parameter. Users are instead recommended to sample their dataset as a preprocessing step before using XGBoost.
2018-06-07 10:25:58 +12:00
Philip Hyunsu Cho
9a8211f668
Update dmlc-core submodule (#3221)
* Update dmlc-core submodule

* Fix dense_parser to work with the latest dmlc-core

* Specify location of Google Test

* Add more source files in dmlc-minimum to get latest dmlc-core working

* Update dmlc-core submodule
2018-05-09 18:55:29 -07:00
Rory Mitchell
ccf80703ef
Clang-tidy static analysis (#3222)
* Clang-tidy static analysis

* Modernise checks

* Google coding standard checks

* Identifier renaming according to Google style
2018-04-19 18:57:13 +12:00
Scott Lundberg
d878c36c84 Add SHAP interaction effects, fix minor bug, and add cox loss (#3043)
* Add interaction effects and cox loss

* Minimize whitespace changes

* Cox loss now no longer needs a pre-sorted dataset.

* Address code review comments

* Remove mem check, rename to pred_interactions, include bias

* Make lint happy

* More lint fixes

* Fix cox loss indexing

* Fix main effects and tests

* Fix lint

* Use half interaction values on the off-diagonals

* Fix lint again
2018-02-07 20:38:01 -06:00