* Add BigDenseMatrix
* ability to create DMatrix with bigger than Integer.MAX_VALUE size arrays
* uses sun.misc.Unsafe
* make DMatrix test work from a jar as well
* Add public group getter for java and scala
* Remove unnecessary param from javadoc
* Fix typo
* Fix another typo
* Add semicolon
* Fix javadoc return statement
* Fix missing return statement
* Add a unit test
* Make train in xgboost4j respect print params
Previously no setting in params argument of Booster::train would prevent
the Rabit.trackerPrint call. This can fill up a lot of screen space in
the case that many folds are being trained.
* Setting "silent" in this map to "true", "True", a non-zero integer, or
a string that can be parsed to such an int will prevent printing.
* Setting "verbose_eval" to "False" or "false" will prevent printing.
* Setting "verbose_eval" to an int (or a String parseable to an int) n
will result in printing every n steps, or no printing is n is zero.
This is to match the python behaviour described here:
https://www.kaggle.com/c/rossmann-store-sales/discussion/17499
* Fixed 'slient' typo in xgboost4j test
* private access on two methods
* Fix early stop with xgboost4j-spark
* Update XGBoost.java
* Update XGBoost.java
* Update XGBoost.java
To use -Float.MAX_VALUE as the lower bound, in case there is positive metric.
* Only update best score if the current score is better (no update when equal)
* Update xgboost-spark tutorial to fix early stopping docs.
* Updates to Booster to support other feature importances
* Add returns for Java methods
* Pass Scala style checks
* Pass Java style checks
* Fix indents
* Use class instead of enum
* Return map string double
* A no longer broken build, thanks to mvn package local build
* Add a unit test to increase code coverage back
* Address code review on main code
* Add more unit tests for different feature importance scores
* Address more CR
* add back train method but mark as deprecated
* add back train method but mark as deprecated
* add back train method but mark as deprecated
* add back train method but mark as deprecated
* fix scalastyle error
* fix scalastyle error
* fix scalastyle error
* fix scalastyle error
* wrap iterators
* enable copartition training and validationset
* add parameters
* converge code path and have init unit test
* enable multi evals for ranking
* unit test and doc
* update example
* fix early stopping
* address the offline comments
* udpate doc
* test eval metrics
* fix compilation issue
* fix example
* add back train method but mark as deprecated
* add back train method but mark as deprecated
* add back train method but mark as deprecated
* add back train method but mark as deprecated
* fix scalastyle error
* fix scalastyle error
* fix scalastyle error
* fix scalastyle error
* update version
* 0.82
* fix early stopping condition
* remove unused
* update comments
* udpate comments
* update test
* add back train method but mark as deprecated
* add back train method but mark as deprecated
* add back train method but mark as deprecated
* add back train method but mark as deprecated
* fix scalastyle error
* fix scalastyle error
* fix scalastyle error
* fix scalastyle error
* temp
* add method for classifier and regressor
* update tutorial
* address the comments
* update
* [jvm-packages] Prevent dispose being called twice when finalize
* Convert SIGSEGV to XGBoostError
* Avoid creating a new SBooster with the same JBooster
* Address CR Comments
* [jvm-packages] Train Booster from an existing model
* Align Scala API with Java API
* Existing model should not load rabit checkpoint
* Address minor comments
* Implement saving temporary boosters and loading previous booster
* Add more unit tests for loadPrevBooster
* Add params to XGBoostEstimator
* (1) Move repartition out of the temp model saving loop (2) Address CR comments
* Catch a corner case of training next model with fewer rounds
* Address comments
* Refactor newly added methods into TmpBoosterManager
* Add two files which is missing in previous commit
* Rename TmpBooster to checkpoint
* Allowed subsampling test from the training data frame/RDD
The implementation requires storing 1 - trainTestRatio points in memory
to make the sampling work.
An alternative approach would be to construct the full DMatrix and then
slice it deterministically into train/test. The peak memory consumption
of such scenario, however, is twice the dataset size.
* Removed duplication from 'XGBoost.train'
Scala callers can (and should) use names to supply a subset of
parameters. Method overloading is not required.
* Reuse XGBoost seed parameter to stabilize train/test splitting
* Added early stopping support to non-distributed XGBoost
Closes#1544
* Added early-stopping to distributed XGBoost
* Moved construction of 'watches' into a separate method
This commit also fixes the handling of 'baseMargin' which previously
was not added to the validation matrix.
* Addressed review comments
* Converted ml.dmlc.xgboost4j.LabeledPoint to Scala
This allows to easily integrate LabeledPoint with Spark DataFrame APIs,
which support encoding/decoding case classes out of the box. Alternative
solution would be to keep LabeledPoint in Java and make it a Bean by
generating boilerplate getters/setters. I have decided against that, even
thought the conversion in this PR implies a public API change.
I also had to remove the factory methods fromSparseVector and
fromDenseVector because a) they would need to be duplicated to support
overloaded calls with extra data (e.g. weight); and b) Scala would expose
them via mangled $.MODULE$ which looks ugly in Java.
Additionally, this commit makes it possible to switch to LabeledPoint in
all public APIs and effectively to pass initial margin/group as part of
the point. This seems to be the only reliable way of implementing distributed
learning with these data. Note that group size format used by single-node
XGBoost is not compatible with that scenario, since the partition split
could divide a group into two chunks.
* Switched to ml.dmlc.xgboost4j.LabeledPoint in RDD-based public APIs
Note that DataFrame-based and Flink APIs are not affected by this change.
* Removed baseMargin argument in favour of the LabeledPoint field
* Do a single pass over the partition in buildDistributedBoosters
Note that there is no formal guarantee that
val repartitioned = rdd.repartition(42)
repartitioned.zipPartitions(repartitioned.map(_ + 1)) { it1, it2, => ... }
would do a single shuffle, but in practice it seems to be always the case.
* Exposed baseMargin in DataFrame-based API
* Addressed review comments
* Pass baseMargin to XGBoost.trainWithDataFrame via params
* Reverted MLLabeledPoint in Spark APIs
As discussed, baseMargin would only be supported for DataFrame-based APIs.
* Cleaned up baseMargin tests
- Removed RDD-based test, since the option is no longer exposed via
public APIs
- Changed DataFrame-based one to check that adding a margin actually
affects the prediction
* Pleased Scalastyle
* Addressed more review comments
* Pleased scalastyle again
* Fixed XGBoost.fromBaseMarginsToArray
which always returned an array of NaNs even if base margin was not
specified. Surprisingly this only failed a few tests.
* add back train method but mark as deprecated
* fix scalastyle error
* first commit in scala binding for fast histo
* java test
* add missed scala tests
* spark training
* add back train method but mark as deprecated
* fix scalastyle error
* local change
* first commit in scala binding for fast histo
* local change
* fix df frame test
* [jvm-packages] Scala implementation of the Rabit tracker.
A Scala implementation of RabitTracker that is interface-interchangable with the
Java implementation, ported from `tracker.py` in the
[dmlc-core project](https://github.com/dmlc/dmlc-core).
* [jvm-packages] Updated Akka dependency in pom.xml.
* Refactored the RabitTracker directory structure.
* Fixed premature stopping of connection handler.
Added a new finite state "AwaitingPortNumber" to explicitly wait for the
worker to send the port, and close the connection. Stopping the actor
prematurely sends a TCP RST to the worker, causing the worker to crash
on AssertionError.
* Added interface IRabitTracker so that user can switch implementations.
* Default timeout duration changes.
* Dependency for Akka tests.
* Removed the main function of RabitTracker.
* A skeleton for testing Akka-based Rabit tracker.
* waitFor() in RabitTracker no longer throws exceptions.
* Completed unit test for the 'start' command of Rabit tracker.
* Preliminary support for Rabit Allreduce via JNI (no prepare function support yet.)
* Fixed the default timeout duration.
* Use Java container to avoid serialization issues due to intermediate wrappers.
* Added tests for Allreduce/model training using Scala Rabit tracker.
* Added spill-over unit test for the Scala Rabit tracker.
* Fixed a typo.
* Overhaul of RabitTracker interface per code review.
- Removed methods start() waitFor() (no arguments) from IRabitTracker.
- The timeout in start(timeout) is now worker connection timeout, as tcp
socket binding timeout is less intuitive.
- Dropped time unit from start(...) and waitFor(...) methods; the default
time unit is millisecond.
- Moved random port number generation into the RabitTrackerHandler.
- Moved all Rabit-related classes to package ml.dmlc.xgboost4j.scala.rabit.
* More code refactoring and comments.
* Unified timeout constants. Readable tracker status code.
* Add comments to indicate that allReduce is for tests only. Removed all other variants.
* Removed unused imports.
* Simplified signatures of training methods.
- Moved TrackerConf into parameter map.
- Changed GeneralParams so that TrackerConf becomes a standalone parameter.
- Updated test cases accordingly.
* Changed monitoring strategies.
* Reverted monitoring changes.
* Update test case for Rabit AllReduce.
* Mix in UncaughtExceptionHandler into IRabitTracker to prevent tracker from hanging due to exceptions thrown by workers.
* More comprehensive test cases for exception handling and worker connection timeout.
* Handle executor loss due to unknown cause: the newly spawned executor will attempt to connect to the tracker. Interrupt tracker in such case.
* Per code-review, removed training timeout from TrackerConf. Timeout logic must be implemented explicitly and externally in the driver code.
* Reverted scalastyle-config changes.
* Visibility scope change. Interface tweaks.
* Use match pattern to handle tracker_conf parameter.
* Minor clarification in JNI code.
* Clearer intent in match pattern to suppress warnings.
* Removed Future from constructor. Block in start() and waitFor() instead.
* Revert inadvertent comment changes.
* Removed debugging information.
* Updated test cases that are a bit finicky.
* Added comments on the reasoning behind the unit tests for testing Rabit tracker robustness.
* Fixed BufferUnderFlow bug in decoding tracker 'print' command.
* Merge conflicts resolution.
* [jvm-packages] Scala implementation of the Rabit tracker.
A Scala implementation of RabitTracker that is interface-interchangable with the
Java implementation, ported from `tracker.py` in the
[dmlc-core project](https://github.com/dmlc/dmlc-core).
* [jvm-packages] Updated Akka dependency in pom.xml.
* Refactored the RabitTracker directory structure.
* Fixed premature stopping of connection handler.
Added a new finite state "AwaitingPortNumber" to explicitly wait for the
worker to send the port, and close the connection. Stopping the actor
prematurely sends a TCP RST to the worker, causing the worker to crash
on AssertionError.
* Added interface IRabitTracker so that user can switch implementations.
* Default timeout duration changes.
* Dependency for Akka tests.
* Removed the main function of RabitTracker.
* A skeleton for testing Akka-based Rabit tracker.
* waitFor() in RabitTracker no longer throws exceptions.
* Completed unit test for the 'start' command of Rabit tracker.
* Preliminary support for Rabit Allreduce via JNI (no prepare function support yet.)
* Fixed the default timeout duration.
* Use Java container to avoid serialization issues due to intermediate wrappers.
* Added tests for Allreduce/model training using Scala Rabit tracker.
* Added spill-over unit test for the Scala Rabit tracker.
* Fixed a typo.
* Overhaul of RabitTracker interface per code review.
- Removed methods start() waitFor() (no arguments) from IRabitTracker.
- The timeout in start(timeout) is now worker connection timeout, as tcp
socket binding timeout is less intuitive.
- Dropped time unit from start(...) and waitFor(...) methods; the default
time unit is millisecond.
- Moved random port number generation into the RabitTrackerHandler.
- Moved all Rabit-related classes to package ml.dmlc.xgboost4j.scala.rabit.
* More code refactoring and comments.
* Unified timeout constants. Readable tracker status code.
* Add comments to indicate that allReduce is for tests only. Removed all other variants.
* Removed unused imports.
* Simplified signatures of training methods.
- Moved TrackerConf into parameter map.
- Changed GeneralParams so that TrackerConf becomes a standalone parameter.
- Updated test cases accordingly.
* Changed monitoring strategies.
* Reverted monitoring changes.
* Update test case for Rabit AllReduce.
* Mix in UncaughtExceptionHandler into IRabitTracker to prevent tracker from hanging due to exceptions thrown by workers.
* More comprehensive test cases for exception handling and worker connection timeout.
* Handle executor loss due to unknown cause: the newly spawned executor will attempt to connect to the tracker. Interrupt tracker in such case.
* Per code-review, removed training timeout from TrackerConf. Timeout logic must be implemented explicitly and externally in the driver code.
* Reverted scalastyle-config changes.
* Visibility scope change. Interface tweaks.
* Use match pattern to handle tracker_conf parameter.
* Minor clarification in JNI code.
* Clearer intent in match pattern to suppress warnings.
* Removed Future from constructor. Block in start() and waitFor() instead.
* Revert inadvertent comment changes.
* Removed debugging information.
* Updated test cases that are a bit finicky.
* Added comments on the reasoning behind the unit tests for testing Rabit tracker robustness.
* create dmatrix with specified missing value
* update dmlc-core
* support for predict method in spark package
repartitioning
work around
* add more elements to work around training set empty partition issue
