Jiaming Yuan
e6088366df
Export Python Interface for external memory. ( #7070 )
...
* Add Python iterator interface.
* Add tests.
* Add demo.
* Add documents.
* Handle empty dataset.
2021-07-22 15:15:53 +08:00
Jiaming Yuan
bd1f3a38f0
Rewrite sparse dmatrix using callbacks. ( #7092 )
...
- Reduce dependency on dmlc parsers and provide an interface for users to load data by themselves.
- Remove use of threaded iterator and IO queue.
- Remove `page_size`.
- Make sure the number of pages in memory is bounded.
- Make sure the cache can not be violated.
- Provide an interface for internal algorithms to process data asynchronously.
2021-07-16 12:33:31 +08:00
Jiaming Yuan
1c8fdf2218
Remove use of device_idx in dh::LaunchN. ( #7063 )
...
It's an unused parameter, removing it can make the CI log more readable.
2021-06-29 11:37:26 +08:00
Jiaming Yuan
43efadea2e
Deterministic data partitioning for external memory ( #6317 )
...
* Make external memory data partitioning deterministic.
* Change the meaning of `page_size` from bytes to number of rows.
* Design a data pool.
* Note for external memory.
* Enable unity build on Windows CI.
* Force garbage collect on test.
2020-11-11 06:11:06 +08:00
Jiaming Yuan
0012f2ef93
Upgrade clang-tidy on CI. ( #5469 )
...
* Correct all clang-tidy errors.
* Upgrade clang-tidy to 10 on CI.
Co-authored-by: Hyunsu Cho <chohyu01@cs.washington.edu>
2020-04-05 04:42:29 +08:00
Rory Mitchell
b745b7acce
Fix memory usage of device sketching ( #5407 )
2020-03-14 13:43:24 +13:00
Rory Mitchell
3ad4333b0e
Partial rewrite EllpackPage ( #5352 )
2020-03-11 10:15:53 +13:00
Rory Mitchell
a38e7bd19c
Sketching from adapters ( #5365 )
...
* Sketching from adapters
* Add weights test
2020-03-07 21:07:58 +13:00
Jiaming Yuan
655cf17b60
Predict on Ellpack. ( #5327 )
...
* Unify GPU prediction node.
* Add `PageExists`.
* Dispatch prediction on input data for GPU Predictor.
2020-02-23 06:27:03 +08:00
Rong Ou
0afcc55d98
Support multiple batches in gpu_hist ( #5014 )
...
* Initial external memory training support for GPU Hist tree method.
2019-11-16 14:50:20 +08:00
Rong Ou
5b1715d97c
Write ELLPACK pages to disk ( #4879 )
...
* add ellpack source
* add batch param
* extract function to parse cache info
* construct ellpack info separately
* push batch to ellpack page
* write ellpack page.
* make sparse page source reusable
2019-10-22 23:44:32 -04:00
