Some guidelines on device memory usage (#5038)

* Add memory usage demo

* Update documentation

demo/gpu_acceleration/README.md

@@ -1,7 +1,5 @@
 # GPU Acceleration Demo
 
-This demo shows how to train a model on the [forest cover type](https://archive.ics.uci.edu/ml/datasets/covertype) dataset using GPU acceleration. The forest cover type dataset has 581,012 rows and 54 features, making it time consuming to process. We compare the run-time and accuracy of the GPU and CPU histogram algorithms.
+`cover_type.py` shows how to train a model on the [forest cover type](https://archive.ics.uci.edu/ml/datasets/covertype) dataset using GPU acceleration. The forest cover type dataset has 581,012 rows and 54 features, making it time consuming to process. We compare the run-time and accuracy of the GPU and CPU histogram algorithms.
 
-This demo requires the [GPU plug-in](https://xgboost.readthedocs.io/en/latest/gpu/index.html) to be built and installed.
-
-The dataset is automatically loaded via the sklearn script. 
+`memory.py` shows how to repeatedly train xgboost models while freeing memory between iterations.

demo/gpu_acceleration/memory.py

@@ -0,0 +1,51 @@
+import xgboost as xgb
+import numpy as np
+import time
+import pickle
+import GPUtil
+
+n = 10000
+m = 1000
+X = np.random.random((n, m))
+y = np.random.random(n)
+
+param = {'objective': 'binary:logistic',
+         'tree_method': 'gpu_hist'
+         }
+iterations = 5
+dtrain = xgb.DMatrix(X, label=y)
+
+# High memory usage
+# active bst objects with device memory persist across iterations
+boosters = []
+for i in range(iterations):
+    bst = xgb.train(param, dtrain)
+    boosters.append(bst)
+
+print("Example 1")
+GPUtil.showUtilization()
+del boosters
+
+# Better memory usage
+# The bst object can be destroyed by the python gc, freeing device memory
+# The gc may not immediately free the object, so more than one booster can be allocated at a time
+boosters = []
+for i in range(iterations):
+    bst = xgb.train(param, dtrain)
+    boosters.append(pickle.dumps(bst))
+
+print("Example 2")
+GPUtil.showUtilization()
+del boosters
+
+# Best memory usage
+# The gc explicitly frees the booster before starting the next iteration
+boosters = []
+for i in range(iterations):
+    bst = xgb.train(param, dtrain)
+    boosters.append(pickle.dumps(bst))
+    del bst
+
+print("Example 3")
+GPUtil.showUtilization()
+del boosters