[doc] Add more detailed explanations for advanced objectives (#10283)

---------

Co-authored-by: Jiaming Yuan <jm.yuan@outlook.com>

R-package/R/xgb.train.R

@@ -102,6 +102,18 @@
 #'            It might be useful, e.g., for modeling total loss in insurance, or for any outcome that might be
 #'            \href{https://en.wikipedia.org/wiki/Tweedie_distribution#Applications}{Tweedie-distributed}.}
 #'   }
+#'
+#'  For custom objectives, one should pass a function taking as input the current predictions (as a numeric
+#'  vector or matrix) and the training data (as an `xgb.DMatrix` object) that will return a list with elements
+#'  `grad` and `hess`, which should be numeric vectors or matrices with number of rows matching to the numbers
+#'  of rows in the training data (same shape as the predictions that are passed as input to the function).
+#'  For multi-valued custom objectives, should have shape `[nrows, ntargets]`. Note that negative values of
+#'  the Hessian will be clipped, so one might consider using the expected Hessian (Fisher information) if the
+#'  objective is non-convex.
+#'
+#'  See the tutorials \href{https://xgboost.readthedocs.io/en/stable/tutorials/custom_metric_obj.html}{
+#'  Custom Objective and Evaluation Metric} and \href{https://xgboost.readthedocs.io/en/stable/tutorials/advanced_custom_obj}{
+#'  Advanced Usage of Custom Objectives} for more information about custom objectives.
 #'  }
 #'   \item \code{base_score} the initial prediction score of all instances, global bias. Default: 0.5
 #'   \item{ \code{eval_metric} evaluation metrics for validation data.

R-package/man/xgb.train.Rd

@@ -144,6 +144,18 @@ It might be useful, e.g., for modeling insurance claims severity, or for any out
 It might be useful, e.g., for modeling total loss in insurance, or for any outcome that might be
 \href{https://en.wikipedia.org/wiki/Tweedie_distribution#Applications}{Tweedie-distributed}.}
 }
+
+For custom objectives, one should pass a function taking as input the current predictions (as a numeric
+vector or matrix) and the training data (as an \code{xgb.DMatrix} object) that will return a list with elements
+\code{grad} and \code{hess}, which should be numeric vectors or matrices with number of rows matching to the numbers
+of rows in the training data (same shape as the predictions that are passed as input to the function).
+For multi-valued custom objectives, should have shape \verb{[nrows, ntargets]}. Note that negative values of
+the Hessian will be clipped, so one might consider using the expected Hessian (Fisher information) if the
+objective is non-convex.
+
+See the tutorials \href{https://xgboost.readthedocs.io/en/stable/tutorials/custom_metric_obj.html}{
+Custom Objective and Evaluation Metric} and \href{https://xgboost.readthedocs.io/en/stable/tutorials/advanced_custom_obj}{
+Advanced Usage of Custom Objectives} for more information about custom objectives.
 }
 \item \code{base_score} the initial prediction score of all instances, global bias. Default: 0.5
 \item{ \code{eval_metric} evaluation metrics for validation data.