nxcals.api.extraction.data.builders.DataFrame.approxQuantile

DataFrame.approxQuantile(col: str, probabilities: Union[List[float], Tuple[float]], relativeError: float) → List[float]
DataFrame.approxQuantile(col: Union[List[str], Tuple[str]], probabilities: Union[List[float], Tuple[float]], relativeError: float) → List[List[float]]

Calculates the approximate quantiles of numerical columns of a DataFrame.

The result of this algorithm has the following deterministic bound: If the DataFrame has N elements and if we request the quantile at probability p up to error err, then the algorithm will return a sample x from the DataFrame so that the exact rank of x is close to (p * N). More precisely,

floor((p - err) * N) <= rank(x) <= ceil((p + err) * N).

This method implements a variation of the Greenwald-Khanna algorithm (with some speed optimizations). The algorithm was first present in [[https://doi.org/10.1145/375663.375670 Space-efficient Online Computation of Quantile Summaries]] by Greenwald and Khanna.

New in version 2.0.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters:

  • col (str, tuple or list) –

    Can be a single column name, or a list of names for multiple columns.

    Changed in version 2.2.0: Added support for multiple columns.

  • probabilities (list or tuple) – a list of quantile probabilities Each number must belong to [0, 1]. For example 0 is the minimum, 0.5 is the median, 1 is the maximum.

  • relativeError (float) – The relative target precision to achieve (>= 0). If set to zero, the exact quantiles are computed, which could be very expensive. Note that values greater than 1 are accepted but gives the same result as 1.

Returns:

the approximate quantiles at the given probabilities.

  • If the input col is a string, the output is a list of floats.

  • If the input col is a list or tuple of strings, the output is also a

    list, but each element in it is a list of floats, i.e., the output is a list of list of floats.

Return type:

list

Notes

Null values will be ignored in numerical columns before calculation. For columns only containing null values, an empty list is returned.