Detailed API#

To go back to the main page, click here.

mauve.compute_mauve(p_features=None, q_features=None, p_tokens=None, q_tokens=None, p_text=None, q_text=None, num_buckets='auto', pca_max_data=-1, kmeans_explained_var=0.9, kmeans_num_redo=5, kmeans_max_iter=500, featurize_model_name='gpt2-large', device_id=-1, max_text_length=1024, divergence_curve_discretization_size=25, mauve_scaling_factor=5, verbose=False, seed=25, batch_size=1, use_float64=False)#

Compute the MAUVE score between two text generations P and Q.

P is either specified as p_features, p_tokens, or p_text. Same with Q.

Parameters:

  • p_features -- numpy.ndarray of shape (n, d), where n is the number of generations.

  • q_features -- numpy.ndarray of shape (n, d), where n is the number of generations.

  • p_tokens -- list of length n, each entry is torch.LongTensor of shape (1, length).

  • q_tokens -- list of length n, each entry is torch.LongTensor of shape (1, length).

  • p_text -- list of length n, each entry is a string.

  • q_text -- list of length n, each entry is a string.

  • num_buckets -- the size of the histogram to quantize P and Q. Options: 'auto' (default, which is n/10) or an integer.

  • pca_max_data -- the number data points to use for PCA. If -1, use all the data. Default -1.

  • kmeans_explained_var -- amount of variance of the data to keep in dimensionality reduction by PCA. Default 0.9.

  • kmeans_num_redo -- number of times to redo k-means clustering (the best objective is kept). Default 5. Try reducing this to 1 in order to reduce running time.

  • kmeans_max_iter -- maximum number of k-means iterations. Default 500. Try reducing this to 100 in order to reduce running time.

  • featurize_model_name -- name of the model from which features are obtained. Default 'gpt2-large'. We support all models which can be loaded from transformers.AutoModel.from_pretrained(featurize_model_name).

  • device_id -- Device for featurization. Supply gpu_id (e.g. 0 or 3) to use GPU or -1 to use CPU.

  • max_text_length -- maximum number of tokens to consider. Default 1024.

  • divergence_curve_discretization_size -- Number of points to consider on the divergence curve. Default 25. Larger values do not offer much of a difference.

  • mauve_scaling_factor -- The constant``c`` from the paper. Default 5. See Best Practices for details.

  • verbose -- If True, print running time updates.

  • seed -- random seed to initialize k-means cluster assignments.

  • batch_size -- Batch size for feature extraction

Returns:

an object with fields p_hist, q_hist, divergence_curve and mauve.

  • out.mauve is a number between 0 and 1, the MAUVE score. Higher values means P is closer to Q.

  • out.frontier_integral, a number between 0 and 1. Lower values mean that P is closer to Q.

  • out.p_hist is the obtained histogram for P. Same for out.q_hist.

  • out.divergence_curve contains the points in the divergence curve. It is of shape (m, 2), where m is divergence_curve_discretization_size