SQwash API summary

To go back to the main page, click here.

Module API of reducers:

class sqwash.MeanReducer

Reduce a batch of values by their mean.

Parameters:

batch (torch.Tensor) – Tensor of values to reduce. Reshaped into a single dimension.

class sqwash.SuperquantileReducer(superquantile_tail_fraction=0.5)

Reduce a batch of values by their superquantile (a.k.a. Conditional Value at Risk, CVaR).

It is given by the value of the linear program

\[\begin{split}\min \quad & \,\, \langle q, \ell\rangle \\ \text{s.t. } \quad & 0 \le q \le \frac{1}{\theta n}, \\ & \sum_{i=1}^n q_i = 1,\end{split}\]

where \(\ell\) represents the input batch to the forward method, \(n\) is the dimension of \(\ell\) (and also of \(q\)) and \(\theta\) is superquantile_tail_fraction and \(\nu\) is smoothing_coefficient.

Parameters:

• batch (torch.Tensor) – Tensor of values to reduce. Reshaped into a single dimension.

• superquantile_tail_fraction (float, default is 0.5) – What fraction of the tail to average over for the superquantile. If 1.0, the function returns batch.mean() and if 0.0, the function returns batch.max().

Examples:

criterion = torch.nn.CrossEntropyLoss(reduction='none')  # Note: must set `reduction='none'`
reducer = SuperquantileReducer(superquantile_tail_fraction=0.6)  # define the reducer

# Training loop
for x, y in dataloader:
    y_hat = model(x)
    batch_losses = criterion(y_hat, y)  # shape: (batch_size,)
    loss = reducer(batch_losses)  # Additional line to use the superquantile reducer
    loss.backward()  # Proceed as usual from here
    ...

class sqwash.SuperquantileSmoothReducer(superquantile_tail_fraction, smoothing_coefficient)

Reduce a batch of values by a L2 smoothing of their superquantile (a.k.a. Conditional Value at Risk, CVaR).

It is given by the value of the quadratic program

\[\begin{split}\min \quad & \, \, \langle q, \ell \rangle - \frac{\nu}{2n} \big\| q - \frac{\mathbf{1}_n}{n} \big\|_2^2 \\ \text{s.t. } \quad & 0 \le q \le \frac{1}{\theta n}, \\ & \sum_{i=1}^n q_i = 1,\end{split}\]

where \(\ell\) represents batch, \(n\) is the dimension of \(\ell\) (and also of \(q\)) and \(\theta\) is superquantile_tail_fraction and \(\nu\) is smoothing_coefficient.

Parameters:

• batch (torch.Tensor) – Tensor of values to reduce. Reshaped into a single dimension.

• superquantile_tail_fraction (float, default is 0.5) – What fraction of the tail to average over for the superquantile. If 1.0, the function returns batch.mean() and if 0.0, the function returns batch.max().

• smoothing_coefficient (float, default=1.0) – How much to smooth by?

Examples:

criterion = torch.nn.CrossEntropyLoss(reduction='none')  # Note: must set `reduction='none'`
reducer = SuperquantileSmoothReducer(  # define the reducer
    superquantile_tail_fraction=0.6, smoothing_coefficient=1.0
) 

# Training loop
for x, y in dataloader:
    y_hat = model(x)
    batch_losses = criterion(y_hat, y)  # shape: (batch_size,)
    loss = reducer(batch_losses)  # Additional line to use the superquantile reducer
    loss.backward()  # Proceed as usual from here
    ...

Functional API of reducers:

sqwash.reduce_mean(batch: Tensor) → Tensor

Reduce a batch of values by their mean.

Parameters:

batch (torch.Tensor) – Tensor of values to reduce. Reshaped into a single dimension.

sqwash.reduce_superquantile(batch: Tensor, superquantile_tail_fraction: float = 0.5) → Tensor

Reduce a batch of values by their superquantile (a.k.a. Conditional Value at Risk, CVaR).

See the class SuperquantileReducer for details.

Parameters:

• batch (torch.Tensor) – Tensor of values to reduce. Reshaped into a single dimension.

• superquantile_tail_fraction (float, default is 0.5) – What fraction of the tail to average over for the superquantile. If 1.0, the function returns batch.mean() and if 0.0, the function returns batch.max().

Examples:

for x, y in dataloader: 
    y_hat = model(x)
    batch_loss = torch.nn.functional.cross_entropy(y_hat, y, reduction='none')  # must set `reduction='none'`
    loss = reduce_superquantile(batch_loss, superquantile_tail_fraction=0.6) 
    loss.backward()  # Proceed as usual from here
    ...

sqwash.reduce_superquantile_smooth(batch: Tensor, superquantile_tail_fraction: float = 0.5, smoothing_coefficient: float = 1.0) → Tensor

Reduce a batch of values by a L2 smoothing of their superquantile (a.k.a. Conditional Value at Risk, CVaR).

See the documentation of the class SuperquantileSmoothReducer for details.

Parameters:

• batch (torch.Tensor) – Tensor of values to reduce. Reshaped into a single dimension.

• superquantile_tail_fraction (float, default is 0.5) – What fraction of the tail to average over for the superquantile. If 1.0, the function returns batch.mean() and if 0.0, the function returns batch.max().

• smoothing_coefficient (float, default=1.0) – How much to smooth by?

Examples:

for x, y in dataloader: 
    y_hat = model(x)
    batch_loss = torch.nn.functional.cross_entropy(y_hat, y, reduction='none')  # must set `reduction='none'`
    loss = reduce_superquantile_smooth(
        batch_loss, superquantile_tail_fraction=0.6, smoothing_coefficient=1.0
    ) 
    loss.backward()  # Proceed as usual from here
    ...

To go back to the main page, click here.