Enterprise AI Analysis

Coherence Robustness Across Temperatures

Traditional sampling methods like min-p and top-p show a sharp decline in generated text coherence (measured by total log-probability) as temperature increases. This indicates their sensitivity and tendency to produce fragmented text at higher temperatures, where diversity is sought. In contrast, Top-H adaptively adjusts its entropy constraint to the next token's probability distribution, allowing it to maintain significantly more consistent and coherent outputs, even in high-temperature settings (see Figure 3 in the paper).

Tuning the Entropy Threshold Coefficient (α)

The parameter α in Top-H directly controls the maximum allowable entropy of the sampled distribution (H(q) ≤ αH(p)), effectively governing the balance between creativity and coherence. Through extensive empirical evaluation using LLM-as-a-judge on development samples, it was found that an α value of 0.4 yields the highest average across creativity and coherence scores, striking an optimal balance. This allows Top-H to modulate randomness precisely according to the model's uncertainty (Figure 4 in the paper).

Despite the NP-hard nature of the ECMM problem, empirical evaluations demonstrate that Top-H's greedy approximation algorithm performs remarkably well. The ratio of Top-H's generated probability mass to the theoretically optimal ECMM solution (Γs/Γs*) consistently hovers around 1.0 across various generation steps and prompts (Figure 5 in the paper). This strong empirical optimality confirms Top-H's practical effectiveness in approximating the ideal balance of creativity and coherence.

Hugging Face Top-H LogitsProcessor

Top-H decoding can be easily integrated into existing Hugging Face-based LLM generation pipelines as a LogitsProcessor. This snippet illustrates the core implementation logic, demonstrating its simplicity and plug-and-play compatibility. Developers can instantiate this class and pass it to their model's generate function, enabling Top-H's dynamic creativity-coherence balancing with minimal effort.

from transformers import LogitsProcessor
import torch
import numpy as np
import torch.nn.functional as F

class EntropyFilteringLogitsProcessor (LogitsProcessor):
    def __init__(self, top_n=100, temperature=1.0):
        super().__init__()
        self.top_n = top_n
        self.temperature = temperature

    @staticmethod
    def calculate_entropy (probs):
        probs = probs [probs > 0]
        probs = probs / np.sum(probs)
        return -np.sum(probs * np.log(probs))

    def __call__(self, input_ids, scores):
        batch_size, vocab_size = scores.shape
        assert batch_size == 1

        scaled_logits = scores
        probs = F.softmax(scaled_logits / self.temperature, dim=-1)
        probs_np = probs [0].cpu().numpy()

        sorted_indices = np.argsort(probs_np) [::-1]
        sorted_probs = probs_np [sorted_indices]

        top_n_indices = sorted_indices[:self.top_n]
        top_n_probs = probs_np [top_n_indices]

        alpha = 0.4
        threshold = self.calculate_entropy (top_n_probs) * alpha

        ind = 1
        valid_indices = []
        for idx, prob in zip (top_n_indices, top_n_probs):
            valid_indices.append(idx)
            ind += 1
            cumulative_probs_np_b = top_n_probs[:ind]
            entropy_diff = self.calculate_entropy(cumulative_probs_np_b)
            if entropy_diff > threshold:
                break
        
        keep_mask = torch.zeros(vocab_size, dtype=torch.bool)
        keep_mask[valid_indices] = True
        updated_scores = scaled_logits.clone()
        updated_scores[:, keep_mask] = float('-inf')
        return updated_scores