How does Safety Change when Context is Compressed?

A replication and extension of LLMLingua-2

My contribution: Domain Specific Compression, Soft Scoring Alignment, and Round-Trip Evaluation: LLMLingua-2

Status: side project. Results are small-scale (one QA model, two datasets) and should be read as exploratory.

At some point, every agent needs to compress their context. Long memories and documents degrade the performance of the model as it reaches its context window, especially if there are multiple turns, steps, tool calls, etc. in the same session.

One way to do this is to make the plain text context shorter, via semantic summary or by pruning unimportant tokens, the latter of which is what the LLMLingua-2 paper did.

The issue is that the model never sees the original, so there is no way of knowing if an important name or specific constraint was dropped. “No” is a single word that flips the meaning of a sentence.

For example, if the agent is ordering a salad for someone with a peanut allergy, the difference between “a salad with no peanuts, please” and “salad peanuts” is life and death.

If an agent’s guardrails live in its context and the context gets compressed, are the guardrails the first to go?

I replicated LLMLingua-2 and also invented a new metric called round-trip reconstruction, where an original prompt is compressed. Then the LLM “decompresses” the compressed prompt to reconstruct the original. The cosine similarity between the original and reconstructed is one way to evaluate the quality of the compression. Was the original meaning preserved through the 3 versions?

This was my setup: GPT-4 compresses text, and is prompted to only delete unimportant words and keep entities, numbers and dates. Then, a very small BERT model learns on GPT-4’s binary choices: 0 = drop, 1 = keep. I tested on SAT English passages (literary) and SQuAD (Wikipedia, facts), where I used llama3.2 to answer questions on the compressed text.

However, it’s important to note that the compressor doesn’t follow our prompt. GPT-4 was only supposed to delete words, but it often paraphrased, reordered, summarized, and cut out chunks from the middle of SAT passages. The data that we trained BERT with was already not so great.

Main finding: The same prompt compresses very differently by domain. SAT passages were compressed ~4.5x; SQuAD was compressed ~1.5x. Also, the factual encyclopedic texts maintained much more of its meaning (cosine similarity 0.95 vs. 0.81).

Also super interesting: There seems to be no strong relationship between performance of compressed vs. uncompressed, but sometimes, compressed passages outperform uncompressed passages. Compression is not necessarily harmful.

A limitation to my round-trip reconstruction metric: the compression and reconstruction were already misaligned, with paraphrasing and summaries. This means that the similarity scores between original-compressed-reconstructed, are also not accurate.

Also, cosine score is just the angle between two vectors, which is how a token is represented in the embedding space. But I learned that cosine scoring is not a perfect representation for similarity in meaning.

For example, the sentences “The meeting is at 3pm” and “The meeting is at 3am” are actually very similar in the latent space (96.8% similarity), although for a human they mean very different things for our mental healths.

High cosine similarity =/= information accuracy.

Next Questions

My question was, if an agent’s guardrails live in its context and the context gets compressed, are the guardrails the first to go? However, we did not dive into security-relevant content. I want to see if negations and constraints hold up in compression.

I’d also like to find a better faithfulness metric than cosine similarity, like QA probes on facts.