How to Break a Vigenère Cipher Without the Key: A Worked Example

For three centuries the Vigenère cipher was called *le chiffre indéchiffrable* — the indecipherable cipher. It defeats the simple frequency analysis that breaks a Caesar cipher, because the same plaintext letter encrypts to different ciphertext letters depending on its position. Then, in 1863, Friedrich Kasiski published a general attack (Charles Babbage had found one privately a decade earlier). Today it's a standard CTF challenge, and breaking one by hand is the best introduction to real cryptanalysis there is.

This guide breaks a real ciphertext step by step. Every number below was computed with the same functions that power our Vigenère tool and frequency analysis tool, so you can follow along.

How Vigenère works (in one paragraph)

Pick a keyword, say KEY. Write it repeatedly under the plaintext, then shift each plaintext letter forward by the alphabet position of the key letter above it (A=0, B=1, … Z=25). With key KEY, the 1st, 4th, 7th… letters are shifted by 10 (K), the 2nd, 5th, 8th… by 4 (E), and the 3rd, 6th, 9th… by 24 (Y). It's simply several interleaved Caesar ciphers — and that's its weakness.

The challenge

Here is our ciphertext. Spaces and punctuation were left in place, which is common in puzzles (and makes life easier):

Ehr fijfpntxv crqt gai yncbbk fvszrr wena naekczar a fxecro lrmxve qoe mlv tzvrkrfe. Yoohhp by tux wyva kaxa nult gai cretrk wrvo, bhm imrcybgi bapw vm qrgeeexh, srnahli kup cnixrvy hnw hfhmlrw xyr hagvl rao fbkfzqoea trp ytgumw fa oepd ewgpr fnrjre. Oa mlv gsiew rztst n lxfex cnfi fhe os mlv jpsg tru gse pkin ylsuxh kup cnkkf gz tux qrfe. Wuxr kup sxr gcrlrrw xyr xeflietpr jtw xbye, ngh jb haf mlv yptgxv. Kup cnixrvy srtvtupd roiil naobr rao fbnru byll tr vzatl vlrvc aaw e tnydyx xyne wnl wkvwl jtvd.

First, a quick sanity check with the frequency analysis tool: the 405 letters have an index of coincidence of 0.0448. English text scores around 0.066–0.073 (the plaintext here scores 0.0728); uniformly random letters score 0.0385. A value in between is the classic signature of a polyalphabetic cipher like Vigenère — a Caesar or simple substitution cipher would keep the English value.

Step 1a: Kasiski examination

When the same plaintext fragment happens to line up with the same part of the key, it produces the same ciphertext. Common words like *the* make this likely. So: find repeated sequences of three or more letters (ignoring spaces) and measure the distances between them.

Almost every distance is a multiple of 7. Two aren't — 118 and 204 — which is normal: some repeats are coincidences, not alignments. Kasiski gives you strong candidates, not certainty; the next test confirms.

Step 1b: index of coincidence by key length

If the key length is L, then letters 1, L+1, 2L+1… were all shifted by the same key letter. Each such column is a Caesar cipher of English, so its index of coincidence should look like English (~0.066). At the wrong length, each column mixes several shifts and looks random (~0.038). Compute the average column IoC for each candidate length:

There's no ambiguity: length 7 jumps to 0.073, matching English, while every other length up to 12 stays near 0.044. Multiples of the true length — 14 and 21 — also score high (21 even higher, because its columns are so short that the statistic gets noisy), so always prefer the smallest length that spikes.

Step 2: recover each key letter with chi-squared

Now split the 405 letters into 7 columns of 57–58 letters. Each column is a Caesar cipher. For each one, try all 26 shifts and measure how close the resulting letter frequencies are to English using the chi-squared statistic:

χ² = Σ (observed_count − expected_count)² / expected_count
     over the 26 letters, where expected = English frequency × column length

Lower χ² = closer to English.

In every column the best shift beats the runner-up by a wide margin — a χ² in the 15–30 range against 87–213. Read the winning shifts top to bottom: L-A-N-T-E-R-N. Decrypt with key LANTERN:

The messenger left the harbor before dawn carrying a sealed letter for the
governor. Nobody on the ship knew what the letter said, but everyone knew it
mattered, because the captain had doubled the watch and forbidden any lights
on deck after sunset. …

You can reproduce this: paste the ciphertext into the Vigenère tool, choose *decrypt*, and enter LANTERN. Then try a wrong key of the same length and watch the output turn to noise.

Doing it by hand: the mutual-shift shortcut

Without a computer, computing 26 chi-squared scores per column is tedious. A faster manual trick: in each column, find the most common letter and assume it's E. Column 4's most frequent letter is X (10 times), so the shift is X − E = 23 − 4 = 19 = T — correct.

Five of seven letters come out right, and the misses are instructive: with only 58 letters per column, T or A can outnumber E by chance. That's why chi-squared, which uses all 26 frequencies at once, is more reliable. By hand, check whether the top two or three letters can play the roles of E, T and A consistently, then fix doubtful letters by reading the partial decryption. The shortcut's key LPNTENN decrypts to text where only the 2nd and 6th letter of every 7-letter group is wrong, and it's easy to see which shift turns Tse back into The.

When it doesn't work

• Autokey variant — the key is the keyword followed by the plaintext itself, so there's no repeating period and Kasiski/IoC find nothing. Attack it by guessing the keyword length and probable words instead. Our tool supports autokey too.
• Running key / book cipher — a long text as the key. Breakable with statistics on both texts, but much harder.
• Key as long as the message and truly random — that's a one-time pad, and it's provably unbreakable.
• Very short ciphertext — with only a few letters per column, there isn't enough statistical signal. Try dictionary attacks on likely keywords instead.
• Spaces removed and letters grouped in fives — no problem for these methods, which ignore non-letters anyway.

Why this still matters

Vigenère's failure teaches a lesson every modern cipher is designed around: any repetition in the keystream is exploitable. The same mistake reappears in reused one-time pads, in repeating-key XOR (which falls to exactly this attack — try our XOR bruteforcer), and in stream ciphers like AES-CTR or ChaCha20 when a nonce is reused. Modern ciphers avoid it by making the keystream effectively infinite and never repeating it under the same key.

Practice workflow for CTFs

1. Run frequency analysis: IoC near 0.066 → Caesar/substitution; near 0.04–0.05 → polyalphabetic.
2. If it's monoalphabetic and only shifted, use the Caesar bruteforce.
3. If polyalphabetic, find the period with Kasiski + IoC.
4. Solve each column with chi-squared, decrypt, and sanity-check the plaintext.
5. If nothing works, consider transposition (rail fence), Playfair, or an encoding layer such as Base64 on top.