The Weird Storage card

Store files where files don’t go! Pictures, sound, DNS records, a spreadsheet, a Minecraft world… as long as the file comes back out in one piece.

What counts

A program that takes any file, hides it in something that was never meant to hold files, and gets the exact same bytes back out. “Any file” really means any: try it with a photo or a .zip, not just a text file. It’s fine if big files take ages or need a pile of carriers, as long as a real file makes the trip.

The carrier should still be a real one of whatever it is. The picture opens in an image viewer, the chess games are legal, the Minecraft world loads in Minecraft.

Zipping a file, base64-ing it into a .txt, renaming it to .png or uploading it to a normal file host doesn’t count. Neither does sticking a zip on the end of a JPEG with cat. It’s a fun classic, but the file is just sitting there in one piece!

Plain old steganography (hiding a file inside a normal-looking photo) counts. The weirder the carrier, the better though. If you’re not sure about yours, ask in #crescent-help.

Getting started, step by step

Every weird storage project does the same few things underneath, so let’s build the loop once with a boring carrier: the pixels of a PNG. Every 3 bytes of your file become one pixel’s red, green and blue. This is a warm-up, so please don’t ship it as it is! The carrier is your job.

You’ll need Python and Pillow (pip install pillow).

1. Read the file as bytes

Forget that it’s a picture or a song. To your program, every file is just a list of numbers from 0 to 255.

import hashlib, math, struct
from pathlib import Path
from PIL import Image

data = Path("cat.jpg").read_bytes()
print(len(data), data[:8])

2. Add a header

When you read the bytes back out, you’ll get extra junk at the end (the last row of pixels is padded with zeros), and you’ll have forgotten what the file was called. So put a little header in front: a magic value so you know it’s one of yours, the length so you know where the file ends, and the filename. struct packs numbers into bytes for you.

MAGIC = b"WSTR"

def pack(path):
    data = path.read_bytes()
    name = path.name.encode()
    header = MAGIC + struct.pack(">BI", len(name), len(data)) + name
    return header + data

def unpack(blob):
    if blob[:4] != MAGIC:
        raise ValueError("this isn't one of ours!")
    name_len, data_len = struct.unpack(">BI", blob[4:9])
    name = blob[9 : 9 + name_len].decode()
    data = blob[9 + name_len : 9 + name_len + data_len]
    if len(data) != data_len:
        raise ValueError("the file got cut off")
    return name, data

>BI means “big-endian, one byte, then a four-byte number”, so filenames can be up to 255 bytes long and files up to 4 GB.

3. Encode it into the carrier

Make the smallest square image that fits, pad the leftover space with zeros, and let Pillow turn the bytes into pixels.

def encode(blob, out):
    side = math.ceil(math.sqrt(len(blob) / 3))
    blob += bytes(side * side * 3 - len(blob))
    Image.frombytes("RGB", (side, side), blob).save(out)

You’ll get a square of colourful static. Open it, it’s a real PNG!

4. Decode it back

The same thing backwards: pixels to bytes, then let the header tell you what’s real.

def decode(path):
    return Image.open(path).convert("RGB").tobytes()

5. Prove it round trips

“It looks the same” isn’t good enough. One flipped bit can break a zip! Compare a SHA-256 hash of the original with the one that came out. If they match, the bytes are identical.

encode(pack(Path("cat.jpg")), "cat.png")
name, data = unpack(decode("cat.png"))

Path("recovered").mkdir(exist_ok=True)
Path("recovered", Path(name).name).write_bytes(data)

print(hashlib.sha256(Path("cat.jpg").read_bytes()).hexdigest())
print(hashlib.sha256(data).hexdigest())

(Path(name).name throws away any folders in the stored name, so a sneaky file called ../../something can’t write outside recovered/.)

You can double check from the terminal with shasum -a 256 on macOS, sha256sum on Linux, or Get-FileHash in PowerShell.

Now try this: open cat.png in an image editor and save it as a JPEG, then decode that. You’ll get “this isn’t one of ours!”, because JPEG nudged nearly every byte a tiny bit. Keep that in mind, it matters a lot later.

If you’d rather work in the browser, a <canvas> and getImageData() give you the same thing with four bytes per pixel. Keep the alpha at 255, because canvas quietly changes the colours of see-through pixels.

Now it’s yours

That’s the whole loop: bytes, header, encode, decode, hash. Everything from here is up to you, so swap the PNG for something weird and work out how to squeeze bytes into it. That’s the fun part, and nobody else should end up with the same one!

Picking your carrier

Some ideas to get you going:

• Sound. Turn bytes into beeps like an old modem, play it out loud and record it back in. ggwave and minimodem are good to look at.
• A video of QR codes. One QR code holds up to 2,953 bytes, so a few seconds of video can carry a whole file. segno makes the codes and ffmpeg glues them together.
• Chess games. In most positions there’s about 30 legal moves, so every move you pick can stand for about 5 bits. chessencryption turns files into PGN games. If you store a file in a chess game, I need to see it.
• A Minecraft world. One block type per byte, and the file is a building! Amulet can read and write worlds from Python.
• DNS TXT records on a domain you own. Look at DNSFS for a gloriously cursed take on the idea.
• A font. Hide bytes in the shape of each letter. fontTools opens fonts up for you.
• A spreadsheet’s cells. Numbers, colours, cell comments… openpyxl can write all of it.
• Git commit messages, or commit timestamps, or the order of the commits.
• A playlist’s song order. 20 songs can be shuffled in enough ways to hold about 7 bytes. That’s not much, but it’s very funny.
• Paper. Print it, scan it back in. PaperBack did this years ago with a regular printer.
• Emoji. On their own they’re base64 in a costume, so put them somewhere funny.

Tom 7’s Harder Drive is a whole tour of hard drives nobody asked for, and pingfs keeps files in ping packets while they’re flying around the internet. Both are great for getting ideas. And if you want to see how far it goes, Benn Jordan saved a PNG to a bird! He turned a picture into sound, played it to a real starling, and got the picture back out of its song.

People have done this before!

Other teenagers have already stored files in some gloriously wrong places. A few of our favourites:

• Check-MateData turns any file into legal chess games you can import into Lichess, right in your browser, and cheese goes one further and actually plays the games on Lichess.
• encode-image-to-minecraft writes files into a Minecraft region file, one block per byte, so you can walk around your data.
• cryptags hides files in the messages of git tags, all hanging off one empty commit, so the repo looks completely empty.
• InvisisbleBanana stores files inside NaNs! A floating-point NaN has spare bits nobody looks at, so the file turns into a long list of math errors.
• image-encrypt hides files in a photo by nudging each colour by 1/256, so the picture looks exactly the same.
• Someone stored files in a Twitch stream’s chat, one message every 1.6 seconds… until the streamers started banning them (more on that below).

Go look at what they made, then try to find a carrier nobody’s thought of yet.

Going further

• Error correction. Lossy carriers (JPEG, YouTube, a recording from your phone’s microphone) change your data a little bit on the way. Use big chunky blocks instead of one byte per pixel, then add Reed-Solomon codes so you can fix the few bytes that still come out wrong. With reedsolo, RSCodec(32) adds 32 extra bytes to every 223, and can repair up to 16 broken ones.
• Compress first. zlib.compress() before encoding means less to hide. It’s a huge win when your carrier only holds a few bytes per thing!
• Chunking. Big files won’t fit in one carrier, so split them up and give each piece its own small header (which piece it is, and how many there are), so you can put them back together in any order.
• Encryption. Anyone who finds your file and knows the trick can read it, so encrypt it first (with age, for example) if you want it to stay private.

Other people’s servers

Storing files in YouTube videos, Discord messages or a Google Sheet is really funny, and a small demo of it is totally fine! Just remember it’s usually against those services’ terms, and they can delete your stuff or ban your account. Keep it small and use your own files. Something built to dump terabytes onto someone else’s servers isn’t what this card is about, and we won’t approve it.

Before you ship

There’s a few things every project needs before we can approve it:

• It works. It doesn’t have to be finished, but a file should go in and come back out with the same hash.
• All of your code is public, on GitHub or somewhere like it.
• Anyone can try it, even someone who doesn’t code, in under two minutes of setup.
• Someone else could rebuild it from your repo and README.

What that looks like depends on where your project lives:

• On the web: a public link that stays up. A page where we can drop a file in, get the weird thing out, then drop that back in and get our file back is the dream demo! GitHub Pages, Vercel, Render, Railway and Nest are all great, but please don’t use Streamlit. Don’t put it behind a login either! (Pyodide can run Python and Pillow in the browser, if you don’t want to rewrite it in JavaScript.)
• As a command line tool: publish it as a package (on PyPI, crates.io, npm…) or as a download people can run straight away.
• As a desktop app: a GitHub release with something people can download and run (an .exe, a .dmg, an .AppImage…). The source code on its own isn’t enough.

A video of a file going in and coming back out is awesome, but it can’t be your demo. Put videos on YouTube or Vimeo, or upload an .mp4, rather than Google Drive.

Shipping one

• Demo: a web page where we can try our own files is ideal. Otherwise, include a carrier with a file already hidden in it, so we can pull it out ourselves!
• Screenshot: the carrier itself. The static, the chess board, your Minecraft build…
• README: how the encoding works, what your header looks like, and how much fits (bytes per pixel, per move, per block). Show the SHA-256 before and after, and brag about the biggest file you’ve managed to store.