TEXT and JSON Columns: Where the Schema Goes to Hide

Thu, 23 Apr 2026 00:00:00 +0000

TL;DR

A TEXT or JSON column moves the schema out of the database catalog and into application code — the data inside has a shape, but the DDL won’t tell you what it is. Readers can’t query into it without knowing the format, planners can’t reason about it, and the shape drifts across years of writes with no signal to the next reader. The fix isn’t “don’t use JSON”; it’s to promote the fields that actually get queried into real columns and treat the rest as genuinely opaque.

An AI assistant is asked to “find customers who upgraded to enterprise in the last quarter.” It reads the catalog, finds api_logs(id, endpoint VARCHAR, payload LONGTEXT, created_at DATETIME), and generates the reasonable query:

SELECT JSON_EXTRACT(payload, '$.action') AS action, created_at
FROM api_logs
WHERE JSON_EXTRACT(payload, '$.action') = 'upgrade'
  AND JSON_EXTRACT(payload, '$.plan')   = 'enterprise'
  AND created_at >= NOW() - INTERVAL 90 DAY;

Runs clean. Returns zero rows. The actual key was renamed from action to event.type two years ago when the team adopted a shared event schema — new rows match $.event.type, old rows still match $.action, and no one migrated the historical data because it wasn’t queryable anyway. Neither column nor catalog said any of this. The query is syntactically perfect, semantically correct for the key it guessed, and wrong because the key doesn’t exist in most of the rows.

The obvious fix is “switch to JSONB, validate with a JSON schema, add a GIN index.” Each one helps at the margin and none of them close the gap. JSONB tells you the blob is valid JSON, not what keys are in it. CHECK constraints with JSON_SCHEMA_VALID or jsonb_matches_schema work prospectively, but the six years of rows already in the table were written against five format generations and no validator reaches back in time. A GIN index accelerates key lookups but only if you know which keys to look up. The problem isn’t the storage format — it’s that the schema emigrated to application code, and changing the column type doesn’t bring it back.

What leaves the catalog when the column becomes a blob

DDL is the contract between the database and everything that reads it. A typed column says “this value is an integer between 0 and 2³¹−1, and here’s the index I’ve built over it.” A TEXT or JSON column says “this value is a string the application decided on, and the application can tell you what that means.” The second contract is thinner in ways that compound.

Readers can’t discover the shape from the schema. information_schema.COLUMNS for a JSON column returns COLUMN_TYPE = 'json' and nothing else. Every tool that reads catalog metadata — MCP servers, ERD generators, typed-client code generators, AI assistants, new engineers running \d+ — sees a blob. The shape lives in the serializer class, the protobuf definition, the TypeScript interface, or nowhere. Whichever of those the reader happens to find is the shape they’ll assume. See Comment Your Schema for the lowest-effort way to leave a trail, but comments can describe the shape; they can’t make the catalog enforce it.

Generational drift is silent. Year one the payload is {action, user}. A migration adds nested metadata: {action, user, metadata: {source}}. A rewrite flattens and renames: {event: {type, user_id}, source}. A new service standardizes with a version field: {version: 3, event: {...}}. All four versions are sitting in the same column with nothing to distinguish them at read time except the keys they happen to have. A JSON_EXTRACT path written against today’s producer hits the newest generation and silently misses the older ones. The failure mode is exactly the one described in Legacy Schemas Are Sediment: the schema’s history is compressed into the data, and the data can’t decompress itself.

Writes are untyped. Without CHECK constraints or a JSON-schema validator, the writer is the only guardrail. A service deployed last Tuesday that emits amount as the string "9900" instead of the integer 9900 silently poisons the column — downstream queries comparing amount > 1000 work on new rows and misbehave on the poisoned batch, because JSON-extract returns a string and the comparison is lexicographic. The same class of mismatch a typed column would reject on INSERT.

The planner is working blind. Row-count estimates on JSON_EXTRACT(payload, '$.event.type') = 'upgrade' have no histogram to consult; the planner falls back to a default selectivity estimate that’s usually wrong. Plans for queries filtered on JSON fields are routinely pessimistic or optimistic by an order of magnitude, and there’s no ANALYZE to fix that because the statistics don’t exist for the interior of the blob.

Indexes are per-key, not per-column. A functional index on JSON_EXTRACT(payload, '$.event.type') accelerates one path. The next query filters on $.source and scans the table. Generated columns are the cleaner version of this — payload_event_type VARCHAR(50) GENERATED ALWAYS AS (JSON_EXTRACT(payload, '$.event.type')) STORED — but each one is a schema change with a backfill, and you have to know in advance which keys matter. GIN indexes on JSONB cover arbitrary keys but are large, slow to update, and still don’t tell the reader what keys exist.

Untyped writes + untyped reads = silent schema drift

A TEXT or JSON column accepts anything the writer emits and returns exactly that on read. Two services writing to the same column with slightly different shapes don’t conflict at the database level — they just produce a column whose contents depend on which service wrote the row. The divergence is invisible until a query tries to read uniformly across both.

Plausible paths, empty results

Schema-reading LLMs generate JSON_EXTRACT paths the same way they generate column names in a typed schema — by pattern-matching the column name and the question. Asked about “upgrade actions,” the model guesses $.action = 'upgrade' because the English-to-JSON-path mapping is obvious. It has no way to know that the key was renamed, that three generations coexist, or that the canonical name is now buried under two layers of nesting. The catalog gives it a column type of json and nothing else, and the model’s best guess is reasonable and wrong.

The failure pattern is familiar from other schema-hiding designs. Polymorphic references hide which table a foreign-key-shaped column points at; bare id primary keys hide which identifier is being compared; TEXT/JSON columns hide what’s in the column at all. All three are cases where the LLM generates a plausible query against a schema that isn’t telling it enough, and the query returns plausibly-shaped but semantically empty results.

The fix, and where it stops being free

The lever isn’t “avoid JSON” — which is both impractical and sometimes wrong — it’s to be honest about what’s inside and pick the right storage per field.

Promote fields that get queried. If the application filters on event.type more than occasionally, that’s a real column. Generated columns are the low-friction middle path: derive a typed, indexable column from the JSON, keep the raw payload as the audit trail.

ALTER TABLE api_logs
  ADD COLUMN event_type VARCHAR(50) GENERATED ALWAYS AS
    (JSON_UNQUOTE(JSON_EXTRACT(payload, '$.event.type'))) STORED,
  ADD INDEX idx_event_type (event_type);

The trade-off: every promoted field is a migration, and generated columns don’t retroactively rewrite rows written with a different shape — you still need the COALESCE(JSON_EXTRACT(payload, '$.event.type'), JSON_EXTRACT(payload, '$.action')) cleanup for the old generations, and you’re doing that exactly once as part of the promotion rather than in every query.

Enforce new writes with a JSON schema. PostgreSQL’s pg_jsonschema and MySQL 8.0’s JSON_SCHEMA_VALID let a CHECK constraint reject writes that don’t match a named schema. Doesn’t fix existing rows; does stop the next silent format change from landing. If the team doesn’t already have a shared event schema, a CHECK constraint is the forcing function that produces one.

Version the payload explicitly. {"version": 3, "payload": {...}} at the top lets every reader dispatch on version instead of inferring it from which keys happen to be present. Doesn’t help rows written before versioning started, but bounds the drift going forward and turns “which generation is this row?” from archaeology into a lookup.

Document what stays inside. Comments on the column — “see github.com/org/events for the schema; versions 1–3 coexist in rows older than 2024-Q2” — won’t replace types, but they give the reader a place to look. Comments on the schema are cheap, in-place, and propagate through every tool that reads the catalog; for genuinely-opaque columns this is the best available signal.

When JSON is actually the right answer

The pattern earns its keep in specific shapes where the alternative — typed columns — is worse.

Truly variable shape per row. User-supplied settings blobs, custom-field configurations, extension points where the keys are genuinely per-tenant or per-user. Modeling each variant as a column produces a wide table full of NULLs; see God Tables for the cost of that direction. The column is honest about being schemaless because the data is schemaless.

Audit payloads nobody queries. Raw API request/response bodies retained for compliance, debug traces, incident forensics. Written once, read by humans one row at a time, never aggregated. The lack of a queryable schema is fine because no query needs one. A sensible default here is to keep the payload compressed and add a small set of typed columns (endpoint, status_code, user_id, created_at) for the predicates the operational queries actually use.

Short-lived staging. Job queues, idempotency cache payloads, outbox entries — where the producer and consumer are deployed together, the payload is read once, and the row is deleted on completion. Drift can’t accumulate in rows that don’t stay around.

Document stores on purpose. PostgreSQL JSONB with a stable schema, validated on write, with functional indexes on the paths that matter. This is a real design; it’s not the unspoken default that most TEXT columns represent. If the team is reaching for JSONB and treating it as a document store, it should look like one — with validation, indexes, and documentation — not like a TEXT column that happens to parse.

The bigger picture

A TEXT or JSON column is a specific architectural choice: move part of the schema out of the catalog, in exchange for cheaper writes and looser contracts between producer and consumer. When the trade is deliberate — genuinely variable data, write-once audit, short-lived buffer — it’s the correct shape. When it’s the path of least resistance because typed columns would require a migration, the cost is deferred to every future reader who has to reconstruct the format from commit history.

Databases are good at enforcing the contracts they know about. The column types are how they know. Every field that matters to a query deserves to be in the part of the schema the database can see; everything else is honestly opaque and should look it. The default drift — “stick it in the payload, we’ll parse it later” — produces columns whose contents nobody fully knows, including the team that wrote them, and the cost is paid in the form of queries that return plausible answers to questions the data can’t actually answer.

Json on EXPLAIN ANALYZE

TEXT and JSON Columns: Where the Schema Goes to Hide

What leaves the catalog when the column becomes a blob

Plausible paths, empty results

The fix, and where it stops being free

When JSON is actually the right answer

The bigger picture