Why I set `dynamic: false` on my Elasticsearch audit-log stream

Most Elasticsearch advice is about getting more out of it: better relevance, faster queries, smarter aggregations. This is about the opposite. It's a place where I deliberately told Elasticsearch to do less, and why that was the right call.

👌 The short version: I built an audit-log pipeline, a record of what changed on our entities over time, and I set dynamic: false on the mapping so Elasticsearch would stop trying to index every field it had never seen before. Run an append-only log of arbitrary change events through Elasticsearch with the default settings and the mapping grows without bound. A growing mapping doesn't announce itself. You find out it's a problem once it's already hurting you.

What we were building

The job was an audit log: track what changed on records over time. An entity gets created, updated, deleted, and we write an event describing that change. Which entity, what fields, the old and new values, who did it and when. It's append-only by nature. You never update an audit event, you only add more of them. And you read it far less often than you write it, mostly when someone asks "what happened to this record, and when?"

That shape matters, because it's the shape that makes Elasticsearch's friendliest default turn against you.

The default that will quietly hurt you

By default, Elasticsearch uses dynamic mapping. A document arrives with a field Elasticsearch hasn't seen, it guesses a type, and it adds that field to the index mapping automatically. For most use cases this is genuinely great. You throw documents at it, it figures out the schema, you move on.

Now think about what that means for an audit log.

An audit log's payload is, by design, the union of every field that has ever changed on every entity you track. Every entity type, every column, every nested attribute. Anything that can change is something that can show up in a change event. Run that through dynamic mapping and Elasticsearch faithfully adds a field to the mapping for each new leaf it sees. The mapping stops describing one document shape and slowly accretes every shape your application has ever produced.

A normal index has a roughly stable set of fields. An audit log over a real application does not. It trends toward "every field in the system, ever." So the feature that's a convenience everywhere else is, for this one data shape, a slow leak that you only opt out of if you saw it coming.

What a mapping explosion actually costs

"Mapping explosion" sounds dramatic, so I want to be concrete about the mechanism. The cost is real and it isn't obvious.

The mapping is part of the cluster state, the metadata that lives in memory and is replicated across the cluster. It isn't free per field. As the field count climbs into the thousands, that metadata grows, cluster-state updates get heavier, and the cost is paid by the whole cluster, not just the one index. That's why Elasticsearch ships with a default limit of 1,000 fields per index (index.mapping.total_fields.limit). The limit isn't arbitrary. It's a guardrail against exactly this.

When you hit that limit, you don't get a gentle warning. Indexing fails and the document is rejected. For an audit log that's a nasty failure mode, because the whole point of the system is that it captures everything, and the way dynamic mapping breaks is by silently raising the field count until writes start getting rejected. You can "fix" it by raising the limit, but that's treating the symptom. You're buying time until a bigger number while making every cluster-state update heavier along the way.

There's a second, subtler cost before you ever hit the limit. Every dynamically mapped field is a field Elasticsearch builds indexing structures for. You're paying, in memory and in indexing work, to make fields queryable that, for an audit log, almost nobody will ever query directly.

The decision: dynamic: false 🥳

So I locked the mapping down. The setting that matters:

{
  "mappings": {
    "dynamic": false,
    "properties": {
      "@timestamp":  { "type": "date" },
      "entity_id":   { "type": "keyword" },
      "entity_type": { "type": "keyword" },
      "trace_id":    { "type": "keyword" },
      "author": {
        "dynamic": false,
        "properties": {
          "id":       { "type": "keyword" },
          "username": { "type": "keyword" },
          "source":   { "type": "keyword" },
          "name":     { "type": "text" }
        }
      }
    }
  }
}

Notice the same dynamic: false on the nested author object. The author of a change is a small, known set of fields, id, username, source, name, so I map those and lock that sub-object down too. Nothing unexpected creeps in there either.

It's worth being precise about what dynamic: false does, because there are three options and they aren't the same:

• dynamic: true (the default): unknown fields get auto-mapped and become searchable. This is the explosion path.
• dynamic: false: unknown fields are stored in _source and returned with the document, but they aren't added to the mapping and aren't indexed, so you can't search or aggregate on them. The document is accepted. The field just isn't queryable.
• dynamic: strict: a document containing an unknown field is rejected outright.

I chose false, not strict, on purpose. strict would have meant that every time the application started tracking changes on a new field, the audit pipeline would start rejecting events until someone updated the mapping. That couples the audit log's health to every schema change upstream. It's fragile, and it defeats the point of an audit log, which is to never drop a change event. false keeps the full payload in _source, so the complete record of what changed is still there and still retrievable. It just isn't turned into a searchable field. I explicitly map the handful of fields I actually query on, the entity identity, the timestamp, a trace id that ties an event back to the request that caused it, and who made the change, and everything else rides along in _source without bloating the mapping.

That's the whole trade in one sentence: I keep the data, I drop the ability to query arbitrary fields, and in exchange the mapping stays small and the cluster stays healthy.

The trade I accepted

I want to be honest about the cost, because dynamic: false isn't free.

What I gave up: I can't run an ad-hoc query like "find every audit event where some arbitrary nested field changed to value X," because that field was never indexed. If I genuinely needed that, I'd have to map the field explicitly, ahead of time.

Why it was the right trade for this data: an audit log is overwhelmingly read by entity. The real query is "show me the change history for this record," and that's served by the fields I did map, the entity type and id and timestamp. The rare case where someone wants the full detail of a specific event is served by _source, which still has everything. I'm not losing information. I'm losing a query pattern I don't actually use, in exchange for a system that doesn't degrade as the application grows.

The general principle I took from it: dynamic mapping is a convenience that assumes a bounded set of fields. The moment your data's field set is unbounded by design, and an audit log is the textbook example, that convenience works against you, and you should opt out deliberately rather than discover the field limit in production.

Where this lives: data streams, not a plain index

One more piece, because "set dynamic: false" is only half the design. An audit log is append-only and time-ordered. You write a continuous stream of events and basically never modify what's already there. That's the case Elasticsearch data streams are built for.

Instead of writing to a single ever-growing index, a data stream sits in front of a series of backing indices and rolls over to a new one as they grow, while you read and write through one stable name. You configure the whole thing, the dynamic: false mapping included, once, in an index template, and every backing index the stream creates inherits it. So the mapping discipline isn't something you have to remember to reapply. It's in the template, and every new index the stream rolls into starts out locked down.

That pairing is the actual design: a data stream for append-only, time-series writes, plus an index template that carries the dynamic: false mapping so the field set stays controlled no matter how long the log runs or how many indices it rolls through.

The takeaway 📝

If you're putting an append-only log into Elasticsearch, audit events, entity-change history, anything where the field set grows with your application, decide your mapping strategy before the field count forces the decision for you. The default will happily index every field your app has ever produced, and the way you find out is the day writes start failing.

dynamic: false plus a small set of explicitly mapped query fields, sitting on a data stream driven by an index template, is the configuration I'd reach for again without thinking twice. It's a small, boring decision, and boring is what you want here. It's the difference between an audit log that quietly runs for years and one that turns into a cluster-state problem you have to firefight later.