Fixing Broken Chain-of-Thought in Constrained JSON Schemas

November 14, 2026·4 min read·Prompt Airchitect
  • json
  • llm
  • constrained-decoding
  • prompt-engineering
  • machine-learning
A sequence of tiles with vertical dividers forcing a marble to follow a specific path.

In short

  • Constrained decoding engines prioritize the fulfillment of required JSON fields, often ignoring field order.
  • Optional reasoning fields are frequently bypassed or moved to the end of output because they are not strictly required for schema validation.
  • Moving all fields into the required array forces a sequential dependency that ensures reasoning precedes conclusions.
  • Adding specific instructions within the field description can further guide the model to follow the desired logical order.

Why Schema-Level Required Field Ordering Breaks Dependent Reasoning

Constrained decoding engines facilitate JSON structure by restricting token generation to valid syntax. When you define a schema where a reasoning field is optional but a decision field is required, the execution flow can result in the model generating the final output before the chain-of-thought scratchpad. This structural constraint forces the model to finalize a decision key, often leaving the optional reasoning field unpopulated or placed after the decision.

Technical takeaways

  • Constrained decoding logic often mandates the fulfillment of the required array to ensure schema validity.
  • Placing a reasoning field as optional can result in it being bypassed or trailing the final output keys, rendering it ineffective for logical validation.
  • To ensure reasoning precedes conclusions within a single schema, every field should be marked as required, turning the schema into a strict sequential dependency.
  • Aligning the required array order with the desired logical flow encourages the model to generate the fields as defined in the schema sequence.

The mechanics of constrained decoding priority

Constrained decoding engines ensure valid JSON by intercepting token generation and restricting the vocabulary to tokens that maintain syntax integrity. To guarantee the schema is satisfied, these engines focus on fulfilling the required list. If you define your schema with decision as a required field and reasoning as optional, the decoder manages the generation process to ensure decision appears before the final closing brace of the JSON object.

A mechanical sieve forcing a required block into a slot while an optional block falls away.
A mechanical sieve forcing a required block into a slot while an optional block falls away.

This process can influence the model's output order. The decoder's requirement to satisfy the mandatory decision key at the earliest possible index may cause the reasoning field to be generated after the decision or omitted if the model encounters the closing brace to finalize the required schema.

Why optional fields collapse chain-of-thought

When reasoning is optional, the model does not face a hard constraint requiring the reasoning tokens to be generated before the final decision tokens. Since the constrained decoder manages the required decision key, the model is incentivized to resolve the decision-making logic internally to satisfy the schema constraints. This creates a trap where the reasoning block, intended to host the chain-of-thought, becomes an afterthought rather than a preceding computation.

We avoid using standard text prompts before the JSON block because it frequently causes format drift, where the model outputs introductory text that breaks JSON parsers. By forcing the reasoning field into the JSON schema, we ensure the output is machine-readable. However, if that field is not strictly required, the logic collapses, as the model treats the reasoning field as a lower-priority task that does not influence the actual extraction process.

Converting schemas to enforce sequential logic

The solution is to make every field in your schema required, creating a "forced-order" dependency. By moving your reasoning field into the required array, you force the constrained decoder to treat it as a mandatory prerequisite for the final JSON object. This ensures that the tokens generated for the reasoning field must precede the tokens generated for the final decision field, preserving the logical flow of the chain-of-thought.

Interlocking gears where every tooth is required for the mechanism to turn.
Interlocking gears where every tooth is required for the mechanism to turn.
{
  "type": "object",
  "properties": {
    "reasoning": {
      "type": "string",
      "description": "Step-by-step logical analysis leading to the conclusion."
    },
    "decision": {
      "type": "string",
      "enum": ["approve", "reject"],
      "description": "The final classification result."
    }
  },
  "required": ["reasoning", "decision"]
}

If the model still ignores the reasoning, we suggest adding an explicit instruction inside the description field of the reasoning key, such as: "You must generate this field in full detail before providing the decision field."

Dealing with field ordering edge cases

Edge cases occur when decoding engines attempt to optimize generation, potentially causing non-deterministic ordering. We address this by explicitly structuring the prompt instructions to warn the model that the schema represents a strict, sequential process.

A pipe system splitting a single flow into two distinct sequential stages.
A pipe system splitting a single flow into two distinct sequential stages.

Addressing decoding conflicts

If the constrained decoding engine is aggressive in its prioritization of the schema's terminal tokens, the model may bypass the reasoning block. In these instances, we recommend splitting the task into two distinct calls: a chain-of-thought generation call using a standard text-based output, followed by a structured extraction call. While this increases latency, it acts as a reliable method to bypass the priority enforcement of a single-pass constrained decoder when the schema complexity exceeds the model's ability to maintain sequence under these constraints.

Next steps

Review your current JSON schemas to identify optional fields that contain reasoning or scratchpad elements. Move these fields into the required array of your schema definition and verify that your parser handles these fields as mandatory strings. You can test these updated schemas by running them through your interface to observe if the token generation sequence now aligns with your logical requirements.

Want the prompt this article describes, built for your own objective?

Engineer one now — free, no account needed