FlexJSON: A Robust Partial and Streaming JSON Parser for Go

FlexJSON parses incomplete or streaming JSON data, like JSON coming from an LLM. Unlike standard JSON parsers that require valid, complete JSON input, FlexJSON gracefully handles partial JSON fragments and streams of characters, extracting as much structured data as possible.

🌟 Features

• Partial JSON Parsing: Extract data from incomplete JSON fragments

  • {"key": 123 → map[string]any{"key": 123}
  • {"key": 1234, "key2": → map[string]any{"key": 1234, "key2": nil}

• Character-by-Character Streaming: Process JSON one character at a time

  • Ideal for network streams, telemetry data, or large files
  • Updates an output map in real-time as data arrives

• Nested Structure Support: Handles complex nested objects and arrays

  • Properly tracks hierarchy in deeply nested structures
  • Maintains context across partial fragments

• Resilient Parsing: Recovers gracefully from unexpected input

  • No panic on malformed input
  • Extracts maximum valid data even from corrupted JSON

• LLM Integration: Perfect for processing streaming JSON responses from LLMs

  • Extracts structured data as tokens arrive
  • Enables real-time UI updates with partial LLM outputs

• Zero Dependencies: Pure Go implementation with no external dependencies

📦 Installation

go get github.com/jpoz/flexjson

🚀 Quick Start

Parsing Partial JSON

package main

import (
    "fmt"
    "github.com/jpoz/flexjson"
)

func main() {
    // Parse incomplete JSON
    partialJSON := `{"name": "John", "age": 30, "city":`
    
    result, err := flexjson.ParsePartialJSONObject(partialJSON)
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        return
    }
    
    fmt.Printf("Parsed result: %v\n", result)
    // Output: Parsed result: map[name:John age:30 city:<nil>]
}

Streaming JSON Character-by-Character

package main

import (
	"fmt"

	"github.com/jpoz/flexjson"
)

func main() {
	// Example JSON string
	jsonStrs := []string{`{"name":"John Doe"`, `,"age":30,`, `"email":"[email protected]"}`}

	// Create output map
	output := map[string]any{}

	// Create streaming parser
	sp := flexjson.NewStreamingParser(&output)

	// Process each string
	for _, str := range jsonStrs {
		fmt.Printf("Processing %s\n", str)
		err := sp.ProcessString(str)
		if err != nil {
			fmt.Printf("Error: %v\n", err)
			return
		}

		// The output map is updated after each string
		fmt.Printf("Current state: %v\n", output)
	}

	fmt.Printf("Final result: %v\n", output)
}

🤖 LLM Integration Benefits

FlexJSON is particularly well-suited for applications working with LLMs:

• Real-time Processing: Parse JSON data as it streams from LLM APIs token by token
• Immediate Feedback: Update UIs with structured data before the LLM completes its response
• Resilient Handling: Continue processing even if the LLM produces malformed JSON
• Progressive Rendering: Display complex nested structures as they become available
• Efficient Resource Usage: Begin processing data immediately rather than waiting for complete responses

⚙️ How It Works

FlexJSON uses a custom lexer and parser system for the partial JSON parsing, and a state machine approach for streaming parsing:

1. Lexer: Tokenizes the input string into JSON tokens (strings, numbers, booleans, etc.)
2. Parser: Converts tokens into a structured map representation
3. StreamingParser: Maintains stacks of containers and keys to track position in the JSON hierarchy

The library intelligently handles incomplete input by:

• Treating unexpected EOF as valid termination
• Providing default values (nil) for incomplete key-value pairs
• Maintaining context across nested structures

🧪 Testing

The library includes comprehensive test coverage for both partial and streaming parsing:

go test -v github.com/jpoz/flexjson