How to validate JSON schemas in Python for data
This tutorial shows how to validate JSON schemas in Python for data using jsonschema, a useful task to ensure that JSON files have the expected keys and types before ETL processing or ingestion into databases. Validation helps prevent late-stage pipeline failures, produce consistent error records, and enforce quality rules before transformation or loading.
Prerequisites
- Install Python 3.8+ (3.9 or 3.10 recommended for modern compatibilities)
- Basic knowledge of Python and JSON (object/array structures, primitive types)
- jsonschema library (install with pip)
Step 1: Why validate JSON schemas?
JSON files can vary: missing fields, unexpected types, or different nested structures. Validating schemas avoids errors later in the pipeline and allows rejecting, correcting, or logging invalid records early. In a practical example, if you have 10,000 records per day and 2% have format issues, validating early reduces by hundreds the records that would cause ETL processing failures. Additionally, a well-defined schema serves as a contract between data producers and consumers, making clear which fields are required, which are optional, and which formats (for example ISO8601 for dates) are acceptable.
Step 2: Install jsonschema
Use jsonschema because it is simple, compatible with common Drafts, and widely used in data pipelines. To install:
pip install jsonschema
Note: jsonschema supports multiple Drafts (e.g. Draft4, Draft7). Most examples use Draft7, which is sufficient for typical requirements (type, required, format, minimum, etc.).
Step 3: Define the JSON schema (example)
Create a schema that describes the expected structure. Here is an example for sales records with required fields and clear types. This schema also specifies limits (minimum) for quantity and price, which prevents accepting negative numbers by mistake.
sales_schema = {
"type": "object",
"required": ["sale_id", "date", "customer", "items", "total"],
"properties": {
"sale_id": {"type": "string"},
"date": {"type": "string", "format": "date"},
"customer": {
"type": "object",
"required": ["id", "name"],
"properties": {
"id": {"type": "string"},
"name": {"type": "string"}
}
},
"items": {
"type": "array",
"items": {
"type": "object",
"required": ["sku", "qty", "price"],
"properties": {
"sku": {"type": "string"},
"qty": {"type": "integer", "minimum": 1},
"price": {"type": "number", "minimum": 0}
}
}
},
"total": {"type": "number", "minimum": 0}
}
}
Concrete examples: a valid record has total 123.45 and items length 2; an invalid record may have "qty": 0 or "date": "2020/01/01" (wrong format).
Step 4: Validate a single JSON file
Minimal example to validate a JSON file with jsonschema. The code reads the file, validates and handles common errors like ValidationError. To validate formats (e.g.: date) use FormatChecker.
import json
from jsonschema import validate, ValidationError, FormatChecker
with open('sale_example.json', 'r', encoding='utf-8') as f:
data = json.load(f)
try:
validate(instance=data, schema=sales_schema, format_checker=FormatChecker())
print('Validation successful')
except ValidationError as e:
print('Validation error:', e.message)
# For debugging: print(e.path, e.schema_path)
If the file has a single record, this flow leaves it ready to load into the database. If it fails, it returns a readable message with the reason (e.g. "'date' is a required property" or "0 is less than the minimum of 1").
Step 5: Validate multiple records and log errors
In a pipeline it's common to receive many files or a file with multiple records. Iterate, validate with Draft7Validator to obtain all errors and log them to a log file. Here is a function that separates valid and invalid records and writes errors to an organized json.
from jsonschema import Draft7Validator, FormatChecker
import json
def validate_records(records, schema):
validator = Draft7Validator(schema, format_checker=FormatChecker())
valid = []
invalid = []
for i, rec in enumerate(records):
errors = list(validator.iter_errors(rec))
if errors:
invalid.append({
'index': i,
'errors': [e.message for e in errors]
})
else:
valid.append(rec)
return valid, invalid
with open('sales_batch.json', 'r', encoding='utf-8') as f:
batch = json.load(f) # assume é uma lista de registos
valid, invalid = validate_records(batch, sales_schema)
print(f'Válidos: {len(valid)}, Inválidos: {len(invalid)}')
with open('validation_errors.json', 'w', encoding='utf-8') as out:
json.dump(invalid, out, indent=2, ensure_ascii=False)
Practical example: in a batch of 5,000 records, you quickly get a count and a validation_errors.json file with entries like {"index": 37, "errors": ["'total' is a required property"]} for later analysis.
Step 6: Common errors and how to fix them
Frequent errors: wrong types (string instead of number), missing fields, dates in unknown formats. Resolution: adjust the JSON producer, normalize data before validating (for example convert strings to numbers), or update the schema to reflect optional fields. Some concrete strategies:
- Pre-processing: convert "qty": "2" → int(2) before validating if the producer sends strings.
- More tolerant schema: use "anyOf" to accept string or number for legacy fields.
- Log metrics: count % of invalid records per day and set a threshold (e.g., above 5% stop automatic ingestion).
Verify the result
Confirm that validation worked by checking the output: validation_errors.json file with detailed errors and the number of valid/invalid records. For a quick test, change a field in sale_example.json to a wrong type and confirm that a validation error appears. In a test environment, create datasets with 10, 100 and 1,000 records to measure performance: jsonschema is fast enough for hundreds of thousands of checks per minute on a modern CPU, but for millions consider validating in parallel or using streaming validation.
Conclusion
Validating JSON schemas in Python for data with jsonschema reduces pipeline failures and makes handling invalid records easier. Next steps: integrate this validation into an ETL step, normalize data before validating, or use more advanced schemas (e.g.: custom formats). Practical tip: start by creating unit tests with valid and invalid examples (e.g. 10 cases of each) to ensure the schema covers real cases and monitor the error percentage after integration.