Chapter 34 — Data Validation and Configuration: A Doorkeeper Called Pydantic

Series I4 — Advanced Python Language

NoteIn this chapter, you will:

• Understand why data from outside must be validated, and the limits of a dataclass
• Build Pydantic models: validation, type conversion, and error messages
• Add rules with Field, field_validator, and model_validator
• Read app settings from environment variables and .env (pydantic-settings)
• Validate field data from five sectors at once: the Series I4 project

All scripts for this chapter are in the kode/ch34/ folder. On the web version, you can also download them all at once (zip).

In Chapter 33, type hints acted as labels read by a checking tool. But while the program runs, those labels guard nothing. Yet field data often comes from outside: forms, CSV files, JSON from an API, a clerk’s typing. It can have typos, be empty, or be beyond common sense (water pH 17, a negative tree diameter).

A doorkeeper is needed: a checker that works while the program runs, rejects bad data with a clear message, and tidies data that is still acceptable. In Python, the most popular doorkeeper is Pydantic.

%%{init: {"flowchart": {"diagramPadding": 40}}}%%
flowchart LR
  A["Raw data<br/>(dict, JSON, .env)"] --> B["Pydantic model<br/>(types + rules)"]
  B -- "passes" --> C["Clean object<br/>with correct types"]
  B -- "fails" --> D["ValidationError<br/>with a clear message"]

Pydantic as a doorkeeper: raw data is checked, then becomes a clean object or is rejected with an error message.

A dataclass is tidy but does not guard

In Chapter 16 you met dataclass: a compact way to make a class that holds data. See what happens when the content makes no sense:

01_dataclass_no_checks.py

# Chapter 34 - A dataclass is tidy, but does not check the content
from dataclasses import dataclass


@dataclass
class Tree:
    species: str
    diameter_cm: float
    healthy: bool


good = Tree("Teak", 32.5, True)
print(good)

messy = Tree("Teak", "thirty", "maybe")   # no complaint at all
print(messy)
print(type(messy.diameter_cm).__name__)

Tree(species='Teak', diameter_cm=32.5, healthy=True)
Tree(species='Teak', diameter_cm='thirty', healthy='maybe')
str

The object messy is created without complaint: the diameter holds the text 'thirty', healthy holds 'maybe'. Its type is str, although we wrote float. A dataclass only tidies up writing the class; it does not check the content. The mistake is felt later, perhaps far from its source, when that number gets calculated.

A Pydantic model

Pydantic is a data-validation library that uses type hints as rules. The version installed on the author’s computer has major version 2 (the script prints only the “major version”, because the full number changes over time). This book uses the Pydantic 2 style.

02_pydantic_model.py

# Chapter 34 - A Pydantic model: checks and converts types
import pydantic
from pydantic import BaseModel

print("Pydantic major version:", pydantic.VERSION.split(".")[0])


class Tree(BaseModel):
    species: str
    diameter_cm: float
    healthy: bool = True          # default value


t = Tree(species="Teak", diameter_cm="32.5", healthy="true")   # text becomes number and bool
print(t)
print(type(t.diameter_cm).__name__, type(t.healthy).__name__)
print(t.model_dump())

t2 = Tree(species="Meranti", diameter_cm=45)   # int becomes float
print(t2.diameter_cm, t2.healthy)

Pydantic major version: 2
species='Teak' diameter_cm=32.5 healthy=True
float bool
{'species': 'Teak', 'diameter_cm': 32.5, 'healthy': True}
45.0 True

• A model is made as a subclass of BaseModel, and each attribute is given a type. healthy: bool = True has a default value.
• Pydantic converts when it makes sense: the text "32.5" becomes the float 32.5, the text "true" becomes True, and the whole number 45 becomes 45.0.
• The result is an object with correct types: float and bool (not str).
• model_dump() turns the object back into an ordinary dict.

There is a small lesson here. When I tried to fill healthy="ya" (the Indonesian word for “yes”), Pydantic rejected it (bool_parsing): it does not know that word. What it knows are standard forms such as "true". Pydantic’s conversion is limited and documented, not guessing.

Clear error messages

When data cannot be accepted, Pydantic raises a ValidationError that explains all the problems at once, not only the first.

03_error_messages.py

# Chapter 34 - Clear error messages
from pydantic import BaseModel, ValidationError


class Tree(BaseModel):
    species: str
    diameter_cm: float
    healthy: bool


try:
    Tree(species="Teak", diameter_cm="thirty", healthy="maybe")
except ValidationError as e:
    print("number of problems:", e.error_count())
    for problem in e.errors(include_url=False, include_input=False):
        print(problem["loc"], "|", problem["type"], "|", problem["msg"])

try:
    Tree(species="Teak")
except ValidationError as e:
    for problem in e.errors(include_url=False, include_input=False):
        print(problem["loc"], "|", problem["type"], "|", problem["msg"])

number of problems: 2
('diameter_cm',) | float_parsing | Input should be a valid number, unable to parse string as a number
('healthy',) | bool_parsing | Input should be a valid boolean, unable to interpret input
('diameter_cm',) | missing | Field required
('healthy',) | missing | Field required

In the first attempt there are two problems: diameter_cm cannot be read as a number (float_parsing) and healthy cannot be read as yes/no (bool_parsing). In the second attempt, two required fields are not filled in (missing: “Field required”). Each problem has:

• loc: where the problem is (the field name),
• type: a stable error category that a program can process,
• msg: an explanatory sentence for humans.

The options include_url=False and include_input=False remove the link and the copy of the input from the display, so the output is short and stable.

Extra rules: Field and validators

A type alone is not enough. “A number” does not yet mean “sensible”. Extra rules can be attached with Field (simple limits) and validators (your own rules).

04_validation_rules.py

# Chapter 34 - Rules through Field and validators
from typing import Literal

from pydantic import BaseModel, Field, ValidationError, field_validator, model_validator


class Fish(BaseModel):
    pond: str = Field(min_length=1, max_length=3)
    species: Literal["catfish", "tilapia"]
    count: int = Field(gt=0)
    avg_weight_g: float = Field(ge=1, le=5000)
    water_ph: float = Field(ge=0, le=14)

    @field_validator("pond")
    @classmethod
    def upper_case(cls, value: str) -> str:
        return value.upper()          # tidies up at the same time

    @model_validator(mode="after")
    def check_catfish_weight(self):
        if self.species == "catfish" and self.avg_weight_g > 2000:
            raise ValueError("an average catfish weight above 2000 g is not realistic")
        return self


print(Fish(pond="a", species="catfish", count=800, avg_weight_g=540.5, water_ph=7.2))

for data in [
    dict(pond="B", species="carp", count=100, avg_weight_g=300, water_ph=7),
    dict(pond="B", species="tilapia", count=0, avg_weight_g=300, water_ph=15),
    dict(pond="C", species="catfish", count=10, avg_weight_g=3000, water_ph=7),
]:
    try:
        Fish(**data)
    except ValidationError as e:
        for m in e.errors(include_url=False, include_input=False):
            print(m["loc"], "|", m["msg"])
        print("--")

pond='A' species='catfish' count=800 avg_weight_g=540.5 water_ph=7.2
('species',) | Input should be 'catfish' or 'tilapia'
--
('count',) | Input should be greater than 0
('water_ph',) | Input should be less than or equal to 14
--
() | Value error, an average catfish weight above 2000 g is not realistic
--

• Field(gt=0) means “greater than 0”, ge/le mean “greater or equal”/“less or equal”, and min_length/max_length limit text length.
• Literal["catfish", "tilapia"] limits the fish species. "carp" is rejected: “Input should be ‘catfish’ or ‘tilapia’”.
• @field_validator("pond") checks (and may tidy) one field: the pond name "a" becomes "A".
• @model_validator(mode="after") checks a combination of fields after all are valid: catfish with an average weight above 2000 g is rejected, and the error location is () (empty) because the problem belongs to the whole model, not one field.

WarningWARNING

Limits such as avg_weight_g <= 5000 or “catfish no more than 2000 g” in this script are only sample numbers for practice, not farming advice. Set the real limits together with an extension officer or an experienced fish farmer.

App settings: environment variables and .env

An application has settings that differ on each computer: file locations, download limits, test mode, service keys. Putting them inside the code is a nuisance and, for secret keys, dangerous. A common practice: keep them in environment variables (settings outside the program) or in a .env file (a list of NAME=value), and read them when the program starts.

The pydantic-settings package reads settings into a class that is validated like a Pydantic model. This script uses fake values (no real secrets) and writes a sample .env file into _tmp/.

05_settings_env.py

# Chapter 34 - App settings from environment variables and a .env file (fake values)
import os
from pathlib import Path

from pydantic import SecretStr
from pydantic_settings import BaseSettings, SettingsConfigDict

folder = Path(__file__).parent / "_tmp"
folder.mkdir(exist_ok=True)
env_file = folder / "example.env"
env_file.write_text("FIELD_UNIT_NAME=Example Pond\nFIELD_API_TOKEN=fake-token-not-a-secret\n"
                    "FIELD_DOWNLOAD_LIMIT=5\n", encoding="utf-8")


class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="FIELD_", env_file=env_file, env_file_encoding="utf-8")

    unit_name: str = "No name"
    download_limit: int = 10
    api_token: SecretStr = SecretStr("")
    test_mode: bool = False


s = Settings()
print("from .env       :", s.unit_name, s.download_limit, s.test_mode)
print("token (masked):", s.api_token)
print("token (opened on purpose):", s.api_token.get_secret_value() == "fake-token-not-a-secret")

os.environ["FIELD_DOWNLOAD_LIMIT"] = "25"          # an environment variable beats .env
os.environ["FIELD_TEST_MODE"] = "true"
s2 = Settings()
print("env beats .env:", s2.download_limit, s2.test_mode)

print("a direct argument is strongest:", Settings(download_limit=1).download_limit)

from .env       : Example Pond 5 False
token (masked): **********
token (opened on purpose): True
env beats .env: 25 True
a direct argument is strongest: 1

• env_prefix="FIELD_" means the setting download_limit is read from FIELD_DOWNLOAD_LIMIT. The prefix prevents clashes with variables of other programs.
• Values from .env are converted automatically: "5" becomes the int 5.
• SecretStr masks the value when printed (**********), so keys do not leak into logs. The real value opens only through get_secret_value(), which we use here only to compare.
• The order of strength, according to the pydantic-settings documentation: direct argument > environment variable > .env file > default value. The script proves it: .env gives 5, the environment variable overrides it to 25, and the argument Settings(download_limit=1) beats them all.

TipTIP

A .env file holds personal and often secret settings. Do not keep it in Git (add it to .gitignore, Chapter 20), and provide a sample file (.env.example) without secret values. In this book all values are fake.

JSON, lists of models, and schemas

06_json_and_schema.py

# Chapter 34 - JSON, lists of models, and JSON Schema
import json

from pydantic import BaseModel, TypeAdapter


class Record(BaseModel):
    sector: str
    unit: str
    value: float


raw = '{"sector": "pond", "unit": "A", "value": "540.5"}'
r = Record.model_validate_json(raw)           # JSON text -> validated object
print(r)
print(r.model_dump_json())

records = TypeAdapter(list[Record]).validate_python([
    {"sector": "forest", "unit": "F1", "value": 32.5},
    {"sector": "orchard", "unit": "O2", "value": "245.5"},
])
print(len(records), records[1].value)

schema = Record.model_json_schema()           # basis of API docs and AI agent tools
print(json.dumps(schema, indent=2))

sector='pond' unit='A' value=540.5
{"sector":"pond","unit":"A","value":540.5}
2 245.5
{
  "properties": {
    "sector": {
      "title": "Sector",
      "type": "string"
    },
    "unit": {
      "title": "Unit",
      "type": "string"
    },
    "value": {
      "title": "Value",
      "type": "number"
    }
  },
  "required": [
    "sector",
    "unit",
    "value"
  ],
  "title": "Record",
  "type": "object"
}

• model_validate_json validates JSON text directly ("540.5" becomes a number). model_dump_json() turns the object back into JSON.
• TypeAdapter(list[Record]) validates a list of models at once.
• model_json_schema() produces a JSON Schema: a standard description of the shape of the data (fields, types, which are required). This schema is the basis of an API’s automatic documentation.

Dataclass or Pydantic?

	dataclass	Pydantic BaseModel
Checks content while running	no	yes
Converts types ("32.5" to 32.5)	no	yes
Detailed error messages	no	yes
Part of the standard library	yes	no (extra library)
Suits	data inside a program that is already trusted	data coming in from outside

Rule of thumb: data crossing the “border” (forms, files, APIs, settings) is validated with Pydantic; clean internal data may be a dataclass.

The link to FastAPI and AI agents

This checking at the border is the basis of the next two topics in the book. FastAPI (Chapter 36), a web framework, uses Pydantic: the FastAPI package installed on the author’s computer lists pydantic as a dependency (result of pip show fastapi), and in Chapter 36 you will see for yourself Pydantic models checking API input (in this chapter that is not run). In the discussion of AI agents at the Advanced level, similar schemas are used to describe the shape of data to a language model; that part has not been tested in this chapter.

Running Project I4: validating field data from five sectors

Field data from the five sectors has different shapes: trees have a diameter, the orchard has a yield, the ricefield has grain, the pond has weight and pH, the barn has livestock weight. We build five models, then one “gate” that picks the model by reading the sector field (called a discriminated union).

07_project_five_sector_validation.py

# Chapter 34 - Running project I4: validating field data from five sectors
from typing import Annotated, Literal

from pydantic import BaseModel, Field, TypeAdapter, ValidationError


class Forest(BaseModel):
    sector: Literal["forest"]
    unit: str
    diameter_cm: float = Field(gt=0, le=300)


class Orchard(BaseModel):
    sector: Literal["orchard"]
    unit: str
    yield_kg: float = Field(ge=0)


class Ricefield(BaseModel):
    sector: Literal["ricefield"]
    unit: str
    grain_kg: float = Field(ge=0)


class Pond(BaseModel):
    sector: Literal["pond"]
    unit: str
    avg_weight_g: float = Field(gt=0)
    water_ph: float = Field(ge=0, le=14)


class Barn(BaseModel):
    sector: Literal["barn"]
    unit: str
    weight_kg: float = Field(gt=0)


FieldData = Annotated[Forest | Orchard | Ricefield | Pond | Barn, Field(discriminator="sector")]
checker = TypeAdapter(FieldData)

raw = [
    {"sector": "forest", "unit": "F1", "diameter_cm": "32.5"},
    {"sector": "orchard", "unit": "O1", "yield_kg": 245.5},
    {"sector": "ricefield", "unit": "R1", "grain_kg": -10},
    {"sector": "pond", "unit": "A", "avg_weight_g": 540.5, "water_ph": 7.2},
    {"sector": "pond", "unit": "B", "avg_weight_g": 498, "water_ph": 17},
    {"sector": "barn", "unit": "D1", "weight_kg": 412.5},
    {"sector": "swamp", "unit": "E1", "avg_weight_g": 100},
    {"sector": "barn", "unit": "D2"},
]

valid, rejected = [], []
for number, row in enumerate(raw, start=1):
    try:
        valid.append(checker.validate_python(row))
    except ValidationError as e:
        problem = e.errors(include_url=False, include_input=False)[0]
        rejected.append((number, ".".join(str(x) for x in problem["loc"]) or "sector", problem["msg"]))

print("valid:", len(valid), "| rejected:", len(rejected))
print("valid object types:", sorted({type(o).__name__ for o in valid}))
for number, place, reason in rejected:
    print(f"row {number}: {place} -> {reason}")

valid: 4 | rejected: 4
valid object types: ['Barn', 'Forest', 'Orchard', 'Pond']
row 3: ricefield.grain_kg -> Input should be greater than or equal to 0
row 5: pond.water_ph -> Input should be less than or equal to 14
row 7: sector -> Input tag 'swamp' found using 'sector' does not match any of the expected tags: 'forest', 'orchard', 'ricefield', 'pond', 'barn'
row 8: barn.weight_kg -> Field required

Eight raw rows go in, four pass and four are rejected:

• Row 1: diameter_cm is the text "32.5", converted and accepted as a Forest.
• Row 3: grain_kg is -10, rejected because it must be >= 0.
• Row 5: pH 17, rejected because it is above 14.
• Row 7: the sector "swamp" is not one of the five, rejected, and the message names the valid choices.
• Row 8: the field weight_kg is missing (“Field required”).
• Object types that passed: Barn, Forest, Orchard, Pond, each with a different shape yet all correctly typed.

Rejected data does not stop the program: it is recorded with its reason, and the rest is processed. That is the safe way of working with dirty field data.

Exercises

1. Make a model Livestock with fields barn: str, kind: Literal["cattle", "goat", "chicken"], age_months: int (must be >= 0), and weight_kg: float (must be > 0). Try one correct record and one wrong one.
2. Change 03_error_messages.py: give diameter_cm=-5 to a model that has Field(gt=0). Read the error message.
3. In 05_settings_env.py, add a setting output_folder: str with default "results", then override it through an environment variable and through .env. See which one wins.
4. Challenge: add a sixth model Equipment to 07_project_five_sector_validation.py (with sector equal to "tool"), put it in the union, and add two raw rows (one correct, one wrong). Recount the passed and rejected rows.

Chapter Summary

• Data from outside must be validated while the program runs; type hints alone guard nothing.
• A dataclass does not check content; a Pydantic BaseModel checks, converts, and gives per-field error messages.
• Extra rules: Field (limits), field_validator (one field), model_validator (combined fields).
• pydantic-settings reads settings from environment variables and .env; order of strength: argument > environment > .env > default; SecretStr masks secrets.
• TypeAdapter, model_validate_json, and model_json_schema validate lists, JSON, and produce schemas.
• Project I4: five-sector data is validated through a discriminated union; four pass, four are rejected with reasons.

Next, in Chapter 35: async and concurrency, a way to do many waiting jobs at once, and the close of Series I4.