HTTP APIs with urllib
Treat transport, decoding, schema validation, and retries as separate responsibilities.
By the end, you’ll be able to…
- Send requests with explicit timeouts and response limits.
- Distinguish HTTP status errors from transport failures.
- Validate decoded JSON before domain code uses it.
01HTTP success has several layers
An API call crosses multiple boundaries: establishing a connection, receiving an HTTP response, decoding bytes, parsing JSON, and checking an application schema. A successful TCP connection is not proof of a successful HTTP response, and valid JSON is not proof of valid business data. Keep these steps separate so failures have useful context and can be tested independently.
urllib.request belongs to the standard library and is sufficient for many simple HTTP integrations. Supply an explicit timeout, expected response type, and a response-size limit. A socket timeout does not establish one absolute deadline across every redirect and every read. Operations with strict overall budgets need additional deadline design, particularly when retries or pagination can multiply the elapsed time.
02Bound the transport adapter
The adapter below checks a successful status, a JSON media type, and a byte limit before decoding. urllib raises HTTPError for many unsuccessful HTTP statuses; HTTPError is also a URLError subclass, so catch it first when separate reporting is useful. A timeout or DNS failure is a different category from a well-formed response saying the request was rejected.
This network example is for a terminal and is marked non-runnable in the lesson sandbox. Use HTTPS and trusted endpoint configuration. If users can provide URLs, validate allowed schemes, destinations, redirects, and network access before fetching; a timeout alone does not prevent server-side request forgery. Never attach an authorization token to an arbitrary user-selected host.
import jsonfrom urllib.error import HTTPError, URLErrorfrom urllib.request import Request, urlopendef fetch_catalog(url): request = Request(url, headers={'Accept': 'application/json'}) try: with urlopen(request, timeout=5) as response: if response.status != 200: raise ValueError(f'unexpected status: {response.status}') if response.headers.get_content_type() != 'application/json': raise ValueError('expected JSON response') payload = response.read(100_001) except HTTPError as error: raise RuntimeError(f'HTTP status {error.code}') from error except URLError as error: raise RuntimeError('transport failed') from error if len(payload) > 100_000: raise ValueError('response exceeds byte limit') return json.loads(payload.decode('utf-8'))03Validate untrusted JSON explicitly
JSON objects become dictionaries and arrays become lists, but neither carries your application’s type contract. Check the top-level shape, required fields, accepted types, numeric ranges, and string lengths before persisting data. In Python, bool is an int subclass; type(value) is int is useful when a field must reject true and false rather than interpret them as one and zero.
Set limits before expensive work when possible. A byte limit reduces exposure but is not a complete defense against deeply nested or expensive inputs. Decide whether unknown fields are rejected or ignored for forward compatibility. The local fixture example deliberately accepts only the fields it needs and constructs a fresh validated record instead of passing an untrusted dictionary through the application.
04Retries need a policy
Retry only failures that your service contract identifies as transient, with bounded attempts, delays, and an overall deadline. Rate-limit responses may provide Retry-After information. An invalid schema is usually a compatibility or data problem, not a reason to hammer the same endpoint. Authentication failures likewise need configuration correction rather than repeated identical requests.
Distinguish safe reads from operations that create effects. A timed-out POST might already have created an order, so an automatic retry can duplicate it unless an idempotency mechanism is part of the API contract. Test the parser with bytes and the transport with a local fixture or injected adapter. Keep network access out of fast correctness tests to avoid depending on availability and live data.
The ideas worth keeping.
- Bound transport time and response size explicitly.
- Parse syntax and validate domain data separately.
- Retry only under a documented, effect-aware policy.
Make it yours.
Type and test your code right here in the interactive workspace. Run your solution, check the output, and compare with the worked guide when ready.
Reject the wrong numeric type
Validate a stock record containing quantity. Accept nonnegative integers, reject bool and negative values, and return only the validated quantity.
Explore a worked solution
def quantity(record): if not isinstance(record, dict): raise ValueError('expected object') value = record.get('quantity') if type(value) is not int or value < 0: raise ValueError('quantity must be a nonnegative integer') return valueassert quantity({'quantity': 0}) == 0for bad in [True, -1, '3']: try: quantity({'quantity': bad}) except ValueError: pass else: raise AssertionError('invalid quantity accepted')An exact int check excludes booleans, preventing JSON true from quietly becoming a quantity of one.
Test with an injected fetcher
Write load_title(fetch) that consumes fixture bytes from an injected zero-argument fetch callable. Reject missing or empty titles.
Explore a worked solution
import jsondef load_title(fetch): data = json.loads(fetch().decode('utf-8')) if not isinstance(data, dict): raise ValueError('expected object') title = data.get('title') if not isinstance(title, str) or not title.strip(): raise ValueError('title is required') return title.strip()assert load_title(lambda: b'{"title":" Python "}') == 'Python'try: load_title(lambda: b'{}')except ValueError: passelse: raise AssertionError('missing title accepted')Dependency injection exercises decoding and schema behavior without a real request. A separate adapter test should cover HTTP statuses, headers, and timeouts.
What stayed with you?
Check the ideas while they’re fresh. Your answers stay in this page and feedback appears as you go.
Reject the wrong numeric type
Validate a stock record containing quantity. Accept nonnegative integers, reject bool and negative values, and return only the validated quantity.
[Process exited with code 0 - No output]
Keep exploring.
The official references behind this chapter, when you want a second angle or the full details.
One chapter closer.
Reading is a start. Trying it yourself makes the connection.