Automation and web data
Turn repetitive file work, public data, and HTML pages into a checked workflow that can run again tomorrow.
By the end, you’ll be able to…
- Plan file automation with dry-run output and collision rules.
- Fetch JSON or HTML with timeouts, validation, and polite request limits.
- Write a reproducible report with explicit failure handling.
01Turn a routine into a contract
Automation starts with a written routine: which files are inputs, which folder receives them, and what happens when a name already exists? Write the plan as a pure function when possible. A function that groups paths by suffix can be tested without moving anything. Once the plan is correct, a separate function can execute it with a dry-run flag.
pathlib represents paths as objects and avoids manual string concatenation. shutil handles moves and copies. Before moving files, decide whether to overwrite, skip, or add a suffix. A dry run should print the same decisions the real run would make. This habit prevents an automation script from becoming a destructive surprise.
from pathlib import Pathdef group_by_suffix(paths): groups = {} for path in paths: suffix = path.suffix.lower() or "(none)" groups.setdefault(suffix, []).append(path.name) return groupsfiles = [Path("notes.md"), Path("photo.JPG"), Path("script.py"), Path("README")]for suffix, names in sorted(group_by_suffix(files).items()): print(suffix, names)(none) ['README'] .jpg ['photo.JPG'] .md ['notes.md'] .py ['script.py']
02Fetch only what you can validate
urllib.request is in the standard library and is enough for a straightforward GET request. Set a timeout, send a helpful User-Agent, check the HTTP status, and treat the response body as untrusted until your code has validated its shape. A network call can fail because of DNS, TLS, a timeout, a redirect, or a server error; each is a different failure to report.
Keep retrieval separate from normalization. fetch_json can return a decoded object, while normalize_items can reject a missing key or an unexpected type. This makes the transformation testable with a local dictionary and keeps a temporary network failure from being confused with bad data. For heavier projects, requests is a common third-party alternative, but the same boundary rules apply.
import jsondef normalize_items(payload): if not isinstance(payload, dict) or not isinstance(payload.get("items"), list): raise ValueError("expected an object with an items list") return [int(item["id"]) for item in payload["items"] if "id" in item]raw = json.loads('{"items": [{"id": "3"}, {"id": 7}, {"name": "skip"}]}')print(normalize_items(raw))[3, 7]
03Parse HTML with a small parser
HTML is a tree, but you rarely need a full browser to read a few links. html.parser provides a streaming parser: subclass HTMLParser, handle the tags you care about, and ignore the rest. This avoids regular expressions that break when attribute order or whitespace changes. For complex pages, BeautifulSoup or a browser automation tool may be appropriate, but simpler pages deserve simpler tools.
A polite scraper respects the site's robots.txt rules, identifies itself, caches repeated requests, and leaves a delay between calls. Do not turn a learning exercise into a load test. Check the terms of the source, prefer an official API or download when one exists, and store only the data you need. The parser should extract facts, not execute scripts from the page.
from html.parser import HTMLParserclass LinkParser(HTMLParser): def __init__(self): super().__init__() self.links = [] def handle_starttag(self, tag, attrs): if tag == "a": href = dict(attrs).get("href") if href: self.links.append(href)parser = LinkParser()parser.feed('<a href="/docs">Docs</a> <a href="https://example.com">Example</a>')print(parser.links)['/docs', 'https://example.com']
04Write a report you can rerun
A useful automation leaves a small, inspectable artifact. CSV is easy to open in a spreadsheet, JSON preserves nested shapes, and Markdown is readable in a repository. Write to a temporary file in the destination directory, then replace the old report only after the new content is complete. os.replace makes the final switch atomic on the same filesystem.
Log the source, counts, and failures so a second run can be compared with the first. If a scheduled job fails, the log should say whether the network call, the validation, or the write failed. Keep credentials in environment variables, not in the script. A report that can be regenerated from the same inputs is more valuable than a one-off file no one can reproduce.
import csvimport osfrom pathlib import Pathrows = [{"name": "Ada", "score": 92}, {"name": "Lin", "score": 81}]out = Path("report.csv")temporary = out.with_suffix(".csv.tmp")with temporary.open("w", newline="", encoding="utf-8") as handle: writer = csv.DictWriter(handle, fieldnames=["name", "score"]) writer.writeheader() writer.writerows(rows)os.replace(temporary, out)print(out.read_text(encoding="utf-8").strip())name,score Ada,92 Lin,81
The ideas worth keeping.
- Separate planning, execution, retrieval, validation, and reporting so each part can be tested.
- Treat downloaded HTML and JSON as untrusted input and bound every network wait.
- Write reports atomically and log enough context to explain a failed run.
Make it yours.
Type and test your code right here in the interactive workspace. Run your solution, check the output, and compare with the worked guide when ready.
Plan a folder move
Write plan(paths) that groups file names by lowercase suffix and returns a dictionary sorted by suffix. Do not move anything. Demonstrate a dry-run print.
Explore a worked solution
from pathlib import Pathdef plan(paths): groups = {} for path in paths: groups.setdefault(path.suffix.lower() or "(none)", []).append(path.name) return dict(sorted(groups.items()))files = [Path("A.TXT"), Path("b.md"), Path("README"), Path("c.txt")]for suffix, names in plan(files).items(): print(f"{suffix}: {', '.join(names)}")The suffix and name are derived from Path objects, so the plan is deterministic. A real executor can consume the same dictionary after the plan is reviewed.
Extract unique links
Write a LinkCollector that gathers href values and returns them in first-seen order without duplicates. Test it with repeated and missing href attributes.
Explore a worked solution
from html.parser import HTMLParserclass LinkCollector(HTMLParser): def __init__(self): super().__init__() self.links = [] self.seen = set() def handle_starttag(self, tag, attrs): if tag != "a": return href = dict(attrs).get("href") if href and href not in self.seen: self.seen.add(href) self.links.append(href)parser = LinkCollector()parser.feed('<a href="/a">A</a><a href="/a">Again</a><a>No link</a><a href="/b">B</a>')print(parser.links)The set makes the deduplication check constant time, while the list preserves the order in which links first appeared.
What stayed with you?
Check the ideas while they’re fresh. Your answers stay in this page and feedback appears as you go.
Plan a folder move
Write plan(paths) that groups file names by lowercase suffix and returns a dictionary sorted by suffix. Do not move anything. Demonstrate a dry-run print.
suffix: ', '.join(names)
Keep exploring.
The official references behind this chapter, when you want a second angle or the full details.
One chapter closer.
Reading is a start. Trying it yourself makes the connection.