Sources & methodology

How we collect & process data

• Collection. A scheduled scraper reads public city and school-district websites, document APIs, business-directory records, news feeds, and state/federal data services. It identifies itself with a contact address, obeys each site's robots.txt, and waits at least 3 seconds between requests to the same host.
• Cleaning. Text is normalized — HTML entities decoded, Unicode standardized, invisible characters removed, whitespace collapsed — so summaries read cleanly.
• Deduplication. Repeated items are merged: documents by web address, news and agenda items by title and date, school records by school (combining the Future Ready PA profile with federal enrollment figures).
• Summaries. News snippets are tidied (mastheads and ALL-CAPS letterheads removed, trimmed to a sentence or two).
• Verification. Records are schema-validated, and a weekly link checker flags documents whose links go dead.

Known limitations

• Meetings & votes. We don't yet parse a meeting schedule or vote records; we link to the official agendas, minutes, and BoardDocs instead.
• CASD document host & robots.txt. The CASD file host returns an error for its robots.txt rather than rules. Under RFC 9309 §2.3.1.4, an unavailable robots.txt means no crawl restrictions are published, so we treat those documents as link-only — we show the official link for you to click through, and do not download or re-host the files.
• BoardDocs. The CASD BoardDocs site disallows automated access in its robots.txt, so we honor that and link to it directly rather than scraping it.
• Accuracy. Data is collected automatically and may contain errors, delays, or omissions. Always confirm anything important with the linked official source.

Source status

Curated official links