Councils polled389VEvery night for subscribers, less often for the rest Applications held2.9mMProduction database, Sep 2026 Classification124 tagsVAcross 6 categories, versioned rules Automated tests4,158VAcross 266 test files What it does
- Pulls new planning applications every night from PlanIt, with the Planning London Datahub as a second source for London boroughs.
- Classifies each one with a versioned rules engine: is it residential, is the address deliverable, what kind of work is it.
- Matches applications to subscribers by trade and radius, creates the lead and sends the letter through Stannp. A follow-up goes out automatically when the application is approved.
- Builds each builder's letterhead from their own website: it pulls the logo, colours and company details, and an LLM drafts the letter copy for them to edit.
- Enriches a lead on demand with the full postal address, EPC data and an automated valuation.
- Handles subscriptions with metered overage, and publishes lead pages by county and town for search.
What made it hard
- Upstream rate limits. The main data source locks callers out for 20 to 40 minutes at a time, so polling runs through a queue that retries when the lockout clears and resumes from a cursor, never from scratch.
- Keeping 389 councils fresh without polling them all nightly: councils with subscribers go nightly, warm ones every third night, the rest weekly, and an alert fires if one goes 40 hours untouched.
- Two data sources in one shape. London records arrive in a different format and are normalised into the same pipeline, with checks that reject impossible coordinates.
- A classifier that can be trusted: 22 gate rules, 10 scope rules and 25 classify rules held as data, versioned, with every decision recorded so the whole table can be re-run when a rule changes.
- Geography without PostGIS: a bounding-box filter and a distance check in code, plus index and planner tuning that took the biggest lead pages from 1.7 seconds to about 0.1.
- Print rules: artwork has to stay clear of the postal address window and the tracking code, so every letter is checked against those zones before it goes to print.
- Running 31 job queues reliably, with their own retry schedules, a dead-letter queue and a guard against jobs running twice.
Stack
- App
- TypeScriptNext.js 16React 19Tailwind 4TiptapMapLibre GL
- Data and jobs
- PostgreSQL 16Prisma 6Redis 7BullMQ (31 queues)
- Rendering
- Playwright (PDF pool)pdf-libsharp
- Data sources
- PlanItPlanning London Datahubpostcodes.ioIdeal Postcodes (PAF)EPC RegisterChimnie AVM
- Services
- Stannp (print and post)Stripe (metered)ResendAnthropic ClaudeSentry
- Infrastructure
- DockerCaddyDigitalOceanGitHub ActionsVitest
Candidates found200k+MJun 2026 snapshot Address points joined41.5mVOS Open UPRN, read in slices Postcode districts2,292VEngland and Wales Automated tests525VPython and web What it does
- Runs a six-stage pipeline per postcode district, joining energy certificates, address points and Land Registry title plots into one picture of what sits on each piece of land.
- Flags the title plots that fit the pattern it's looking for, and works out how big each one is from the address data.
- Grades how sure it is about ownership (gold, silver or bronze) from title and sales history, then scores each candidate on size, time since last sale, tenure, energy rating and age.
- Finds the owner from Land Registry's company ownership data, and their directors from Companies House.
- Buys official title documents through the Land Registry Business Gateway and posts letters to owners.
- Shows it all on a map, filtered by area and score.
What made it hard
- National datasets on a modest server: 41.5 million address points and 31 million sales are never loaded whole. Queries are pushed down into Parquet files so only the slice needed is read.
- Spatial joins done properly in British National Grid (EPSG:27700), then served in WGS84, with a rule for address points that fall inside overlapping title polygons.
- Council reorganisations: EPC records sit under council codes that no longer exist, so there is a map from old codes to new, and Land Registry downloads are matched against about 318 council file names.
- Matching owners fails closed. Postcode, building and street must all agree, because a wrong match means a letter to the wrong person.
- Money paths that spend real fees are locked so a job can never pay twice, with spend caps per organisation.
- Land Registry's new sales-to-address lookup only covers new sales, so it is archived daily to build a history.
Stack
- Backend
- Python 3.12FastAPISQLAlchemyGeoAlchemy2CeleryRedis
- Geospatial
- PostGIS 3.4GeoPandasShapelypyprojFionapyarrow / Parquet
- Frontend
- Next.js 16React 19MapLibre GLTurf
- Open data
- EPC registerOS Open UPRNHMLR INSPIREHMLR Price PaidHMLR CCOD/OCOD
- Services
- Companies House APIHMLR Business Gateway (SOAP)StannpWeasyPrint
Postcode district boundaries: missinglink/uk-postcode-polygons, CC BY-SA 3.0.
Properties valued3,712MAug 2026 snapshot Address match rate94.4%MNew sales linked to a UPRN Automated tests1,400+VTypeScript and Python What it does
- Surfaces houses listed at least 10% below a market value worked out from several sources, and grades them A to D on discount, seller motivation and area.
- Values each property as a confidence-weighted mix of automated valuations and its own comparables: Land Registry prices times EPC floor area, re-priced with the house price index.
- Reads listing text with an LLM for condition, motivation and red flags, checked against a strict schema.
- Checks planning constraints, flood and coal-mining risk, schools, deprivation and rents for every property.
- Builds a branded desktop valuation report per property from an evidence pack.
What made it hard
- Joining datasets that share no common key. The address matcher fails closed: same postcode, same house number, street must agree, and two possible matches means no match.
- Proving the valuation works. A backtest showed the number of comparables predicts nothing, but how much they disagree does, so confidence is now banded on their spread: 12.3% median error when they agree, 19.3% when they don't.
- Keeping paid lookups under control: every call is logged against a ledger.
- Knowing where each source falls short, such as a police force that publishes no crime data, and letting that part of the score drop out rather than guess.
Stack
- App
- TypeScriptNext.js 16React 19Prisma 7BullMQMapLibre GLTurf
- Data
- PostgreSQL 16Redis 7
- Open government data
- HMLR Price PaidUK HPI (SPARQL)EPC registerONSPDIMD 2019 and 2025ONS private rentsplanning.data.gov.ukEnvironment Agency floodCoal AuthorityGIAS and OfstedOS Open UPRNHMLR INSPIRE
- Reports
- Pythonheadless Chrome (DevTools protocol)poppler
Companies7VEach with its own month-end Lettable units66VAcross 21 properties Reconciliation1pVTolerance before anything is posted Automated tests1,248VAcross 81 files What it does
- Reads the letting agent's statement PDFs, matches every line to a tenancy and flags missing rent, short payments and anything it can't place.
- Runs a month-end for each company as a state machine, from draft to closed, with a proper reversal if a month is reopened.
- Posts to Xero with a tracking category per property, and to FreeAgent by project, including bank transaction explanations.
- Tracks 11 kinds of safety certificate with renewal alerts at 30 days, 7 days and expiry.
- Holds insurance policies, sums insured and premium history, checked against valuations, plus mortgage facilities, rates and loan-to-value.
What made it hard
- Statement PDFs whose multi-column rows come out of a normal parser in the wrong order. Rows are rebuilt from each word's position on the page, and the parse fails loudly if a section total is out by more than a penny.
- Nothing posts to the accounts unless the lines add up, to the penny, to the money that actually arrived.
- A parser for each insurer's documents, tested against 60 real schedules, certificates and invoices, and used to back-fill about eight years of history.
- Money in integer pence everywhere, encrypted OAuth tokens, and jobs designed so running one twice changes nothing.
- No AI in the money path: parsing is deterministic, so the same PDF always gives the same answer.
Stack
- App
- TypeScriptNext.js 15React 19TailwindAuth.js
- Data and jobs
- PostgreSQL 16Prisma 5Redis 7BullMQ
- Documents
- pdf.js (positional extraction)pdf-parsecsv-parse
- Finance
- Xero APIFreeAgent API
- Infrastructure
- DockerCaddy 2HetznerVitest
It replaced a paid property management package.
Fee invoices reconciled653VTwo years of Amazon fees Recoverable VAT6 figuresVIdentified for reclaim SKUs forecast36VAcross three brands What it does
- Forecasts 12 months of demand per SKU and flags stock risk, then sizes each restock shipment for Amazon fulfilment and writes the plan to the team sheet.
- Watches inbound shipments for any that have stalled.
- Builds weekly, monthly and quarterly scorecards for the account.
- Reconciled Amazon fee invoices against settlement reports and the ERP month by month, to find the input VAT that was never claimed.
What made it hard
- Forecasting through stockouts: a month counts as a stockout if the buy box share or traffic collapses, so lost sales don't read as lost demand. Recent months weigh more, seasonality is damped and new lines follow a launch curve.
- Restock rules that match how the warehouse works: aim for six weeks of cover, round up to a full pallet once a shipment is 70% of one, and hold off if there are already six weeks in stock.
- Amazon's fee VAT invoices have no API, so 653 of them were collected from Seller Central by browser automation and matched against settlements.
- Reconciliation traps: a line-level tax test threw up 383,768 false alarms because net amounts are after discount and tax is before it; and paging a SuiteQL query without an order silently corrupts totals.
- Rate limits handled properly, retrying on the times Amazon asks for.
- Writing to a shared sheet safely: it re-reads before writing, stops if a colleague has changed it, and keeps a backup.
Stack
- Code
- Python 3.12FastAPIHTMXJinja2SQLite (WAL)matplotlibpytest
- Amazon SP-API
- FBA InventoryProduct PricingFulfillment Inbound v0Finances v0Settlement reportsLWA auth
- Other APIs
- NetSuite SuiteQL (TBA, read-only)Shopify GraphQL (ShopifyQL)Google Sheets
Client described rather than named: a contract manufacturer in nutrition and supplements.
Discount codes audited7,352VWith live checkout tests Orders verified1,300+MAfter a sync outage, none missed Change requests8VEach approved before any write What it does
- Audited the store: discount codes, product catalogue, redirects and theme.
- Made fixes through the Admin API on a duplicate theme, then published: Liquid template edits and metafield updates, with nothing edited live.
- Archived dead products, added redirects and moved subscribers off retired products in Recharge.
- Diagnosed an outage in the Shopify to ERP order sync and verified, order by order, that every order was imported once it was fixed.
- Traced why stock levels kept reverting between Shopify and the ERP, and designed a replacement sync.
What made it hard
- Change control: the scripts that write to the store refuse to run without an approved change request, so every change is logged and reversible.
- A deprecated API field: from the 2025-01 API, Shopify reports every variant as manually fulfilled, which broke how the connector built its item map.
- A connector script that allowed one active mapping per item per store, when items needed both a UK and an EU variant, so manual fixes were undone every two hours.
- Quantity semantics: copying the ERP's "available" into Shopify's "available" double-counts committed stock and shows kits as sold out.
- Subscription swaps where the API silently ignores a filter, which would have moved 40 pack-size subscriptions to the wrong product had it not been caught.
- Designing the new sync to be safe: compare-and-set writes that only change stock if it is still what the sync last saw, and idempotent requests.
Stack
- Shopify
- Admin GraphQL APIREST redirectsLiquid themesmetafieldsinventorySetQuantities
- Subscriptions
- Recharge API
- ERP and integration
- NetSuite REST and SuiteQL (TBA)Celigo
- Scripts
- Bash (curl, jq)Python
Client described rather than named: a sports nutrition brand selling direct to consumers.
Source records46,122VFrom search, maps and crawling Prospects kept13,358VAfter de-duplication and checks Towns in the sweep5,229VWorked through in population order Guard test cases50+VAgainst invented facts and injection What it does
- Works through 5,229 UK towns in population order, finding firms in the target trades from search results and maps.
- Crawls each firm's website for contact details, company number and services, and uses an LLM to decide whether it really is a prospect rather than a directory.
- Verifies every email before use and confirms the company with Companies House.
- Scores every new record against existing prospects and merges duplicates automatically above a threshold, with a review queue below it.
- Writes the opener around a real event in their area from the last 30 days, taken from the product's own data, then checks every fact in it against the record before it can be sent.
- Sends through a dedicated sending platform, sweeps replies every 20 minutes and drafts answers for a human to approve, feeding a small CRM.
What made it hard
- Keeping directories and aggregator sites out of the list, and getting the prospect-or-not call right.
- Stopping the LLM inventing facts, or being steered by text on a scraped page. A deterministic guard rejects anything not in the source record and falls back to a template.
- Controlling spend across several paid data services: a daily budget in pence, a deadline per run, and one attempt per job so a retry never pays twice.
- Doing it responsibly: suppression checked on email, domain, company number and phone; erasure that keeps only a hash so a person is never re-imported; and a written legitimate-interest assessment.
- Merging prospects safely when they already have send history attached.
Stack
- Code
- TypeScriptBullMQ workersjsdomPlaywright
- Data sources
- DataForSEOGoogle PlacesCompanies House APIHunter.ioMillionVerifier
- AI
- Claude Haiku 4.5Message Batches API
- Sending
- Instantly API v2webhooks
Sites moved13VAll serving from Cloudflare Entries migrated14,800M13,718 of them on one large content site Redirects kept23,700+VEvery old URL still resolves URL parity13,709 of 13,710VThe largest site, checked before cutover What it does
- Exports everything from WordPress: the database directly, WP-CLI on the server or the REST API, whichever the site allows.
- Converts posts and pages to MDX in Astro content collections, with schemas that fail the build if any SEO field is missing.
- Recreates titles, descriptions, canonicals, robots rules and structured data from the old SEO plugin, and turns its redirects into Cloudflare redirect rules.
- Adds static search, freezes old comments as HTML and replaces PHP forms with Cloudflare functions.
- Crawls old and new sites side by side before the DNS switch, so nothing is lost at cutover.
What made it hard
- Redirects at scale: 22,887 rules exported from the SEO plugin, some with several sources each, became a 2.2 MB map bundled into a Cloudflare Worker.
- Platform limits: Pages caps a site at 20,000 files, so one site's 65,773 images went to R2 storage; the largest deploys about 71,000 files, and its 30,000 forum pages are generated after the build because rendering them normally ran out of memory.
- Keeping old .html URLs working, which Pages redirects by default. The largest site moved to Workers with that switched off and its own index handling.
- Different URL conventions on every site: trailing slashes, language prefixes, category bases.
- Content that doesn't convert cleanly: embedded audio, page-builder markup, form shortcodes, literal SEO template variables, stray non-breaking spaces and missing image alt text.
- Life without PHP: scheduled posts published by a cron trigger that fires a deploy.
Stack
- Build
- Astro 5HugoTypeScriptMDXZod schemasPagefindsharp
- Migration
- WP-CLIMySQL 8 (Docker)WordPress REST APITurndown
- Cloudflare
- PagesPages FunctionsWorkersR2Cron TriggersWeb AnalyticsWrangler
- Delivery
- GitHub ActionsResend (forms)
Something similar?
If one of these sounds like the problem you've got, tell me about it.