Grant Scraper: Internal Funding Discovery for LTMA Consultancy

An internal web scraping and matching tool I built for LTMA Consultancy, a grant-writing firm I co-founded. It automates the discovery of government and arts funding opportunities and helps the team match clients — mostly artists — to grants they're actually eligible for.

This was never a commercial product. It was tooling for our own workflow, built because the manual version of the job was eating our time.

The Problem

Grant writing starts with grant finding, and finding grants manually is slow. Australian funding opportunities are scattered across federal, state, and local portals, plus arts-specific bodies, each with its own site structure and no unified feed. Application windows are often short, so an opportunity you discover late is an opportunity you can't use.

Before this tool, our research process was exactly what you'd imagine: someone opening a dozen browser tabs, checking the same portals week after week, and keeping notes in a spreadsheet. It worked, but it didn't scale with the client list, and it depended on whoever was doing the checking not missing anything.

The Approach

The system does three things:

1. Scrape — Scheduled Scrapy spiders crawl the funding portals we care about and extract grant listings: title, description, eligibility text, funding amount, and closing date. Each portal gets its own spider subclassing a shared base, since every site structures things differently.
2. Store — Everything lands in PostgreSQL with deduplication, so the team has one searchable database instead of a dozen bookmarked portals. Celery and Redis handle the scheduling and job queue.
3. Match — Text-similarity scoring compares grant descriptions and eligibility text against client profiles (discipline, location, career stage) to surface likely fits, so the team reviews a shortlist instead of everything.

The matching is deliberately assistive rather than authoritative — it ranks candidates for a human to review, it doesn't make eligibility decisions. Grant eligibility criteria are too messy and too legalistic to trust to keyword scoring alone.

Scraping Responsibly

The most important design decision was being a polite scraper. These are mostly government sites, and getting blocked would defeat the whole purpose.

class BaseGrantScraper(scrapy.Spider):
    """Shared base for all portal spiders."""

    custom_settings = {
        'DOWNLOAD_DELAY': 2,
        'RANDOMIZE_DOWNLOAD_DELAY': True,
        'ROBOTSTXT_OBEY': True,
        'USER_AGENT': 'LTMA Grant Research Bot (+contact URL)',
        'CONCURRENT_REQUESTS_PER_DOMAIN': 2,
    }

Rate limiting, robots.txt compliance, an identifiable user agent, and low per-domain concurrency. Boring, but it's why the scrapers kept working.

The other hard-earned pattern was defensive parsing. Government sites change their markup without notice and are wildly inconsistent between (and within) portals, so every extraction is wrapped in validation — a failed parse logs a warning and skips the record rather than poisoning the database with half-empty rows.

Deduplication mattered more than I expected. The same grant frequently appears on multiple portals (a state program relisted on a federal aggregator, for example), so records are keyed on a normalised source URL with fuzzy title matching as a backstop. Without that, the database fills up with near-duplicates and the team stops trusting it.

Matching Clients to Grants

The matching layer is straightforward text similarity rather than anything exotic. Each client has a profile — discipline, location, career stage, and a free-text description of their practice — and each grant has its scraped title, description, and eligibility text. TF-IDF vectors over both sides give a ranked shortlist:

def rank_grants(client_profile: str, grants: list[Grant]) -> list[tuple[Grant, float]]:
    """Rank open grants by text similarity to a client profile."""
    corpus = [client_profile] + [g.searchable_text() for g in grants]
    matrix = TfidfVectorizer(stop_words='english', ngram_range=(1, 2)).fit_transform(corpus)
    scores = cosine_similarity(matrix[0:1], matrix[1:]).flatten()
    return sorted(zip(grants, scores), key=lambda pair: pair[1], reverse=True)

A few hard filters run before scoring — closing date in the future, location eligibility where the grant states one explicitly — because no similarity score should surface a grant that closed last week. Everything past those filters is a suggestion, not a verdict.

Stack

• Python + Scrapy + BeautifulSoup for the scraping layer
• PostgreSQL for grant storage, with full-text search for the team's ad-hoc queries
• Celery + Redis for scheduled crawls and processing jobs
• Basic NLP / text similarity (TF-IDF-style scoring) for client-to-grant matching
• AWS for hosting

What It Changed for Us

Honest version: I don't have before/after metrics — we didn't instrument the old manual process, and this was an internal tool for a small team, not a product with analytics.

What I can say is what it replaced. Grant discovery went from a recurring manual chore across many portals to checking one database that refreshed itself. The team stopped worrying about missing opportunities and spent research time on the part that actually needs judgment: assessing fit and writing the applications. For a grant-writing firm, that's the whole game — the writing is the billable work; the searching never was.

Lessons Learned

• Scrape politely or don't bother. Rate limits, robots.txt, and an honest user agent are what keep a scraper alive long-term against government sites.
• Expect broken markup. Portal HTML changes silently and often. Validation and graceful skipping mattered more than any clever extraction logic.
• Keep matching assistive. Eligibility criteria are too nuanced for confident automation. Ranking candidates for human review was the right ceiling for the tool.
• Internal tools are underrated portfolio work. Building for your own company means you feel every rough edge yourself — the feedback loop is immediate and brutal, and the tool only survives if it genuinely helps.