Our crawler
studyjob.ai indexes public information so that students can see which student jobs fit their degree. This page says exactly what our crawler does, so site operators can decide how to treat it.
- User agent
- studyjob-bot/1.0 (+https://studyjob.ai/bot; hello@studyjob.ai)
- What it fetches
- University programme pages, programme regulations (studieordninger) and curriculum PDFs (course titles, codes, ECTS, semesters, prerequisites, programme objectives), and the universities' public course catalogues. Course descriptions are read as extraction input and are never republished. Job postings are collected from public job boards through the Apify scraping platform, whose own crawlers fetch them, so the pace, user agent and robots.txt rules below describe studyjob-bot's requests, not those. Every posting links back to the original.
- How politely
- One request at a time per host, never more than one every two seconds, always with the user agent above. A full pass over a university is at most a few thousand requests, most of them course catalogue pages. Each university is re-collected once per semester: one run in spring and one in autumn, after the universities publish their course catalogues. A run can span a few nights, and between runs we send only small test fetches (at most a hundred pages per site a day) when a parser changes. Re-collections use conditional requests where the site supports them, so unchanged pages are not downloaded again. The crawler never logs in, never uses personal accounts and never circumvents access controls, CAPTCHAs or Cloudflare challenges.
- robots.txt
Honoured per path, with one disclosed exception: the IT University of Copenhagen's public LearnIT course base, whose robots.txt disallows crawlers. We read the course descriptions linked from ITU's own programme structure pages once per semester, as internal input for matching only. That text is never republished or shown on this site, and ITU can have it removed by emailing hello@studyjob.ai.
The Copenhagen Business School and University of Copenhagen sources we read (programme pages on www.cbs.dk, studieordninger.cbs.dk, kursuskatalog.cbs.dk, www.ku.dk and kurser.ku.dk, plus the admission statistics on ufsn.dk) allow crawlers on the pages we fetch, and we never request a path a site's robots.txt disallows, such as the search pages and the parameter URLs (?key=value) on www.ku.dk.
- Personal data
- Staff names on course pages are dropped on ingest. Recruiter and job-poster metadata is stripped from postings before storage. Individuals are never shown; only postings, programmes and aggregates.
- Opting out
- Add a Disallow rule for studyjob-bot in your robots.txt, or email hello@studyjob.ai. Opt-out requests are applied within 24 hours and the affected content is removed.