scrapers

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 22, 2025 License: MIT Imports: 32 Imported by: 0

Documentation

Index

Constants

View Source
const API_KEY string = "0001085cc708b9cef47080f064612ca5"

API_KEY is the Concept3D API key observed from map.utdallas.edu traffic.

View Source
const BASE_URL string = "https://profiles.utdallas.edu/browse?page="

BASE_URL is the root listing endpoint for UTD professor profiles.

View Source
const CAL_URL string = "https://calendar.utdallas.edu/api/2/events"
View Source
const END_URL string = "/?map=" + UTD_MAP_ID + "&key=" + API_KEY

END_URL appends the map and key query parameters for Concept3D requests.

View Source
const START_URL string = "https://api.concept3d.com"

START_URL points to the Concept3D API host.

View Source
const UTD_MAP_ID string = "1772"

UTD_MAP_ID references the Concept3D map identifier for the UTD campus map.

Variables

View Source
var MAX_EVENTS_PER_DAY = 5000

MAX_EVENTS_PER_DAY caps the Astra API results to guard against truncated responses.

Functions

func ScrapeAcademicCalendars

func ScrapeAcademicCalendars(outDir string)

func ScrapeAstra

func ScrapeAstra(outDir string)

ScrapeAstra iterates day-by-day through Astra events and persists the raw JSON output.

func ScrapeCometCalendar

func ScrapeCometCalendar(outDir string)

ScrapeCometCalendar retrieves calendar events through the API

func ScrapeCoursebook

func ScrapeCoursebook(term string, startPrefix string, outDir string, resume bool)

ScrapeCoursebook scrapes utd coursebook for the provided term (semester)

func ScrapeMapLocations

func ScrapeMapLocations(outDir string)

ScrapeMapLocations downloads Concept3D responses and writes raw map data to disk.

func ScrapeMazevo

func ScrapeMazevo(outDir string)

ScrapeMazevo pulls Mazevo calendar events via the public API and stores the raw response.

func ScrapeOrganizations

func ScrapeOrganizations(outdir string)

ScrapeOrganizations authenticates with SharePoint and exports the student organization directory CSV.

func ScrapeProfiles

func ScrapeProfiles(outDir string)

ScrapeProfiles navigates UTD profile listings and writes professor metadata to JSON.

Types

type APICalendarResponse

type APICalendarResponse struct {
	Events []RawEvent        `json:"events"`
	Page   map[string]int    `json:"page"`
	Date   map[string]string `json:"date"`
}

APICalendarResponse models the calendar API pagination envelope.

type AcademicCalendar

type AcademicCalendar struct {
	Title string
	Href  string
	Time  string
}

type RawEvent

type RawEvent struct {
	Event map[string]any `json:"event"`
}

RawEvent mirrors the nested event payload returned by the calendar API.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL