Dashboard
Scrape Berita
Spesifikasi teknis modul Scrape Berita (/dashboard/scraper), kontrol job perayapan portal, serta pemetaan database PostgreSQL, Elasticsearch, dan Kafka.
Scrape Berita (/dashboard/scraper)
Modul Scrape Berita menyediakan kontrol pemicu scraping ad-hoc (on-demand crawl) berdasarkan kata kunci target, URL portal tertentu, serta audit log eksekusi scraper GolangScraper A1 dan B1.
Tabel Pemetaan Komponen ke PostgreSQL & Elasticsearch
| Komponen / Bagian UI | PostgreSQL (Tabel & Kolom Kunci) | Elasticsearch (Indeks & Field Kunci) | Kafka (Topik & Alur Event) |
|---|---|---|---|
Kontrol Job Scraping & TargetPOST /api/v1/scraper/triggerGET /api/v1/master/scraping-targets | • scraping_targets↳ id, organization_node_id, domain_name, base_url, scraping_config, is_active• scraping_target_keywords↳ scraping_target_id, taxonomy_keyword_id | • scraper_audit_logs↳ target_domain, urls_fetched, http_status_distribution, duration_ms, timestamp | queue-jobqueue-control-signal(Scraper Job Bus) |
Data Source Artikel
Spesifikasi teknis modul Data Source Artikel (/dashboard/data-source-artikel), volume artikel per domain portal media, serta pemetaan database PostgreSQL, Elasticsearch, dan Kafka.
Services
Spesifikasi teknis modul Services (/dashboard/services), kontrol dan jadwal service pipeline, serta pemetaan database PostgreSQL, Elasticsearch, dan Kafka.