Distributed observing infrastructure
BHTOM
A web platform coordinating time-domain observations and automated photometric processing across a global telescope network.
- Software
- Infrastructure
- Delivery
01 / Context
Problem
Time-domain astronomy needs observations from many independent instruments to be coordinated, processed and made useful on short timescales.
02 / Scope
My responsibility
I worked across the software and infrastructure boundary, including containerisation and pipeline integration, as part of international collaboration within BHTOM.
03 / System
How it works
A deliberately simplified view of system boundaries and data movement.
- 01Telescope network
- 02Portal and data intake
- 03Workflow orchestration
- 04CCD reduction and calibration
- 05Standardised photometry
04 / Delivery
What I built
- Containerised services and processing components for reproducible operation.
- Integrated the complete processing pipeline with Prefect during engineering collaboration within BHTOM.
- Worked on automated CCD image reduction within the wider BHTOM system.
- Collaborated internationally within BHTOM on system and pipeline integration.
05 / Decisions
Constraints and trade-offs
- Reproducibility and operability mattered more than exposing internal implementation detail.
- Independent instruments produce heterogeneous inputs, so the public case study focuses on boundaries and data flow rather than private deployment specifics.
06 / Tools
Project stack
The technologies used across the system, grouped by the job they perform.
Application and API
-
Python
Service and processing code
-
Django
Portal backend and application framework
-
Django REST Framework
Programmatic access and service interfaces
-
TOM Toolkit
Astronomical target and observation management foundation
Data and workflows
-
PostgreSQL
Relational application data
-
MongoDB
Supporting platform data services
-
Prefect
Processing workflow orchestration
-
Apache Kafka
Distributed event and data transport
Deployment and observability
-
Docker
Reproducible service and pipeline deployment
-
Linux
Operating environment for distributed services
-
Nginx
Reverse proxy and static delivery
-
Gunicorn
Python application serving
-
Prometheus
Operational metrics
-
Graylog
Centralised log management
-
Elasticsearch
Search and log storage