Concepts

Architecture

Koala consists of multiple separate processes (red) and middleware components (blue).

Middleware

Data Management (DB)

Primary datastore. Contains asset metadata, details about ingest and retrieval requests, as well as known filetypes, schemas, and user accounts.

Currently supported: MySQL

Message Queue (MQ)

The message queue is used to asynchronously route ingest and retrieval requests to loader/retriever applications.

Currently supported: RabbitMQ

Archival Storage

Physically stores the assets. Multiple implementations of an Archival Storage client are available:

• TSMClient — Full IBM Spectrum Protect integration via ctypes bindings to the native C API
• NFSHSMClient — Local filesystem-based archival for testing or lightweight deployments

The client is selected via the LTP_IMPLEMENTATION configuration variable.

Cache

Stores heartbeat information about running applications, shown on the web application. Retrieves and caches statistics about archived packages from the main database.

Currently supported: Redis

Applications

Scheduler

Monitors the upload directory for new SIPs and publishes the data for further processing to the message queue. Only one scheduler must be running at the same time.

Loader

Multiple loaders can be started simultaneously, connecting to the message broker and processing SIPs concurrently.

Processing includes the following steps:

• Unzipping/unpacking
• Virus scanning (ClamAV)
• Validating files and metadata
• Calculating checksums
• Storing metadata in DB and assets in Archival Storage

The status of the ingest process is continuously updated and can be monitored through the Web UI or API.

Proxy

A reverse proxy (nginx or Apache) for SSL offloading and serving of static content. Not a koala application — deployed externally.

Web

Provides an HTTP API for third-party applications and a web UI for admin operations. Includes WebSocket/SocketIO support for real-time dashboard updates.

Retriever

Multiple retrievers can be started simultaneously, connecting to the message broker and processing retrieval requests from clients. AIPs are fetched from the Archival Storage system, converted to a DIP, and saved in the HTTP-accessible downloadarea.

The status of the retrieval process is continuously updated and can be monitored through the Web application.

Purger

Monitors the downloadarea filesystem size and removes old packages once the high watermark is reached. Also handles recovery from fatal loader states by locking workareas and cleaning up incomplete ingest transactions.

Statistics

Queries the main database, calculates statistics, and saves the information in the cache. Supports five resolution levels: minute, hour, day, week, and month.

Processes

Ingest

The ingest process is modeled as a state machine with done, error, and fatal end states. As soon as the client gets a "done" ticket from the web application, the AIP is saved on stable storage (archival storage) and every participating system has the required data (e.g., MDQI, data management).

Every ingest of a SIP is a "transaction" — a failing step triggers a recovery/rollback operation. Recovery is handled by:

• LoaderErrorState — for recoverable errors during normal processing
• Purger — for fatal errors where the loader crashed mid-operation

During recovery, the purger locks the corresponding workarea and blocks the loader from restarting until cleanup is complete.

Retrieve

When a client requests a specific package, a retriever saves the data to the HTTP-accessible downloadarea. DIPs remain accessible until a configured threshold is reached (typically 70% downloadarea usage). Then DIPs are removed from oldest to newest by the purger.

Recovery

When a loader crashes during ingest (e.g., due to OOM, container restart, or hardware failure), the purger detects the orphaned workarea and initiates recovery:

1. Purger acquires a lock on the workarea
2. Rolls back partial changes (removes from DB, archival storage, MDQI)
3. Cleans up temporary files
4. Releases the lock, allowing the loader to resume

This ensures data consistency even in the face of unexpected failures.

Circuit Breaker

The loader includes a circuit breaker that automatically pauses the scheduler when repeated errors of the same type exceed a configurable threshold. This prevents cascading failures and gives operators time to investigate systemic issues.

The circuit breaker configuration:

Variable	Purpose
LOADER_CIRCUIT_BREAKER_ENABLED	Enable/disable the circuit breaker
LOADER_CIRCUIT_BREAKER_EXCEPTION_TYPE_MAX_FAILED_COUNT	Max errors before opening the circuit
LOADER_CIRCUIT_BREAKER_EXCEPTION_TYPE_EXPIRATION_IN_SEC	Seconds before the circuit resets

Heartbeat

Each koala application sends periodic heartbeat signals to Redis. The web UI displays the current status, uptime, and processing state of all running applications. Heartbeats are also used by container orchestrators to detect unhealthy instances.

Permissions

The following permissions control access to UI and API functions:

Permission	Description
admin	Full access to all UI functions and API endpoints
monitor	View-only access to the dashboard
api	Execute all API functions
api_restricted	Execute all API functions except deletion of assets and AIPs

Permissions are assigned when creating or editing users via the web UI. See Authentication for details.