Getting Started¶

How to use koala from a client perspective.

Regular SIP Workflow¶

Steps:

Set up metadata extraction tool
Extract metadata from file
Pack metadata file (mets.xml) and content into a SIP archive
Upload SIP via SFTP or REST API
Check ingest progress via the ticket system or callback webhook

Step 1: Setup metadata extraction toolkit (FITS)¶

To ensure long-term archivability, metadata must be extracted beforehand.

root@host:~/getting-started# wget https://github.com/harvard-lts/fits/releases/download/1.5.0/fits-1.5.0.zip
root@host:~/getting-started# unzip fits-1.5.0.zip

Step 2: Create the SIP¶

The following script extracts metadata from a PDF and creates a SIP archive:

root@host:~/getting-started# cat mets-template.sh

#!/bin/bash

FITS_TOOL_PATH=/root/getting-started/fits.sh

__FILENAME__="littledorrit0000dick_o8l6.pdf" # the file to archive

__CREATE_DATE__="$(date --iso-8601=seconds)"
__AGENT_NAME__="GWDG"
__AGENT_NOTE__="Automatically generated metadata"
__PID__="urn:nbn:de:gwdg:test" # must be generated by PID service
__MASTER_CREATION_DATE__="$(date --iso-8601=seconds)"
__METADATA_CREATION_DATE__="$(date --iso-8601=seconds)"
__METADATA_RECORD_CREATOR__="GWDG"
__NR_OF_FILES__=1
__FORMAT__="urn:diasid:fty:kopal:0200507050000000000064"
__FILE_CREATED__="$(date --iso-8601=seconds)"
__SIZE__="$(stat --printf="%s" ${__FILENAME__})"
__CHECKSUM__="$(sha1sum ${__FILENAME__} | awk '{ print $1 }')"
__XOR_OF_FILE_CHECKS=${__CHECKSUM__}
__CHECKSUMTYPE__="SHA-1"
__MIMETYPE__="application/pdf"

UUID=$(uuidgen)

mkdir -p ${UUID}/content
cp -v ${__FILENAME__} ${UUID}/content
cd ${UUID}

cat << EOF > mets.xml

<?xml version="1.0" encoding="UTF-8"?>
<mets OBJID="" TYPE="kopal Submission Information Package" PROFILE="DNB" xsi:schemaLocation="http://www.loc.gov/METS/ http://www.loc.gov/standards/mets/mets.xsd    http://www.ddb.de/LMERfile http://files.dnb.de/standards/lmer/lmer-file.xsd    http://www.ddb.de/LMERobject http://files.dnb.de/standards/lmer/lmer-object.xsd    http://www.ddb.de/LMERprocess http://files.dnb.de/standards/lmer/lmer-process.xsd    http://purl.org/dc/elements/1.1/ http://dublincore.org/schemas/xmls/qdc/2003/04/02/dc.xsd" xmlns="http://www.loc.gov/METS/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:lmerFile="http://www.ddb.de/LMERfile" xmlns:lmerObject="http://www.ddb.de/LMERobject" xmlns:lmerProcess="http://www.ddb.de/LMERprocess" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <metsHdr CREATEDATE="${__CREATE_DATE__}" RECORDSTATUS="TEST">
    <agent ROLE="ARCHIVIST" TYPE="ORGANIZATION">
      <name>${__AGENT_NAME__}</name>
      <note>${__AGENT_NOTE__}</note>
    </agent>
  </metsHdr>
  <amdSec ID="AmdSec-0001">
    <techMD ID="TechMD-LMER-Object">
      <mdWrap ID="TechMD-LMER-Object-MdWrap" MIMETYPE="text/xml" MDTYPE="OTHER" OTHERMDTYPE="lmerObject" LABEL="LMERobject">
        <xmlData>
          <lmerObject:persistentIdentifier>${__PID__}</lmerObject:persistentIdentifier>
          <lmerObject:transferChecksum CHECKSUMTYPE="xor of sha1 file checksums">${__XOR_OF_FILE_CHECKS__}</lmerObject:transferChecksum>
          <lmerObject:masterCreationDate>${__MASTER_CREATION_DATE__}</lmerObject:masterCreationDate>
          <lmerObject:metadataCreationDate>${__METADATA_CREATION_DATE__}</lmerObject:metadataCreationDate>
          <lmerObject:metadataRecordCreator>${__METADATA_RECORD_CREATOR__}</lmerObject:metadataRecordCreator>
          <lmerObject:numberOfFiles>${__NR_OF_FILES__}</lmerObject:numberOfFiles>
        </xmlData>
      </mdWrap>
    </techMD>
    <techMD ID="TechMD-File--0">
      <mdWrap ID="TechMD-File--0-MDWRAP" MIMETYPE="text/xml" MDTYPE="OTHER" OTHERMDTYPE="lmerFile">
        <xmlData>
          <lmerFile:format REGISTRYNAME="DIAS">${__FORMAT__}</lmerFile:format>
          <lmerFile:xmlData MDTYPE="FITS">

            $(./fits.sh -i ${__FILENAME__})

          </lmerFile:xmlData>
        </xmlData>
      </mdWrap>
    </techMD>
  </amdSec>
  <fileSec>
    <fileGrp ID="ASSET" ADMID="TechMD-LMER-Object">
      <file ID="FILE-0" ADMID="TechMD-File--0" CREATED="${__FILE_CREATED__}" SIZE="${__SIZE__}" CHECKSUM="${__CHECKSUM__}" CHECKSUMTYPE="${__CHECKSUMTYPE__}" MIMETYPE="${__MIMETYPE__}">
        <FLocat LOCTYPE="URL" xlink:type="simple" xlink:href="file://./content/${__FILENAME__}"/>
      </file>
    </fileGrp>
  </fileSec>
  <structMap TYPE="ASSET">
    <div ORDER="1" LABEL="File list" TYPE="ASSET">
      <fptr FILEID="FILE-0"/>
    </div>
  </structMap>
</mets>

EOF

zip -r sip-${UUID}.zip *
echo done

Step 3: Upload the SIP¶

Via SFTP¶

First, request transport parameters from the API:

curl -u user:password https://<koala-host>/api/ftpinfo

Then upload using SFTP:

root@host:~/getting-started# export SSHPASS=your-password-here
sshpass -e sftp -oBatchMode=no -b - ftpuser@<koala-host> << !
   cd <upload-area-path>
   put 7f55da15-6551-4e6a-ac3a-13f6df1328ff.zip.tmp
   mv 7f55da15-6551-4e6a-ac3a-13f6df1328ff.zip.tmp 7f55da15-6551-4e6a-ac3a-13f6df1328ff.zip
   bye
!

Important: Use a unique temporary filename (e.g., <uuid>.tmp) during transfer. The scheduler only processes files matching the configured SCHEDULER_SIP_FILE_PATTERN. Renaming the file triggers ingestion.

Via REST API (Simple Blob Storage)¶

For simple bitstream preservation without metadata:

curl -u user:password -X POST --data-binary @"myfile.pdf" https://<koala-host>/api/sbs

Response:

{"internal_asset_id": "63b54de9-dc2b-4bff-be28-e2ad92d02b5c"}

Step 4: Check ingest progress¶

Via REST API¶

curl https://<koala-host>/api/ingeststatus/<sip-basename>

Response:

{
  "create_ts": "2018-06-28T09:11:45",
  "modify_ts": "2018-06-28T09:11:45",
  "originator": "loader@24d737fc147f",
  "status": "done",
  "substate": "done",
  "suppl_info": null
}

Valid statuses: uploading, busy, done, error.

Via DIAS XML endpoint¶

curl https://<koala-host>/IngestStatus/Ingest?TicketId=<sip-basename>

Via callback webhook¶

Configure LOADER_CALLBACK_URL to receive HTTP POST notifications when ingest completes (success or error). See Configuration and API.

Step 5: Retrieve the asset¶

After successful ingest, retrieve the asset:

curl -u user:password -X POST -H "Content-Type: application/json" \
  -d '{"type": "asset", "format": "zip"}' \
  https://<koala-host>/api/retrieve/internal/<internal-asset-id>

This returns a session_id. Poll the status endpoint until status is ready:

curl https://<koala-host>/api/retrievestatus/<session_id>

Error Scenarios¶

Validation failure¶

If the SIP metadata fails validation (missing external_asset_id, invalid XSD, etc.), the ingest status will show status: "error" with details in suppl_info. The SIP is moved to the errorarea.

Virus detected¶

If ClamAV detects malware, the ingest status will show status: "error" with the infected file path in suppl_info. The SIP is moved to the errorarea.

File type not allowed¶

If a content file has a type not in the allowed filetypes list, the ingest fails with status: "error".

Callback webhook failure¶

If the configured LOADER_CALLBACK_URL is unreachable, the error is logged and recorded in suppl_info, but the ingest itself is not affected.