docuteam:feeder_steps
Unterschiede
Hier werden die Unterschiede zwischen zwei Versionen angezeigt.
Nächste Überarbeitung | Vorhergehende Überarbeitung | ||
docuteam:feeder_steps [2015/04/20 15:41] – angelegt christian | docuteam:feeder_steps [2023/02/06 14:52] (aktuell) – gelöscht Administrator | ||
---|---|---|---|
Zeile 1: | Zeile 1: | ||
- | ====== Allgemein ======= | ||
- | |||
- | Die Liste der Operationen kann über targets im build.xml erstellt werden: '' | ||
- | |||
- | ==== Hilfe-Output ==== | ||
- | |||
- | Um die Parameter einer Operation zu sehen, kann die entsprechende Klasse ohne Parameter auf der Shell aus dem Verzeichnis '' | ||
- | < | ||
- | Ergibt folgenden Output: | ||
- | < | ||
- | INFO 2014-01-02T10: | ||
- | Parameters: | ||
- | [path/ | ||
- | </ | ||
- | |||
- | ===== Admin ===== | ||
- | |||
- | ==== Version ==== | ||
- | |||
- | The Version class returns the current version number of the docuteam feeder library and the version numbers of the docuteam libraries it depends on. | ||
- | |||
- | < | ||
- | docuteam feeder: 2.4.9 (17.04.2015) | ||
- | docuteam darc: 2.14.2 (10.03.2015) | ||
- | docuteam converter: 1.0.4 (10.03.2015) | ||
- | docuteam tools: 1.10.6 (25.02.2015) | ||
- | AIPCreatorETH: | ||
- | |||
- | ===== Access ===== | ||
- | |||
- | ==== FedoraExporter ==== | ||
- | |||
- | The FedoraExporter class gets the record of a given PID with all its child objects, if there ary any. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | PID[, | ||
- | [/ | ||
- | [zip]: if ' | ||
- | |||
- | ===== ETH ===== | ||
- | |||
- | ==== CreateAIP ==== | ||
- | |||
- | The CreateAIP operation will create Rosetta Objects out of a SIP. The operation looks for a specific levels.xml file in a " | ||
- | config/ | ||
- | levels.xml | ||
- | dropbox/ | ||
- | [sip]/ | ||
- | mets.xml | ||
- | |||
- | < | ||
- | Parameters: | ||
- | SIPPath: Absolute path to the SIP to convert | ||
- | executionId: | ||
- | [testMode]: | ||
- | |||
- | ==== GetIngestFeedback ==== | ||
- | |||
- | The GetIngestFeedback operation gets feedback about the stored AIP. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | SIPPath: Absolute path to the SIP to convert | ||
- | executionId: | ||
- | |||
- | ===== Ingest ===== | ||
- | |||
- | ==== BARSIPConverter ==== | ||
- | |||
- | The BARSIPConverter converts a BARSIP SIP to a SIP, that the Matterhorn Profile. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | [targetFolder]: | ||
- | |||
- | ==== CheckWorkbenchSpace ==== | ||
- | |||
- | The CheckWorkbenchSpace checks whether the workbench has the necessary space available for the processing of the given SIP. By default, it calculates with three potential copies for a given SIP on the workbench at the same time. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | [targetFolder]: | ||
- | |||
- | ==== Cleanup ==== | ||
- | |||
- | The Cleanup is used to delete a given SIP from the work folder in the workbench. It will use the workbench defined in the '' | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | [prep]: if ' | ||
- | |||
- | ==== CreateEADFile ==== | ||
- | |||
- | The CreateEADFile operation creates a single EAD file from the EAD chunks of each node of a given SIP and puts it into the '' | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | |||
- | ==== ExtentCalculator ==== | ||
- | |||
- | The ExtentCalculator will calculate the amount of files within each folder node and assign its value to the respectives folder element ' | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | |||
- | ==== SIPFileMigrator ==== | ||
- | |||
- | The SIPFileMigrator compares the files of a SIP with the settings of a configuration file (migration-config.xml) and converts the files according to the definitions in that file. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | keepOriginals: | ||
- | |||
- | ==== SIPRemoveFromDropbox ==== | ||
- | |||
- | The SIPRemoveFromDropbox operation will remove a given SIP from the dropbox to an indicated folder or deletes it if no target folder is given. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | [targetFolder]: | ||
- | |||
- | ===== LIPDA ===== | ||
- | |||
- | ==== Confirmation ==== | ||
- | |||
- | The Confirmation class will send a GET request to the LIPDA website with the PIDs for a given publication. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | sip: name of the sip to process | ||
- | lipda_url: URL pointing to the LIPDA main page, e.g. http:// | ||
- | proxyHost: Hostname or IP of the proxy to use | ||
- | proxyPort: Port of the proxy to use</ | ||
- | |||
- | ==== Preparation ==== | ||
- | |||
- | The Preparation operation checks the ingest.lipda page of the LIPDA website | ||
- | for newly transfered documents. For each one, it will download the respective | ||
- | content and create a SIP based on a given submission agreement. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | lipda_url: URL pointing to the LIPDA main page, e.g. http:// | ||
- | no_sips: maximum number of SIPs to process; 0 = all | ||
- | agreement: ID of the submission agreement to use, e.g. sa_mv-4330-02 | ||
- | [proxyHost]: | ||
- | [proxyPort]: | ||
- | |||
- | ===== POYRY ===== | ||
- | |||
- | ==== CreateSIPsFromFileOrFolder ==== | ||
- | |||
- | The CreateSIPsFromFileOrFolder operation will create SIPs from a given file | ||
- | or folder. If the source is a folder, a parameter will define whether a | ||
- | single SIP or separate SIPs for each child should be created. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | source: file or folder for which an SIP should be generated | ||
- | split: if ' | ||
- | saID: value to use for referencing a submission agreement in the SIP | ||
- | dssID: value to use for referencing a data submission session of the respective submission agreement | ||
- | author: value to use as the creator for the SIP | ||
- | [dropbox]: optional location where to put the SIPs; if omitted the property ' | ||
- | |||
- | ===== Quality Assurance ===== | ||
- | |||
- | ==== FilePathLengthCheck ==== | ||
- | |||
- | Check if any canonical filepath within a given folder exceeds a given number | ||
- | of allowed characters. \\ | ||
- | Any file or folder exceeding the maximal allowed path lengh will be logged. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | / | ||
- | maxAllowedFilePathLength: | ||
- | |||
- | ==== SIPConfirmation ==== | ||
- | |||
- | Connect to the Fedora repository and get a single PID to identify the SIP. \\ | ||
- | In the sequence, this PID will be used as the main entry point in the | ||
- | repository for the submission. The value will be stored in the < | ||
- | element. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | [PID namespace[:### | ||
- | |||
- | ==== SIPConvertToSafeFileNames ==== | ||
- | |||
- | Rename files containing special characters. \\ | ||
- | Safe filenames contain only the characters A-Z, a-z, 0-9, and " | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | |||
- | ==== SIPDeleteBackupFiles ==== | ||
- | |||
- | Delete backup files of a SIP. \\ | ||
- | A list of filename patterns must be supplied to specify which files to delete. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | [filenamePattern filenamePattern ...]: a list of filename patterns (NOT case-sensitive, | ||
- | |||
- | ==== SIPExtractor ==== | ||
- | |||
- | Extract a zipped SIP into the work folder of the workbench. \\ | ||
- | The optional second argument can be used to indicte a different target folder. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | [targetdir]: | ||
- | |||
- | ==== SIPFixityCheck ==== | ||
- | |||
- | Check the files contained for conformance with the checksums in the METS | ||
- | file. \\ | ||
- | The results of the check will be written in the form of PREMIS events as | ||
- | inline xml code into the METS file. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | |||
- | ==== SIPPathLengthCheck ==== | ||
- | |||
- | Check if any canonical file path within the SIP exceeds the maximal allowed path length. \\ | ||
- | Any file or folder exceeding the maximal allowed path lengh will be logged. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | maxAllowedFilePathLength: | ||
- | |||
- | ==== SIPRootRemover | ||
- | |||
- | Remove the root node and set its child as the new root node. \\ | ||
- | This operation has only been used during early development | ||
- | and never been used in production environments. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | |||
- | ==== SIPSubmissionAgreementCheck ==== | ||
- | |||
- | Check the files formats contained for compliance with the submission agreement. \\ | ||
- | There are two modes: In the first mode (removeBadFiles = false), any files not | ||
- | complying with the submission agreement will be listed (using WARN log entries) and an | ||
- | error code will be returned. \\ | ||
- | In the second mode (removeBadFiles = true), any files not complying with the | ||
- | submission agreement will be deleted from the SIP, and the modified METS.xml | ||
- | will be saved (the original SIP remains untouched as a backup). | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | [removeBadFiles]: | ||
- | |||
- | ==== SIPVirusCheck ==== | ||
- | |||
- | Check each file of the SIP for viruses using a ClamAV (www.clamav.net). \\ | ||
- | A running clamav daemon is required. Depending on the second argument, it | ||
- | will either throw exceptions or automatically delete infected files. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | deleteInfected: | ||
- | |||
- | ===== Storage ===== | ||
- | |||
- | ==== ChecksumChecker ==== | ||
- | |||
- | The ChecksumChecker will check the objects with ORIGINAL datastreams in fedora against the generated checksum. \\ | ||
- | It will use the FEEDER_JAVA system variable to locate configuration files | ||
- | |||
- | < | ||
- | Parameters: | ||
- | mailto: | ||
- | Optional Parameters: | ||
- | [namespace] [namespace] ...: Fedora namespaces separated by space. If no namespace is given, all datastreams are checked.</ | ||
- | |||
- | ==== DIPDeliverer ==== | ||
- | |||
- | The DIPDeliverer gets the datastream(s) for a provided fedora PID or file format PUID, repackages each as a DIP | ||
- | and stores them in a given directory. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [' | ||
- | [PID|PUID] (PID = fedora persistent unique identifier, PUID = pronom persistent unique identifier) | ||
- | Optional Parameters: | ||
- | [targetLocation]: | ||
- | |||
- | ==== FedoraObjectUpdater ==== | ||
- | |||
- | This operation uploads new versions of object to the fedora server. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | |||
- | ==== FOXMLCreator ==== | ||
- | |||
- | The FOXMLCreator converts a given METS package into separate FOXML | ||
- | (Fedora Object) files. \\ | ||
- | It will use the current working directory or - if not available - the FEEDER_JAVA system variable to locate configuration files | ||
- | and the workbench defined in the docuteamFeeder.properties file. \\ | ||
- | The code makes use of the Directory Ingest tool which is available from the | ||
- | Fedora website. Modifications were done to support distinctions between root | ||
- | folders and folders through conversion rules (crules.xml) and to be able to | ||
- | handle already gotten PIDs during the ingest process. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | |||
- | ==== FOXMLIngester ==== | ||
- | |||
- | The FOXMLIngester will transfer a given list of FOXML (Fedora Object) files to a Fedora repository for storage. \\ | ||
- | It will use the FEEDER_JAVA system variable to locate configuration files | ||
- | and the workbench defined in the docuteamFeeder.properties file. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | keepFOXML: One of { true | false }, indicating whether to keep the FOXML files after a successful ingest; defaults to ' | ||
- | |||
- | ==== METSValidator ==== | ||
- | |||
- | The METSValidator validates the mets xml file with the linked schema | ||
- | definitions and places the namespace declarations from the root element to | ||
- | the respective elements. \\ | ||
- | This is a necessary preparation for the mets xml when it has to be split up | ||
- | into different parts, as is the case when several foxml files are being | ||
- | created out of the SIP (FOXMLCreator). | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | [withEAD]: whether to include EAD as descriptive metadata and create a datastream in the fedora objects; defaults to false</ | ||
- | |||
- | ==== PIDAssigner ==== | ||
- | |||
- | The PIDAssigner will obtain PIDs from fedora and assign them to the nodes in the sip. \\ | ||
- | It will use the FEEDER_JAVA system variable to locate configuration files and | ||
- | the workbench defined in the docuteamFeeder.properties file. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | |||
- | ==== PIDListPublisher ==== | ||
- | |||
- | The PIDListPublisher saves/sends the file ' | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | receiverURL: | ||
- | |||
- | ==== RenameSIPasAIPforIaas ==== | ||
- | |||
- | The operation RenameSIPasAIPforIaas renames an SIP using the pid of the root element of the mets file as prefix. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | [targetFolder]: | ||
- | |||
- | ==== UpdateExcelWithPID ==== | ||
- | |||
- | This class will write PIDs from an SIP's nodes into excel sheet(s). The excel | ||
- | sheet(s) must have a column with a label of either ' | ||
- | the first row. PIDs will be written into the column with the header string | ||
- | ' | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [path/ | ||
- | path/ | ||
- | |||
- | ===== Submission ===== | ||
- | |||
- | ==== AgreementsOverviewGenerator ==== | ||
- | |||
- | Creates a simple overview of submission agreements located in a given folder. | ||
- | This is done by XSL transformations, | ||
- | submission agreement files in the given directory and lists them in a simple | ||
- | xml structure: | ||
- | < | ||
- | < | ||
- | <sa_1 /> \\ | ||
- | <sa_2 /> \\ | ||
- | ... \\ | ||
- | <sa_x /> \\ | ||
- | </ | ||
- | |||
- | < | ||
- | Parameters: | ||
- | agreements_directory: | ||
- | type: one of { Hierarchy | Flat | CSV }, defining the structure of the resulting overview file | ||
- | output_directory: | ||
- | |||
- | ==== CheckFolder ==== | ||
- | |||
- | Check if sip size and size of each file in folder and file paths within the SIP exceeds the maximal allowed provided value. \\ | ||
- | Any file or folder exceeding the maximal allowed size will be logged. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | [/ | ||
- | maxTotalSize: | ||
- | maxSingleFileSize: | ||
- | maxFilePathLength: | ||
- | |||
- | ==== CreateSIPsFromFileOrFolder ==== | ||
- | |||
- | The CreateSIPsFromFileOrFolder operation will create SIPs from a given file | ||
- | or folder. If the source is a folder, a parameter will define whether a | ||
- | single SIP or separate SIPs for each child should be created. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | source: file or folder for which an SIP should be generated | ||
- | split: if ' | ||
- | saID: value to use for referencing a submission agreement in the SIP | ||
- | dssID: value to use for referencing a data submission session of the respective submission agreement | ||
- | author: value to use as the creator for the SIP | ||
- | [dropbox]: optional location where to put the SIPs; if omitted the property ' | ||
- | |||
- | ==== CreateSIPsFromFolderContent (DEPRECATED!) ==== | ||
- | |||
- | The CreateSIPsFromFolderContent is deprecated and superseded by the CreateSIPsFromFileOrFolder which offers enhanced functionality. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | source: file or folder for which an SIP should be generated | ||
- | saID: value to use for referencing a submission agreement in the SIP | ||
- | dssID: value to use for referencing a data submission session of the respective submission agreement | ||
- | author: value to use as the creator for the SIP | ||
- | singleSIPs: | ||
- | dropbox: optional location where to put the SIPs</ | ||
- | |||
- | ==== SubmitSIPsFromFolder ==== | ||
- | |||
- | The SubmitSIPsFromFolder will use the given arguments for selecting SIPs in a | ||
- | folder and submitting them to a number of workflows using the feeder | ||
- | REST-interface. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | dropbox: path to the folder containing the SIPs | ||
- | errorbox: path to the folder where to put unsuccessful SIPs | ||
- | filter: regex filter string for the SIPs within the dropbox; put the regex expression into quotation marks! | ||
- | feeder_url: | ||
- | workflows: comma separated list of workflows to execute on each SIP | ||
- | user: username for feeder | ||
- | password: password for feeder | ||
- | useAbsolutePaths: | ||
- | checkEmptyQueue: | ||
- | [maxNumberSIPs] (optional): maximum number of SIPs to send to feeder; if omitted, all SIPs matching the filter string will be submitted</ | ||
- | |||
- | ==== WebjaxeAgreementCollector ==== | ||
- | |||
- | The WebjaxeAgreementCollector will look for the any submission agreements created within the | ||
- | webjaxe editor and copy them to a given directory. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | target_directory: | ||
- | [webjaxe_home]: | ||
- | |||
- | ===== Util ===== | ||
- | |||
- | ==== MailSender ==== | ||
- | |||
- | This MailSender operation sends an email to the given recipient with optional attachments. | ||
- | |||
- | < | ||
- | Parameters: | ||
- | receiver: the receiver' | ||
- | subject: the mail subject | ||
- | text: the message text | ||
- | attachments: | ||
- | |||
docuteam/feeder_steps.1429537278.txt.gz · Zuletzt geändert: 2019/01/07 11:12 (Externe Bearbeitung)