Application description
The QCG-PilotJob system is designed to schedule and execute many small jobs inside one scheduling system allocation. Direct submission of a large group of jobs to a scheduling system can result in long aggregated time to finish as every single job is scheduled independently and waits in a queue. On the other hand, the submission of a large group of jobs can be restricted or even forbidden by administrative policies defined on clusters. One can argue that there are available job array mechanisms in many systems, however, the traditional job array mechanism allows to run only a group of jobs having the same resource requirements while many jobs within a larger workflow can vary in requirements and therefore need more flexible solutions.