Dynamic rules for routing Galaxy entities to destinations

TotalPerspectiveVortex (TPV) provides an installable set of dynamic rules for the Galaxy application that can route entities (Tools, Users, Roles) to appropriate destinations based on a configurable yaml file. The aim of TPV is to build on and unify previous efforts, such as Dynamic Tool Destinations, the Job Router and Sorting Hat, into a configurable set of rules that that can be extended arbitrarily with custom Python logic.

How it works

TPV provides a dynamic rule that can be plugged into Galaxy via job_conf.yml. The dynamic rule will also have an associated configuration file, that maps entities (tools, users, roles) to specific destination through a flexible tagging system. Destinations can have arbitrary scheduling tags defined, and each entity can express a preference or aversion to specific scheduling tags. Based on this tagging, jobs are routed to the most appropriate destination. In addition, admins can also plugin arbitrary python based rules for making more complex decisions, as well as custom ranking functions for choosing between matching destinations. For example, a ranking function could query influx metrics to determine the least loaded destination, and route jobs there, providing a basic form of “metascheduling” functionality.

Shared database

A shared database of TPV rules are maintained in: https://github.com/galaxyproject/tpv-shared-database/ These rules are based on typical settings used in the usegalaxy.* federation, which you can override based on local resource availability.

Getting Started

1. pip install total-perspective-vortex into Galaxy’s python virtual environment

2. Configure Galaxy to use TPV’s dynamic destination rule

3. Create the TPV job mapping yaml file, indicating job routing preferences

4. Submit jobs as usual

Standalone Installation

If you wish to install TPV outside of Galaxy’s virtualenv (e.g. to use the tpv lint command locally or in a CI/CD pipeline), use the [cli] pip requirement specifier to make sure the necessary Galaxy dependency packages are also installed. This should not be used in the Galaxy virtualenv:

$ pip install 'total-perspective-vortex[cli]'

Contents:

• TPV by example
  • Simple configuration
  • Default inheritance
  • Explicit inheritance
  • Scheduling tags
  • Rules
  • User and Role Handling
  • Metascheduling
  • Custom contexts
  • Multiple matches
  • Job Environment
  • Job Resubmission
  • Using the shared database
  • Clamping resources
  • Giving a parameterized, custom name to a destination
• Concepts and Organisation
  • Object types
  • Operations
  • Job Dispatch Process
  • Expressions
  • Scheduling
• Configuring Galaxy
  • Simple configuration
  • Combining multiple remote and local configs
• Shell Commands
  • lint
  • dry-run
• Migration Guide
  • Migrating from v1.x to v2.x

Indices and tables

• Index

• Module Index

• Search Page