Skip to main content

DaskTask

This class transforms local Python code for execution within a Dask execution context. It configures Dask jobs with specified scheduler and worker resources, enabling distributed computation. The class also integrates with Flyte's cluster execution environment by registering code bundles with the Dask client.

Attributes​

  • plugin_config: Dask

    • The Dask plugin configuration.

  • task_type: string = dask

    • The type of the task, defaulting to 'dask'.

Methods​

def pre(args: Any, kwargs: Any) - > Dict[str, Any]

  • This method is called before the task execution. It sets up the Dask client and registers a plugin for uploading the code bundle if the task is running in a cluster environment with a code bundle available. It returns an empty dictionary.

  • Parameters

    • args: Any
      • Positional arguments.
    • kwargs: Any
      • Keyword arguments.

  • Return Value: Dict[str, Any]

    • An empty dictionary.
def custom_config(sctx: [SerializationContext](src_flyte_models_serializationcontext)) - > Dict[str, Any]

  • This method generates a DaskJob configuration based on the plugin's configuration. It extracts scheduler and worker details, creates DaskScheduler and DaskWorkerGroup objects, and then constructs a DaskJob. The job configuration is returned as a dictionary.

  • Parameters

  • Return Value: Dict[str, Any]

    • A dictionary representing the DaskJob configuration.