Skip to content

Plugins

This page describes how to use and write plugins for checker pipelines.

You can refer to the course-template repository for examples of plugins usage and custom plugins development.

What is the Plugin

  tasks_pipeline:
    - name: "Check forbidden regexps"
      fail: fast  # fast, after_all, never
      run: "check_regexps"
      args:
        origin: "${{ global.temp_dir }}/${{ task.task_sub_path }}"
        patterns: ["**/*.py"]
        regexps: ["exit(0)"]

    - name: "Run linter"
      run_if: ${{ parameters.run_linting }}
      fail: after_all  # fast, after_all, never
      run: "run_script"
      args:
        origin: ${{ global.temp_dir }}
        script: "python -m ruff --config=pyproject.toml ${{ task.task_sub_path }}"

Plugin is a single stage of the pipeline, have arguments, return exclusion result. In a nutshell, it is a Python class overriding abstract class checker.plugins.PluginABC:

Bases: ABC

Abstract base class for plugins. :ivar name: str plugin name, searchable by this name

Source code in checker/plugins/base.py 
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
class PluginABC(ABC):
    """Abstract base class for plugins.
    :ivar name: str plugin name, searchable by this name
    """

    name: str

    class Args(BaseModel):
        """Base class for plugin arguments.
        You have to subclass this class in your plugin.
        """

        pass

    def run(self, args: dict[str, Any], *, verbose: bool = False) -> PluginOutput:
        """Run the plugin.
        :param args: dict plugin arguments to pass to subclass Args
        :param verbose: if True should print teachers debug info, if False student mode
        :raises BadConfig: if plugin arguments are invalid
        :raises ExecutionFailedError: if plugin failed
        :return: PluginOutput with stdout/stderr and percentage
        """
        args_obj = self.Args(**args)

        return self._run(args_obj, verbose=verbose)

    @classmethod
    def validate(cls, args: dict[str, Any]) -> None:
        """Validate the plugin arguments.
        :param args: dict plugin arguments to pass to subclass Args
        :raises BadConfig: if plugin arguments are invalid
        :raises BadStructure: if _run method is not implemented
        """
        try:
            cls.Args(**args)
        except ValidationError as e:
            raise BadConfig(f"Plugin {cls.name} arguments validation error:\n{e}")

        if not hasattr(cls, "_run"):
            raise BadStructure(f"Plugin {cls.name} does not implement _run method")

    @abstractmethod
    def _run(self, args: Args, *, verbose: bool = False) -> PluginOutput:
        """Actual run the plugin.
        You have to implement this method in your plugin.
        In case of failure, raise ExecutionFailedError with an error message and output.
        :param args: plugin arguments, see Args subclass
        :param verbose: if True should print teachers debug info, if False student mode
        :return: PluginOutput with stdout/stderr and percentage
        :raises ExecutionFailedError: if plugin failed
        """
        pass

Args

Bases: BaseModel

Base class for plugin arguments. You have to subclass this class in your plugin.

Source code in checker/plugins/base.py 
30
31
32
33
34
35
class Args(BaseModel):
    """Base class for plugin arguments.
    You have to subclass this class in your plugin.
    """

    pass

run(args, *, verbose=False)

Run the plugin. :param args: dict plugin arguments to pass to subclass Args :param verbose: if True should print teachers debug info, if False student mode :raises BadConfig: if plugin arguments are invalid :raises ExecutionFailedError: if plugin failed :return: PluginOutput with stdout/stderr and percentage

Source code in checker/plugins/base.py 
37
38
39
40
41
42
43
44
45
46
47
def run(self, args: dict[str, Any], *, verbose: bool = False) -> PluginOutput:
    """Run the plugin.
    :param args: dict plugin arguments to pass to subclass Args
    :param verbose: if True should print teachers debug info, if False student mode
    :raises BadConfig: if plugin arguments are invalid
    :raises ExecutionFailedError: if plugin failed
    :return: PluginOutput with stdout/stderr and percentage
    """
    args_obj = self.Args(**args)

    return self._run(args_obj, verbose=verbose)

validate(args) classmethod

Validate the plugin arguments. :param args: dict plugin arguments to pass to subclass Args :raises BadConfig: if plugin arguments are invalid :raises BadStructure: if _run method is not implemented

Source code in checker/plugins/base.py 
49
50
51
52
53
54
55
56
57
58
59
60
61
62
@classmethod
def validate(cls, args: dict[str, Any]) -> None:
    """Validate the plugin arguments.
    :param args: dict plugin arguments to pass to subclass Args
    :raises BadConfig: if plugin arguments are invalid
    :raises BadStructure: if _run method is not implemented
    """
    try:
        cls.Args(**args)
    except ValidationError as e:
        raise BadConfig(f"Plugin {cls.name} arguments validation error:\n{e}")

    if not hasattr(cls, "_run"):
        raise BadStructure(f"Plugin {cls.name} does not implement _run method")

Note that each plugin should override checker.plugins.PluginABC.Args class to provide arguments validation. Otherwise, empty arguments will be passed to run method.

Bases: BaseModel

Base class for plugin arguments. You have to subclass this class in your plugin.

Source code in checker/plugins/base.py 
30
31
32
33
34
35
class Args(BaseModel):
    """Base class for plugin arguments.
    You have to subclass this class in your plugin.
    """

    pass

Each plugin output checker.plugins.PluginOutput class when executed successfully.

Plugin output dataclass. :ivar output: str plugin output :ivar percentage: float plugin percentage

Source code in checker/plugins/base.py 
12
13
14
15
16
17
18
19
20
@dataclass
class PluginOutput:
    """Plugin output dataclass.
    :ivar output: str plugin output
    :ivar percentage: float plugin percentage
    """

    output: str
    percentage: float = 1.0

In case of error, checker.exceptions.PluginExecutionFailed have to be raised.

Bases: TestingError

Exception raised when plugin execution failed

Source code in checker/exceptions.py 
43
44
45
46
47
48
49
@dataclass
class PluginExecutionFailed(TestingError):
    """Exception raised when plugin execution failed"""

    message: str = ""
    output: str | None = None
    percentage: float = 0.0

Note

Base Plugin class will handle all ValidationErrors of Args and raise error by itself.
So try to move all arguments validation to Args class in pydantic way.

How to use plugins

Plugins are used in the pipelines described in .checker.yml file. When running a pipeline the checker will validate plugin arguments and run it.

The following plugins are available out of the box, here is the list with their arguments:

  • run_script - execute any script with given arguments

    Bases: Args

    Source code in checker/plugins/scripts.py 
    14
15
16
17
18
    class Args(PluginABC.Args):
    origin: str
    script: Union[str, list[str]]  # as pydantic does not support | in older python versions
    timeout: Union[float, None] = None  # as pydantic does not support | in older python versions
    env_whitelist: Optional[list[str]] = None

  • safe_run_script - execute script withing firejail sandbox

    Bases: Args

    Source code in checker/plugins/firejail.py 
    24
25
26
27
28
29
30
31
32
    class Args(PluginABC.Args):
    origin: str
    script: Union[str, list[str]]  # as pydantic does not support | in older python versions
    timeout: Union[float, None] = None  # as pydantic does not support | in older python versions

    env_whitelist: list[str] = list()
    paths_whitelist: list[str] = list()
    lock_network: bool = True
    allow_fallback: bool = False

  • check_regexps - error if given regexps are found in the files

    Bases: Args

    Source code in checker/plugins/regex.py 
    12
13
14
15
    class Args(PluginABC.Args):
    origin: str
    patterns: list[str]
    regexps: list[str]

  • aggregate - aggregate results of other plugins (e.g. sum/mean/mul scores)

    Bases: Args

    Source code in checker/plugins/aggregate.py 
    14
15
16
17
    class Args(PluginABC.Args):
    scores: list[float]
    weights: Union[list[float], None] = None  # as pydantic does not support | in older python versions
    strategy: Literal["mean", "sum", "min", "max", "product"] = "mean"

  • report_score_manytask - report score to manytask

    Bases: Args

    Source code in checker/plugins/manytask.py 
    26
27
28
29
30
31
32
33
34
35
    class Args(PluginABC.Args):
    origin: Optional[str] = None  # as pydantic does not support | in older python versions
    patterns: list[str] = ["*"]
    username: str
    task_name: str
    score: float  # TODO: validate score is in [0, 1] (bonus score is higher than 1)
    report_url: AnyUrl
    report_token: str
    check_deadline: bool
    send_time: datetime = datetime.now().astimezone()

  • check_gitlab_merge_request - [WIP] check gitlab MR is valid (no conflicts, no extra files, has label etc.)

    Bases: Args

    Source code in checker/plugins/gitlab.py 
    13
14
15
16
17
18
    class Args(PluginABC.Args):
    token: str
    task_dir: str
    repo_url: AnyUrl
    requre_approval: bool = False
    search_for_score: bool = False

  • collect_score_gitlab_merge_request - [WIP] search for score by tutor in gitlab MR comment

    Bases: Args

    Source code in checker/plugins/gitlab.py 
    34
35
36
37
38
39
    class Args(PluginABC.Args):
    token: str
    task_dir: str
    repo_url: AnyUrl
    requre_approval: bool = False
    search_for_score: bool = False

How to write a custom plugin

To write a custom plugin you need to create a class inheriting from checker.plugins.PluginABC and override _run method, Args inner class and set name class attribute.

from random import randint
from checker.plugins import PluginABC, PluginOutput
from checker.exceptions import PluginExecutionFailed
from pydantic import AnyUrl

class PrintUrlPlugin(PluginABC):
    """Plugin to print url"""

    name = "print_url"

    class Args(PluginABC.Args):
        url: AnyUrl

    def _run(self, args: Args, *, verbose: bool = False) -> PluginOutput:
        if randint(0, 1):
            if verbose:
                raise PluginExecutionFailed("Verbose error, we got randint=1")
            else:
                raise PluginExecutionFailed("Random error")

        return PluginOutput(
            output=f"Url is {args.url}",
            percentage=1.0,  # optional, default 1.0 on success
        )

Important

The Plugin must implement verbose functionality!
If verbose is True the plugin should provide all info and possible debug info.
If verbose is False the plugin should provide only public-friendly info, e.g. excluding private test output.

Note

It is a nice practice to write a small tests for your custom plugins to be sure that it works as expected.