scorers

Scorers

get_scorer_version

def get_scorer_version(self,
                       scorer_id: UUID,
                       version: int) -> Unset | BaseScorerVersionResponse

Arguments

• name (str): Name of the scorer
• version (int): Version of the scorer.

Returns

• Scorer response if found, otherwise None.:

list

def list(self,
         name: str | None=None,
         types: list[ScorerTypes] | None=None) -> list[ScorerResponse]

Lists scorers, optionally filtering by name and/or type.

The filters are combined with an AND condition, while the values within the types list are combined with an OR condition.

For example, calling list(name="my_scorer", types=[ScorerTypes.llm, ScorerTypes.code]) will return scorers where the name is “my_scorer” AND the type is either “llm” OR “code”.

Arguments

• name: Name of the scorer to filter by.
• types: List of scorer types to filter by. Defaults to all scorers.

Returns

• A list of scorers that match the filter criteria.:

list_by_ids

def list_by_ids(self,
                scorer_ids: builtins.list[str]) -> builtins.list[ScorerResponse]

List scorers by their UUIDs.

Arguments

• scorer_ids: UUID strings of scorers to retrieve.

Returns

• A list of scorers matching the given IDs.:

list_by_labels

def list_by_labels(self,
                   labels: builtins.list[str],
                   strict: bool=False) -> builtins.list[ScorerResponse]

List scorers by label.

Arguments

• labels: Label values to search for.
• strict: When False (default), also matches by scorer name as a fallback for custom scorers that have no label set.

Returns

• A list of scorers matching the label (or name) criteria.:

ScorerSettings

create

def create(self,
           project_id: str,
           run_id: str,
           scorers: list[ScorerConfig]) -> HTTPValidationError | RunScorerSettingsResponse | None

Arguments

• project_id: ID of the project
• run_id: ID of the run
• scorers: List of scorer configurations

Returns

• Upserted scorer settings.: