Skip to main content
POST
/
v2
/
scorers
Create
curl --request POST \
  --url https://api.galileo.ai/v2/scorers \
  --header 'Content-Type: application/json' \
  --header 'Galileo-API-Key: <api-key>' \
  --data '
{
  "name": "<string>",
  "scorer_type": "llm",
  "description": "",
  "tags": [
    "<string>"
  ],
  "defaults": {
    "model_name": "<string>",
    "num_judges": 123,
    "filters": [
      {
        "operator": "eq",
        "value": "<string>",
        "name": "node_name",
        "case_sensitive": true
      }
    ],
    "scoreable_node_types": [
      "<string>"
    ],
    "cot_enabled": true,
    "output_type": "boolean",
    "input_type": "basic"
  },
  "deprecated": true,
  "model_type": "slm",
  "ground_truth": true,
  "default_version_id": "<string>",
  "user_prompt": "<string>",
  "scoreable_node_types": [
    "<string>"
  ],
  "output_type": "boolean",
  "input_type": "basic",
  "multimodal_capabilities": [
    "vision"
  ],
  "required_scorers": [
    "<string>"
  ],
  "roll_up_method": "average",
  "metric_color_picker_config": {
    "constraints": [
      {
        "color": "red",
        "operator": "eq",
        "value": 123
      }
    ],
    "type": "numeric"
  }
}
'
{
  "id": "<string>",
  "name": "<string>",
  "scorer_type": "llm",
  "tags": [
    "<string>"
  ],
  "defaults": {
    "model_name": "<string>",
    "num_judges": 123,
    "filters": [
      {
        "operator": "eq",
        "value": "<string>",
        "name": "node_name",
        "case_sensitive": true
      }
    ],
    "scoreable_node_types": [
      "<string>"
    ],
    "cot_enabled": true,
    "output_type": "boolean",
    "input_type": "basic"
  },
  "latest_version": {
    "id": "<string>",
    "version": 123,
    "scorer_id": "<string>",
    "generated_scorer": {
      "id": "<string>",
      "name": "<string>",
      "chain_poll_template": {
        "template": "<string>",
        "metric_system_prompt": "<string>",
        "metric_description": "<string>",
        "value_field_name": "rating",
        "explanation_field_name": "explanation",
        "metric_few_shot_examples": [
          {
            "generation_prompt_and_response": "<string>",
            "evaluating_response": "<string>"
          }
        ],
        "response_schema": {}
      },
      "instructions": "<string>",
      "user_prompt": "<string>"
    },
    "registered_scorer": {
      "id": "<string>",
      "name": "<string>",
      "score_type": "<string>"
    },
    "finetuned_scorer": {
      "id": "<string>",
      "name": "<string>",
      "lora_task_id": 123,
      "prompt": "<string>",
      "lora_weights_path": "<string>",
      "luna_input_type": "span",
      "luna_output_type": "float",
      "class_name_to_vocab_ix": {},
      "executor": "action_completion_luna"
    },
    "model_name": "<string>",
    "num_judges": 123,
    "scoreable_node_types": [
      "<string>"
    ],
    "cot_enabled": true,
    "output_type": "boolean",
    "input_type": "basic"
  },
  "model_type": "slm",
  "ground_truth": true,
  "default_version_id": "<string>",
  "default_version": {
    "id": "<string>",
    "version": 123,
    "scorer_id": "<string>",
    "generated_scorer": {
      "id": "<string>",
      "name": "<string>",
      "chain_poll_template": {
        "template": "<string>",
        "metric_system_prompt": "<string>",
        "metric_description": "<string>",
        "value_field_name": "rating",
        "explanation_field_name": "explanation",
        "metric_few_shot_examples": [
          {
            "generation_prompt_and_response": "<string>",
            "evaluating_response": "<string>"
          }
        ],
        "response_schema": {}
      },
      "instructions": "<string>",
      "user_prompt": "<string>"
    },
    "registered_scorer": {
      "id": "<string>",
      "name": "<string>",
      "score_type": "<string>"
    },
    "finetuned_scorer": {
      "id": "<string>",
      "name": "<string>",
      "lora_task_id": 123,
      "prompt": "<string>",
      "lora_weights_path": "<string>",
      "luna_input_type": "span",
      "luna_output_type": "float",
      "class_name_to_vocab_ix": {},
      "executor": "action_completion_luna"
    },
    "model_name": "<string>",
    "num_judges": 123,
    "scoreable_node_types": [
      "<string>"
    ],
    "cot_enabled": true,
    "output_type": "boolean",
    "input_type": "basic"
  },
  "user_prompt": "<string>",
  "scoreable_node_types": [
    "<string>"
  ],
  "output_type": "boolean",
  "input_type": "basic",
  "multimodal_capabilities": [
    "vision"
  ],
  "required_scorers": [
    "<string>"
  ],
  "deprecated": true,
  "roll_up_config": {
    "roll_up_methods": "<unknown>"
  },
  "label": "",
  "included_fields": [
    "<string>"
  ],
  "description": "<string>",
  "created_by": "<string>",
  "created_at": "2023-11-07T05:31:56Z",
  "updated_at": "2023-11-07T05:31:56Z",
  "roll_up_method": "average",
  "metric_color_picker_config": {
    "constraints": [
      {
        "color": "red",
        "operator": "eq",
        "value": 123
      }
    ],
    "type": "numeric"
  }
}

Authorizations

​
Galileo-API-Key
string
header
required

Body

application/json
​
name
string
required
​
scorer_type
enum<string>
required
Available options:
llm,
code,
luna,
preset
​
description
string
default:""
​
tags
string[]
​
defaults
ScorerDefaults · object
​
deprecated
boolean | null
​
model_type
enum<string> | null
Available options:
slm,
llm,
code
​
ground_truth
boolean | null
​
default_version_id
string<uuid4> | null
​
user_prompt
string | null
​
scoreable_node_types
string[] | null
​
output_type
enum<string> | null

Enumeration of output types.

Available options:
boolean,
categorical,
count,
discrete,
freeform,
percentage,
multilabel,
retrieved_chunk_list_boolean,
boolean_multilabel
​
input_type
enum<string> | null

Enumeration of input types.

Available options:
basic,
llm_spans,
retriever_spans,
sessions_normalized,
sessions_trace_io_only,
tool_spans,
trace_input_only,
trace_io_only,
trace_normalized,
trace_output_only,
agent_spans,
workflow_spans
​
multimodal_capabilities
enum<string>[] | null
Available options:
vision,
audio
​
required_scorers
string[] | null
​
roll_up_method
enum<string> | null

Roll up methods for aggregating numeric metrics up the session/trace/span hierarchy.

Available options:
average,
sum,
max,
min
​
metric_color_picker_config
MetricColorPickerNumeric · object

Color picker configuration for numeric metrics.

Each constraint maps a numeric condition to a color. The UI uses these constraints to color-code metric values (e.g. green for high scores, red for low scores).

Example: { "type": "numeric", "constraints": [ {"color": "green", "operator": "gte", "value": 0.8}, {"color": "yellow", "operator": "between", "value": [0.3, 0.8]}, {"color": "red", "operator": "lt", "value": 0.3} ] }

Response

Successful Response

​
id
string<uuid4>
required
​
name
string
required
​
scorer_type
enum<string>
required
Available options:
llm,
code,
luna,
preset
​
tags
string[]
required
​
defaults
ScorerDefaults · object
​
latest_version
BaseScorerVersionDB · object

Scorer version from the scorer_versions table.

​
model_type
enum<string> | null
Available options:
slm,
llm,
code
​
ground_truth
boolean | null
​
default_version_id
string<uuid4> | null
​
default_version
BaseScorerVersionDB · object

Scorer version from the scorer_versions table.

​
user_prompt
string | null
​
scoreable_node_types
string[] | null
​
output_type
enum<string> | null

Enumeration of output types.

Available options:
boolean,
categorical,
count,
discrete,
freeform,
percentage,
multilabel,
retrieved_chunk_list_boolean,
boolean_multilabel
​
input_type
enum<string> | null

Enumeration of input types.

Available options:
basic,
llm_spans,
retriever_spans,
sessions_normalized,
sessions_trace_io_only,
tool_spans,
trace_input_only,
trace_io_only,
trace_normalized,
trace_output_only,
agent_spans,
workflow_spans
​
multimodal_capabilities
enum<string>[] | null
Available options:
vision,
audio
​
required_scorers
string[] | null
​
deprecated
boolean | null
​
roll_up_config
BaseMetricRollUpConfigDB · object

Configuration for rolling up metrics to parent/trace/session.

​
label
string | null
default:""
​
included_fields
string[]

Fields that can be used in the scorer to configure it. i.e. model, num_judges, etc. This enables the ui to know which fields a user can configure when they're setting a scorer

​
description
string | null
​
created_by
string<uuid4> | null
​
created_at
string<date-time> | null
​
updated_at
string<date-time> | null
​
roll_up_method
enum<string> | null

Roll up methods for aggregating numeric metrics up the session/trace/span hierarchy.

Available options:
average,
sum,
max,
min
​
metric_color_picker_config
MetricColorPickerNumeric · object

Color picker configuration for numeric metrics.

Each constraint maps a numeric condition to a color. The UI uses these constraints to color-code metric values (e.g. green for high scores, red for low scores).

Example: { "type": "numeric", "constraints": [ {"color": "green", "operator": "gte", "value": 0.8}, {"color": "yellow", "operator": "between", "value": [0.3, 0.8]}, {"color": "red", "operator": "lt", "value": 0.3} ] }