Skip to main content
POST
/
v2
/
scorers
Create
curl --request POST \
  --url https://api.galileo.ai/v2/scorers \
  --header 'Content-Type: application/json' \
  --header 'Galileo-API-Key: <api-key>' \
  --data '{
  "name": "<string>",
  "description": "",
  "tags": [
    "<string>"
  ],
  "defaults": {
    "model_name": "<string>",
    "num_judges": 123,
    "filters": [
      {
        "name": "node_name",
        "filter_type": "string",
        "value": "<string>",
        "operator": "eq",
        "case_sensitive": true
      }
    ],
    "scoreable_node_types": [
      "<string>"
    ],
    "cot_enabled": true,
    "output_type": "boolean",
    "input_type": "basic"
  },
  "scorer_type": "llm",
  "model_type": "slm",
  "ground_truth": true,
  "default_version_id": "<string>",
  "user_prompt": "<string>",
  "scoreable_node_types": [
    "<string>"
  ],
  "output_type": "boolean",
  "input_type": "basic"
}'
{
  "id": "<string>",
  "name": "<string>",
  "scorer_type": "llm",
  "defaults": {
    "model_name": "<string>",
    "num_judges": 123,
    "filters": [
      {
        "name": "node_name",
        "filter_type": "string",
        "value": "<string>",
        "operator": "eq",
        "case_sensitive": true
      }
    ],
    "scoreable_node_types": [
      "<string>"
    ],
    "cot_enabled": true,
    "output_type": "boolean",
    "input_type": "basic"
  },
  "latest_version": {
    "id": "<string>",
    "version": 123,
    "scorer_id": "<string>",
    "generated_scorer": {
      "id": "<string>",
      "name": "<string>",
      "instructions": "<string>",
      "chain_poll_template": {
        "metric_system_prompt": "<string>",
        "metric_description": "<string>",
        "value_field_name": "<string>",
        "explanation_field_name": "<string>",
        "template": "<string>",
        "metric_few_shot_examples": [
          {
            "generation_prompt_and_response": "<any>",
            "evaluating_response": "<any>"
          }
        ],
        "response_schema": {}
      },
      "user_prompt": "<string>"
    },
    "registered_scorer": {
      "id": "<string>",
      "name": "<string>"
    },
    "finetuned_scorer": {
      "id": "<string>",
      "name": "<string>",
      "lora_task_id": 123,
      "prompt": "<string>",
      "luna_input_type": "span",
      "luna_output_type": "float",
      "class_name_to_vocab_ix": {},
      "executor": "action_completion_luna"
    },
    "model_name": "<string>",
    "num_judges": 123,
    "scoreable_node_types": [
      "<string>"
    ],
    "cot_enabled": true,
    "output_type": "boolean",
    "input_type": "basic"
  },
  "model_type": "slm",
  "ground_truth": true,
  "default_version_id": "<string>",
  "default_version": {
    "id": "<string>",
    "version": 123,
    "scorer_id": "<string>",
    "generated_scorer": {
      "id": "<string>",
      "name": "<string>",
      "instructions": "<string>",
      "chain_poll_template": {
        "metric_system_prompt": "<string>",
        "metric_description": "<string>",
        "value_field_name": "<string>",
        "explanation_field_name": "<string>",
        "template": "<string>",
        "metric_few_shot_examples": [
          {
            "generation_prompt_and_response": "<any>",
            "evaluating_response": "<any>"
          }
        ],
        "response_schema": {}
      },
      "user_prompt": "<string>"
    },
    "registered_scorer": {
      "id": "<string>",
      "name": "<string>"
    },
    "finetuned_scorer": {
      "id": "<string>",
      "name": "<string>",
      "lora_task_id": 123,
      "prompt": "<string>",
      "luna_input_type": "span",
      "luna_output_type": "float",
      "class_name_to_vocab_ix": {},
      "executor": "action_completion_luna"
    },
    "model_name": "<string>",
    "num_judges": 123,
    "scoreable_node_types": [
      "<string>"
    ],
    "cot_enabled": true,
    "output_type": "boolean",
    "input_type": "basic"
  },
  "user_prompt": "<string>",
  "scoreable_node_types": [
    "<string>"
  ],
  "output_type": "boolean",
  "input_type": "basic",
  "label": "<string>",
  "tags": [
    "<string>"
  ],
  "included_fields": [
    "<string>"
  ],
  "description": "<string>",
  "created_by": "<string>",
  "created_at": "2023-11-07T05:31:56Z",
  "updated_at": "2023-11-07T05:31:56Z"
}

Authorizations

​
Galileo-API-Key
string
header
required

Body

application/json
​
name
string
required
​
scorer_type
enum<string>
required
Available options:
llm,
code,
luna,
preset
​
description
string
default:""
​
tags
string[]
​
defaults
object | null
​
model_type
enum<string> | null
Available options:
slm,
llm,
code
​
ground_truth
boolean | null
​
default_version_id
string<uuid4> | null
​
user_prompt
string | null
​
scoreable_node_types
string[] | null
​
output_type
enum<string> | null

Enumeration of output types.

Available options:
boolean,
categorical,
count,
discrete,
freeform,
percentage,
multilabel
​
input_type
enum<string> | null

Enumeration of input types.

Available options:
basic,
llm_spans,
retriever_spans,
sessions_normalized,
sessions_trace_io_only,
tool_spans,
trace_input_only,
trace_io_only,
trace_normalized,
trace_output_only,
agent_spans,
workflow_spans

Response

Successful Response

​
id
string<uuid4>
required
​
name
string
required
​
scorer_type
enum<string>
required
Available options:
llm,
code,
luna,
preset
​
tags
string[]
required
​
defaults
object | null
​
latest_version
object | null

Scorer version from the scorer_versions table.

​
model_type
enum<string> | null
Available options:
slm,
llm,
code
​
ground_truth
boolean | null
​
default_version_id
string<uuid4> | null
​
default_version
object | null

Scorer version from the scorer_versions table.

​
user_prompt
string | null
​
scoreable_node_types
string[] | null
​
output_type
enum<string> | null

Enumeration of output types.

Available options:
boolean,
categorical,
count,
discrete,
freeform,
percentage,
multilabel
​
input_type
enum<string> | null

Enumeration of input types.

Available options:
basic,
llm_spans,
retriever_spans,
sessions_normalized,
sessions_trace_io_only,
tool_spans,
trace_input_only,
trace_io_only,
trace_normalized,
trace_output_only,
agent_spans,
workflow_spans
​
label
string | null
default:""
​
included_fields
string[]

Fields that can be used in the scorer to configure it. i.e. model, num_judges, etc. This enables the ui to know which fields a user can configure when they're setting a scorer

​
description
string | null
​
created_by
string<uuid4> | null
​
created_at
string<date-time> | null
​
updated_at
string<date-time> | null