Skip to content

AutoML API

This document describes each endpoint of VecML RESTful API for Automated Maching Learning (AutoML), including:

  • Endpoint URL
  • Expected Request Body (JSON)
  • Response Format (JSON)
  • Relevant Details or Constraints
  • Example Requests / Responses

All RESTful API calls to VecML cloud database should be sent to: https://aidb.vecml.com/api. All endpoints require a JSON body unless otherwise noted.

Rate and request size limit

For resource allocation and server stability, the server enforces a rate limit on the number of API calls per second. If exceeded, the server responds with: 

400 Bad Request
Request limit reached, please try again later
The clustering result endpoints (/delete_clustering_model, /get_clustering_labels, /get_kmeans_centroids, /get_dbscan_core_points, /get_dbscan_kdist_plot, /compute_clustering_metrics, /compute_tsne_embeddings) instead respond with 429 Too Many Requests and a JSON body {"error": "Request limit reached, please try again later"}. Please avoid high-frequency API calls. For example, when inserting multiple vectors, please use relatively large batch size in /add_data_batch and avoid calling the API too frequently.

The size of each request cannot pass 200MB.

Field types

Send each field with the JSON type shown in this document (strings as strings, numbers as numbers, booleans as booleans). A field with the wrong JSON type (e.g. "cluster_model_name": 5) is generally answered with HTTP 400 and the plain-text body Invalid request field: <reason> (plain text on every endpoint, including the clustering result endpoints whose other errors are JSON); integer fields that are range-checked (e.g. "n_clusters": "10") report Invalid <field>! Must be an integer between <min> and <max>. instead. The exceptions are noted per endpoint.

Managing Unique IDs for vectors

Vectors are identified by UNIQUE string IDs. While VecML provide a way to maintain auto-generated string IDs internally, we highly encourage the users to maintain and specify unique IDs for the vectors, for the convenience of database operations. Multiple ways are available to specify the IDs, please check the data inserting functions for details.

Authentication (API key):

VecML RESTful API requests are authenticated through user's API key. The API key can be generated as follows:

  1. Go to https://account.vecml.com/user-api-keys , sign up a free VecML account.
  2. After registration, you can get an API key by clicking "Create New API Key".
  3. This unique API key will only be seen once at the creation time, so please keep the API key safely. If you need to re-generate an API key, simply delete the previous one and then create a new key.

Pass the key in the user_api_key field of every request body. A request without the field is rejected as a missing field (HTTP 400). If the key is invalid, the server responds with 401 Unauthorized and the message Unauthorized user. Please check your VecML API key. (a plain-text body for most endpoints; a JSON body {"error": "..."} for the clustering result endpoints listed above).

Getting Started: Project and Dataset Management

To use AutoML APIs, the user first needs to upload/insert a vector collection (dataset) to VecML cloud.

See the vector DB API documentation for Project Management and Dataset Management. Also check Job Management for managing the async job status for data insertion/upload.

An Example WorkFlow

VecML Database provides fast and performant built-in ML tools for your project and bussiness. Below is a standard workflow using VecML RESTful API for AutoML model training:

  1. Call /create_project to create a new project.

  2. Create a dataset (or collection, interchangably) within the project and upload data matrix X. The recommended way is:

    • Upload a data file (supported types: csv, json, binary format, libsvm) as a dataset using /upload_automl_X. You can iteratively upload files to a data collection.

    Note: When uploading data, the user needs to specify the categorical features that will be included when training the model.

  3. Attach the label (response) to the dataset. Use /attach_automl_label to upload the Y to the dataset. Now (X, Y) form the AutoMl training problem.

  4. Train an AutoML model for the dataset using /train_automl_model.

    Note: Before starting model training, please call /get_upload_data_status to confirm that X and Y have been uploaded successfull. Otherwise, the model training result might be incorrect.

  5. Make predictions on predict/test dataset /automl_predict.

The following is an example python code to demonstrate the API workflow.

import requests
import json
import numpy as np
import time

# Configuration
API_KEY = "replace_this_with_your_api_key"
BASE_URL = "https://aidb.vecml.com/api"

def make_request(endpoint, data):
    """Helper function to make API calls"""
    url = f"{BASE_URL}/{endpoint}"
    response = requests.post(url, json=data)
    print(f"Request to {endpoint}: HTTP {response.status_code}")

    if response.text:
        try:
            json_response = response.json()
            print(f"Response: {json_response}")
            return response.status_code, json_response
        except requests.exceptions.JSONDecodeError:
            print(f"Response: {response.text}")
            return response.status_code, {"error": "Not JSON", "message": response.text}
    else:
        print("Response: Empty")
        return response.status_code, None

def wait_for_job_completion(job_id, status_endpoint, max_wait_time=60):
    """Wait for an async job to complete"""
    start_time = time.time()

    while True:
        status_data = {"user_api_key": API_KEY, "job_id": job_id}
        status, status_response = make_request(status_endpoint, status_data)

        if status_response and status_response.get("status") == "finished":
            return True
        elif status_response and status_response.get("status") == "failed":
            return False

        if time.time() - start_time > max_wait_time:
            return False

        time.sleep(2)

def generate_dataset(num_samples, vector_dim, id_prefix, seed=2025):
    """Generate dataset with linear decision boundary"""
    np.random.seed(seed)
    vectors = np.random.randn(num_samples, vector_dim).tolist()
    categories = [np.random.choice(['A', 'B', 'C']) for _ in range(num_samples)]

    labels = []
    for vec, category in zip(vectors, categories):
        # Linear combination of first few components plus category weight
        score = sum(vec[:20]) + {'A': 1.0, 'B': -0.5, 'C': 0.0}[category]
        label = '1' if score > 0 else '0'
        labels.append(label)

    # Generate IDs and attributes
    ids = [f"{id_prefix}_{i:03d}" for i in range(num_samples)]
    attributes = [{"label": str(label), "category": category} for label, category in zip(labels, categories)]

    return vectors, ids, attributes

# Clean up any existing project
status, response = make_request("delete_project", {"user_api_key": API_KEY, "project_name": "AutoML-Demo"})

# 1. Create a project
project_data = {"user_api_key": API_KEY, "project_name": "AutoML-Demo", "application": "Machine Learning"}
status, response = make_request("create_project", project_data)

# 2. Initialize training dataset
init_data = {"user_api_key": API_KEY, "project_name": "AutoML-Demo", "collection_name": "training_data",
             "vector_type": "dense", "vector_dim": 64}
status, response = make_request("init", init_data)

# 3. Generate and add training data using add_data_batch
vectors, ids, attributes = generate_dataset(num_samples=1000, vector_dim=64, id_prefix="train", seed=2025)

# Add training data in batch
batch_data = {"user_api_key": API_KEY, "project_name": "AutoML-Demo", "collection_name": "training_data",
              "string_ids": ids, "data": vectors, "attributes": attributes}
status, response = make_request("add_data_batch", batch_data)
train_upload_job_id = response["job_id"]

# Wait for training data upload to complete
if not wait_for_job_completion(train_upload_job_id, "get_upload_data_status", max_wait_time=30):
    exit(1)

# 4. Train AutoML model
train_data = {"user_api_key": API_KEY, "project_name": "AutoML-Demo", "collection_name": "training_data",
              "model_name": "model1", "training_mode": "high_speed", "task_type": "classification",
              "label_attribute": "label", "train_categorical_features": ["category"]}
status, response = make_request("train_automl_model", train_data)
train_job_id = response["job_id"]

# Wait for training to complete
if not wait_for_job_completion(train_job_id, "get_automl_training_status", max_wait_time=60):
    exit(1)

# 5. Initialize prediction dataset
pred_init_data = {"user_api_key": API_KEY, "project_name": "AutoML-Demo", "collection_name": "prediction_data",
                  "vector_type": "dense", "vector_dim": 64}
status, response = make_request("init", pred_init_data)

# 6. Generate and add prediction data
prediction_vectors, prediction_ids, prediction_attributes = generate_dataset(num_samples=100, vector_dim=64, id_prefix="pred", seed=2026)

# Add prediction data in batch
pred_batch_data = {"user_api_key": API_KEY, "project_name": "AutoML-Demo", "collection_name": "prediction_data",
                   "string_ids": prediction_ids, "data": prediction_vectors, "attributes": prediction_attributes}
status, response = make_request("add_data_batch", pred_batch_data)
pred_upload_job_id = response["job_id"]

# Wait for prediction data upload to complete
if not wait_for_job_completion(pred_upload_job_id, "get_upload_data_status"):
    exit(1)

# 7. Make predictions using the existing dataset
predict_data = {"user_api_key": API_KEY, "project_name": "AutoML-Demo", "collection_name": "training_data",
                "model_name": "model1", "prediction_dataset": "prediction_data"}
status, prediction_results = make_request("automl_predict", predict_data)

Upload AutoML Datasets

/upload_automl_X

Description: Uploads a file containing dataset vectors (CSV, JSON, libsvm or binary format) into a dataset. If the dataset does not exist it is created; if it exists, the vectors are appended (the vector type and, except for sparse data, the dimension must match the existing data). The upload runs asynchronously; you receive a job_id to query via /get_upload_data_status.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",   // The dataset's name
  "X": "string",   // base64-encoded data (optionally gzip compressed)
  "file_format": "string", // "csv", "json", "libsvm", "binary"
  "has_field_names": true | false,   // required for every file format
  "vector_type": "dense" | "dense1Bit" | "dense2Bit" | "dense4Bit" | "dense8Bit" | "sparse",

  // If JSON with field names:
  "vector_data_field_name": "string",

  // If CSV or JSON with field names (optional):
  "vector_id_field_name": "string",

  // If CSV or JSON (optional):
  "categorical_features": ["attr1", "attr2", ...],

  // If "binary" file_format:
  "binary_dtype": "uint8" | "float32",
  "binary_dim": 123,

  // Optional compression method
  "compression_type": "string",    // now only "gzip" is supported

  // Optional checksum for data integrity check
  "checksum": "string",           // SHA256 hash of the "X" string

  // Optional custom string IDs
  "string_id_txt": "string"        // base64-encoded text file, one ID per line
}

Required:

  • user_api_key: Your VecML API key for authentication.
  • project_name: The project that the uploaded dataset belongs to. The project must already exist.
  • collection_name: The name of the data collection in VecML DB (at most 128 characters; the characters \ / : * ? " ' < > | and control characters are not allowed, and names that consist only of dots, such as . and .., and the reserved names project_infos.txt and dataset_infos.txt are rejected).
  • X: The Base64 encoded raw data matrix (set of vectors) for AutoML model training.
  • file_format: "csv", "json", "libsvm" or "binary". If another value is given, the server tries to detect the format from the file content (if detection fails: 400 Unable to detect file format. Please specify the format explicitly.). "jsonl" (given or detected) is accepted by the request, but the upload job fails with File format not supported!.
  • has_field_names (boolean): whether the data file contains column headers (csv) or field names (json). It is required for every file format (for libsvm and binary files, set it to false); a request without it is rejected with 400 Missing required field: has_field_names.
  • vector_type: the type of the vector. Supported types:
    • dense: the standard float32 dense vector. For example, [0, 1.2, 2.4, -10 ,5.7]. Standard embedding vectors from language or vision models can be saved as this type.
    • dense8Bit: uint8 dense vectors, with integer vector elements ranging in [0, 255]. For example, [0, 3, 76, 255, 152]. 8-bit quantized embedding vectors can be saved as this type for storage saving.
    • dense4Bit: 4-bit quantized dense vectors, with integer vector elements ranging in [0, 15].
    • dense2Bit: 2-bit quantized dense vectors, with integer vector elements ranging in [0, 3].
    • dense1Bit: 1-bit quantized dense vectors, with binary vector elements.
    • sparse: sparse vector formatted as a set of index:value pairs. Please use this for libsvm file format. This is useful for high-dimensional sparse vectors.

Conditional:

  • For json format data with has_field_names == true:
    • vector_data_field_name (required): the json field that contains the vector data.
  • For csv and json format data with has_field_names == true:
    • vector_id_field_name (optional): the column/field that holds each vector's string ID. Without it, IDs are generated automatically.
  • For csv and json format data:
    • categorical_features (optional): [attr1, attr2, ..., attrN], the columns/fields that are stored as (categorical) attributes of the vectors instead of vector elements. These attributes can later be used as categorical features in /train_automl_model. For JSON files without field names this list is ignored. (vector_attributes is accepted as an alias; if both are given, categorical_features is used.)
  • For libsvm format data: the label of each line is stored automatically as an attribute named "label", so it can be used directly as the label_attribute for training (no need to call /attach_automl_label).
  • For binary format data (vectors concatenated as raw bytes):
    • binary_dtype (required): "uint8" or "float32". The stored vector type follows this value (uint8 → dense8Bit, float32 → dense), regardless of vector_type.
    • binary_dim (required): the vector dimension, a positive integer.
    • If binary_dtype or binary_dim is missing, the request is accepted (a job_id is returned) but the upload job fails. An invalid value (a dtype other than "uint8"/"float32", or a non-positive or non-integer dim) makes the request fail with HTTP 500.

Optional:

Auto-generation of column names (for CSV)

For csv files, if has_field_names == false, you can specify categorical feature columns by their column number (1-based), such as "categorical_features": ["column 59", "column 60"]. The digits in each name select the column, and the attribute keeps the name you gave (e.g. "column 59"). A name without digits does not select a column and is skipped; a number outside the file's columns makes the upload job fail.

  • compression_type: to further reduce the file size and speed up communication, the user can gzip the file before converting to Base64 format. If gzip is applied, set this argument to "gzip". Currently, only "gzip" is supported. When set, string_id_txt must be gzip compressed as well.

  • checksum: the SHA256 hash of the X string. If it does not match the hash computed by the server, the request is rejected with HTTP 400.

  • string_id_txt: a Base64 encoded text file of string IDs separated by whitespace (one per line is typical; IDs therefore cannot contain spaces), used as the IDs of the uploaded vectors in order. If the number of IDs differs from the number of vectors, the IDs are ignored and default IDs are used (the job's status message says so).

Notes

  • A dataset can hold at most 1,000,000 vectors. An upload that would exceed this limit fails (the job status becomes "failed").
  • If the uploaded vectors do not match an existing dataset's vector type or (for non-sparse data) dimension, the job fails and nothing is inserted.
  • For the low-bit types (dense8Bit, dense4Bit, dense2Bit, dense1Bit), a csv or json value that is not an integer in the type's range makes the job fail, e.g. Invalid value 300 for vector type dense8Bit at line 2, column 3: values must be integers in [0, 255].
  • Validation errors are returned as HTTP 400 with a plain-text message (a missing X is reported as Missing required field: file_data); an invalid API key returns HTTP 401 Unauthorized user. Please check your VecML API key.

Response 

{
  "success": true,
  "job_id": "string",
  "error_message": "none",
  "checksum_server": "string"
}

  • job_id: Use this ID to query the status of the upload job via /get_upload_data_status. Format: <user_id>||<project_name>||<collection_name>||InsertDataJob||<uuid>.

  • checksum_server: the SHA256 checksum of X computed on the server side, for data integrity check. It is always returned.

Example

POST /upload_automl_X
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "X": "BASE64_ENCODED_DATA==",
  "file_format": "csv",
  "has_field_names": true,
  "vector_type": "dense",
  "categorical_features": ["date", "region", "category"]
}
Response: 
{
  "success": true,
  "job_id": "user_123||ProjectA||MyDataset||InsertDataJob||1b4e28ba-2fa1-11d2-883f-0016d3cca427",
  "error_message": "none",
  "checksum_server": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
}

/attach_automl_label

Description: Attaches label data to an existing dataset by uploading a file containing the label values. The upload runs asynchronously; you receive a job_id to query via /get_upload_data_status.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",   // The existing dataset's name
  "Y": "string",   // base64-encoded label data file (optionally gzip compressed)
  "label_name": "string",   // The name of the label attribute to be added

  // Optional compression method
  "compression_type": "string"    // now only "gzip" is supported
}

Required:

  • user_api_key: Your VecML API key for authentication.
  • project_name: The project that contains the dataset.
  • collection_name: The name of the existing data collection in VecML DB.
  • Y: The Base64 encoded label data file, a plain text (or csv) file in one of two formats. The format and the delimiter (comma, tab or space) are detected from the first non-empty line:
    • Two columns id,label per line: each label is assigned to the vector with that string ID.
    • One column (one label per line): line i (counting from 0) gets the ID "i", i.e. "0", "1", "2", ..., and each label is assigned to the vector with that string ID. This matches the default IDs generated when a file is uploaded without an ID column; if your vectors have custom string IDs, use the two-column format.
    • Because the delimiter is detected from the first line, a one-column label that contains a comma, tab or space would be read as two columns.
    • Numeric labels are normalized (e.g. "2.0" is stored as "2", "1.50" as "1.5").
  • label_name: The name to assign to this label attribute in the dataset.

Optional:

  • compression_type: To further reduce the file size and speed up communication, the user can gzip the file before converting to Base64 format. If gzip is applied, set this argument to "gzip". Currently, only "gzip" is supported.

Response 

{
  "success": true,
  "job_id": "string",
  "error_message": "none",
  "num_vectors_parsed": 123
}

  • job_id: Use this ID to query the status of the upload job via /get_upload_data_status. Format: <user_id>||<project_name>||<collection_name>||AddAttributesFromFileJob||<uuid>.
  • num_vectors_parsed: The number of ID-label pairs successfully parsed from the uploaded file.

Notes

  • IDs in the file that do not exist in the dataset are reported in the job's error_message, as The following vectors failed to update attributes: [{"error":"Vector not found with string_id: <id>","string_id":"<id>"}, ...]; the job still finishes ("finished") and the other labels are attached.
  • Errors in the request or in the file format are returned as HTTP 400 with a plain-text message; a request missing a field is rejected with Missing user_api_key, project_name, collection_name, attribute_name, or file_data (attribute_name and file_data are the internal names of label_name and Y). An unknown project or dataset returns Project does not exist! / Dataset does not exist! (HTTP 400).
  • Non-ASCII characters in the file are converted to ASCII: accented Latin letters are transliterated ("café" becomes "cafe"), typographic quotes, dashes and … become their ASCII equivalents, and other characters become ?.

Example 

POST /attach_automl_label
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "Y": "BASE64_ENCODED_LABEL_DATA==",
  "label_name": "sentiment",
  "compression_type": "gzip"
}

Response: 

{
  "success": true,
  "job_id": "user_123||ProjectA||MyDataset||AddAttributesFromFileJob||1b4e28ba-2fa1-11d2-883f-0016d3cca427",
  "error_message": "none",
  "num_vectors_parsed": 1000
}

AutoML API Endpoints

/train_automl_model

Description: Initiates training of an AutoML model on a specified dataset. This is an asynchronous operation that returns a job ID for tracking progress.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string", 
  "model_name": "string",
  "training_mode": "string",
  "task_type": "string",
  "label_attribute": "string",
  "train_categorical_features": ["string", ...],  // optional
  "validation_dataset": "string",                 // optional
  "feature_augmentation": boolean,                // optional, default false
  "do_retrain": boolean                           // optional, default true
}

Request Fields

  • model_name: Unique name for the model (at most 128 characters; the characters \ / : * ? " ' < > | and control characters are not allowed). It must not consist only of dots (., ..), must not be one of the names reserved for the dataset's own files (see index_name in /attach_index), and must not already be used by an index or a clustering model of the dataset.
  • training_mode: Training mode configuration: "linear_model", "high_speed", "balanced", "high_accuracy". Any other value is rejected with HTTP 400 Invalid training_mode! Should be linear_model, high_speed, balanced or high_accuracy.
  • task_type: Type of ML task - either "classification" or "regression". Any other value is rejected with HTTP 400 Invalid task_type! Should be classification or regression.
  • label_attribute: Name of the attribute/column to use as the target label.
  • train_categorical_features: (Optional) Array of attribute names to treat as categorical features when training the model. Use attributes that were uploaded with the dataset (e.g. the categorical_features of /upload_automl_X).
  • validation_dataset: (Optional) Name of separate dataset to use for validation. If not set, 20% of the training dataset is held out for validation. If validation_dataset is provided, it must have the same vector type and (for non-sparse data) the same vector dimension as the training dataset, and contain every attribute listed in train_categorical_features.
  • feature_augmentation: (Optional, default false) Whether to enable feature augmentation for better performance. It takes some extra time but usually gives better performance. Accepts a boolean ("true"/"1" and non-zero integers are also accepted; numbers are truncated to integers, so 0.5 counts as false). It only takes effect when the project's application is "autoML" or "both". When it is true, the server prepares the augmented features before returning the job_id, so this request can take a while (minutes for large datasets) to respond.
  • do_retrain: (Optional, default true) After the best model is selected on the validation data, retrain it on the training and validation data combined. Accepts the same value forms as feature_augmentation.

Response Success Response (200 OK): 

{
  "success": true,
  "job_id": "string"
}

  • job_id: Format: <user_id>||<project_name>||<collection_name>||<model_name>||TrainJob.

Error responses are HTTP 400 with a plain-text message, for example:

  • Missing fields in JSON: user_api_key, project_name, or collection_name
  • Missing training configuration: model name, mode, task type (when model_name, training_mode, task_type or label_attribute is missing)
  • Model with same name already exists! / A model with same name is already training!
  • The name is already used by an index or a clustering model in this dataset!
  • Project does not exist! / Dataset does not exist! (checked before anything is written to disk)
  • The vector type of validation dataset does not match the training data. / The vector dimension of validation dataset does not match the training data. / Categorical feature <name> not found in validation data.

An invalid API key returns HTTP 401 Unauthorized user. Please check your VecML API key.

Notes

  • Model names must be unique within a dataset.
  • Training is asynchronous - use /get_automl_training_status with the returned job_id to monitor progress.
  • The training dataset must contain the specified label_attribute. For regression, samples without a (numeric) label are not included in training. For classification, samples without the label attribute are not dropped: they form an extra class whose label is the empty string "", which can then appear in predictions, class_labels and probability_matrix of /automl_predict. Make sure every training vector has a label.

Example 

POST /train_automl_model
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "MLProject",
  "collection_name": "customer_data",
  "model_name": "customer_segmentation_v1",
  "training_mode": "balanced",
  "task_type": "classification", 
  "label_attribute": "customer_segment",
  "train_categorical_features": ["region", "subscription_type"],
  "validation_dataset": "customer_validation_data"
}

Response: 

{
  "success": true,
  "job_id": "user_123||MLProject||customer_data||customer_segmentation_v1||TrainJob"
}

/automl_predict

Description: Use a trained AutoML model to make predictions on new data. Can accept data as a file upload or reference an existing dataset.

Method - POST

Request Body

Option 1: Using file upload 

{
  "user_api_key": "string",
  "project_name": "string", 
  "collection_name": "string",
  "model_name": "string",
  "file_data": "string",          // base64 encoded file
  "file_format": "string",        // "csv", "json", or "libsvm"
  "has_field_names": boolean,     // optional: defaults to the training dataset's setting
  "compression_type": "string",   // optional: "gzip"
  "return_all_probabilities": boolean  // optional, classification only (default false)
}

Option 2: Using existing dataset 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string", 
  "model_name": "string",
  "prediction_dataset": "string",
  "return_all_probabilities": boolean  // optional, classification only (default false)
}

Request Fields

  • collection_name: Name of the dataset the model was trained on
  • model_name: Name of the trained model to use for prediction
  • file_data: (Option 1) Base64 encoded file containing prediction data
  • file_format: (Option 1, required with file_data) Format of uploaded file: "csv", "json", or "libsvm". "jsonl" and "binary" are not supported for prediction uploads and are rejected with 400 No valid vectors found in file. Any other value makes the server detect the format from the file content: a detected csv, json or libsvm file is processed, a detected jsonl or binary file is rejected as above, and an undetectable file returns 400 Unable to detect file format. Please specify the format explicitly.
  • has_field_names: (Option 1, optional) Whether the CSV file contains column headers (or the JSON file contains field names). If omitted, the setting of the training dataset is used. The file is parsed with the training dataset's column layout: its vector columns, ID column, attribute columns and (for JSON) vector data field name
  • compression_type: (Option 1, optional) Set to "gzip" if file is gzip compressed
  • prediction_dataset: (Option 2) Name of existing dataset to make predictions on. It must have the same vector type and (for dense data) the same dimension as the training dataset. If both prediction_dataset and file_data are given, prediction_dataset is used
  • return_all_probabilities: (Optional, default false) Classification models only. When true (or 1), the response also includes class_labels and probability_matrix, the predicted probability of every class for every row. Ignored for regression models

Response For Classification Models: 

{
  "success": true,
  "num_samples": 1000,
  "predictions": ["cat", "dog", "bird", ...],
  "prediction_metric": {
    "accuracy": 0.85,
    "micro_precision": 0.86, 
    "macro_precision": 0.84,
    "micro_recall": 0.85,
    "macro_recall": 0.83,
    "micro_f1": 0.855,
    "macro_f1": 0.835,
    "auc": 0.92
  }
}

For Regression Models: 

{
  "success": true,
  "num_samples": 1000,
  "predictions": [12.5, 8.3, 15.7, ...],
  "prediction_metric": {
    "mse": 0.45,  // MSE value
    "mae": 0.62
  }
}

For Classification Models with "return_all_probabilities": true, two more fields are returned: 

{
  "success": true,
  "num_samples": 1000,
  "predictions": ["cat", "dog", "bird", ...],
  "class_labels": ["bird", "cat", "dog"],
  "probability_matrix": [
    [0.031, 0.902, 0.067],
    [0.105, 0.210, 0.685],
    [0.874, 0.051, 0.075],
    ...
  ],
  "prediction_metric": { ... }
}

Error Response:

If the prediction itself fails, the server returns HTTP 200 with: 

{
  "success": false,
  "error_message": "Server exception while autoML prediction: <description>"
}
Request errors are returned as HTTP 400 with a plain-text message, for example Missing training configuration: model name, or data file is not uploaded., A prediction dataset is upload, file_format must be specified., return_all_probabilities must be true/false or 1/0., Project does not exist!, Dataset does not exist!, Model does not exist! (a model_name that does not exist on the dataset), Prediction dataset does not exist!, AutoML predict: prediction dataset and train data mismatch in dimension. ..., or No valid vectors found in file. An invalid API key returns HTTP 401 Unauthorized user. Please check your VecML API key.

Two other HTTP 200 responses can occur: {"success": false, ...} without error_message when the computed metrics are invalid, and {"error_code": "<code>"} (no success field) when the temporary dataset for an uploaded file cannot be created.

Response Fields

  • num_samples: Number of rows that have a label (the model's label_attribute) and are used to compute the prediction metrics
  • predictions: Array of prediction values, one per input row in input order. For classification these are the original label strings from the training data (e.g. "cat", "1"); for regression, numbers
  • class_labels: (Only with return_all_probabilities, classification) The class labels, in the column order of probability_matrix. This order is fixed per model but is not sorted, so always read it rather than assuming one. If some training vectors had no label, the empty string "" is one of the classes (see /train_automl_model)
  • probability_matrix: (Only with return_all_probabilities, classification) One row per input row, in the same order as predictions; probability_matrix[i][c] is the predicted probability that row i belongs to class_labels[c]. Each row sums to 1 (values are rounded to 6 decimals), and its largest entry is the class given in predictions[i]
  • prediction_metric: Performance metrics if true labels are available in the data; null when no row has a label (num_samples is 0), in which case predictions is still returned
    • For classification: detailed metrics object
    • For regression: MSE (Mean Squared Error) and MAE (Mean Absolute Error)

Notes

  • The prediction data must have the same vector dimensions and format as the training data
  • When uploading files, supported formats are CSV, JSON and LibSVM
  • Prediction metrics are only calculated if at least one true label is present in the prediction data (an attribute/column with the model's label_attribute name). Data without labels can still be predicted: predictions is returned and prediction_metric is null
  • return_all_probabilities adds roughly (number of rows) x (number of classes) numbers to the response; leave it off when only the predicted labels are needed
  • A CSV prediction file without a header is read with the training dataset's column layout: each attribute whose name contains digits is read as that column (e.g. "column 5"), and names without digits (such as an attached label) are skipped. An attribute added later whose name happens to contain digits is also read as a column, so for such a dataset use prediction_dataset or a JSON file instead
  • Uploaded prediction files are stored in a temporary dataset that is deleted when the request finishes, whether it succeeds or fails

Example 

POST /automl_predict
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "MLProject",
  "collection_name": "customer_data", 
  "model_name": "customer_segmentation_v1",
  "prediction_dataset": "new_customers"
}

Response: 

{
  "success": true,
  "num_samples": 393,
  "predictions": ["premium", "standard", "basic", "standard", "premium", ...],
  "prediction_metric": {
    "accuracy": 0.89,
    "micro_precision": 0.91,
    "macro_precision": 0.88,
    "micro_recall": 0.89,
    "macro_recall": 0.87,
    "micro_f1": 0.90,
    "macro_f1": 0.875,
    "auc": 0.94
  }
}

/get_feature_importance

Description: Retrieves the feature importance scores for a trained AutoML model. Returns the top features ranked by their contribution to the model's predictions.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "model_name": "string"
}

Required Fields:

  • user_api_key: Your VecML API key for authentication
  • project_name: The project containing the model
  • collection_name: The dataset name used to train the model
  • model_name: The name of the trained model

Response Success Response (200 OK): 

{
  "success": true,
  "feature_importance": [
    {
      "feature": "string",
      "importance": 0.0
    },
    ...
  ]
}

Response Fields:

  • success: Boolean indicating if the request was successful
  • feature_importance: Array of feature objects, sorted by importance (descending order)
    • feature: Name of the feature: the column name from the dataset, or the name of a categorical feature. For sparse datasets, and dense datasets without column names, vector elements are named "Dim 1", "Dim 2", ...
    • importance: Importance score relative to the most important feature: scores are divided by the largest one, so the top feature has 1.0 and the others are in [0, 1] (if every score is 0, all are returned as 0). The scores do not sum to 1

Notes:

  • Returns at most the top 50 features (fixed; there is no parameter to change it)
  • Feature names correspond to the column names in the original training dataset; a categorical feature is reported once, combining all of its values
  • The importance scores come from the best linear (LibLinear) or tree (ABCBoost) candidate trained during AutoML, which is not necessarily the model that was finally selected. If no such candidate was trained, feature_importance is an empty array
  • Missing fields return HTTP 400 (Missing fields in JSON: ...), as do Project does not exist! and Dataset does not exist!. A model_name that does not exist returns HTTP 500

Example 

POST /get_feature_importance
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "model_name": "classification_model_1"
}

Response: 

{
  "success": true,
  "feature_importance": [
    {
      "feature": "income",
      "importance": 1.0
    },
    {
      "feature": "age",
      "importance": 0.7377
    },
    {
      "feature": "credit_score",
      "importance": 0.5631
    },
    {
      "feature": "employment_length",
      "importance": 0.3881
    },
    {
      "feature": "debt_ratio",
      "importance": 0.2572
    }
  ]
}

/delete_automl_model

Description: Deletes a specific AutoML model from a dataset. This action cannot be undone.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string", 
  "collection_name": "string",
  "model_name": "string"
}

Response Success Response (200 OK): 

{
  "success": true
}

If the model cannot be deleted (for example, no model with that name exists), the server still returns HTTP 200, with: 

{
  "success": false,
  "error_code": "string"
}

Missing fields, an unknown project (Project does not exist!) or an unknown dataset (Dataset does not exist!) return HTTP 400 with a plain-text message.

Example 

POST /delete_automl_model
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "model_name": "old_classification_model"
}

Response: 

{
  "success": true
}

/get_automl_training_status

Description: Retrieves the current status and progress of an AutoML training job.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "job_id": "string"
}

Response For In-Progress Jobs: 

{
  "success": true,
  "task_type": "classification",
  "model_name": "model_123",
  "label_attribute": "category",
  "status": "in_progress",
  "start_time": "2025/01/15, 14:30:25",
  "end_time": "2025/01/15, 14:35:55",
  "duration": "5 minutes 30 seconds"
}

For Completed Jobs: 

{
  "success": true,
  "task_type": "classification",
  "model_name": "model_123", 
  "label_attribute": "category",
  "status": "finished",
  "start_time": "2025/01/15, 14:30:25",
  "end_time": "2025/01/15, 14:43:10",
  "duration": "12 minutes 45 seconds",
  "validation_metric": {
    "accuracy": 0.85,
    "micro_precision": 0.86,
    "macro_precision": 0.84,
    "micro_recall": 0.85,
    "macro_recall": 0.83,
    "micro_f1": 0.855,
    "macro_f1": 0.835,
    "auc": 0.92
  }
}

For Failed Jobs: 

{
  "success": true,
  "task_type": "regression",
  "model_name": "model_456",
  "label_attribute": "price", 
  "status": "failed",
  "start_time": "2025/01/15, 15:00:10",
  "end_time": "2025/01/15, 15:02:25",
  "duration": "2 minutes 15 seconds",
  "error": "Training failed"
}

If the job does not exist, or belongs to another user, the server returns HTTP 404: 

{
  "success": false,
  "error": "Job not found"
}

Notes

  • Status values: "pending", "in_progress", "finished", "failed"

  • For classification tasks, validation_metric contains detailed metrics

  • For regression tasks, validation_metric contains MSE (Mean Squared Error) and MAE (Mean Absolute Error)

  • Times are formatted as "YYYY/MM/DD, HH:MM:SS" (server local time). For a running job, end_time is the current time; for a pending job, start_time and end_time are the current time and duration is "0 seconds"

  • Duration is a readable string such as "1 hour 5 minutes 2 seconds" or "2 days 3 hours" (zero units are omitted; durations under a second are shown as e.g. "350 milliseconds")

  • For failed jobs, error is always "Training failed"

  • Missing job_id or user_api_key returns HTTP 400 (Missing job_id in request / Missing user_api_key.)

Example 

POST /get_automl_training_status
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "job_id": "user_123||MLProject||customer_data||customer_segment_model||TrainJob"
}

Response: 

{
  "success": true,
  "task_type": "classification",
  "model_name": "customer_segment_model",
  "label_attribute": "segment",
  "status": "finished", 
  "start_time": "2025/01/15, 14:30:25",
  "end_time": "2025/01/15, 14:39:07",
  "duration": "8 minutes 42 seconds",
  "validation_metric": {
    "accuracy": 0.91,
    "micro_precision": 0.92,
    "macro_precision": 0.90,
    "micro_recall": 0.91,
    "macro_recall": 0.89,
    "micro_f1": 0.915,
    "macro_f1": 0.895,
    "auc": 0.96
  }
}

/get_model_validation_metric

Description: Retrieves validation metrics for a specific trained AutoML model.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string", 
  "model_name": "string"
}

Response: For Classification Models: 

{
  "success": true,
  "validation_metric": {
    "accuracy": 0.85,
    "micro_precision": 0.86,
    "macro_precision": 0.84,
    "micro_recall": 0.85,
    "macro_recall": 0.83,
    "micro_f1": 0.855,
    "macro_f1": 0.835,
    "auc": 0.92
  }
}

For Regression Models: 

{
  "success": true,
  "validation_metric": {
    "mse": 0.45,   // MSE value
    "mae": 0.61
  }
}

Notes

  • Missing fields return HTTP 400 with a plain-text message (Missing fields in JSON: ...), as do Project does not exist! and Dataset does not exist!.
  • A model_name that does not exist on the dataset returns HTTP 500.

Example 

POST /get_model_validation_metric
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA", 
  "collection_name": "MyDataset",
  "model_name": "classification_model_1"
}

Response: 

{
  "success": true,
  "validation_metric": {
    "accuracy": 0.89,
    "micro_precision": 0.90,
    "macro_precision": 0.88,
    "micro_recall": 0.89,
    "macro_recall": 0.87,
    "micro_f1": 0.895,
    "macro_f1": 0.875,
    "auc": 0.94
  }
}

/list_automl_model_infos

Description: Retrieves metadata for all AutoML models within a specific dataset.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string"
}

Response Success Response (200 OK): 

{
  "success": true,
  "model_infos": [
    {
      "model_name": "string",
      "task_type": "string",
      "training_mode": "string",
      "label_attribute": "string",
      "feature_augmentation": bool,
      "validation_dataset": "string",
      "create_time": "string"
    },
    ...
  ]
}

Response Fields: - model_name: Name of the trained model - task_type: Type of machine learning task ("classification" or "regression") - training_mode: Training mode used ("linear_model", "high_speed", "balanced", or "high_accuracy") - label_attribute: Name of the label/target attribute used for training - feature_augmentation: (boolean) Whether the selected model uses feature augmentation - validation_dataset: Name of the validation dataset used ("cross_validation" if auto-split was used) - create_time: Time when the model training was started, formatted as "YYYY/MM/DD, HH:MM:SS"

Missing fields, an unknown project (Project does not exist!) or dataset (Dataset does not exist!) return HTTP 400 with a plain-text message.

Example 

POST /list_automl_model_infos
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset"
}

Response: 

{
  "success": true,
  "model_infos": [
    {
      "model_name": "classification_model_1",
      "task_type": "classification",
      "training_mode": "high_speed",
      "label_attribute": "category",
      "feature_augmentation": true,
      "validation_dataset": "cross_validation",
      "create_time": "2025/01/15, 14:30:25"
    },
    {
      "model_name": "regression_model_1",
      "task_type": "regression",
      "training_mode": "balanced",
      "label_attribute": "price",
      "feature_augmentation": false,
      "validation_dataset": "validation_set",
      "create_time": "2025/01/16, 09:15:02"
    }
  ]
}

Clustering API Endpoints

VecML provides built-in clustering algorithms (K-Means and DBSCAN) that run directly on your vector collections. Training is asynchronous — you receive a job_id and poll for status.

Clustering Workflow

A typical clustering workflow:

  1. Create a project and upload a vector dataset (see Project Management and Dataset Management).
  2. (Optional) Use /get_dbscan_kdist_plot to explore your data and choose DBSCAN parameters.
  3. Train a clustering model using /train_kmeans or /train_dbscan.
  4. Check training status with /get_clustering_training_status.
  5. Retrieve cluster labels with /get_clustering_labels.
  6. Evaluate results with /compute_clustering_metrics.
  7. Visualize clusters with /compute_tsne_embeddings.

Error responses of the clustering result endpoints

/delete_clustering_model, /get_clustering_labels, /get_kmeans_centroids, /get_dbscan_core_points, /get_dbscan_kdist_plot, /compute_clustering_metrics and /compute_tsne_embeddings return request errors as JSON objects {"error": "<message>"} (some also include "error_code"): HTTP 400 for invalid JSON, a missing field (Missing required field: <name>) or Project or dataset does not exist!, HTTP 401 for an invalid API key, HTTP 429 (not 400) when the rate limit is exceeded, and HTTP 500 {"error": "Failed to initialize dataset", "error_code": "..."} if the dataset cannot be loaded. /train_kmeans, /train_dbscan, /get_clustering_training_status and /list_clustering_model_infos use plain-text error messages like the other endpoints.

/train_kmeans

Description: Initiates training of a K-Means clustering model on a specified dataset. This is an asynchronous operation that returns a job ID for tracking progress.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "cluster_model_name": "string",
  "kmeans_method": "string",
  "n_clusters": 10,
  "n_init": 3,
  "max_train_samples": 10000,
  "max_iterations": 100,
  "tolerance": 1e-4,
  "num_threads": 4,
  "random_seed": 1234567,
  "initialization_method": "string",
  "dist_type": "string"
}

Required Fields:

  • user_api_key: Your VecML API key for authentication.
  • project_name: The project containing the dataset.
  • collection_name: The dataset to cluster.
  • cluster_model_name: Unique name for this clustering model within the dataset. The same length, character and reserved-name rules as for an AutoML model_name apply (see /train_automl_model), otherwise HTTP 400; see the notes below for names already in use.
  • kmeans_method: K-Means variant to use. Supported values: "standard" or "lloyd" (standard Lloyd's algorithm), "hamerly" (Hamerly's accelerated algorithm, faster for low-dimensional data). Any other value is accepted by the request but the training job fails with Invalid kmeans_method.
  • n_clusters: Number of clusters to form, an integer >= 1.

Optional Fields:

The integer fields must be JSON integers: n_init, max_train_samples and max_iterations >= 1, num_threads between 1 and 1024, random_seed within the 32-bit signed range; tolerance must be a number >= 0. Other values return HTTP 400 (Invalid <field>! Must be an integer between <min> and <max>. or Invalid tolerance! Must be a number >= 0.).

Field Default Description
n_init 3 Number of times to run K-Means with different random seeds. The best result (lowest inertia) is kept.
max_train_samples 10000 Number of samples used for training: the actual count is min(max(max_train_samples, 10 × n_clusters), dataset size), drawn at random when smaller than the dataset.
max_iterations 100 Maximum number of iterations per K-Means run.
tolerance 1e-4 Convergence threshold. Training stops when the change in centroids falls below this value.
num_threads 4 Number of threads for parallel computation.
random_seed 1234567 Random seed for reproducibility.
initialization_method "k-means++" Centroid initialization strategy. "k-means++" provides smarter initialization; "random" selects random data points. Any other value makes the training job fail.
dist_type "Euclidean" Distance metric: "Euclidean" / "euclidean" / "L2" / "l2", "Manhattan" / "manhattan" / "L1" / "l1", "Cosine" / "cosine", "Inner Product" / "inner product" / "InnerProduct". "standard"/"lloyd" support all four (with Manhattan, training runs on the CPU: points are assigned by L1 distance and each centroid is the mean of its points); "hamerly" supports only Euclidean and Manhattan: another distance with "hamerly" returns HTTP 400 kmeans_method "hamerly" supports only Euclidean and Manhattan distance. Hamming is not supported by K-Means: HTTP 400 Hamming is not supported by K-Means. An unrecognized value is treated as "Euclidean".

Response 

{
  "success": true,
  "job_id": "string"
}

  • job_id: Format: <user_id>||<project_name>||<collection_name>||<cluster_model_name>||KMeansTrainJob||<uuid>.

Notes:

  • Model names must be unique within a dataset. A name already used by an index, an AutoML model or a clustering model of the dataset is rejected with HTTP 400 The name is already used by an index or a model in this dataset!. (A model of the same name that is still training is not yet listed; the later job then fails with Clustering model already exists, please specify another model name!.)
  • A training failure (for example n_clusters larger than the number of vectors) makes the job fail; see /get_clustering_training_status.
  • Use /get_clustering_training_status with the returned job_id to monitor progress.
  • For high-dimensional embedding vectors (e.g., 768 or 1024 dimensions), "Cosine" distance is often the best choice; use "standard"/"lloyd" for it, since "hamerly" does not support Cosine.
  • n_init > 1 improves result quality at the cost of longer training time.
  • Errors: a missing required field returns HTTP 400 Missing required field: <name>; an unknown project or dataset returns HTTP 400 (Project does not exist! / Dataset does not exist!); an invalid name or parameter value returns HTTP 400 (see above); a dataset that cannot be loaded returns HTTP 400 Failed to initialize the dataset (or HTTP 200 {"success": false, "error_code": "..."} if its vector type cannot be read); an invalid API key returns HTTP 401.

Example 

POST /train_kmeans
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "EmbeddingProject",
  "collection_name": "document_embeddings",
  "cluster_model_name": "topic_clusters_v1",
  "kmeans_method": "lloyd",
  "n_clusters": 20,
  "n_init": 5,
  "dist_type": "Cosine",
  "initialization_method": "k-means++"
}

Response: 

{
  "success": true,
  "job_id": "user_123||EmbeddingProject||document_embeddings||topic_clusters_v1||KMeansTrainJob||1b4e28ba-2fa1-11d2-883f-0016d3cca427"
}

/train_dbscan

Description: Initiates training of a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering model. DBSCAN automatically discovers the number of clusters based on data density and can identify noise points that don't belong to any cluster.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "cluster_model_name": "string",
  "eps": 0.5,
  "min_samples": 5,
  "dist_type": "string",
  "num_threads": 4,
  "random_seed": 1234567
}

Required Fields:

  • user_api_key: Your VecML API key for authentication.
  • project_name: The project containing the dataset.
  • collection_name: The dataset to cluster.
  • cluster_model_name: Unique name for this clustering model. Same rules as for /train_kmeans.
  • eps: The maximum distance between two points for them to be considered neighbors. Must be a number > 0 (otherwise HTTP 400 Invalid eps! Must be a number > 0.); very small values such as 1e-7 are used as given in training (the params of /get_clustering_training_status print values below 1e-4 with full precision, e.g. "1.00000001e-07"). This is the most important parameter — see Choosing DBSCAN Parameters below.
  • min_samples: Must be a JSON integer >= 1 (otherwise HTTP 400 Invalid min_samples! Must be an integer between 1 and 2147483647.). The minimum number of points required to form a dense region (core point). Points in regions with fewer neighbors are classified as noise.
  • dist_type: Distance metric. Supported: "Euclidean" / "euclidean" / "L2" / "l2", "Manhattan" / "manhattan" / "L1" / "l1", "Cosine" / "cosine". Inner product is not supported: "Inner Product" / "inner product" / "InnerProduct" returns HTTP 400, because DBSCAN would compare −⟨x, y⟩ (usually negative) against eps, which must be > 0. Use "Cosine" instead; on normalized vectors it orders neighbors the same way. Hamming is not supported either: "Hamming" / "hamming" returns HTTP 400 Hamming is not supported by DBSCAN. An unrecognized value is treated as "Euclidean".

Optional Fields:

Field Default Description
num_threads 4 Number of threads for parallel computation. An integer between 1 and 1024 (otherwise HTTP 400).
random_seed 1234567 Random seed for reproducibility. An integer within the 32-bit signed range (otherwise HTTP 400).
create_time current time Creation time string to record for the model (shown by /list_clustering_model_infos).

Response 

{
  "success": true,
  "job_id": "string"
}

Notes:

  • job_id: Format: <user_id>||<project_name>||<collection_name>||<cluster_model_name>||DBSCANTrainJob||<uuid>.
  • Errors: a missing required field returns HTTP 400 Missing required field: <name>; an unknown project or dataset returns HTTP 400; a dataset that cannot be loaded returns HTTP 400 Failed to initialize the dataset (or HTTP 200 {"success": false, "error_code": "..."} if its vector type cannot be read); an invalid API key returns HTTP 401. A name already used by an index, an AutoML model or a clustering model of the dataset is rejected with HTTP 400 The name is already used by an index or a model in this dataset! (a model of the same name that is still training is only detected by the job, which then fails).
  • Unlike K-Means, DBSCAN does not require specifying the number of clusters — it discovers them automatically.
  • Points that don't belong to any cluster are labeled as noise (label = -1).
  • DBSCAN works well for datasets with clusters of varying shapes and sizes.
  • The eps and min_samples parameters significantly affect results. Use the K-distance plot (/get_dbscan_kdist_plot) to guide your parameter selection.

Example 

POST /train_dbscan
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "AnomalyProject",
  "collection_name": "sensor_readings",
  "cluster_model_name": "anomaly_detection_v1",
  "eps": 0.3,
  "min_samples": 10,
  "dist_type": "Euclidean"
}

Response: 

{
  "success": true,
  "job_id": "user_123||AnomalyProject||sensor_readings||anomaly_detection_v1||DBSCANTrainJob||6fa459ea-ee8a-3ca4-894e-db77e160355e"
}

Choosing DBSCAN Parameters

DBSCAN's quality depends heavily on two parameters: eps (neighborhood radius) and min_samples (minimum density). Here's how to choose them:

min_samples:

  • A good starting point is min_samples = 2 * dimension of your data, but for high-dimensional embeddings this may be too large.
  • For most practical use cases with embedding vectors, values between 5 and 20 work well.
  • Higher values produce fewer, denser clusters and more noise points. Lower values produce more clusters and less noise.

eps (using the K-distance plot):

The K-distance plot is the recommended way to choose eps. Here's how it works:

  1. For each point in the dataset, compute the distance to its k-th nearest neighbor (where k = min_samples).
  2. Sort these distances in descending order and plot them.
  3. Look for the "elbow" — the point where the curve transitions from steep to flat. The distance value at this elbow is a good candidate for eps.

Interpreting the K-distance plot:

  • Steep region (left side): These are noise points or outliers — they have large distances to their k-th neighbor.
  • Flat region (right side): These are points inside dense clusters — their k-th neighbor distances are similar and small.
  • Elbow point: The transition between noise and clusters. Setting eps at this value separates meaningful clusters from noise.

Use /get_dbscan_kdist_plot to compute the K-distance values and slope, then look for the elbow in kdist_grid_desc where slope_grid_desc shows the steepest change.

Tips:

  • If too many points are noise: increase eps or decrease min_samples.
  • If everything is one big cluster: decrease eps or increase min_samples.
  • Run DBSCAN with a few different eps values around the elbow and compare results using /compute_clustering_metrics.

/get_clustering_training_status

Description: Retrieves the current status of a clustering training job (K-Means or DBSCAN).

Method - POST

Request Body 

{
  "user_api_key": "string",
  "job_id": "string"
}

Response For In-Progress Jobs: 

{
  "success": true,
  "job_id": "string",
  "model_name": "string",
  "model_type": "kmeans",
  "kmeans_method": "lloyd",
  "status": "in_progress",
  "error_code": "Unknown",
  "params": { ... },
  "start_time": "2025/06/15, 14:30:25",
  "end_time": "2025/06/15, 14:30:55",
  "duration": "30 seconds"
}

For Completed Jobs: 

{
  "success": true,
  "job_id": "string",
  "model_name": "string",
  "model_type": "kmeans",
  "kmeans_method": "lloyd",
  "status": "finished",
  "error_code": "Success",
  "params": {
    "kmeans_method": "lloyd",
    "n_clusters": "20",
    "n_init": "3",
    "dist_type": "Cosine",
    "initialization_method": "k-means++",
    "max_iterations": "100",
    "tolerance": "0.000100",
    "num_threads": "4",
    "random_seed": "1234567",
    "max_train_samples": "10000"
  },
  "start_time": "2025/06/15, 14:30:25",
  "end_time": "2025/06/15, 14:32:10",
  "duration": "1 minute 45 seconds"
}

For Failed Jobs: 

{
  "success": true,
  "job_id": "string",
  "model_name": "string",
  "model_type": "dbscan",
  "status": "failed",
  "error_code": "Unknown",
  "error_message": "Description of the failure",
  "params": { ... },
  "start_time": "2025/06/15, 15:00:10",
  "end_time": "2025/06/15, 15:00:12",
  "duration": "2 seconds"
}

If the job does not exist, or belongs to another user, the server returns HTTP 404 with {"success": false, "error": "Job not found"}. A missing job_id or user_api_key returns HTTP 400 (Missing job_id in request / Missing user_api_key.).

Response Fields:

  • status: One of "pending", "in_progress", "finished", "failed".
  • model_type: "kmeans" (for every kmeans_method, including "hamerly") or "dbscan".
  • kmeans_method: (K-Means only) The variant requested.
  • error_code: "Success" when training finished; "Unknown" while the job is pending or running; for a failed job, the training error code, or "Unknown" if training threw an exception.
  • params: The training parameters as key-value strings. For K-Means: kmeans_method, n_clusters, n_init, dist_type, initialization_method, max_iterations, tolerance, num_threads, random_seed, max_train_samples, where dist_type is the normalized name ("Euclidean", "Manhattan", "Cosine" or "InnerProduct"). For DBSCAN: eps, min_samples, dist_type (the string given in the request), num_threads, random_seed.
  • error_message: (Failed jobs only) Description of the error, e.g. "Exception while training KMeans: <message>", "Exception while training DBSCAN: <message>", or "train_kmeans returned <error code>" / "train_dbscan returned <error code>".
  • Times are formatted as "YYYY/MM/DD, HH:MM:SS"; duration is a readable string such as "1 minute 45 seconds". For a running job, end_time is the current time; for a pending job, start_time and end_time are the current time and duration is "0 seconds".

/list_clustering_model_infos

Description: Retrieves metadata for all clustering models within a specific dataset.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string"
}

Response 

{
  "success": true,
  "model_infos": [
    {
      "model_name": "topic_clusters_v1",
      "model_type": "kmeans",
      "distance_type": "Cosine",
      "create_time": "2025/06/15, 14:30:25",
      "model_path": "path/to/model",
      "params": {
        "n_clusters": 20,
        "n_init": 3,
        "initialization_method": "k-means++",
        "max_iterations": 100,
        "max_train_samples": 10000,
        "tolerance": 9.999999747378752e-05,
        "num_threads": 4,
        "random_seed": 1234567
      }
    },
    {
      "model_name": "anomaly_v1",
      "model_type": "dbscan",
      "distance_type": "Euclidean",
      "create_time": "2025/06/16, 09:15:02",
      "model_path": "path/to/model",
      "params": {
        "eps": 0.30000001192092896,
        "min_samples": 10,
        "num_threads": 4,
        "random_seed": 1234567
      }
    }
  ]
}

Response Fields:

  • model_type: "kmeans" (standard / Lloyd), "kmeans_hamerly", or "dbscan". Note that /get_clustering_training_status reports "kmeans" for both K-Means variants.
  • distance_type: The distance metric used for training: "Euclidean", "Manhattan", "Cosine" or "InnerProduct".
  • create_time: Formatted as "YYYY/MM/DD, HH:MM:SS" (for DBSCAN, the create_time given in the training request, if any).
  • params: Algorithm-specific parameters. K-Means: n_clusters, n_init, max_iterations, tolerance, num_threads, random_seed, max_train_samples, initialization_method. DBSCAN: eps, min_samples, num_threads, random_seed. tolerance and eps are stored as 32-bit floats, so they come back with float rounding (e.g. 0.3 as 0.30000001192092896).

Missing fields, an unknown project (Project does not exist!) or dataset (Dataset does not exist!) return HTTP 400 with a plain-text message.

/delete_clustering_model

Description: Deletes a specific clustering model from a dataset. This action cannot be undone.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "model_name": "string"
}

Response 

{
  "success": true,
  "status": "finished",
  "error_code": "Success"
}

If the model cannot be deleted (e.g. it does not exist), the response is still HTTP 200, with "success": false, "status": "failed" and the reason in error_code.

Errors from this endpoint are JSON objects {"error": "<message>"} (plus "error_code" for some): HTTP 400 for invalid JSON, a missing field (Missing required field: <name>) or Project or dataset does not exist!; HTTP 401 for an invalid API key; HTTP 429 when the rate limit is exceeded; HTTP 500 if the dataset cannot be loaded.

Example 

POST /delete_clustering_model
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "model_name": "old_kmeans_model"
}

/get_clustering_labels

Description: Retrieves the cluster label assigned to each vector in the dataset by a trained clustering model.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "model_name": "string"
}

Response 

{
  "success": true,
  "error_code": "Success",
  "labels": [0, 1, 2, 0, 3, -1, 2, 1, ...]
}

Response Fields:

  • labels: Array of integer cluster labels, one per vector in the dataset. The order matches the internal vector ordering.
    • For K-Means: labels range from 0 to n_clusters - 1.
    • For DBSCAN: labels range from 0 to num_clusters - 1, with -1 indicating noise points.
  • If the model exists but is not trained yet, the response is HTTP 200 with "success": false, "error_code": "ModelNotTrained", and no labels.
  • If no clustering model with that name exists, the server returns HTTP 400 with {"error": "Invalid model name, clustering model does not exist."}.

Example 

POST /get_clustering_labels
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "model_name": "topic_clusters_v1"
}

/get_kmeans_centroids

Description: Retrieves the centroid vectors of a trained K-Means model.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "model_name": "string"
}

Response 

{
  "success": true,
  "error_code": "Success",
  "centroids": [
    [0.12, -0.45, 0.78, ...],
    [0.56, 0.23, -0.91, ...],
    ...
  ]
}

Response Fields:

  • centroids: 2D array of shape [n_clusters, vector_dim]. Each row is the centroid vector for one cluster.

Notes:

  • This endpoint is only available for K-Means models. Calling it on a DBSCAN model, or with a model name that does not exist, returns HTTP 400 with a JSON error {"error": "<message>"}. If the model exists but is not trained yet, the response is HTTP 200 with "success": false, "error_code": "ModelNotTrained", and no centroids.
  • Centroids can be used for nearest-centroid classification or as representative vectors for each cluster.

/get_dbscan_core_points

Description: Retrieves the indices of core points identified by a trained DBSCAN model. Core points are data points that have at least min_samples neighbors within eps distance.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "model_name": "string"
}

Response 

{
  "success": true,
  "error_code": "Success",
  "core_points": [0, 3, 5, 7, 12, 15, ...]
}

Response Fields:

  • core_points: Array of integer indices identifying which vectors in the dataset are core points.

Notes:

  • This endpoint is only available for DBSCAN models. Calling it on a K-Means model, or with a model name that does not exist, returns HTTP 400 with a JSON error {"error": "<message>"}. If the model exists but is not trained yet, the response is HTTP 200 with "success": false, "error_code": "ModelNotTrained", and no core_points.
  • Core points are the "anchors" of each cluster. Non-core points that are within eps of a core point are border points; the rest are noise.

/get_dbscan_kdist_plot

Description: Computes the K-distance plot data for a dataset, which is used to determine the optimal eps parameter for DBSCAN. The K-distance plot shows, for each point, the distance to its k-th nearest neighbor, sorted in descending order.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "k": 5,
  "max_num_samples": 2000,
  "max_num_grids": 200,
  "dist_type": "string",
  "num_threads": 3,
  "random_seed": 1234567
}

Required Fields:

  • user_api_key: Your VecML API key.
  • project_name: The project containing the dataset.
  • collection_name: The dataset to analyze.

Optional Fields:

Field Default Description
k 5 The k-th nearest neighbor to compute distances for. Should match the min_samples value you plan to use for DBSCAN.
max_num_samples 2000 Maximum number of samples to use for computing the plot. A random subset is selected if the dataset is larger.
max_num_grids 200 Number of evenly spaced grid points in the output arrays. Controls the resolution of the plot.
dist_type "Euclidean" Distance metric. Should match the dist_type you plan to use for DBSCAN. Supported: "Euclidean" / "euclidean" / "L2" / "l2", "Manhattan" / "manhattan" / "L1" / "l1", "Cosine" / "cosine". Not supported: inner product ("Inner Product" / "inner product" / "InnerProduct") returns HTTP 400 {"error": "InnerProduct is not supported by DBSCAN: ..."}, and "Hamming" / "hamming" returns HTTP 400 {"error": "Hamming is not supported by DBSCAN."}. An unrecognized value is treated as "Euclidean".
num_threads 3 Number of threads for parallel computation (values ≤ 0 are treated as 1).
random_seed 1234567 Random seed for reproducibility.

Response 

{
  "success": true,
  "error_code": "Success",
  "k": 5,
  "dist_type": "Euclidean",
  "max_num_samples": 2000,
  "max_num_grids": 200,
  "kdist_grid_desc": [2.45, 2.41, 2.38, ..., 0.12, 0.08],
  "slope_grid_desc": [-0.02, -0.03, -0.05, ..., -0.8, -1.2]
}

Response Fields:

  • dist_type: The normalized name of the distance metric used: "Euclidean", "Manhattan" or "Cosine".
  • kdist_grid_desc: K-distance values in descending order. The server picks min(max_num_samples, dataset size) random points, computes each one's distance to its k-th nearest neighbor among all vectors in the dataset, and returns these distances at G evenly spaced quantiles, where G = min(number of points with a valid distance, max_num_grids). Both arrays have length G. These are the y-axis values of the K-distance plot.
  • slope_grid_desc: Array of slope values at each grid point (first derivative of the K-distance curve). Large negative slopes indicate the "elbow" region.
  • If the computation fails, the response is HTTP 200 with "success": false, the reason in error_code, and no kdist_grid_desc / slope_grid_desc. k, max_num_samples or max_num_grids ≤ 0 returns HTTP 400 {"error": "Invalid parameter: ... must be > 0"}. The integer fields (k, max_num_samples, max_num_grids, num_threads, random_seed) must be JSON integers within the 32-bit signed range; otherwise the response is HTTP 400 {"error": "Invalid <field>! Must be an integer between -2147483648 and 2147483647."}.

How to Use the Response:

  1. Plot the curve: Use kdist_grid_desc as y-values (x-axis is just the index from 0 to max_num_grids - 1).
  2. Find the elbow: Look for where slope_grid_desc has the most negative value (steepest descent). The corresponding kdist_grid_desc value at that index is a good candidate for eps.
  3. Alternatively: Visually inspect the plot for the "knee" — the transition point from steep to flat.

Example 

POST /get_dbscan_kdist_plot
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "k": 10,
  "dist_type": "Cosine",
  "max_num_samples": 5000,
  "max_num_grids": 300
}

Response: 

{
  "success": true,
  "error_code": "Success",
  "k": 10,
  "dist_type": "Cosine",
  "max_num_samples": 5000,
  "max_num_grids": 300,
  "kdist_grid_desc": [0.95, 0.93, 0.91, 0.88, ...],
  "slope_grid_desc": [-0.01, -0.01, -0.02, -0.03, ...]
}

/compute_clustering_metrics

Description: Computes evaluation metrics for a trained clustering model. Supports both unsupervised metrics (no ground truth needed) and supervised metrics (when ground truth labels are available).

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "model_name": "string",
  "ground_truth_attribute": "string",
  "ground_truth_labels": ["string", ...]
}

Required Fields:

  • user_api_key: Your VecML API key.
  • project_name: The project containing the dataset.
  • collection_name: The dataset.
  • model_name: The trained clustering model to evaluate.

Optional Fields:

  • ground_truth_attribute: Name of a vector attribute containing true cluster labels. Required for supervised metrics.
  • ground_truth_labels: Array of ground truth label strings, one per vector. Alternative to ground_truth_attribute. Required for supervised metrics.

Response 

{
  "success": true,
  "status": "finished",
  "metrics": {
    "distance_type": "Euclidean",
    "ground_truth_attribute": "",
    "ground_truth_labels_provided": false,
    "noise_ratio": 0.05,
    "inertia_sse": 1234.56,
    "davies_bouldin": 1.23,
    "calinski_harabasz": 456.78,
    "silhouette": 0.45,
    "adjusted_rand_index": 0.67,
    "normalized_mutual_info": 0.72,
    "best_match_accuracy": 0.81,
    "homogeneity": 0.75,
    "completeness": 0.70,
    "v_measure": 0.72
  }
}

Each metric is included only if it could be computed and its value is finite; otherwise its key is omitted. distance_type is the model's distance metric ("Euclidean", "Manhattan", "Cosine" or "InnerProduct"), and ground_truth_attribute / ground_truth_labels_provided echo the request. If no clustering model with that name exists, the server returns HTTP 400 with {"error": "Invalid model name, clustering model does not exist."}. If the computation throws an error, the server returns HTTP 500 with {"error": "Exception in compute_clustering_metrics: ..."}.

Unsupervised Metrics (no ground truth needed):

Metric Description Good Values
noise_ratio Fraction of points labeled as noise (-1). Only meaningful for DBSCAN. Lower is usually better, but some noise is expected.
inertia_sse / L1_dispersion / cosine_dispersion Within-cluster sum of distances; the key is inertia_sse for Euclidean, L1_dispersion for Manhattan and cosine_dispersion for Cosine models. Not returned for other distance types. Lower is better. Not comparable across different n_clusters.
davies_bouldin Ratio of within-cluster to between-cluster distances. Lower is better. 0 is perfect.
calinski_harabasz Ratio of between-cluster to within-cluster variance. Higher is better.
silhouette How similar each point is to its own cluster vs. nearest neighboring cluster. Range [-1, 1]. Higher is better. > 0.5 is good.

Supervised Metrics (only when ground truth is provided):

Metric Description Range
adjusted_rand_index Similarity between predicted and true labels, adjusted for chance. [-1, 1]. 1 = perfect.
normalized_mutual_info Mutual information between predicted and true labels, normalized. [0, 1]. 1 = perfect.
best_match_accuracy Best one-to-one matching accuracy between predicted and true labels. [0, 1]. 1 = perfect.
homogeneity Whether each cluster contains only members of a single true class. [0, 1]. 1 = perfect.
completeness Whether all members of a true class are assigned to the same cluster. [0, 1]. 1 = perfect.
v_measure Harmonic mean of homogeneity and completeness. [0, 1]. 1 = perfect.

Example 

POST /compute_clustering_metrics
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "model_name": "topic_clusters_v1",
  "ground_truth_attribute": "true_category"
}

/compute_tsne_embeddings

Description: Computes 2D t-SNE embeddings for visualizing clustering results. Each point is projected to 2D space and labeled with its cluster assignment.

Method - POST

Request Body 

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "model_name": "string",
  "max_samples": 2000,
  "perplexity": 30.0,
  "theta": 0.5,
  "max_iter": 600,
  "learning_rate": 200.0
}

Required Fields:

  • user_api_key: Your VecML API key.
  • project_name: The project containing the dataset.
  • collection_name: The dataset.
  • model_name: The trained clustering model (used for coloring points by cluster label).

Optional Fields:

Field Default Description
max_samples 2000 Maximum number of points to include. A random subset is selected if the dataset is larger.
perplexity 30.0 Balances attention between local and global structure. Typical range: 5–50. Lower values emphasize local structure. Must be at most (N − 1) / 3, where N = min(max_samples, number of vectors in the dataset) — see the note below.
theta 0.5 Speed/accuracy trade-off for Barnes-Hut approximation. 0 = exact (slow), 1 = fast (approximate).
max_iter 600 Maximum number of optimization iterations.
learning_rate 200.0 Learning rate of the t-SNE optimizer. Must be positive.

Response 

{
  "success": true,
  "status": "finished",
  "num_points": 2000,
  "parameters": {
    "max_samples": 2000,
    "perplexity": 30.0,
    "theta": 0.5,
    "max_iter": 600,
    "learning_rate": 200.0
  },
  "embeddings": [
    {"label": 0, "x": 12.34, "y": -5.67},
    {"label": 2, "x": -8.91, "y": 3.45},
    {"label": -1, "x": 45.12, "y": 22.33},
    ...
  ]
}

Response Fields:

  • num_points: Number of points in the embedding.
  • embeddings: Array of 2D points, each with:
    • label: Cluster assignment from the clustering model (-1 = noise for DBSCAN).
    • x, y: 2D coordinates from t-SNE.

Notes:

  • parameters echoes max_samples, perplexity, theta, max_iter and learning_rate.
  • Invalid parameters return HTTP 400 with a JSON error: Invalid max_samples! ... / Invalid max_iter! ... (these two must be JSON integers within the 32-bit signed range), max_samples must be positive, perplexity must be positive, theta must be in range [0, 1], max_iter must be positive, learning_rate must be positive, or Invalid t-SNE parameters: perplexity must be at most (number of points - 1) / 3 = ... (see below).
  • An unknown model returns HTTP 400 with {"error": "Invalid model name, clustering model does not exist."}.
  • If the embedding cannot be computed, the server returns HTTP 500 with a JSON error {"error": "Failed to compute t-SNE embeddings: <error code>"} (e.g. fewer than 2 points).
  • The result is deterministic: the same request on the same data returns the same layout (fixed random seed).
  • For large datasets, use max_samples to limit computation time.

Perplexity must be small enough for the number of points

perplexity must be at most (N − 1) / 3, where N = min(max_samples, number of vectors in the dataset); for example, with 100 vectors use a perplexity of at most 33. A larger value is rejected with HTTP 400 {"error": "Invalid t-SNE parameters: perplexity must be at most (number of points - 1) / 3 = ..."}.

  • Points with label -1 (DBSCAN noise) are typically scattered away from cluster centers in the visualization.

Example 

POST /compute_tsne_embeddings
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "model_name": "topic_clusters_v1",
  "max_samples": 5000,
  "perplexity": 40.0
}