Vector DB API

This document describes each endpoint of VecML RESTful API, including:

• Endpoint URL
• Expected Request Body (JSON)
• Response Format (JSON for successful calls; many error responses are plain text, see below)
• Relevant Details or Constraints
• Example Requests / Responses

All RESTful API calls to VecML cloud database should be sent to: https://aidb.vecml.com/api. All endpoints require a JSON body unless otherwise noted.

Error responses

Request validation errors (missing fields including a missing user_api_key, invalid values, unknown project or dataset, ...) are returned as an HTTP 400 with a plain-text body whose message varies by endpoint. An invalid API key is answered with HTTP 401 and the plain-text body Unauthorized user. Please check your VecML API key. A field of the wrong JSON type (for example a number where a string is expected) is generally answered with HTTP 400 and the plain-text body Invalid request field: <reason>; the exceptions are noted per endpoint. Some operations report failures inside a JSON body instead (for example {"success": false, "error_code": "..."} with HTTP 200); these cases are noted per endpoint. Client code should therefore not assume that every response body is JSON.

Rate and request size limit

For resource allocation and server stability, the server enforces a per-user rate limit (at most 1000 API calls in any 3-second window). If exceeded, the server responds with:

400 Bad Request
Request limit reached, please try again later

Please avoid high-frequency API calls. For example, when inserting multiple vectors, please use relatively large batch size in /add_data_batch and avoid calling the API too frequently.

The size of each request cannot pass 200MB.

Managing Unique IDs for vectors

Vectors are identified by UNIQUE string IDs. While VecML provide a way to maintain auto-generated string IDs internally, we highly encourage the users to maintain and specify unique IDs for the vectors, for the convenience of database operations. Multiple ways are available to specify the IDs, please check the data inserting functions for details.

---

Authentication (API key):

VecML RESTful API requests are authenticated through user's API key. The API key can be generated as follows:

1. Go to https://account.vecml.com/user-api-keys , sign up a free VecML account.
2. After registration, you can get an API key by clicking "Create New API Key".
3. This unique API key will only be seen once at the creation time, so please keep the API key safely. If you need to re-generate an API key, simply delete the previous one and then create a new key.

Pass the key in the user_api_key field of every request body. A missing user_api_key field is rejected with HTTP 400; an invalid key with HTTP 401 and the plain-text body Unauthorized user. Please check your VecML API key.

---

An Example WorkFlow

With VecML API, managing vector database and searching for queries become easy. Below is a standard workflow using VecML RESTful API:

1. Call /create_project to create a new project.

2. Create a dataset (or collection, interchangably) within the project. Two convenient options are:

   (i) Call /init to initialize/create a new dataset. Then use /add_data_batch to (iteratively) add batch of vectors to the dataset.

   (ii) Upload a data file (supported types: csv, json, binary format, libsvm) as a dataset using /upload. You can iteratively upload files to a data collection.

3. Build an index for the dataset using /attach_index. The dataset must be loaded on the server; if /attach_index answers This collection hasn't been initialized with POST or GET /init (for example after a long idle period), call /init for the dataset first with the dataset's vector_type (vector_dim can be omitted for an existing dataset). Note: Before indexing, please call /get_upload_data_status to confirm that the data insertion has finished. Otherwise, indexing an actively incrementing dataset may cause unexpected behavior.

4. Conduct approximate nearest neighbor search using /search. Note: Before search, please call /get_attach_index_status to confirm the index construction has finished. Otherwise, search might not return full results.

The following is an example code to demonstrate the workflow. The code is in Python, Javascript, and cURL.

PythonJavaScriptcURL

import requests
import json
import numpy as np
import time

# Configuration
API_KEY = "replace_this_with_your_api_key"
BASE_URL = "https://aidb.vecml.com/api"

def make_request(endpoint, data):            # helper function to make API calls
    url = f"{BASE_URL}/{endpoint}"
    response = requests.post(url, json=data)
    print(f"Request to {endpoint}: HTTP {response.status_code}")
    if not response.text:
        return response.status_code, None
    try:
        return response.status_code, response.json()
    except ValueError:                       # many error responses are plain text
        print(f"Response: {response.text}")
        return response.status_code, {"error": response.text}

# 1. Create a project
project_data = {"user_api_key": API_KEY, "project_name": "RAG-example", "application": "search"}
status, response = make_request("create_project", project_data)

# 2. Initialize dataset
init_data = {"user_api_key": API_KEY, "project_name": "RAG-example", "collection_name": "embeddings",
             "vector_type": "dense", "vector_dim": 128 }
status, response = make_request("init", init_data)

# 3. Generate and add 100 random vectors with /add_data_batch
vectors = np.random.randn(100, 128).tolist()                             # some random vectors
string_ids = ['ID_' + str(i) for i in range(100)]                        # dummy ids for each vector
attributes = [{"text": f"example text {i}"} for i in range(100)]         # attributes of each vector, which can be arbitrary

batch_data = {"user_api_key": API_KEY, "project_name": "RAG-example", "collection_name": "embeddings",
             "string_ids": string_ids, "data": vectors, "attributes": attributes }
status, response = make_request("add_data_batch", batch_data)

# 4. Attach index with L2 distance
index_data = {"user_api_key": API_KEY, "project_name": "RAG-example", "collection_name": "embeddings",
              "index_name": "L2_index", "index_type": "standard", "dist_type": "L2" }
status, response = make_request("attach_index", index_data)
index_job_id = response["job_id"]
print(f"Got index job ID: {index_job_id}")

# 5. Checking the indexing job status every 1 sec, query after the job is done
print("Waiting for indexing to complete...")
max_wait_time = 20
start_time = time.time()
while True:
    status_data = {"user_api_key": API_KEY, "job_id": index_job_id}
    status, status_response = make_request("get_attach_index_status", status_data)

    if status_response.get("status") == "finished":
        print("Indexing completed successfully")
        break

    if time.time() - start_time > max_wait_time:
        print("Server busy - indexing timeout after 20 seconds")
        exit(1)

    print("Indexing in progress, checking again...")
    time.sleep(1)

# 6. Search with a random query vector
query_vectors = np.random.randn(2, 128).tolist()     # query vector
search_data = {
    "user_api_key": API_KEY, "project_name": "RAG-example", "collection_name": "embeddings",
    "index_name": "L2_index", "query_type": "vectors",
    "top_k": 5, "query_vectors": query_vectors
}
status, search_results = make_request("search", search_data)

print("Search results:")
for result in search_results["results"]:
    matches = result["matches"]
    query_id = result["query_vector_id"]

    print("Query id: ", query_id)
    for pair in matches:
      print(f"ID: {pair['idx']}, L2 Distance: {pair['distance']}")

const fetch = require('node-fetch');

// Configuration
const API_KEY = "replace_this_with_your_api_key";
const BASE_URL = "https://aidb.vecml.com/api";

// Helper function to make API calls
async function makeRequest(endpoint, data) {
    const url = `${BASE_URL}/${endpoint}`;
    const response = await fetch(url, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(data)
    });
    console.log(`Request to ${endpoint}: HTTP ${response.status}`);
    const text = await response.text();
    if (!text) {
        return [response.status, null];
    }
    try {
        return [response.status, JSON.parse(text)];
    } catch (e) {                          // many error responses are plain text
        console.log(`Response: ${text}`);
        return [response.status, { error: text }];
    }
}

// Main function to run the workflow
async function runWorkflow() {
    try {
        // 1. Create a project
        const projectData = { "user_api_key": API_KEY, "project_name": "RAG-example", "application": "search" };
        let [status, response] = await makeRequest("create_project", projectData);

        // 2. Initialize dataset
        const initData = { "user_api_key": API_KEY, "project_name": "RAG-example", "collection_name": "embeddings",
                           "vector_type": "dense", "vector_dim": 128 };
        [status, response] = await makeRequest("init", initData);

        // 3. Generate and add 100 random vectors with /add_data_batch
        //    (uniform in [-1, 1], but that's okay for demonstration)
        const vectors = Array(100).fill().map(() =>
            Array(128).fill().map(() => (Math.random() * 2 - 1))
        );
        const stringIds = Array(100).fill().map((_, i) => `ID_${i}`);
        const attributes = Array(100).fill().map((_, i) => ({"text": `example text ${i}`}));

        const batchData = { "user_api_key": API_KEY, "project_name": "RAG-example", "collection_name": "embeddings",
                            "string_ids": stringIds, "data": vectors, "attributes": attributes };
        [status, response] = await makeRequest("add_data_batch", batchData);

        // 4. Attach index with L2 distance
        const indexData = { "user_api_key": API_KEY, "project_name": "RAG-example", "collection_name": "embeddings",
                            "index_name": "L2_index", "index_type": "standard", "dist_type": "L2" };
        [status, response] = await makeRequest("attach_index", indexData);
        const indexJobId = response.job_id;
        console.log(`Got index job ID: ${indexJobId}`);

        // 5. Checking the indexing job status every 1 sec, query after the job is done
        console.log("Waiting for indexing to complete...");
        const maxWaitTime = 20; // seconds
        const startTime = Date.now();
        while (true) {
            const statusData = {
                "user_api_key": API_KEY,
                "job_id": indexJobId
            };
            [status, response] = await makeRequest("get_attach_index_status", statusData);

            if (response && response.status === "finished") {
                console.log("Indexing completed successfully");
                break;
            }
            if ((Date.now() - startTime) / 1000 > maxWaitTime) {
                console.log("Server busy - indexing timeout after 20 seconds");
                process.exit(1);
            }
            console.log("Indexing in progress, checking again...");
            await new Promise(resolve => setTimeout(resolve, 1000));
        }

        // 6. Search with a random query vector
        const queryVector = Array(128).fill().map(() => (Math.random() * 2 - 1));
        const searchData = { "user_api_key": API_KEY, "project_name": "RAG-example", "collection_name": "embeddings",
                             "index_name": "L2_index", "query_type": "vector", "top_k": 5, "query_vector": queryVector };
        let [_, searchResults] = await makeRequest("search", searchData);

        console.log("Search results:");
        // results holds one entry per query vector; a "vector" query has exactly one
        searchResults.results.forEach(result => {
            console.log(`Query id: ${result.query_vector_id}`);
            result.matches.forEach(match => {
                console.log(`ID: ${match.idx}, L2 Distance: ${match.distance}`);
            });
        });
    } catch (error) {
        console.error("Error in workflow:", error);
    }
}

runWorkflow();

#!/bin/bash

# Configuration
API_KEY="replace_this_with_your_api_key"
BASE_URL="https://aidb.vecml.com/api"

# Helper function to make requests
make_request() {
    local endpoint=$1
    local data=$2
    echo "Request to $endpoint"
    response=$(curl -s -X POST "$BASE_URL/$endpoint" \
        -H "Content-Type: application/json" \
        -d "$data")
    echo "$response"
    echo "$response" > last_response.json
}

# 1. Create a project
project_data='{ "user_api_key": "'"$API_KEY"'", "project_name": "RAG-example", "application": "search" }'
make_request "create_project" "$project_data"

# 2. Initialize dataset (8-dimensional)
init_data='{ "user_api_key": "'"$API_KEY"'", "project_name": "RAG-example", "collection_name": "embeddings",
             "vector_type": "dense", "vector_dim": 8 }'
make_request "init" "$init_data"

############################################
# 3. Add vectors (3 vectors, each 8D).
#    You can expand to more vectors if needed.
############################################

# Make sure each vector has exactly vector_dim (8) elements!

batch_data='{
  "user_api_key": "'"$API_KEY"'", "project_name": "RAG-example", "collection_name": "embeddings",
  "string_ids": ["ID_0", "ID_1", "ID_2"],
  "data": [
    [0.10, 0.20, 0.30, 0.40, 0.50, 0.60, 0.70, 0.80],
    [-0.10, -0.20, -0.30, -0.40, -0.50, -0.60, -0.70, -0.80],
    [0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.05, -0.05]
  ],
  "attributes": [
    { "text": "example text 0" },
    { "text": "example text 1" },
    { "text": "example text 2" }
  ]
}'
make_request "add_data_batch" "$batch_data"

# 4. Attach index with L2 distance
index_data='{ "user_api_key": "'"$API_KEY"'", "project_name": "RAG-example", "collection_name": "embeddings",
              "index_name": "L2_index", "index_type": "standard", "dist_type": "L2" }'
make_request "attach_index" "$index_data"

# Extract job ID from the response
index_job_id=$(grep -o '"job_id":"[^"]*"' last_response.json | cut -d'"' -f4)
echo "Got index job ID: $index_job_id"

# 5. Check indexing status with a 20-second timeout
echo "Waiting for indexing to complete..."
start_time=$(date +%s)
max_wait_time=20
while true; do
    status_data='{ "user_api_key": "'"$API_KEY"'", "job_id": "'"$index_job_id"'" }'
    make_request "get_attach_index_status" "$status_data"
    current_status=$(grep -o '"status":"[^"]*"' last_response.json | cut -d'"' -f4)
    if [ "$current_status" = "finished" ]; then
        echo "Indexing completed successfully"
        break
    fi
    current_time=$(date +%s)
    elapsed=$((current_time - start_time))
    if [ $elapsed -gt $max_wait_time ]; then
        echo "Server busy - indexing timeout after $max_wait_time seconds"
        exit 1
    fi
    echo "Indexing in progress, checking again..."
    sleep 1
done

# 6. Search with an 8D query vector
search_data='{ "user_api_key": "'"$API_KEY"'", "project_name": "RAG-example", "collection_name": "embeddings",
               "index_name": "L2_index", "query_type": "vector", "top_k": 5,
               "query_vector": [0.12, 0.20, 0.33, -0.15, 0.40, 0.11, 0.05, 0.06] }'
make_request "search" "$search_data"

# Parse and display search results from last_response.json
echo "Search results:"
# Extract all IDs and distances into arrays
ids=($(grep -o '"idx":"[^"]*"' last_response.json | cut -d'"' -f4))
distances=($(grep -o '"distance":[0-9.\-e]*' last_response.json | cut -d':' -f2))

# Process them using array indexing
for i in ${!ids[@]}; do
    echo "ID: ${ids[$i]}, L2 Distance: ${distances[$i]}"
done

---

Project Management

/create_project

Description: Creates a new project for a given user.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "application": "string" // e.g., "autoML", "both", or any descriptive text
}

• All three fields are required; a missing field returns 400.
• The project_name must be unique for the user (an existing name returns 400 Project already exists!), non-empty, at most 128 characters, and must not contain any of \ / : * ? " ' < > | or control characters. Names that consist only of dots (., ..) and the reserved names project_infos.txt and dataset_infos.txt are rejected with 400 as well.
• application describes how the project will be used: search, autoML, or both. AutoML feature augmentation is only applied in projects whose application is autoML or both.

Response

{
  "success": true,
  "error_code": "Success"
}

If the project could not be created, success is false and error_code names the error.

Example

POST /create_project
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "application": "search"
}

Response:

{
  "success": true,
  "error_code": "Success"
}

---

/delete_project

Description: Deletes an existing project and all of its datasets. This action cannot be undone.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string"
}

Response

{
  "success": true
}

On failure the HTTP status is still 200 and the body is {"success": false, "error_code": "..."}. A project_name that does not name a directory inside the user's own directory (for example "", .. or ../x) or that is a reserved name (project_infos.txt, dataset_infos.txt) returns {"success": false, "error_code": "InvalidParameter"}, and nothing is deleted.

Example

POST /delete_project
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA"
}

---

/fetch_projects

Description: Returns a list of all project(s) belonging to a user.

Method - POST

Request Body

{
  "user_api_key": "string"
}

Response

{
  "success": true,
  "projects": [
    {
      "project_name": "...",
      "project_id": "...",
      "application":  "...",
      "num_of_datasets": 0
    },
    ...
  ]
}

project_id is reserved and currently always an empty string. num_of_datasets is the number of datasets in the project.

Example

POST /fetch_projects
Content-Type: application/json

{
  "user_api_key": "api_key_123"
}

Response:

{
  "success": true,
  "projects": [
    {
      "project_name": "ProjectA",
      "project_id": "",
      "application": "search",
      "num_of_datasets": 2
    },
    {
      "project_name": "ProjectB",
      "project_id": "",
      "application": "autoML",
      "num_of_datasets": 1
    }
  ]
}

---

/list_all_items

Description: Retrieves a complete hierarchical structure of all projects, datasets, and indices (including vector indices and text field indices) for a user. Returns organized data showing the relationship between projects, their datasets, and associated indices.

Method - POST

Request Body

{
  "user_api_key": "string"
}

Request Fields - user_api_key: Valid API key for authentication

Response Success Response (200 OK):

{
  "success": true,
  "projects": [
    {
      "project_name": "string",
      "project_id": "string", 
      "application": "string",
      "datasets": [
        {
          "dataset_name": "string",
          "type": "string",
          "size": 1024,
          "create_time": "string",
          "last_insert_time": "string",
          "dimension": "(1000, 128)",
          "indices": [
            {
              "index_name": "string",
              "distance_type": "string",
              "type": "string"
            }
          ],
          "text_field_indices": [
            {"attribute_name": "string"}
          ]
        }
      ]
    }
  ]
}

Response Fields

• success: Boolean indicating operation success
• projects: Array of project objects containing:
• project_name: Name of the project
• project_id: Reserved; currently always an empty string
• application: Application type or category

• datasets: Array of dataset objects containing:

  • dataset_name: Dataset (collection) name
  • type: Vector type of the dataset: "dense", "sparse", "dense8Bit", "dense4Bit", "dense2Bit" or "dense1Bit"
  • size: On-disk size of the dataset's vectors and attributes, in bytes
  • create_time: When the dataset was created, formatted YYYY/MM/DD, HH:MM:SS
  • last_insert_time: Last time data was inserted (same format), or "None" if nothing was inserted yet
  • dimension: String of the form "(num_vectors, vector_dimension)"; see /fetch_datasets for how both numbers are counted

  • indices: Array of index objects containing:

    • index_name: Index name
    • distance_type: Distance metric: "Euclidean", "Cosine", "InnerProduct", "Manhattan" or "Hamming"
    • type: the type of index: "standard", "fast_indexing", "standard-large_index", or "unknown"

  • text_field_indices: Array of objects {"attribute_name": ...}, one per attribute with a text field index

Notes

• Returns complete hierarchy in a single request
• Useful for dashboard views and project overview
• Dataset size is returned in bytes; convert to KB/MB/GB as needed
• Empty arrays are returned for projects/datasets with no children
• Rate limiting applies to this endpoint

Example

POST /list_all_items
Content-Type: application/json

{
  "user_api_key": "api_key_123"
}

Response:

{
  "success": true,
  "projects": [
    {
      "project_name": "RecommendationEngine",
      "project_id": "",
      "application": "search",
      "datasets": [
        {
          "dataset_name": "user_embeddings",
          "type": "dense",
          "size": 2048576,
          "create_time": "2025/01/15, 10:30:00",
          "last_insert_time": "2025/01/20, 14:45:00",
          "dimension": "(10000, 128)",
          "indices": [
            {"index_name": "user_similarity_index", "distance_type": "Cosine", "type": "standard"}
          ],
          "text_field_indices": []
        },
        {
          "dataset_name": "item_features",
          "type": "dense",
          "size": 1572864,
          "create_time": "2025/01/15, 11:00:00",
          "last_insert_time": "2025/01/19, 09:30:00",
          "dimension": "(5000, 256)",
          "indices": [{"index_name": "index1", "distance_type": "Cosine", "type": "standard"},
                      {"index_name": "index2", "distance_type": "Euclidean", "type": "fast_indexing"}],
          "text_field_indices": [{"attribute_name": "text_chunk"},
                                 {"attribute_name": "title"}]
        }
      ]
    }
  ]
}

---

Dataset Management

/init

Description: Initializes a dataset in the user's project. If the dataset does not exist, it is created. If the dataset already exists and the specified parameters match the existing configuration, it materializes the dataset into memory.

Methods - POST (also accepted as GET with the same JSON body)

Request Body Parameters

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "vector_type": "string",   // "dense" or "sparse" (no other values supported)
  "vector_dim": int          // Required for "dense"; must be > 0
}

- user_api_key: The user's API key.

• project_name: The name of the project to which the dataset belongs. Must already exist.

• collection_name: The name of the dataset/collection to initialize. It must follow the same rules as in /upload (otherwise 400).

• vector_type: "dense" or "sparse" to create a dataset. Low-bit dense datasets ("dense8Bit", "dense4Bit", "dense2Bit", "dense1Bit") can only be created by /upload; an existing low-bit dataset can be re-initialized with its own type. Any other value, or a low-bit type for a dataset that does not exist, returns 400 Unsupported vector type.

• vector_dim: Positive integer, at most 2147483647 (a numeric string such as "128" is also accepted). A larger or non-integer JSON number returns 400 Invalid vector dimension, must be positive integer. Required when creating a "dense" dataset; a missing vector_dim returns 400. Not used for "sparse".

• If the dataset already exists, vector_type must match the existing dataset (otherwise 400). vector_dim is optional in this case; if it is given for a non-sparse dataset it must equal the dataset's dimension (the second number of dimension returned by /fetch_datasets), otherwise 400 is returned. For sparse datasets vector_dim is ignored.

Response Success Response (200 OK):

{
  "error_code": "Success",
  "success": true
}

Error Responses: 400 status code with a plain-text message.

Example

POST /init
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "vector_type": "dense",
  "vector_dim": 100
}

Response:

{
  "error_code": "Success",
  "success": true
}

or, code 400 with (an example failure message)

The dataset already exists, but the input vector dim does not match the existing vector dim: 128

---

/add_data_batch

Description: Inserts a batch of vectors. Expects arrays of string_ids, data, and (optionally) attributes with the same length. The dataset must already exist (create it with /init); otherwise 400 is returned. This is an asynchronous operation - use /get_upload_data_status to check job status.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "string_ids": [
    "id_1",
    "id_2",
    ...
  ],
  "data": [
    // for each index i, the vector in the dataset's format
    [0.15, -0.23, 0.44, 0.11],           // dense dataset: array of numbers
    [[0, 1.8], [4, 2.3], [53, -1.2]]     // sparse dataset: array of [index, value] pairs
  ],
  "attributes": [
    // array of objects, parallel to data
    {"attr1": "val1", "attr3": "val3", ...},
    {"attr2": "val2", ...},
    ...
  ],
  "checksum": "SHA256(data)",         // optional data integrity check
  "global_indices": [0, 1, 2, ...]    // optional, for distributed systems
}

• string_ids and data are required arrays (otherwise 400).

• attributes is optional. The keys are attribute names and the values should be strings, for example

  • {"age": "19"}
  • {"weight": "175.6"}
  • {"Title": "The Kite Runner"}
  • Each vector can have different attribute fields
  • Non-string values are stored as their JSON text (e.g. 19 is stored as "19").
  • Numeric-looking values are stored in a canonical form: a leading + and trailing zeros after the decimal point are removed, e.g. "+1.200" is stored as "1.2" and "2.0" as "2". Use the canonical form when filtering on such values.

• checksum is optional. It should be the SHA256 hash of the data field in string format.

• global_indices is optional. Used for distributed indexing to specify global vector IDs.

• All arrays must have the same length (otherwise 400). The i-th entry in string_ids matches the i-th entry in data and attributes. An attributes entry that is not a JSON object is ignored.

• A dataset can hold at most 1,000,000 vectors, counting removed vectors too. A request that would exceed this returns 400.

• Vector dimensions: once the dataset holds vectors, each non-sparse vector must have the dataset's dimension. In a request to an empty dataset, every non-sparse vector must have the dimension of the request's first vector. Sparse vectors have no fixed dimension (the dataset's dimension is the largest index used so far plus one).

• Element values: for dense8Bit / dense4Bit / dense2Bit / dense1Bit datasets every element must be an integer in [0, 255] / [0, 15] / [0, 3] / [0, 1]; sparse indices must be integers in [0, 4294967294]. Other values make the vector malformed (see the job status below).

• A string ID that was removed with /remove_data cannot be inserted again: the insert is reported with VectorAlreadyExists for that ID.

• If checksum is given and does not match the server's checksum of data, the request is rejected with 400.

Response

{
  "success": true,
  "job_id": "string",
  "error_message": "none",
  "checksum_server": "string"
}

The checksum_server is the SHA256 of the data field (compact JSON) computed on the server side; it is always returned so the client can verify data integrity.

Job Status Response (via /get_upload_data_status)

{
  "success": true,
  "status": "finished",               // "pending", "in_progress", "finished", or "failed"
  "start_time": "2024/12/04, 10:30:00",
  "end_time": "2024/12/04, 10:30:15",
  "duration": "15 seconds",
  "error_message": ""                 // Empty on success, contains details otherwise
}

A job whose vectors were all processed reports "finished", even if some vectors were rejected; the rejected ones are listed in error_message:

"error_message": "The following vectors failed to be inserted: [{\"body\":\"Failed to add vector: VectorAlreadyExists\",\"string_id\":\"id_1\"}, ...]"

If an entry of data or attributes is malformed (wrong format or dimension, a low-bit element or sparse index outside its range), or global_indices does not have the same length as string_ids, processing stops at that batch: the job still reports "finished", the error is reported as {"error": "Batch processing failed", "body": "..."} in error_message, and vectors of earlier batches (of 1000) remain inserted and are counted in /fetch_datasets. An element of a dense vector (or a sparse value) that is not a number makes the job report "failed" with "Job failed with error: ...". The job also reports "failed" if the data could not be persisted ("Data added, but failed during flush.").

NOTE: The server computes the checksum in compact JSON format. For data integrity check to work properly, on the user side, the checksum field should also be computed on data field in compact JSON format, i.e., there should be no spaces after colons or commas.

Example

POST /add_data_batch
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "BatchDS",
  "string_ids": ["row1", "row2"],
  "data": [
    [0.25, 0.75],
    [0.6, 0.3]
  ],
  "attributes": [
    {"date": "2024-01-01", "category": "A"},
    {"author": "John", "source": "wikipedia"}
  ]
}

Response Example

{
  "success": true,
  "job_id": "<user_id>||ProjectA||BatchDS||InsertDataJob||550e8400-e29b-41d4-a716-446655440000",
  "error_message": "none",
  "checksum_server": "a3c5f7..."
}

---

/add_vector_attributes

Description: Adds or updates attributes for existing vectors without modifying the vector data itself. Use this endpoint to enrich vectors with metadata after they have been inserted. Each vector can have a different set of attribute fields. This is an asynchronous operation - use /get_upload_data_status to check job status.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "string_ids": [
    "id_1",
    "id_2",
    ...
  ],
  "attributes": [
    {"attr1": "val1", "attr2": "val2", ...},
    {"attr3": "val3", "attr4": "val4", ...},
    ...
  ]
}

• string_ids: Array of existing vector IDs to update (required)
• attributes: Array of attribute objects, parallel to string_ids (required, same length)
• Each vector is matched by its string_id; the order of the arrays does not matter. If an ID appears more than once, its last entry is used.
• The keys are attribute names and the values should be strings; non-string values are stored as their JSON text, and empty strings are ignored
• Each vector can have completely different attribute fields
• Existing attributes will be overwritten; new attributes will be added

Response

{
  "success": true,
  "job_id": "string",
  "error_message": "none"
}

Job Status Response (via /get_upload_data_status)

{
  "success": true,
  "status": "finished",               // "pending", "in_progress", "finished", or "failed"
  "start_time": "2024/12/04, 10:30:00",
  "end_time": "2024/12/04, 10:30:15",
  "duration": "15 seconds",
  "error_message": ""                 // Empty on success, contains details otherwise
}

If some vectors could not be updated, the job still reports "finished" and error_message lists them:

"error_message": "The following vectors failed to update attributes: [{\"error\":\"Vector not found with string_id: id_9\",\"string_id\":\"id_9\"}, ...]"

Error Cases: - Vector with string_id does not exist: reported for that ID in the job's error_message; the other vectors are still updated - Missing fields, or string_ids / attributes not arrays: 400 - Mismatched array sizes: 400 - Project does not exist: 400 - Dataset does not exist: 400 Dataset does not exist! - Invalid API key: 401 - Failure to persist the update: the job reports "failed" with "Attributes updated, but failed during flush."

Example

POST /add_vector_attributes
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "documents",
  "string_ids": ["doc_1", "doc_2"],
  "attributes": [
    {
      "title": "Introduction to AI",
      "author": "John Doe",
      "date": "2024-01-15",
      "category": "technology"
    },
    {
      "product_name": "Wireless Mouse",
      "price": "29.99",
      "in_stock": "true"
    }
  ]
}

Response Example

{
  "success": true,
  "job_id": "<user_id>||ProjectA||documents||AddAttributesJob||550e8400-e29b-41d4-a716-446655440000",
  "error_message": "none"
}

---

/upload

Description: Uploads a file containing dataset vectors (CSV, JSON, libsvm or binary format) and creates a new dataset with those vectors, or appends them to an existing dataset of the same name (if that dataset already holds vectors, the vector type and, except for sparse datasets, the dimension must match it). The upload runs asynchronously; you receive a job_id to query via /get_upload_data_status. /upload can be directly called without /init.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",   // The new dataset's name
  "file_data": "string",   // base64-encoded data (optionally gzip compressed)
  "file_format": "string", // "csv", "json", "libsvm", "binary"
  "has_field_names": true | false,   // required for every file_format
  "vector_type": "dense" | "dense1Bit" | "dense2Bit" | "dense4Bit" | "dense8Bit" | "sparse",

  // If JSON with field names:
  "vector_data_field_name": "string",

  // If JSON or CSV with field names:
  "vector_id_field_name": "string",
  "vector_attributes": ["attr1", "attr2", ...],

  // If "binary" file_format:
  "binary_dtype": "uint8" | "float32",   // the dataset type follows binary_dtype: "float32" -> dense, "uint8" -> dense8Bit
  "binary_dim": 123,

  // Optional string ID txt file
  "string_id_txt": "string",     // base64-encoded external string ID .txt file, optional

  // Optional compression method
  "compression_type": "string",    // now only "gzip" is supported

  // Optional checksum for data integrity check
  "checksum": "string"           // this should be the SHA256 hash computed on "file_data" field        
}

Required:

• user_api_key: Your VecML API key.
• project_name: The project that the uploaded dataset belongs to (must exist).
• collection_name: The name of the data collection in VecML DB (non-empty, at most 128 characters, none of \ / : * ? " ' < > | and no control characters; names that consist only of dots, such as . and .., and the reserved names project_infos.txt and dataset_infos.txt are rejected with 400).
• file_data: The Base64 encoded raw data file.
• file_format: "csv", "json", "libsvm" or "binary". Any other value makes the server detect the format from the content; a file detected as JSON Lines is not supported by /upload and the job fails.
• has_field_names (boolean, required for every format, 400 if missing or not a boolean): whether the data file contains column headers (csv) or field names (json). Pass false for libsvm and binary files.
• vector_type: the type of the vector (400 Invalid vector_type! otherwise). Supported types:
  • dense: the standard float32 dense vector. For example, [0, 1.2, 2.4, -10 ,5.7]. Standard embedding vectors from language or vision models can be saved as this type.
  • dense8Bit: uint8 dense vectors, with integer vector elements ranging in [0, 255]. For example, [0, 3, 76, 255, 152]. 8-bit quantized embedding vectors can be saved as this type for storage saving.
  • dense4Bit: 4-bit quantized dense vectors, with integer vector elements ranging in [0, 15].
  • dense2Bit: 2-bit quantized dense vectors, with integer vector elements ranging in [0, 3].
  • dense1Bit: 1-bit quantized dense vectors, with binary vector elements.
  • sparse: sparse vector formatted as a set of index:value pairs. Please use this for libsvm file format. This is useful for high-dimensional sparse vectors.

Conditional:

• For csv and json format data:
  • vector_data_field_name (required if file_format == json and has_field_names == true, otherwise 400): the json field that contains the vector data.
  • vector_attributes (optional, csv and json only): [attr1, attr2, ..., attrN], List of attribute columns/fields associated with the file's vectors. For json files without field names it is ignored.
  • vector_id_field_name (optional, only read when has_field_names == true): Specifies the column/field that should be used as the unique vector IDs of the vectors.
• For libsvm data, the label of each line is stored as an attribute named "label".
• csv and libsvm files are converted to ASCII before parsing: accented Latin letters are transliterated ("café" becomes "cafe"), typographic quotes, dashes and … become their ASCII equivalents, and other non-ASCII characters become ?.
• For binary data:
  • binary_dtype (required): "float32" (dataset type dense) or "uint8" (dataset type dense8Bit). vector_type is not used for binary files.
  • binary_dim (required): the vector dimension, a positive integer. The file is the concatenation of all vectors.
  • An invalid binary_dtype or binary_dim makes the request fail with an HTTP 500 error. Always provide both for binary files.

Optional:

• string_id_txt: A Base64 encoded white-space or newline separated .txt file that specifies the UNIQUE string IDs of the vectors, in file order. If compression_type is "gzip", this file must be gzip-compressed too. When the number of IDs equals the number of vectors in the data file, these IDs override all other ID sources; otherwise they are ignored, the default IDs are used, and the job's error_message says so.

Auto-generation of string ID and column names (for CSV and JSON)

For libsvm and binary data, we will automatically generate numeric IDs for the vectors. For csv and json, if has_field_names == False or vector_id_field_name is not provided, the IDs are also auto-generated. Auto-generated IDs are "0", "1", ... for the first upload into an empty dataset; later uploads continue from the dataset's current vector count (removed vectors included), e.g. the second upload of a 1000-row dataset starts at "1000".

Moreover, for csv files, if has_field_names == False, the attribute columns can still be specified by their 1-based column number, such as "vector_attributes": ["column 59", "column 60"] (the number is taken from the digits in the name). A name without digits does not select a column and is skipped; a number outside the file's columns makes the upload job fail. The ID column cannot be selected this way; IDs are auto-generated.

• compression_type: to further reduce the file size and speed up communication, the user can gzip the file before converting to Base64 format. If gzip is applied, set this argument to "gzip". Currently, only "gzip" is supported.

• checksum: it is the SHA256 hash of the file_data field in string format. If it does not match the server's hash, the request is rejected with 400.

Response

{
  "success": true,
  "job_id": "string",
  "error_message": "none",
  "checksum_server": "string"
}

The checksum_server (SHA256 of the file_data field computed on the server side) is always returned.

Job status (via /get_upload_data_status): on success the job reports "finished" with error_message "Upload completed successfully." (plus a note about string_id_txt when one was given). The job reports "failed" with the reason in error_message when the file format is not supported, the file cannot be parsed, a value of a low-bit type (dense8Bit / dense4Bit / dense2Bit / dense1Bit) in a csv or json file is not an integer in the type's range (e.g. Invalid value 300 for vector type dense8Bit at line 2, column 3: values must be integers in [0, 255]), the dimension (not checked for sparse datasets) or vector type does not match an existing dataset, the dataset would exceed 1,000,000 vectors, the vectors cannot be added ("Error when adding vectors to data collection."), or the data cannot be persisted. Individual vectors rejected during insertion (for example because their ID already exists) do not fail the job: it reports "finished", and error_message ends with The following vectors failed to be inserted: [{"string_id": "...", "body": "..."}, ...]. Rejected vectors are not counted in /fetch_datasets.

Example

POST /upload
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "file_data": "BASE64_ENCODED_DATA==",
  "file_format": "csv",
  "has_field_names": true,
  "vector_type": "dense",
  "vector_id_field_name": "unique_ID",
  "vector_attributes": ["date", "region", "category"]
}

Response:

{
  "success": true,
  "job_id": "<user_id>||ProjectA||MyDataset||InsertDataJob||550e8400-e29b-41d4-a716-446655440000",
  "error_message": "none",
  "checksum_server": "9f2c..."
}

---

/add_vector_attributes_from_file

Description: Adds one attribute to the vectors of an existing dataset by uploading a file containing vector IDs and their corresponding values (for example labels). The upload runs asynchronously; you receive a job_id to query via /get_upload_data_status. (The AutoML endpoint /attach_automl_label does the same and accepts label_name in place of attribute_name.)

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",   // The existing dataset's name
  "file_data": "string",   // base64-encoded attribute file (optionally gzip compressed)
  "attribute_name": "string",   // The name of the attribute to be added

  // Optional compression method
  "compression_type": "string"    // now only "gzip" is supported
}

Required:

• user_api_key: Your VecML API key for authentication.
• project_name: The project that contains the dataset.
• collection_name: The name of the existing data collection in VecML DB.
• file_data: The Base64 encoded attribute file: a plain text (or csv) file with one id,value pair per line, separated by a comma, tab or space (the separator is detected from the first line). It also accepts a single-column file with one value per line; the values are then assigned to the vectors whose string IDs are "0", "1", "2", ... in line order. This matches the auto-generated IDs of a dataset created by a single /upload; for datasets with custom IDs or several uploads (later uploads continue numbering from the existing vector count), use the two-column format. A value containing the separator character is not supported. Non-ASCII characters are converted to ASCII: accented Latin letters are transliterated ("café" becomes "cafe"), typographic quotes, dashes and … become their ASCII equivalents, and other characters become ?. Numeric-looking values are stored in canonical form (see /add_data_batch).
• attribute_name: The name to assign to this attribute in the dataset. A missing field returns 400.

Optional:

• compression_type: To further reduce the file size and speed up communication, the user can gzip the file before converting to Base64 format. If gzip is applied, set this argument to "gzip". Currently, only "gzip" is supported.

Response

{
  "success": true,
  "job_id": "string",
  "error_message": "none",
  "num_vectors_parsed": 123
}

• job_id: Use this ID to query the status of the upload job via /get_upload_data_status. IDs in the file that do not exist in the dataset are listed in the job's error_message; the other vectors are still updated.
• num_vectors_parsed: The number of ID-value pairs successfully parsed from the uploaded file.

A file that cannot be decoded or parsed is rejected with 400 and a plain-text message, as is an unknown project (Project does not exist!) or dataset (Dataset does not exist!).

Example

POST /add_vector_attributes_from_file
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "file_data": "BASE64_ENCODED_ATTRIBUTE_DATA==",
  "attribute_name": "sentiment",
  "compression_type": "gzip"
}

Response:

{
  "success": true,
  "job_id": "<user_id>||ProjectA||MyDataset||AddAttributesFromFileJob||550e8400-e29b-41d4-a716-446655440000",
  "error_message": "none",
  "num_vectors_parsed": 1000
}

---

/remove_data

Description: Remove vectors from the dataset and its indices based on a list of string ids.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "string_ids": ["id1", "id2"]
}

- string_ids must be an array of IDs (a single ID must also be passed as an array). - The project and dataset must exist (otherwise 400). - A removed string ID cannot be inserted into the dataset again: /add_data_batch and /upload report VectorAlreadyExists for that ID.

Response Success Response (200 OK):

{
  "error_code": "Success",        // or "Failed" if any ID could not be removed
  "error_message": "All deleteions succeeded."
}

If any removal fails, error_code is "Failed" and error_message lists each failed ID with its error, starting with The following deletions failed:. IDs that were removed successfully stay removed.

Example

POST /remove_data
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "string_ids": ["Sample_1", "Sample_2"]
}

---

/delete_dataset

Description: Deletes an existing dataset from a project. This action cannot be undone.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string"
}

Response

{
  "success": true
}

A project that does not exist returns 400 (plain text). Other failures return HTTP 200 with {"success": false, "error_code": "..."}. A collection_name that does not name a directory inside the project's directory (for example "", .. or ../x) or that is a reserved name (project_infos.txt, dataset_infos.txt) returns {"success": false, "error_code": "InvalidParameter"}, and nothing is deleted.

Example

POST /delete_dataset
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "ObsoleteData"
}

---

/fetch_datasets

Description: Fetches metadata for all datasets within a given project.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string"
}

Response

{
  "success": true,
  "datasets": [
    {
      "dataset_name": "...",
      "type": "dense",
      "size": 1234567,
      "create_time": "2025/01/02, 15:30:00",
      "last_insert_time": "2025/01/03, 09:12:45",
      "dimension": "(1000, 128)"
    },
    ...
  ]
}

• dataset_name: Dataset (collection) name
• type: Vector type of the dataset ("dense", "sparse", "dense8Bit", "dense4Bit", "dense2Bit" or "dense1Bit")
• size: On-disk size of the dataset's vectors and attributes, in bytes
• create_time / last_insert_time: formatted YYYY/MM/DD, HH:MM:SS; last_insert_time is "None" until data is inserted
• dimension: String of the form "(num_vectors, vector_dimension)". num_vectors is a running count of inserted rows: it is not decreased by /remove_data, and for /upload it counts the rows that were inserted. vector_dimension is the dataset's current dimension (for sparse datasets, the largest index used plus one).

The project must exist (otherwise 400 Project does not exist!).

Example

POST /fetch_datasets
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA"
}

---

Index and Query

/attach_index

Description: Creates an ANN (Approximate Nearest Neighbor) index for the given dataset. The indexing job runs asynchronously. You receive a job_id for tracking progress via /get_attach_index_status. Once attached, whenever new data are added to or removed from the data collection, the index will be updated automatically.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "index_name": "string",     // must be a valid name, <=128 chars, no invalid symbols
  "index_type" : "string",    // "standard" or "fast_indexing"
  "dist_type": "string",      // e.g., "L2", "Cosine", "Inner Product", "Hamming"
  "large_index" : bool        // optional
}

• user_api_key, project_name, collection_name, index_name, index_type and dist_type are all required; a missing field returns 400.

• index_name: non-empty, at most 128 characters, none of \ / : * ? " ' < > | and no control characters, and not already used by an index of the dataset (400 Index with same name already exists!) or by an AutoML or clustering model of the dataset (400 The name is already used by a model in this dataset!). Names that consist only of dots (., ..) and names reserved for the dataset's own files (vector_collection, vector_collection_temp, vector_id_manager, BCWS, interface_metadata.txt, index_metadata.txt, ml_model_metadata.txt, clustering_model_metadata.txt, text_field_index_metadata.txt, bcws_job_status.txt, and any name ending in _text_field_iindex) are rejected with 400.

• index_type: standard or fast_indexing (any other value returns 400). Standard index has moderate indexing time and extremely fast query speed; it supports all vector types. Fast index builds the index very efficiently, but query speed is slower; it only supports dense datasets (other types return 400) with Euclidean or Cosine distance (another metric is accepted with a job_id, but the indexing job then fails). Choose based on the application scenario.

• dist_type: accepted spellings are "L2", "l2", "Euclidean", "euclidean" (Euclidean distance); "L1", "l1", "Manhattan", "manhattan"; "Cosine", "cosine"; "Inner Product", "inner product", "InnerProduct"; "Hamming", "hamming". Any other value is not rejected: the index silently uses inner product. Check the spelling.

  • Hamming only works on low-bit dense datasets (dense8Bit, dense4Bit, dense2Bit, dense1Bit); on dense or sparse datasets the indexing job fails.
  • Manhattan builds on low-bit dense datasets, but searching such an index fails with HTTP 500.

• large_index: (JSON boolean, optional, only for index_type = standard) whether to use advanced approach to optimize the performance for very large standard index. For large indices, we provide techniques to balance the query efficiency, accuracy, and memory usage. It is highly recommended to set large_index to true if index_type = standard and your data collection has more than ~2M vectors. It requires at least 500,000 vectors; with fewer, the indexing job fails with "Attach index failed with error NotEnoughData".

• The dataset must exist and be loaded on the server; otherwise 400 (This collection hasn't been initialized with POST or GET /init means: call /init first).

Response

{
  "job_id": "string",
  "success": true,
  "error_code": "Success"
}

• job_id can be passed to /get_attach_index_status to see if the index build is finished. If the build fails, the job status is "failed" and its error_message gives the error.

Example

POST /attach_index
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "index_name": "index1",
  "index_type": "standard",
  "dist_type": "Cosine"
}

Response:

{
  "job_id": "<user_id>||ProjectA||MyDataset||index1||AttachIndexJob",
  "success": true,
  "error_code": "Success"
}

---

/delete_index

Description: Deletes a specific index from the dataset. This action cannot be undone.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "index_name": "string"
}

Response

{
  "success": true
}

The project and dataset must exist (otherwise 400). If the index cannot be deleted (e.g. it does not exist), the HTTP status is still 200 and the body is {"success": false, "error_code": "..."}.

Example

POST /delete_index
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "index_name": "index1"
}

---

/search

Description: Performs a nearest-neighbor vector search using a built index, with optional filter conditions. This endpoint supports:

• Searching with a single raw query vector ("vector") or a list of them ("vectors"), dense or sparse.
• Searching by referencing an existing row in the dataset (a row_number).
• Searching via a file containing multiple vectors to be searched. Please make sure your file have the same format (vector dimension, attributes) as the database dataset.
• Searching an entire dataset (query_collection_name) for batch queries.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "query_type": "vector" | "vectors" | "row" | "file" | "dataset",
  "index_name": "string",
  "top_k": int,                 // optional, default 5
  "filter": {
     // optional filter structure, see below
  },

  // If query_type == "vector":
  "query_vector": [0.1, -0.2, 0.3],                       // dense dataset
                  [[0, 0.1], [4, 5.2]],                   // sparse dataset: [index, value] pairs
  "string_id": "string",                                  // optional, returned as query_vector_id

  // If query_type == "vectors":
  "query_vectors": [[0.1, -0.2, 0.3], [0, 1, 10]],        // dense dataset
                   [[[0, 0.1], [4, 5.2]], [[2, 0], [9, 0.3], [20, 0.4]]],   // sparse dataset
  "string_ids": ["q1", "q2"],                             // optional, returned as query_vector_id (must match the number of query vectors)

  // If query_type == "row":
  "row_number": 42,

  // If query_type == "file":
  "file_data": "base64-encoded data",
  "file_name": "string",
  "file_format": "json" | "csv" | "libsvm" | "binary",
  // server uses dataset config to parse (field names, etc.)

  // If query_type == "dataset":
  "query_collection_name": "string"
}

• user_api_key, project_name, collection_name, query_type and index_name are required. The project, dataset and index must exist (otherwise 400); any other query_type returns 400 Invalid query_type.
• Query vectors must match the dataset: the same dimension for dense datasets (otherwise 400), integer elements in [0, 255] / [0, 15] / [0, 3] / [0, 1] for dense8Bit / dense4Bit / dense2Bit / dense1Bit datasets, and [index, value] pairs with integer indices in [0, 4294967294] for sparse datasets. For "vector" and "vectors", a low-bit element or sparse index outside its range returns 400.
• top_k: number of neighbors per query, default 5. It must be a JSON integer within the 32-bit signed range (otherwise 400 Invalid top_k! Must be an integer between -2147483648 and 2147483647.). A value <= 0 returns {"error_code": "InvalidData"} for "vector" and "row"; for "vectors", "file" and "dataset" every query fails, so results is empty while error_code is "Success".
• row_number: the 0-based internal position of a vector in the dataset (insertion order). It must be a JSON integer between 0 and 4294967294 (otherwise 400 Invalid row_number! Must be an integer between 0 and 4294967294.); a number beyond the dataset's last row returns 400 Invalid row number, failed to retrieve vector.
• file_data / file_name / file_format (all required for "file"): file_name is not used otherwise. CSV and JSON files are parsed with the dataset's own has_field_names setting (and field names, for files with headers). For a CSV file without a header, the dataset's "column N" attributes are read from column N; attribute names without digits (e.g. a label added later) are not columns and are skipped, and a "column N" beyond the file's columns returns 400 (Error processing file: Field "column N" refers to column N, but the file has M columns). The server does not record which attributes came from the uploaded file's columns, so every attribute name that contains digits is read as a column number, including one added later (for example score2 added by /add_data_batch is read as column 2). On such a dataset a CSV query file without a header fails with a dimension mismatch; use a "binary" or JSON query file instead. A "binary" file is a concatenation of vectors of the dataset's dimension, float32 for dense datasets and uint8 for dense8Bit datasets (other dataset types return 400). A "libsvm" file can only be used with sparse datasets (400 otherwise).
• query_collection_name: another dataset in the same project; each of its vectors is used as a query.

Filter Structure

"filter": {
  "type": "equality" | "range",
  "attribute": "attrName",

  // equality case:
  "target_values": ["someValue1", "someValue2", ...]

  // or range case:
  "start_value": "A",
  "end_value": "Z"
}

• No filtering is applied when filter is absent or null, when attribute is "", or when an equality filter has an empty target_values array.
• Otherwise type and attribute must both be present, as well as target_values for an equality filter; a filter missing one of them is rejected with HTTP 400 (Invalid request field: ...). For a range filter, a missing (or "") start_value or end_value leaves that side open; with both missing or both "", the filter matches every vector that has the attribute. A type other than "equality" or "range" (spelling is case-sensitive) matches nothing, so the search returns no results.
• An equality filter returns items whose attribute value is exactly one of the target values (string comparison; see /add_data_batch for how numeric-looking values are stored).
• A range filter returns items whose attribute is between start_value and end_value (inclusive). An empty or missing start_value or end_value leaves that side open (so does a bound that is not a JSON string, e.g. 5 instead of "5"). If the attribute value and the given bounds all parse as numbers, they are compared numerically; otherwise lexical (string) ordering is used.
• Vectors without the attribute never match.

Response

Each match has idx (the string ID of the matched vector) and distance. Matches are ordered from the nearest (most similar) to the farthest; rely on this order to rank results. The meaning of distance depends on the index's distance type, which is returned in distance_type:

• "Euclidean": the Euclidean (L2) distance; smaller is closer.
• "Cosine" / "InnerProduct": the absolute value of the cosine similarity / inner product. For positive similarities larger means closer, but because the sign is dropped, a negative similarity (e.g. -0.9) is reported as 0.9; use the returned order rather than the value.

• "Manhattan": the L1 distance; "Hamming": the Hamming distance; smaller is closer.

• For query_type: "row":

  {
    "error_code": "Success",
    "results": [
      {"idx": "string_id_of_match", "distance": float},
      ...
    ],
    "distance_type": "Euclidean"
  }
  

• For query_type: "vector", "vectors", "dataset" or "file":

  {
    "error_code": "Success",
    "results": [
      {
        "query_vector_id": "...",
        "matches": [
          {"idx": "string_id_of_match", "distance": float},
          ...
        ]
      },
      ...
    ],
    "distance_type": "Euclidean"
  }
  

  query_vector_id is the string_id / string_ids entry if given (otherwise, or if string_ids has a different length than query_vectors, "0", "1", ... by position) for "vector" and "vectors"; for "file", the value of the dataset's ID column for CSV files when the dataset was uploaded with vector_id_field_name, otherwise the row number "0", "1", ...; and the vector's string ID for "dataset".

• If the search fails, the HTTP status is 200 and error_code names the error (for "vector" and "row"). For "vectors", "file" and "dataset", a query that fails is left out of results, so results can have fewer entries than there were queries.

• Searching a dataset that has no vectors yet returns HTTP 200 with error_code "Success" and no matches, without parsing the queries: for "vector", one result with empty matches; for "vectors", one result with empty matches per entry of query_vectors (query_vector_id as described above); for "file" and "dataset" (and, in this case, an unrecognized query_type too), "results": []. "row" still returns an error, since there is no row to search by.

Examples

• Query with a direct vector

  POST /search
  Content-Type: application/json
  
  {
    "user_api_key": "api_key_123",
    "project_name": "ProjectA",
    "collection_name": "MyDataset",
    "query_type": "vectors",
    "index_name": "index1",
    "top_k": 5,
    "query_vectors": [[0.25, 0.75, 1.25, 2.00]],       // use one vector in the example
    "filter": {
      "type": "equality",
      "attribute": "Category",
      "target_values": ["CatA","CatC"]
    }
  }
  

  Response:

  {
    "error_code": "Success",
    "results": [
      {
        "query_vector_id": "0",
        "matches": [
          {"idx": "Sample_01", "distance": 0.3},
          {"idx": "Sample_07", "distance": 0.8},
          ...
        ]
      }
    ],
    "distance_type": "Euclidean"
  }
  

• Query referencing an existing row

  {
    "query_type": "row",
    "row_number": 123,
    ...
  }
  

• Batch file-based query

  {
    "query_type": "file",
    "file_data": "base64encodedFile",
    "file_name": "Queries.json",
    "file_format": "json",
    ...
  }
  

• Dataset-to-dataset query

  {
    "query_type": "dataset",
    "query_collection_name": "QueryVectorsDS",
    ...
  }
  

---

/list_index_infos

Description: Lists all indexes that have been attached to a particular dataset.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string"
}

Response

{
  "success": true,
  "index_infos": [
    [ "index_name", "distance_type" ],
    ...
  ]
}

Each entry is a two-element array: the index name and its distance type ("Euclidean", "Cosine", "InnerProduct", "Manhattan" or "Hamming"). The index type is not included; /list_all_items returns it. The project and dataset must exist (otherwise 400).

Example

POST /list_index_infos
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset"
}

Response:

{
  "success": true,
  "index_infos": [
    ["index1", "Cosine"],
    ["index2", "Euclidean"]
  ]
}

---

Hybrid Search

VecML DB provides advanced Hybrid Search functionalities that combines semantic search (e.g., through vector embeddings) and keyword search ("string match"). To use hybrid search, the user needs to specify a text field that will be used to perform keyword matching. The text field should be inserted as an attribute for each vector.

An example workflow for hybrid search is:

1. Upload a vector collection with a target text field stored as an attribute (named "attribute" in this example), for example, using /add_data_batch or /upload.

2. Attach a vector index using /attach_index. Check /get_attach_index_status and confirm indexing is successful.

3. Attach a text field index using /attach_text_field_index specifying the target text field as "attribute". Check /get_attach_index_status and confirm indexing is successful.

4. Perform hybrid search with /search API and hybrid search specific request fields. The keyword part of the search uses the query's own text, passed as the same attribute on each query vector.

---

/attach_text_field_index

Description: Creates an keyword matching index for a text field of a given dataset. The text field must be one attribute of the vectors. The indexing job runs asynchronously. You receive a job_id for tracking progress via /get_attach_index_status. Once attached, whenever new data are added to or removed from the data collection, the index will be updated automatically.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "attribute_name": "string"
}

• attribute_name: the text field that we want to index with keyword matching, which must be inserted/uploaded as an attribute of the vector. For example, the text field can be the chunk that generate each embedding vector. All four fields are required (otherwise 400); the name must be non-empty, at most 128 characters, without \ / : * ? " ' < > |, and must not already have a text field index (400). If the dataset has no attribute with this name at all, the indexing job fails with "Attach text field index failed with error AttributeNotFound"; if the attribute name exists in the dataset but no vector has a value for it, the job succeeds with an empty index. In general, vectors whose value for the attribute is empty or missing are simply not indexed.

Response

{
  "job_id": "string",
  "success": true,
  "error_code": "Success"
}

• job_id can be passed to /get_attach_index_status to see if the index build is finished.

Example

POST /attach_text_field_index
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "attribute_name": "text_chunk"
}

Response:

{
  "job_id": "<user_id>||ProjectA||MyDataset||text_chunk||AttachTextFieldIndexJob",
  "success": true,
  "error_code": "Success"
}

---

/delete_text_field_index

Description: Deletes a text field index from the dataset. This action cannot be undone.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "attribute_name": "string"    // the text field whose index will be deleted
}

Response

{
  "success": true
}

The project and dataset must exist (otherwise 400). If the index cannot be deleted, the HTTP status is still 200 and the body is {"success": false, "error_code": "..."}.

Example

POST /delete_text_field_index
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset",
  "attribute_name": "text_chunk"
}

---

/list_text_field_index_infos

Description: Lists all text field indexes that have been attached to a particular dataset.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string"
}

Response

{
  "success": true,
  "text_field_indices": [
    "text_field1", "text_field2"
  ]
}

text_field_indices is an array of attribute names (strings) that have a text field index. The project and dataset must exist (otherwise 400).

Example

POST /list_text_field_index_infos
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "project_name": "ProjectA",
  "collection_name": "MyDataset"
}

Response:

{
  "success": true,
  "text_field_indices": [
    "text_chunk", "title"
  ]
}

---

/search (for hybrid search)

Description:
We make it convenient to perform hybrid search, which combines results from both semantic (vector) and lexical (keyword) retrieval within the standard vector /search API request.

We use Reciprocal Rank Fusion (RRF) to merge the rankings from the two retrievers.
Suppose we want to retrieve \(k\) vectors. Denote by \(V\) the union of the top-\(k\) vectors obtained from both the semantic and lexical searches. The RRF score for each document \(v \in V\) is computed as:

\[ \text{score}(v) = w \cdot \frac{1}{f + \text{rank}_{\text{semantic}}(v)} + (1 - w) \cdot \frac{1}{f + \text{rank}_{\text{lexical}}(v)} \]

where:

• \(w\) — the weight assigned to the semantic (vector-based) score.
• \(\text{rank}_{r}(v)\) — the rank position of vector \(v\) in retriever \(r\) (either semantic or lexical), starting at 1. A vector found by only one retriever gets only that term.
• \(f\) — the damp constant parameter that controls how quickly the contribution of lower-ranked results decreases.
  A smaller \(f\) makes top-ranked documents dominate more strongly, while a larger \(f\) smooths the fusion across ranks.

Method - POST

Request Body

{
  "user_api_key": "string",
  "project_name": "string",
  "collection_name": "string",
  "query_type": "vector" | "vectors" | "row" | "file" | "dataset",
  "index_name": "string",
  "top_k": int,                 // optional, default 5
  "filter": {
     // optional filter structure, see below
  },

  // If query_type == "vector":
  "query_vector": [0.1, -0.2, 0.3],
  "attributes": {"text_chunk": "query text for keyword matching"},

  // If query_type == "vectors":
  "query_vectors": [[0.1, -0.2, 0.3], [0, 1, 10]],                 // dense dataset
                   [[[0, 0.1], [4, 5.2]], [[2, 0], [9, 0.3]]],      // sparse dataset: [index, value] pairs
  "attributes": [{"text_chunk": "query text 1"}, {"text_chunk": "query text 2"}],   // one object per query vector

  // If query_type == "row":
  "row_number": 42,

  // If query_type == "file" (the file includes the hybrid_attribute_name column/field):
  "file_data": "string",        // base64 encoded file
  "file_name": "string",
  "file_format": "csv" | "json" | "libsvm" | "binary",

  // If query_type == "dataset":
  "query_collection_name": "string",

  ============== Additional fields for hybrid search =================
  "hybrid_attribute_name": "string",
  "semantic_weight": float,
  "damp_constant": float
}

• Filter structure: see the guide in /search.

• hybrid_attribute_name: the text field used for keyword matching in hybrid search. Hybrid search is used only when this field is given.
  The field must already have an index created via attach_text_field_index.

• The keyword query text comes from the query vector itself: each query must carry the hybrid_attribute_name attribute with the text to match.

  • "vector": pass it in the attributes object.
  • "vectors": pass attributes, an array with one object per entry of query_vectors.
  • "row" and "dataset": the attribute stored on the query vectors is used.
  • "file": each row's value of the hybrid_attribute_name column (CSV) or field (JSON). The attribute must be one of the dataset's attributes, and the file must contain it (CSV with a header or JSON with field names); libsvm and binary files carry no attributes, so with semantic_weight below 1 their queries fail and are left out of results.

  A query without this attribute fails: it is left out of results (for "vectors", "file", "dataset") or the response carries a non-Success error_code (for "vector", "row"). Attribute values must be strings; other value types make the request fail with HTTP 500.

• semantic_weight: the weight \(w\) for the semantic score (vector similarity) in the RRF formula, a float between 0 and 1. Default 0.5. With exactly 1, only the vector search runs (the query attribute is not needed) and score holds the vector search value (as distance in /search); with exactly 0, only the keyword search runs and score is the absolute keyword-match similarity rather than an RRF score.

• damp_constant: the RRF damp constant \( f \) in the RRF formula. Controls how steeply the influence of lower-ranked documents decays. Default 2.0.

Response

The response has the same structure as /search, except that each match carries score (the RRF score, higher is better; matches are sorted by it) instead of distance.

• For query_type: "row":

  {
    "error_code": "Success",
    "results": [
      {"idx": "string_id_of_match", "score": float},
      ...
    ],
    "distance_type": "Cosine"
  }
  

• For query_type: "vector", "vectors", "file" or "dataset":

  {
    "error_code": "Success",
    "results": [
      {
        "query_vector_id": "...",
        "matches": [
          {"idx": "string_id_of_match", "score": float},
          ...
        ]
      },
      ...
    ],
    "distance_type": "Cosine"
  }
  

Examples

• Query with a direct vector

  POST /search
  Content-Type: application/json
  
  {
    "user_api_key": "api_key_123",
    "project_name": "ProjectA",
    "collection_name": "MyDataset",
    "query_type": "vectors",
    "index_name": "index1",
    "top_k": 5,
    "query_vectors": [[0.25, 0.75, 1.25, 2.00]],       // use one vector in the example
    "attributes": [{"text_chunk": "wireless mouse battery life"}],
    "hybrid_attribute_name": "text_chunk",
    "semantic_weight": 0.7,
    "damp_constant": 2.0,
    "filter": {
      "type": "equality",
      "attribute": "Category",
      "target_values": ["CatA","CatC"]
    }
  }
  

  Response:

  {
    "error_code": "Success",
    "results": [
      {
        "query_vector_id": "0",
        "matches": [
          {"idx": "Sample_01", "score": 0.3333},    // rank 1 in both retrievers: 0.7/3 + 0.3/3
          {"idx": "Sample_07", "score": 0.2500},    // rank 2 in both retrievers: 0.7/4 + 0.3/4
          ...
        ]
      }
    ],
    "distance_type": "Cosine"
  }
  

• Query referencing an existing row (the row's stored text_chunk is the keyword query)

  {
    "query_type": "row",
    "row_number": 123,
    "hybrid_attribute_name": "text_chunk",
    ...
  }
  

• File-based queries (each row's text_chunk column is its keyword query)

  {
    "query_type": "file",
    "file_data": "<base64 CSV with the vector columns and a text_chunk column>",
    "file_name": "queries.csv",
    "file_format": "csv",
    "hybrid_attribute_name": "text_chunk",
    ...
  }
  

• Dataset-to-dataset query (each query vector's stored text_chunk is its keyword query)

  {
    "query_type": "dataset",
    "query_collection_name": "QueryVectorsDS",
    "hybrid_attribute_name": "text_chunk",
    ...
  }
  

---

Job Status Management

VecML DB uses asynchronous job executions on computationally heavy jobs like data insertion, file uploads, and indexing. We provide APIs to check the status of these jobs. It is a good practice to check the job status and confirm success before subsequent DB operations. See example workflow.

/get_upload_data_status

Description: Checks the status of a data job initiated by /upload, /add_data_batch, /add_vector_attributes or /add_vector_attributes_from_file.

Method - POST

Request Body

{
  "user_api_key": "string",
  "job_id": "string"
}

Response

{
  "success": true,
  "status": "pending" | "in_progress" | "finished" | "failed",
  "error_message": "detailed error description",    
  "start_time": "string",
  "end_time": "string",
  "duration": "string"
}

If the job status is "in_progress", the end_time will be the current time stamp, and duration will be the time ellapsed since job started until now. Times are formatted YYYY/MM/DD, HH:MM:SS; duration is a phrase such as "2 minutes 5 seconds".

A missing job_id returns 400. An unknown job_id, or one belonging to another user, returns HTTP 404 with {"success": false, "error": "Job not found"}. Finished and failed jobs are kept for 24 hours (up to about 25 hours, as cleanup runs hourly) after they started.

Example

POST /get_upload_data_status
Content-Type: application/json

{
  "user_api_key": "api_key_123",
  "job_id": "my_job_id"
}

Response:

For data insertion job was initialized by /UPLOAD:

{
  "success": true,
  "status": "finished",
  "error_message": "Upload completed successfully.",
  "start_time": "2025/10/15, 14:02:15",
  "end_time": "2025/10/15, 14:02:28",
  "duration": "13 seconds"
}

or, if the upload failed (for example, a dimension mismatch with the existing dataset):

{
  "success": true,
  "status": "failed",
  "error_message": "Uploaded vector has incorrect dimension, dimension should be 128",
  "start_time": "2025/10/15, 14:02:15",
  "end_time": "2025/10/15, 14:02:16",
  "duration": "1 second"
}

For data insertion job was initialized by /ADD_DATA_BATCH:

If all vectors are added to the data collection successfully:

{
  "success": true,
  "status": "finished",
  "error_message": "",
  "start_time": "2025/10/15, 14:02:15",
  "end_time": "2025/10/15, 14:02:28",
  "duration": "13 seconds"
}

Otherwise, (for example, two vectors failed to be inserted), the job is still "finished" and the failed vectors are listed:

{
  "success": true,
  "status": "finished",
  "error_message": "The following vectors failed to be inserted: [{\"body\":\"Failed to add vector: VectorAlreadyExists\",\"string_id\":\"ID_0\"},{\"body\":\"Failed to add vector: VectorAlreadyExists\",\"string_id\":\"ID_1\"}]",
  "start_time": "2025/10/15, 14:02:15",
  "end_time": "2025/10/15, 14:02:28",
  "duration": "13 seconds"
}

---

/get_attach_index_status

Description: Returns the current status of an index-building job (from /attach_index or /attach_text_field_index).

Method - POST

Request Body

{
  "user_api_key": "string", 
  "job_id": "string"
}

Response

{
  "success": true,
  "status": "pending" | "in_progress" | "finished" | "failed",
  "error_message": "detailed error description",    
  "start_time": "string",
  "end_time": "string",
  "duration": "string"
}

If the job status is "in_progress", the end_time will be the current time stamp, and duration will be the time ellapsed since job started until now. If the job failed, error_message gives the error.

A missing job_id returns 400. An unknown job_id, or one belonging to another user, returns HTTP 404 with {"success": false, "error_message": "Job not found"}. Finished and failed jobs are kept for 24 hours (up to about 25 hours, as cleanup runs hourly) after they started.

Example

POST /get_attach_index_status
Content-Type: application/json

{ 
  "user_api_key": "api_key_123",
  "job_id": "my_job_id"
}

Response:

{
  "success": true,
  "status": "finished",
  "error_message" : "",
  "start_time": "2025/10/15, 14:02:15",
  "end_time": "2025/10/15, 14:02:28",
  "duration": "13 seconds"
}

---

/get_user_jobs

Description: Retrieves all jobs (upload, indexing, AutoML training and clustering) associated with the authenticated user. This endpoint provides a comprehensive overview of all job types with their current status, timing information, and any relevant metadata or error messages. The following jobs will be returned:

• All pending and in_progress jobs
• finished and failed jobs that were started within the last 24 hours (removed by an hourly cleanup, so up to about 25 hours)

Method - POST

Request Body

{
  "user_api_key": "string"
}

Response

{
  "upload_jobs": [
    {
      "job_id": "string",
      "status": "pending" | "in_progress" | "finished" | "failed",
      "start_time": "string",
      "end_time": "string",
      "duration": "string",
      "error_message": "string"  // optional, only if error occurred
    }
  ],
  "indexing_jobs": [
    {
      "job_id": "string", 
      "status": "pending" | "in_progress" | "finished" | "failed",
      "start_time": "string",
      "end_time": "string",
      "duration": "string",
      "error_message": "string"  // optional, only if error occurred
    }
  ],
  "training_jobs": [
    {
      "job_id": "string",
      "status": "pending" | "in_progress" | "finished" | "failed", 
      "model_name": "string",
      "task_type": "string",
      "label_attribute": "string",
      "start_time": "string",
      "end_time": "string",
      "duration": "string",
      "validation_metric": {  // optional, only for finished training jobs
        ...
      }
    }
  ],
  "clustering_jobs": [
    {
      "job_id": "string",
      "status": "pending" | "in_progress" | "finished" | "failed",
      "model_name": "string",
      "model_type": "string",
      "kmeans_method": "string",   // only for k-means jobs
      "params": { ... },           // optional, the job's hyperparameters
      "start_time": "string",
      "end_time": "string",
      "duration": "string",
      "error_message": "string",   // optional, only if error occurred
      "error_code": "string"       // only for finished or failed jobs
    }
  ],
  "summary": {
    "total_upload_jobs": int,
    "total_indexing_jobs": int, 
    "total_training_jobs": int,
    "total_clustering_jobs": int
  }
}

• upload_jobs also include /add_vector_attributes and /add_vector_attributes_from_file jobs; indexing_jobs include text field index jobs.
• validation_metric holds the classification metrics (accuracy, micro_precision, macro_precision, micro_recall, macro_recall, micro_f1, macro_f1, auc) or, for regression models, mse and mae.
• Times are formatted YYYY/MM/DD, HH:MM:SS.

Example

POST /get_user_jobs
Content-Type: application/json

{
  "user_api_key": "api_key_123"
}

Response:

{
  "upload_jobs": [
    {
      "job_id": "<user_id>||ProjectA||MyDataset||InsertDataJob||550e8400-e29b-41d4-a716-446655440000",
      "status": "finished",
      "start_time": "2024/01/15, 10:30:00",
      "end_time": "2024/01/15, 10:30:17",
      "duration": "17 seconds",
      "error_message": "Upload completed successfully."
    }
  ],
  "indexing_jobs": [
    {
      "job_id": "<user_id>||ProjectA||MyDataset||index1||AttachIndexJob",
      "status": "in_progress",
      "start_time": "2024/01/15, 11:00:00",
      "end_time": "2024/01/15, 11:02:25",
      "duration": "2 minutes 25 seconds"
    }
  ],
  "training_jobs": [
    {
      "job_id": "<user_id>||ProjectA||MyDataset||my_classifier||TrainJob",
      "status": "finished",
      "model_name": "my_classifier",
      "task_type": "classification",
      "label_attribute": "category",
      "start_time": "2024/01/15, 09:00:00",
      "end_time": "2024/01/15, 09:06:23",
      "duration": "6 minutes 23 seconds",
      "validation_metric": {
        "accuracy": 0.95,
        "micro_precision": 0.94,
        "macro_precision": 0.93,
        "micro_recall": 0.95,
        "macro_recall": 0.92,
        "micro_f1": 0.94,
        "macro_f1": 0.92,
        "auc": 0.96
      }
    }
  ],
  "clustering_jobs": [],
  "summary": {
    "total_upload_jobs": 1,
    "total_indexing_jobs": 1,
    "total_training_jobs": 1,
    "total_clustering_jobs": 0
  }
}

---