Onnx FAQ & Answers

sentence-transformers' native ONNX backend has compatibility issues with nomic-embed-text due to:

1. Custom code requirement: nomic-embed-text uses trust_remote_code=True for custom model architecture
2. safe_serialization argument: sentence-transformers passes this argument to ORTModel which doesn't accept it
3. Multiple ONNX files: The model has 8 ONNX variants, causing ambiguity warnings

Error message:

ORTModel._from_pretrained() got an unexpected keyword argument 'safe_serialization'

Solution: Use Optimum directly instead:

from optimum.onnxruntime import ORTModelForFeatureExtraction

model = ORTModelForFeatureExtraction.from_pretrained(
    "nomic-ai/nomic-embed-text-v1.5",
    file_name="model_int8.onnx",
    subfolder="onnx",
    provider="CPUExecutionProvider",
    trust_remote_code=True
)

This approach is actually better as it gives you full control over which ONNX variant to load.

This error occurs due to a compatibility issue between sentence-transformers and the optimum library. The sentence-transformers ONNX backend passes safe_serialization to ORTModel which doesn't accept it.

Solution: Use Optimum directly instead of sentence-transformers' ONNX backend:

from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForFeatureExtraction
import torch.nn.functional as F

tokenizer = AutoTokenizer.from_pretrained("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)
model = ORTModelForFeatureExtraction.from_pretrained(
    "nomic-ai/nomic-embed-text-v1.5",
    file_name="model_int8.onnx",
    subfolder="onnx",
    provider="CPUExecutionProvider",
    trust_remote_code=True
)

This bypasses sentence-transformers' ONNX wrapper and uses Optimum directly, avoiding the argument compatibility issue.

Use Optimum's ORTModelForFeatureExtraction directly (not sentence-transformers backend) with manual mean pooling:

from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForFeatureExtraction
import torch
import torch.nn.functional as F

# Load model
tokenizer = AutoTokenizer.from_pretrained("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)
model = ORTModelForFeatureExtraction.from_pretrained(
    "nomic-ai/nomic-embed-text-v1.5",
    file_name="model_int8.onnx",  # INT8 for CPU, or model.onnx for full precision
    subfolder="onnx",
    provider="CPUExecutionProvider",
    trust_remote_code=True
)

# Mean pooling function
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Generate embeddings
texts = ["search_query: What is PostgreSQL?"]
inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
outputs = model(**inputs)
embeddings = mean_pooling(outputs, inputs["attention_mask"])
normalized = F.normalize(embeddings, p=2, dim=1)

Note: nomic-embed-text requires task prefixes like search_query: or search_document:.

Use the file_name and subfolder parameters:

from optimum.onnxruntime import ORTModelForFeatureExtraction

model = ORTModelForFeatureExtraction.from_pretrained(
    "nomic-ai/nomic-embed-text-v1.5",
    file_name="model_int8.onnx",  # Just the filename, not the path
    subfolder="onnx",              # The subfolder containing ONNX files
    provider="CPUExecutionProvider",
    trust_remote_code=True
)

Important:

• file_name should be just the filename (e.g., model_int8.onnx), not a path
• subfolder specifies the directory within the model repo (e.g., onnx)
• Don't combine them like file_name="onnx/model_int8.onnx" - this will fail

Common error: FileNotFoundError: Could not find any ONNX files with target file name onnx/model.onnx - this means you incorrectly included the subfolder in file_name.

ONNX INT8 quantization provides significant speedups on CPU:

Benchmark (nomic-embed-text-v1.5 on CPU):

Metric	PyTorch	ONNX INT8	Speedup
Model load	~3.0s	~1.4s	2.1x
Single query	~80ms	~40ms	2.0x
Batch 10	164ms (16ms/q)	94ms (9ms/q)	1.7x
Batch 100	1270ms (12ms/q)	636ms (6ms/q)	2.0x

Key points:

• INT8 quantization has minimal accuracy loss (<1% typically)
• Speedup is most significant for single queries and small batches
• Large batches see ~2x improvement
• No GPU required - runs efficiently on CPU

To use ONNX INT8:

model = ORTModelForFeatureExtraction.from_pretrained(
    "model-name",
    file_name="model_int8.onnx",
    subfolder="onnx",
    provider="CPUExecutionProvider"
)

nomic-embed-text-v1.5 provides multiple ONNX variants in the onnx/ subfolder:

File	Description	Best For
model.onnx	Full precision (FP32)	Maximum accuracy
model_fp16.onnx	Half precision (FP16)	GPU inference
model_int8.onnx	INT8 quantized	CPU inference (recommended)
model_uint8.onnx	UINT8 quantized	CPU inference
model_quantized.onnx	Dynamic quantization	CPU inference
model_q4.onnx	4-bit quantization	Memory constrained
model_q4f16.onnx	4-bit with FP16	GPU memory constrained
model_bnb4.onnx	BitsAndBytes 4-bit	Extreme compression

For CPU inference, use model_int8.onnx - it provides ~1.5-2x speedup over PyTorch with minimal accuracy loss.

Use the provider parameter in ORTModelForFeatureExtraction:

CPU Inference:

from optimum.onnxruntime import ORTModelForFeatureExtraction

model = ORTModelForFeatureExtraction.from_pretrained(
    "nomic-ai/nomic-embed-text-v1.5",
    file_name="model_int8.onnx",  # INT8 best for CPU
    subfolder="onnx",
    provider="CPUExecutionProvider"
)

GPU Inference (CUDA):

model = ORTModelForFeatureExtraction.from_pretrained(
    "nomic-ai/nomic-embed-text-v1.5",
    file_name="model_fp16.onnx",  # FP16 best for GPU
    subfolder="onnx",
    provider="CUDAExecutionProvider"
)

Available providers:

• CPUExecutionProvider - Default CPU
• CUDAExecutionProvider - NVIDIA GPU
• TensorrtExecutionProvider - NVIDIA TensorRT (fastest GPU)
• CoreMLExecutionProvider - Apple Silicon
• ROCMExecutionProvider - AMD GPU

Install with: pip install optimum[onnxruntime-gpu] for GPU support.

ONNX models output raw token embeddings, not pooled sentence embeddings. You need to apply mean pooling manually:

import torch
import torch.nn.functional as F

def mean_pooling(model_output, attention_mask):
    """
    Mean pooling - average token embeddings weighted by attention mask.
    
    Args:
        model_output: ONNX model output (tuple with token embeddings at index 0)
        attention_mask: Tokenizer attention mask
    
    Returns:
        Pooled embeddings tensor
    """
    token_embeddings = model_output[0]  # First element is token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
    sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
    return sum_embeddings / sum_mask

# Usage
inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
outputs = model(**inputs)
embeddings = mean_pooling(outputs, inputs["attention_mask"])
normalized = F.normalize(embeddings, p=2, dim=1)  # L2 normalize

The attention mask ensures padding tokens don't contribute to the mean.