Dedicated capacity
Dedicated Capacity questions
Answers to common questions about Dedicated Capacity: model size, bundles, sizing, access, data, model lifetime, testing, billing and serving your own customers.
What is a rack?
Section titled “What is a rack?”A rack is one SambaRack: 16 purpose-built AI chips (RDUs) in 8 servers. It is one unit of capacity. We size your capacity in racks.
How large a model fits on one rack?
Section titled “How large a model fits on one rack?”Every model in the list of supported models runs on one rack. What fits on a rack depends on the model’s architecture and precision, not only on its parameter count. For a model that is not on the list, we check the fit with you.
Do several models on one rack run at the same time?
Section titled “Do several models on one rack run at the same time?”The models in a bundle stay loaded together. The rack computes one model at a time and switches between them in milliseconds, per request. The models share the throughput of the rack, so we size a bundle for the combined traffic of its models. See Model bundles.
How many racks do I need?
Section titled “How many racks do I need?”We size your capacity from your workload. Send us:
- the models
- the tokens per day
- the ratio of input to output tokens
- the typical and the maximum context length
- the peak number of concurrent requests
- your latency or throughput targets
- how the traffic is spread over the day
Talk to us with these values.
Can we get lower-level access than the model endpoints?
Section titled “Can we get lower-level access than the model endpoints?”No. Dedicated Capacity gives you model endpoints through the OpenAI-compatible API. You do not get access to the hardware, the operating system or the serving stack. See Connect to your endpoint.
Who can access the system, and does data leave the EU?
Section titled “Who can access the system, and does data leave the EU?”The racks are in our data center in Munich. SambaNova operates the software stack with us, with remote access, as set out in our data processing agreement (DPA). We do not store prompts or outputs persistently. See the Trust Center.
How do we reach the endpoint?
Section titled “How do we reach the endpoint?”Your endpoint is reached over HTTPS on the internet. Talk to us if you need a private network connection.
How long do the models stay? Can we keep a model?
Section titled “How long do the models stay? Can we keep a model?”You choose the models on your capacity. We agree any model change with you.
Can we test before we commit?
Section titled “Can we test before we commit?”Yes. We can arrange a time-limited test on the model you need before you sign. If the model is in the Inference API catalog, you can also start there. Talk to us.
Can we serve our own customers on it?
Section titled “Can we serve our own customers on it?”Yes. It is your reserved capacity. You can build your own service on it, including serving your own clients. System integrators and agencies do this.
How is it billed?
Section titled “How is it billed?”You pay a monthly fee for the reserved capacity, per agreement. Longer terms get better rates. Shorter terms are possible. There is no one-time hardware fee. We set prices per engagement and do not publish them.
Can you run a model that is not on the list?
Section titled “Can you run a model that is not on the list?”An open-weight model with a supported architecture: yes, after a bring-up. Support for a new architecture is a commercial conversation. A proprietary model: no, because we have no access to its weights. See Custom checkpoints.