AI compute
AI Inference Software
Run LLMs reliably and efficiently. One software package covers model deployment, inference serving, scheduling and monitoring. It comes preinstalled on the AI Compute Appliance, or runs on GPU servers you already have.
Key capabilities
One-click model deployment
Pick a model and launch an inference service, with no per-server environment setup. Supports mainstream open-source LLMs, including text generation, embedding and multimodal models.
Inference acceleration
Optimized for concurrency and GPU memory, so the same GPUs serve more requests.
Unified API
Business systems call every model through one OpenAI-compatible API, which keeps integration work low.
Multi-model scheduling
Multiple models share one set of GPUs. Resources are allocated by load and released automatically when idle.
Permissions and usage
Grant API access by department or application, and track call volume and resource usage.
Monitoring and alerts
See GPU utilization, request latency and error rates in real time, with prompt alerts on anomalies.
Where it fits
The inference software sits between compute and your applications: it schedules GPUs below and serves models to every AI application above.
Three deployment options
Preinstalled on the appliance
The AI Compute Appliance ships with the inference software installed, so you can deploy models on arrival.
On your own servers
If you already have GPU servers, you can install it on its own. Supports mainstream NVIDIA data center and professional GPUs; our consultants will confirm specific models.
Hosted in a HyperSource data center
Place your servers in HyperSource's Beijing, Guangzhou, Dongguan or Guiyang data centers, with software and hardware managed together.
Specifications
| Item | Details |
|---|---|
| Supported models | Mainstream open-source LLMs; custom models can be added |
| Supported hardware | NVIDIA GPUs, single to multi-GPU; specific models and quantity confirmed based on your needs |
| API | OpenAI-compatible REST API |
| Deployment | Single node / multi-node cluster |
| Licensing | Preinstalled with the appliance, or licensed per node; subject to quotation |
| System requirements | Linux, Docker or bare-metal installation; detailed requirements provided with the proposal |
FAQ
Do I have to buy a HyperSource appliance?
No. The inference software can be installed on your existing GPU servers. Our consultants will confirm compatibility with your hardware.
How is this different from an open-source inference framework?
With an open-source framework, you set up the environment, tune parameters, and build monitoring and permissions yourself. Our inference software provides these as ready-made features, and we provide upgrades and technical support.
Which LLMs are supported?
All mainstream open-source LLMs can be deployed. The model size you can run depends on the GPU memory configuration, and you can test it at our data center.
How is it priced?
When preinstalled on an appliance, it is included in the system price. Standalone installs are licensed per node, including version upgrades and technical support, subject to quotation.
Related
Request a trial of AI Inference Software
A consultant will reply during business hours (China Standard Time) with site, configuration and pricing recommendations.
- leo@hs.com.cn
- 24/7 technical support