Launching an inference instance

Launching a model as a managed OpenAI-compatible endpoint. The dialog is titled Deploy Inference Endpoint in the console.

Open the dialog

Console → Compute → Model Instances → Deploy Endpoint.

Step 1 — Engine

Pick how the model is served:

Engine	When to use
vLLM	You want the platform vLLM image. Works out of the box for LLM and vision-LLM (VLM) models.
Custom Container	Your own image from the Images registry (or any public Docker image). Required for image / audio / video models, or for any runtime that isn't vLLM (TGI, llama.cpp, Triton, custom servers).

Step 2 — Model source

Three mutually-exclusive options:

Source	What it points at	Replicas / regions
HuggingFace	Any HF repo ID (e.g. Tongyi-MAI/Z-Image-Turbo). Pulls at first launch, cached per region.	2 replicas across 2 regions (redundancy)
EcoLink Registry	A model you registered in Registry → Models — typically a folder on a shared filesystem (fine-tunes, custom weights).	1 replica in the filesystem's region (weights are pinned)
EcoLink Models	A built-in platform model (same catalog you see in the Playground).	2 replicas across 2 regions

For HuggingFace: paste the repo ID. A HuggingFace Token field appears — required for gated repos (Llama, Gemma, …), optional otherwise. Your token is stored server-side; it isn't visible to anyone else on your account.

For EcoLink Registry: pick from a dropdown of the models you've registered. The Model Path shown next to it (e.g. /mnt/shared/Z-Image-Turbo-fine-tuning) is the path the container sees at start-up.

Step 3 — Container Image (Custom Container only)

Two ways to specify the image:

• Pick from a dropdown of images you've saved in Registry → Images (convenient — same image used across many launches), or
• Type a full Docker image URL, e.g. docker.io/vllm/vllm-omni:v0.18.0.

Step 4 — Runtime fields

Field	Notes
Name	Friendly label (optional). Becomes part of the model ID (<name>:<instance_id>).
Startup Command	Shell command the container runs. Examples: vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8000 or just --tensor-parallel-size 2 if your image has an ENTRYPOINT that prepends vllm serve.
Service Port	Port your container listens on (default 8000). The platform routes /v1/* and /health to this port.
OpenAI-compatible	Default on. Uncheck if your container has a custom HTTP surface (then fill in Custom API path). See OpenAI-compatible vs custom.
Custom API path	Required only when OpenAI-compatible is off — e.g. /generate.
GPU Type	Shows live free-GPU count per region under each choice. Pick the type that has capacity where you want to run.
GPUs / replica	1, 2, 4, or 8. Multi-GPU replicas use tensor parallelism — your startup command may need --tensor-parallel-size <n> (vLLM).
Max replicas	Upper bound on total replicas across all target regions. Default 2.

The bottom of the dialog shows Per replica $X/hr and Total ($X/hr × N regions) — this is exactly what you'll be charged (as a 24h credit hold, auto-renewed) while the instance runs.

Step 5 — Deploy

Click Deploy.

What happens after launch

1. The instance goes to pending.
2. For each target region, EcoLink creates a pod with your model and container.
3. For each pod:
   • Image is pulled (~1–3 min)
   • Model is downloaded (HF-backed, first-time-per-region) or mounted (FS-backed)
   • Container starts; platform waits for /health to report ready
4. Regional status transitions pending → loading → running as pods become Ready.
5. Instance-level status goes pending → degraded → running as regions come up. degraded means some regions are still loading; the endpoint is already callable (routes to healthy regions only).

Typical time to fully-running: 2–5 minutes for small HF models, 5–15 minutes for 30B–70B HF models, 30–90 seconds for FS-backed.

After it's running — get your endpoint

Go to the instance detail page. You'll see:

• Model ID: my-llama:142 (where 142 is the instance ID)
• Endpoint URL: https://api.ecohash.com/v1 — same for all user inference instances; differentiation is via the model ID
• Proxy URL (if non-OpenAI-compatible): https://api.ecohash.com/inference-instances/142/proxy/<custom_path>

See Calling your endpoint for actual API usage.

Example — deploy Z-Image-Turbo from HuggingFace

Walkthrough for an image-generation model:

1. Console → Compute → Model Instances → Deploy Endpoint.
2. Name: Z-Image-Turbo.
3. Engine: Custom Container.
4. Model source: HuggingFace → Tongyi-MAI/Z-Image-Turbo. Paste your HF token only if the repo is gated.
5. Container Image: docker.io/vllm/vllm-omni:v0.18.0 (or pick from your saved Images).
6. Startup Command: vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8000.
7. GPU Type: pick one with capacity.
8. Deploy.

Once it shows as running, test it (the instance detail page shows the endpoint URL, e.g. https://45.inference.ecohash.com):

curl https://45.inference.ecohash.com/v1/images/generations \
  -H "Authorization: Bearer eco_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A futuristic city with flying cars, high resolution, cinematic lighting",
    "n": 1,
    "size": "512x512"
  }'

From the API

curl https://api.ecohash.com/inference-instances \
  -X POST \
  -H "Authorization: Bearer eco_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-llama-70b",
    "user_model_id": 5,
    "gpu_type": "<gpu-type-for-region>",
    "gpu_count": 2,
    "container_image": "vllm/vllm-openai:latest",
    "startup_command": "--tensor-parallel-size 2",
    "service_port": 8000,
    "openai_compatible": true
  }'

Regions and replica count are derived from the registered model's source — you don't specify them in the request.

Credit hold at launch

The platform places a 24h hold sized for the full multi-region deployment: hourly_rate_per_gpu × gpu_count × total_replicas × 24h. If your balance can't cover it, launch fails with 402. The launch dialog shows the per-GPU rate and total hold for your selection.

See Cost and lifecycle for the full renewal / partial-hold / terminate-at-$0 story.

Startup gotchas

• "Model not found" inside the container — check that the container's --model argument matches the model path EcoLink provides at MODEL_PATH env var (HF repo ID or mounted filesystem path).
• OOM at load time — the model doesn't fit in the GPU count you picked. Options: use more GPUs with tensor parallelism, or quantize the model (fp8, int8, int4).
• Container crashes with missing cuDNN — the container image needs cuDNN installed. Standard vLLM / TGI images come with it. Custom images sometimes miss it.

Save a working deployment as a template

Once an instance is serving, click Save as template on its detail page. The engine, image, GPU shape and runtime fields are recorded, and the next deployment is one click — yours, or your colleagues'. See Compute templates.

Next steps

• Call your endpoint
• Save this deployment as a template
• OpenAI-compatible vs custom containers
• Cost and lifecycle