Inference gateway

Choose your fighter:

Follow the steps in the web app.

The inference gateway supports model listing and OpenAI-compatible Chat Completions, including streaming, at https://gateway.str.ing. Requests use an organization credential and are attributed to that organization’s usage.

List models and send a request

Gateway requests are made through HTTP clients. Select curl above for a model listing and a Chat Completions example.

Set STRING_TOKEN to a credential for your organization. Replace acme with that organization’s slug, then list available models:

curl https://gateway.str.ing/acme/models \
  -H "Authorization: Bearer $STRING_TOKEN"

Set STRING_MODEL to a model ID from the response and send a completion:

curl https://gateway.str.ing/acme/chat/completions \
  -H "Authorization: Bearer $STRING_TOKEN" \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"$STRING_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}]}"

Set stream to true in the request for a streamed response.

str api calls the application API. For requests to gateway.str.ing, select curl above and use the gateway examples.

You can also use an OpenAI-compatible client. Set its base URL to https://gateway.str.ing/acme and its API key to your organization credential.

Response caching

The gateway caches eligible identical requests within your organization for 24 hours. A cache hit returns the stored completion without another model call, including when the request has a nonzero temperature.

Control caching with the X-String-Cache request header:

Value	Behavior
`enabled` (default)	Read and write the cache
`read-only`	Read cached responses without storing new ones
`write-only`	Store responses without reading the cache
`disabled`	Skip cache reads and writes

Use disabled when you want a fresh response for each request. The x-string-cache-status response header reports hit, miss, or bypass.

See the gateway API specification for request schemas and error responses.