Your App Now Has a Non-Deterministic Dependency
Traditional API monitoring rests on a comforting assumption: same input, same output, predictable latency. You send a request to a payment processor, you get back a charge confirmation. You query a database API, you get back rows. The response is deterministic, the latency falls within a known range, and when something breaks, it breaks obviously -- a 500, a timeout, a malformed payload.
LLM APIs break all three assumptions.
Send the same prompt to GPT-4 twice and you will get two different responses. Latency can swing from 800ms to 15 seconds depending on output length, model load, and whether the provider is having a rough afternoon. And failure is no longer binary. An LLM endpoint can return a 200 with a confident, fluently written answer that is completely wrong.
If you are building on top of OpenAI, Anthropic, Google, or any other LLM provider, your monitoring needs a fundamentally different approach. The tools you use for REST APIs are a starting point, not a solution.
What Makes LLM APIs Different to Monitor
Non-Deterministic Responses
A traditional API contract says: given input X, expect output Y. LLM APIs have no such contract. The same prompt can produce different wording, different structure, and occasionally different meaning across calls. This makes response validation harder. You cannot assert on exact content. You have to validate structure, format compliance, and the absence of failure signals like refusals or hallucinated tool calls.
Streaming vs. Non-Streaming Failure Modes
Most LLM integrations use streaming (Server-Sent Events) to reduce perceived latency. Streaming introduces failure modes that traditional monitoring misses:
- The connection opens successfully (HTTP 200) but no tokens arrive -- a silent hang.
- Tokens stream for several seconds, then the stream terminates mid-sentence.
- The stream completes but the final response is truncated due to hitting the max token limit.
A standard HTTP monitor sees a 200 and moves on. A streaming-aware monitor needs to track whether tokens actually arrived and whether the stream completed.
Token-Based Pricing
Every monitoring probe you send to an LLM API costs money. A synthetic health check that sends a prompt and reads the response burns input and output tokens. If you run it every 60 seconds across multiple models, you are paying for monitoring on top of paying for inference. This creates a tension that does not exist with traditional APIs: the cost of observing the system is non-trivial compared to the cost of using it.
Model Versioning and Silent Behavior Changes
LLM providers update models without changing the API contract. OpenAI can update gpt-4o and your prompt that worked reliably for months starts returning different formatting, refusing certain inputs, or producing longer outputs that blow past your token budget. Your code did not change. Your tests pass. But production behavior shifted because the model underneath changed.
This is the equivalent of a database vendor silently changing query optimizer behavior in a minor patch -- except it happens regularly and affects the actual content your users see.
Cascading Failures Across Providers
Many teams implement fallback chains: try Claude, fall back to GPT-4, fall back to a smaller model. When the primary provider hits rate limits or degrades, traffic shifts to the fallback. But the fallback model might behave differently -- shorter context window, different formatting tendencies, different refusal patterns. Your app "stays up" but the user experience silently degrades. Monitoring the switch itself is as important as monitoring each provider.
Key Metrics to Track
Time to First Token (TTFT)
For streaming responses, TTFT is the metric your users feel. It measures how long they stare at a blank screen before text starts appearing. TTFT varies dramatically based on prompt length, model load, and whether the provider caches your system prompt. Track P50, P75, and P95 -- averages hide the tail latency that makes your app feel broken.
Typical ranges: 200-600ms for cached, short prompts. 2-15 seconds for long-context requests. If your P95 TTFT spikes from 2 seconds to 10, your users will notice before your dashboards do unless you are measuring this explicitly.
Total Response Time
End-to-end time from request sent to final token received (or full response for non-streaming). This depends heavily on output length, which you cannot predict or control. A prompt that usually generates 200 tokens might occasionally generate 2,000. Track this metric alongside output token count to distinguish between "the model is slow" and "the model is verbose."
Token Usage
Track input tokens and output tokens separately per request. Input tokens are predictable (you control the prompt), but output tokens are not. Watch for:
- Output token spikes that indicate the model is being more verbose than expected.
- Input token creep as conversation histories or RAG context grows.
- Requests hitting the context window limit, which causes truncated responses.
Error Rates by Type
Not all errors are equal. Break them down:
- Rate limit errors (429): You are sending too much traffic. Needs backoff logic or a higher tier.
- Context length errors (400): Your input is too long for the model. Needs prompt trimming.
- Content filter errors: The model or provider refused the request. May indicate prompt injection or edge-case inputs.
- Server errors (500/503): The provider is having issues. Nothing you can fix -- you need a fallback.
- Timeout errors: The request took too long. Could be your timeout is too aggressive for LLM latency, or the provider is under load.
Each error type has a different remediation path. Lumping them into a single "error rate" metric hides the signal.
Cost per Request and Spend Tracking
Multiply input tokens by the input price and output tokens by the output price for each request. Aggregate daily and weekly. Set alerts on spend thresholds. A single bad prompt template that generates excessive output can double your daily spend before anyone notices.
Response Quality Signals
This is where LLM monitoring diverges most from traditional API monitoring. A 200 response with valid JSON can still be a failure if:
- The model refused to answer ("I'm sorry, I can't help with that").
- The response does not match your expected schema or format.
- The output is abnormally short or long compared to baseline.
- The model hallucinated a function call or tool use that does not exist.
Track refusal rate, format compliance rate, and response length distribution as first-class metrics.
Monitoring Strategies
Synthetic Probes: Test Structure, Not Content
Send known prompts on a schedule and validate the response structure rather than the exact content. A good synthetic probe for an LLM API:
import time
import openai
def llm_health_check():
"""Synthetic probe: validate the LLM responds with expected structure."""
start = time.time()
try:
response = openai.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "Respond with valid JSON only."},
{"role": "user", "content": 'Return: {"status": "ok", "model": "<your model name>"}'}
],
max_tokens=50,
temperature=0,
timeout=10
)
elapsed = time.time() - start
content = response.choices[0].message.content
# Validate structure, not exact content
import json
parsed = json.loads(content)
assert "status" in parsed, "Missing 'status' field"
return {
"healthy": True,
"latency_ms": round(elapsed * 1000),
"input_tokens": response.usage.prompt_tokens,
"output_tokens": response.usage.completion_tokens,
}
except openai.RateLimitError:
return {"healthy": False, "error": "rate_limited"}
except openai.APITimeoutError:
return {"healthy": False, "error": "timeout"}
except (json.JSONDecodeError, AssertionError) as e:
return {"healthy": False, "error": f"invalid_response: {e}"}
except openai.APIError as e:
return {"healthy": False, "error": f"api_error: {e.status_code}"}
A similar probe for Anthropic:
import anthropic
import time
def anthropic_health_check():
"""Synthetic probe for Anthropic Claude API."""
client = anthropic.Anthropic()
start = time.time()
try:
message = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=50,
messages=[
{"role": "user", "content": 'Respond with only: {"status": "ok"}'}
]
)
elapsed = time.time() - start
content = message.content[0].text
import json
parsed = json.loads(content)
return {
"healthy": True,
"latency_ms": round(elapsed * 1000),
"input_tokens": message.usage.input_tokens,
"output_tokens": message.usage.output_tokens,
}
except anthropic.RateLimitError:
return {"healthy": False, "error": "rate_limited"}
except anthropic.APITimeoutError:
return {"healthy": False, "error": "timeout"}
except Exception as e:
return {"healthy": False, "error": str(e)}
Keep the prompts short and set max_tokens low to minimize cost. Use temperature=0 where supported to reduce response variance. Run these every 2-5 minutes rather than every 60 seconds -- the cost adds up and LLM providers rarely recover from outages in under a minute anyway.
Canary Deployments for Prompt Changes
Treat prompt changes like code deployments. When you update a system prompt or few-shot examples, roll it out to a small percentage of traffic first and monitor:
- Response format compliance rate
- Average output token count (did the new prompt make the model more verbose?)
- Refusal rate
- User-facing error rate
A prompt change that looks fine in testing can behave differently under the distribution of real user inputs.
Provider Status Monitoring
Before you debug your code, check whether your LLM provider is having an incident. OpenAI and Anthropic both publish status pages, but checking them manually during an incident wastes time.
Automate it. Most provider status pages expose a JSON API:
# Check OpenAI status
curl -s https://status.openai.com/api/v2/summary.json | jq '.status'
# Check Anthropic status
curl -s https://status.anthropic.com/api/v2/summary.json | jq '.status'
Better yet, use a service that monitors these status pages for you and alerts your team when a provider reports degraded performance. This is exactly the kind of dependency monitoring that matters -- knowing that OpenAI is degraded 30 seconds after it starts saves you from a 45-minute debugging rabbit hole.
Circuit Breakers and Fallback Monitoring
Implement circuit breakers that trip when error rates or latency exceed thresholds:
class LLMCircuitBreaker:
def __init__(self, failure_threshold=5, recovery_timeout=60):
self.failure_count = 0
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.state = "closed" # closed, open, half-open
self.last_failure_time = None
def record_failure(self):
self.failure_count += 1
self.last_failure_time = time.time()
if self.failure_count >= self.failure_threshold:
self.state = "open"
def record_success(self):
self.failure_count = 0
self.state = "closed"
def can_execute(self):
if self.state == "closed":
return True
if self.state == "open":
if time.time() - self.last_failure_time > self.recovery_timeout:
self.state = "half-open"
return True
return False
return True # half-open: allow one request through
The critical monitoring point: track when circuit breakers trip and when traffic shifts to fallbacks. A circuit breaker that trips daily at 2pm PT might indicate a recurring capacity issue at your provider. A fallback that activates for 10% of requests is a cost and quality signal that deserves its own alert.
Putting It Together: A Practical Monitoring Stack
LLM monitoring is not one tool. It is three layers working together:
Layer 1 -- Endpoint Reachability. Is the provider's API reachable? Can you establish a connection? This is traditional uptime monitoring. Check every 1-2 minutes from multiple regions. Alert on consecutive failures.
Layer 2 -- Response Validation. Does the API return a well-formed response that matches your expected structure? This is your synthetic probe layer. Run every 2-5 minutes with lightweight prompts. Validate JSON structure, check for refusals, measure TTFT and total latency.
Layer 3 -- Dependency Health. Is the provider reporting any incidents? Are other customers seeing issues? This is status page monitoring. Subscribe to provider status pages and get alerted when they report degraded performance or active incidents.
Most teams start with Layer 1 and stop there. But an LLM API that returns 200 with garbage is arguably worse than one that returns 503 -- at least the 503 triggers your error handling.
How Alert24 Fits
Alert24 covers all three layers. Set up HTTP monitors on your AI-powered endpoints to check reachability and response time. Use response body validation to verify that your LLM endpoints return properly structured output. And use Alert24's directory of 2,000+ third-party status pages -- including OpenAI, Anthropic, Google AI, and other providers -- to get alerted the moment your AI dependencies report an incident.
When your LLM provider goes down at 3am, your on-call engineer gets an alert from Alert24 that tells them the root cause is upstream, not in your code. That context alone can cut incident response time from 45 minutes of confused debugging to 5 minutes of activating your fallback plan.
Combine uptime monitoring for your endpoints, response validation for your LLM integrations, and dependency monitoring for your providers. One platform, three layers of coverage for the AI infrastructure your product depends on.
The Monitoring Mindset Shift
The fundamental shift is this: with traditional APIs, monitoring answers "is it up?" With LLM APIs, monitoring answers "is it up, is it fast enough, is it too expensive, and is it still behaving the way we expect?"
That is four questions instead of one, and each requires different instrumentation. Teams that treat LLM monitoring as a standard uptime check will miss the slow degradations -- the model that gradually gets more verbose, the provider that quietly increases latency during peak hours, the prompt that works until it encounters an input in a language it was not tested with.
Start with the basics. Monitor reachability and latency. Add response structure validation. Track token costs. Watch your providers' status pages. Then build up to quality signals as your LLM integration matures.
The non-determinism is not going away. Your monitoring strategy should account for it from day one.