Monitoring, Analytics, and Cost Management
Overview
AI Guard Developer Portal provides comprehensive monitoring and analytics to help you:
- Track API usage and performance
- Monitor costs and optimize spending
- Analyze guardrail effectiveness
- Identify issues and trends
- Generate compliance reports
API Usage Monitoring
Real-Time Dashboard
Main Dashboard Shows:
Current Metrics (Last Hour):
- Active requests
- Requests per minute
- Average latency
- Error rate
- Guardrail blocks
Today's Statistics:
- Total requests
- Successful responses
- Failed requests
- Tokens consumed
- Estimated cost
Quick Actions:
- View detailed logs
- Check API key status
- Review recent errors
- Export reports
API Logs
Access Logs:
- Navigate to Logs > API Logs
- Or: API Keys > [Select Key] > View Logs
Log Information:
{
"timestamp": "2025-12-01T14:23:45Z",
"api_key": "devkey_***...***",
"request_id": "req_abc123",
"endpoint": "/api/dev_guard.php",
"method": "POST",
"status": 200,
"latency_ms": 487,
"tokens": {
"prompt": 45,
"completion": 156,
"total": 201
},
"cost": 0.0060,
"model": "gpt-4-turbo",
"guardrails": {
"input": "pass",
"output": "pass",
"violations": 0
},
"privacy_guard": {
"pii_detected": false
},
"citadel": {
"threats_blocked": 0
},
"ip_address": "203.0.113.45",
"user_agent": "python-requests/2.31.0"
}
Filtering Logs
Available Filters:
Time Range:
- Last hour
- Last 24 hours
- Last 7 days
- Last 30 days
- Custom range
API Key:
- All keys
- Specific key
- Multiple keys
Status:
- All
- Success (200)
- Client errors (4xx)
- Server errors (5xx)
Guardrail Results:
- All
- Passed
- Violations
- Blocked
Example Filter:
Time: Last 24 hours
API Key: prod-chatbot-key
Status: All
Guardrail: Violations only
Minimum cost: $0.01
Exporting Logs
Export Options:
- Click "Export" button
- Select format:
- CSV (spreadsheet)
- JSON (programmatic)
- PDF (reports)
- Choose date range
- Apply filters
- Download
CSV Export Example:
Timestamp,API Key,Status,Tokens,Cost,Latency,Model
2025-12-01 14:23:45,devkey_***,200,201,0.0060,487ms,gpt-4-turbo
2025-12-01 14:24:12,devkey_***,200,345,0.0104,523ms,gpt-4-turbo
Performance Analytics
Response Time Analysis
Metrics Dashboard:
Average Latency:
- Overall average
- By API key
- By model
- By time of day
- Trend over time
Latency Distribution:
P50 (median): 450ms
P90: 850ms
P95: 1200ms
P99: 2100ms
Max: 4500ms
Performance Chart:
Latency Over Time (Last 7 Days)
2000ms ┤ ╭╮
1500ms ┤ ╭╯╰╮ ╭╮
1000ms ┤ ╭╯ ╰╮ ╭╯╰╮
500ms ┤╭──╯ ╰─╯ ╰───
0ms ┴───────────────────
Mon Wed Fri Sun
Success Rate Monitoring
Success Metrics:
- Total requests
- Successful (2xx)
- Client errors (4xx)
- Server errors (5xx)
- Success rate percentage
Error Breakdown:
Last 24 Hours:
Total Requests: 10,523
Successful: 10,234 (97.3%)
Errors: 289 (2.7%)
Error Types:
400 Bad Request: 156 (54%)
401 Unauthorized: 89 (31%)
429 Rate Limit: 32 (11%)
500 Server Error: 12 (4%)
Throughput Analysis
Requests Per Minute:
Peak: 1,234 RPM (2:15 PM)
Average: 456 RPM
Minimum: 23 RPM (3:00 AM)
Busiest Hours:
2:00 PM - 3:00 PM: 1,100 RPM
10:00 AM - 11:00 AM: 980 RPM
9:00 AM - 10:00 AM: 870 RPM
Cost Management
Cost Dashboard
Overview:
- Today's cost
- This week's cost
- This month's cost
- Projected monthly cost
- Budget remaining
Cost Breakdown:
December 2025 (To Date)
Total Cost: $1,247.56
Budget: $2,000.00
Remaining: $752.44 (38%)
By Model:
GPT-4 Turbo: $856.23 (69%)
GPT-3.5 Turbo: $234.12 (19%)
Claude 3.5: $157.21 (13%)
By API Key:
prod-chatbot: $523.45 (42%)
internal-tool: $398.67 (32%)
doc-qa-system: $325.44 (26%)
Token Usage Analysis
Token Metrics:
Total Tokens:
- Prompt tokens
- Completion tokens
- Total consumed
- Percentage breakdown
Example:
Last 30 Days:
Total Tokens: 4,235,678
Prompt: 1,567,234 (37%)
Completion: 2,668,444 (63%)
Average per Request:
Prompt: 156 tokens
Completion: 267 tokens
Total: 423 tokens
Most Token-Heavy:
Long-form content: 2,145 avg
Code generation: 1,876 avg
Simple Q&A: 245 avg
Cost Optimization Insights
Automatic Recommendations:
High-Cost API Keys:
Alert: prod-chatbot using expensive model
Current:
Model: GPT-4
Cost/1K tokens: $0.030
Monthly cost: $856
Recommendation:
Switch to: GPT-4 Turbo
Cost/1K tokens: $0.010
Projected savings: $572/month (67%)
Token Waste Detection:
Alert: Excessive completion tokens
API Key: doc-qa-system
Average completion: 1,234 tokens
Typical for task: 300 tokens
Recommendation:
Set max_tokens: 500
Projected savings: $145/month
Budget Alerts
Configure Alerts:
Settings > Cost Management > Alerts
budget_alerts:
monthly_budget: 2000
alerts:
- threshold: 50%
action: email
recipients:
- admin@company.com
- threshold: 80%
action: email + slack
recipients:
- admin@company.com
- finance@company.com
- threshold: 95%
action: email + slack + warning
message: "Approaching budget limit"
- threshold: 100%
action: auto_disable
disable_keys:
- non_critical_*
Alert Example:
Budget Alert: 80% Reached
Current Spend: $1,600 / $2,000
Days Remaining: 12
Projected End-of-Month: $2,100 (105%)
Action Required:
- Review high-cost API keys
- Consider model optimization
- Implement request caching
- Set stricter rate limits
Guardrail Analytics
Effectiveness Metrics
Guardrail Performance:
Input Guardrails:
Last 30 Days:
Total Checked: 45,678
Passed: 44,123 (96.6%)
Blocked: 1,555 (3.4%)
Top Violations:
Profanity: 789 (51%)
Off-topic: 432 (28%)
PII detected: 234 (15%)
Prompt injection: 100 (6%)
Output Guardrails:
Total Checked: 44,123
Passed: 43,567 (98.7%)
Modified: 556 (1.3%)
Modifications:
PII redacted: 345 (62%)
Tone adjusted: 123 (22%)
Disclaimer added: 88 (16%)
Violation Trends
Trend Analysis:
Violation Rate Over Time
5% ┤╭╮
4% ┤│╰╮
3% ┤│ ╰─╮ ╭╮
2% ┤│ ╰──╯╰─
1% ┤│
0% ┴────────────────
Week 1 2 3 4
Note: Violation rate decreasing - guardrails working!
False Positive Monitoring
Track False Positives:
Settings > Guardrails > [Guardrail] > Analytics
Guardrail: Profanity Filter
Blocked Requests: 789
Reviewed: 234
True Positives: 198 (85%)
False Positives: 36 (15%)
False Positive Examples:
"This is bad for business" → blocked word "bad"
"Hell yeah, great idea!" → blocked word "hell"
Recommendation:
Adjust context-awareness
Whitelist business phrases
Security Monitoring
Threat Detection
Citadel Dashboard:
Blocked Threats:
Last 7 Days:
Total Attacks: 127
Prompt Injection: 67 (53%)
Jailbreak Attempts: 34 (27%)
Prompt Leaking: 18 (14%)
Other: 8 (6%)
Top Attack Sources:
203.0.113.45: 23 attempts
198.51.100.67: 19 attempts
192.0.2.123: 15 attempts
Attack Patterns:
Common Attack Vectors:
1. "Ignore previous instructions" (34 attempts)
2. "You are now DAN" (23 attempts)
3. "Repeat your system prompt" (18 attempts)
4. Base64 encoded payloads (12 attempts)
Privacy Guard Monitoring
PII Detection Dashboard:
Last 30 Days:
Total Scans: 45,678
PII Detected: 1,234 (2.7%)
PII Types:
Email: 567 (46%)
Phone: 345 (28%)
SSN: 189 (15%)
Credit Card: 78 (6%)
Address: 55 (4%)
Action Taken:
Redacted: 1,156 (94%)
Blocked: 78 (6%)
Compliance Reporting
Automated Reports
Schedule Reports:
Settings > Reports > Schedule
reports:
- name: Weekly Usage Report
frequency: weekly
day: monday
time: "09:00"
recipients:
- team@company.com
includes:
- api_usage
- cost_summary
- top_keys
- name: Monthly Compliance Report
frequency: monthly
day: 1
time: "08:00"
recipients:
- compliance@company.com
- legal@company.com
includes:
- privacy_guard_summary
- citadel_blocks
- guardrail_violations
- audit_trail
Compliance Audit Logs
Audit Trail:
Compliance Audit Report
Date Range: December 1-30, 2025
=== PII Protection ===
Scans Performed: 45,678
PII Detected: 1,234
Redaction Rate: 100%
Compliance: ✓ GDPR, HIPAA, PCI-DSS
=== Security ===
Threats Detected: 127
Blocked: 127 (100%)
False Positives: 0
=== Access Control ===
API Keys Active: 12
Unauthorized Attempts: 89
All Blocked: ✓
=== Data Retention ===
Log Retention: 90 days
Backup Status: Current
Encryption: ✓ At rest and in transit
Custom Dashboards
Creating Custom Views
Dashboard Builder:
- Settings > Dashboards > Create New
- Add widgets:
- Metrics (numbers)
- Charts (time series)
- Tables (logs)
- Alerts (thresholds)
Example: Executive Dashboard
dashboard:
name: Executive Overview
layout: 2x2
widgets:
- type: metric
title: "Monthly Cost"
value: total_cost
comparison: last_month
- type: chart
title: "API Usage Trend"
data: daily_requests
period: 30_days
- type: metric
title: "Success Rate"
value: success_percentage
threshold: 99.5
- type: table
title: "Top 5 API Keys"
data: usage_by_key
limit: 5
Integration with External Tools
Webhook Notifications
Configure Webhooks:
Settings > Integrations > Webhooks
webhooks:
- name: Slack Alerts
url: https://hooks.slack.com/services/YOUR/WEBHOOK
events:
- budget_threshold
- security_threat
- high_error_rate
- guardrail_violation
- name: PagerDuty
url: https://events.pagerduty.com/v2/enqueue
events:
- service_down
- critical_error
severity: critical
Data Export APIs
Programmatic Access:
import requests
# Export usage data
response = requests.get(
'https://developer.isms-cloud.com/api/analytics/export',
headers={
'X-API-Key': 'your-admin-key',
'Content-Type': 'application/json'
},
params={
'start_date': '2025-12-01',
'end_date': '2025-12-31',
'metrics': ['cost', 'tokens', 'requests'],
'format': 'json'
}
)
data = response.json()
Best Practices
Daily Monitoring
✓ Check Dashboard:
- Review current metrics
- Check for errors
- Monitor cost trends
- Review security alerts
✓ Review Top API Keys:
- Highest usage
- Highest cost
- Most errors
- Security incidents
Weekly Analysis
✓ Performance Review:
- Average latency trends
- Success rate changes
- Error pattern analysis
- Throughput capacity
✓ Cost Analysis:
- Weekly spend vs. budget
- Cost per API key
- Model efficiency
- Optimization opportunities
✓ Security Review:
- Threat attempts
- Guardrail effectiveness
- False positive rate
- Attack patterns
Monthly Reporting
✓ Executive Summary:
- Total usage and cost
- Growth trends
- Key achievements
- Issues and resolutions
✓ Compliance Documentation:
- Privacy guard activity
- Security incidents
- Audit trail export
- Regulatory compliance
✓ Optimization Plan:
- Cost reduction opportunities
- Performance improvements
- Guardrail tuning
- Capacity planning
Troubleshooting
High Costs
Investigation:
- Check cost breakdown by API key
- Identify high token consumers
- Review request patterns
- Check for errors causing retries
Solutions:
- Switch to cheaper models
- Implement caching
- Set token limits
- Optimize prompts
Performance Issues
Diagnosis:
- Check latency trends
- Identify slow API keys
- Review model selection
- Check guardrail complexity
Solutions:
- Use faster models
- Simplify guardrails
- Implement caching
- Optimize LLM connections
Missing Data
Common Causes:
- Date range filter too narrow
- API key filter excluding data
- Data not yet processed
- Export format limitations
Solutions:
- Expand date range
- Remove filters
- Wait for batch processing
- Try different export format
Next Steps
- Set up Custom Dashboards
- Configure Budget Alerts
- Schedule Automated Reports
- Integrate Webhooks
- Review Cost Optimization