# Critical Lesson: IBM TargetProcess Pagination Requirements
## Instance: usibmsandbox.tpondemand.com
### ACTUAL Entity Counts (January 2025)
- **Projects**: 768 (NOT 1000 limit)
- **Work Allocations**: 6,652 (NOT 1000 limit)
- **Users**: 1000+ estimated
- **Show All Response**: 44,298 tokens (too large for single response)
## The Pagination Problem
**NEVER assume the 1000 item default limit is the total count!**
### What Happens Without Pagination:
1. API returns first 1000 items by default
2. You miss 85% of work allocations (5,652 out of 6,652)
3. Reports show incorrect resource utilization
4. Critical allocations are invisible
5. Management makes decisions on incomplete data
### Correct Pagination Implementation
```python
def fetch_all_targetprocess_entities(entity_type, instance_url, api_key):
"""
ALWAYS paginate through ALL results for accurate data.
"""
all_entities = []
page_size = 1000 # API max per request
skip = 0
while True:
# Build request with pagination
params = {
"access_token": api_key,
"take": page_size,
"skip": skip,
"format": "json"
}
response = requests.get(f"{instance_url}/api/v1/{entity_type}", params=params)
data = response.json()
items = data.get("Items", [])
all_entities.extend(items)
print(f"Fetched {skip + len(items)} of unknown total {entity_type}...")
# Check if there are more pages
if len(items) < page_size:
break # Last page
skip += page_size
print(f"✅ Total {entity_type} fetched: {len(all_entities)}")
return all_entities
```
### Using MCP Tools with Pagination
```javascript
// Use show_more for pagination
let result = await search_entities({
type: 'WorkAllocation',
take: 100
});
// Check for pagination key
if (result.includes("show_more with key:")) {
// Extract key and fetch more
let key = extractKey(result);
let moreResults = await show_more({cacheKey: key});
}
// For complete data (but watch token limits!)
// show_all may exceed token limits with large datasets
```
## Resource Allocation Report Requirements
For accurate resource allocation reports:
1. **Fetch ALL work allocations** (6,652 in this instance)
2. **Fetch ALL active users** (likely 1000+)
3. **Fetch ALL projects** (768 in this instance)
4. **Cross-reference by Talent ID custom fields**
5. **Group by Product/Service (Project)**
6. **Calculate true utilization rates**
## Token Limit Considerations
When using `show_all` with large datasets:
- 6,652 work allocations = ~44,298 tokens (exceeds 25,000 limit)
- Solution: Use pagination with `show_more` or batch processing
- Process data in chunks to avoid token limits
## Key Takeaways
1. **Default ≠ Total**: The 1000 item default is just the first page
2. **Always Paginate**: Implement skip/take loops for complete data
3. **Verify Counts**: Check "X more items available" messages
4. **Token Limits**: Large datasets may exceed response token limits
5. **Incremental Processing**: Process in batches for large instances
## IBM Instance Specifics
- URL: `https://usibmsandbox.tpondemand.com`
- Authentication: API Key as `access_token` parameter
- Custom Fields: "Talent ID" and "Connected User Talent ID" for mapping
- Scale: Enterprise-level with thousands of entities
- Products/Services: 768 active projects to filter by
---
**Remember**: In IBM's TargetProcess instance, assuming the 1000 limit would mean missing 85% of work allocations. Always paginate!
