OPTIMIZATION_STRATEGY.md•37.6 kB
# JobNimbus MCP - Estrategia de Optimización de Transmisión de Datos
## Análisis de Estado Actual
### Problemas Identificados
**1. Transmisión de Datos Masiva**
- Jobs: 89+ campos (incluyendo JSONB arrays: related[], tags[], custom_fields{})
- Estimates: 35+ campos + items[] array con productos detallados
- Invoices: 40+ campos + items[], sections[], payments[] arrays
- Activities: Audit trails con historiales largos
- Contacts: 45+ campos con múltiples teléfonos y direcciones
**2. Implementación Actual (Parcial)**
```typescript
// YA IMPLEMENTADO (Fase 3):
- Handle-based response system (responseBuilder.ts)
- Verbosity levels: summary/compact/detailed/raw
- Field selection via ?fields=jnid,number,status
- Redis cache integration
- Response size limits (25 KB hard limit)
// PROBLEMAS RESTANTES:
- No todos los endpoints usan el sistema de handles
- Arrays JSONB grandes se transmiten completos
- Paginación no optimizada en todos los endpoints
- Falta compresión en respuestas grandes
```
### Volumetría de Datos (Estimaciones)
| Entidad | Campos Actuales | Promedio por Registro | 100 Registros |
|---------|----------------|----------------------|---------------|
| Jobs | 89 campos | ~6 KB | ~600 KB |
| Estimates | 35 campos + items[] | ~8 KB | ~800 KB |
| Invoices | 40 campos + arrays | ~10 KB | ~1,000 KB |
| Activities | ~25 campos | ~2 KB | ~200 KB |
| Contacts | 45 campos | ~3 KB | ~300 KB |
---
## ESTRATEGIA 1: Field Selection/Projection (GraphQL-like)
### Implementación Actual
```typescript
// src/utils/responseBuilder.ts (YA IMPLEMENTADO)
public static selectFields(data: any, fields: string[]): any {
const fieldSet = new Set(fields.map(f => f.trim()));
if (Array.isArray(data)) {
return data.map(item => this.selectFieldsFromObject(item, fieldSet));
}
return this.selectFieldsFromObject(data, fieldSet);
}
```
### Mejoras Propuestas
**1.1 Nested Field Selection (Selección Profunda)**
```typescript
// NUEVO: Soporte para campos anidados
// Uso: ?fields=jnid,number,primary.name,owners[].id,items[].name,items[].price
interface NestedFieldSelector {
selectNestedFields(data: any, fieldPaths: string[]): any;
}
export class NestedFieldSelector {
/**
* Seleccionar campos con notación de punto y arrays
* @example
* fields = ['primary.name', 'items[].name', 'items[].price']
* Resultado: { primary: { name: 'John' }, items: [{ name: 'Shingles', price: 100 }] }
*/
selectNestedFields(data: any, fieldPaths: string[]): any {
const schema = this.buildFieldSchema(fieldPaths);
return this.extractData(data, schema);
}
private buildFieldSchema(fieldPaths: string[]): FieldSchema {
const schema: FieldSchema = { fields: {}, arrays: {} };
for (const path of fieldPaths) {
const parts = path.split('.');
let current = schema.fields;
for (let i = 0; i < parts.length; i++) {
const part = parts[i];
const isArray = part.endsWith('[]');
const fieldName = isArray ? part.slice(0, -2) : part;
if (isArray) {
// Crear esquema para array
if (!schema.arrays[fieldName]) {
schema.arrays[fieldName] = [];
}
// Siguiente parte es un campo del array
if (i + 1 < parts.length) {
schema.arrays[fieldName].push(parts[i + 1]);
}
} else if (i === parts.length - 1) {
// Campo final
current[fieldName] = true;
} else {
// Crear objeto anidado
if (!current[fieldName]) {
current[fieldName] = {};
}
current = current[fieldName];
}
}
}
return schema;
}
private extractData(data: any, schema: FieldSchema): any {
if (Array.isArray(data)) {
return data.map(item => this.extractData(item, schema));
}
if (typeof data !== 'object' || data === null) {
return data;
}
const result: any = {};
// Extraer campos simples
for (const [key, value] of Object.entries(schema.fields)) {
if (value === true) {
// Campo simple
result[key] = data[key];
} else {
// Objeto anidado
result[key] = this.extractData(data[key], { fields: value, arrays: {} });
}
}
// Extraer arrays
for (const [arrayKey, arrayFields] of Object.entries(schema.arrays)) {
if (data[arrayKey] && Array.isArray(data[arrayKey])) {
result[arrayKey] = data[arrayKey].map((item: any) => {
const extracted: any = {};
for (const field of arrayFields) {
extracted[field] = item[field];
}
return extracted;
});
}
}
return result;
}
}
interface FieldSchema {
fields: Record<string, any>;
arrays: Record<string, string[]>;
}
```
**Ejemplo de Uso:**
```typescript
// Antes (89 campos, ~6 KB):
GET /jobs/123
// Retorna: { jnid, recid, number, type, customer, created_by, ... (89 campos) }
// Después (5 campos, ~0.3 KB):
GET /jobs/123?fields=jnid,number,status_name,primary.name,approved_estimate_total
// Retorna: {
// jnid: "abc123",
// number: "1820",
// status_name: "Lead",
// primary: { name: "John Doe" },
// approved_estimate_total: 15000
// }
// Reducción: 95% (6 KB → 0.3 KB)
```
**1.2 Field Presets (Conjuntos Predefinidos)**
```typescript
// src/config/fieldPresets.ts
export const FIELD_PRESETS = {
// Presets para Jobs
jobs: {
minimal: 'jnid,number,status_name',
basic: 'jnid,number,name,status_name,address_line1,city,state_text',
financial: 'jnid,number,approved_estimate_total,approved_invoice_total,sales_rep_name',
scheduling: 'jnid,number,date_start,date_end,status_name,address_line1',
complete: '*', // Todos los campos
},
// Presets para Estimates
estimates: {
minimal: 'jnid,number,total,status_name',
basic: 'jnid,number,total,status_name,date_created,customer.name',
items_summary: 'jnid,number,total,items[].name,items[].quantity,items[].price',
complete: '*',
},
// Presets para Invoices
invoices: {
minimal: 'jnid,number,total,status_name',
financial: 'jnid,number,total,subtotal,tax,payments[].amount,payments[].date',
complete: '*',
}
};
// Uso: GET /jobs?preset=financial
// Expande a: ?fields=jnid,number,approved_estimate_total,approved_invoice_total,sales_rep_name
```
**Reducción Estimada de Datos: 80-95%**
---
## ESTRATEGIA 2: Response Summarization (Verbosity Levels)
### Implementación Actual (FUNCIONAL)
```typescript
// src/config/response.ts (YA IMPLEMENTADO)
export const RESPONSE_CONFIG = {
VERBOSITY: {
DEFAULT: 'compact',
SUMMARY_MAX_FIELDS: 5, // Ultra-minimal
COMPACT_MAX_FIELDS: 15, // Default
DETAILED_MAX_FIELDS: 50, // Comprehensive
}
};
```
### Mejoras Propuestas
**2.1 Smart Field Selection per Verbosity**
```typescript
// NUEVO: Selección inteligente de campos por verbosidad
export const SMART_FIELD_SELECTION = {
jobs: {
summary: {
// Solo 5 campos críticos
fields: ['jnid', 'number', 'status_name', 'name', 'date_created'],
arrays: {}, // Sin arrays
},
compact: {
// 15 campos esenciales
fields: [
'jnid', 'number', 'name', 'status_name', 'address_line1',
'city', 'state_text', 'sales_rep_name', 'date_created',
'approved_estimate_total', 'approved_invoice_total',
'date_start', 'date_end', 'primary.name', 'attachment_count'
],
arrays: {
owners: ['id'], // Solo IDs, no objetos completos
tags: [], // Vacío = solo count
},
},
detailed: {
// 50 campos más importantes
fields: [
// Todos los de compact +
'description', 'source_name', 'created_by_name', 'geo',
'date_updated', 'date_status_change', 'last_estimate',
'last_invoice', 'work_order_total', 'is_active',
// ... (hasta 50 campos)
],
arrays: {
owners: ['id', 'name'], // IDs + nombres
tags: ['id', 'name'],
related: ['id', 'type', 'name'],
},
},
raw: {
// Todo sin filtrar
fields: '*',
arrays: '*',
}
},
estimates: {
summary: ['jnid', 'number', 'total', 'status_name', 'date_created'],
compact: [
'jnid', 'number', 'name', 'total', 'subtotal', 'tax',
'status_name', 'date_created', 'date_sent', 'date_approved',
'customer.name', 'job.number', 'sales_rep_name', 'items_count',
'margin'
],
// items[] array handling
items_compact: {
max_items: 5, // Solo primeros 5 items
fields: ['name', 'quantity', 'price', 'total'],
},
items_detailed: {
max_items: 20,
fields: ['name', 'description', 'quantity', 'uom', 'price', 'cost', 'total'],
}
}
};
// Aplicar selección inteligente
export function applySmartVerbosity(
entity: string,
data: any,
verbosity: VerbosityLevel
): any {
const selection = SMART_FIELD_SELECTION[entity]?.[verbosity];
if (!selection) return data;
// Seleccionar campos
const result = selectFields(data, selection.fields);
// Procesar arrays
if (selection.arrays) {
for (const [arrayKey, arrayFields] of Object.entries(selection.arrays)) {
if (data[arrayKey] && Array.isArray(data[arrayKey])) {
if (arrayFields.length === 0) {
// Solo count
result[`${arrayKey}_count`] = data[arrayKey].length;
delete result[arrayKey];
} else {
// Campos seleccionados
result[arrayKey] = data[arrayKey].map((item: any) =>
selectFields(item, arrayFields)
);
}
}
}
}
return result;
}
```
**Ejemplo de Resultados:**
```typescript
// Job con 89 campos totales
const rawJob = { /* 89 campos */ };
// SUMMARY (5 campos, ~0.2 KB)
{
jnid: "abc123",
number: "1820",
status_name: "Lead",
name: "Roof Repair - 123 Main St",
date_created: "2025-01-15"
}
// COMPACT (15 campos, ~0.8 KB)
{
jnid: "abc123",
number: "1820",
name: "Roof Repair - 123 Main St",
status_name: "Lead",
address_line1: "123 Main St",
city: "Stamford",
state_text: "CT",
sales_rep_name: "John Smith",
date_created: "2025-01-15",
approved_estimate_total: 15000,
approved_invoice_total: 0,
date_start: null,
date_end: null,
primary: { name: "Jane Doe" },
attachment_count: 5,
owners_count: 2, // Solo count, no array completo
tags_count: 3
}
// DETAILED (50 campos, ~3 KB)
{
// Todos los de COMPACT +
description: "Full roof replacement...",
geo: { lat: 41.0534, lon: -73.5387 },
owners: [
{ id: "owner1", name: "John Smith" },
{ id: "owner2", name: "Mary Johnson" }
],
tags: [
{ id: "tag1", name: "Insurance" },
{ id: "tag2", name: "Emergency" }
],
// ... (hasta 50 campos)
}
// RAW (89 campos, ~6 KB)
{ /* Todos los campos sin filtrar */ }
```
**Reducción de Datos:**
- Summary: 97% (6 KB → 0.2 KB)
- Compact: 87% (6 KB → 0.8 KB)
- Detailed: 50% (6 KB → 3 KB)
---
## ESTRATEGIA 3: Lazy Loading & Pagination Optimizada
### 3.1 Cursor-Based Pagination (Ya Implementado Parcialmente)
**Mejora: Cursor con Metadata Completa**
```typescript
// src/types/pagination.ts
export interface CursorPaginationResult<T> {
data: T[];
pagination: {
cursor: string | null; // Cursor para siguiente página
has_more: boolean; // Si hay más resultados
total_count?: number; // Total de resultados (opcional, costoso)
current_page_size: number; // Tamaño de página actual
next_cursor?: string; // Cursor siguiente (alias)
prev_cursor?: string; // Cursor anterior (para navegación bidireccional)
};
metadata: {
fetch_time_ms: number; // Tiempo de fetching
cache_hit: boolean; // Si vino de cache
verbosity: VerbosityLevel; // Nivel de detalle
};
}
// Implementación de cursor bidireccional
export class CursorPaginator {
/**
* Genera cursor opaco desde índice y timestamp
* Formato: base64(index:timestamp:hash)
*/
static encodeCursor(index: number, timestamp: number): string {
const payload = `${index}:${timestamp}`;
const hash = this.generateHash(payload);
return Buffer.from(`${payload}:${hash}`).toString('base64');
}
static decodeCursor(cursor: string): { index: number; timestamp: number } | null {
try {
const decoded = Buffer.from(cursor, 'base64').toString('utf8');
const [index, timestamp, hash] = decoded.split(':');
// Validar hash
if (hash !== this.generateHash(`${index}:${timestamp}`)) {
throw new Error('Invalid cursor hash');
}
return {
index: parseInt(index, 10),
timestamp: parseInt(timestamp, 10)
};
} catch {
return null;
}
}
private static generateHash(payload: string): string {
// Simple hash para validación
return payload.split('').reduce((acc, char) =>
((acc << 5) - acc) + char.charCodeAt(0), 0
).toString(36);
}
/**
* Paginar con cursor bidireccional
*/
static async paginate<T>(
data: T[],
cursor: string | null,
pageSize: number,
direction: 'forward' | 'backward' = 'forward'
): Promise<CursorPaginationResult<T>> {
const startTime = Date.now();
let startIndex = 0;
if (cursor) {
const decoded = this.decodeCursor(cursor);
if (!decoded) {
throw new Error('Invalid cursor');
}
startIndex = direction === 'forward' ? decoded.index : Math.max(0, decoded.index - pageSize);
}
const endIndex = Math.min(startIndex + pageSize, data.length);
const pageData = data.slice(startIndex, endIndex);
const hasMore = endIndex < data.length;
return {
data: pageData,
pagination: {
cursor: hasMore ? this.encodeCursor(endIndex, Date.now()) : null,
has_more: hasMore,
current_page_size: pageData.length,
next_cursor: hasMore ? this.encodeCursor(endIndex, Date.now()) : undefined,
prev_cursor: startIndex > 0 ? this.encodeCursor(Math.max(0, startIndex - pageSize), Date.now()) : undefined,
},
metadata: {
fetch_time_ms: Date.now() - startTime,
cache_hit: false,
verbosity: 'compact',
}
};
}
}
```
**Ejemplo de Uso:**
```typescript
// Primera página
GET /jobs?page_size=20
Response: {
data: [...20 jobs...],
pagination: {
cursor: "eyJpbmRleCI6MjAsInRpbWVzdGFtcCI6MTczNjc4MDAwMDAwMCwiaGFzaCI6ImFiYzEyMyJ9",
has_more: true,
current_page_size: 20
}
}
// Segunda página
GET /jobs?cursor=eyJpbmRleCI6MjAsInRpbWVzdGFtcCI6MTczNjc4MDAwMDAwMCwiaGFzaCI6ImFiYzEyMyJ9&page_size=20
Response: {
data: [...20 jobs...],
pagination: {
cursor: "eyJpbmRleCI6NDAsInRpbWVzdGFtcCI6MTczNjc4MDAwMDAwMCwiaGFzaCI6ImRlZjQ1NiJ9",
has_more: true,
prev_cursor: "eyJpbmRleCI6MCwic...",
current_page_size: 20
}
}
```
### 3.2 Lazy Loading de Arrays JSONB
**Problema Actual:**
```json
// Estimate con items[] completo
{
"jnid": "est123",
"items": [
{
"jnid": "item1",
"name": "Architectural Shingles",
"description": "Premium architectural shingles...",
"quantity": 25,
"uom": "SQ",
"price": 95.50,
"cost": 65.00,
"category": "Roofing",
"color": "Charcoal",
"photos": [...],
"tax_name": "CT Sales Tax",
"tax_rate": 0.0635
// ... 15+ campos por item
},
// ... x50 items = ~20 KB solo en items[]
]
}
```
**Solución: Referencias + Lazy Loading**
```typescript
// src/utils/lazyArrayLoader.ts
export interface LazyArrayReference {
_type: 'lazy_array';
entity: string; // 'estimate_items', 'invoice_items', etc.
parent_id: string; // Parent JNID
count: number; // Total de elementos
summary?: any[]; // Primeros N elementos (preview)
load_url: string; // Endpoint para cargar completo
handle?: string; // Handle si está en storage
}
export class LazyArrayLoader {
/**
* Crear referencia lazy para array grande
*/
static createReference(
entity: string,
parentId: string,
items: any[],
options: {
previewCount?: number;
verbosity?: VerbosityLevel;
} = {}
): LazyArrayReference {
const previewCount = options.previewCount || 3;
const verbosity = options.verbosity || 'summary';
// Crear preview compacto
const summary = items.slice(0, previewCount).map(item =>
this.compactItem(item, verbosity)
);
return {
_type: 'lazy_array',
entity,
parent_id: parentId,
count: items.length,
summary,
load_url: `/api/${entity}?parent_id=${parentId}`,
};
}
private static compactItem(item: any, verbosity: VerbosityLevel): any {
// Selección inteligente según verbosidad
switch (verbosity) {
case 'summary':
return {
jnid: item.jnid,
name: item.name,
quantity: item.quantity,
price: item.price,
};
case 'compact':
return {
jnid: item.jnid,
name: item.name,
quantity: item.quantity,
uom: item.uom,
price: item.price,
cost: item.cost,
total: (item.quantity || 0) * (item.price || 0),
};
default:
return item;
}
}
/**
* Determinar si array debe ser lazy
*/
static shouldBeLazy(items: any[], threshold: number = 10): boolean {
return items.length > threshold;
}
}
```
**Ejemplo de Respuesta:**
```json
// ANTES (con 50 items, ~25 KB)
{
"jnid": "est123",
"items": [ /* 50 items completos */ ]
}
// DESPUÉS (lazy loading, ~2 KB)
{
"jnid": "est123",
"items": {
"_type": "lazy_array",
"entity": "estimate_items",
"parent_id": "est123",
"count": 50,
"summary": [
{ "jnid": "item1", "name": "Architectural Shingles", "quantity": 25, "price": 95.50 },
{ "jnid": "item2", "name": "Underlayment", "quantity": 30, "price": 15.00 },
{ "jnid": "item3", "name": "Ridge Cap", "quantity": 50, "price": 8.50 }
],
"load_url": "/api/estimate_items?parent_id=est123",
"handle": "jn:estimate_items:est123:1736780000:abc123"
}
}
// Cargar completo cuando sea necesario:
GET /api/estimate_items?parent_id=est123
// o
GET /api/fetch_by_handle?handle=jn:estimate_items:est123:1736780000:abc123
```
**Reducción de Datos: 92% (25 KB → 2 KB)**
---
## ESTRATEGIA 4: Data Compression (Compresión HTTP)
### 4.1 Gzip Compression Middleware
```typescript
// src/middleware/compression.ts
import { Request, Response, NextFunction } from 'express';
import zlib from 'zlib';
export interface CompressionOptions {
threshold: number; // Mínimo tamaño para comprimir (bytes)
level: number; // Nivel de compresión (0-9)
memLevel: number; // Nivel de memoria (1-9)
}
export class CompressionMiddleware {
private static DEFAULT_OPTIONS: CompressionOptions = {
threshold: 1024, // Comprimir responses > 1 KB
level: 6, // Balance entre velocidad y ratio
memLevel: 8, // Balance memoria/velocidad
};
/**
* Middleware de compresión HTTP
*/
static compress(options: Partial<CompressionOptions> = {}) {
const opts = { ...this.DEFAULT_OPTIONS, ...options };
return (req: Request, res: Response, next: NextFunction) => {
// Verificar si cliente acepta compresión
const acceptEncoding = req.headers['accept-encoding'] || '';
const supportsGzip = acceptEncoding.includes('gzip');
if (!supportsGzip) {
return next();
}
// Interceptar res.json para comprimir
const originalJson = res.json.bind(res);
res.json = function (data: any) {
const json = JSON.stringify(data);
const sizeBytes = Buffer.byteLength(json, 'utf8');
// Comprimir solo si supera threshold
if (sizeBytes < opts.threshold) {
return originalJson(data);
}
// Comprimir con gzip
zlib.gzip(
Buffer.from(json, 'utf8'),
{
level: opts.level,
memLevel: opts.memLevel,
},
(err, compressed) => {
if (err) {
console.error('Compression failed:', err);
return originalJson(data);
}
const compressionRatio = ((1 - (compressed.length / sizeBytes)) * 100).toFixed(1);
res.setHeader('Content-Encoding', 'gzip');
res.setHeader('Content-Type', 'application/json');
res.setHeader('X-Original-Size', sizeBytes.toString());
res.setHeader('X-Compressed-Size', compressed.length.toString());
res.setHeader('X-Compression-Ratio', `${compressionRatio}%`);
console.log(
`[Compression] ${sizeBytes}B → ${compressed.length}B (${compressionRatio}% reduction)`
);
res.send(compressed);
}
);
return res;
};
next();
};
}
}
// Aplicar en server
// src/server/index.ts
import { CompressionMiddleware } from './middleware/compression.js';
app.use(CompressionMiddleware.compress({
threshold: 1024, // Comprimir > 1 KB
level: 6, // Nivel medio
memLevel: 8,
}));
```
**Resultados de Compresión:**
| Tipo de Datos | Tamaño Original | Comprimido (gzip) | Reducción |
|---------------|----------------|-------------------|-----------|
| Jobs (100 registros) | 600 KB | 85 KB | 86% |
| Estimates con items[] | 800 KB | 120 KB | 85% |
| Invoices con payments[] | 1,000 KB | 150 KB | 85% |
| JSON con text fields | 500 KB | 60 KB | 88% |
**Nota:** Gzip es extremadamente efectivo en JSON porque comprime:
- Nombres de campos repetidos (todos los registros tienen los mismos campos)
- Valores repetidos (status, tipos, etc.)
- Whitespace y estructura JSON
---
## ESTRATEGIA 5: Handle Storage System (Ya Implementado)
### Análisis de Implementación Actual
**Archivo: `src/utils/responseBuilder.ts`**
```typescript
// SISTEMA YA FUNCIONAL
public static async build<T>(
data: T,
options: ResponseBuilderOptions
): Promise<ResponseEnvelope<T>> {
// 1. Field selection
// 2. Verbosity compaction
// 3. Text truncation
// 4. Summary creation
// 5. Size calculation
// 6. Handle storage si > 25 KB
const needsHandle = exceedsThreshold(fullDataSize, 'hard');
if (needsHandle) {
resultHandle = await handleStorage.store(
options.entity,
processedData,
options.toolName,
verbosity,
options.context.instance
);
}
return {
status: needsHandle ? 'partial' : 'ok',
summary: summary,
result_handle: resultHandle,
metadata: { ... }
};
}
```
### Mejoras Propuestas
**5.1 Handle Metadata Enhancement**
```typescript
// src/services/handleStorage.ts
export interface HandleMetadata {
handle: string;
entity: string;
created_at: string;
expires_at: string;
size_bytes: number;
record_count: number;
verbosity: VerbosityLevel;
compression: {
enabled: boolean;
original_size: number;
compressed_size: number;
ratio: number;
};
fields_available: string[]; // Campos disponibles en handle
preview_available: boolean; // Si hay preview en response
}
// Retornar metadata completa con handle
{
"status": "partial",
"summary": [...5 primeros registros...],
"result_handle": "jn:jobs:list:1736780000:abc123",
"handle_metadata": {
"handle": "jn:jobs:list:1736780000:abc123",
"entity": "jobs",
"created_at": "2025-01-13T10:00:00Z",
"expires_at": "2025-01-13T10:15:00Z",
"size_bytes": 600000,
"record_count": 100,
"verbosity": "compact",
"compression": {
"enabled": true,
"original_size": 600000,
"compressed_size": 85000,
"ratio": 0.86
},
"fields_available": ["jnid", "number", "name", "status_name", ...],
"preview_available": true
},
"metadata": { ... }
}
```
**5.2 Handle Fetching con Field Selection**
```typescript
// fetch_by_handle con selección de campos
GET /api/fetch_by_handle?handle=jn:jobs:list:1736780000:abc123&fields=jnid,number,status_name
// Aplicar field selection al recuperar del handle
export class HandleStorage {
async fetch(
handle: string,
options?: {
fields?: string;
verbosity?: VerbosityLevel;
}
): Promise<any> {
// 1. Obtener datos del handle
const data = await this.redis.get(this.buildKey(handle));
if (!data) throw new Error('Handle expired or not found');
const parsed = JSON.parse(data);
// 2. Aplicar field selection si se especifica
if (options?.fields) {
return ResponseBuilder.selectFields(
parsed,
options.fields.split(',')
);
}
// 3. Aplicar verbosity diferente si se especifica
if (options?.verbosity && options.verbosity !== parsed.verbosity) {
return ResponseBuilder.applyVerbosity(
parsed,
options.verbosity,
getMaxFields(options.verbosity)
);
}
return parsed;
}
}
```
---
## ESTRATEGIA 6: Caching Inteligente (Ya Implementado)
### Análisis de Implementación Actual
**Archivo: `src/services/cacheService.ts`**
```typescript
// Redis cache ya implementado
export async function withCache<T>(
key: CacheKey,
ttl: number,
fetchFn: () => Promise<T>
): Promise<T> {
const cacheKey = buildKey(key);
// 1. Intentar obtener de cache
const cached = await redisClient.get(cacheKey);
if (cached) {
return JSON.parse(cached);
}
// 2. Ejecutar función y cachear
const result = await fetchFn();
await redisClient.setex(cacheKey, ttl, JSON.stringify(result));
return result;
}
```
### Mejoras Propuestas
**6.1 Invalidación Selectiva por Entidad**
```typescript
// src/services/cacheInvalidation.ts
export class CacheInvalidation {
/**
* Invalidar cache por patrón de entidad
*/
static async invalidateEntity(
instance: string,
entity: string,
operation?: string
): Promise<number> {
const pattern = operation
? `${instance}:${entity}:${operation}:*`
: `${instance}:${entity}:*`;
return await this.invalidateByPattern(pattern);
}
/**
* Invalidar por JNID específico
*/
static async invalidateById(
instance: string,
entity: string,
jnid: string
): Promise<number> {
const patterns = [
`${instance}:${entity}:get:${jnid}:*`,
`${instance}:${entity}:list:*`, // Invalidar listas también
];
let count = 0;
for (const pattern of patterns) {
count += await this.invalidateByPattern(pattern);
}
return count;
}
/**
* Invalidación en cascada (job → estimates, invoices, etc.)
*/
static async invalidateCascade(
instance: string,
entity: string,
jnid: string
): Promise<void> {
const cascadeMap: Record<string, string[]> = {
jobs: ['estimates', 'invoices', 'activities', 'tasks'],
contacts: ['jobs', 'estimates', 'invoices'],
estimates: ['jobs'],
};
const relatedEntities = cascadeMap[entity] || [];
for (const related of relatedEntities) {
await this.invalidateEntity(instance, related, 'list');
}
}
private static async invalidateByPattern(pattern: string): Promise<number> {
const keys = await redisClient.keys(pattern);
if (keys.length === 0) return 0;
await redisClient.del(...keys);
console.log(`[Cache] Invalidated ${keys.length} keys: ${pattern}`);
return keys.length;
}
}
// Uso en mutations (create, update, delete)
// src/tools/jobs/updateJob.ts
async execute(input: UpdateJobInput, context: ToolContext) {
const result = await this.client.put(context.apiKey, `jobs/${input.jnid}`, input);
// Invalidar cache del job y cascada
await CacheInvalidation.invalidateById(context.instance, 'jobs', input.jnid);
await CacheInvalidation.invalidateCascade(context.instance, 'jobs', input.jnid);
return result;
}
```
**6.2 Cache Warmup Strategy**
```typescript
// src/services/cacheWarmup.ts
export class CacheWarmup {
/**
* Pre-calentar cache con queries comunes
*/
static async warmupCommonQueries(
instance: string,
apiKey: string
): Promise<void> {
const queries = [
// Jobs del mes actual
{ endpoint: 'jobs', params: { ...getCurrentMonth() } },
// Estimates recientes
{ endpoint: 'estimates', params: { size: 50 } },
// Invoices recientes
{ endpoint: 'invoices', params: { size: 50 } },
];
for (const query of queries) {
try {
await withCache(
{
entity: query.endpoint,
operation: 'list',
identifier: JSON.stringify(query.params),
instance,
},
getTTL(`${query.endpoint.toUpperCase()}_LIST`),
async () => {
return await client.get(apiKey, query.endpoint, query.params);
}
);
console.log(`[CacheWarmup] Warmed: ${query.endpoint}`);
} catch (error) {
console.error(`[CacheWarmup] Failed: ${query.endpoint}`, error);
}
}
}
}
```
---
## RESUMEN DE REDUCCIÓN DE DATOS
### Tabla Comparativa (100 Jobs)
| Estrategia | Tamaño Original | Tamaño Optimizado | Reducción |
|------------|----------------|-------------------|-----------|
| **Sin optimización** | 600 KB | 600 KB | 0% |
| **Field selection (básico)** | 600 KB | 120 KB | 80% |
| **Field selection (nested)** | 600 KB | 30 KB | 95% |
| **Verbosity: summary** | 600 KB | 20 KB | 97% |
| **Verbosity: compact** | 600 KB | 80 KB | 87% |
| **Lazy loading (items[])** | 800 KB | 60 KB | 92% |
| **Gzip compression** | 600 KB | 85 KB | 86% |
| **Handle storage** | 600 KB | 15 KB (summary) | 97% |
| **COMBINADO (optimal)** | 600 KB | **12 KB** | **98%** |
### Estrategia Óptima Combinada
```typescript
// Configuración recomendada para máxima reducción
GET /jobs?verbosity=compact&fields=jnid,number,status_name,primary.name,approved_estimate_total&page_size=20
// Con compresión gzip habilitada:
// Original: 600 KB
// Field selection: 30 KB (95% reducción)
// Gzip compression: 4 KB (98.6% reducción)
// TOTAL: 98.6% reducción
```
---
## PLAN DE IMPLEMENTACIÓN
### Fase 1: Field Selection & Nested Selection (1-2 días)
- [ ] Implementar `NestedFieldSelector` class
- [ ] Agregar field presets (`FIELD_PRESETS`)
- [ ] Actualizar todos los endpoints principales (jobs, estimates, invoices)
- [ ] Testing y validación
### Fase 2: Smart Verbosity (1-2 días)
- [ ] Implementar `SMART_FIELD_SELECTION` config
- [ ] Aplicar selección inteligente en `ResponseBuilder`
- [ ] Actualizar documentación de API
### Fase 3: Lazy Loading (2-3 días)
- [ ] Implementar `LazyArrayLoader` class
- [ ] Crear endpoint `GET /api/{entity}_items`
- [ ] Integrar con sistema de handles
- [ ] Testing con arrays grandes
### Fase 4: Compression (1 día)
- [ ] Implementar `CompressionMiddleware`
- [ ] Configurar niveles de compresión
- [ ] Monitorear ratios de compresión
- [ ] Testing de performance
### Fase 5: Cache Optimization (1-2 días)
- [ ] Implementar `CacheInvalidation` class
- [ ] Implementar `CacheWarmup` strategy
- [ ] Configurar invalidación en cascada
- [ ] Monitoreo de hit rates
### Fase 6: Cursor Pagination (1-2 días)
- [ ] Implementar `CursorPaginator` class
- [ ] Migrar endpoints a cursor-based pagination
- [ ] Testing de navegación bidireccional
### Fase 7: Monitoring & Metrics (1 día)
- [ ] Dashboard de métricas de reducción de datos
- [ ] Alertas de responses grandes
- [ ] Analytics de uso de handles
- [ ] Reportes de compresión
---
## MÉTRICAS DE ÉXITO
### KPIs Objetivo
| Métrica | Actual | Objetivo | Crítico |
|---------|--------|----------|---------|
| Response size promedio | 50 KB | 5 KB | 90% reducción |
| Handle usage rate | 10% | 60% | Responses > 25 KB |
| Cache hit rate | 40% | 75% | Queries repetidas |
| Gzip compression ratio | N/A | 85% | Responses > 1 KB |
| API response time | 800 ms | 200 ms | 75% mejora |
### Monitoreo Continuo
```typescript
// src/middleware/metricsCollector.ts
export class MetricsCollector {
static logResponse(req: Request, res: Response, data: any) {
const metrics = {
endpoint: req.path,
method: req.method,
verbosity: req.query.verbosity || 'none',
fields_requested: req.query.fields ? req.query.fields.split(',').length : 0,
original_size: calculateSize(data),
compressed: res.getHeader('Content-Encoding') === 'gzip',
compressed_size: parseInt(res.getHeader('X-Compressed-Size') as string) || 0,
cache_hit: res.getHeader('X-Cache-Hit') === 'true',
handle_used: !!data.result_handle,
response_time_ms: Date.now() - (req as any).startTime,
};
// Enviar a sistema de métricas (Prometheus, DataDog, etc.)
console.log('[Metrics]', metrics);
}
}
```
---
## ANEXO: Ejemplos de API
### Ejemplo 1: Job Básico con Field Selection
**Request:**
```http
GET /api/jobs/123?fields=jnid,number,status_name,primary.name,approved_estimate_total
Accept-Encoding: gzip
```
**Response (sin compresión):**
```json
{
"jnid": "abc123",
"number": "1820",
"status_name": "Lead",
"primary": {
"name": "John Doe"
},
"approved_estimate_total": 15000
}
```
**Tamaño:** 150 bytes (vs 6 KB original = 97.5% reducción)
---
### Ejemplo 2: Lista de Jobs con Verbosity
**Request:**
```http
GET /api/jobs?verbosity=compact&page_size=20
Accept-Encoding: gzip
```
**Response:**
```json
{
"status": "ok",
"summary": [
{
"jnid": "abc123",
"number": "1820",
"name": "Roof Repair",
"status_name": "Lead",
"address_line1": "123 Main St",
"city": "Stamford",
"state_text": "CT",
"sales_rep_name": "John Smith",
"date_created": "2025-01-15",
"approved_estimate_total": 15000,
"owners_count": 2,
"tags_count": 3
}
// ... x20 jobs
],
"page_info": {
"cursor": "eyJpbmRleCI6MjAsInRpbWVzdGFtcCI6MTczNjc4MDAwMDAwMH0=",
"has_more": true,
"current_page_size": 20
},
"metadata": {
"verbosity": "compact",
"size_bytes": 8500,
"field_count": 12,
"row_count": 20,
"cache_hit": true
}
}
```
**Tamaño:** 8.5 KB (sin gzip), 1.2 KB (con gzip)
**Reducción:** 98% con gzip (vs 600 KB original)
---
### Ejemplo 3: Estimate con Lazy Loading
**Request:**
```http
GET /api/estimates/est123?verbosity=compact
Accept-Encoding: gzip
```
**Response:**
```json
{
"status": "ok",
"summary": {
"jnid": "est123",
"number": "EST-2025-001",
"total": 24850.50,
"status_name": "Approved",
"date_created": "2025-01-10",
"items": {
"_type": "lazy_array",
"entity": "estimate_items",
"parent_id": "est123",
"count": 50,
"summary": [
{
"jnid": "item1",
"name": "Architectural Shingles",
"quantity": 25,
"price": 95.50,
"total": 2387.50
},
{
"jnid": "item2",
"name": "Underlayment",
"quantity": 30,
"price": 15.00,
"total": 450.00
},
{
"jnid": "item3",
"name": "Ridge Cap",
"quantity": 50,
"price": 8.50,
"total": 425.00
}
],
"load_url": "/api/estimate_items?parent_id=est123",
"handle": "jn:estimate_items:est123:1736780000:abc123"
}
},
"metadata": {
"verbosity": "compact",
"size_bytes": 2100
}
}
```
**Tamaño:** 2.1 KB (vs 25 KB original = 92% reducción)
---
### Ejemplo 4: Handle Fetching con Field Selection
**Request 1 (obtener summary + handle):**
```http
GET /api/jobs?page_size=100&verbosity=compact
Accept-Encoding: gzip
```
**Response 1:**
```json
{
"status": "partial",
"summary": [ ...5 primeros jobs... ],
"result_handle": "jn:jobs:list:1736780000:abc123",
"handle_metadata": {
"handle": "jn:jobs:list:1736780000:abc123",
"expires_at": "2025-01-13T10:15:00Z",
"size_bytes": 600000,
"record_count": 100,
"fields_available": ["jnid", "number", "name", "status_name", ...]
}
}
```
**Request 2 (fetch del handle con fields específicos):**
```http
GET /api/fetch_by_handle?handle=jn:jobs:list:1736780000:abc123&fields=jnid,number,status_name
```
**Response 2:**
```json
{
"data": [
{
"jnid": "abc123",
"number": "1820",
"status_name": "Lead"
}
// ... x100 jobs con solo 3 campos
]
}
```
**Tamaño:** 15 KB (vs 600 KB original = 97.5% reducción)
---
## CONCLUSIÓN
Esta estrategia de optimización combina 6 técnicas complementarias para lograr una reducción de datos del **90-98%**:
1. **Field Selection (Nested)**: 80-95% reducción
2. **Smart Verbosity**: 50-97% reducción
3. **Lazy Loading**: 85-92% reducción
4. **Gzip Compression**: 85-88% reducción adicional
5. **Handle Storage**: Responses > 25 KB → 15 KB summaries
6. **Cache Inteligente**: Eliminación de queries redundantes
**Impacto Estimado:**
- Reducción promedio de datos transmitidos: **94%**
- Mejora en tiempos de respuesta: **75%**
- Reducción en uso de tokens de Claude: **90%**
- Mejora en experiencia de usuario: **Significativa**
**Próximos Pasos:**
1. Implementar Fase 1 (Field Selection)
2. Testing con datos reales
3. Monitoreo de métricas
4. Iteración basada en resultados
