TinySystems Documentation

Appearance

Data Transformation ​

This guide covers common data transformation patterns using TinySystems expressions. Learn how to reshape, filter, and manipulate data as it flows through your pipelines.

Basic Transformations ​

Field Selection ​

Extract specific fields from a message:

yaml
# Input: { "id": 1, "name": "John", "email": "john@example.com", "age": 30 }
data:
  name: "{{$.name}}"
  email: "{{$.email}}"
# Output: { "name": "John", "email": "john@example.com" }

Field Renaming ​

Rename fields for downstream compatibility:

yaml
# Input: { "user_id": 123, "user_name": "alice" }
data:
  userId: "{{$.user_id}}"
  userName: "{{$.user_name}}"
# Output: { "userId": 123, "userName": "alice" }

Field Restructuring ​

Reorganize nested structure:

yaml
# Input: { "name": "John", "street": "123 Main", "city": "NYC", "zip": "10001" }
data:
  name: "{{$.name}}"
  address:
    street: "{{$.street}}"
    city: "{{$.city}}"
    zip: "{{$.zip}}"
# Output: { "name": "John", "address": { "street": "123 Main", "city": "NYC", "zip": "10001" } }

Flattening ​

Flatten nested objects:

yaml
# Input: { "user": { "name": "John", "address": { "city": "NYC" } } }
data:
  userName: "{{$.user.name}}"
  userCity: "{{$.user.address.city}}"
# Output: { "userName": "John", "userCity": "NYC" }

String Transformations ​

Concatenation ​

yaml
data:
  fullName: "{{$.firstName + ' ' + $.lastName}}"
  greeting: "{{'Hello, ' + $.name + '!'}}"

Template Strings ​

yaml
data:
  message: "{{`User ${$.name} (${$.email}) logged in at ${$.timestamp}`}}"

Case Conversion ​

yaml
data:
  upper: "{{$.text.toUpperCase()}}"
  lower: "{{$.text.toLowerCase()}}"

Trimming ​

yaml
data:
  cleaned: "{{$.input.trim()}}"

Substring ​

yaml
data:
  first10: "{{$.text.substring(0, 10)}}"
  afterFirst: "{{$.text.substring(1)}}"

Replace ​

yaml
data:
  replaced: "{{$.text.replace('old', 'new')}}"
  sanitized: "{{$.html.replace(/<[^>]*>/g, '')}}"  # Strip HTML

Split ​

yaml
data:
  parts: "{{$.csv.split(',')}}"
  firstPart: "{{$.path.split('/')[0]}}"

Number Transformations ​

Arithmetic ​

yaml
data:
  sum: "{{$.a + $.b}}"
  product: "{{$.price * $.quantity}}"
  percentage: "{{$.value / $.total * 100}}"
  rounded: "{{Math.round($.value)}}"

Formatting ​

yaml
data:
  fixed: "{{$.price.toFixed(2)}}"
  integer: "{{parseInt($.stringNum)}}"
  float: "{{parseFloat($.stringFloat)}}"

Bounds ​

yaml
data:
  clamped: "{{Math.min(Math.max($.value, 0), 100)}}"
  absolute: "{{Math.abs($.diff)}}"

Array Transformations ​

Map ​

Transform each element:

yaml
# Input: { "users": [{"id": 1, "name": "A"}, {"id": 2, "name": "B"}] }
data:
  names: "{{$.users.map(u => u.name)}}"
  ids: "{{$.users.map(u => u.id)}}"
# Output: { "names": ["A", "B"], "ids": [1, 2] }

Filter ​

Select matching elements:

yaml
# Input: { "items": [{"status": "active"}, {"status": "inactive"}] }
data:
  active: "{{$.items.filter(i => i.status === 'active')}}"

Find ​

Get first matching element:

yaml
data:
  admin: "{{$.users.find(u => u.role === 'admin')}}"

Reduce ​

Aggregate values:

yaml
data:
  total: "{{$.items.reduce((sum, item) => sum + item.price, 0)}}"
  concatenated: "{{$.strings.reduce((acc, s) => acc + s, '')}}"

Sort ​

Order elements:

yaml
data:
  sorted: "{{$.items.sort((a, b) => a.name.localeCompare(b.name))}}"
  byAge: "{{$.users.sort((a, b) => a.age - b.age)}}"

Slice ​

Get subset:

yaml
data:
  first5: "{{$.items.slice(0, 5)}}"
  last3: "{{$.items.slice(-3)}}"

Join ​

Combine to string:

yaml
data:
  csv: "{{$.names.join(',')}}"
  sentence: "{{$.words.join(' ')}}"

Unique ​

Remove duplicates:

yaml
data:
  unique: "{{[...new Set($.tags)]}}"

Flatten ​

Flatten nested arrays:

yaml
data:
  flattened: "{{$.nestedArrays.flat()}}"
  deepFlat: "{{$.deepNested.flat(Infinity)}}"

Object Transformations ​

Merge ​

Combine objects:

yaml
data: "{{...$.defaults, ...$.overrides}}"

Pick Properties ​

Select specific properties:

yaml
data:
  picked: "{{(({id, name}) => ({id, name}))($.user)}}"

Omit Properties ​

Exclude specific properties:

yaml
data:
  safe: "{{(({password, ...rest}) => rest)($.user)}}"

Keys and Values ​

yaml
data:
  keys: "{{Object.keys($.config)}}"
  values: "{{Object.values($.config)}}"
  entries: "{{Object.entries($.config)}}"

From Entries ​

Build object from array:

yaml
data:
  rebuilt: "{{Object.fromEntries($.pairs)}}"

Conditional Transformations ​

Ternary ​

yaml
data:
  status: "{{$.count > 0 ? 'has items' : 'empty'}}"
  displayName: "{{$.nickname || $.name || 'Anonymous'}}"

Null Handling ​

yaml
data:
  safe: "{{$.value ?? 'default'}}"
  optional: "{{$.nested?.deeply?.value ?? 'missing'}}"

Type-based ​

yaml
data:
  isArray: "{{Array.isArray($.value) ? $.value : [$.value]}}"
  stringified: "{{typeof $.value === 'object' ? JSON.stringify($.value) : $.value}}"

Date Transformations ​

Current Time ​

yaml
data:
  now: "{{Date.now()}}"
  isoNow: "{{new Date().toISOString()}}"

Parsing ​

yaml
data:
  timestamp: "{{Date.parse($.dateString)}}"
  date: "{{new Date($.timestamp)}}"

Formatting ​

yaml
data:
  iso: "{{new Date($.timestamp).toISOString()}}"
  locale: "{{new Date($.timestamp).toLocaleDateString()}}"

JSON Transformations ​

Parse ​

yaml
data:
  parsed: "{{JSON.parse($.jsonString)}}"

Stringify ​

yaml
data:
  json: "{{JSON.stringify($.object)}}"
  pretty: "{{JSON.stringify($.object, null, 2)}}"

Complex Examples ​

API Response Normalization ​

yaml
# Input: API response with nested data
# { "data": { "users": [{"id": 1, "attributes": {"name": "John"}}] } }
data:
  users: "{{$.data.users.map(u => ({id: u.id, name: u.attributes.name}))}}"
# Output: { "users": [{"id": 1, "name": "John"}] }

Error Handling ​

yaml
data:
  result:
    success: "{{$.status === 'ok'}}"
    data: "{{$.status === 'ok' ? $.data : null}}"
    error: "{{$.status !== 'ok' ? $.error : null}}"

Pagination Metadata ​

yaml
data:
  items: "{{$.results}}"
  pagination:
    page: "{{$.page}}"
    perPage: "{{$.per_page}}"
    total: "{{$.total}}"
    totalPages: "{{Math.ceil($.total / $.per_page)}}"
    hasNext: "{{$.page < Math.ceil($.total / $.per_page)}}"

Statistics Calculation ​

yaml
data:
  count: "{{$.items.length}}"
  sum: "{{$.items.reduce((a, b) => a + b.value, 0)}}"
  average: "{{$.items.reduce((a, b) => a + b.value, 0) / $.items.length}}"
  min: "{{Math.min(...$.items.map(i => i.value))}}"
  max: "{{Math.max(...$.items.map(i => i.value))}}"

Grouping ​

yaml
data:
  grouped: "{{$.items.reduce((acc, item) => ({...acc, [item.category]: [...(acc[item.category] || []), item]}), {})}}"

Best Practices ​

Keep It Simple ​

yaml
# Good: Simple, readable
data:
  name: "{{$.firstName + ' ' + $.lastName}}"

# Avoid: Complex inline logic
# data:
#   result: "{{$.items.filter(i => i.active).map(i => ({...i, processed: true})).sort((a, b) => a.order - b.order)}}"

Use Intermediate Nodes ​

For complex transformations, use multiple transform nodes:

Input → Filter Node → Map Node → Sort Node → Output

Handle Missing Data ​

yaml
data:
  name: "{{$.name ?? 'Unknown'}}"
  items: "{{$.items ?? []}}"
  count: "{{$.items?.length ?? 0}}"

Validate Types ​

yaml
data:
  ensuredArray: "{{Array.isArray($.items) ? $.items : []}}"
  ensuredString: "{{typeof $.value === 'string' ? $.value : String($.value)}}"

Next Steps ​

Build flow-based applications on Kubernetes

Copyright © TinySystems