Kubernetes Architecture
TinySystems modules are Kubernetes operators built with Kubebuilder. Understanding the Kubernetes architecture is essential for advanced module development.
Operator Pattern
Each module runs as a Kubernetes operator:
┌─────────────────────────────────────────────────────────────────────────────┐
│ MODULE OPERATOR │
│ │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ CONTROLLER MANAGER │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ TinyNode │ │ TinyModule │ │ TinySignal │ │ TinyFlow │ │ │
│ │ │ Controller │ │ Controller │ │ Controller │ │ Controller │ │ │
│ │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │
│ │ │ │ │ │ │ │
│ │ └────────────────┴────────────────┴────────────────┘ │ │
│ │ │ │ │
│ └────────────────────────────────────┼───────────────────────────────────┘ │
│ │ │
│ ┌────────────────────────────────────┼───────────────────────────────────┐ │
│ │ SCHEDULER │ │
│ │ │ │ │
│ │ ┌─────────────────────────┴─────────────────────────┐ │ │
│ │ │ │ │ │
│ │ ┌─────┴─────┐ ┌───────────┐ ┌───────────┐ │ │ │
│ │ │ Runner │ │ Runner │ │ Runner │ ... │ │ │
│ │ │(node-abc) │ │(node-def) │ │(node-ghi) │ │ │ │
│ │ └───────────┘ └───────────┘ └───────────┘ │ │ │
│ │ │ │ │
│ └──────────────────────────────────────────────────────────────┘ │ │
│ │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ gRPC SERVER │ │
│ │ (Cross-module communication) │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
│ Watch/Update
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ KUBERNETES API SERVER │
│ │
│ TinyNode CRDs TinyModule CRDs TinySignal CRDs TinyFlow CRDs │
└─────────────────────────────────────────────────────────────────────────────┘Custom Resource Definitions (CRDs)
TinySystems defines several CRDs:
| CRD | Purpose | Controller |
|---|---|---|
| TinyNode | Component instance configuration | TinyNodeReconciler |
| TinyModule | Module service discovery | TinyModuleReconciler |
| TinySignal | Trigger node execution | TinySignalReconciler |
| TinyFlow | Group nodes in a flow | TinyFlowReconciler |
| TinyProject | Top-level project container | TinyProjectReconciler |
| TinyTracker | Execution tracing | TinyTrackerReconciler |
Controller-Runtime
Modules use controller-runtime:
go
import (
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/manager"
)
func main() {
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
MetricsBindAddress: ":8080",
HealthProbeBindAddress: ":8081",
LeaderElection: true,
LeaderElectionID: "my-module-lock",
})
// Register controllers
(&TinyNodeReconciler{}).SetupWithManager(mgr)
(&TinyModuleReconciler{}).SetupWithManager(mgr)
(&TinySignalReconciler{}).SetupWithManager(mgr)
// Start manager
mgr.Start(ctrl.SetupSignalHandler())
}Reconciliation Loop
Controllers implement the reconciliation pattern:
┌─────────────────────────────────────────────────────────────────┐
│ RECONCILIATION LOOP │
└─────────────────────────────────────────────────────────────────┘
┌───────────────┐
│ Watch Events │
│ (Create/Update│
│ /Delete) │
└───────┬───────┘
│
▼
┌───────────────┐
│ Work Queue │
│ (Deduplication│
│ Rate limiting)│
└───────┬───────┘
│
▼
┌───────────────┐
│ Reconcile() │
│ │
│ 1. Get CR │
│ 2. Check state│
│ 3. Take action│
│ 4. Update │
│ status │
└───────┬───────┘
│
┌─────────────┴─────────────┐
│ │
▼ ▼
┌─────────────┐ ┌─────────────┐
│ Success │ │ Requeue │
│ (No requeue)│ │ (After delay)│
└─────────────┘ └─────────────┘Module Startup Flow
1. INITIALIZATION
┌────────────────────────────────────────────────────────────────┐
│ - Load kubeconfig │
│ - Create controller manager │
│ - Register component handlers │
│ - Setup leader election │
└────────────────────────────────────────────────────────────────┘
│
▼
2. LEADER ELECTION
┌────────────────────────────────────────────────────────────────┐
│ - Acquire Lease lock │
│ - If leader: Start reconcilers, update CRs │
│ - If follower: Watch only, no status updates │
└────────────────────────────────────────────────────────────────┘
│
▼
3. MODULE REGISTRATION
┌────────────────────────────────────────────────────────────────┐
│ - Create/update TinyModule CR │
│ - Leader updates status with: │
│ - gRPC address │
│ - Version │
│ - Available components │
└────────────────────────────────────────────────────────────────┘
│
▼
4. CONTROLLER STARTUP
┌────────────────────────────────────────────────────────────────┐
│ - Start watching TinyNode CRs │
│ - Start watching TinySignal CRs │
│ - Start watching TinyModule CRs (for discovery) │
│ - Start gRPC server for cross-module communication │
└────────────────────────────────────────────────────────────────┘
│
▼
5. RECONCILIATION
┌────────────────────────────────────────────────────────────────┐
│ - Process TinyNode CRs matching this module │
│ - Create Runner instances for each node │
│ - Handle incoming signals │
│ - Periodic reconciliation (every 5 minutes) │
└────────────────────────────────────────────────────────────────┘Key Components
Controller Manager
Manages all controllers and shared resources:
go
mgr, _ := ctrl.NewManager(config, ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
HealthProbeBindAddress: healthAddr,
// Leader election
LeaderElection: true,
LeaderElectionID: fmt.Sprintf("%s-lock", moduleName),
// Client configuration
NewClient: func(config *rest.Config, options client.Options) (client.Client, error) {
return client.New(config, options)
},
})Scheduler
Routes messages to component instances:
go
type Scheduler struct {
instancesMap *cmap.ConcurrentMap[string, *runner.Runner]
components *cmap.ConcurrentMap[string, module.Component]
clientPool *client.Pool
}
func (s *Scheduler) Handle(ctx context.Context, msg runner.Msg) error {
// Find runner for target node
r, exists := s.instancesMap.Get(nodeName)
if !exists {
// Route to remote module
return s.routeToRemoteModule(ctx, msg)
}
return r.MsgHandler(ctx, msg)
}Runner
Wraps a component instance:
go
type Runner struct {
node *v1alpha1.TinyNode
component module.Component
scheduler *Scheduler
tracer trace.Tracer
}
func (r *Runner) MsgHandler(ctx context.Context, msg Msg) error {
// Create trace span
ctx, span := r.tracer.Start(ctx, "handle-message")
defer span.End()
// Deserialize and evaluate expressions
data := r.deserialize(msg.Data, msg.To)
// Call component
return r.component.Handle(ctx, r.outputHandler, msg.To, data)
}RBAC Requirements
Modules need these Kubernetes permissions:
yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tinysystems-module
rules:
# CRD access
- apiGroups: ["operator.tinysystems.io"]
resources: ["tinynodes", "tinymodules", "tinysignals", "tinyflows"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# Status updates
- apiGroups: ["operator.tinysystems.io"]
resources: ["tinynodes/status", "tinymodules/status"]
verbs: ["get", "update", "patch"]
# Leader election
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "create", "update"]
# Service/Ingress management (for http-module)
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "create", "update", "patch"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get", "list", "create", "update", "patch"]Health Probes
Modules expose health endpoints:
go
// Liveness probe
mgr.AddHealthzCheck("healthz", healthz.Ping)
// Readiness probe
mgr.AddReadyzCheck("readyz", healthz.Ping)Kubernetes probes:
yaml
livenessProbe:
httpGet:
path: /healthz
port: 8081
initialDelaySeconds: 15
periodSeconds: 20
readinessProbe:
httpGet:
path: /readyz
port: 8081
initialDelaySeconds: 5
periodSeconds: 10Next Steps
- TinyNode CRD - Node configuration details
- Controller Reconciliation - Reconciliation patterns
- Leader Election - Multi-replica coordination