Kubernetes Architecture
TinySystems modules are Kubernetes operators built with Kubebuilder. Understanding the Kubernetes architecture is essential for advanced module development.
Operator Pattern
Each module runs as a Kubernetes operator:
+-----------------------------------------------------------------------------+
| MODULE OPERATOR |
| |
| +------------------------------------------------------------------------+ |
| | CONTROLLER MANAGER | |
| | | |
| | +-------------+ +-------------+ +-------------+ +-------------+ | |
| | | TinyNode | | TinyModule | | TinySignal | | TinyFlow | | |
| | | Controller | | Controller | | Controller | | Controller | | |
| | +------+------+ +------+------+ +------+------+ +------+------+ | |
| | | | | | | |
| | +----------------+----------------+----------------+ | |
| | | | |
| +------------------------------------+-----------------------------------+ |
| | |
| +------------------------------------+-----------------------------------+ |
| | SCHEDULER | |
| | | | |
| | +-------------------------+-------------------------+ | |
| | | | | |
| | +-----+-----+ +-----------+ +-----------+ | | |
| | | Runner | | Runner | | Runner | ... | | |
| | |(node-abc) | |(node-def) | |(node-ghi) | | | |
| | +-----------+ +-----------+ +-----------+ | | |
| | | | |
| +--------------------------------------------------------------+ | |
| |
| +------------------------------------------------------------------------+ |
| | gRPC SERVER | |
| | (Cross-module communication) | |
| +------------------------------------------------------------------------+ |
+-----------------------------------------------------------------------------+
|
| Watch/Update
v
+-----------------------------------------------------------------------------+
| KUBERNETES API SERVER |
| |
| TinyNode CRDs TinyModule CRDs TinySignal CRDs TinyFlow CRDs |
+-----------------------------------------------------------------------------+Custom Resource Definitions (CRDs)
TinySystems defines several CRDs:
| CRD | Purpose | Controller |
|---|---|---|
| TinyNode | Component instance configuration | TinyNodeReconciler |
| TinyModule | Module service discovery | TinyModuleReconciler |
| TinySignal | Trigger node execution | TinySignalReconciler |
| TinyFlow | Group nodes in a flow | TinyFlowReconciler |
| TinyProject | Top-level project container | TinyProjectReconciler |
| TinyTracker | Execution tracing | TinyTrackerReconciler |
Controller-Runtime
Modules use controller-runtime:
go
import (
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/manager"
)
func main() {
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
MetricsBindAddress: ":8080",
HealthProbeBindAddress: ":8081",
LeaderElection: true,
LeaderElectionID: "my-module-lock",
})
// Register controllers
(&TinyNodeReconciler{}).SetupWithManager(mgr)
(&TinyModuleReconciler{}).SetupWithManager(mgr)
(&TinySignalReconciler{}).SetupWithManager(mgr)
// Start manager
mgr.Start(ctrl.SetupSignalHandler())
}Reconciliation Loop
Controllers implement the reconciliation pattern:
+-----------------------------------------------------------------+
| RECONCILIATION LOOP |
+-----------------------------------------------------------------+
+---------------+
| Watch Events |
| (Create/Update|
| /Delete) |
+-------+-------+
|
v
+---------------+
| Work Queue |
| (Deduplication|
| Rate limiting)|
+-------+-------+
|
v
+---------------+
| Reconcile() |
| |
| 1. Get CR |
| 2. Check state|
| 3. Take action|
| 4. Update |
| status |
+-------+-------+
|
+-------------+-------------+
| |
v v
+-------------+ +-------------+
| Success | | Requeue |
| (No requeue)| | (After delay)|
+-------------+ +-------------+Module Startup Flow
1. INITIALIZATION
+----------------------------------------------------------------+
| - Load kubeconfig |
| - Create controller manager |
| - Register component handlers |
| - Setup leader election |
+----------------------------------------------------------------+
|
v
2. LEADER ELECTION
+----------------------------------------------------------------+
| - Acquire Lease lock |
| - If leader: Start reconcilers, update CRs |
| - If follower: Watch only, no status updates |
+----------------------------------------------------------------+
|
v
3. MODULE REGISTRATION
+----------------------------------------------------------------+
| - Create/update TinyModule CR |
| - Leader updates status with: |
| - gRPC address |
| - Version |
| - Available components |
+----------------------------------------------------------------+
|
v
4. CONTROLLER STARTUP
+----------------------------------------------------------------+
| - Start watching TinyNode CRs |
| - Start watching TinySignal CRs |
| - Start watching TinyModule CRs (for discovery) |
| - Start gRPC server for cross-module communication |
+----------------------------------------------------------------+
|
v
5. RECONCILIATION
+----------------------------------------------------------------+
| - Process TinyNode CRs matching this module |
| - Create Runner instances for each node |
| - Handle incoming signals |
| - Periodic reconciliation (every 5 minutes) |
+----------------------------------------------------------------+Key Components
Controller Manager
Manages all controllers and shared resources:
go
mgr, _ := ctrl.NewManager(config, ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
HealthProbeBindAddress: healthAddr,
// Leader election
LeaderElection: true,
LeaderElectionID: fmt.Sprintf("%s-lock", moduleName),
// Client configuration
NewClient: func(config *rest.Config, options client.Options) (client.Client, error) {
return client.New(config, options)
},
})Scheduler
Routes messages to component instances:
go
type Scheduler struct {
instancesMap *cmap.ConcurrentMap[string, *runner.Runner]
components *cmap.ConcurrentMap[string, module.Component]
clientPool *client.Pool
}
func (s *Scheduler) Handle(ctx context.Context, msg runner.Msg) error {
// Find runner for target node
r, exists := s.instancesMap.Get(nodeName)
if !exists {
// Route to remote module
return s.routeToRemoteModule(ctx, msg)
}
return r.MsgHandler(ctx, msg)
}Runner
Wraps a component instance:
go
type Runner struct {
node *v1alpha1.TinyNode
component module.Component
scheduler *Scheduler
tracer trace.Tracer
}
func (r *Runner) MsgHandler(ctx context.Context, msg Msg) error {
// Create trace span
ctx, span := r.tracer.Start(ctx, "handle-message")
defer span.End()
// Deserialize and evaluate expressions
data := r.deserialize(msg.Data, msg.To)
// Call component
return r.component.Handle(ctx, r.outputHandler, msg.To, data)
}RBAC Requirements
Modules need these Kubernetes permissions:
yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tinysystems-module
rules:
# CRD access
- apiGroups: ["operator.tinysystems.io"]
resources: ["tinynodes", "tinymodules", "tinysignals", "tinyflows"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# Status updates
- apiGroups: ["operator.tinysystems.io"]
resources: ["tinynodes/status", "tinymodules/status"]
verbs: ["get", "update", "patch"]
# Leader election
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "create", "update"]
# Service/Ingress management (for http-module)
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "create", "update", "patch"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get", "list", "create", "update", "patch"]Health Probes
Modules expose health endpoints:
go
// Liveness probe
mgr.AddHealthzCheck("healthz", healthz.Ping)
// Readiness probe
mgr.AddReadyzCheck("readyz", healthz.Ping)Kubernetes probes:
yaml
livenessProbe:
httpGet:
path: /healthz
port: 8081
initialDelaySeconds: 15
periodSeconds: 20
readinessProbe:
httpGet:
path: /readyz
port: 8081
initialDelaySeconds: 5
periodSeconds: 10Next Steps
- TinyNode CRD - Node configuration details
- Controller Reconciliation - Reconciliation patterns
- Leader Election - Multi-replica coordination