Termination Sequence

The termination sequence provides a controlled and graceful process for removing workload replicas from the load balancer pool and handling container termination. This process uses preStop hooks (either default or custom) to manage termination timing and ensure proper connection handling based on your workload’s specific requirements.

​

When Termination Occurs

Workload termination typically happens in these scenarios:

• Scaling down: When reducing the number of workload replicas.
• Version updates: When replacing an old version with a new version after a successful deployment.
• Capacity AI: Regular rollouts similar to version updates when Capacity AI is enabled.
• Maintenance: Rare cases where maintenance activities require workload replicas to be rescheduled.

​

Termination Process Overview

The following steps are performed for each workload replica. The shutdown process occurs simultaneously across all containers in a workload, with default configurations giving in-flight requests time to complete and load balancers time to update before containers receive shutdown signals. Termination Grace Period By default, spec.rolloutOptions.terminationGracePeriodSeconds is set to 90 seconds. This controls the total time available for the workload replica to shut down gracefully before all containers receive a SIGKILL signal.

​

1. Load Balancer Update

At the start of workload termination, load balancers receive a command to remove the workload replica from the pool. This update process typically takes a few seconds but can take up to 10 seconds. Once updated, new incoming requests are routed to the remaining healthy replicas.

​

2. Workload Sidecar and Container Termination

The workload sidecar (managed by Control Plane) and all other workload containers receive commands to begin their termination process. This occurs nearly simultaneously with the load balancer update.

​

Sidecar Termination Process

The Control Plane-managed sidecar shutdown process consists of three sequential phases: Hold → Monitoring → Drain.

​

Phase 1: Sidecar Hold

The sidecar continues running normally for 80 seconds by default. This is calculated as 10 seconds less than the termination grace period, which defaults to 90 seconds. The termination grace period can be adjusted in the workload rollout options. As an example, if the termination grace period is reduced to 10 seconds or less, the sidecar hold period would be 0 seconds.

​

Phase 2: Sidecar Monitoring

The Control Plane-managed sidecar continues monitoring inbound and outbound network activity from workload containers. The sidecar remains running and waits to drain until no active requests exist. This phase can continue until the termination grace period expires or until no more connections are found.

​

Phase 3: Sidecar Drain

In the drain phase, the sidecar stops accepting new connections and once again verifies that all existing connections are complete. Once all connections are completed, the sidecar shuts down.

​

Workload Container Termination Process

​

Default PreStop Hook

If no custom preStop hook is defined for workload containers, Control Plane applies a default preStop hook that pauses the shutdown process by running sh -c "sleep <duration>". The sleep duration is half of terminationGracePeriodSeconds; with the default 90-second grace period, the hook runs sh -c "sleep 45". After the sleep completes, the container receives a SIGTERM signal and has the remaining half of the termination grace period to shut down gracefully before receiving a SIGKILL signal.

​

Custom PreStop Hook (Optional)

Consider implementing a custom preStop hook only if your workload requires specific termination logic, such as:

• Ensuring connections are gracefully terminated.
• Managing shutdown delays when sh and sleep binaries are unavailable in the container.
• Implementing custom request handling during shutdown.

Important: If you implement a custom preStop hook, ensure it includes a delay or checks for ongoing requests before exiting. This allows external load balancers sufficient time to update. After the preStop hook completes, the container receives a SIGTERM signal to terminate gracefully, with the full terminationGracePeriodSeconds (default: 90 seconds) allocated from the start of the shutdown process.

​

Summary

Implementing a custom preStop hook is only recommended when additional logic is necessary for your specific workload termination requirements. The default preStop hook provides adequate termination handling for most use cases.