Termination Sequence
The termination sequence ensures a controlled and graceful process for removing workload replicas from the load balancer pool and handling container termination. The use of a preStop hook, either default or custom, can be leveraged to manage termination and connections as per the workload’s requirements.
Termination occurs most often when a workload is being scaled down or when an old version of the workload is taken offline after a successfull update to a new version. In rare cases, maintenance activities can cause workload replicas to be rescheduled and terminated.
The steps below are performed for each replica of the workload.
-
Load Balancer Update: At the initiation of workload termination, load balancers are sent a command to remove the workload replica from the pool.
-
Workload Sidecar and Containers Termination: The workload sidecar (managed by Control Plane) and all other workload containers are sent a command to begin their termination process. This happens at nearly the same time as step 1.
-
Load Balancer Routing: Once the load balancer is updated, any new incoming requests will be routed to the remaining healthy workload replicas for the workload.
-
Sidecar Monitoring: The sidecar managed by Control Plane continues to monitor network activity. It will remain running until no more active requests exist and connections terminate, for a maximum duration of up to 90 seconds.
-
Default PreStop Hook: If no customer-defined preStop hook is provided, a default preStop hook is applied to all workload containers. This default hook consists of the “sleep” command with a duration of 45 seconds. Once the sleep command completes the container is sent a SIGTERM signal and allowed to gracefully shutdown for an additional 45 seconds before receiving a SIGKILL signal.
If the “sleep” executable is not available in the customer’s workload container, the container will be sent a graceful termination signal immediately. Requests might still reach it before the load balancer is fully updated.
- Custom PreStop Hook (Optional): It is recommended to add a custom preStop hook only if specific custom logic is required, such as gracefully terminating connections that would not otherwise terminate. If a custom preStop hook is implemented, ensure it includes a delay or a check for existing requests before exiting. Once the preStop hook completes the container will receive a signal to terminate gracefully (SIGTERM) and will have up to 90 seconds from the start from the start of the shutdown process to complete.
Please note that implementing a custom preStop hook is only advisable when additional logic is necessary for your specific workload termination needs.