Troubleshooting: Failed Deployment Requests in ReleaseOpsSummary<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } A step-by-step visual walkthrough for investigating why a Deployment Request failed in ReleaseOps. This guide traces a real DR through the system, from the DR record all the way down to inter-instance operation tracking. Release<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } All Releases Instructions<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Overview When a Deployment Request fails, the troubleshooting path follows this chain: DR Record (work notes, state, tasks) | vPlaybook Execution (sys_pd_context) | vActivity Executions (sys_pd_activity_context) | vExecution Logs (sys_flow_log via sys_pd_context) | vSubflow in Workflow Studio (individual actions) | vOperation Tracker (sn_releaseopsbe_operation_tracker) | vExecution Tracker on remote instance (sys_execution_tracker) Each step drills deeper. Most failures are resolved in the first 2-3 steps. The deeper steps are for inter-instance operation failures (retrieve, preview, commit). Step 1: Open the Deployment Request Record Start by opening the failed DR. Check: State — confirms the DR is in Failed (80) stateWork Notes — failure subflows always write a work note describing what went wrong. This is the single most useful piece of information.DR Tasks related list — check for open preview_issue, test_issue, scan_issue, or runbook tasks that may be blocking the playbook A completed DR used as our walkthrough example. For a failed DR, the State field would show "Failed" and work notes would contain the failure reason. Step 2: Find the Playbook Execution Each DR has two associated playbooks: an assessment playbook that is related to the DR record itself, and a release playbook that is related to the Release record. Both are tracked in the sys_pd_context table. How to find it: Navigate to sys_pd_context.LISTFilter by Input Record = the DR's sys_id Important: Filter by input_record, NOT document_id. The input_record field stores the DR's sys_id and is the reliable way to find the playbook execution. sys_pd_context list filtered by input_record = the DR's sys_id. Shows the assessment playbook execution. The playbook execution record shows: State — Completed, In Progress, Error, or CancelledStarted / Ended — timing for the overall playbook If the playbook state is Error or Cancelled, this confirms the failure. If it's In Progress but not progressing, the DR may be stuck. Step 3: Examine Activity Executions Open the playbook execution record (sys_pd_context). The Activity Executions related list shows every step the playbook performed. Activity Executions related list on the playbook execution record. Each row is a playbook step with its state, lane, and dependencies. Each activity execution shows: Label — the step name (e.g., "Integrate Deployment Request", "Before Test Deployment", "Run ATF Tests")State — Complete, Error, Skipped, Cancelled, Pending, In ProgressLane — which phase of the playbook (e.g., "Move To Test", "Test Assessment")Dependencies — which activities must complete before this one runs What to look for: Activities in Error state — these are the failure pointsActivities in Cancelled state — downstream activities that were cancelled because an upstream activity failedThe order of activities tells you how far the playbook got before failing Tip: The default filter on Activity Executions is State != Pending, which hides activities that were never reached. This is fine for most troubleshooting. To see ALL activities (including ones that never ran), remove this filter by clicking the ">" button next to the filter breadcrumb. Bottom half of the Activity Executions list showing later stages of the playbook. Step 4: Examine Execution Logs The playbook execution record (sys_pd_context) also has an Execution Logs related list that shows messages from the playbook engine. Execution Logs tab on the playbook execution record. The log entries include: Level — Debug, Information, Warning, ErrorMessage — what the engine was doing at that pointTimestamp — when it happened What to look for: Error entries — direct failure messagesWarning entries — conditions that may have contributed to failureThe last few entries before the error — these show what the engine was attempting when it failed By default, the playbook engine only logs Warning and Error level messages. To get detailed execution logs for troubleshooting, you need to change the log level property. Property: com.glide.hub.flow_engine.log_level Setting What it logs ERROR Only errors WARN (default) Warnings and errors INFO Informational messages, warnings, and errors DEBUG All messages including detailed execution flow How to set it: Navigate to sys_properties.LISTSearch for com.glide.hub.flow_engine.log_levelChange the value to DEBUGReproduce the failure Note: Set this back to WARN after troubleshooting to avoid excessive logging. This property replaced the older boolean property com.glide.hub.flow_engine.debug (which was removed via a migration script — if the old property was true, the migration set the new property to DEBUG). Step 5: Drill into a Specific Activity Execution Click on an activity execution that is in Error state (or the last one in Complete state before the failure) to see its details. Activity execution record for "Integrate Deployment Request" -- shows the label, lane, state, activity type, and dependencies. The key fields are in the lower section: Inputs — JSON showing what was passed to this activity (e.g., destination instance, source deployment instance, deployment request reference)Locals — JSON containing execution context, including:fdScopedName — the subflow name (e.g., sn_releaseops.integrate_deployment_request)rw_id — the worker execution tracker ID (links to sn_releaseopsbe_operation_tracker.worker_execution_tracker_id)fdType — "SUBFLOW" confirms this activity runs a Flow Designer subflowwait_for_completion — whether the playbook waits for this subflow to finish Step 6: Open the Subflow in Workflow Studio From the activity execution record, click the "Open Flow in Operations View" link in the Related Links section. This opens Workflow Studio showing every action the subflow performed. Complete subflow view showing all 49 actions. Scroll down to find the action that failed or the last successful action. What to look for in Workflow Studio: Actions with a red/error indicator — these are the failure pointsConditional branches — the subflow has if/else logic; follow the branch that was actually takenTiming — how long each action took (slow actions may indicate timeout issues)Action names — descriptive names like "Pull Update Set", "Preview Update Set", "Commit Update Set" tell you exactly which operation failed Step 7: Investigate the Operation Tracker For failures involving inter-instance operations (such as retrieving, previewing, or committing update sets), the sn_releaseopsbe_operation_tracker table records every request/response interaction between instances. Finding the Operation Tracker Records Navigate to sn_releaseopsbe_operation_tracker_list.do on the controller instance (the instance that manages the pipeline). Operation tracker list on the controller instance showing the sequence of operations for a DR: Instance Scan, Pull Update Set, Preview, Commit, and Update Set Info. Matching Operation Tracker Records to a DR There is no direct foreign key from the operation tracker to the DR. Use these approaches: Approach 1 — Timestamp correlation (quickest for triage): Filter by requestor_instance = the controller instance and sys_created_on within the playbook execution's time window. Approach 2 — Via Update Set ID in Request JSON (definitive): The request field contains JSON with an updateSetId: {"@type":"PullRemoteUpdateSetRequest","updateSetId":"0ce18f1e2b4d83905721f66eec91bfe2",...} That updateSetId corresponds to records in sn_releaseops_deployment_request_update_set, which has a deployment_request reference field pointing to the DR. Approach 3 — Via worker execution tracker ID from activity locals: The activity execution's locals JSON contains an rw_id field. Filter operation trackers by worker_execution_tracker_id = that rw_id to find all operations spawned by that specific subflow execution. Understanding the Operation Tracker Record Operation tracker record on the requestor side showing a PullRemoteUpdateSetRequest. Key fields: Field Description Correlation Id Unique ID linking the requestor and replier records (same on both instances) Request JSON payload sent to the remote instance (contains the operation type and parameters) Response JSON response from the remote instance (contains results or error details) Status Current state: Requesting, Requested, Pending, Running, Successful, Failed, Cancelled Requestor Instance The instance that initiated the operation Replier Instance The instance that executed the operation Worker Execution Tracker Id Links to sys_execution_tracker on the replier instance Percent Complete Progress of the operation (0-100) For failed operations: The response field contains the error details. For example: {"@type":"OperationErrorResponse","message":"Failed to export update set","details":"..."} How Operation Tracker Records Are Updated The operation tracker follows this lifecycle: startOperation() on requestor | v[requested] -- record created on requestor, MIF message sent to replier | vhandleRemoteOperationRequest() on replier | v[requested] -- second record created on replier (same correlation_id) | vWatcherJob picks up the request (runs every 15 seconds) | +--> handler.processRequest() starts the work | v[pending] -- worker started, worker_execution_tracker_id set | vWatcherJob polls sys_execution_tracker on each cycle | v[running] -- percent_complete updated as work progresses | vWorker completes | v[successful/failed/cancelled] -- handler.processResponse() builds response JSON | vMIF update message sent back to requestor | vRequestor's record updated to match (same final status + response) Timeout: If a tracker's sys_updated_on is older than sn_releaseopsbe.operation_timeout_in_hours (default: 24 hours), the watcher job marks it as failed with the message "Operation cancelled due to timeout." Step 8: Cross-Instance Investigation via Correlation ID The correlation ID is the bridge between instances. The same correlation ID exists on both the requestor and replier instance in their respective sn_releaseopsbe_operation_tracker tables. Finding the Matching Record on the Replier Instance Copy the Correlation Id from the operation tracker record on the requestor instanceNavigate to sn_releaseopsbe_operation_tracker_list.do on the replier instanceFilter by correlation_id = the copied value Operation tracker list on the replier instance filtered by the same correlation ID -- shows the matching record. Operation tracker record on the replier instance showing the same correlation ID, request/response, and worker_execution_tracker_id. Notice the record has the same data but from the replier's perspective. The replier's record may have more detailed error information in the response field, since it's the instance that actually executed the operation. Checking the Execution Tracker on the Replier The worker_execution_tracker_id field maps to a sys_execution_tracker record on the replier instance. This tracker contains: Name — the operation type (e.g., "Instance Scan", "Update Set Commit")State — Pending, Running, Successful, Failed, CancelledMessage — often contains the most detailed error information for failuresResult — JSON with operation-specific results (e.g., scan result ID, commit details)Percent Complete — progress trackingSource — reference to the source record (e.g., scan result, update set)Start Time / Completion Time — timing information A sys_execution_tracker record showing an Instance Scan operation. The Message field shows "Scan completed with 0 warning(s), 0 error(s), and 52 finding(s)." For failed operations, this field contains the error details. Note: sys_execution_tracker records are cleaned up over time. If you're investigating an older failure, the tracker may have been deleted. Investigate promptly after a failure occurs. "Record not found" when navigating to a sys_execution_tracker record that has been cleaned up (the DR ran 3 days earlier). Troubleshooting Decision Tree Use this quick reference to decide which step to start at: Symptom Start at DR shows "Failed" with a clear work note Step 1 -- the work note may be sufficient Work note is vague or says "unknown error" Step 2 -- check the playbook execution Playbook shows "Error" but no detail Step 3 -- check activity executions for the failed step Need engine-level detail Step 4 -- check execution logs (enable DEBUG if needed) Activity failed but no clear reason Step 5 -- drill into the activity execution details Need subflow-level detail Step 6 -- open Workflow Studio for the subflow Inter-instance operation failed Step 7 -- check operation tracker Operation tracker shows failed but no response detail Step 8 -- check the replier instance's tracker and execution tracker DR stuck, not progressing Step 2 -- check if playbook is in progress; Step 7 -- check if operation tracker is in a non-final state Summary of Key Tables Table Purpose Instance sn_releaseops_deployment_request The DR record itself Controller sys_pd_context Playbook execution record Controller sys_pd_activity_context Individual playbook steps Controller sys_flow_log Execution logs (via related list on sys_pd_context) Controller sn_releaseopsbe_operation_tracker Inter-instance request/response tracking Both (requestor + replier) sys_execution_tracker Worker progress tracking Replier sn_releaseops_deployment_request_update_set Update sets linked to a DR Controller sn_releaseops_deployment_request_task DR Tasks (preview issues, test failures, etc.) Controller Key Properties Property Default Description com.glide.hub.flow_engine.log_level WARN Playbook engine log verbosity (DEBUG/INFO/WARN/ERROR) sn_releaseopsbe.operation_timeout_in_hours 24 Hours before a non-final operation tracker is marked as timed out