How MID Server File Synchronisation works - Troubleshooting guideSeveral features and applications share the same MID Server platform code for getting records and attachments in the instance synchronised to files on the MID Servers. This article aims to describe the steps and code involved to aid troubleshooting. Table of Contents The Tables involvedHow a synchronisation job is triggered AttachmentsFields Applications using this systemHow the MID Server processes the job How it realizes a file was changedScripted SOAP ServicesECC Agent Script File "Checksum" field Troubleshooting and Known issues Attachments deleted by ClonesTrouble uploading the attachment to the formCheck MID Server agent logCheck the disk to see what was synchronisedForce a re-sync of any table without making ChangesMID Server syncs the Folders, and Files from fields, but not Files from AttachmentsRecord fields or attachments changed, but sys_updated_on didn'tMultiple attachments on the same recordGeneral tipsKnown LimitationsRelated Problems The Tables involved For a table to use the system, it has to extend MID Server Synchronized File [ecc_agent_sync_file]. Below is the schema map for the extensions of the MID Server Synchronized File table in Orlando, with ITOM plugins installed. More have been added since. How a synchronisation job is triggered Attachments When an attachment is added to a record in any of these tables, that file is automatically placed on the disk within the MID Server install folder tree of all MID Servers. The file is then automatically re-synchronised with all MID Servers as the files are changed in the instance. The synchronisation is usually triggered by the system events created when records are added/changed/deleted from the Attachment [sys_attachment] table. The names of the Events [sysevent] are below, and Parm1 of the event is the table name to which the attachment belongs: attachment.uploadedattachment.renamedattachment.deleted For each event, there is a Script Action [sysevent_script_action], all named "MID Server file synchronization check", which tells the MID Servers to re-synchronise that table. The Script Actions call "MIDServerFileSync" Script Include, "notifyMIDServers" function, using the table name from the event's Parm1, to insert an ECC Queue table [ecc_queue] output record: agent = mid.server.*source = FileChangename = <table the attachment is on>topic = SystemCommandqueue = outputstate = readypriority = 0 (interractive) The "Probe - All MIDs" Business Rule on ecc_queue then copies that "mid.server.*" record, and inserts an output specific to each MID Server. All MID Servers will have the job created, regardless of whether they are Up, or whether there are any existing duplicates still in Ready state in the queue. Fields Since 2013, the contents of fields can also be synchronised to files on the MID Server. Where this is the case, Business rules on the specific table trigger the FileChange Jobs. Applications using this system The following table lists each of the extended table classes, their names, applications which use them, and anything different to the above normal event triggered synchronisation. Many other things are synchronised to memory caches in the MID Server, but don't end up as files. TableUsed byAttachment or FieldDirectoryAgent Client Collector Plugin [sn_agent_asset]ITOM Features, IntegrationHub, Vulnerability response, etc.Attachmentstatic\acc_plugin\Notify MID on Agent Asset Changes" Business Rule on sn_agent_asset calls "MIDNotificationHandler " Script Include "notifyMIDServers" function to insert a similar ECC Record as above.Agent Client Collector Configuration File [sn_agent_configuration_file]Data files for ACC pluginsAttachmentstatic\acc_configDiscovery Patterns [sa_pattern]DiscoveryService MappingField: Pattern [ndl]work\ndl "Notify MID server on NDL change" Business Rule on sa_pattern calls "PatternLibrary" Script Include "notifyMIDs" function. "Synchronize with MID Servers" UI Action on sa_pattern forces synchronisation by updating the sys_updated_on value of the pattern. This also calls "PatternLibrary" Script Include "notifyMIDs" function. This is a List action, available after ticking the checkbox by the pattern, on the /sa_pattern_list.do List. "Pattern Sync to Mid" UI Action on MID Server form uses "PatternLibrary" Script Include functions to synch patterns and other Pattern-related synchronisations. These 3 methods all use "PatternLibrary" Script Include "notifyMIDs" function, which calls "MIDNotificationHandler" Script Include "notifyMIDServers" function to insert a similar ECC Record as above. MID Server Extension JAR File [ecc_agent_ext_jar]TBCAttachmentTBCMID Server JAR File [ecc_agent_jar]Import Set JDBC DriversOrchestrationExternal Credential StoreCloud ManagementAttachmentextlibMID Server MIB File [ecc_agent_mib]SNMP DiscoveryAttachmentwork\mibsMID Server Script File [ecc_agent_script_file]DiscoveryOrchestration Attachment or Script [script] field. scripts\<parent directories> The target directory will depend on the parent(s) of the record. Records with directory=true define the folder structure, and are not actual files. A couple of other Business Rules in the attachment table update the Synchronized File record: Insert/Update: Update MID script on attachment changeDelete: Update MID script on attachment deletion These update or clear/inactivate the following fields on the Synchronized File record: script_attachment - sys_id, referencing the Attachment [sys_attachment] record. This is important if more than one file is attached.checksum - MD5 Checksum of the file in the Attachment tableuse_attachment - Tells the MID Server to use the attachment, rather than a field valueactive - set false if no file is attached Uploaded File [sa_uploaded_file]Discovery and Service Mapping PatternsAttachment work/uploaded_files "Synchronize with MID Servers" UI Action can be used to force a synchronisation, which calls "SaUploadedFiles" Script Include to call "MIDNotificationHandler " Script Include "notifyMIDServers" function to insert a similar ECC Record as above. The "GetAllUploadedFiles" Scripted SOAP Service is used by the MID Server to get all service mapping uploaded files. How the MID Server processes the job Each MID Server has its own FileChange output in the ecc_queue, which will be specific to a table. If the MID Server is UP, connected to the instance, and has available interactive threads, then it will puck up the job from the ECC Queue and execute it. Note: the following is all in the Java code compiled into the MID Server, and not code customers can inspect. There is some logging added to the MID Server Agent log as this progresses. The SystemCommand object checks the name field of the ECC Queue output, and if these are FileChange, it creates a SyncedFileChangeEvent where the type is the table to sync. This then creates a new FileSyncWorker thread for the actual Sync, using a Syncher specific to the type, which predefines some attributes such as useAttachment, and target folder, and field to use. JarSyncer, MIBSyncer, ScriptFileSyncer, AssetFileSyncer extend AFileSyncerNdlFileSyncer and SaUploadedFileSyncer, extend DelayedFileSyncer. Delayed file syncher will not sync files until the enabled flag is changed. The FileSyncWorker code in the MID server then manages the process. It starts by grabbing a Snapshot of a specified set of files to be synchronized to the MID server. This is requested from the "MIDFileSyncSnapshot" Scripted SOAP Service [sys_web_service] in the instance. Evidence of that request happening can be seen in the Transaction Log [syslog_transaction] with URL "/MIDFileSyncSnapshot.do?SOAP". It processes the directories recursively, one directory at a time. Within the directory, each file is processed separately, and from the Snapshot knows these attributes: useAttachment - true if an attachment exists. This will determine which mode is used. An Attachment takes precedence over the script field, if both exist. MIDFileSyncSnapshot will only return the most recent attachment for a file, if there are more than one. False will be given if a field is to be used instead.name - file_name of the sys_attachment, or name of the Synchronized File record if no attachment.id - sys_id of the sys_attachment, or sys_id of the Synchronized File record if no attachmenttime - sys_updated_on/sys_created_on of the sys_attachment, or sys_updated_on of the Synchronized File record if no attachment.checksum - MD5 checksum of the sys_attachment from the Synchronized File record field, if a ecc_agent_script_file record. Depending on the mode, the IDownloadedFileWriter function uses InstanceFormFieldToFileDownloader or InstanceFileDownloader to request from one or other of these Scripted SOAP Service to retrieve the field or attachment from the instance, which can also be confirmed in the instance transaction logs: MIDFieldForFileProvider - Requested by InstanceFormFieldToFileDownloader. Fetches the value of a given field for the MID server to write as a file, along with the field record's created and updated timesMIDServerFileProvider - Requested by InstanceFileDownloader. Provides file contents for files that are synchronized to MID servers. /sys_attachment.do may also be used to download the attachment to the MID Server from the instance. If the instance record includes a checksum, which it should for all ecc_agent_script_file attachments, then the MID Server checks this against the file it has downloaded. How it realizes a file was changed The logic to decide if a file needs replacing is if the local file exists and its "last modified" time is more than 4 seconds different (older or newer) than the file's timestamp in the instance (record/attachment sys_updated_on value), then we consider it to be different. The checksum or size is not used for deciding if a file should be replaced. Files that are missing from the disk because they have been manually deleted will be replaced by this process. Files and Directories may also be deleted from the MID Server by this process if: A file is left over or has been placed into one of the MID Server folders manually, and is not in the instance. Those are considered unauthorised files, and deleted.The attachments or records have been deleted from the instance, or the record with a field has been deleted.If Directory records have been deleted in the instance, all records with that parent are cascade deleted in the instance. On the disk, the folder and its contents are deleted.The file has been replaced. The file will be deleted first, then replaced by the new one. On Windows, JAR files need to be deleted after the MID server is shut down, due to a file lock put by the JVM class loader on startup. Things like this are deleted externally using a Powershell script, and this automated delete and replace process may cause up to 3 restarts before it is complete (with the batch file, PS script TBC). If checking the transaction logs of the instance, remember a MID Server will have a different session ID after a restart. Prior to versions San Diego, Rome Patch 4, Quebec Patch 10, a batch file "bin\filesyncdelete.bat" was used instead. Logs from that external delete powershell script/batch file are saved in agent/logs/filesyncdelete.log Scripted SOAP Services MIDFileSyncSnapshotTells the MID Server what there is to Sync in general, including the Directory trees.Then for each file, one or the other of the following is requested individually:- MIDFieldForFileProviderFor records with content in the FieldMIDServerFileProvider For records with attachments ECC Agent Script File "Checksum" field From the Utah release, any MID Server Script File based on the script field (not attachments ones) that doesn't have a value for the Checksum field will have one added by: business rule "Generate Checksum For Scripts"fix script "Generate Checksum For Script File"Events related to app installs/upgrades:- plugin.upgraded, processed by script action "Generate Checksum on Plugin Upgrade"plugin.activated, processed by script action "Generate Checksum on Plugin Activation" For anything with an attachment, and all ECC Agent Script File, the MID Server will verify these checksums during file sync, so that the checksum of the file created on disk matches what we calculated that it would be from the instance record fields. If there is a problem, a MID Server Issues record will be created with the message:"Checksum validation on MID server for <filepath> failed with checksum - <checksum>" Troubleshooting and Known issues An update for a single record will create a job for re-synchronising of all the records in its table. That duplication can cause performance problems when a large number records are updated at the same time, for a large number of MID Servers. Attachments deleted by Clones It is possible for the attachments to be missing after a clone, if the attachments are excluded and not preserved by the clone, which has been seen several times for sa_uploaded_file. See:KB0786475 MID Servers and Clones Trouble uploading the attachment to the form Possible Errors uploading the attachment: File type not permitted or mime type does not match the file content.If the instance is not configured to allow JAR files as attachments, perhaps because jar is listed in system property glide.attachment.blacklisted.extensions. Content too large for field payload; max length (storage): 16,777,215, compressed length: 16,940,311 (or whatever the file is)e.g. from large JDBC drivers. This does not prevent the attachment being added, nor synced to the MID Server. This is a platform error from the Update set/Versions system, and refers to the mediumtext "payload" field of the sys_update_xml / sys_update_version tables. It can be avoided by turning off versioning/update set tracking of the ecc_agent_jar table, but is not necessary, as the error can be ignored. Problem ticket for MID Server case tracking PRB1499103. Check MID Server agent log Restart the MID Server, and on startup all tables will be re-synched. Every table is synched, and within that table every file will be checked and synched if needed. Files for a specific folder are done together. Agent log, even without debug, will show each table being synchronised at startup, or when a FileChange SystemCommand runs. 2023-10-09T18:37:44.430+0200 INFO (FileSync:sn_agent_asset) [FileSyncWorker:81] Starting file synchronization: sn_agent_asset...2023-10-09T18:37:45.231+0200 INFO (FileSync:sn_agent_asset) [FileSyncWorker:141] Synchronizing 2 files to C:\MID SERVER\ServiceNow MID Server mid_server_dev\agent\static\acc_plugin\all\all\all\all2023-10-09T18:37:45.231+0200 INFO (FileSync:sn_agent_asset) [FileSyncWorker:174] Synchronizing C:\MID SERVER\ServiceNow MID Server mid_server_dev\agent\static\acc_plugin\all\all\all\all\app-ci-metrics-acc-commons.tar.gz2023-10-09T18:37:45.241+0200 INFO (FileSync:sn_agent_asset) [FileDownloader:35] Downloading C:\MID SERVER\ServiceNow MID Server mid_server_dev\agent\static\acc_plugin\all\all\all\all\app-ci-metrics-acc-commons.tar.gz from https://<instance>.service-now.com/sys_attachment.do?sys_id=ae33d183530221102f10ddeeff7b124b2023-10-09T18:37:45.241+0200 DEBUG (FileSync:sn_agent_asset) [AAttachmentFileDownload:45] Downloading attachment using UnsignedAttachmentFileDownload2023-10-09T18:37:45.610+0200 INFO (FileSync:sn_agent_asset) [AAttachmentFileDownload:84] Attachment successfully downloaded 1908 bytes2023-10-09T18:37:45.610+0200 INFO (FileSync:sn_agent_asset) [FileSyncWorker:174] Synchronizing C:\MID SERVER\ServiceNow MID Server mid_server_dev\agent\static\acc_plugin\all\all\all\all\acc-f-commons.tar.gz2023-10-09T18:37:45.610+0200 INFO (FileSync:sn_agent_asset) [FileDownloader:35] Downloading C:\MID SERVER\ServiceNow MID Server mid_server_dev\agent\static\acc_plugin\all\all\all\all\acc-f-commons.tar.gz from https://<instance>.service-now.com/sys_attachment.do?sys_id=f0378b99d1047110f877eca2213bddee2023-10-09T18:37:45.610+0200 DEBUG (FileSync:sn_agent_asset) [AAttachmentFileDownload:45] Downloading attachment using UnsignedAttachmentFileDownload2023-10-09T18:37:45.833+0200 INFO (FileSync:sn_agent_asset) [AAttachmentFileDownload:84] Attachment successfully downloaded 12619 bytes...2023-10-09T18:38:05.534+0200 INFO (FileSync:sn_agent_asset) [FileSyncWorker:92] Finishing file synchronization: sn_agent_asset Check the disk to see what was synchronised An ecc_queue output, for mid.server.*, with this DIR command will show what was synchronised to the various folders in the input. Agent: mid.server.*Topic: CommandName: dir /b /s static\*.*dir /b /s scripts\*.*dir /b /s work\mibs\*.*etc. Queue: outputPayload: <parameters><parameter name="skip_sensor" value="true"/></parameters> Any dir, ls, chmod, or other commands and combination of switches can be used, to check dates, sizes, permissions etc. Force a re-sync of any table without making Changes An ecc_queue output, for mid.server.*, with this SystemCommand will force a re-sync of the table, without having to make any changes to a customer instance. Agent: mid.server.*Topic: SystemCommandName: sn_agent_assetecc_agent_jarecc_agent_script_fileetc. Source: FileChangeQueue: output MID Server syncs the Folders, and Files from fields, but not Files from Attachments If the attachments do exist, but are not synched, this will probably be due to ACL problems for the Attachments, for the MID Server login user. The MID Server login user will need to pass ACL for the table records, but also for the sys_attachment table ACL for the attachments. e.g. for sn_agent_asset, which out-of-box has a directory tree defined, plus some attachment files. Turn on Debug SecurityAs admin/maint, Open a list of sys_attachment, filtered for table IS the table, to confirm the attachments existImpersonate the MID Server user (Web service access only=false on the User for impersonation to be possible)refresh the list and see which ACLs are failing. Look for "record/sys_attachment/read" ACLs where the CONTEXT is the attachment name.Compare those ACLs to a clean instance Out-of-box, it is this ACL that gives permission to read the attachments, if you can read the record it's attached to. /sys_security_acl.do?sys_id=0bcf23740a6a38d400c7e02590038464 That ACL just has a script. It should have no conditions, but depending on what other plugins are installed may have a Role requirement. If the Explicit Roles plugin is installed, the mid server user will need the snc_internal adding. Look out for Query business rules too. Record fields or attachments changed, but sys_updated_on didn't The following product enhancement problem exists for this scenario: PRB1639539 MID Server sync files with the instance only based on "sys_updated_on" field in the new insert/update xml This means if an out-of-box record is changed by servicenow development, but without changing the sys_updated_on value, the MID Server doesn't realise it has changed. Existing MID Severs will remain on the old version of the file. Only new MID Servers sychronised for the first time since the change will have the new file. Multiple attachments on the same record If you end up with multiple attachments on the same record, it will be random which of the attachments is synched to the mid servers. They won't all be synched. If using attachments, you must only have 1 attachment per record. General tips Clicking the UI Actions mentioned above usually resolve most synchronisation problems. If the ecc_queue output is not seen, then session debug can be used to confirm the business rules ran, and checking the sysevent table will show the attachment events were created or not. If the ecc_queue output is seen, is it stuck in ready state? Is the MID Server up and healthy? Does the agent log show the job running? You would expect to see the ecc_queue output record sys_id in the agent log for any log entries for the start and end of the job. Logs from the FileSyncWorker thread will show any problems with the snapshot, download and file operations. The ecc_queue input may show error messages from the MID Server code. There is a lot of logging that goes into the MID Server agent log if mid.log.level MID Server parameter is set to debug. Check only one attachment exists per record. Check all relevant Scripted SOAP Services, for Package=MID Server or Pattern Designer, are the current OOTB versions. Compare behaviour with other MID Servers on the same host, or other hosts. MID Servers with the same and different instance and service users. Installing a new clean MID Server can also help narrow down where the issue might be. Folder permissions are now checked on MID Server startup, but could be the cause on older versions. gs.log() can be added to various places in the scripted SOAP Services, to get more visibility of what files and attachments are being processed. You can remotely check the files in the target folder using a Topic=Command ecc_queue output, using a command in the Name field such as "dir /s" or "ls -lsR" and the ecc input will contain a directly listing that includes the timestamps and file sizes. Avoid installing a new MID Server by copying an existing install folder. It's best to install from scratch and allow the MID Server to do a full synchronization on startup. In one customer case, Patterns failed to get updated due to running an old external JVM, with no obvious errors in the agent log. Edit wrapper-override.conf to use the bundled agent/jre/bin instead to resolve this. Failure to fully download the attachment, or download a corrupted attachment, could be an issue with the instance database. You can download the attachments manually from the instance records to confirm the files are correct. Do you see the requests to the SOAP web services in the transaction logs? Do you see any errors from that in the app node localhost logs? Known Limitations KB0813383 Can a JAR file be pushed to a specific MID Server instead of all MID Servers? - quick answer: No. This applies to per-Domain JAR files too.KB1182832 Add or replace Java Classes in the MID Server, without using the JAR File synchronisation from the instance Related Problems PRB1433170 / KB0862383 The jar files for Oracle/MSSQL/MySQL JDBC drivers on MID Servers are in the lib folder, not the JAR Files table, and so can't be swapped out for different versions to be compatible with all versions of those databasesPRB1441208 / KB0862614 Oracle JDBC Driver 11.2.0.3.0 included in MID Server lib folder is old, and incompatible with at least Oracle 8iKB1192296 3rd party Evanios EVAgent JAR file causes Mid Server stack crash for various threads due to log4j and slf4j classes calling each other in an infinite loopThere has also been a problem with a Pattern, where the new version of the pattern has the same updated_on value as the old version (TBC).