How MID Server File Synchronisation works, to help when Troubleshooting


Several features and applications share the same MID Server platform code for getting records and attachments in the instance synchronised to files on the MID Servers. This article aims to describe the steps and code involved to aid troubleshooting.

The Tables involved

For a table to use the system, it has to extend MID Server Synchronized File [ecc_agent_sync_file]. Below is the schema map for the extensions of the MID Server Synchronized File table in Orlando, with ITOM plugins installed.

How a synchronisation job is triggered

Attachments

When an attachment is added to a record in any of these tables, that file is automatically placed on the disk within the MID Server install folder tree of all MID Servers. The file is then automatically re-synchronised with all MID Servers as the files are changed in the instance.

The synchronisation is usually triggered by the system events created when records are added/changed/deleted from the Attachment [sys_attachment] table. The names of the Events [sysevent] are below, and Parm1 of the event is the table name to which the attachment belongs:

For each event, there is a Script Action [sysevent_script_action], all named "MID Server file synchronization check", which tells the MID Servers to re-synchronise that table.

The Script Actions call "MIDServerFileSync" Script Include, "notifyMIDServers" function, using the table name from the event's Parm1, to insert an ECC Queue table [ecc_queue] output record:

The "Probe - All MIDs" Business Rule on ecc_queue then copies that "mid.server.*" record, and inserts an output specific to each MID Server. All MID Servers will have the job created, regardless of whether they are Up, or whether there are any existing duplicates still in Ready state in the queue.

Fields

Since 2013, the contents of fields can also be synchronised to files on the MID Server. Where this is the case, Business rules on the specific table trigger the FileChange Jobs.

Applications using this system

The following table lists each of the extended table classes, their names, applications which use them, and anything different to the above normal event triggered synchronisation. Many other things are synchronised to memory caches in the MID Server, but don't end up as files. 

TableUsed byAttachment or FieldDirectory
Agent Client Collector Plugin [sn_agent_asset]Event ManagementAttachmentstatic\acc_plugin\
Notify MID on Agent Asset Changes" Business Rule on sn_agent_asset calls "MIDNotificationHandler " Script Include "notifyMIDServers" function to insert a similar ECC Record as above.
Discovery Patterns [sa_pattern]Discovery
Service Mapping
Field: Pattern [ndl]work\ndl

"Notify MID server on NDL change" Business Rule on sa_pattern calls "PatternLibrary" Script Include "notifyMIDs" function.

"Synchronize with MID Servers" UI Action on sa_pattern forces synchronisation by updating the sys_updated_on value of the pattern. This also calls "PatternLibrary" Script Include "notifyMIDs" function.

"Pattern Sync to Mid" UI Action on MID Server form uses "PatternLibrary" Script Include functions to synch patterns and other Pattern-related synchronisations.

These 3 methods all use "PatternLibrary" Script Include "notifyMIDs" function, which calls "MIDNotificationHandler" Script Include "notifyMIDServers" function to insert a similar ECC Record as above.

MID Server Extension JAR File [ecc_agent_ext_jar]TBCAttachmentTBC
MID Server JAR File [ecc_agent_jar]Import Set JDBC Drivers
Orchestration
External Credential Store
Cloud Management
Attachmentextlib
MID Server MIB File [ecc_agent_mib]SNMP DiscoveryAttachmentwork\mibs
MID Server Script File [ecc_agent_script_file]Discovery
Orchestration

Attachment  or Script [script] field.

scripts\<parent directories>

The target directory will depend on the parent(s) of the record. Records with directory=true define the folder structure, and are not actual files.

A couple of other Business Rules in the attachment table update the Synchronized File record:

  • Insert/Update: Update MID script on attachment change
  • Delete: Update MID script on attachment deletion

These update or clear/inactivate the following fields on the Synchronized File record:

  • script_attachment - sys_id, referencing the Attachment [sys_attachment] record. This is important if more than one file is attached.
  • checksum - MD5 Checksum of the file in the Attachment table
  • use_attachment - Tells the MID Server to use the attachment, rather than a field value
  • active - set false if no file is attached
Uploaded File [sa_uploaded_file]Discovery and Service Mapping PatternsAttachment

work/uploaded_files

"Synchronize with MID Servers" UI Action can be used to force a synchronisation, which calls "SaUploadedFiles" Script Include to call "MIDNotificationHandler " Script Include "notifyMIDServers" function to insert a similar ECC Record as above.

The "GetAllUploadedFiles" Scripted SOAP Service is used by the MID Server to get all service mapping uploaded files.

How the MID Server processes the job

Each MID Server has its own FileChange output in the ecc_queue, which will be specific to a table.

If the MID Server is UP, connected to the instance, and has available interactive threads, then it will puck up the job from the ECC Queue and execute it.

Note: the following is all in the Java code compiled into the MID Server, and not code customers can inspect. There is some logging added to the MID Server Agent log as this progresses.

The SystemCommand object checks the name field of the ECC Queue output, and if these are FileChange, it creates a SyncedFileChangeEvent where the type is the table to sync. This then creates a new FileSyncWorker thread for the actual Sync, using a Syncher specific to the type, which predefines some attributes such as useAttachment, and target folder, and field to use.

The FileSyncWorker code in the MID server then manages the process. It starts by grabbing a Snapshot of a specified set of files to be synchronized to the MID server. This is requested from the "MIDFileSyncSnapshot" Scripted SOAP Service [sys_web_service] in the instance. Evidence of that request happening can be seen in the Transaction Log [syslog_transaction] with URL "/MIDFileSyncSnapshot.do?SOAP".

It processes the directories recursively, one directory at a time. Within the directory, each file is processed separately, and from the Snapshot knows these attributes:

Depending on the mode, the IDownloadedFileWriter function uses InstanceFormFieldToFileDownloader or InstanceFileDownloader to request from one or other of these Scripted SOAP Service to retrieve the field or attachment from the instance, which can also be confirmed in the instance transaction logs:

/sys_attachment.do may also be used to download the attachment to the MID Server from the instance.

If the instance record includes a checksum, which it should for all ecc_agent_script_file attachments, then the MID Server checks this against the file it has downloaded. 

When changes to files will happen

The logic to decide if a file needs replacing is if the local file exists and its "last modified" time is more than 4 seconds different (older or newer) than the file's timestamp in the instance (record/attachment sys_updated_on value), then we consider it to be different. The checksum or size is not used for deciding if a file should be replaced.

Files that are missing from the disk because they have been manually deleted will be replaced by this process.

Files and Directories may also be deleted from the MID Server by this process if:

On Windows, JAR files need to be deleted after the MID server is shut down, due to a file lock put by the JVM class loader on startup. Things like this are deleted externally using a batch file "bin\filesyncdelete.bat", and this automated delete and replace process may cause up to 3 restarts before it is complete. If checking the transaction logs of the instance, remember a MID Server will have a different session ID after a restart.

MID Server code signing and validation

From the Quebec release, these script files will be signed and need validating or they will not be synchronised. More information is in the docs:
https://docs.servicenow.com/bundle/quebec-servicenow-platform/page/product/mid-server/concept/mid-code-sign-validation.html

Troubleshooting and Known issues

An update for a single record will create a job for re-synchronising of all the records in its table. That duplication can cause performance problems when a large number records are updated at the same time, for a large number of MID Servers.

It is possible for the attachments to be missing after a clone, if the attachments are excluded and not preserved by the clone, which has been seen several times for sa_uploaded_file. See:
KB0786475 MID Servers and Clones

Possible Errors uploading the attachment:

Clicking the UI Actions mentioned above usually resolve most synchronisation problems.

If the ecc_queue output is not seen, then session debug can be used to confirm the business rules ran, and checking the sysevent table will show the attachment events were created or not.

If the ecc_queue output is  seen, is it stuck in ready state? Is the MID Server up and healthy? Does the agent log show the job running? You would expect to see the ecc_queue output record sys_id in the agent log for any log entries for the start and end of the job. Logs from the FileSyncWorker thread will show any problems with the snapshot, download and file operations.

The ecc_queue input may show error messages from the MID Server code. There is a lot of logging that goes into the MID Server agent log if mid.log.level MID Server parameter is set to debug.

Check only one attachment exists per record.

Check all relevant Scripted SOAP Services, for Package=MID Server or Pattern Designer, are the current OOTB versions.

Compare behaviour with other MID Servers on the same host, or other hosts. MID Servers with the same and different instance and service users. Installing a new clean MID Server can also help narrow down where the issue might be.

Folder permissions are now checked on MID Server startup, but could be the cause on older versions.

You can remotely check the files in the target folder using a Topic=Command ecc_queue output, using a command in the Name field such as "dir /s" or "ls -lsR" and the ecc input will contain a directly listing that includes the timestamps and file sizes.

Avoid installing a new MID Server by copying an existing install folder. It's best to install from scratch and allow the MID Server to do a full synchronization on startup.

In one customer case, Patterns failed to get updated due to running an old external JVM, with no obvious errors in the agent log. Edit wrapper-override.conf to use the bundled agent/jre/bin instead to resolve this.

Failure to fully download the attachment, or download a corrupted attachment, could be an issue with the instance database. You can download the attachments manually from the instance records to confirm the files are correct. Do you see the requests to the SOAP web services in the transaction logs? Do you see any errors from that in the app node localhost logs?

Related Problems