TOI : File based Discovery through Agent<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: block; max-width: ; width: auto; height: auto; } } File based Discovery through AgentOverview : File-based Discovery helps identify software running on Windows and UNIX servers and devices. It enables organizations to manage software licenses, detect unlicensed or forbidden files, and evaluate potential security threats. The collected file information is stored in the File Information (cmdb_file_information) table with a reference to the server's Configuration Item (CI). If Software Asset Management Professional (SAMP) is active, File-based Discovery creates or updates software product records in the Software Installation (cmdb_sam_sw_install) table and updates license information for matched software packages. Without SAMP, only file information is stored, and no software records are created. Any file that remains unmatched by the normalization process is inserted into the Unidentified File Set (cmdb_unidentified_file_set) table. Users can update this table by providing additional details, such as the product name and publisher. If these values are provided, and SAMP settings permit, File-based Discovery can use this information for future software identification. The agent machine collects the following file attributes: File nameFile sizeFile version (not always available)File path For more information on the File Signature Normalization plugin, refer to the official documentation. Plugins to be installed : Discovery (com.snc.discovery)File based Discovery (com.snc.discovery.file_based_discovery)Agent Client Collector for Visibility Content (sn_acc_vis_content)Software Asset Management Professional (com.snc.samp) Configuration : The user needs to configure the following settings for file-based discovery on the configuration console: Enable File Based Discovery: Switch to enable file-based discovery.Folders to scan: Enter the path to the folders that need to be scanned.Folders to ignore: Enter the path to the folders that need to be ignored..Extension exclusion list: Enter the list of extensions that should not be scanned.Extension wildcards: Enter the list of extensions that should be specifically scanned.Sleep time: Enter the amount of time the process should sleep after scanning a specific number of files.File Throttle: Enter the number of files that need to be scanned before the process sleeps.Skip Hidden Folders: Switch to determine whether hidden folders should be skipped during scanning. If enabled, they will be skipped; otherwise, they will be scanned.Scan Running Process Paths: Switch to determine whether running process paths should be scanned. If enabled, they will be scanned; otherwise, they will be skipped.Maximum Files: Enter the maximum number of files to be scanned. For Linux, exclusion wildcards are not supported as a configuration option. However, users can create a corresponding record for the file they want to discover in the samp_custom_file_name table by specifying the file name and setting the platform as "Linux." Additionally, all other options available in the Configuration Console are not supported for File-based Discovery via the Agent. Files Added/Changed as a part of this feature Instance Level: Business Rule: Added to enable or disable the newly created policies.Scheduled Job: A new job runs every 24 hours to collect File-Based Discovery (FBD) parameters from the Configuration Console and updates the configuration files. ["AgentFBDWindowsParameters.json" and "AgentFBDUnixParameters.json"]Policies/Check Definitions: Added two new policies along with their corresponding check definitions. [File based discovery background policy and File based discovery policy]Script Includes: EnhancedDiscoveryHandler: Modified to process the FBD payload.EndpointFileBasedDiscoveryHelper: Newly added script, called from the EnhancedDiscoveryHandler, to persist data in the relevant tables. Agent Level: Ruby Scripts: Added two new Ruby scripts, one for each policy. Flows : Prerequisite Flow for File-Based Discovery to Work I. Data Services Scheduled Jobs: "Download Software Content: File Name""Download Software Content: File Set""Download Software Content: File Map"To verify whether all three scheduled jobs have been executed, go to the link below and ensure that all records have a "Last Updated on" value that is not null.https://<instance_name>.service-now.com/cds_client_schedule_list.do?sysparm_query=table%3D1778fd43e7731300ba990558d2f6a9fd%5EORtable%3D1a67fd43e7731300ba990558d2f6a908%5EORtable%3Df859f183e7731300ba990558d2f6a9e0&sysparm_first_row=1&sysparm_view=data_services_schedule II. File-Based Discovery Scheduled Jobs: "Discovery - File Discovery Whitelist Update Notification" (Runs automatically based on schedule)"Discovery - File Discovery Whitelist Update Notification (Force Update)" (Can be manually triggered for an immediate update) III. ACCVC Specific Scheduled Job: "Refresh FBD Config Files" All scheduled jobs run automatically at their scheduled intervals. However, users can run them on demand if immediate results are required. Scheduled Jobs I and II: Only need to run once after installation of plugins.Scheduled Job III: Required if any changes are made in the Configuration Console.Scheduled Job II: Required if any changes are made to the samp_custom_file_name table. File-Based Discovery Background Policy Flow This policy performs the following tasks: 1. Deploying Configuration Files Configuration files are shipped from the instance to the host machines. Each policy (associated with a check instance) includes three attached configuration files: AgentFBDWindowsParameters.json – Contains configuration settings for Windows file scanning.AgentFBDUnixParameters.json – Contains configuration settings and a list of whitelisted file names for Unix file scanning.WindowsWhitelist.txt – Contains a list of whitelisted file names for Windows file scanning. All three files are deployed on the agent host machine. 2. File Scan The system performs a file scan using the parameters from the configuration file and stores the output on the agent.Flowchart for the ruby file associated with this policy is : Flowchart for the "Scan process" is File-Based Discovery Policy Flow This policy performs the following tasks: Collecting Output Data – Retrieves the output file from the agent and sends the data to the instance. (Handled by the Ruby script)Persisting Data – Stores the collected information in the respective tables on the instance. (Handled by the EndpointFileBasedDiscoveryHelper script include) Flowchart for the ruby file associated with this policy is : Flowchart for the instance side script include is : Note : For Windows File-Based Discovery, the ServiceNow user do not have the necessary permissions. In this case, the user needs to manually grant the 'List Folder Contents' permission to the folder that needs to be scanned. Important Debugging Points / Common Issues 1. File-Based Discovery Policy is Not Returning Any Data The "File-Based Discovery Background" policy must run first, as it collects information and stores it in an output file. The File-Based Discovery policy retrieves this data.If the check is not returning any data, allow additional time for the "File-Based Discovery Background" policy to complete its execution or wait for it to trigger before troubleshooting further.If the "File-Based Discovery Background" policy is currently running, a file named scan_process.pid will be present in the config-files folder on the host machine.In this case, the "File-Based Discovery" check will return the status "Previous job in execution." 2. File Missing in File Information Table Ensure that the path to the root directory where the file is located is correctly specified in the Configuration Console.Verify that the file name is valid and not a dummy file (e.g., test1.txt or similar). Such files are filtered out by the whitelist file.If the file is being filtered out, add the particular file extension to the extension wildcards and then run the "Refresh FBD Config Files" scheduled job for Windows, for immediate testing.For Unix, add the file name to the samp_custom_file_name table and run the "Discovery - File Discovery Whitelist Update Notification (Force Update)" scheduled job for immediate testing. 3. Software record is not created but corresponding file record is created. Software records are only created for files that are normalized properly. Normalization is done by a SAM API.The output of the API will be printed in the system logs.Look for a record in the system log containing the string "FBD API result" along with the agent ID of the agent.