[Knowledge Managent] Demand Insights is not creating records in table [kb_task_knowledge_coverage] as during the execution of the KBCurationProcessor it throws "Read timed out from prediction call for ml_sn_global_similar_incidents_and_kbs solution"Issue The Scheduled Job [Knowledge Curation]: Generate Incident Clusters, which is Scheduled Script Execution, when executed calls the script include KBCurationProcessor with the sys_id of the record in the Demand Insights Configuration [kb_curation_config] table as an argument in this script. The Demand Insight configuration record for "incident" uses two Predictive Intelligence solutions, as follows - Demand Insights: Similar Incidents and Knowledge [ml_sn_global_similar_incidents_and_kbs solution]Demand Insights: Incident Clusters Need Knowledge [ml_sn_global_incidents_need_knowledge_cluster] The KBCurationProcessor script include uses the records in the Curation table with the Query condition configured in the Demand Insights Configuration [kb_curation_config] record to determine which records to process for the knowledge curation. For these records, it will check tables [m2m_kb_task] and [sys_journal_field] for referenced knowledge and it will also trigger a prediction on the configured Similarity solution to determine if there are knowledge articles that are similar in nature to the record being processed. For each record that does not have attached knowledge or the prediction on the Similarity solution for the record falls below the threshold, it will create a record in table [kb_task_knowledge_coverage], which is used as an Input table for the Clustering solution configured in the Demand Insights Configuration [kb_curation_config] record. Once it has processed all the incidents for knowledge curation, it will then execute the Clustering solution training that generates the Demand Insights Dashboards. However, during the execution of the KBCurationProcessor script include, it can throw the error "Read timed out from prediction call for ml_sn_global_similar_incidents_and_kbs solution" and it will stop the knowledge curation process resulting in considerably less records being generated in the Clustering solution Input table [kb_task_knowledge_coverage], hence the resulting Demand Insights Dashboard is missing data. The localhost logs has the following stack trace when this error occurs - 2022-07-04 10:00:59 (456) worker.0 worker.0 txid=1fa19ba897e0 SEVERE *** ERROR *** DxC_ML: dxc_id=2485787319809428 Received Read timed out from prediction call for ml_sn_global_similar_incidents_and_kbs solutionjava.lang.Exception: Read timed outat com.glide.platform_ml.api.predictionserver.SimilarityPredictionClient.predict(SimilarityPredictionClient.java:285)at com.glide.platform_ml.api.predictionserver.AbstractGlidePredictionClient.predict(AbstractGlidePredictionClient.java:127)at com.glide.platform_ml.api.solution.MLSolution.jsFunction_predict(MLSolution.java:118)at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.base/java.lang.reflect.Method.invoke(Method.java:566)at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138)at org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:670)at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:614)at org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2609)at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)at org.mozilla.javascript.gen.sys_script_include_da5a4ab2d72121004792a1737e6103c5_script_133._c_anonymous_20(sys_script_include.da5a4ab2d72121004792a1737e6103c5.script:337)2022-07-04 10:00:59 (461) worker.0 worker.0 txid=1fa19ba897e0 SEVERE *** ERROR *** sn_km_ml (): KBCuration :: Error occurred!!! ConversionError: The undefined value has no properties. The undefined value has no properties. at sys_script_include.23fc071773000010f84de1d28bf6a73f.script:164at sys_script_include.23fc071773000010f84de1d28bf6a73f.script:72at <refname>:2 The KBCurationProcessor script include also has Info logging enabled and you can search for "knowledge coverage" in the Message field using the module System Logs > System Log > all KBCuration :: 1 Started : Delete Task Knowledge Coverage Records for Curation Configuration:incident e6dac78973d40010f84de1d28bf6a7b6KBCuration :: 2 Finished : Delete Task Knowledge Coverage Records for Curation Configuration:sn_customerservice_case 6b8cd58384408410f877a5851c8d3870KBCuration :: 3 Started : Mark Tasks that need knowledge coverage KBCuration :: 4 Finished : Mark Tasks that need knowledge coverage KBCuration :: 5 Started : Trigger Clustering Job on the Tasks that need knowledge coverageKBCuration :: 6 Finished : Triggered Clustering Job on the Tasks that need knowledge coverage. Wait for the clustering job ml_sn_global_cases_need_knowledge_cluster to complete. When this prediction read timed out error occurs, the last Info log entry will be "KBCuration :: 3 Started : Mark Tasks that need knowledge coverage", as it will never finish this process. CauseThe default prediction timeout is 10 seconds, as the majority of prediction calls are synchronous, as the end user is waiting on the prediction to be returned before they can continue, which typically takes around 1 second to complete. For some Predictive Intelligence solutions, the prediction can take longer and in extreme cases can exceed the default timeout, which results in a prediction "Read timed out" error. However for the Demand Insights knowledge curation, as the predictions are triggered in a background process when executing the KBCurationProcessor script include, the prediction call can be considered as asynchronous, hence the default prediction timeout is irrelevant. However, the default prediction timeout can cause the Demand Insights knowledge curation process to fail, as it takes too long to make a prediction on the Similarity solution Demand Insights: Similar Incidents and Knowledge [ml_sn_global_similar_incidents_and_kbs solution].ResolutionThe prediction timeout can be set at the Predictive Intelligence solution level, so that you can override the default prediction timeout set to 10 seconds, as follows - In the Global scope, create two new "integer" system properties [glide.mlpredictor.ml_sn_global_similar_incidents_and_kbs.predict.request_timeout] and [glide.mlpredictor.ml_sn_global_similar_incidents_and_kbs.push.request_timeout]Set the value to 200000 [in ms] for both system properties. This sets the prediction request and the push request timeout to 200 seconds for the solution [ml_sn_global_similar_incidents_and_kbs] to handle very large models. The push request is when it needs to load the model from the instance into the Prediction Server memory Once created, trigger the Scheduled Job [Knowledge Curation]: Generate Incident Clusters again and check the logs to see if the knowledge curation completes successfully. Important note: You can create a system property to change the default prediction timeout for any predictive intelligence solution, as per the above steps using the solution name in the system property name. However, we do not recommend setting it lower than the default value of 10 seconds and increasing the timeout should only be done in certain circumstances such as the one described in this article. As always, test in sub-production first.Related LinksIf you need to exclude certain types of incidents from the Demand Insights knowledge curation process, you will need to make changes to the Similarity solution in the Demand Insights Configuration [kb_curation_config] record, as per KB1112338: How to remove specific record types, such as incidents automatically created from system events from the Similarity Solution definitions, so that they are excluded from showing in the Similarity Examples after training the solution.