Flow execution fails with error : Failed to iterate on data stream: com.glide.transform.transformer.exceptions.RecordSizeTooLargeException: Size of full record exceeds configured max buffer allocation of 10000000 bytes - Support and Troubleshooting

Flow execution fails with error : Failed to iterate on data stream: com.glide.transform.transformer.exceptions.RecordSizeTooLargeException: Size of full record exceeds configured max buffer allocation of 10000000 bytes Issue Flow execution fails with error "Failed to iterate on data stream: com.glide.transform.transformer.exceptions.RecordSizeTooLargeException: Size of full record exceeds configured max buffer allocation of 10000000 bytes"

Stack:

SEVERE *** ERROR *** Flow Designer: Operation<> failed with error: java.lang.RuntimeException: com.glide.transform.transformer.exceptions.RecordSizeTooLargeException: Size of full record exceeds configured max buffer allocation of 10000000 bytes at com.glide.transform.transformer.TransformerResultIterator.getNextRow(TransformerResultIterator.java:50) at com.glide.transform.transformer.TransformerResultIterator.hasNext(TransformerResultIterator.java:65) at com.snc.process_flow.stream.SplittingDatastream.getNextRecord(SplittingDatastream.java:105) at com.snc.process_flow.stream.SplittingDatastream.hasNext(SplittingDatastream.java:90) at com.snc.process_flow.stream.ScriptParserStream.getCo(ScriptParserStream.java:127) at com.snc.process_flow.stream.ScriptParserStream.hasNext(ScriptParserStream.java:99) at com.snc.process_flow.stream.VirtualPage.hasNext(VirtualPage.java:54) at com.snc.process_flow.stream.PaginatingDataStream.get(PaginatingDataStream.java:136) at com.snc.process_flow.stream.PaginatingDataStream.get(PaginatingDataStream.java:150) at com.snc.process_flow.stream.PaginatingDataStream.hasNext(PaginatingDataStream.java:78) at com.snc.process_flow.stream.PaginatingDataStreamProxy.hasNext(PaginatingDataStreamProxy.java:65) at com.snc.process_flow.stream.ComplexObjectDataStream.hasNext(ComplexObjectDataStream.java:31) at com.glide.plan.runners.scriptable.ScriptableDataStream.hasNext(ScriptableDataStream.java:79) at com.glide.plan.runners.scriptable.ScriptableDataStream.jsFunction_hasNext(ScriptableDataStream.java:48) at sun.reflect.GeneratedMethodAccessor2701.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138) at org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:670) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:614) at org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2609) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1501) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:829) at org.mozilla.javascript.InterpretedFunction.lambda$call$0(InterpretedFunction.java:152) at org.mozilla.javascript.Context$ScriptCaller.call(Context.java:2941) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:151) at org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2678) at org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2617) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1501) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:829) at org.mozilla.javascript.InterpretedFunction.lambda$call$0(InterpretedFunction.java:152) at org.mozilla.javascript.Context$ScriptCaller.call(Context.java:2941) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:151) at org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2678) at org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2617) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1501) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:829) at org.mozilla.javascript.InterpretedFunction.lambda$call$0(InterpretedFunction.java:152) at org.mozilla.javascript.Context$ScriptCaller.call(Context.java:2941) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:151) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:563) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3459) at org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:164) at com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:279) at com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:118) at com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:82) at com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:321) at com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:179) at com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:158) at com.glide.db.impex.custom.CustomLoader.loadDataFromScript(CustomLoader.java:71) at com.glide.db.impex.custom.CustomLoader.load(CustomLoader.java:52) at com.glide.db.impex.AbstractLoader.startWork(AbstractLoader.java:163) at com.glide.worker.AbstractProgressWorker.startAndWait(AbstractProgressWorker.java:126) at com.glide.worker.HierarchicalProgressWorker.startAndWait(HierarchicalProgressWorker.java:26) at com.glide.worker.AbstractProgressWorker.start(AbstractProgressWorker.java:101) at com.glide.processors.ImportProcessorWorker.startWorker(ImportProcessorWorker.java:331) at com.glide.processors.ImportProcessorWorker.loadDataSource(ImportProcessorWorker.java:172) at com.glide.processors.ImportProcessorWorker.startWork(ImportProcessorWorker.java:110) at com.glide.worker.AbstractProgressWorker.startAndWait(AbstractProgressWorker.java:126) at com.glide.worker.HierarchicalProgressWorker.startAndWait(HierarchicalProgressWorker.java:26) at com.glide.worker.BackgroundProgressJob.execute(BackgroundProgressJob.java:59) at com.glide.schedule.JobExecutor.lambda$executeJob$0(JobExecutor.java:115) at com.glide.schedule.JobExecutor.executeJob(JobExecutor.java:118) at com.glide.schedule.JobExecutor.execute(JobExecutor.java:102) at com.glide.schedule_v2.SchedulerWorkerThread.executeJob(SchedulerWorkerThread.java:300) at com.glide.schedule_v2.SchedulerWorkerThread.lambda$process$0(SchedulerWorkerThread.java:188) at com.glide.worker.TransactionalWorkerThread.executeInTransaction(TransactionalWorkerThread.java:35) at com.glide.schedule_v2.SchedulerWorkerThread.process(SchedulerWorkerThread.java:188) at com.glide.schedule_v2.SchedulerWorkerThread.run(SchedulerWorkerThread.java:102) Caused by: com.glide.transform.transformer.exceptions.RecordSizeTooLargeException: Size of full record exceeds configured max buffer allocation of 10000000 bytes at com.glide.transform.transformer.JsonStreamParserStrategy.fillBuffer(JsonStreamParserStrategy.java:312) at com.glide.transform.transformer.JsonStreamParserStrategy.getNextObject(JsonStreamParserStrategy.java:111) at com.glide.transform.transformer.JsonStreamParserStrategy.getNextObject(JsonStreamParserStrategy.java:100) at com.glide.transform.transformer.JsonStreamParserStrategy.getNextObject(JsonStreamParserStrategy.java:19) at com.glide.transform.transformer.StreamParserContext.getNextObject(StreamParserContext.java:20) at com.glide.transform.transformer.Transformer.nextRecord(Transformer.java:792) at com.glide.transform.transformer.TransformerResultIterator.getNextRow(TransformerResultIterator.java:42) ... 68 more

Cause By default the system considers the CHAR_BUFFER_MAX_SIZE_PROPERTY_NAME as 10000000(10 MB).

If the payload size is greater than 10 MB the flow fails with this exception.

Resolution Check if the payload is in JSON then,

Create a new property in sys_properties table as follows:

Name:com.glide.transform.json.char_buffer_max_size

Type:Integer

Value: greater than 10000000 and large enough to handle the XML returned

If the payload is in XML, then please follow the solution provided in KB0951728