ADDM staging tool may fail to stage data and report the error: System.IO.IOException: Stream was too long

Description

Summary

The BMC Atrium Discovery and Dependency Mapping (ADDM) staging tool may fail to stage data and report an error like the following:

2019-09-20 10:43:13.727 : ERROR : *** ERROR: An error occurred when attempting to read data from ADDM or when trying to write data to SQL Server: System.IO.IOException: Stream was too long.

Symptoms & Cause

The following is an example of logging that can be seen associated with this error:

2019-09-20 9:40:35.557 : INFO : ======== NetworkInterface Nodes
2019-09-20 9:41:12.974 : INFO : Response from ADDM in 0 seconds.
2019-09-20 9:41:13.146 : INFO : Staging ADDM query to file './Files/NetworkInterface.xml' complete
2019-09-20 9:41:13.146 : INFO : Transferred 27997259 bytes of data from ADDM
2019-09-20 9:41:13.146 : INFO : Import completed in 37.573 seconds
2019-09-20 9:41:13.146 : INFO : ======== Package Nodes
2019-09-20 10:43:13.727 : ERROR : *** ERROR: An error occurred when attempting to read data from ADDM or when trying to write data to SQL Server: System.IO.IOException: Stream was too long.
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
   at ADDMSource.ADDMQueryResultStream..ctor(Stream stream)
   at ADDMSource.ADDMDataReader.HTTPStream(String protocol, String server, String username, String password, Int32 timeout, String query, Boolean httpsCheckServerCertificate)
   at ADDMSource.Program.Main(String[] args)
2019-09-20 10:43:13.742 : ERROR : *** ERROR: Failed to stage ADDM query to file './Files/DiscoveredPackages.xml'
2019-09-20 10:43:13.742 : INFO : Retrying query with 2 attempts remaining for this mapping.
2019-09-20 10:43:13.742 : INFO : Import completed in 3720.276 seconds
2019-09-20 10:43:13.742 : INFO : ======== Package Nodes

This error can occur when too much data is returned from a query used when extracting data from ADDM.

Workaround

Within the FnmpADDMSettings.xml file, the “Package Node” query (or any other queries that results in this error occurring) can be split into multiple parts. This will look like the following:

<!--
Package nodes
Raw installer evidence gathered by ADDM from the operating system
supported installer technologies. This query can take a very long
time to run and return a very large number of records.
-->
<mapping description="Package Nodes A-M">
	<query>
		<![CDATA[
		SEARCH Host
		STEP IN Host:HostedSoftware:InstalledSoftware:Package
		WHERE #:Host:Host.key MATCHES '^(?i)[a-m]' 
		SHOW
		#:InstalledSoftware:Package.name AS 'Name',
		#:InstalledSoftware:Package.version AS 'Version',
		#:InstalledSoftware:Package.vendor AS 'Vendor',
		#:Host:Host.key as 'HostKey'
		]]>
	</query>
	<table filename="DiscoveredPackages.xml" db-name="DiscoveredPackages_ci">
		<column src="Name" dest="Package" data-size="255" data-type="System.String"/>
		<column src="Version" dest="Version" data-size="255" data-type="System.String"/>
		<column src="Vendor" dest="Vendor" data-size="255" data-type="System.String"/>
		<column src="HostKey" dest="HostKey" data-size="64" data-type="System.String"/>
	</table>
</mapping>

<mapping description="Package Nodes N-Z,0-9">
	<query>
		<![CDATA[
		SEARCH Host
		STEP IN Host:HostedSoftware:InstalledSoftware:Package
		WHERE #:Host:Host.key MATCHES '^(?i)[n-z,0-9]' 
		SHOW
		#:InstalledSoftware:Package.name AS 'Name',
		#:InstalledSoftware:Package.version AS 'Version',
		#:InstalledSoftware:Package.vendor AS 'Vendor',
		#:Host:Host.key as 'HostKey'
		]]>
	</query>
	<table filename="DiscoveredPackages.xml" db-name="DiscoveredPackages_ci" append="true">
		<column src="Name" dest="Package" data-size="255" data-type="System.String"/>
		<column src="Version" dest="Version" data-size="255" data-type="System.String"/>
		<column src="Vendor" dest="Vendor" data-size="255" data-type="System.String"/>
		<column src="HostKey" dest="HostKey" data-size="64" data-type="System.String"/>
	</table>
</mapping>

<mapping description="Package Nodes NOT A-Z,0-9">
	<query>
	<![CDATA[
		SEARCH Host
		STEP IN Host:HostedSoftware:InstalledSoftware:Package
		WHERE #:Host:Host.key NOT MATCHES '^(?i)[a-z,0-9]' 
		SHOW
		#:InstalledSoftware:Package.name AS 'Name',
		#:InstalledSoftware:Package.version AS 'Version',
		#:InstalledSoftware:Package.vendor AS 'Vendor',
		#:Host:Host.key as 'HostKey'
		]]>
	</query>
	<table filename="DiscoveredPackages.xml" db-name="DiscoveredPackages_ci" append="true">
		<column src="Name" dest="Package" data-size="255" data-type="System.String"/>
		<column src="Version" dest="Version" data-size="255" data-type="System.String"/>
		<column src="Vendor" dest="Vendor" data-size="255" data-type="System.String"/>
		<column src="HostKey" dest="HostKey" data-size="64" data-type="System.String"/>
	</table>
</mapping>

Fix Details

The above example divides the Package Nodes query into 3 parts, each of which retrieve a portion of the overall data set to be returned. The specific queries to use may need some modifications depending on what the data looks like.

The split query would have the following components:

The MATCHES clause where it matches the specified letters(case insensitive) and numbers at the start of the Host.Key field.
The other very important part is the Package Nodes NOT A-Z,0-9 where it will catch everything that the first two queries didn’t capture. Having the NOT MATCHES query at last is very important as it will catch any Host.Keys that start with symbols, punctuation or other non-alphanumeric characters.
The final important part is the append="true" attribute on all but the very first query of a particular type; failure to use that attribute correctly will result in the table being cleared out when that queries data is imported.

We may need to break this up into multiple queries depending on data size and specific requirements. This can be done by adding more queries and updating the MATCHES clause to narrow the matching criteria further. For example, the first query will just work on [A-M], then the next work on [N-Z], same can be done with numbers [0-1], [1-2].. etc.

Fix status

There are no plans to address this issue.

Was this helpful?

Support Hub

Product

Flexera One

Deployment type

Cloud

Visibility

Guest

URL Name

addm-staging-tool-may-fail-to-stage-data-and-report-the-error-system-io-ioexception-stream-was-too-long-IOJ-2077955

Are you sure you want to cancel the case creation?

Case closed successfully

Action performed successfully

Upload Attachment

Maximum file size allowed is 3 MB.

File type not supported.

Supported file types:

Documents (.txt, .doc, .docx, .pdf), Images (.jpg, .png), Comma Separated Files (.csv) Speadsheets (.xlsx, .xls)

Are you sure you want to close this case

Get Hotline support

Products	Region	Phone Numbers
FlexNet Operations FlexNet Embedded FlexNet Publisher FlexNet Connect FlexNet Code Insight InstallAnywhere InstallShield	North America *	+1 630-332-2513 (toll) +1 877-279-2853 (toll-free in North America)
	Europe *	+44 1925 944367 (toll) +44 800 047 8642 (toll-free in Europe)
	Japan *	+81 3-4540-5335 (select option 2)
	Australia *	+61 3 9895 2177 +61 1800 560 603 (toll-free in Australia)
Usage Intelligence (formerly Revulytics) Compliance Intelligence	Please use the Case Portal to submit your support ticket or reach out to your Revenera contact.

Summary

Symptoms & Cause

Workaround

Fix Details

Related Articles

Please wait...

Case id: 00001065