LDW 2.3.23: Data lineage improvements for TEXTTABLE/XMTLABLE

We have again a large number of improvements in the Studio. Most noticeably, the rows of a result set are now numbered for easier reference:

On the backend-side, we made important improvements to the column-level data lineage feature. The usage of XML/CSV/JSON was previously not supported when generating the lineage, which was caused by the usage of the XMLTABLE/TEXTTABLE functions.

As an example, this is a simple view which parses a JSON file using the XMLTABLE function:

CREATE VIEW views.salestaxrate_from_json AS
SELECT "xmlTable".*
FROM "files".getFiles('./salestaxrate.json') f,
XMLTABLE(XMLNAMESPACES('http://www.w3.org/2001/XMLSchema-instance' AS "xsi"), '/root/salestaxrate/row' PASSING JSONTOXML('root', TO_CHARS(f.file, 'UTF-8'))
	COLUMNS 
	"CountryRegionCode" STRING PATH 'CountryRegionCode',
	"Name" STRING PATH 'Name',
	"ModifiedDate" STRING PATH 'ModifiedDate',
	"TaxType" STRING PATH 'TaxType'
) "xmlTable";;

The lineage for this view can now be generated successfully and shows the column (for which the lineage was generated), the view, the procedure of the data source reading the file, as well as the base directory configured in the data source.

Studio

• DVCORE-7296 (Improvement): rename "View history" to "View audit log"
• DVCORE-7277 (Improvement): remove varcharReserveAdditionalSpacePercent from defaults of "Add Analytical Storage" wizard for Snowflake
• DVCORE-7270 (Improvement): use the editor tab name as a suggested file name upon saving a script
• DVCORE-7261 (Improvement): restructure the statistics menu and remove the statistics page from the data source creation wizard
• DVCORE-7253 (Improvement): add row numbers to the query result set returned in the SQL Editor tab
• DVCORE-7251 (Improvement): align the row-level security dialogs opened from different places in the application
• DVCORE-7208 (Improvement): make "Analytical Storage" item in the "Data explorer" tab expandable
• DVCORE-6041 (Improvement): change the icon for tables/views with disabled optimization and failed materialization
• DVCORE-5952 (Improvement): make "From Address" a mandatory field in SMTP settings and use its value for manual sendMail and job notifications procedures
• DVCORE-5210 (Improvement): add a possibility to skip creating a schedule while creating an SQL, replication or materialization job
• DVCORE-7288: data source parameters are not refreshed when opening the edit wizard
• DVCORE-7271: the "Detach" right-click menu item for tabs is duplicated
• DVCORE-7254: some column checkboxes are not visible in the replication wizard for objects with large number of fields
• DVCORE-7233: wrong icons are shown for materialized tables in Snowflake Analytical Storage if useFullSchemaName is set to TRUE
• DVCORE-7219: NullPointerException when pressing "Alt Tab" after clicking a drop-down in the "Target data source" field of the replication wizard
• DVCORE-7170: changing a script via "Replace all" does not mark the script as unsaved
• DVCORE-7155: the "Preferences" button is active after logging out
• DVCORE-7137: code completion generates procedure calls without listing InOut parameters
• DVCORE-6316: pressing "Alt B" in dialogues opens a view builder dialogue
• DVCORE-6008: the right curly bracket used in procedure parameter annotation gets removed on a procedure call via double-click
• DVCORE-5418: list of jobs in the dependent schedule creation wizard contains the job for which it is opened
• DVCORE-7295: disabled autocompletion still works for ending quotes
• DVCORE-7310: the "Constraint" checkbox default values are different depending on where the "Set Permissions" wizard is called

Connectors

• SQL-353 (Improvement) YouTube Data: remove the deprecated GuideCategories procedure and update the Videos procedure to allow showing owner info
• SQL-352 (Improvement) YouTube Analytics: add the video_id parameter to the BasicUserActivityStatisticsForVideo procedure
• SQL-309 YouTube Data: snippet_publishedat parameter returns null values
• DVCORE-7272 Data Virtuality Synchronization Tool: dv_sync views and procedures creation fails due to a missing column in SYS.Tables
• DVCORE-7292 Snowflake: MONTHNAME returns the name with a "th" suffix
• DVCORE-6480 SQL Server: SELECT INTO of a BOOLEAN value fails for upload modes other than default one
• DVCORE-7298 SQL Server: inserting a STRING(1) value into a CHAR column fails
• DVCORE-5412 Excel: NullPointerException while creating a data source
• DVCORE-7302 Azure Synapse Analytics: strings containing line feed or carriage return cannot be written via BLOB upload mode

Backend

• DVCORE-7308 (Improvement): Improve the Java garbage collector log format and output configuration
• DVCORE-7304 (Improvement): Data Virtuality REST API: Add a property to configure request timeouts
• DVCORE-7275 (Improvement): Disable the recalculation of the column size when varcharReserveAdditionalSpacePercent is set to a non-default value
• DVCORE-7183 (Improvement): Column Level Data Lineage: add support for file data sources including usage with TEXTTABLE and XMLTABLE functions
• DVCORE-7100 (Improvement): Improve SSH Tunnels creation workflow by adding public and private key pair generation
• DVCORE-7174: Data Virtuality Exporter: exporting JBoss settings fails if a transport method is not present
• DVCORE-7128: Query plan for LEFT OUTER JOIN of several tables from the same data source and one table from a different one contains multiple unnecessary JOIN nodes
• DVCORE-7048: Column Level Data Lineage: getDataLineageWithRelationsOnly returns an incorrect result for special symbols in identifiers
• DVCORE-6995: Column Level Data Lineage: TABLE node contains duplicate column names when LAG or LEAD window functions are used
• DVCORE-6757: UTILS.upsert procedure cannot handle double underscore in column names
• DVCORE-5525: SYSADMIN.renameVirtualSchema procedure does not change the view/procedure references in DROP statements of updated view/procedure definitions