Data Virtuality Platform 3.0.4 (Beta): new job queue feature, OData bug fix, and other improvements

With this release, we bring to you a new great feature for the Server and Studio: job queue. To support it, we have implemented several additions:

• one job status: QUEUE
• one option: MAX_JOB_QUEUE_SIZE option
• two tables: SYSADMIN.jobQueue and SYSADMIN.jobQueueStats
• two columns: inQueue in the SYSADMIN.ScheduleJobs table and queueHandler in the SYSADMIN.Schedules, SYSADMIN.createSchedule, SYSADMIN.updateSchedule, and SYSLOG.JobLogs tables
• four procedures: SYSADMIN.cleanupJobQueue,  SYSADMIN.createQueueHandler, SYSADMIN.editQueueHandler, and SYSADMIN.dropQueueHandler
• two properties had to be deprecated: org.quartz.jobStore.class and com.datavirtuality.quartz.DVJobStore.misfireThreshold, and are not supported by the Exporter anymore.

We have updated our documentation with a detailed description of how to work with job queues and queue handlers: you can consult it here.

For the Server, we have also fixed a bug affecting OData in which foreign keys set in a view caused NullPointerException on requesting data from other objects - now everything should work fine. As for other improvements on the Server side, most importantly, we have fixed a bug affecting work with Google Ads API: previously, running several queries simultaneously might cause out-of-memory errors. Now you can safely run multiple Google Ads jobs without encountering errors. Another important bug fix improves REST API queries performance: we have resolved a bug causing the JsonStreamer.write method to not use a buffer and send the data byte by byte to the RESTEasy output stream. 

A whole bunch of fixes concerns Column Level Data Lineage: the path is now highlighted for a view if the query contains aggregation or DISTINCT or arithmetic operation or concatenation; the SYSADMIN.getDataLineageWithRelationsOnly procedure will not display temporary tables in the plan and will not fail with an assertion error anymore and will return the correct information for materialized views with enabled optimization; and WITH queries (also known as CTEs) will return correct results on generation of data lineage.

As for the Studio, apart from adding support for job queues, we have fixed three bugs, thus improving performance and security. The job tab will no longer freeze if there is a job with a comma in the description; if a Dependency graph or an Impact analysis tab is open for a non-existent object, no error message will be shown upon reconnecting (the tab will now display a relevant message, 'The %procedure|table|view|schema' does not exist;); and last but not least, virtual schemas will not be shown in the Data Sources section if permissions are set to objects in those schemas but not the schemas themselves.

And finally, an improvement and a bug fix for the connectors: for Amazon Selling Partner, the Orders procedures will now save data into subordinate tables, and for Google Ads API, reports with the bidding_strategy and accessible_bidding_strategy parameters will yield correct results.

Here are all issues in this release:

Server

• DVCORE-7672 (New Feature): Create a new procedure to immediately trigger job execution
• DVCORE-7384 (New Feature): Implement the job queue
• DVCORE-7755 (Improvement): Google Ads API: fix mapping of date fields and add non-pushing down for specific tables
• DVCORE-7694 (Improvement): MongoDB: update the driver to version 4.5.1
• DVCORE-7665 (Improvement): Introduce job queue handlers
• DVCORE-7657 (Improvement): Make it possible to directly assign values to array items
• DVCORE-7646 (Improvement): MySQL: update the JDBC driver to version 8.0.28
• DVCORE-7389 (Improvement): Display job interruption reason for interrupted jobs
• DVCORE-5173 (Improvement): Add an INFO log message for situations when maximum number of running jobs is already reached and a new one is added in the queue
• DVCORE-7756 (Bug Fix): OData: foreign keys set in a view cause NullPointerException on requesting data from other objects
• DVCORE-7754 (Bug Fix): REST API: JsonStreamer.write method does not use a buffer and sends the data byte by byte to the RESTEasy output stream
• DVCORE-7744 (Bug Fix): Column Level Data Lineage: a path is not highlighted for a view if a query contains aggregation or DISTINCT or arithmetic operation or concatenation
• DVCORE-7741 (Bug Fix): Dependency graph generation fails when there is an uppercased virtual schema with a procedure respectively eponymous to an uppercased schema and table in a data source configured with importer.useFullSchemaName set to true
• DVCORE-7728 (Bug Fix): Column Level Data Lineage: SYSADMIN.getDataLineageWithRelationsOnly procedure returns temporary tables in the plan and in some cases fails with an assertion error
• DVCORE-7727 (Bug Fix): Redshift: no tables are shown for a data source with importer.schemaPattern value consisting of only wildcards
• DVCORE-7723 (Bug Fix): Google Ads API: running several queries in parallel may result in out of memory errors
• DVCORE-7574 (Bug Fix): SYSADMIN.getClientId adds a linebreak to client IDs defined with Windows line endings
• DVCORE-7395 (Bug Fix): Column Level Data Lineage: an empty targetColumnSchema value is returned when plan contains a materialized table in Snowflake
• DVCORE-6827 (Bug Fix): Column Level Data Lineage: WITH queries cause incorrect result on generation of data lineage

Studio

• DVCORE-7460 (New Feature): reflect new job queuing functionality
• DVCORE-7693 (Bug Fix): job tab freezes if there is a job with a comma in the description
• DVCORE-7674 (Bug Fix): an error message is shown upon reconnecting if a Dependency graph or an Impact analysis tab is open for a non-existent object
• DVCORE-7009 (Bug Fix): virtual schemas are shown in Data Sources section if permissions are set to objects in those schemas but not the schemas themselves

Exporter

• DVCORE-7666 (Improvement): Data Virtuality Exporter: add export for job queue handlers

Connectors

• SQL-497 (Improvement): Amazon Selling Partner: Orders procedures should save data into subordinate tables
• SQL-537 (Bug Fix): Google Ads API: fix bidding_strategy and accessible_bidding_strategy parameters in reports