We're happy to announce ExMeX Core Framework 3.4.0, a significant release that brings enhanced stability, improved performance, and better operational control to the framework. This version focuses on robustness improvements, module management capabilities, and comprehensive system reliability enhancements.

Key highlights include enhanced error handling with automatic rollback capabilities, flexible module orchestration with enable/disable functionality, improved performance through optimized data types, and streamlined installation processes with better logging and metadata management.

Upgrading to version 3.4 follows our commitment to Semantic Versioning principles. This minor release maintains full backward compatibility, ensuring you can upgrade confidently without disrupting existing functionality.

Enhanced Installation & Error Handling

Comprehensive Rollback Functionality

One of the most significant improvements in version 3.4 is the introduction of complete rollback capabilities. When an installation fails, the system can now automatically revert all changes, including data, tables, stored procedures, and other database objects, back to the previous release state.

This enhancement provides unprecedented safety during upgrades, ensuring your system remains stable even if unexpected issues occur during the installation process. For detailed guidance, refer to our new Rollback Guide documentation.

Improved Installation Logging

Installation processes now feature enhanced logging with separate log files for metadata installation and module compilation, providing clearer insights into each phase of the upgrade process.

Advanced Module Orchestration

Flexible Module Management

ExMeX 3.4 introduces powerful module orchestration capabilities, allowing you to enable or disable modules at the batch level. This granular control means the same module can be active in one batch while remaining disabled in another, providing unprecedented flexibility in system management.

Key benefits include:

• Independent module states across different batches
• Enhanced system maintenance capabilities
• Improved resource management
• Streamlined troubleshooting processes

Disabled modules are automatically ignored during batch execution, optimizing performance and resource utilization.

RowsRead Performance Control

A new capability allows you to disable RowsRead calculation for specific modules. This is particularly useful when reading from complex views where determining the row count can be time-consuming.

Background: Normally, the framework can determine the underlying table and retrieve row counts from statistics. However, in rare cases with complex business logic encapsulated in views, this is not possible. When RowsRead is disabled, the Data Logistics Framework writes NULL to the RowsRead column in the MetadataZone.ProcessStepVapourTrail table, clearly indicating that the number of rows read is unknown to the framework.

This setting operates at the module-in-batch level providing granular control over performance optimization.

Load Window Management per Module

The ModuleUsesLoadWindow setting now applies to a unique module independent of the batch. When multiple instances of the same module exist in the BatchExecuteLoadProcessStep table, all related rows are updated simultaneously.

When ModuleUsesLoadWindow = 1, the load window generation function is activated in the module. This writes a new load window to the MetadataZoneCore.ModuleLoadWindow table.

Current Implementation: The load window can be used in views to filter PSA/HistoryZone data:

-- In view as join within the existing system
        FROM <source_data_object> st
        INNER JOIN [TEDAMOH-ExMeX-Release-Feature].MetadataZone.ProcessStepDeltaExport psde
        ON psde.SourceEntityClassCode = N'<your view/table>'
        -- Apply load window
        AND st.InscriptionTimestamp >  psde.DeltaExportAssertionTimeFrom
        AND st.InscriptionTimestamp <= psde.DeltaExportAssertionTimeBefore

Planned for Future Release: Direct filtering within modules without intermediate views:

-- In module with variables (recommended)
        FROM <source_data_object> st
        WHERE 1=1
        -- Apply load window
        AND st.InscriptionTimestamp >  @LoadWindowAfterTimestamp
        AND st.InscriptionTimestamp <= @LoadWindowToTimestamp

Improved manual batch execution

This enhancement provides a clearer output and compatibility with ExMeX release 3.4.0.

New version 2.4.0: ./Manual Batch.orig.sql

Intelligent Restart Logic

The framework now implements smarter restart mechanisms that ignore successful "TRY restart" attempts, focusing only on the latest execution state to determine whether a module should be executed or skipped.

Performance & Data Structure Optimizations

Database Performance Enhancements

ExMeX 3.4 includes significant performance improvements through optimized data types alignment, enhancing join performance in DAPIs by addressing root causes rather than applying temporary fixes in DAPI views.

Additionally, the main batch startup performance has been optimized, reducing overhead during batch initialization.

Enhanced Link Functionality

Link objects now support context columns (payload) that can be mapped as data item mappings. This enhancement enables advanced scenarios such as YearMonth partitioning for improved performance optimization.

SQL Agent Compatibility

Added automatic SET QUOTED_IDENTIFIER ON to each module, ensuring compatibility with SQL agents that set QUOTED_IDENTIFIER to OFF by default. This change enables the use of enhanced database options like filtered indexes.

Data Logistics Control Improvements

Optimized Load Window Management

Resolved duplicate load window issues by ensuring consistent granularity in the MetadataForExecuteDataLogisticLoadProcessStep view. Each module now maintains the same time period independent of batch assignment, eliminating data duplication from overlapping time periods.

Intelligent Subbatch Handling

During main batch restarts, the system now intelligently ignores subbatches that contain no modules requiring execution, optimizing processing efficiency.

Load Batch Dependency Fixes

Resolved issues in trLoadBatchDependency trigger by properly deleting records in edSatLoadBatchDependency, ensuring data consistency in batch dependency management.

Schema and Structure Updates

New Temporary Schema

Introduced MetadataZoneTemp schema for temporary data and objects, improving system organization and performance.

Enhanced Documentation

Documentation has been significantly improved with integrated search functionality and version selection capabilities, making it easier to find relevant information for your specific ExMeX version.

Data Structure Refinements

Multiple tables and views have been updated with optimized data types and streamlined structures:

• Enhanced stored procedures including new uspRegisteredModule
• Updated BatchExecuteLoadProcessSTep table with new module control columns
• Improved version tracking with the new MetadataZoneCore.ExMeXVersionHistory table
• Deprecated columns removal from various hub and metadata tables

General Improvements

System Organization & Documentation

• Revision Number Removal: Eliminated revision numbers from installed releases log (Table: MetadataZone.TEDAMOHExMeXRelease) to reduce confusion and improve clarity
• Enhanced Installation Output: Improved messaging during update installations for better visibility into the process
• New Temporary Schema: Introduced MetadataZoneTemp schema specifically for temporary data and objects, improving system organization
• Advanced Documentation: Significantly enhanced documentation system with integrated search functionality and version selection capabilities, making it easier to find relevant information for your specific ExMeX version

Resolved Issues

This release addresses numerous tickets and improvements:

Installation & Framework Core:

• 0000700: Enhanced error and logfile metadata installation
• 0000683: Complete rollback functionality for failed installations
• 0000703: Performance improvements through optimized data types
• 0000705: Performance start of main batch

Solution Patterns:

• 0000697: Context columns (payload) support for Links
• 0000712: SQL Agent QUOTED_IDENTIFIER compatibility
• 0000701: Resolved duplicate load windows for modules
• 0000714: Activate load window per module
• 0000713: Deactivate RowsRead

Data Logistics Control:

• 0000699: Improved TRY restart error handling
• 0000686: Flexible module orchestration capabilities
• 0000684: Intelligent subbatch restart management
• 0000710: Current manual batch to be delivered
• trLoadBatchDependency fixed: Delete record in edSatLoadBatchDependency

Upgrading to Version 3.4.0

Important: Before upgrading, create comprehensive backups of both your database and ExMeX framework folder. This release includes structural changes that require careful migration planning.

Pre-Upgrade Checklist

1. Backup Creation: Ensure complete database and framework folder backups
2. Column Usage Review: Check for usage of deprecated columns (AssertionTimeFrom and HighEndTimestamp) using the provided SQL query in the release notes
3. Custom Integration Assessment: Review any custom integrations that might be affected by data type changes

Migration Considerations

• Deprecated columns will be removed in ExMeX release 3.5
• Data type changes affect multiple tables and views
• Load window behavior may change for modules using time-based filtering

For detailed upgrade instructions, consult our Upgrade Guide.

New Installation

For fresh installations, follow the comprehensive steps outlined in our Install Framework Guide.

Looking Forward

ExMeX 3.4.0 represents a significant step forward in system reliability, performance, and operational flexibility. These enhancements lay the foundation for even more powerful features in future releases while maintaining our commitment to stability and backward compatibility.

For technical support or questions about this release, please refer to our documentation or contact our support team.

---

Appendix: Database Structure Changes

Stored Procedures

New Procedures

• uspRegisteredModule - New stored procedure for module registration and management

Modified Procedures

• uspSetDeltaExportTimePeriod - Added parameter @DeltaExportAssertionTimeBefore DATETIME2 (Default: NULL). If parameter is not set, SYSUTCDATETIME() is used instead of high-end-timestamp to prevent issues with load windows.

Tables

BatchExecuteLoadProcessStep

New Columns:

• ModuleIsActive - Controls module active/inactive state
• ModuleUsesLoadWindow - Indicates if module uses load window functionality
• ModuleUsesRowsRead - Controls whether RowsRead calculation is performed for the module

Column Behavior Notes:

• ModuleIsActive: Operates at module-in-batch level (independent per batch)
• ModuleUsesLoadWindow: Applies to unique module independent of batch (all related rows updated simultaneously)
• ModuleUsesRowsRead: Operates at module-in-batch level (independent per batch)

Version Tables

Replaced:

• Old: MetadataZone.TEDAMOHExMeXRelease
• New: MetadataZoneCore.ExMeXVersionHistory

Reason: Better consistency and removed revision numbers for less confusion

Data Type Optimizations

The following tables received data type changes to improve join performance:

Core Tables:

• [MetadataZoneCore].[HubLoadProcessStep] - Data type changes, dropped column EntityId (contained only -1, no added value)
• [MetadataZoneCore].[HubMapping] - Data type changes
• [MetadataZone].[HubEntityClassAttribute] - Data type changes, dropped column EntityId (contained only -1, no added value)
• [MetadataZoneCore].[SatMapping] - Data type changes
• [MetadataZoneCore].[SatModel] - Data type changes
• [MetadataZoneCore].[HubModel] - Data type changes
• [MetadataZone].[SatProcessStepDeltaExport] - Data type changes

Views

Deprecated Columns

MetadataForExecuteDataLogisticLoadProcessStep:

• [AssertionTimeFrom] - DEPRECATED (will be removed in ExMeX 3.5)
• [HighEndTimestamp] - DEPRECATED (will be removed in ExMeX 3.5)

Reason: Columns are not used in system and provide no added value

Updated Views (Data Type Inheritance)

The following views were updated to inherit data type changes from underlying tables:

• [MetadataZoneApp].[AppGetListOfMapping]
• [MetadataZoneApp].[AppGetFilterConditionAmount]
• [MetadataZoneApp].[AppGetListOfModel]
• [MetadataZone].[ProcessStepDeltaExport]
• [MetadataZoneStage].[GetMappingInBatchFromJson]
• [MetadataZoneStage].[GetTargetModelInBatchFromJson]
• [MetadataZoneMart].[DimLoadProcessStep]
• [MetadataZoneMart].[DimBatch]

New Schemas

MetadataZoneTemp

New schema introduced for temporary data and objects, improving system organization and performance.

Impact Assessment Query

To check if deprecated columns are used outside of the ExMeX framework, use this SQL query:

-- Search for column names in all object definitions
-- Example: Search for deprecated columns
DECLARE @ColumnName NVARCHAR(128) = N'AssertionTimeFrom'
--DECLARE @ColumnName NVARCHAR(128) = N'HighEndTimestamp'
DECLARE @ObjectName NVARCHAR(128) = N'MetadataForExecuteDataLogisticLoadProcessStep'

SELECT DISTINCT
    o.type_desc AS ObjectType,
    SCHEMA_NAME(o.schema_id) AS SchemaName,
    o.name AS ObjectName,
    m.definition
FROM sys.objects o
INNER JOIN sys.sql_modules m ON o.object_id = m.object_id
WHERE m.definition LIKE '%' + @ColumnName + '%'
    AND m.definition LIKE '%' + @ObjectName + '%'
    AND o.type IN ('P', 'V', 'FN', 'IF', 'TF', 'U') -- Stored Procedures, Views, Functions, User Tables
ORDER BY o.type_desc, SchemaName, ObjectName

Important: Review any custom code or integrations that might reference the deprecated columns before upgrading to avoid compatibility issues.