Skip to main content
Version: 3.4.0 (Latest Stable)

How to Execute ExMeX Framework Batches

The ExMeX Framework Core provides a comprehensive batch execution system for running data loading and processing workflows. This guide shows you how to configure and execute batches using the framework's core script.

Overview​

The ExMeX Framework Core script allows you to:

  • Execute predefined batches with various configurations
  • Run in Trace Mode (T) or Execute Mode (E)
  • Handle parallel execution with semaphore control
  • Restart failed batches from specific points
  • Execute single process steps for debugging
  • Monitor execution with detailed logging

Prerequisites​

  • Access to the ExMeX Framework database
  • SQL Server Management Studio or similar SQL client
  • Appropriate permissions for MetadataZone schema
  • Understanding of your available batches

Basic Configuration​

1. Set Your Batch Label​

The most important setting is choosing which batch to execute:

-- Set main (master) batch or batch for single batch execution
SET @BatchLabel = '<YourMainBatch>' -- e.g. SET @BatchLabel = N'BigOne'

2. Choose Execution Mode​

-- Set Execution Mode: T = Trace, E = Execute
SET @BatchExec = 'T'

Trace Mode (T): Simulates execution without making changes - perfect for testing Execute Mode (E): Actually executes the batch operations

3. Configure Parallel Processing​

-- Parallel Execution: 0 = Off, 1 = Parallel execution enabled
SET @ParallelExecution  = 1

-- Use semaphore for parallel execution: 0 = Off, 1 = On
SET @ModuleSemaphoreOn  = 1

-- Exit on error in batch: 0 = Off, 1 = Exit on Error
SET @ExitOnError        = 0

Common Use Cases​

Finding Available Batches​

Before executing, check which batches are available:

-- Available batches query
SELECT * FROM MetadataZone.LoadBatch;

-- Available main batches query
SELECT * FROM MetadataZone.LoadBatch WHERE ParentLoadBatchId IS NULL;

Standard Batch Execution​

For normal batch execution with default settings:

-- Configuration
SET @BatchLabel = N'YourBatchName'
SET @BatchExec = 'E'  -- Execute mode
SET @ParallelExecution = 1
SET @ExitOnError = 0
SET @ModuleSemaphoreOn = 1

-- Execute the script...

Testing a Batch (Trace Mode)​

To test a batch without making changes:

-- Configuration for testing
SET @BatchLabel = N'YourBatchName'
SET @BatchExec = 'T'  -- Trace mode only
SET @Debug = 0        -- Disable debug output

Restart a Failed Batch​

If a batch fails and you need to restart from a specific point:

-- Set BatchRunVapourTrailId for restart
SET @BatchRestartRunVapourTrailId = 23526  -- Use the VapourTrailId from failed run
SET @BatchExec = 'E'

Execute Single Process Step​

For debugging specific process steps:

-- Set ProcessStepId for single LPS execution
SET @SingleProcessStepId = 12345  -- Specific process step ID
SET @BatchExec = 'E'

Multiple Batch Execution​

To execute multiple batches in sequence:

-- Set batches for multiple batch execution
INSERT INTO @MultipleBatchLabel VALUES ('Default - Hub');
INSERT INTO @MultipleBatchLabel VALUES ('Special - Bitemporal Satellite Type');

Advanced Configuration​

SSIS Integration Override​

Override global SSIS settings if needed:

-- Override SSIS environment
SET @SSISLinkedServer     = 'YourSSISServer';
SET @SSISProjectName      = 'YourSSISProject';
SET @SSISFolderName       = 'YourSSISFolder';
SET @SSISProxyAccountName = 'YourProxyAccount';

Debug Mode​

Enable detailed debug output:

-- Debug: 1 = on; 0 = off
DECLARE @Debug BIT = 1;

Execution Phases​

The script executes in three distinct phases:

Phase 1: Batch Initialization​

  • Validates batch exists
  • Starts vapour trail logging (if in Execute mode)
  • Displays configuration summary

Phase 2: Batch Execution​

  • Executes the actual batch operations
  • Handles parallel processing and semaphore control
  • Manages error handling based on configuration

Phase 3: Batch Completion​

  • Reports final status
  • Closes vapour trail logging
  • Provides execution summary

Output and Monitoring​

The script provides comprehensive output including:

==========================================
ExMeX Framework Core - Batch Execution
Version: 2.4.0
Date: 2025-09-18
==========================================

Configuration Summary:
- Batch Label: BigOne
- Execution Mode: T (Trace)
- Parallel Execution: Enabled
- Exit on Error: Disabled
- Module Semaphore: Enabled
- Debug Mode: Disabled

==========================================
Phase 1: Batch Initialization
==========================================
SUCCESS: Batch "BigOne" found (ID: 42)

==========================================
Phase 2: Batch Execution
==========================================
Executing load process steps for batch: BigOne

==========================================
Phase 3: Batch Completion
==========================================
BATCH COMPLETED SUCCESSFULLY:
- Return Status: 0
- Success Message: SUCCESSFUL: Batch "BigOne" (Id: 42) finished.

==========================================
EXECUTION SUMMARY
==========================================
Batch: BigOne (ID: 42)
Mode: T (Trace Mode)
Final Status: SUCCESS
Status Number: 0
Execution Time: 14:30:25
==========================================

Error Handling​

Batch Not Found​

If the specified batch doesn't exist:

ERROR: Batch "InvalidBatch" not found in MetadataZone.LoadBatch
Available batches:
[List of available batches displayed]
Script execution aborted.

Execution Failure​

If batch execution fails:

BATCH FAILED:
- Return Status: -1
- Error Message: ERROR: In batch "BatchName" (Id: 42) at least error -1 occurred.

Best Practices​

1. Always Test First​

Run in Trace mode before Execute mode:

SET @BatchExec = 'T'  -- Test first
-- Review output, then change to 'E' for execution

2. Use Appropriate Error Handling​

For critical batches, consider enabling exit on error:

SET @ExitOnError = 1  -- Stop on first error

3. Monitor VapourTrail IDs​

Keep track of VapourTrail IDs for restart capabilities:

-- Note the VapourTrail ID from successful runs for potential restarts

4. Configure Parallel Processing Carefully​

Consider your system resources when enabling parallel execution:

SET @ParallelExecution = 1   -- Enable for better performance
SET @ModuleSemaphoreOn = 1   -- Control resource usage

Troubleshooting​

Common Issues​

Issue: "Batch not found" error Solution: Check available batches with SELECT * FROM MetadataZone.LoadBatch

Issue: Permission errors during execution Solution: Ensure proper permissions on MetadataZone schema

Issue: SSIS integration failures Solution: Verify SSIS configuration in MetadataZone.ConfigExMeXFramework

Issue: Parallel execution conflicts Solution: Enable semaphore control with @ModuleSemaphoreOn = 1

Getting Help​

  • Check the ExMeX Framework documentation: https://exmex.tedamoh.com
  • Review batch configuration in MetadataZone.LoadBatch
  • Use Trace mode to debug execution flow
  • Examine VapourTrail logs for detailed execution history

This guide covers the essential aspects of ExMeX Framework batch execution. For more detailed information about specific stored procedures and advanced configuration options, refer to the complete ExMeX Framework documentation.