Creating A Simple Batch Apex Process Queue

Working with the Salesforce platform presents many challenges. One of the most common is avoiding the platforms many governor limits. As a developer, hitting governor limits on the platform is a sure fire way to turn a good day into a bad day. If you are working on a complex code base it can be a very frustrating experience.

So, you start to look for ways to get more out of the platform for that complex business process you’re working on. You may get a bit a further pushing your process into a future job or you may resort to using staging tables and multiple scheduled batch jobs polling and updating every 15 minutes.

Trying to eek out a bit more processing power can often lead to code getting more and more complicated and less and less maintainable.

I recently saw an inspiring example of a Message Bus implemented on the platform by Matt Bingham and Neil Reid. You can check it out here: http://www.ratsaas.com/eda

Often, however, you don’t have the time to re-architect your whole code base and need something to plug in quickly to solve your immediate problem. The pattern I’m going to show you takes some ideas from Matt and Neil’s presentation and builds a simple stateful process queue that allows you to break your process up into steps and give each step its own request and therefore its own set of governors.

With this pattern you can:

  • Split your operation into smaller manageable process steps.
  • Sequence and execute your process steps in a specific order.
  • Pass state between process steps.
  • Decide whether a failing process step should halt the process chain.

This is achieved with little more that a Batch Apex class and a simple Interface.

We’ll begin with taking a look at the Interface …

All the process steps we will later define will implement a common Interface that looks like this:

public interface IProcessStep
{
	void execute(Map<String, Object> stateMap);

	Boolean haltOnError();
}

You’ll see that the interface has 2 methods.

The execute() method takes a Map<String, Object> as an argument. This map is used to persist state between process steps. You’ll see how this works when we get to the implementation.

The haltOnError() method allows the process step to flag that a critical error has occurred and processing of further steps should be prevented.

The Batch Process …

public class ProcessStepsBatch
	implements Database.Batchable<Object>, Database.Stateful
{
	private List<IProcessStep> m_processSteps;		// List of Process Steps to Iterate
	private Map<String, Object> m_stateMap;			// State Map

	private Boolean m_halt;
	private List<String> m_messages;

	/**
	 * Constructor
	 *
	 * Arguments:	List<IProcessStep>		List of process steps to execute
	 *				Map<String, Object>		State Map to persist
	 */
	public ProcessStepsBatch(List<IProcessStep> processSteps, Map<String, Object> stateMap)
	{
		Messaging.reserveSingleEmailCapacity(1);
		
		m_processSteps = processSteps;
		m_stateMap = stateMap;

		m_messages = new List<String>();
		m_halt = false;
	}

	/**
	 * Start Method
	 */
	public Iterable<Object> start(Database.BatchableContext info)
	{
		return (Iterable<Object>)m_processSteps;
	}

	/**
	 * Execute Method
	 */
	public void execute(Database.BatchableContext info, Object scope)
	{
		List<Object> scopeList = (List<Object>)scope;

		// Check the batch size
		if (scopeList.size() > 1)
		{
			throw new ProcessException('Maximum batch size is 1.');
		}

		// Cast to our Interface
		IProcessStep processStep = (IProcessStep)scopeList[0];

		// if we called a halt to the process do not process the step, just report it
		if (m_halt == true)
		{
			m_messages.add( 'NOT PROCESSED: Not Processed Action: ' + processStep);
			return;
		}

		try
		{
			// Execute the Step
			processStep.execute(m_stateMap);
			
			// Report the Success
			m_messages.add( 'SUCCESS: Processed Action: ' + processStep);
		}
		catch (Exception e)
		{
			// Report the Failure
			m_messages.add( 'FAILED: Error Processing Step: ' + processStep + '; Exception: ' + e.getMessage());
			
			// Check whether to halt the process or continue
			if (processStep.haltOnError())
			{
				m_halt = true;
			}
		}
	}

	/**
	 * Finish method
	 */
	public void finish(Database.BatchableContext info)
	{
		// Send out an Email with a Report
		Messaging.SingleEmailMessage mail = new Messaging.SingleEmailMessage();
		mail.setToAddresses(new String[] { UserInfo.getUserEmail() });
		mail.setSubject('Process Steps Batch - Report');

		String body = 'Report: \n\n';
		for (String m : m_messages)
		{
			body += '    ' + m + '\n';
		}

		mail.setPlainTextBody(body);

		Messaging.sendEmail(new Messaging.SingleEmailMessage[] { mail });
	}

	public class ProcessException extends Exception {}
}

Most important is that the batch apex class implements the Database.Stateful interface as well as the Database.Batchable interface.

The constructor takes 2 arguments. The first is a list of Process Steps that implement the IProcessStep Interface and the second is the state map. Both of these are stored as member variables to ensure they are persisted across the entire batch.

The start() method returns the list of process steps (which itself is an iterator).

The execute() method first validates the scope passed to it to ensure it has a batch size of 1. You could, and should, experiment with larger batch sizes but the point of this exercise is to give maximum resource to each step. (A little modification will be required if you want to increase the batch size.) It then executes the step and handles any errors.

The finish() method simply sends an email to report its progress.

Some simple process steps …

Next we can define some process step classes that implement our IProcessStep Interface.

For this example we going to use two simplistic process steps, one that creates an Account and another that creates an Opportunity.

The create Account process step class looks like this:

public without sharing class CreateAccountStep 
	implements IProcessStep
{
	// Member variables
	private String m_accountIdStateKey;
	private String m_accountName;

	/**
	 * Constructor
	 *
	 * Arguments:	String	Key to the State Map to store the Account Id
	 *				String	Account Name to create the Account with
	 */
	public CreateAccountStep(String accountIdStateKey, String accountName)
	{
		m_accountIdStateKey = accountIdStateKey;
		m_accountName = accountName;
	}

	/**
	 * Execute method (from Interface)
	 */
	public void execute(Map<String, Object> stateMap)
	{
		// Create the account
		Account a = new Account(
			Name = m_accountName
		);

		insert a;

		// Store the Id in the State Map using the supplied key
		stateMap.put(m_accountIdStateKey, a.Id);
	}

	/**
	 * Halt on Error? (from Interface)
	 */
	public Boolean haltOnError()
	{
		return true;
	}
}

You’ll see that the constructor takes 2 arguments.

The first is a key to the State Map. It uses this key to store the Id of the Account is has created as we are going to need this in a later step when we create the Opportunity. We could equally pass any sort of object into this map but bear in mind that any state we store is going to count towards our overall heap size as we will be persisting this map in the batch apex implementation.

The second argument is the Name we want to use for the Account.

We store both these values as member variables, again this will count towards our heap so we want to give careful consideration to what we are storing.

The execute() method creates and inserts an Account and stores its Id in the state map using the specified key.

The haltOnError() method returns true as we don’t want to do anything else if it should fail.

In this example I’m arbitrarily setting it to true but you could equally define logic that determines its value.

The create Opportunity process step class looks like this:

public without sharing class CreateOpportunityStep 
	implements IProcessStep
{
	// Member variables
	private String m_accountIdStateKey;
	private String m_opportunityName;

	/**
	 * Cosntructor
	 *
	 * Arguments:	String	Key to the State Map holding the Account Id
	 *				String	Opportuninty Name to create the Opportunity with
	 */
	public CreateOpportunityStep(String accountIdStateKey, String opportunitytName)
	{
		m_accountIdStateKey = accountIdStateKey;
		m_opportunityName = opportunitytName;
	}

	/**
	 * Execute method (from Interface)
	 */
	public void execute(Map<String, Object> stateMap)
	{
		// Get the Account Id from the stateMap
		Id accountId = (Id)stateMap.get(m_accountIdStateKey);

		if (accountId == null)
		{
			throw new ProcessStepException('No Account Id passed to process step.');
		}

		// Create the opportunity
		Opportunity o = new Opportunity(
			AccountId = accountId,
			Name = m_opportunityName,
			CloseDate = System.today(),
			StageName = 'Closed Won'
		);

		insert o;
	}

	/**
	 * Halt on Error? (from Interface)
	 */
	public Boolean haltOnError()
	{
		return false;
	}

	public class ProcessStepException extends Exception {}
}

Similarly to the CreateAccountStep class the constructor takes 2 arguments.

Once again the first is a key to the state map.

The second is the name of the Opportunity to create.

In similar fashion it stores these arguments as member variables.

The execute() method uses the key to resolve the Id of the Account before creating the Opportunity and inserting it. If we wanted to do something with the Opportunity in a future step we would need to store its Id in the state map and add another key to the constructor arguments.

The haltOnError() method returns false to indicate processing may continue in the event of a failure.

Putting it all together …

Now we have a couple of process steps we can wrap them up in a process and send it to the batch queue. For this example I’m going to put it all in a standalone class inside a small static method but you could integrate this into a service class with some logic behind it:

public without sharing class MyProcessSteps
{
	/**
	 * Static Sumbit Method to get things going
	 */
	public static void submit()
	{
		// Create a state map
		Map<String, Object> stateMap = new Map<String, Object>();

		// Create a list of Process Steps
		List<IProcessStep> processSteps = new List<IProcessStep>();

		// Add the process steps to execute
		processSteps.add(new CreateAccountStep('Acc1', 'Yorkshire Coffee'));
		processSteps.add(new CreateOpportunityStep('Acc1', 'The Great Teabag Swindle'));
		processSteps.add(new CreateAccountStep('Acc2', 'Yorkshire Toffee'));
		processSteps.add(new CreateOpportunityStep('Acc2', 'Curry and Chocolate Promotion'));

		// Instantiate the Batch Job
		ProcessStepsBatch batch = new ProcessStepsBatch(processSteps, stateMap);
		
		// Execute the Batch with a size of one
		Database.executeBatch(batch, 1);
	}
}

You’ll see we first instantiate the state map and list of IProcesStep‘s then we add the required process steps to the list.

In the above example we are creating 2 Accounts and 2 Opportunities. We pass a common key to the state map between the process steps that need to share information. So step 1 and 2 both use the key ‘Acc1’ while steps 2 and three share the key ‘Acc2’.

Next we instatiate the batch apex class and pass in the process steps and state map.

Finally we execute the batch and and let it process each step in turn, et voilà!

A couple of things to think about before we go …

  • Consider carefully what member variables you hold in your process step classes. It will all count towards your heap. Can you pass Ids instead of objects and requery them inside the execute method?
  • You’re working outside of a single request. If a process step fails down the line all the process steps before will be commited. What are the implications of this? How will you handle errors?

Thanks also to Phil Hawthorn for helping to inspire this approach. Hope you find it useful.

Advertisements

Author, brainstormer, coder, dad, explorer, four chord trickster, gig goer, home worker, inquisitor, joker, knowledge seeker, likes: marmite, note scribbler, opinionator, poet, quite likes converse, roller skater, six music listener, tea drinker, urban dweller, vinyl spinner, word wrangler, x-factor hater, Yorkshireman (honorary), zombie slayer (lie).

Tagged with: , , , ,
Posted in apex, code, force.com, salesforce
6 comments on “Creating A Simple Batch Apex Process Queue
  1. Andrew Fawcett says:

    Nice use of Iterator Tony! Adds much more flexibility to Batch Apex processing. Using ‘global’ is optional thank goodness these days, not sure if you want to call that out and/or adjust your sample above. Great work! 🙂

  2. […] Post Credits Finally, I’d like to give a nod to an past work associate of mine, Tony Scott, who has taken this type of approach down a similar path, but added process control semantics around it. Check out his blog here! […]

  3. Great post Tony! Love the creative use of an iterator to walk the flow steps.

  4. Matt Addy says:

    Thanks for sharing Tony! I’m curious why you needed to use a custom iterator in this example. Couldn’t you have just returned the List (the m_processSteps reference) from the start() method of your batch class?

    • Tony Scott says:

      Hi Matt, The post is a couple of years old now. Not sure where my brain was going with custom iterator, but you’re right – no need for it. It seems obvious now re-reading it. Anyway, thanks for the observation. Have updated the article and ditched the custom iterator. Cheers, Tony

Comments are closed.

About Me
Product Services Developer at:
FinancialForce.com
All views expressed here are my own. More about me and contact details here.

Enter your email address to follow this blog and receive notifications of new posts by email.

Copyright (there isn’t any, feel free to reuse!)

CC0
To the extent possible under law, Tony Scott has waived all copyright and related or neighboring rights to MeltedWires.com Examples and Code Samples. This work is published from: United Kingdom.

%d bloggers like this: