FAQ: Large Data Sets

Why can't I retrieve all security at once? And what can I do about it?

One of the challenges with Salesforce security is the sheer volume of data.  Security is the intersection of Profiles or Permission Sets and the entities (Objects, Fields, etc) that they are managing.  Adding one more Profile to a retrieve operation means retrieving security settings for each entity selected.  If an Object has 50 fields, that means 51 additional security settings must be retrieved per security file added.  Adding security files and entities can quickly explode the volume of data to retrieve.  For small environments, you may be able to retrieve all your security, but most real world implementations quickly become too large to retrieve everything.  So, what can be done?  

1. Adjust retrieve options:

If you are getting "Request timed out" messages during retrieve operations, the first thing to check are the retrieve options.  When a retrieve operation is performed, Security Zen will periodically poll Salesforce to determine the status of the retrieve operation.

The "Max Number of Polls" is the maximum number of times Security Zen will poll Salesforce before giving up.  For large data sets where you expect it to take a long time, you can increase this number.

The "Poll Time (seconds)" is the time to wait between Polls.  A larger Poll Time means Security Zen will update the  status of the retrieve operation less frequently, but also increases the total amount of time that is available to perform the retrieve operation.

For example, Max Number of Polls = 50 and Poll Time = 2 means that status of the retrieve operation is reported once every 2 seconds and will give up after 100 seconds (i.e. 50x2), which is less than 2 minutes.

Max Number of Polls = 500 and Poll Time = 10 means that status of the retrieve operation is reported once every 10 seconds and will give up after 5000 seconds (500x10), which is about 83 minutes.

2. Adjust Display Options:

If you appear to be getting stuck on the "Building Grid" step of the retrieval process, the dynamic column width feature may be the cause.  Up to Version 1.8 of Security Zen, Column Width was determined based on the cell contents.  Starting with Version 1.9, there is an Option for specifying Column Width.  Setting Column Width to 0 turns on the automatic Column Width sizing based on the cell contents, however, this is not recommended for large grids.

If you appear to be getting stuck on the "Sorting Grid" step of the retrieval process, you can turn the feature off during the retrieval process.  Go to the Option "Sort Rows Upon Retrieve" and uncheck this feature prior to retrieving data.

3. Retrieve security incrementally:

If there is already data in the Security Zen workspace when you do a retrieve operation, you will be prompted to "Clear workspace?".  If you answer No, then the workspace will not be cleared and the retrieved data will be added to the existing workspace.  This allows you to retrieve data incrementally and build up the workspace.  

4. Start with small retrieves and slowly increase it:

If you are still having problems, try starting small.  Select one Profile and a few fields (not the entire object) to get an idea of how long it takes to perform a small retrieve.  Try adding one more Profile and/or a couple more fields to see how this changes the retrieval times.

Remember, selecting an object includes all the fields and record types of that object, so be cautious in adding too many objects.  Try adjusting the retrieve options as you increase the number of security files and entities to understand the impact of the retrieve options on performance.  

What's the best way to deploy a large data set?

Deploying large data sets presents similar challenges to retrieving large data sets.  In order to best meet the needs of different use cases, there are two methods available for deploying security to Salesforce:

  • API performs the deployment using API calls to Salesforce.  This method is faster for small security changes.
  • ANT creates a zip file that is deployed to Salesforce as an independent process.  This method is faster for large security changes.  If using this method, it is recommended that you monitor the deployment on Salesforce in Setup->Deploy Status

If you are getting timeout issues using the API method and you have a lot of security files, you should reduce the batch size.  This will create more batches each of a smaller size, but is less likely to time out.  If you are getting time out issues with a batch size of one, or have a very large data set, then it is recommended you switch to the ANT deployment method.

The deployment method is selected in the Options menu.  This is further documented in the Options Documentation in the section titled "Deploy Options".