ServiceNow auditing (history) feature for a table and troubleshoot related issues

ServiceNow auditing (history) feature:

ServiceNow auditing (history) feature for a table can be enabled using below steps:

1.       Navigate to System Definition > Dictionary.

2.      Select the table to audit. For example, alm_asset.

3.      Select the dictionary entry for the table. The table name always has an empty column name and the type of collection.

4.      Check the Audit check box.

5.      Click Update.

It should be noted that, by default, the system does not audit records from system tables. To audit a system table, add it to the list of tables in the glide.ui.audit_deleted_tables property list.

Once these steps are performed, ideally ServiceNow should start auditing field value changes in corresponding table. One can view the historical data for that table using below options.         

• Users can view History detail in Calendar format for that table by right click on the form and selecting History à Calendar
• Users can view Record History in List format for that table by right click on the form and selecting History àList
• Audit History section can be added at the bottom of form (for that table) by Configure à Related Lists and then moving Audit History to Selected list.

Note: In Fuji version, Configure option is selected on right click on the form. Prior to Fuji version, we use Personalize option

In some cases, audit feature doesn’t work as expected. At high level, it appears as if audit feature is not working. It may also appear that ServiceNow audit feature is working, perhaps some functionality is either restricting the audited data from saving and/or wiping out audit history data.

In midst of all these doubts, you need to be very clear that whether audit feature is not working for any specific table or all tables. If it is not working for any specific table, there can be multiple root cause for that behavior. It might take some reasonable amount of time and effort to identify the exact reason behind the auditing failure for that table.

Before deep dive into the issue, first please note below information from ServiceNow Wiki documentation:

·         ServiceNow tracks incident, change, and problem history in the sys_audit table.

·         Enabling auditing tracks the creation, update, and deletion of audited records.

·         In addition, auditing activates the History context menu entry.

·         It is not possible to audit individual fields without auditing the entire table.

·         It is possible, however, to hide certain fields from the audit using a dictionary attribute.

·         Auditing certain system tables that receive a large amount of traffic, such as Workflow Contexts [wf_context], can impact performance and is not recommended.

Auditing feature tracks only following record changes.

·         The Unique Record Identifier (sys_id) of the record that changed

·         The field that changed

·         The new field value

·         The old field value

·         The number of times this record and field have been updated

·         The date and time when the change occurred

·         The user who made the change

·         The reason for the change (if any reason is associated with the change)

·         The record's internal checkpoint ID (if the record has multiple versions)

Some updates are not audited despite enabling auditing on a table. Auditing excludes the following information:

·         Any updates made by an upgrade.

·         Any updates made through import sets.

·         Any records in parent or child tables.

·         Any field with the no_audit dictionary attribute.

·         Any system tables not listed in the glide.ui.audit_deleted_tables property list.

·         Any field that begins with the sys_ prefix (system fields) except the sys_class_name and sys_domain_id columns.

·         Any time an inactivity monitor touches a record (this prevents you seeing possibly hundreds of updates listed against an incident, with the noise drowning out the useful data).

If your case is not addressed by above details from ServiceNow Wiki documentation, you can explore further. To save your time, you can initially check if below statements are used in ServiceNow scripts related to the target table. One or more custom scripts using below statement might be preventing the sys_audit table from being updated for that table.

current.autoSysFields(false);

current.setWorkflow(false);

ServiceNow Wiki documentation explains,

autoSysFields:

public void autoSysFields(boolean e)

Enables or disables the update to the fields sys_updated_by, sys_updated_on, sys_mod_count, sys_created_by, and sys_created_on. This is often used for manually updating field values on a record while leaving historical information unchanged.

Note: Use caution if you use this method. When you use this method the sys_mod_count field will not be incremented, and other sys_ fields will not be updated. This can break functionality including, but not limited to, the Activity Formatter, History Sets, and Metrics.

Parameters:

e - Boolean variable that if false disables updates to sys_updated_by, sys_updated_on, sys_mod_count, sys_created_by, and sys_created_on.

setWorkflow:

public void setWorkflow(boolean e)

Enables or disables the running of business rules that might normally be triggered by subsequent actions. If the e parameter is set to false, an insert/update will not be audited. Auditing only happens when the parameter is set to true for a GlideRecord operation.

Parameters:

e - Boolean variable that if true (default) enables business rules, and if false to disables them.

The idea here is to examine your ServiceNow instance's Script Includes, UI Actions, and business rules for autoSysFields(false) or setWorkflow(false) calls related to that table. You can set active = false one by one and retest, or set them all to active = false and retest. That should help you to isolate the cause of the issue.

For example, to identify the business rules at fault, you can enable debugging trace of business rule. You can follow below steps:

1.       Navigate to Debug and click

·         Debug Business Rule (Details)

·         Debug Log

·         Debug SQL (Detailed).

2.      Update a record (from that table) by clicking 'Save' (Right-mouse-click header and click Save, if necessary), so that you stay on the same page.

3.      Examine the trace the bottom of the form. Note that

=== means a business rule was skipped because the condition did not evaluate to true

==> entering the business rule

<== exiting the business rule

In case you have already identified those script lines where autoSysFields(false) or setWorkflow(false)  are called, examine those lines. Whether those statements have been used purposely to achieve any desired business functionality? There can be also a possibility of developers overlook while using these statements. No matter what is the reason, autoSysFields(false) or setWorkflow(false) statements need to be used carefully and avoided, if not needed. Else it will override the audit feature on the related table. In addition to this, it might also impact metrics definition and instance data for related table and fields.

You need to identify and fix the custom scripts that are turning off the sys_audit updates on that table. For testing purpose, you can comment those lines and see if audit feature works on the required table by doing some record updates. In case, it still not work and you are sure that audit feature need to work on that table in all scenarios, you can follow a workaround as an alternate solution. This will be a brute-force sort of fix. You can create a business rule on that table for “update” and “insert” action with “onbefore” event. Include following statements in that business rule and keep its “order” to maximum in relation to other business rule for that table.

current.autoSysFields(true);

current.setWorkflow(true);

This will ensure to record the sys_audit table entries. If the problem still persists, it will be advisable to involve and consult ServiceNow team. 

QA testing of ServiceNow Data Sources, Import Sets and Transform Maps

ServiceNow Data Sources, Import Sets & Transform Maps

Technically Data Sources in ServiceNow are used to create intermediate import set. Import set table data is mapped to production table. However import set data can be processed prior mapping to production table. Data for data sources can come in three ways viz. 1) Recognized file format 2) Data accessed via JDBC and 3) Data from LDAP server.

As import sets table acts as staging area for records imported from data sources, import set table data can be manipulated for testing of intended data before it gets updated into production table via respective transform map field mappings. As a ServiceNow expert, these points look familiar and might be simple. However it has been observed many times that QA testers find these simple feature difficult while testing newly created (or modified) data sources in ServiceNow.

Mostly QA testers prefer to take the help of ServiceNow developers or admin to run the data source and its respective transform map and then validate the results on destination table using reports, list view or in layout view. They find testing the data source part complex might be because:

·        Import Sets can have thousands of rows and it might take long time to execute

·        QA testers need to run the data source multiple times to validate different test cases and scenarios. To complete all scenarios, QA testers might put lot of hours. However developers have more knowledge and know tricks to run and test intended rows data quickly.

·        QA testers might need admin or security_admin roles/permissions to perform some of steps. Instead they might prefer developers or admins to check those steps

QA testers can find below steps useful in testing ServiceNow data sources, provided they have security_admin elevated rights

1.      Note down details in form of reports or screenshots of required table or form involved in data source testing

2.     Choose the data source and select “Load All Records” (in Related Links) to run the data source and load all data rows

3.     If the data source involves large amount of data, it takes some more time to process. Once the State is Complete and Completion Code is Success, go to Import sets. Note down the Import Set number.

4.     Identify the desired rows from the Import Set to test further.

5.     Once rows from Import Set have been identified, delete other Import Set rows with below script using “Scripts – Background”.  Note:Prior to this, security_admin elevated privilege need to be activated

For e.g. Data Source name is "Test Import Assets"

Import Set table name is "u_test_import_assets"

Import Set number is "ISET1234567"

var gr = new GlideRecord('u_test_import_assets'); //Import Set table name

gr.addQuery('sys_import_set.number', 'ISET0017093'); //Import Set number

gr.addQuery('sys_import_state', 'pending');

//Delete Import Set table rows except row numbers 944,5612,8881

gr.addQuery('sys_import_row','NOT IN' , '944,5612,8881');

            gr.deleteMultiple();

6.     With above script, unwanted rows can be deleted from Import Set table and then respective transform map for that Import Set can be run to complete the data update in production table.

7.     Import set rows data can be changed or manipulated to perform positive or negative testing as per required scenario.

8.    With this approach testing can be done with less number of rows, and QA testers can run multiple tests in less time. Otherwise the source data need to be less to execute and complete the data source run quickly.

ServiceNow System Properties

ServiceNow Properties

To customize and update ServiceNow System Properties search by typing ‘sys_properties.list’ in filter text box. Then Search by the respective property name and select the record to customize and update the details of that property.

glide.db.soft_join_cap

Smaller values cause the system to issue a larger number of less complex queries. Larger values reduce the number of queries at the cost of additional complexity per query. In the absence of known database issues stemming from large join counts, this property should remain unchanged.

This property is used to limit database join complexity and size in a particular query. Though ServiceNow uses relational database model, however it is better to use fewer joins in order to prevent any database performance issue. By default, this parameter is set to 10 which should remain unchanged so that database queries work efficiently without any performance issue.

glide.dashboard.refresh_intervals

This property can be used to set dashboard refresh intervals in seconds (for e.g. 900,1800,3600). Accordingly refresh intervals will be available for dashboard for users.

glide.escalation.notes

This property can be used to set the Boolean flag (True/False) in order to turn on/off the feature of logging the information/comment in task Work Notes field.

glide.escalation.sticky

This property can be used to set the Boolean flag (True/False) to turn on/off the feature of re-computation of escalations on every Save/Update action for any ticket.

glide.escalation.verbose

This property can be used to set the Boolean flag (True/False) to turn on/off the feature of logging verbose escalation messages.

glide.history.initial_updates_when_truncated

Maximum numerical limit value can be set in relation to historical updates availability for any record. These number of updates shows up from the point when record was actually created initially until some records are omitted, when the maximum field entry value exceed. (For e.g. the default value is set to 10).

glide.history.max_entries

Activity formatter will use the maximum number set for this property and display up to those number of field entries in record history. Please note that most recent field entries are shown at the starting entries list in Activity formatter. The maximum number can be set as 250. By the way, the default maximum value is 250 for this property.

glide.history.role

History access can be provided to specific users by setting list of roles (comma-separated) in this property. Users having those roles will be able to access the history.

glide.itil.assign.number.on.insert

This Boolean property can be set to turn on/off the feature to generate task number only upon insert action. This way it helps to prevent the generation of unused task numbers.

glide.product.apple_icon

This property can be used to set the icon used for iPhone home page bookmarks. It can be overridden in the company record with a custom user image.

glide.product.description

This property can be used to set the window title at the rightmost portion of the page header and browser title.

glide.product.description.style

This property can be used to set the style for product description next to banner. (For e.g. padding-bottom: 0px; padding-top: 4px;)

glide.product.help_icon

This property can be used to set the icon for the 'Help' link in the welcome banner. Please note that it must be uploaded first before the property setting.

glide.product.help_url

This property value act as global setting for the URL the help icon directs to. Value for this property must be empty for context-sensitive help to work. Users simply click the help icon to open the help page provided by ServiceNow. For any page that does not have context-sensitive help defined, a search for relevant content on the ServiceNow Wiki opens (starting with the Eureka release). In versions prior to Eureka, the main ServiceNow Wiki page opens.

glide.product.icon

This property can be set for the icon image to be displayed in bookmarks and browser address bar.

glide.product.image

This property can be set for banner image displayed at the top of the page. This property value can be overridden by banner text and banner image defined for the company that a user is assigned to.

glide.product.name

This property can be set for window title part appearing at the leftmost portion of the page header and browser title

glide.product.name.style

This property can be set CSS properties to display product name next to banner. Often, this style is set to “display: none” so that the product name is not displayed, but still used as part of the Window Title.

glide.short.labels

This property can be used to specify the type of labels that are displayed for all reference fields on any form. This global property can be overridden for any field by setting the short_label=true or long_label=true attribute for the field in the Dictionary.

Set this property to use short labels for all fields. For example, if a form contains the caller's email address, use the "Email" label rather than the full label of "Caller Email".

glide.sys.date_format

This property can be used to set the date format used for date representation.

(For e.g. yyyy-MM-dd)

Note that MM is months, where mm indicates minutes. The format string consists of the following abbreviations:

Field	Full Form	Short Form
Year	yyyy (4 digits)	yy (2 digits), y (2 or 4 digits)
Month	MMM (name or abbr.)	MM (2 digits), M (1 or 2 digits)
Day of Month	dd (2 digits)	d (1 or 2 digits)

glide.sys.time_format

This property can be used to set the time format for time representation.

(For e.g. HH:mm:ss).

The format string consists of the following abbreviations:

Field	Full Form	Short Form
Hour (1-12)	hh (2 digits)	h (1 or 2 digits)
Hour (0-23)	HH (2 digits)	H (1 or 2 digits)
Minute	mm (2 digits)	m (1 or 2 digits)
Second	ss (2 digits)	s (1 or 2 digits)
AM/PM	a

glide.sys.default.tz

This property can be used to set the System timezone, used as default for calendars and users. Mostly Olson/zoneinfo timezone values are accepted.

glide.sys.logout_prompt

This property can be set to prompt user for logout request confirmation.

glide.sys.servlet_path

This property can be used to set the prefix path that is used while accessing user’s ServiceNow instance.

glide.sys.sparse.update

This property can be set to allow physical update of the database only when fields have changed in value.

glide.sys.upgrade_script

This property can be set for upgrade script used in particular instance.

glide.ui.autoclose.time

This property can be used to set the number of days (integer) after which Resolved incidents are automatically closed.

If set as Zero (0), it will disable this feature.

glide.ui.per_page

This property can be used to set 'Items per page' drop-down options (comma separated, no spaces). (For e.g. 10,15,20,50,100)

glide.ui.record.level.history

This property can be used to turn on/off the feature to maintain a history of records visited by each user.

glide.xmlhttp.ac_wait_time

This property can be used to set the default wait time between keystrokes for a dynamic drop-down choice list (milliseconds). (For e.g. 250)

glide.xmlhttp.max_choices

This property can be set for maximum number of items to show for a dynamic drop-down choice list.

glide.use_tiny_urls

This Boolean property allow to turn on/off the feature to use tiny URLs when a redirect URL becomes too large. This ensures that URLs that are too large for Internet Explorer (greater than 2083) are not used. Instead, they are converted to a tiny URL as a workaround for the Internet Explorer issue.

glide.tiny_url_min_length

This property is set for minimum length of a redirect URL that is turned into a tiny URL. The default value is to 1024.

glide.ui.activity.email.use_display

This Boolean property, if set to “true”, will turn on the ServiceNow feature to show user IDs instead of email addresses in Activity header for emails. Basically the instance searches for a user record based on matching email address in User table. If any matching user record is found, it displays user IDs in email header. If no matching user record is found, the email address is displayed. Default value is “false”.

glide.ui.activity.email_roles

This property allows to customize the list of roles that can view mail in the Activity formatter when including "Sent/Received Emails". The list of roles need to be specified in the format of comma-separated string values. Default value is “itil”.

glide.ui.activity.image.assigned_to

This property, if set with appropriate image filename path, enables to use "Assigned To" image in Activity formatter. For e.g. “images/icons/user.gifx”

glide.ui.activity.image.assignment_group

This property, if set with appropriate image filename path, enables to use Assignment group image in Activity formatter.

glide.ui.activity.image.comments

This property, if set with appropriate image filename path, enables to use Additional comments image used in Activity formatter.

glide.ui.activity.image.work_notes

This property, if set with appropriate image filename path, enables to use Work notes image used in Activity formatter.

glide.ui.activity.style.comments

This property allows to change the style of additional comments and provide customized background color in Activity formatter. For e.g. “background-color: WhiteSmoke”

glide.ui.activity.style.work_notes

This property allows to change the style of Work Notes in Activity formatter. Other fields can also be customized by creating new UI Properties.

glide.ui.advanced

This Boolean property, if set to “true”, enable the advanced UI and allows to show "Save", "Insert" and "Insert and Stay" buttons on forms.

glide.ui.app_menu_context

This Boolean property, if set to “true”, enable the right click to edit context menu on applications in the navigator.

glide.ui.attachment_drag_and_drop

This Boolean property, if set to “true”, allow attachment drag and drop in supported HTML5 browsers.

glide.ui.audit_deleted_tables

This property can be set with list of system tables (beginning with "sys_", comma separated) for which the audit history tracks deletions. Please note that by default, system tables do not have the delete audited. So for any system table to audited for deletions need to be included in the comma separated string value for this property. Default value for this property is “sys_user,sys_user_group,sys_user_role,sys_user_has_role,sys_user_grmember,sys_group_has_role,sys_security_acl_role”

glide.ui.autoclose.time

This property enables a feature wherein integer value set decides the number of days for automatic closure of Resolved incidents. If Zero (0) is set, it disables this feature.

glide.ui.buttons_bottom

This Boolean property, if set to “true”, enables a feature wherein buttons like Submit, Update, etc. are shown at end of form.

glide.ui.buttons_top

This Boolean property, if set to “true”, enables a feature wherein buttons like Submit, Update, etc. are shown in form header.

glide.ui.can_search

This property help to manage the use of Global Text Search feature in ServiceNow by allowing the access to specific roles. The list of roles is set as string in comma-separated format in this property. The default value for this property is “itil,text_search_admin,admin”

glide.ui.choicelist.defaultwidth

This property can be customized to set the width (in terms of pixels) for all choice list on the instance. Default value for this property is set to 160 that means by default all choice lists use a width of 160 pixels.

glide.ui.choices.show_missing

This property enables the appearance of missing choice list entries. It is advisable to set this Boolean property to “true” to avoid any issue related to dynamically created choice list entries.

glide.ui.clickthrough.popup

This property, if set to “true”, turn on the feature of opening a new window when the related object icon is clicked on any form in a ServiceNow instance. An additional icon is shown beside the field for referenced object. To avoid the creation of new window, keep this property as “false”.

glide.ui.clickthrough.replace

This property, if set to “true”, allow replacing the current screen for a related object. It works along with glide.ui.clickthrough.popup property. If both property are “true”, two different icons are shown for referenced object. Icon for glide.ui.clickthrough.popup property (when clicked) opens new window for referenced details. Icon for glide.ui.clickthrough.replace property (when clicked) loads referenced object details on same screen. If glide.ui.clickthrough.replace property is “false”, it can’t replace referenced details on same screen as its icon is not displayed.

glide.ui.client.integer.check

This Boolean property, if set to “true”, enable client side validation of integer fields.

glide.ui.client.mandatory.check

This Boolean property, if set to “true”, turn on a feature wherein client validates mandatory field population.

glide.ui.client.numeric.check

This Boolean property, if set to “true”, enable client side validation of numeric fields.

glide.ui.dirty_form_support

This Boolean property, if set to “true”, show a popup to user while navigating away from a modified form. The pop up shows two options. One to proceed further without saving the modifications. Second option to cancel the navigation and stay on same form.

glide.ui.email_client.autocomplete.count

This property help to control the email client autocomplete search result behaviors. The integer value set decides the maximum number of autocomplete matches to return to the email client. If glide.ui.email_client.autocomplete.group property is turned on, in that case, email client autocomplete search result feature applies to users and groups both. However the count applies separately for users and groups.

glide.ui.email_client.autocomplete.group

This Boolean property, if set to “true”, allows to include groups in email client autocomplete results. The count set for glide.ui.email_client.autocomplete.count property applies separately for users and groups.

glide.ui.email_client.from

This property allows to override the email 'From:' address in the email client. Normally “From” field is not displayed in the email client and it uses default email property values of Outgoing email display name (glide.email.username) and User email (glide.email.user). This property, if set to “true”, activate the feature wherein an editable “From” field is displayed in the email client.

glide.ui.email_client.reply_to

This property allows to override the email 'Reply to:' address in the email client. This property , if set to “true”, activate the feature wherein 'Reply to:' address line is displayed in the email client.

glide.ui.enforce.mandatory.on.update

This property, if set to “true”, validates mandatory field value and produce a 'mandatory' error during an update when a mandatory field begins with a null value.

glide.ui.filters

This Boolean property enable User Interface filters on ServiceNow instance.

glide.ui.focus_first_element

Setting this property to “false” disable the form focus on form load. If “true”, on form load, focus is on first writable element on the form.

glide.ui.forgetme

If this property set to “true”, it remove "Remember me" checkbox from login page.

glide.ui.form_multiple_splits

This property allow multiple splits within a single form section. If this property is set to true, three options "|- begin_split -|","|- split -|" and "|- end_split -|" appear in the Available column of the configure form slushbucket. Each of these split options creates a form group. Such form group contain two columns. It enables to create elements that span the form at the top of the form and also multiple two-column split element groups within the form.