RADE

Overview

As programs grow and become more and more complex, ensuring code quality becomes a more and more challenging task. To ensure a good software quality, one might consider a multiple level approach: formal analysis of the source code, automated tests, and manual tests.

ITC is a product dedicated to automated tests. Basically, it allows an interactive scenario to be recorded and replayed. The scenario needs only to be recorded once, and can be replayed as many times as necessary. This provides the developer with a way to make impact analysis or to check for regressions. The tester saves a tedious repetition of manual tests.

The V5 record engine works at "V5 device level": what are recorded are mouse clicks, mouse moves and keyboard events that are translated into V5 system events. Contrary to other commercial automatic test tools, it takes advantage of its full integration into the V5 platform to provide a good stability:

• Replay is not sensitive to the operating system. For instance, record can be done on Windows and replay can be done on both Windows and Unix
• Replay is not sensitive to screen resolution
• Replay is not sensitive to window position, dialog box layout, toolbar position, and toolbar layout
• Replay is not sensitive to the number of items or the order of items in combo-boxes or list-boxes.

The purpose of ITC is to detect changes between code levels. This happens when a scenario cannot be replayed anymore. When such a thing happens, it is left to the developer's responsibility to analyze whether the failure is a regression symptom or not. In some cases, the failure is normal because the software user interface or its internals have changed. In other cases, the failure is a true regression symptom. A discussion on how to make both stable and relevant automated tests can be found later in this document.

In particular, ITC does not provide any mechanism to ensure stability between a V5 code level and another. No stability is insured between V5 releases, service packs or hot fixes.

Other restrictions are:

• ITC is built on a technology that applies to Wintop based products only (no Webtop support)
• Only file based scenario are supported. No VPM interoperability is supported
• Scenarios must be captured and replayed on V5 using the English language, installed on a machine that uses the US-English locale.

[Top]

Basic Tasks

Recording and Replaying

ITC product is accessed through mkodt tool. "-C" option is used for capture. No specific option is necessary for replay.

1. Create a shell file containing at least the "SetOdtParam TYPE=RECORD" command and the name of the V5 executable to launch. In case of CATIA V5 it will be:

  SetOdtParam TYPE=RECORD
  CNEXT

Save the file under a given name in the TestCases directory, for instance TestCases/MyRecord.sh.

In a licensed environment you need to activate the license for CNEXT. Create a FunctionTest/InputData/MyRecord.rec directory, and drop a valid licensing.CATSettings files that will activate the necessary license for capturing and replaying the scenario.

2. Launch the command

     mkodt -s MyRecord -C

   This starts a V5 interactive session.

3. Perform your scenario in the V5 session and end it by exiting the application.

   The returned error code should be 0, meaning the record has successfully been captured.

   Notice that the FunctionTests/InputData/MyRecord.rec directory has been created. This directory contains three files: capture.rec, capture.env, and capture.ver.

   • capture.rec contains the record data. This is a binary file.
   • capture.env contains the list of authorized paths for the record. This is a text file.
   • capture.ver specifies the level used for the record. This is a text file and is used for record versioning.

   More explanations of the content of those files will be given in the record engine internal description section.

4. Replay the test by launching the command:

   mkodt -s MyRecord

   Notice that the test is replayed on screen. The return code should be 0 if the test succeed.

[Top]

Using Settings

When recording a session, all V5 defaults settings values are used. If your scenario needs to be recorded and replayed with non default setting values, simply put the necessary *.CATSettings files in the .rec directory.

Those setting files are the same that are created and used when launching a normal V5 session.

In a licensed environment, a valid licensing.CATSettings file is needed to activate the necessary licenses for the record and replay scenario.

[Top]

Using P2 UI Level

By default, in capture mode, V5 sessions are launched using the P1 UI level. This is a behavior difference between a normal V5 session and a record capture session. To obtain the P2 UI level you need to put a copy of the FrameGeneral.CATSetting file into the *.rec directory.

It is recommended to use a clean FrameGeneral.CATSettings file. To obtain it, remove your FrameGeneral.CATSettings file in your setting directory, launch a normal V5 session and exit immediately.

[Top]

Opening Documents

Access the necessary files using the File->Open command. Authorized paths for document files are:

• "FW.tst/FunctionTests/InputData/"
• "FW.tst/FunctionTests/Output/"
• $ADL_ODT_TMP

Where:

• FW.tst is the test framework
• $ADL_ODT_TMP is a temporary directory that exists only while the scenario is capture or replayed. You can choose it to open or save documents in the file selection box.

The record engine automatically makes file name and path name conversions to ensure stability when the replay environment differs from the record environment (Unix vs. Windows).

However, to ensure stability when replaying Windows recorded scenario on Unix, make sure that file path entered in the "File Open" or directory chooser dialog box at record time have exactly the same case as the actual directory structure on the machine. Otherwise, the scenario might capture and replay correctly on Windows but fail on Unix.

[Top]

Saving Documents

Saving files can be used to test the save functionality itself or to obtain documents that can be compared to a reference later in the automated test. ITC does not provide such tools for document comparison.

Save the file using the File->Save command. Authorized path for files are:

• $ADL_ODT_TMP
• "FW.tst/FunctionTests/Output/"

The $ADL_ODT_TMP directory is deleted at the end of mkodt whereas the content of "Output" is kept. To avoid disk space saturation when replaying a large number of ODTs, use $ADL_ODT_TMP to save documents.

[Top]

Using the Cache System

Cache system location is stored in CATIAV5Cache.CATSettings file. Enter ${ADL_ODT_TMP} in Cache Management directory to make sure that cgr files are generated to a valid directory. Also, this directory is automatically cleaned, avoiding risk of disk saturation.

[Top]

Capture Recommendations, Limitations and Tips

• Play your scenario once before capturing it to check that it works properly. Do not forget that a record ending with an abend is not captured.
• Do not move your mouse too often since every movement is recorded. The less interactions, the better.
• Do not use the wheel or mouse scroll button to scroll the specification tree, otherwise you will not be able to replay the record. The wheel interaction is not supported by the record/replay engine.
• Do not resize the MDI Client Window (document window). On the Unix platform it is not possible to do it at capture time.
• Do not use the Window menu. Window size management controls ("maximize" and "minimize" buttons, for instance) at the top-right corner of the window are not available.
• Do not use the MRU in the File or Start menu to load documents
• Bear in mind that any specification tree alteration (such as the insertion of a node) may prevent the record to work properly.
• Check that a creation command works correctly by recording a selection and a deletion through contextual menu of the created object afterwards
• Interactions with the compass can be recorded on condition that a local transformation (such as a translation) of the 3D viewer is done before doing the first interaction.
• Some elements cannot be recorded:
  • Preselection navigator (also known as drill selection)
  • Drag and Drop
  • Contextual menus on toolbars
  • Context-sensitive help (also known as tool tips)
• The timing of interactions is not recorded. At replay, recorded interactions are launched one after the other. If a scenario depends on a precise timing sequence of interactions, then the replay will probably fail. In other words the elapsed time between two interactions is not a stable value, and might depend on machine hardware configuration, environment, etc
• Macros cannot be recorded and replayed.

[Top]

Replay Limitations and Tips

• CATDlgNotify dialog boxes launched using the CATDlgNotify::DisplayBlocked method are never displayed at replay. The interactions made in those boxes are recorded and replayed but the box itself is never shown at replay.
• Menus and contextual menus are visible at replay. Interactions menus are recorded and replayed, but menus are not visually expanded at replay.
• Modal dialog boxes are not modal at replay. If a modal dialog box appears at replay while it was not there at capture time, it does not prevent the record engine going on replaying the remaining interactions.

[Top]

Common Return Code

Tip

The return code is the return code of the shell passed to mkodt using -s argument [1]. The return code generated by the record engine can be overloaded in this shell. Before making an analysis based on a given return code please check that it corresponds to the actual return code generated by the replay engine.

Code 1

Code 1 is used by the replay engine when the scenario cannot be replayed anymore. Most common situations are:

• "CATCommand not found": a notification went through a CATCommand at capture time, but this CATCommand was not there at replay time. Since all Dialog objects are CATCommands, this happens when any kind of element is missing in the UI: icon, menu or contextual menu item, dialog box, etc...
• "Invalid record buffer": the capture.rec file is corrupted or its end has been reached in an unexpected way. This situation might happen when an unexpected CATDlgNotify message box is launched at replay time.
• "You want to create a new CATCommand named xxx, but you already have in your path a CATCommand with the same name": two instances of CATCommand with the same name share the same parent. This situation is not supported by the replay engine. Either this is a design error, or one of the instances has not been deleted, revealing a life-cycle object problem. A common mistake leading to this error, is to use the same toolbar in two different workbenches. The solution is to rename one of the toolbar.

Code 10

This code applies to problems encountered with dialog objects. Please read the associated message to get a description of the problem. Common situations are:

• Visibility or sensitivity stability check fails

  To ensure non-regression of UI, V5 record/replay checks that visibility and sensitivity of dialog objects on which interactions are made is stable. At capture time the visibility and sensitivity status of the interacted objects is stored along with the nature of the interaction. If this status is different at replay time, then the replay fails.

  An exception to this is the contextual menu, on which only the sensitivity check is applied.

• Missing lines in a CATDlgCombo or a CATDlgSelectorList

  When selecting a line in a combo-box or a list-box, the record engine stores the content of the line as a string. This ensures stability towards the change of the order and number of items in the list. When the string is missing at replay time, the replay fails.

• Missing line in a CATDlgMultilist

  When selecting a line in a multi-column list, the record engine stores the whole line as a separate string for each column. At replay time, the selected line is restored based on the content of the first visible column of the CATDlgMultiList. If more than one line matches the item in the first visible column, then the second column is considered, and so on until a match is found.

  When a match is found, the content of the remaining column is not tested. If no match is found the replay fails.

Code 99

When the allowed time for the test has been spent, mkodt kills the process. This can happen in case of CPU loops, or when the allowed time is not enough to replay the recorded scenario.

Allowed time defaults to 5 minutes. To change this use the SetOdtParam command in your ODT shell. See mkodt documentation [1]. For example to change this value to 10 minutes use:

SetOdtParam max_time=10

Code 100

A CATSession object has not been deleted properly at the end of the process. If the process had a normal termination then this is the symptom of a real CATSession problem. Otherwise, the true cause is to be looked in the abnormal termination of the process.

Other codes

The V5 code includes many integrity checks. Those checks are performed on various domains such as memory management, geometrical modeler, UI implementation, etc. Many of them will generate only a warning or a trace when using a standard V5 session but an error in capture or replay mode. The effect of this is that a scenario that seems to behave well in a standard interactive session will fail to be recorded or fail to be replayed.

This is a powerful tool to ensure good implementation of the V5 infrastructure and non regression when code changes. When this happens an error code will be generated and an error message displayed explaining what is wrong.

[Top]

Debugging Tips

Slowing Down the Replay

Use the CATRecordReplayTimer environment variable to slow down the replay. For example, to have one interaction every 150 milliseconds:

set CATRecordReplayTimer=150      # Windows
export CATRecordReplayTimer=150   # Unix

According to the captured scenario, you can adjust the timer in order to improve the visualization and facilitate error detection.

[Top]

Displaying the Mouse Position

Set the CATRecordVisualDebug environment variable to any value to display mouse position in windows containing a CATViewer. For example:

set CATRecordVisualDebug=1      # Windows
export CATRecordVisualDebug=1   # Unix

Each recorded mouse position is displayed using a red cross. This mode does not correspond to the normal usage of V5, it should be used for debug purposes only and never set in the ODT shell.

[Top]

Advanced Tasks

Using RecordTools

RecordTools is a command line utility that allows you to display and modify the capture.rec file.

Overview

Interactive scenarios are stored in a binary file named "capture.rec". This file contains all the user interactions and necessary information to replay them. RecordTools gives access to the content of this file and provides some editing functionality. It can translate the binary file into a readable xml file format and change the content of an elementary interaction.

RecordTools can be useful to:

• Reverse-engineer the scenario of an automated test
• Modify interactions when UI changes has been made: suppress "OK" click on a warning panel that does not appear any more, change path for menu items that has been moved, etc...
• Make an impact analysis of code change.

Usage

Simply go to the V5 install directory (OS_a\code\bin) and launch RecordTools from the command line.

To display the capture.rec file in xml format, use the -Check option. Use the -h for detailed instructions and display the list of available functionalities.

capture.rec File Content

When recording automated tests, all interactions are stored sequentially in the capture.rec file. An interaction consists in an identifier, a CATCommand path, an event name and optionally a set of data.

Once displayed in xml, the content of the capture.rec file is quite self-explicit.

The following example contains a commented capture.rec file, obtained with the simple following scenario:

• Start V5
• Launch "Help | About" command
• Click "OK" on the "Help About" panel
• Exit V5.

<?xml version="1.0" encoding="UTF-8"?>
<Record Version="V5 1.0">
<Source                                                       // Source section
  File="capture.rec"                                          // name of the source capture.rec file
  Vendor="DASSAULT-SYSTEMES"
  Tool="RECORD"
  Version="1"
  Date="">
</Source>                                                     
<Interactions>                                               // Interaction section
  <Interaction
    ID="CR1"                                                 // First interaction is triggered automatically 
                                                             // when the V5 main window appears.
    Path="CATCommandSelector/CATApplication/CNEXT/"          // The parent of all CATCommands is a CommandSelector, so all
                                                             // paths begin with "CATCommandSelector/"
    Event="CATDlgResizeNotification">                        // This interaction is a resize of the main V5 window.
                                                             
    <Datas>
      <Data Num="1" Type="CHAR" Value="CATDlgDocument"/>
      <Data Num="2" Type="ULONG" Value="65546"/>
    </Datas>
  </Interaction>
  <Interaction
    ID="CR2"                                                 // Interaction number 2 is activating the "HelpAbout"
                                                             // menu item
    Path="CATCommandSelector/CATApplication/CNEXT/MenuBar/HELP/CmdButton_CATHelpAboutCommand/CATHelpAboutCommand/"
                  // The names in the path are the internal names of the CATCommand.
                  // There are no generic naming rules for the internal name, they might be related
                  // to the End User name or not.
    Event="CATDlgMenuIActivateNotification">
    <Datas>
      <Data Num="1" Type="CHAR" Value="CATDlgPushItem"/>
      <Data Num="2" Type="ULONG" Value="2147483658"/>
      <Data Num="3" Type="INT" Value="0"/>
    </Datas>
  </Interaction>
  <Interaction
    ID="CR3"                                                 // Interaction number 3 is clicking on "OK" button
                                                             // of the "Help About window"
    Path="CATCommandSelector/CATApplication/CNEXT/HelpAboutWindow/"
    Event="CATDlgDiaOKNotification">
    <Datas>
      <Data Num="1" Type="CHAR" Value="CATFrmDialog"/>
      <Data Num="2" Type="ULONG" Value="787978"/>
      <Data Num="3" Type="INT" Value="0"/>
    </Datas>
  </Interaction>
  <Interaction
    ID="CR4"                                                 // Interaction number 4 is clicking on the 
                                                             // "File | Exit" menu item.
    Path="CATCommandSelector/CATApplication/CNEXT/MenuBar/START/CmdButton_EXIT/EXIT/"
    Event="CATDlgMenuIActivateNotification">
    <Datas>
      <Data Num="1" Type="CHAR" Value="CATDlgPushItem"/>
      <Data Num="2" Type="ULONG" Value="2147483658"/>
      <Data Num="3" Type="INT" Value="0"/>
    </Datas>
  </Interaction>
</Interactions>
</Record>

[Top]

Understanding the Internals of the Record/Replay Engine

The record/replay engine is at the heart of ITC. It is an extension of the send/receive model events that is used for all user interactions in V5. Being part of the V5 core provides platform independency all well as some stability related to UI changes.

Principle

How does it work? When user makes an interaction with the keyboard or the mouse, an event is sent by the system to the V5 application. This event is translated into a CATNotification that propagates in V5 using the CATCommand send/receive mechanism. In general this first notification triggers a whole set of cascading CATNotification related to UI changes. What is recorded in the capture.rec file is the first CATNotification of all, and this one only.

At replay time, the record engine just has to make sure that this first CATNotification is sent again, and all the cascading CATNotifications and corresponding associated code will execute as if all was triggered by a real user interaction.

In short, V5 translates the operating system events into its own model event. Those events can be recorded and replayed to simulate real user interactions.

[Top]

Record Information

The V5 record needs to store the whole information necessary to send the CATNotification at replay time. This information serves three purposes:

1. Identify the instance of the CATCommand that sends the notification
2. Identify the kind of notification that needs to be sent
3. Provide additional data necessary to perform the interaction (for example, if the interaction is a mouse move, provide mouse coordinates).

CATCommand Identification

To identify a CATCommand instance, the record engine takes advantage of the CATCommand tree. At capture time, the record engine stores the path from the CATCommand instance to the root of the tree. At replay time, the engine follows this path the reverse way, from the root to the CATCommand instance.

A CATCommand path is a string made of CATCommand names. It begins with the CATCommand parent of all, whose name is "CATCommandSelector" (this CATCommand has the same name as its class name). CATCommand names are separated by "/". The CATCommand instance that sends the notification corresponds to the last of the path.

For the record/replay engine to work, the CATCommand tree must have the following requirements:

• A CATCommand instance cannot have two children with the same name.

  When this situation arise, an indetermination exists to find the correct CATCommand instance. This situation is handled by the replay engine by generating an error message and making the replay fail. This situation is not detected at capture time, so in those cases the recording step will work but not the replay.

• A CATCommand must have a non empty name.

  Empty names are not supported in CATCommand path at replay time. If this situation arises, the record will succeed but the replay will fail.

In both cases, V5 applications will work fine interactively. However, it will not be possible to make interactive tests using ITC.

The CATCommand path is stored in the capture.rec file for each notification. When the replay engine fails to find the instance of a CATCommand using the path, a "CATCommand not found" error message is generated and the replay fails with code 1.

Notification

The notification corresponds to the event that is to be sent by the last CATCommand of the CATCommand path. Each Dialog object is a CATCommand and has its own set of notifications, depending on the kind of interaction they support.

For instance, pressing OK button on a CATDlgNotify dialog box corresponds to a CATDlgNfyOKNotification. Clicking with any button of the mouse in a CATViewer corresponds to a CATPressEvent.

Additional Data

Additional data is information necessary to correctly replay the notification. The values stored differ according to the kind of notification that is stored.

For example, in case of a CATMotionEvent, which corresponds to a mouse move over a CATViewer, the data number 3 and 4 are mouse X and Y coordinates. Data number 5 and 6 are the graphic window size.

[Top]

Replay Time

At replay time, interactions are read sequentially in the capture.rec file, and the read notifications are sent by the CATCommand, thus simulating user interactions.

A new notification is sent at each OnIdle event that is generated by the operating system, thus synchronizing the speed of the replay at the maximum machine speed.

During replay all draw requests in the visualization area are not taken into account, except those that come from the CATUpdateEvent which corresponds to draws that were recorded at capture time.

[Top]

Tips to Write Code Compliant with the Record/Replay Engine

• Give names to all instantiated CATCommand
• Give explicit and unique CATCommand names. Since the CATCommand name appears in the CATCommand path, having a pertinent name facilitates the finding of the corresponding source code in case of problem
• Do not instantiate two CATCommands with the same name and with the same parent
• Do not use timer or thread synchronization mechanisms
• Avoid to make record/replay dependent code. In this case, the standard session code will not be covered.

[Top]

Tips to Make Pertinent Tests

• Create a text file that describes each interaction of the scenario. This will be of great help for analyzing in case the test fails
• Do not forget that the replay engine does not check the result of the interactions. As long as proper CATCommands are found to send the notifications, the replay goes on
  In particular, the result of all interactions in a visualization area (CATViewer) are not checked. For example, if a mouse click results in a selection at capture time, the replay engine does not check that the selection is indeed performed at replay. All the replay engine needs is the CATCommand associated with the click to be able to simulate it
• Use contextual menu commands (such as Delete) to make sure that an object is under the mouse
• Save your model and compare it to a reference file (you need to write our own model comparator)
• Add your own debug/check commands.

[Top]

References

History

Copyright © 2004, Dassault Systèmes. All rights reserved.