Eiffel Home Page (Web) -- Getting started with Eiffel (local)

Methodological observations

Although the field of software metrics is a rich one with an abundant literature, its methodological basis is sometimes subject to question. One should resist the tendency to believe numbers just because they are numbers ("lies, damn lies, and metrics").

Software engineers and their managers expect, however, to reap at least some of the benefits that precise quantification has brought to other engineering fields. Such is the purpose of software metrics, defined as quantitative estimates of product and project properties . Object-oriented development, with the rich software structures that it induces, is a particularly amenable to metric analysis. Even when some of the measures do not seem to bring much by themselves, comparing them to those of other projects may reveal significant peculiarities of a system or of some of its parts.

The metrics capabilities of EiffelStudio were designed with these observations in mind. They result from a conservative approach, where no metric is provided without a credible assumption that it reflects some meaningful project or product attribute. For example, you will find a way to define a new metric as a linear combination of existing ones, but not a way to compute arbitrary arithmetic operations, since it isn't clear that -- say -- multiplying two metrics ever makes sense.

Metric terminology

The following terms are used in the presentation of EiffelStudio metric mechanisms.

A metric -- not to be confused with a measure -- is a quantitative property of software products or processes whose possible values are numbers. A measure is the value of a metric for a certain product or process.

For example, we can evaluate the metric "number of classes in the system", called just Classes , by counting the classes in our system. This yields a measure.

We may distinguish between product metrics , which measure properties of the elements being turned out (code, designs, documentation, bug reports...) and process metrics , which measure properties of the process used to turn them out (salaries, expenses, time spent, delays...). The current metric facilities of EiffelStudio are primarily product-oriented but include a process metric: "number of compilations".

Any metric should be relevant: related to some interesting property of the processes or products being measured, such as cost, estimated number of bugs, ease of maintenance... A metric theory is a set of metric definitions accompanied with a set of convincing arguments to show that the metrics are relevant. Neither EiffelStudio nor this manual provide a metric theory.

The numbers yielded by measures are meaningless unless we describe what they refer to. Every metric is relative to a certain unit , specified as part of its definition. For example the unit for a metric that counts classes, such as Classes , is called CLASS .

EiffelStudio provides a set of predefined units. Some simply serve to count occurrences of certain construct specimens in the software; examples include CLASS , CLUSTER , FEATURE , LINE , … The metric RATIO describes metrics whose values are divisions, for example "average number of classes per cluster", obtained by dividing the number of classes by the number of classes.

Any metric applies over one or more scope types . A scope type is a type of product or process over which the metric is measured; for product metric, examples include feature (meaning that we will compute a metric over a single feature), class , cluster , system . They obey an order relation corresponding to containment order: a feature belongs to a class, a class to a cluster and so on.

A scope is an instance of a scope type. For example a given cluster is an instance of the scope type cluster .

To compute a measure is to apply a certain metric over a certain scope of an applicable scope type. For example we may compute the value of the metric Classes over a certain system.

Kinds of metrics

The EiffelStudio metric framework provides a number of predefined metrics but also enables users to define their own metrics in terms of the predefined ones according to a taxonomy illustrated as follows:

Metrics are divided into elementary and composite:

An elementary metric measures the number of occurrences of a certain pattern in the product or process. An example is the number of precondition clauses in a class.

A composite metric, defined by a user of EiffelStudio, applies a mathematical or logical formula involving other metrics (elementary or previously defined metrics).

Among elementary metrics, we make a further distinction between raw and derived:

Raw metrics are simple counts, built-in into EiffelStudio, of occurrences of certain basic elements. For example Classes , the number of classes, is raw.

It is often useful to define a new metric by subjecting a raw metric to one or more selection criteria . For example a class may be either deferred (abstract) or effective (concrete, i.e. fully implemented). This is a selection criterion. Separately, a class may have an invariant, or not; this is another selection criterion, Invariant_equipped . You might want to know the number of classes that are deferred and have no invariant clause; if so, you may define a derived metric by submitting the raw metric Classes to both of these criteria connected by an "and".

The precise definition of selection criterion for a raw metric is: a property with a fixed set of possible values (two or more) characterizing the patterns being counted by the metric. Without the notions of selection criteria and derived metrics, EiffelStudio would need to have predefined (raw) metrics including all possible combinations, such as "deferred and no invariant". This would quickly grow out of hand.

Defining a derived metric

Let us define a new derived metric. In the Context Tool, choose the Metrics Tab. Click the Define new metric button:

In the dialog that comes up, the first tab, selected by default, is the one we want, Derived :

In the Raw Metric entry, currently set to Classes , bring up the menu and select Features . Note the large number of selection criteria applicable to the raw metric Features (number of features), reflecting the many angles under which it is possible to classify features:

Assume you want a metric called Public_functions_full_contracts that counts the number of features that are functions, are exported, and have both a precondition and a postcondition. Enter its name and check the selected criteria:

Click OK to add this new metric to the set of available metrics. It's actually been entered as the default metric to be computed, although we'll compute it only in a short while:

Note that the Unit field indicates the unit automatically assigned to this derived metric, Feature , the same as the unit for the raw metric Features from which you have derived it.

Composite metrics

A composite metrics applies one or more mathematical operators to a set of metrics, themselves either elementary (raw or derived) or already composite. In line with the conservative approach mentioned earlier, EiffelStudio permits only three kinds of mathematical operators:

Linear metrics of the form ∑ k i . m i , where the k i are real values and the m i existing metrics (either elementary or basic) with the same unit, other than RATIO . (It would be improper to add two RATIO values since they might be ratios of incompatible things.)

Ratio metrics of the form m 1 / m 2 where both m i are previously defined metrics, not necessarily with the same unit, neither of which a RATIO (again because RATIO is a catch-all category for all divisions, so you can't divide further without courting incoherence). The resulting unit is RATIO .

Scope comparison metrics that measure the ratio of the value of a given non-ratio metric over two different scope types. For example by choosing the metric Classes and the scope types "cluster" and "system" we can measure the proportion of classes in a system that belong to the current cluster.

Let's define a linear metric Direct_dependents that counts heirs (direct descendants) and direct clients of a class. Click Define new metric again and choose the tab Linear . Bring up the Metric menu:

To define the first term of the sum, select Clients . Do not click the button marked OK (we are not done yet as we need a second term) but instead click the + button next to Metric :

The field at the bottom reflects the current state of the linear combination, which only includes the term Clients , with the default coefficient 1. Now bring up the Metric menu again, choose Heirs this time, and click + again. The expression for the sum appears in the bottom field:

The name of our new linear metric will be Direct_dependents . Enter it into the top field, New_metric_name (if you forget, you will be reminded), and click OK . The name of the new metric and its unit become the defaults in the corresponding fields of the Context Tool.

Let's define one more metric, this time a ratio. We want to know the "branching factor": average number of heirs per class. It will be the ratio of metrics Heirs and Classes . Click Define new metric then Metric ratio :

Select Heirs in the Numerator menu on the left, Classes in the Denominator menu on the right; do not check the box "Display as percentage" at the bottom (we prefer to see the branching factor as a number); enter the metric name Average_heirs , more clear than "branching factor", in the field at the top, and click OK . You now have three user-defined metrics available in addition to predefined metrics, as you can check by bringing up the Metric menu on the Context Tool:

Applicable scope types

The reason Average_heirs is grayed out on the last figure is that it's not applicable to the current scope. Each metric indeed has a minimum "applicable scope type"; for example it doesn't make sense to compute the number of attributes in a feature, since the smallest construct in which attributes appear is a class, bigger than a feature. Similarly, the minimum scope type for Classes (the denominator of Average_heirs ) is Cluster , since we can only start counting classes at that level.

Computing measures

Let's now compute a few measures. These will all be relative to the entire system, so the first thing to do is to change the Scope field of the Context Tool (the second field from the left at the top on the preceding figure) to System , the next-to-last entry on the corresponding menu.

The measures that you get should be identical or very close to those shown next. Differences, if any, may result from changes in the EiffelStudio delivery, especially EiffelBase, or operations that you have performed on the Guided Tour system.

First let's use a predefined metric to compute the number of classes in our system. In the Metrics field bring up the menu and choose Classes --> All . The Unit field is updated to the unit of that metric:

For more clarity, change the name of the next measure in the leftmost top field, set by default to Result1 , to Class_count . Now compute that measure by clicking the Calculate metric button, at the place shown on the above figure. This displays the resulting measure in the field just below:

As you see, our system has 183 classes. Refining the measure -- for example by restricting the scope to certain clusters -- would show that all but a handful belong to the EiffelBase library. But you already know this, so don't perform these measures yet (you can do them later).

Saving measures

When you compute a measure, it's not kept anywhere, and the next measure will displace it. To keep the one you just made, click the button labeled Add measure to the list and save it on the last figure:

The operation has had two effects:

It has added the latest measure to the table of currently displayed measure.

The first time around, as here, it creates a metrics file; a messages tells us this:

Metric information will be kept in the subdirectory Metrics of the project directory. The generated metric_file.xml file contains both the metrics you have defined, and the measures you have performed. This makes it possible to use it both for:

Applying to a new project the metrics that you have defined for an existing one (so that, as you go, you can accumulate a portfolio of useful metrics).

Using existing measures as a comparison for new projects.

To obtain more measures, apply the three metrics that you defined earlier -- with names Public_functions_full_contracts , Direct_dependents and Average_heirs -- to the entire system and save the corresponding measures. You can use appropriate names for the results or use the EiffelStudio defaults. The result will be as follows:

Operations on metrics

Click the button labeled Manage metrics on the last figure. This brings up the metric management dialog:

You can use it to rearrange the order of the metrics you have defined (by selecting one of them and clicking Up or Down ) and delete any of them. The Formula field shows the definition -- arithmetic, or boolean -- of the selected metric. The Import button will let you, if you have defined metrics in another project, select the corresponding XML file to add them to those of your current project.

Here (possibly after having tried rearranging the metrics) just click Exit .

Metric archives

What does a measure mean? You don't necessarily know in the absolute, but you might want to compare your results to those of other projects. For example, if you have measured the average number of invariants in classes of your system, you might be curious to know how this compares to ISE's EiffelBase library.

The notion of metric archive addresses this need. You may:

As noted above, archive all current measures into an XML file, called a measurement archive.

Make this measurement archive available in a shared directory, or as a URL on the Internet.

At any time in a project, select any measurement archive, local or at a URL, as the reference archive; in that case all the measures you perform will be compared to those of the reference archive. You may select various comparison formats: percentage (the default), difference percentage, plain value.

ISE has established a Web site, http://metrics.eiffel.com , as a publicly available reference for metric collections on ISE's own libraries (EiffelBase, EiffelVision, ...). This provides an invaluable source of comparisons for other projects within and without ISE.

Let's see how this works. We'll compare the number of classes in our system to the number of classes in EiffelVision. Click the last button, Compare to archive , in the Context Tool:

This brings up the archive comparison dialog:

You can Browse … to select a local archive (a .xml file) for comparison, but if you have access to the Internet enter the following URL in the File or URL file:

http://metrics.eiffel.com/vision.xml

If you do not have access to the Web, Browse … instead to the file vision.xml of the documentation directory, which has the same information, although possibly less up to date.

Click the Set button next to Ratio of computed measure to archive to select this format for displaying the comparisons.

Click Close . This updates the display of computed measures as shown by the following figure. All future measures that you perform will be immediately compared to the values in the selected archive if the metric is available on both sides.

The option that we saw in the Compare to archive dialog enables you to define an existing archive as reference for your current and future measurements.

The other two self-explanatory options, shown in the picture of the dialog on the previous page, enable you to create your own archive, and to update an existing one. You can then make this archive available, in a local file or on the Web, for other projects to perform comparisons.

An archive, stored as an XML file, includes both the project metrics and its current measures. The metrics will be made available for import to projects selecting the archive for comparison.