Objective Practice Adherence Measurement

Practice adherence data based on concrete project data.

Description

Objective metrics are drawn from evaluation of the project data, given our expectation that the security practices of a team will be reflected in the documentation the team creates, and the logs of activity the team generates.

We collect the following objective practice adherence metrics for each practice:

• Presence: whether we can find evidence of the practice.

  • Values: True, False.

• Prevalence: Proportion of the team applying the practice, the ratio of all practice users to all team members.

  • Values: 0 - 1.00.
  • Alternate Values: Low, Medium, High.

When recording practice adherence manually, it is sufficient to record the following data elements:

• Practice - Name of security practice associated with document.
• Practice date: Date for which evidence of practice use is claimed by the researcher.
• Presence - as described above
• Prevalance - as described above

When recording practice adherence events automatically from emails, issues, commits, we recorded the following data elements:

• Practice - Name of security practice associated with document.
• Event Date - Date on which document was created.
• Source - Data source for document. Possible Values: Version Control, Defect Tracker, Email.
• Document Id - Id of document in its source, e.g. commit hvi ash, bug tracker id, email id.
• Creator - Role of the author of the source document.
• Assignee - For defect report documents, the person assigned the defect, where applicable.