Content Managers Advisory Group

Page tree

Summary:

Development of prioritisation criteria to support prioritisation of the items on the SNOMED International Content Tracker JIRA project

Scope of work: (proposed)

This sub group has been set up to develop prioritisation criteria for the issues on the IHTSDO Content tracker. This work involves:

  • Identification of the prioritisation criteria
  • A recommendation of how to apply the criteria
  • A recommendation on the meaning of each of the priority levels
    • The number of priority levels available will be 3.
  • Recommendations by the group on the priority of each 'component' on the IHTSDO Content Tracker 
  • Recommending a time period for review of the priorities to confirm they are still current. 

It is noted that the work here may support prioritisation of the patterns on the Pre-coordination Pattern tracker. 

Out of scope:
  • Prioritisation of individual issues on the IHTSDO Content Tracker
  • Work to support the clean up of the IHTSDO Content Tracker (excluding that specified in the scope of work statement)


Timeline for work:

 

MonthActivitiesComments
August  
September  
October  
November  

 

 

Group Members:

GoToMeeting Information

Please join my meeting from your computer, tablet or smartphone.
https://global.gotomeeting.com/join/975014981

Actions: 

DateRequested actionRequester(s)Response required by:Comments
4 August 2016
  • Input to support defining the scope of work
  • Input to support development of the timeline for the work
  • Input to determine the frequency of meetings

Cathy Richardson

  • Daniel Karlsson   Prioritisation criteria:scope of work, timeline for work, meeting frequency
  • Elaine Wooler    Prioritisation criteria:scope of work, timeline for work, meeting frequency
  • John Fountain    Prioritisation criteria:scope of work, timeline for work, meeting frequency
  • Linda Parisien    Prioritisation criteria:scope of work, timeline for work, meeting frequency
 
4 August 2016
  • Doodle poll for next meeting
Cathy Richardson
 
8 August 2016
  • Review documents (as noted below) and add comments to this page to support criteria development
Cathy Richardson
  • Daniel Karlsson   Review Content Tracker Guidance and Stakeholder engagement documents. Provide initial ideas for priorisation criteria.
  • Elaine Wooler    Review Content Tracker Guidance and Stakeholder engagement documents. Provide initial ideas for priorisation criteria.
  • John Fountain    Review Content Tracker Guidance and Stakeholder engagement documents. Provide initial ideas for priorisation criteria.
  • Linda Parisien    Review Content Tracker Guidance and Stakeholder engagement documents. Provide initial ideas for priorisation criteria.
  • Cathy Richardson Review Content Tracker Guidance and Stakeholder engagement documents. Provide initial ideas for priorisation criteria.
 
22 August 16
  • Outcome and efforts based framework development

 
 
26 September 16
 
 
26 September 16
  • Prep for next meeting
 
 
3 October 2016 
 
17 October 2016

Framework update

Review of ratings

 
  • Cathy Richardson  Seek input from Jim Case on the Clinical Impact aspect of the criteria
  • Cathy Richardson  Update the framework ready for meeting next week
  • Daniel Karlsson  Analysis of the ratings to determine inter-rater reliability
 

Documents:

  File Modified
Microsoft Word Document Criteria and Guidance for Initiating a SNOMED CT Content Project V1.0.docx 2016-Aug-08 by Elaine Wooler
Microsoft Word Document A_Guide_to_Stakeholder_Engagement_v1_0_20130303 (6).docx 2016-Aug-08 by Cathy Richardson

Meeting 4 August 2016: Plan and notes

  • Showing of the Content tracker, Pre-coordination Patterns tracker (covered)
  • Scoping of work and timeline for development (Input to be provided)
  • Scheduling of meetings (input to be provided)
  • Setting of actions in prep for next meeting (Completed)

Meeting 8 August 2016: Plan and notes

  • Scope agreed.
  • Timeline to be determined
  • Meeting frequency and dates. Monday 2000UTC seems good for all. 
    • Fortnightly: 2nd and 4th Mondays
  • Start development of criteria.
  • CMAG update: CRI to provide.
  • Recommendations will go to the broader CMAG group. They will also be provided to the Head of Content and Head of Terminology for sign off. 
  • Link to SIRS request data presented April 2016: CMAG_SIRS_ContentTrackersFeb16_Meeting20160418.xlsx.xlsx

Meeting 22 August 2016: Plan and notes

  • Discussion on input:
    • Potential to work on low hanging fruit- value of clearing fast track
    • Major changes- take longer 
    • Size- estimated and can change (still of value) Estimated upon inception at this point and at the end of the inception phase.
      • Effort is dependant on the number of concepts, complexity of issue, amount of stakeholder engagement required and whether the change can be managed in an automated way or needs to be done manually. 
        • E.g 10 concepts but very complex or 1000 concepts but simple.. potential to separate out
      • Ability to measure - qualitative and quantitive e.g. number of descriptions
    • What issues are insolvable at that this point?
        • Parked until solvable
        • Issues should be reviewed- 1-2 years. 
        • Degrees of in-solvability
        • List kept
    • Test the criteria as developing. 
    • Major and minor criteria different to major and minor change
    • Member Forum priorities - issues that are within the scope of work for a MF priority - should be managed as part of that stream of work. 
      • SIRS requests not within that scope. 
    • In prep for next meeting:
      • John- redraft of major/minor changes - draft a framework. Outcome and effort focused. Will post for group discussion. Aim for 2nd September. Group to feedback on document. 

Meeting 26th September: Plans and notes 

  • Agreement on framework structure
  • Submitters to be asked to provide predefined information e.g. number of concepts
  • Guidance material should be provided to support submission of information
  • Framework needs to be easy to apply
    • Needs to allow current issues to be managed quickly and suitable for future issues. 

 

  • Submitter to provide (this is still in development)
    • Identification of existing concepts - concepts affected, number used as a target value in an attribute relationship
      • New- more difficult (may be an educated guess)

  • Plan:

    • Prior to next meeting on 3 October:

      • Group to review the criteria in framework document over next week

      • John to update document on 3 October (NZ time)

      • Cathy to select Content Tracker issues ready for testing

    • 3 -17 October Group to test criteria

      • 17 October: meet to discuss results 

    • Update to be provided at Face to Face meeting. 

Meeting 3 October: Plans and notes

  • Note- Our meeting needs to finish 10mins to the hour as there is a back to back meeting on the same GTM account and access is required a few minutes prior to the start of that meeting.  
  • Discussion on framework- decision to use the revised version. 
    • Explanation section
      • Needs to be further developed so it is clearer for those using it.
        • Build in further examples.
      • Clinical impact- needs a decision on how clinical priority is to be determined. Cathy to raise with Jim. 
    • Efforts section
      • Keep at 3 levels in scoring.
      • Add tooling to resource section.
    • Group to add any additional comments and Cathy will finalise.
  • Criteria to be tested over next two weeks. Cathy to finalise tracker list on Tuesday 4th October. 

Meeting 17 October: Plans and notes

  • Comments and feedback on the framework
  • Time taken to review trackers?- not asked. CRI noted easy to apply.
  • Review of results
  • Plan for:
    • Cathy to update framework and seek input from Jim on Clinical Impact
    • Daniel to review the individual ratings for each tracker reviewed to determine consistency. 
  • Comment: Process for applying? CRI noted recommendation by submitter and review and determination by an internal Content team member. Suggestion raise that it may be of value to have more than one person do the rating to ensure consistency. 

Meeting 12 Jan: Plans and Notes

  • Information to be sought on member priorities
    • Managed by Linda, Elaine and Cathy
      • This smaller group will potentially meet next week - 19th Jan 2000UTC. Need to confirm all can attend at this point.
    • Last 2 years Member priority submissions to be looked at.
    • Member Forum to be approached for input
      • Will be provided with a spreadsheet listing the components and and each country asked for
        • priorities
        • comments if there are specific areas within that component. For example with the category neoplasms there may be a project within a country focusing on the genetic aspect or a type of neoplasm.
      • Consideration needs to be given to whether this is held until the next meeting Feb 14th or they are approached directly. 
  • A report on which content trackers are holding up CRS tickets is being developed by Rory. Coming out of this a confluence page will be set up as a reporting page that is automatically updated. The report will be required by the end of January to support the work.
    • A request will be made to also provide this information at a component level given this work will be undertaken at that level from an external perspective. Cathy to raise this with Rory.
  • The due date for overall deliverable is 28th Feb. Timing may need to be extended. Cathy to speak to Lesley. 
  • This sub group as a whole will next meet in the next couple of weeks. Potential times are 23 Jan, 30 Jan or 31 Jan at 2000UTC.

 

 

 

 

 

 

  • No labels

5 Comments

  1. Elaine Wooler

    Hi Cathy

    Thanks for organising.  I was thinking it would be useful to circulate the guidance document we produced for initiating a project as part of the Consultant Terminologist programme which you mentioned in the last meeting.  I can see it on the CTP site but not sure if it is available elsewhere so have attached.  There is mention of initial priority setting so I think we need to link in to some of this as well as developing it further.

    Cheers

    Elaine

  2. Cathy Richardson

    Here is the Guide to Stakeholder Engagement document developed by the 2012 cohort of Consultants. Please note the purpose of this document was " to provide a framework for stakeholder engagement that may be applied to either the Fast Track or standard content development process" - so the focus is the Content tracker items. 

  3. Linda Parisien

    Hi Cathy and Elaine,

    Thank you for providing us with the two documents. This is helpful to set the mindset, although it contains a lot of information...

    I think as a starting point, some items should be more defined such as: 

    • what is a minor change vs a major change
    • when is the fast track process used over the 'regular' one
    • what are the outcomes we'd want to measure:
      • process content that is simpler sooner in the workflow?
      • increase number of RFCs resolution using the fast track process?
      • decrease the number of unassigned RFCs on the tracker?
      • complete more RFCs in a cycle?

    While looking at the Content tracker, I had a hard time to relate the 'Component' items to the 'Unresolved:By Priority' information on this page: /projects/IHTSDO/summary/statistics

    First, to me the Member Forum Priorities defined on a yearly basis is a top-1 priority, then if a request is being made for an imminent implementation, that would be a top-2 priority, which would be followed by QA issues (top-3 priority) and other RFCs (top-4 priority). Of course there are more complex use cases that could potentially modify the priority such as:

    • When a request for same content is made by more than 1 country, this should increase the priority
    • Any content that went through the approval for concept modeling change or editorial change and SME guidance that meet the above criteria should be added to the worklists and prioritised appropriately. This would require an agile process to update tracker asap and get the work started on the RFCs

    Then there is the level of effort required to achieve the changes to X number of concepts. This information will be useful to determine when and if the changes can be achieved within a time frame. That information coupled with the priority would perhaps be easier to manage... I am not sure about the categorisation but let's say level of effort (LE) 1= 1-99 concepts, 2= 100-999 concepts, 3=1000-4999 concepts 4= + 5000 concepts.

    From the Content Project document, we have clear criteria for when a project is assigned. That information is not currently available from the tracker information. I would suggest a mention to the 'Component' or otherwise that identifies what is requiring:

    • a concept model change (CM)
    • an editorial guidelines change (EG)
    • a SME guidance (SG)
    • Alignment to editorial guidelines (AEG)

    We could use for the Content Project the same LE indicator for the level of effort required or number of concepts that need to be changed.

     

  4. John Fountain

    Identification of the prioritisation criteria

     

    I wonder if this can be approach with a “major” and “minor” criteria approach.

     

    Major Criteria

     

    These would encompass highest level priorities, possibly in the following order:

     

    1.            Errors in existing content

    2.            Member requests

    3.            SIRS requests

     

    A certain value can be ascribed to each of these levels. However, to further refine the priority process there can be divisions within each of these levels which also are valued.

     

    Minor criteria

     

    Within the major levels can be the minor criteria, e.g.:

     

    1.            Errors in existing content

    a.            major error with potentially significant clinical implications

    b.            error impacting existing implementations

    c.            error impacting an intended implementation

    d.            error with little effect on the terminology but needs correction at some time

     

    2.            Member requests

    a.            request made by multiple Members with high international value

    b.            request made by two or more Members but without international significance

    c.            single Member request with little impact internationally

     

    3.            SIRS requests

    a.            Request that will negatively impact active/intended implementations internationally if not addressed

    b.            Request that will impact a national implementation if not addressed

    c.            Request that will impact a single vendor / organisation implementation if not addressed

    d.            Request with not impact other than on implementations under consideration

    e.            Routine SIRS requrest

     

    Such an approach, if the values are assigned appropriately between the levels, would allow the ‘movement’ of a lower (major) priority request to a higher level if the underlying implications are of high impact e.g. a multiple Member request with international value could outweigh a recognised error with little practical significance.

     

    Clearly assigning the actual major and minor criteria and their relative values will be interesting...

     

    1. Elaine Wooler

      I like John's approach to priority setting as it makes the process much more transparent.  I know from requesting changes to content in terms of errors in SNOMED that IHTSDO currently have to prioritise these due to resource demands so this seems to fit with their working practices.  

      We probably need to fit into this some of the content from the two documents posted on initiating a content project and stakeholder engagement, mainly:

      • Project that requires change to the concept model
      • Project that requires change to editorial policy
      • Project that requires alignment between editorial policy and content 
      • Project that requires extensive stakeholder engagement

      The first three could be incorporated in to the major criteria perhaps with the later being one of the minor criteria in John's example.  I've not had a chance unfortunately to think this through yet but I would imagine it would work with John's type of approach.  In addition the following are also important in terms of impact to the terminology:

      The impact of a change will generally be higher if: 

      • It is a high-level concept (as opposed to a low-level or leaf concept)
      • It is referenced in a large number of concept definitions
      • If it is referenced in sufficiently defined concepts (particularly in high-level concepts)
      • If it is referenced in high use concepts

       

      Do we also need to consider those projects that are currently "Fast Track" projects - generally simple, non-contentious changes that are usually raised by IHTSDO authors but can be in response to SIRs requests?  

      Sorry this is a bit unstructured but hoping it will help when we get our heads together.