SNOMED International Mapping Tool

Space shortcuts

Child pages
  • GMDN Mapping

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 35 Next »

Overview

Information on the GMDN mapping project.

  • ModuleId: 466707005
  • RefsetId: 467614008

Setting Up

The following steps were used to create the projects:

Monthly GMDN Update

The data files for GMDN get updated monthly and should be loaded in a timely fashion.  To get the data files onto the prod mapping server the following process is used:

  1. A .zip file of GMDN data is put on to the mapping.backup.ihtsdo Amazon S3 bucket

  2. The "pullGmdn.csh" script on the prod server (in /home/ihtsdo/bin, and also in the repository as part of the prod config package) a cron job every 20 min to check for the availablility fo a GMDN package.  NOTE: this requires the cron user to be able to run "sudo aws s3 ls mapping.backup.ihtsdo"

    * 20 * * 0,1,2,3,4,5,6 csh /home/ihtsdo/config/bin/pullGmdn.csh > /home/ihtsdo/logs/pullGmdn.log 2>&1
  3. Any files containing "gmdn" in the filename are downloaded
  4. For each
    1. A file name with the pattern "gmdn.*(\d\d_\d\d).zip" is expected and a directory of the digits is created (e.g. 17_1) in the /home/ihtsdo/data/GMDN folder.
    2. If the filename does not follow this pattern, it gets unzipped into a directory matching the filename without .zip
    3. It is expected that this top-level directory contains the actual XML GMDN files

  5. For example,

    % ls -lt /home/ihtsdo/data/GMDN/17_1
total 93700
-rw-rw-r-- 1 ihtsdo ihtsdo 58681302 Jan  3 12:02 termcollectiveterm17_1.xml
-rw-rw-r-- 1 ihtsdo ihtsdo 34357440 Jan  3 11:57 term17_1.xml
-rw-rw-r-- 1 ihtsdo ihtsdo  1889072 Jan  3 11:56 collectiveterm17_1.xml
-rw-rw-r-- 1 ihtsdo ihtsdo  1003824 Jan  3 11:56 cttreenode17_1.xml

  6. As a separate (and currently manual process), the GMDN version should be loaded (the server must be taken down for this process). NOTE: if there are any problems, download the GMDN version (e.g. zip and copy to the "doc" folder and the file should be accessible, e.g. https://mapping.ihtsdotools.org/doc/17_1.zip) and try loading it into a local dev environment to track down the errors.  The loader assumes the structure of the .xml files is constant - if it changes that may cause problems.

    # This process takes about 15 min
service tomcat7 stop
cd ~/code/admin/loader
mvn install -PGMDN -Dinput.dir=/home/ihtsdo/data/GMDN/17_1 \
  -Dterminology=GMDN -Dversion=17_1 -Drun.config=/home/ihtsdo/config/config.properties

  7. Determine the older version(s) of GMDN

    mysql> select distinct destinationTerminologyVersion from map_projects where destinationTerminology='GMDN';
+-------------------------------+
| destinationTerminologyVersion |
+-------------------------------+
| 16_10                         |
+-------------------------------+

  8. Remove the older version, then restart the server

    # This process takes under 5 min
cd ~/code/admin/remover
mvn install -PTerminology -Dterminology=GMDN -Dversion=16_10 -Drun.config=/home/ihtsdo/config/config.properties
service tomcat7 start
  9. Navigate to the mapping tool (e.g. https://mapping.ihtsdotools.org).
    1. Log in as "mappingadmin"
    2. Change to the GMDN project
    3. Find the GMDN project in the "map projects" accordion
    4. Update the destination terminology version the latest version (e.g. 17_1)
    5. Save the map project.
       
  10. Concepts from the updated GMDN version should now be available to mappers.

Helpful Admin Commands

Remove

cd ~/code/admin/remover
mvn install -PMapRecords -D$rc -Drefset.id=467614008
mvn install -PTerminology -D$rc -Dterminology=GMDN -Dversion=latest

Load

cd ~/code/admin/loader
mvn install -PGMDN -Dinput.dir=/home/ihtsdo/data/GMDN/16_10 -Dterminology=GMDN -Dversion=17_1 -D$rc
cd ~/code/admin/loader
set file = /home/ihtsdo/data/doc/release/20160731/gmdn/der2_sRefset_SimpleMapActiveSnapshot_INT_20160731.txt
/bin/rm -f /tmp/x.txt
perl -ne '@_=split/\t/; print if $_[4] eq "467614008"' $file > /tmp/x.txt
mvn install -PSimpleMapRecords -D$rc -Dinput.file=/tmp/x.txt -Dmember.flag=true -Drecord.flag=false

Compute

cd ~/code/admin/loader
mvn install -PComputeWorkflow -D$rc -Drefset.id=467614008

Release

cd ~/code/admin/release
mvn install -PBeginRelease -D$rc -Drefset.id=467614008 -Dtest.mode.flag=true
mvn install -PRelease -D$rc -Drefset.id=467614008 -Doutput.dir=. -Dtime=20170131 -Dmodule.id=466707005 -Dtest.mode.flag=true
mvn install -PFinishRelease -D$rc -Drefset.id=467614008
mvn install -PBeginEditingCycle -D$rc -Drefset.id=467614008

 

Loading "Human Readable" View

Similar approach to ICD10 but using the "simple" map loader.  e.g.

cd ~/code/admin/loader
set file = ~/data/GMDN/*2016*txt
grep -v 900000000000207008 $file | perl -pe 's/467614008/P467614008/;' >! x.txt
mvn install -PSimpleMapRecords -Drun.config=/home/ihtsdo/config/config.properties -Dinput.file=x.txt -Dmember.flag=false -Drecord.flag=true >&! mvn.log

Derivative Release Process

  1. Determine the source terminology version (SNOMEDCT), the destination terminology version (GMDN) and the effective time for the release.  The effective time and the source terminology version will be the same release date as the immediately previous SNOMED International release (e.g. 20XX0731 or 20XX0131).  The destination version should include the last GMDN content that got included in the International Edition before the content cut off date (for July release, this would be early May, e.g. 17_5).  Examples below are based on SNOMEDCT 20170731 to GMDN 17_5 release. 
  2. Stop editing on Prod
  3. Check Prod database for no tracking records on the project.
  4. Clone the final prod backup to the release server.
  5. Begin next editing cycle on the Prod server.
  6. Compute Workflow on Prod.

  7. On release server, stop the server

    supervisorctl stop mapping-service

  8. On release server, load previous release simple map refset members with GMDN refsetId from previous (e.g.20170131) release.  Andrew will provide this file, or it should be on server in previous release file.

    cd /opt/mapping-service-admin/admin/loader
perl -ne '@_=split/\t/; print if $_[4] eq "467614008"' /home/ihtsdo/data/doc/release/20170131/gmdn/der2_sRefset_SimpleMapActiveSnapshot_INT_20170131.txt > /tmp/x.txt
mvn install -PSimpleMapRecords -Drun.config=/opt/mapping-service/conf/config.properties -Dinput.file=/tmp/x.txt -Dmember.flag=true -Drecord.flag=false

  9. Remove SNOMEDCT 20170131.

  10. Add SNOMEDCT 20170731

    cd /opt/mapping-service-admin/admin/loader
mvn install -PRF2-snapshot -Drun.config=/opt/mapping-service/conf/config.properties -Dterminology=SNOMEDCT -Dversion=20170731 -Dinput.dir=/home/ihtsdo/data/SNOMEDCT/SnomedCT_InternationalRF2_PRODUCTION_20170731T120000Z/Snapshot > /opt/mapping-service/logs/loadSnomed20170731.log

  11. Add GMDN 17_05.  (OK if later versions remain in the db as long as the project references the correct release version)

    cd /opt/mapping-service-admin/admin/loader
mvn install -PGMDN -Dinput.dir=/home/ihtsdo/data/GMDN/17_1 \
  -Dterminology=GMDN -Dversion=17_05 -Drun.config=/opt/mapping-service/conf/config.properties > /opt/mapping-service/logs/loadGmdn17_05.log
  12. Reindex (is this necessary?)

  13. Start the server 

    supervisorctl start mapping-service
  14. Update the project to point to the SNOMEDCT and GMDN release versions.
  15. Compute Workflow
  16. Begin release
  17. Check QA report
  18. Have mappers fix any errors.
  19. Compute Workflow
  20. Check database for no tracking records on the project.
  21. Begin release
  22. Process release
  23. Copy ActiveSnapshot file back to Prod
  24. On Prod, run Preview Finish
  25. Check Release Finalization report
  26. Run Finish Release.
  27. Question: we loaded sources onto the release server.  What needs to be loaded into prod?

NOTE: No need to create 'Published SNOMED to GMDN'.  GMDN project is private.

  • n/a