编辑反馈的内容

Anatomical entity recognition with a hierarchical framework augmented by external resources PLOS ONE

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit, but is not suitable for publication as it currently stands. Therefore, my decision is "Major Revision."

We invite you to submit a revised version of the manuscript that addresses all of the concerns raised by the two reviewers. It is critical that you specifically address the following issues: 1) Provide more details on your methodology and data sources (possibly with examples), so that the reviewers can better evaluate the summary results provided in the tables; 2) Describe precisely what will be publicly available; 3) Thoroughly edit your revised manuscript before submission. Please note that PLoS ONE does not provide copy editing.

We encourage you to submit your revision within forty-five days of the date of this decision.

When your files are ready, please submit your revision by logging on to and following the Submissions Needing Revision link. Do not submit a revised manuscript as a new submission. Before uploading, you should proofread your manuscript very closely for mistakes and grammatical errors. Should your manuscript be accepted for publication, you may not have another chance to make corrections as we do not offer pre-publication proofs.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

In addition, when submitting your revision please include the following items:

A rebuttal letter that responds to each point brought up by the academic editor and reviewer(s). This letter should be uploaded as a 'Response to Reviewers' file.

∙ A clean revised manuscript as your 'Manuscript' file.

∙ A marked-up copy of the changes made from the previous article file

as a 'Revised Manuscript with Track Changes' file. This can be done using 'track changes' in programs such as MS Word and/or

highlighting any changes in the new document. ∙

For more information on how to upload your revised submission, see our video:

http://blogs.plos.org/everyone/2011/05/10/how-to-submit-your-revised-manuscript/

If you choose not to submit a revision, please notify us.

Yours sincerely,

Ramin Homayouni, Ph.D.

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements:

1. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold your manuscript until you get in touch with us with the accession numbers or DOIs necessary to access your data. If you wish to make changes to your data availability statement, please describe these changes in your cover letter and we will make them on your behalf.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

3. Does the manuscript adhere to the PLOS Data Policy?

Authors must follow the , which requires authors to make all data underlying the findings described in their manuscript fully available without restriction. Please refer to the author’s Data Availability Statement in the manuscript. All data and related metadata must be deposited in an appropriate public repository, unless already provided as part of the submitted article or supporting information. If there are restrictions on the ability of authors to publicly share data —e.g. privacy or use of data from a third party— these reasons must be specified.

Reviewer #1: Yes

Reviewer #2: No

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This paper presents an interesting hierarchical framework to recognize anatomical entities, which is important in healthcare domain. Authors also bring the importance and the challenges of this task. To the best of my knowledge, I summarize my comments and suggestions as follows:

1) Features for the sequence labeling problems under CRF are comprehensive and acceptable. Authors include baseline natural language features, semantic features from external knowledge about Wikipedia and WordNet, co-reference, and dictionary matching.

2) Authors conducted relatively comprehensive experiments to show the contribution of each individual features and combination of features to the overall precision and recall.

3) Problem introduction and annotation are good too.

However, some major points need to be fixed:

1) The writing of this paper is really poor. All table references are not correct, grammar errors can be seen almost every paragraph. It is very very difficult to read. It took me hundreds of hours to understand what authors try to deliver. Let me just show examples based on the abstract: a) The first sentence is not a complete sentence. "To develop....in medical records."

b) "They infer relevant anatomical...in the record but also by other diverse..." ==> "They infer relevant anatomical entities based on both explicit anatomical expressions in the record and other diverse... "

c) "The hierarchical framework was demonstrated..." ==> "The hierarchical framework was demonstrated...in F1 comparing to ???"

many others in the paper!!!!!

2) For the annotation, authors used A3 to check (A1, A2), then obtain the coefficient. Why not A3->(A1, A2), A1->(A2, A3), and A2->(A1, A3), then obtain the average coefficient? What if there is a annotation conflict, meaning that all 3 annotators do not agree? In addition, authors claim that their golden standard is not perfect, then why you still use them to do evaluations?

3) From the experimental results, CF seems to be the smallest contribution to the precision in table 5 and table 8, then why adding CF gets a lot increase in table 6 and 9? I don't believe this result. Can you give some explanations.

In addition, some suggestions,

It would be great if the paper gives some formal definition of each concept and shows some real or toy examples in figure. They can help readers to catch the point.

Reviewer #2: The manuscript by Yan Xu et al. describes the construction of an anatomical entity recognition framework based on a machine learning algorithm. This framework can recognize not only explicit expressions of anatomical entities, but also implicit expressions such as diseases, clinical treatments, and clinical tests. The authors insisted that the recognition of the implicit expressions was important because the implicit expressions are abundant in clinical records and it is from these implicit expressions that medical experts can infer the anatomical entities described in the documents.

The framework consists of three layers of entity recognizers, all of which are based on conditional random field (CRF) models. The first layer is the multi-class CRF recognizer developed for the 2009 and 2010 I2B2 challenge; this layer recognizes entities of three semantic classes: diseases, clinical treatments, and clinical tests. The other two recognizer layers are developed in this study. One (the second layer) is for explicit anatomical expression and the other (the third layer) is for implicit expression.

For use in the training and testing of the CRF models, the authors carefully made an annotated corpus of 300 clinical records (i.e., the discharge summaries in this study). The resulting annotations include 16690 explicit anatomical entity tokens and 5564 implicit anatomical entity tokens.

The authors used the following features for the construction of the CRF models and considered the relative impact on the recognition performance using precision, recall, and F-score: baseline features (a standard set of useful features for general named entity recognition tasks), ontological features DF1 and DF2 (based on some of the representative anatomical ontologies: UMLS, MeSH, RadLex, and BodyParts3D), coreference features, and world knowledge features WF1, WF2, WF3, and HF, which is based on the dictionary constructed from the terms in Wikipedia and WordNet,

whose definition sentences contain explicit anatomical entities, for the purpose of extracting implicit anatomical entities; HF is referred to as a hierarchical feature.

This study is original and addresses an important task in processing medical documents in general. Their analytical approach seems to be sound in the sense of ordinal research on natural language processing. Therefore, this manuscript seems to warrant publication in PLOS ONE.

The main criticism I have is the lack of consideration of concrete instances of anatomical dictionaries, clinical record corpuses, annotations, and experiment results. The authors only provided several numerical tables of the precision, recall, and F-score. All the main conclusions were drawn from observation of these numerical tables. Although I know that this style is common in NLP research papers, I believe that without an investigation of concrete instances, readers cannot evaluate the relative impact of the many factors that will affect the final performance.

With only a little thought, one can list up many factors that affect the final results: data sources selection for the construction of the anatomical dictionaries, relative contribution of the (four) data sources on the performance, whether there exists some particular anatomical term in the four dictionaries that has a significant effect on the performance, the total size of anatomical dictionaries, semantic type of terms included in the anatomical dictionaries, type of clinical records, total number of clinical records and sentences which are annotated by the experts, target semantic types, the choices of machine learning algorithms, and the selection of the features for the CRF models, as well as many other factors. However, observation of the series of numerical tables yields only limited information about the impact of the factors and what entities can/cannot be recognized under the proposed framework.

Therefore, at very least, the authors should provide a part of the list of 16690 ―explicit anatomical entity tokens‖ and 5564 ―implicit anatomical entity tokens‖ with their numbers of occurrences in the corpus, because these define the problem that this manuscript is addressing.

In addition, the authors should discuss what terms in the anatomical dictionary match the annotated tokens and/or the results of the Begin/Inside/Outside (BIO) calling by the CRF model. Then some explanation of the relative impact of the framework components should be provided based on the concrete instances of matching results.

A second criticism concerns the reproducibility of this study. Although the authors wrote at the end of the abstract section, ―The resources constructed for this research will be made publicly available.‖ since the resources needed for the reproduction of this study are not provided at this time, I could not evaluate whether the results can be reproduced using the resources that the authors say will be eventually provided. I know that the authors have made a great contribution to the NLP research field, not only by introducing novel concepts, but also by providing many useful resources, including software and annotated corpuses, and so I believe that the resources that will be available to the public will be quite useful for NLP researchers, but I believe that it is quite important to meet the reproducibility criteria stated in the publication criteria of PLOS ONE

(―described in sufficient detail for another researcher to reproduce the experiments described‖), and in order to meet these criteria, I expect that the authors will need to write additional paragraphs describing in sufficient detail how to reproduce the result tables. I believe that the results have been largely affected by the content of the dictionaries and annotated corpuses constructed by the authors, and therefore, without these resources, it will be quite difficult for other researchers to reproduce exactly the results described in the tables.

Minor points

Page 8, lines 7–10

I do not understand the meaning of the numbers described in Table 4.

What is the denominator of ―Coverage of explicit named entity‖? Total number of annotated tokens in the corpus? Or number of unique tokens annotated? In typical cases, rather simple anatomical terms such as ―brain‖, ―liver‖, and ―blood‖ frequently appear in the corpus, and of course these are matched readily to the anatomical dictionaries.

Page 12, lines 7–13.

The table numbering in the main text is not consistent with the actual table numbers. (Table 4, ..., Table 9 in the main text should be Table 5, …, Table 10.)

Page 14, lines 3–5

Near the top of the DISCUSSION section, the author wrote: ―While the features based on the dictionary of anatomical entity expressions greatly improved the performance on explicit anatomical entities, they do not enhance the performance on explicit anatomical entities.‖ But the second occurrence of the word ―explicit‖ should be ―implicit‖.

6. If you would like your identity to be revealed to the authors, please include your name here (optional).

Your name and review will not be published with the manuscript.

Reviewer #1: (No Response)

Reviewer #2: (No Response)

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]