Search tips
Search criteria 


Logo of jmajLink to Publisher's site
Japan Med Assoc J. 2016 September; 59(2-3): 125–139.
Published online 2016 September 1.
PMCID: PMC5333618

Healthcare and the Roles of the Medical Profession in the Big Data Era*1


The accumulation of large amounts of healthcare information is in progress, and society is about to enter the Health Big Data era by linking such data. Medical professionals’ daily tasks in clinical practice have become more complicated due to information overload, accelerated technological development, and the expansion of conceptual frameworks for medical care. Further, their responsibilities are more challenging and their workload is consistently increasing. As medical professionals enter the Health Big Data era, we need to reevaluate the fundamental significance and role of medicine and investigate ways to utilize this available information and technology.

For example, a data analysis on diabetes patients has already shed light on the status of accessibility to physicians and the treatment response rate. In time, large amounts of health data will help find solutions including new effective treatment that could not be discovered by conventional means. Despite the vastness of accumulated data and analyses, their interpretation is necessarily conducted by attending physicians who communicate these findings to patients face to face; this task cannot be replaced by technology.

As medical professionals, we must take the initiative to evaluate the framework of medicine in the Health Big Data era, study the ideal approach for clinical practitioners within this framework, and spread awareness to the public about our framework and approach while implementing them.

Keywords: Big data era, Essence of medical care, Care range expansion, Data interpretation and traps, Initiative by professionals

What Big Data Means

In the following presentation, I will be discussing the practical challenges related to and providing examples of how Big Data should be used by physicians in clinical or general practice, ultimately leading up to my proposals on how we physicians should act as medical professionals in the Big Data era.

First, I would like to start by briefly describing what Big Data means. It seems that the phrase Big Data is also becoming popular among the mass media. However, the definition of Big Data is quite vague. As I understand it, the definition of Big Data or what it implies vary depending on the person who uses it. According to some, data that enables us to predict any future is called Big Data. Others say that any data that surpasses the processing ability and memory faculties of humans should be called Big Data.

Neither description is wrong. Maybe Big Data can anticipate anything someday. In the healthcare setting, however, even if we have data from the over 120 million people of Japan suggesting something, we physicians will always question whether we can honestly say to any given patient in front of us that this suggestion from Big Data applies to him or her. In clinical trials and research, the amount of data handled already exceeds the humanly possible capacity for processing. In fact, when we adopt this second definition of Big Data—that is, an amount of data that cannot be processed by our natural human faculties alone—we find that Big Data is already being applied in some healthcare fields. However, there are some fields of healthcare to which Big Data cannot be applied so easily.

Current Healthcare Practice Settings Are Overloaded with Information

With that in mind, I believe that current medi cal practice is already overloaded with the “currently available” data. I have several points regarding this (Fig. 1).

Fig. 1
Tasks for the medical profession in the era of complicated healthcare and information overload

For one thing, we must constantly make clinical decisions in our daily practice, but the amount of evidence, or the grounds for a decision, is increasing explosively. As already mentioned repeatedly, the amount of available data has far surpassed the human processing capacity.

As shown in Fig. 1, for example, thoroughly reviewing all published papers on echocardiography would take about 20 years for one person, assuming that that person reads 5 papers per hour. So, how can cardiologists begin practicing if they have to complete the current curriculum of 6 years of medical school and 2 years of clinical training followed by specialist training, while also reviewing all papers on echocardiography? There is just too much information.

Another thing is that the range of medical practice has been expanding. What I mean is, medical practice is no longer the simple process, if I may say so, of making a diagnosis and providing treatment. It now includes disease prevention and management, and health promotion.

One example of the expansion is the range of data linkage from clinical tests to a patient’s health information, such as body weight changes, pedometer records, and calorie intake, which many people have to constantly manage nowadays. Alternatively, locomotive data such as a walking speed can be linked to clinical data in daily life function assessment or long-term care. “Linking data” of this nature is becoming increasingly popular, and it is overloading us.

In the world of healthcare—where tasks require the collaboration of many professionals—physicians must make various judgments. However, not only is the amount of information increased by such “linking data” but also is the range of duties for which physicians are responsible. Thus, an issue of considerable significance to us physicians in the Big Data era is how to consider and manage this expansion of medicine as a medical profession.

In addition, technology is progressing very quickly. Technological development in medicine is more like each passing second and minute rather than each passing day—but how can we learn to use the technology produced? Also, how can the effectiveness of the used technology be verified? We are facing overwhelming tasks. When we consider these issues to be part of our contribution to society as a medical profession, we must revisit “what constitutes the essence of medical care” and reevaluate the significance of medical care and medical professions in society. I believe now is a good opportunity to do that, and it is also a demand of society.

How We Can Utilize Big Data

Based on these understandings, today, I think we should consider how we can autonomously utilize Big Data in medical care and how we need to take initiatives in this issue, rather than how Big Data can predict the future or how large-scale health information can change medical practice.

Medical practice has always existed alongside information, and it has developed by accumulating information. Now, as shown on the left side of Fig. 2, we are part of a cycle of technological development in medical practice and promoting those developments in society. It is time that we start from scratch in thinking about how Big Data should be used.

Fig. 2
The “problem” that clinical practitioners must consider

I would like to start with the very basics of healthcare—namely, that there are several players in the healthcare community (Fig. 3). Many of the people who are here today probably belong to the “care providers” box in the center of this figure, but, as I mentioned, the range of care provided has expanded quite considerably in recent years. Also, along with the range of care, the number of industries that people consider to be involved in healthcare is increasing.

Fig. 3
The 5 players and the power balance in the healthcare industry

The good side is that self-recording of lifestyle factors and daily habits is becoming increasingly possible. Thanks to this, data such as blood glucose levels and blood pressure can be collected more easily than before.

On the other hand, concern has arisen over the continuous appearance and disappearance of pseudo-healthcare services with poor grounds. How should we respond to this as a medical profession? As previous presentations have mentioned regarding the protection of personal information, we must think about the safe handling of medical information as professionals.

The people involved in the development of new medical technology, including pharmaceutical companies and research institutions such as medical schools, have also begun using Big Data. As I understand it, the era for considering how Big Data can be used to devise a new framework for medical practice and new healthcare services has already begun for these people as well.

Let’s take the example of developing a drug. In the past, hypotheses were made and the drug efficacy was verified only in an experimental environment under certain controlled settings. Nowadays, the trend is to verify the drug effectiveness in an environment that closely resembles the real world. Study results obtained in that fashion can actually change diagnostic criteria and even the concept of diagnosis itself. In the world of cancer treatment, the way a diagnosis is made is gradually shifting from using an individual’s conditions and test results to identify the disease to using genomic pathology and the subject’s reactivity to specific treatment methods.

Now, we name the cancer by the organ that presents malignant neoplasms. Someday, we may begin to categorize cancer types by the types of drugs that affect them, such as a type of cancer on which a gene-targeting drug works well and that happens to be on a certain organ.

With regard to the “payers,” a number of previous speakers today have talked about the topics of health insurance claim data and DPC data. The data that payers have been using when processing insurance claims to reimburse medical fees are likely the easiest entry point for Big Data application.

Although there are limits due to the nature of the data, there is a lot to learn from Big Data, as Dr. Ishikawa mentioned earlier. If we can learn what “best practice” is, promoting it in the medical practice community as quickly as possible may require the collaborative efforts of both us care providers and the payers who accumulate data.

The government would want to prioritize financial resources in order to prevent cost soaring and encourage them to pursue the financially ideal healthcare. It is highly important to know who will take the initiative in Big Data application.

The people of Japan are patients to us and beneficiaries of healthcare. As medical practitioners, we must not forget that one of our goals is providing proper medical practice and healthcare as promptly and effectively as possible to these people. That said, there are still many issues regarding the purposes and challenges of using information in the era of Big Data. For example, the genetic testing business is popular these days, but how can one separate the good ones from the bad? How and what kind of information becomes a part of Big Data is also one of the many issues that must be resolved.

The Paradigm of Healthcare Is Changing

Now that I have talked about the players in healthcare, I would like to list 3 points regarding what I consider to be minor changes in recent healthcare and medical practice (Fig. 4).

Fig. 4
The paradigm of healthcare is changing

The first point is that healthcare has changed from being “extraordinary” to being “ordinary.” Medical care in the past, either for inpatients or for outpatients, was usually practiced in a world separated from everyday life. Nowadays, however, the borders that separate everyday life from life with illness have become quite ambiguous, as the example of lifestyle-related disease shows. So, how do we go about collecting health information that closely corresponds to everyday life? Also, who would use this information, and how can it be used? In that sense, medical care is increasingly becoming a part of ordinary everyday life rather than an extraordinary part of life. Another thing to investigate, which is a slightly different topic from today’s theme, is the rationale of using an insurance system for paying healthcare practice. This should be examined in light of the fact that we live in an age wherein almost everyone receives care from a physician at least once in his/her lifetime.

The second point is that the trend of healthcare is shifting from treating disease to managing overall health. The Growth Strategies of the Government of Japan have also gradually presented this element. In modern medicine, we can prevent and/or predict disease to a certain degree, especially chronic diseases. In this era, what can data show about the timing of disease onset? Also, when should a predicted risk inferred from data be told to a potential patient, and when should his/her care start? The time to study those issues will soon arrive.

The third point is that healthcare as a concept is shifting from being a cost to being an investment. For some reason, the healthcare expenditure has long been considered a cost, or a social burden, which has always made me feel unsettled. Healthcare is nothing more or less than an investment in health. We physicians are making contributions to national health in the form of medical care, and calling it a “cost” is rather unfortunate.

On the other hand, I also believe that the medical profession should be held accountable for explaining to the Government and communities the contribution we are making using specific figures in the future. I consider this to be an additional task for physicians in the era of Big Data.

Filling the Gaps in Healthcare

So far, I have raised various issues and problems, but now I would like to turn to bridging the gap between the healthcare that is actually being practiced now and the ideal healthcare that should be practiced using Big Data, along with some specific examples (Fig. 5).

Fig. 5
Used as a tool to bridge a gap that cannot be overlooked

On Fig. 5, the right-hand side shows some of these examples. Please note that these are purely my personal opinions on what ideal healthcare comprises; they do not have official or public significance in any away.

The left-hand side of the figure depicts the healthcare that is currently practiced, such as the problem of the shortage of healthcare professionals compared to the number of patients. The shortage of physicians is an issue in many places in Japan, but how can we solve this problem of quantity using Big Data?

Another issue is the problem of quality, as I have repeatedly mentioned today. The demand and expectations of society for healthcare is only growing larger, but are we really equipped with the resources and information to meet them? Also, is the system in place capable of handling them? These are issues we have to consider.

For example, how can we incorporate health maintenance and support into current healthcare practice? Also, how can we utilize Big Data when we want to take a step beyond “free access” healthcare, which is an extremely precious asset for the Japanese healthcare system, to “best access”?

Another thing relates to the medical fee schedule—namely, whether Big Data can be used to secure remuneration and appreciation for physicians, which physicians naturally deserve as a profession for providing high-quality medical care. Also, how can we offer fair and accountable medicine? I would like to explain these issues one by one using data from a few case studies.

How We Can Promote the Best Access Approach

How can we promote health maintenance and support and the best access approach in medical care? I have been entrusted with certain health data for some statistical analyses, one of which is a retrospective cohort study on the health checkup data of 450,000 people (Fig. 6).

Fig. 6
Accessibility to medicine: who, when, and where

These results are from 150,000 people with 3 years of health checkup records, and they show that about 90% of people whose LDL cholesterol was higher than a certain level answered that they had remained untreated. Their health checkup reports all had comments specifically mentioning that their level was abnormal and that they should seek medical consultation; however, only around 10% of them actually visited physicians.

The following year, their LDL-c level was still high, but again only 10% sought medical consultation. The same goes for the year after that. Health checkups are very popular in Japan and people see their health data every year—and yet, only a small proportion of people actually access medical care. Even a well-known risk factor such as cholesterol goes untreated in 90% of the people who should be treated.

So, those of us who are physicians involved in clinical practice must think of how we can provide available information to patients in a way that prompts them to act. We must also think of who should be telling them. While a single personal computer, which almost everyone has nowadays, can do the relevant data computation, it cannot tell the user how to communicate the results to others.

A Gap between What the Data Show and the Reality of Seeking Medical Consultation

A case study of diabetes is shown on Fig. 7; the Y-axis shows the hemoglobin-A1c (HbA1c) level, which corresponds to severity of diabetes. The higher it is on the Y-axis, the more severe the diabetes is. The horizontal dotted line in red is the severity standard of the Japan Diabetes Society. The computation formula is slightly different now, but this line basically indicates that the score is not very good for a diabetic patient—namely, a person is not managing his diabetes well if his/her score is above this line.

Fig. 7
Status of hospital/clinic visits for poorly controlled diabetes

You can see numerous blue circles and red dots; each represents one person. The size of each blue circle represents the medical expenses related to his/her diabetes for a year. Red dots indicate no expense, meaning that the person has not visited any doctor. The X-axis is the age. You can see that a certain number of people go untreated even though their score is high enough to indicate severe diabetes and they are above a certain age. More specifically, about 1/3 are red dots.

When physicians see these data, I believe that they would naturally develop a sense of crisis that this gap between what the data show and the reality of seeking medical care must be somehow filled.

Now, let’s continue to the drug treatment data based on response rate. The data shown on Fig. 8, which are not from Japan, suggest that the response rate was unexpectedly low. Of course this does not apply to all cases, but I believe you all have experienced a difference in efficacy among drugs, such as “Company A’s cough medicine works better than Company B’s,” or “this brand of painkiller works better than that one.” However, even analgesics only work 80% of the time overall. Although it varies by the type of cancer, the data show that cancer drugs only work in 1 out of 4 cases.

Fig. 8
The tools necessary for medical practice are also insufficient

One dilemma for physicians in clinical practice is that patients simply will not come to clinics or hospitals. Even when people do come to a clinic or hospital, the data we have available on the response rate tell them that we have only these weapons with low response rates—so, what should we do? This is one of the many things we must consider in the upcoming era of Big Data.

Making Use of Data in the Health Management of Local Communities

Next, I will talk about how we can use data for value-based evaluation and resource allocation as part of the universal health coverage to protect health.

Using the data I have been entrusted from the insurer from the previously mentioned diabetes case, I can create a status map of treatment and access status (Fig. 9). Please note that the data shown on the figure do not really belong to the community shown on the map; I am simply using a map I found from the Internet homepage of this community. So, please take this merely as an example.

Fig. 9
Data for self-management of health and community development

The data show the number of patients across areas, including not only those who are actually receiving care but also those who should receive care but remain untreated, as well as those who are potential patients. When such information becomes available, local medical associations will realize that there are many people with high LDL cholesterol who do not seek treatment in their areas, and thereby can develop a community healthcare plan, such as adding 1 more day for outpatient care by specialists.

If local medical associations can show the effects of such effort in real numbers, it will help convince local governments and residents that the best access to medical care has been improved and that the health level of the community as a whole has been elevated. This particular example can also help convince people that healthcare is not a social cost but a public investment.

How to Approach Medicine in the Big Data Era

Let’s move on to the next topic: fair and accountable medicine.

I became a physician in 1999. Many lawsuits on medical malpractice were taking place then, and my generation was exposed to public distrust toward medicine. I believe professional accountability will only become increasingly significant for physicians from now on.

Until now, the accountability of medical evidence began with case reports, as shown in Fig. 10. Experts gathered at the meeting of a medical society, and the most respected ones shared their case studies and rules of thumb with the rest. This is how medicine gradually grew, and I believe everyone will probably agree with that.

Fig. 10
The technology and concept of obtaining new medical knowledge

Then, the idea of evidence-based care sunk in, and treatment and its effect in clinical research or trials, wherein real-world conditions were simulated, began to be highlighted at a certain experimental level. Guidelines were developed in this fashion by medical societies, wherein they aggregated experimental results that can be called scientific expectations and efficacy. This approach lasted for years.

In the era of Big Data, however, I believe we can come up with a better concept of healthcare—or best practice—by viewing it as collective knowledge and collecting what is really happening in society, how well guidelines are made known or observed, and how various medical institutions actually carry out the practice of medicine. Those experimental values that were once considered so-called “efficacy (or potential effectiveness)” will actually be regarded as “effectiveness (or practical effectiveness).”

For example, let us assume that there is a drug that is supposed to work on 60% of patients according to some studies, and the drug was given to 5 to 6 people but produced no effect at all. Should the 7th patient be given the same drug? This is always a dilemma for the medical profession. Big Data would enable us to see that the studies with good results were rather atypical, and that the overall opinion among clinical practitioners is that this drug does not work too well—or vice versa. When such a time comes, the current approaches for guidelines and clinical research designs will begin to change. Approaches for selecting medications and the speed of spreading best practice may change as well.

Meanwhile, medical professionals must work on how to explain those medical findings obtained from Big Data to the patients face to face, which is a never-ending mission for us. No matter how large the Big Data grows or how much technology advances, it is the duty of medical professionals to carry out this task.

So, the technology of Big Data can be a very powerful tool in medicine. On the other hand, it will at the same time force us to seriously consider what the purpose of medicine is—and what the ultimate responsibilities of medical professionals are.

Gaps in Medical Practice That Must Not Be Overlooked

Let me elaborate on the topic of accountability a little further. Figure 11 shows 3 examples of gaps between the reality and ideals in medical practice that were discussed around 10 or more years ago.

Fig. 11
Gaps in medical practice that must not be overlooked

Shown on the left is the overuse of medical care, or overcare, to put it simply. The middle is the underuse of care, meaning that care is needed but is not sufficiently provided. This could simply be the case that a patient right in front of you deserves more cordial care, or the number of outpatients in a certain area is beyond the local capability and therefore outpatients do not receive enough care or suffer from poor accessibility. Either case can be considered underuse.

Then, there is the misuse of medicine in the first place—that is, providing inappropriate care. This is not a main topic of my presentation today, but I believe such misuse will likely be reduced to a certain extent if Big Data or IT is applied in clinical/hospital practice. It is also possible that what was once considered the standard treatment will be proven ineffective and a new treatment is established, which means that such misuse will simply become a case of the knowledge being outdated in the future.

Then, how should we update our knowledge? As I mentioned at the beginning, one cannot spend 20 years just doing nothing but reading papers. So, how can we provide timely updates for truly important information? This is another aspect of Big Data application that must be taken into consideration if we are to fulfill our accountability or responsibility of verification in medicine that the upcoming era will require us to.

Physicians’ Competence to Be Tested

Figure 12 is about a slightly different topic; it is an extreme example of how medical findings obtained from data can be used in practice. “Is it possible for this group of 4 persons to cross this river?” is the question.

Fig. 12
Numbers can play tricks and mislead one’s decision-making

Data in textbooks or papers are described like this: The average height of this group is 1.7 m, and the average depth of this river in front is 1.6 m. So, one would naturally think that everyone should be able to cross this river. Similarly, we naturally employ a system of medicine based on “this treatment is good on average.” There is nothing wrong with this approach. However, if this group consists of 2 persons who are 1.9 m and 2 persons who are 1.5 m, it poses a problem. It is possible that 2 may die while 2 will successfully cross the river. Now, what if the deepest spot of the river was 2.2 m? Then, everyone would die. Or, maybe, everyone could cross the river if they stayed in the shallow areas, which has a depth of 1.3 m.

When we take a case-by-case approach in decision-making instead of using average values from existing clinical research, these kinds of “details”—called the “grain size”—can make a considerable difference. As the grain size decreases, the chance of coming up with a clinical pathway that no one could imagine or questioning the original diagnosis increases. In other words, the medicine that was previously trusted may be proven untrustworthy. If it actually happens, how we inform patients is a big challenge for those of us in clinical practice.

Let’s say that the deepest spot is 2.2 m and the shallow areas are 1.3 m, but you do not know the safe route. One approach is to leave the decision-making to the individuals; you could inform a 1.9 m person that the deepest spot is 2.2 m and ask if he/she is willing to try to cross. Another approach is to advise the person to not cross the river since he/she will be carried away if the person happens to cross at the deepest spot.

In the era of Big Data, wherein medicine will use information to predict likely outcomes, how physicians inform patients about the new information and how we make our decisions will become very important points. We would need to bear these points in mind in our profession when using this powerful tool called Big Data.

Changes in Technology Development by Big Data

Now, I would like to stop talking about the demand side of the story, and talk more about technology advancement (Fig. 13).

Fig. 13
Changes in technology development by Big Data

When a new drug is developed to treat a disease, a clinical test would be conducted with a certain number of patients (or subjects); say, for example, that 2 out of 6 patients did not show much improvement. We would then conclude that it is probably a good drug since it worked for 2/3 of the subjects, and we would then tell those patients in green that this drug works well. In the Big Data era, however, the number of cases to deal with will instantly increase.

When the proportion of patients that this drug does not work on exceeds a certain number, a new drug must be developed to provide a cure for those who could not previously be treated—this will likely become the mainstream of new drug development in the future. Earlier, I mentioned that there will be a case in which the original diagnosis will be questioned by Big Data; this way of new drug development is an example.

In the upcoming era of Big Data, what was previously believed to be the same disease could be shown to be different diseases based on patients’ responsiveness to a given treatment. The advancement of medications and medical equipment is not really today’s theme, but I want to point out that it will enable a more pinpointed approach in treatment and increases the significance of atypical cases.

Responsibilities Entrusted to the Medical Profession in the Big Data Era

To wrap up my presentation today, when databases are linked and a large volume of data become available as Big Data, medical professionals have many things to consider regarding the success or failure of Big Data application. Notably, those things to consider could be more serious than the use of Big Data itself (Fig. 14).

Fig. 14
Definition of “medical profession” can influence data utilization

As the 2 previous lecturers mentioned, there are many challenges in the Big Data era at different stages, such as how to obtain certain data and how to analyze and use that data. For the medical profession, one challenge is the potential traps in data interpretation—I showed this in the river-crossing example. A dataset itself has certain limitations to begin with, and therefore we cannot simply use whatever Big Data suggests as the golden rule.

We also need to consider the medical system we have fostered up until now and investigate the ways in which Big Data can contribute to health for the population and to medical practice for social development by using the appropriate grain size, knowledge, range of application, timeliness, and manners of utilization.

The last point I want to make is that we will have to take initiative as clinical practitioners or medical professionals in the Big Data era (Fig. 15). Big Data will make what was previously invisible visible and relate things that were believed to be unrelated before. Furthermore, Big Data will allow us to predict the outcome of a clinical course from a clinical diagnosis, and it will change the balance between prediction and decision in medical practice.

Fig. 15
An era in which medical professionals are asked to take the initiative

On the other hand, someone will have to bridge the gaps between human data and the living human in the medical practice of this era. This responsibility is, and always will be, entrusted to medical professionals.

No matter how good a data analysis is, the so-called human traits—originality, pride, laziness, mistakes, etc. —may never become fully encoded as data. Of course, those people in IT or data management may claim that those human traits can be predicted from the data. Nevertheless, how to encode human qualities as data remains a challenge from a technical perspective as well. That being the situation, I believe there are many things that medical professionals must do.

We have no choice but to use Big Data. As we have seen in the last several years, we live in a time wherein the volume and nature of data change quickly, as does the development of methods of analysis and application. There are areas in which we must also change in response to these changing times.

How do we define future medicine now? How do we approach Big Data application as action guidelines for the medical profession? The time has come that we must examine those issues and send out messages related to these issues of our own accord. What we need now or soon is a starting point, a chance to use Big Data as a tool. I would like to conclude my talk with these remarks. Thank you for your attention.


*1 This article is based on the lecture presented in Japanese at the Japan Medical Association Research Institute (JMARI) Symposium on “Current Status and Future of Health Big Data in Japan” held on February 12, 2015. Unless otherwise stated, the figures in the article are originally made by the author in Japanese and translated for JMAJ.

Articles from Japan Medical Association Journal : JMAJ are provided here courtesy of Japan Medical Association