Showing posts with label false negative. Show all posts
Showing posts with label false negative. Show all posts

Monday, December 14, 2009

Human resolution or disambiguation -- Integrating human workflow in BPEL processes -- Errors in statistical matching of attributes to an individual


To locate health records, statistical matching attempts to string together enough identifying information about an individual to substitute for a unique personal identifier. It involves matching attributes, such as last name, first name, birth date, address or zip code, and gender, and it may use medical-record numbers and all or part of the Social Security number.

The problem with personal attribute keys such as name and address is that they are usually not unique to the individual, change over time, and are often entered into different systems in different formats. And data-entry errors, such as misspellings, add to the difficulties with this type of key. Repeated collection, distribution, storage, and use of these data also represent an important identity-theft risk.

Statistical matching can attempt to correct for some of these changes and errors: The most straightforward process is to tag all of the near matches for human resolution, or disambiguation. Such disambiguation imposes significant costs and operational inefficiencies, particularly if the physician must resolve the ambiguities. Advanced approaches “score” matches on “closeness” to the input set. Those with a high score may be accepted as a match. However, all such efforts are subject to the probabilistic errors inherent in statistical matching systems.

As discussed in earlier posts, there are two types of errors - false positives, in which two different persons’ records are declared to be a match, which can lead to such errors as the wrong patient’s health data being obtained; and false negatives, in which two records for the same person are thought to relate to different people, leading to such consequences as some of the patient’s data being excluded. Both of these errors can lead to serious medical errors, waste (e.g., repeats of tests or the wrong tests), and considerable deviation from the promises of continuity and quality of care postulated for a connected digital health care system.

Disambiguation is a process through which multiple potential identification matches are further parsed until the patient can be matched with his or her data with sufficient certainty to allow for the delivery of a health service with reasonable confidence. The complexity of disambiguation varies according to factors such as the number of potential matches and the type of information available for further analyses. When sufficient digital data are not available to further differentiate potential matches, automated disambiguation may not be possible and may require human involvement. This last case will be the focus of the rest of this post and my next post.

Disambiguation entails implementing significant new workflows and may require substantial time and resources. When human involvement is required, many of the potential benefits of automation are lost. For example, at the point of care, disambiguation is often done by asking the patient further questions regarding personal characteristics and/or health care history. In some situation, disambiguation may not be possible, as when the patient is not present and information needed to further facilitate matching may not be accessible.

I will show how one vendor, Oracle, implements human tasks that can provide workflows such as those identified above. However, these Oracle services can be accessed by applications created with development tools from other vendors (my discussion will use Microsoft Visual Studio).

Introduction to BPEL and Human Workflow

Business Process Execution Language (BPEL), one of the key technologies for Service Oriented Architecture (SOA), has become the accepted mechanism for defining and executing business processes in a common vendor-neutral way. Apropos of this discussion, business processes often require human interactions as well.

User Interaction in Business Processes

BPEL business processes are defined as collections of activities that invoke services. BPEL doesn't make a distinction between services provided by applications and other interactions, such as human interactions. And that's important since real-world business processes often integrate not only systems and services but also users. User interactions in business processes can be simple, such as approving certain tasks or decisions, or complex, such as delegation, renewal, escalation, nomination, or chained execution . . . and matching an ID with an individual.

Task approval is the simplest and probably the most common user interaction. In a business process for opening a new account, a user interaction might be required to decide whether the user is allowed to open the account. If the situation is more complex, a business process might require several users to make approvals, either in sequence or in parallel. In sequential scenarios, the next user often wants to see the decision made by the previous user. Sometimes, particularly in parallel user interactions, users aren't allowed to see the other users' decisions. This improves the decision potential. Sometimes one user doesn't even know which other users are involved - or whether any other users are involved at all.

A common scenario for involving more than one user is workflow with escalation. Escalation is typically used in situations where an activity doesn't fulfill a time constraint. In such a case, a notification is sent to one or more users. Escalations can be chained, going first to the first-line employees and advancing to senior staff if the activity isn't fulfilled.

Sometimes it's difficult or impossible to define in advance which user should perform an interaction. In this case, a supervisor might manually nominate the task to other employees; the nomination can also be made by a group of users or by a decision-support system.

In other scenarios, a business process may require a single user to perform several steps that can be defined in advance or during the execution of the process instance. Even more complex processes might require that one workflow is continued with another workflow.

User interactions aren't limited to approvals; they can also include data entries or process management issues, such as process initiation, suspension, and exception management. This is particularly true in long-running business processes, where, for example, user exception handling can prevent costly process termination and related compensation for those activities that have already been successfully completed.

As a best practice for human workflows, it's usually not wise to associate human interactions directly with specific users; it's better to connect tasks to roles and then associate those roles with individual users. This gives business processes greater flexibility, letting any user with a certain role interact with the process and enabling changes to users and roles to be made dynamically.

BPEL and User Interaction

So far we've seen that user interaction in business processes can get quite complex. Several vendors today have created workflow services that leverage the rich BPEL support for asynchronous services. In this fashion, people and manual tasks become just another asynchronous service from the perspective of the orchestrating process and the BPEL processes stay 100% standard.




In my next post, I’ll talk about some of the specifics of how you might implement a BPEL process that includes human workflow/tasks for disambiguation using tools such as Oracles JDeveloper and Microsoft Visual Studio. The next two figures are meant to give you a preview of that discussion.





{ click the figures for a larger view }


Click here for more on Microsoft Visual Studio 2010

Click here for more on Oracle BPEL and Human Workflow


Monday, December 7, 2009

Human Resolution or Disambiguation -- False Positive and False Negative Identification: Heath Care and Information Technology Perspectives


Disambiguation of IDs is the process of resolving multiple potential matches into a match with the correct person. In general, statistical matching algorithms are likely to require substantially more-frequent disambiguation compared to that required by a system that uses theoretically perfect universal IDs; often, disambiguation is done by human intervention. Such disambiguation imposes significant costs and operational inefficiencies, particularly if, for example, a physician must resolve the ambiguities.

Note 1: Many of the efficiency and safety benefits theoretically possible with health information technology (HIT) systems depend on eliminating such human involvement and its concomitant slowness, expense, and propensity for error.

Note 2: What follows applies to IDs in general, even though I’ve chosen the healthcare industry for much of this discussion.

Disambiguation sometimes entails implementing significant new workflows that may require substantial time and resources. When human involvement is required, many of the potential benefits of automation are lost. For example, at the point of care, disambiguation is often done by asking the patient further questions regarding personal characteristics and/or health care history.

The potential for error in the statistical matching methods (see my December 1 post on unique patient IDs) has important safety implications, which are a chief concern for many in the health care profession. Two types of errors are involved in statistical matching: false positives, in which there is a link to the wrong patient’s records, and false negatives, in which not all of a patient’s records are found. A graphic representation of these types of errors and of how they relate to the probabilities and threshold for matching is shown in the figure below.




The horizontal scale shows the score of a particular match. As more and more attributes match and as the match is weighted by its score, or value, the higher is the probability that the patient is correctly matched to that record. A low score indicates a low probability of match (and a high probability that it does not match). It is possible to use a threshold above which the record is assumed to match and below which it is not assumed to match, which leads to the shaded areas above and below the threshold.

The area shaded to the right of the threshold is the region corresponding to false positives, or picking up the wrong patient’s records. The shaded area to the left of the threshold is the region of false negatives, or the records of the patient that are not picked up because of some non-matching personal attributes. Setting a balance between the two types of errors involves tuning.

Another approach illustrated in this figure is to define a region of ambiguity within which possible matches are tagged for human resolution, or disambiguation. Whether matching uses a single threshold or two thresholds, it is not possible to avoid encountering false-positive and false-negative matches. Adjusting the threshold or thresholds can result in a different proportion of false-positive and false-negative errors, but cannot be used to eliminate them because they result from the inherent characteristics of the population that lead to the two S-shaped curves.

As stated above, many end-to-end business processes require human interactions with the process.

Task Assignment and Routing

Human workflow supports declarative assignment and routing of tasks. In the simplest case, a task is assigned to a single participant (user or group). However, there are many situations in which more detailed task assignment and routing is necessary (for example, when a task must be approved by a management chain or worked and voted on by a set of people in parallel, as shown in the figure below). I’ve chosen tools in the Oracle SOA Suite to illustrate (in the figures below) human workflow that can provide declarative pattern-based support for such scenarios.





I’ll briefly elaborate here with an introduction to human workflow and continue the discussion in my next post, where I'll talk about how you might implement such a system.

Participant Type

In simple cases, a participant maps to a user, group, or role. However, workflow supports declarative patterns for common routing scenarios such as management chain and group vote. The following participant types are available:

Single approver

This is the simple case where a participant maps to a user, group, or role. Since at least one human being is involved, much more than his or her looking at a monitor screen and clicking with a mouse is involved.

For example, a vacation request is assigned to a manager. The manager must act on the request task three days before the vacation starts. If the manager formally approves or rejects the request, the employee is notified with the decision. If the manager does not act on the task, the request is treated as rejected. Notification actions similar to the formal rejection are taken.

Parallel

This participant indicates that a set of people must work in parallel. This pattern is commonly used for voting.

For example, multiple users in a hiring situation must vote to hire or reject an applicant. You specify the voting percentage that is needed for the outcome to take effect, such as a majority vote or a unanimous vote.




Serial

This participant indicates that a set of users must work in sequence. While working in sequence can be specified in the routing policy by using multiple participants in sequence, this pattern is useful when the set of people is dynamic. The most common scenario for this is management chain escalation, which is done by specifying that the list is based on a management chain within the specification of this pattern. More on routing later.

FYI (For Your Information)

This participant also maps to a single user, group, or role, just as in single approver. However, this pattern indicates that the participant just receives a notification task and the business process does not wait for the participant's response. FYI participants cannot directly impact the outcome of a task, but in some cases can provide comments or add attachments.

Readers who are interested in learning more about the subject of human resolution or disambiguation in an otherwise automated system might look at the following two books, while waiting for my next post.