BACKGROUND: Using differential display (DD), we discovered a new member of the serine protease family of protein-cleaving enzymes, named protease M. The gene is most closely related by sequence to the kallikreins, to prostate-specific antigen (PSA), and to trypsin. The diagnostic use of PSA in prostate cancer suggested that a related molecule might be a predictor for breast or ovarian cancer. This, in turn, led to studies designed to characterize the protein and to screen for its expression in cancer. MATERIALS AND METHODS: The isolation of protease M by DD, the cloning and sequencing of the cDNA, and the comparison of the predicted protein structure with related proteins are described, as are methods to produce recombinant proteins and polyclonal antibody preparations. Protease M expression was examined in mammary, prostate, and ovarian cancer, as well as normal, cells and tissues. Stable transfectants expressing the protease M gene were produced in mammary carcinoma cells. RESULTS: Protease M was localized by fluorescent in situ hybridization analysis to chromosome 19q13.3, in a region to which other kallikreins and PSA also map. The gene is expressed in the primary mammary carcinoma lines tested but not in the corresponding cell lines of metastatic origin. It is strongly expressed in ovarian cancer tissues and cell lines. The enzyme activity could not be established, because of difficulties in producing sufficient recombinant protein, a common problem with proteases. Transfectants were selected that overexpress the mRNA, but the protein levels remained very low. CONCLUSIONS: Protease M expression (mRNA) may be a useful marker in the detection of primary mammary carcinomas, as well as primary ovarian cancers. Other medical applications are also likely, based on sequence relatedness to trypsin and PSA.