PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of cancersMDPI Open Access Journalshomethis articlethis journalinstructions for authorssubscribecancers
 
Cancers (Basel). 2009 December; 1(1): 23–28.
Published online 2009 December 15. doi:  10.3390/cancers1010023
PMCID: PMC2882109
NIHMSID: NIHMS165500

N1,N3-Dimethyl-N1,N3-bis(phenylcarbonothioyl) Propanedihydrazide (Elesclomol) Selectively Kills Cisplatin Resistant Lung Cancer Cells through Reactive Oxygen Species (ROS)

Abstract

Cisplatin is an important chemotherapeutic agent in lung cancer treatment. The mechanism of drug resistance to cisplatin is complex and historically has been difficult to overcome. We report here that cisplatin resistant lung cancer cell lines possess high basal levels of reactive oxygen species (ROS) when compared to normal cells and their parental cell counterparts. These resistant cells also have low thioredoxin (TRX) levels which may be one of the contributory factors to high ROS. N1,N3-dimethyl-N1,N3-bis(phenylcarbonothioyl) propanedihydrazide (elesclomol), an agent known to increase ROS is selectively toxic to cisplatin-resistant cells, while sparing normal cells and the parental counterpart. The cytotoxic effect of elesclomol in resistant cells is accompanied by further decreases in TRX and glutathione (GSH) antioxidant systems, while opposite results were found in parental cells. The ID50 of elesclomol in cisplatin-resistant cells ranged from 5–10 nM, which is well within clinically achievable ranges. N-Acetylcysteine (NAC), which is known to neutralize ROS, can abolish the cytotoxic effect of elesclomol, suggesting that the cytotoxic effect results from increased ROS. Overall, our data suggest that elesclomol selectively kills cisplatin-resistant tumor cells through increased ROS. This agent may hold potential to overcome cisplatin resistance and should be further explored to treat patients who have failed cisplatin therapy.

Keywords: lung cancer, drug resistant, cisplatin, ROS, elesclomol

1. Introduction

Cisplatin resistance often involves multiple mechanisms including drug transport, inactivation of drug, and cell type dependent DNA repair, thus making resistance to cisplatin difficult to overcome. One of the known pharmacologic actions of cisplatin is disruption of the redox system through inhibition of thioredoxin reductase (TrxR) [1,2]. TrxR is an important antioxidant system, similar to glutathione reductase (GshR), that converts H2O2 to water and O2 [3]. Thus, inhibition of TRX by cisplatin leads to increased ROS resulting in further damage of DNA and subsequent cell death [4,5].

It is well accepted that ROS can have either prosurvival or prodeath effects, depending on the amount and the effectiveness of the antioxidant system. In this report, we have found that cisplatin resistant cell lines harbor high basal levels of ROS which could be exploited as a therapeutic target. N1,N3-dimethyl-N1,N3-bis(phenylcarbonothioyl)propanedihydrazide (elesclomol) is a novel agent derived from a phenotypic screen for small molecules with potent proapoptotic activity through the induction of oxidative stress [6]. Treatment with elesclomol results in rapid generation of ROS and the induction of genes transcription that are responsible for oxidative stress [6]. In this study, we evaluate the antitumor effect of this agent in a panel of small cell lung cancer (SCLC) and non small cell lung cancer (NSCLC) cisplatin resistant cell lines.

2. Results and Discussion

2.1. Cisplatin Resistant Cell Lines Lead to Increased ROS Levels

We have studied two pairs of small cell lung cancer (SCLC), two pairs of non small cell lung cancer (NSCLC) and their cisplatin resistant counterparts as well as normal fibroblast cell line, BJ-1 (see Experimental section for ID50 response). Our results showed that ROS is increased in all cisplatin resistant cell lines when compared to their parental counterpart cells. In addition, we also evaluated the level of ROS in primary tumor cell cultures from a patient who failed cisplatin (cell line LE). Since no parental cells were available for comparison with LE cell line, normal cells BJ-1 were used as control. Similarly to cisplatin-resistant cell lines, primary tumor cells were also found to have high levels of ROS as compared with BJ-1 (Figure 1).

Figure 1
Flow cytometry analysis of ROS in various lung cancer cell lines detected by DCF-DA probe. The cisplatin resistant variants (R) are shown to possess higher basal levels of ROS when compared to parental cells counterpart (P). Each histogram is representative ...

To further confirm that cisplatin resistant cell lines possess high basal ROS levels, we assayed H2O2 levels in the cell line pairs and the cisplatin-resistant primary culture using H2O2 specific APFB probe. The results are shown in Figure 2. All cisplatin-resistant cell lines clearly expressed high levels of H2O2 when compared to their parental cell counterparts. Again, normal BJ-1 cells were used for comparison with the primary cisplatin-resistant cells. Our data strongly indicate that cisplatin resistant cell lines possess intrinsically high basal ROS levels than their parental counterparts and normal cells.

Figure 2
Flow cytometry analysis of H2O2 in lung cancer cell lines detected by APFB probe indicated that there are 2 populations of cells: R1 (low H2O2 production) and R2 (high H2O2 production). Resistant cells SCLCSR2, NSCLCSC, and LE have more R2 population ...

2.2. Growth Inhibitory Effect of Elesclomol on Cisplatin Resistant Cell Lines

Since cisplatin resistant cell lines expressed substantially high level of ROS, we hypothesized that compounds that generate ROS may increase the ROS levels beyond the threshold of tolerability in cisplatin-resistant cells resulting in cell death while sparing normal cells. To confirm this concept, we have treated both parental and resistant cells with elesclomol, a compound known to increase ROS for 48 h [6]. The data are shown in Figure 3.

Figure 3
ID50 dosage of elesclomol in parental and cisplatin resistant lung cancer cell lines. (SD of three experiments, average p values < 0.005; parental vs. resistant). BJ-1 was used as control.

Our results clearly demonstrated that elesclomol selectively kills cisplatin resistant cells. The growth inhibitory dosages (ID50) of elesclomol in cisplatin resistant cell lines are 4–10 times lower than their parental cells counterpart. We have also measured cell death and caspase activity by flow cytometry. Cell death resulting from the addition of 10 nM of elesclomol ranged from 60%–80% in cisplatin resistant cells and only 5%–10% in parental cells using the same dosage and only 2% cell death in BJ-1. A representative assay from each cell pair is shown in Figure 4.

Figure 4
Elesclomol greatly stimulates caspase activity and cell death in cisplatin resistant cell line. Representative groups of SCLC and NSCLC were treated with 10 nM of elesclomol for 48 h and followed by caspase activity probe FMK-Fluoroscein. The percentage ...

2.3. Elesclomol Increased Levels of ROS and H2O2 in Cisplatin Resistant Cell Lines

To determine whether ROS is increased after treatment with elesclomol, we have treated these two pairs of SCLC, NSCLC, and BJ-1 cells with 10 nM of elesclomol for 24h. Despite the short period of exposure time, the ROS levels are increased more in resistant cells after elesclomol treatment while no significant change occurred in parental cells or normal cells, BJ-1 (Figure 5).

Figure 5
Flow cytometry analysis of ROS in a pair of SCLC and NSCLC cell lines detected by DCF-DA probe. 10 nM of elesclomol for 24 h greatly induced intracellular ROS levels in the resistant (R) cell lines SCLCSR2 and NSCLCSC whereas in parental cell lines (SCLC1 ...

To further confirm this finding, we have also assayed H2O2 production pre and post elesclomol treatment. The results are shown in Figure 6. Similar to ROS, H2O2 production also increased after treatment with elesclomol in cisplatin resistant cells. In contrast, H2O2 production is decreased in parental cells and normal cells when treated with this ROS inducing agent. These results are interesting since parental cells and normal cells may have the ability to effectively turn on the antioxidant system to eliminate H2O2 accumulation.

Figure 6
Flow cytometry analysis of H2O2 in lung cancer cell lines detected by APFB probe indicated that there are 2 populations of cell: R1 (low H2O2) and R2 (high H2O2). The levels of H2O2 decreased in parental lines (SCLC1) and normal cells (BJ-1) when treated ...

To further determine whether increasing ROS leads to cell death, we have cotreated a known antioxidant agent N-acetylcysteine (NAC) with elesclomol and found that 0.1 mM of NAC can reverse the growth inhibitory effect of elesclomol as well as caspase-3 cleavage in cisplatin resistant cells (Figure 7 and Figure 8). Thus, our data further validate that elesclomol selectively kills cisplatin resistant lung cancer cells through increasing ROS.

Figure 7
0.1 mM of N-acetylcysteine (NAC) is able to reverse the growth inhibitory effect of elesclomol in cisplatin resistant cell lines (R).
Figure 8
Immunoblot analysis of cleaved caspase-3 on SCLCSR2. Treatment with 10 nM of elesclomol for 72 h induced caspase 3 cleavages (arrow) which can be reversed with cotreatment of 0.1 mM of NAC (72 h).

2.4. The effect of Elesclomol on Antioxidant System

It has been previously reported by several laboratories that cisplatin as well as other alkylating agents inhibit TrxR [1,7,8]. This inhibition results in lowering TRX levels and thereby increased ROS levels leading to further DNA damage and cell death. Therefore, we investigated the levels of TRX/TrxR before and after treatment with elesclomol. Interestingly, the TRX/TrxR is lower in all cisplatin resistant cell lines and further reduced after treatment with elesclomol (Figure 9). In contrast, the basal TRX/TrxR levels in parental cells are higher and further increased after elesclomol. The question remains why TRX/TrxR is lower in cisplatin resistant cells and further decreased after elesclomol. It is possible that repeated exposure to cisplatin results in lower TRX due to the fact that cisplatin is known to target TrxR. Thus, decreasing the amount of TRX while increasing the amount of other antioxidant system(s) may be a means to evade the cytotoxic effect of cisplatin.

Figure 9
Immunoblot of lung cancer cell lines before and after treated with 10 nM of elesclomol for 24 h. TRX and TrxR activity are decreased in resistant lines and further decreased when treated with elesclomol. There is no significant change in catalase level. ...

Why elesclomol further decreases TRX activity is not known; however, the answer may lie with the level of SOD1, since SOD1 is also attenuated after elesclomol treatment (to be discussed later).

We then investigated the other two important antioxidant systems, catalase and glutathione (GSH) levels, before and after exposure to elesclomol. As shown in Figure 9 and Figure 10, the basal levels of catalase are similar in the parental and resistant cells and showed no significant change upon exposure to elesclomol. However, we have found that the basal levels of GSH are higher in cisplatin resistant cells, but upon exposure to elesclomol, the levels decreased. Conversely, although the basal levels of GSH are lower in parental cells, these cells are able to activate GSH system after exposure to elesclomol (Figure 10). Overall, our data suggest that parental cells are able to activate the antioxidant system readily upon exposure to oxidative stress induced by elesclomol and are thereby able to evade apoptosis at doses that kill resistant cells. It is not clear why cisplatin resistant cells can not increase intracellular GSH under elesclomol treatment. It is possible that cisplatin resistant cells may already express maximal levels of GSH in order to compensate for high basal ROS level and are unable to increase these levels any further.

Figure 10
Total cellular of GSH in lung cancer cell lines treated with 10 nM of elesclomol for 24 h. Note: After treatment with elesclomol, the levels of GSH are decreased in all of the resistant cell lines (R) (Mean SD of three experiments).

Another important factor that may explain high basal levels of ROS in cisplatin resistant cells and its collateral sensitivity to elesclomol is superoxide dismutase (SOD). Consequently, we investigated SOD1 in this panel of cisplatin sensitive and resistant cells, both before and after elesclomol treatment. SOD is a metalloenzyme that is important to capture superoxide (O2) and convert O2 into H2O2 and oxygen. H2O2 is then detoxified by TRX, catalase and/or GSH. Thus, the amount of SOD1 present is crucial for the cellular antioxidant defense mechanism. Figure 9 shows that the basal levels of SOD1 are much higher in cisplatin resistant cell lines; however, upon exposure to elesclomol, the levels of SOD1 increased only in parental cells but decreased in cisplatin resistant cells. In contrast, when compared to parental cell lines, TRX/TrxR is lower in resistant cells and the levels further reduced when treated with elesclomol (Figure 9).

We have already shown that cisplatin resistant cells have high basal levels of ROS/O2. Thus, they require high levels of SOD1 to capture ROS/O2. However, upon exposure to elesclomol, SOD1 is reduced, but ROS and H2O2 continue to rise which reflects high levels of oxidative stress. The underlying mechanism(s) for this is not clear. It is possible that cisplatin resistant cells may not be able to effectively remove H2O2 due to inability to increase TRX and GSH expression. In order to avoid the cytotoxic effect of excessive intracellular H2O2, the cisplatin resistant cells will generate less intracellular SOD1 in an attempt to decrease H2O2 formation upon exposure to elesclomol. However, despite decreasing SOD1, the amount of H2O2 generated is still too excessive and ultimately lead to cell death. In contrast, parental cells can turn on SOD1 readily to capture ROS/O2 and the resulting H2O2 is rapidly detoxified by the antioxidant system upon exposure to elesclomol.

3. Experimental Section

3.1. Cell Lines

SCLC1&SCLCSR2, SCLCB&SCLCBC, NSCLCS&NSCLCSC have been previously characterized [9,10,11,12]. NSCLCF was derived from metastatic adenocarcinoma of the lung to the brain. The NSCLCFC line was established from NSCLCF by repeated exposure to cisplatin. Cell line LE was established from the pleural fluid of a patient who failed cisplatin. An immortalized normal fibroblast cell line (BJ-1) was obtained from Clontech (Mountain View, CA, USA). Note: SCLCSR2 exhibits 20 fold resistances to cisplatin, NSCLCSC exhibits 7-fold resistance to both cisplatin and carboplatin. SCLCBC exhibits 10-fold resistance to cisplatin and carboplatin, NSCLCFC exhibits 8-fold resistance to cisplatin and carboplatin.

3.2. Compounds and Antibodies

N1,N3-Dimethyl-N1,N3-bis(phenylcarbonothioyl)propanedihydrazide (elesclomol) was kindly provided by Synta Pharmaceuticals (Lexington, MA, USA). Antibodies for Caspase-3 and SOD1 were purchased from Cell Signaling Inc. (Danvers, MA, USA). Antibodies for TRX and catalase were purchased from BD Bioscience (San Jose, CA, USA). Antibody for Actin was purchased from Sigma Chemicals (St. Louis, MO, USA).

3.3. Growth Inhibitory Assay

Cells were seeded in 24-well dishes and treated with various concentrations of elesclomol for 24, 48, or 72 h. The procedure was described previously [9,13]. Briefly, the culture mediums as well as the trypsinized cells were collected and this mixture was centrifuged at 400× g for 5 min. The supernatant was discarded, and resuspended in 1 mL of Hank's buffer and assayed for live cells using a Vi-Cell cell viability analyzer (Beckman Coulter, Inc.)

3.4. Apoptosis Assay

Untreated and treated cells were collected and apoptosis was analyzed with caspase fluorescein-conjugated V-D-FMK (R&D Systems, Minneapolis, MN, USA). The cell pellet was collected and suspended in 100 µL of staining solution (10 µL of fluorescein and 20 µL of propidium iodide (PI) in 1 mL of PBS). The suspension was incubated at 37 °C for 30 min, washed with PBS, centrifuged to remove unbound reagent and resuspended in 500 µL of PBS. Samples were analyzed using a Coulter XL Flow Cytometer.

3.5. Assay for Intracellular ROS and H2O2

Untreated and treated cells were collected and intracellular ROS or H2O2 were measured by incubating with 20 µM of CM-H2DCF-DA (Invitrogen, Carlsbad, CA, USA) or with 10 µM of acetyl-pentafluorobenzenesulfonyl fluorescein (APFB) (EMD, La Jolla, CA, USA), respectively, at 37 °C for 30 min in the dark. Then the cells were washed once with PBS and centrifuged to remove impermeable reagents. Cells were resuspended in 500 µL of PBS and analyzed in a Coulter XL Flow Cytometry unit (excitation at 495 nm and emission at 529 nm).

3.6. Assay of Thioredoxin Reductase (TrxR)

A TrxR kit (Cayman Chemical, Ann Arbor, MI, USA) was used to measured total cellular thioredoxin reductase. Cells were seeded at 4 × 105 and cell lysate was prepared by sonication using the conditions recommended by the manufacturer. Total TrxR activity was detected by measuring the reduction of DTNB with NADPH to TNB by UV spectrophotometer at 405 nm. The cellular TrxR contents were calculated using the standard curve generated in parallel experiments. Protein concentrations of each condition were verified using the Micro BCA Protein Assay (Thermo Scientific, Rockford, IL, USA).

3.7. Assay of Glutathione (GSH)

A GSH assay kit (Cayman Chemical, Ann Arbor, MI, USA) was used to measure total cellular glutathione. Cells were seeded at 4 × 105 and cell lysate was prepared by sonication and deproteination using the conditions recommended by the manufacturer. Total GSH was detected by measuring the product of glutathionylated DTNB by UV spectrophotometer at 405 nm. The cellular GSH contents were calculated using the standard curve generated in parallel experiments.

3.8. Western Blot Analysis

Cells were seed at 1 × 105/ml onto 100 mm dishes, treated, collected, lysed and immunoblotted with indicated antibody. Detail procedure was described in our previous publications [9,13]. Briefly, cell lysis was completed by sonication and centrifugation. The total protein was separated on an SDS-PAGE, transferred onto a nitrocellulose membrane (Amersham Biosciences, Piscataway, NJ) and immunoblotted with the indicated antibodies. Bands were measured using a molecular imager Chemidoc system with Quality One software (Bio-Rad, Richmond, CA).

4. Conclusions

In this report, we have presented a novel alternative approach to selectively kill cisplatin resistant cells. Cisplatin is known to inhibit TrxR which leads to elevated ROS levels and repeated exposure to cisplatin leads to lower TrxR/TRX levels and high basal ROS levels (see Scheme 1). Cisplatin resistant cells may adapt to survive under high basal levels of ROS by activating other antioxidant systems to compensate. Exposure to elesclomol leads to further increase in ROS levels and results in cell death while normal and parental cells which have lower levels of ROS can survive at the same dosage of elesclomol.

Scheme 1
Repeat exposure to cisplatin elevates ROS production in lung cancer. The use of elesclomol to further increase ROS can push cisplatin resistant cells beyond the threshold of tolerability and hence create a potential therapeutic window.

Targeting cancer cells by ROS mediated mechanisms was initially reported by Trachootham et al. as an effective strategy to treat cancer cells that possess unique redox mechanisms [14]. This concept has been shown to effectively kill fludarabine resistant CLL and imitanib resistant CML cells that have high basal ROS levels [15]. The investigators used PEITC which interferes with GSH as a mean to increase ROS. There are multiple means of increasing ROS, either by direct generation of ROS or interfering with the antioxidant system. The efficacy of each method is greatly dependent on both the active redox system of the tumor cells and the cause of increasing ROS.

In our cisplatin resistant lung cancer cell lines, we have found that elesclomol is very effective in generating ROS and selectively killing them. This finding can be translated into future clinical trial to treat patients who fail cisplatin.

Acknowledgments

This work is supported by a VA Merit Review award to NS. We would like to thank Synta Pharmaceuticals for providing elesclomol and TRX/GSH activity kits.

References and Notes

1. Witte A.B., Anestal K., Jerremalm E., Ehrsson H., Arner E.S. Inhibition of thioredoxin reductase but not of glutathione reductase by the major classes of alkylating and platinum-containing anticancer compounds. Free Radic. Biol. Med. 2005;39:696–703. doi: 10.1016/j.freeradbiomed.2005.04.025. [PubMed] [Cross Ref]
2. Arner E.S., Nakamura H., Sasada T., Yodoi J., Holmgren A., Spyrou G. Analysis of the inhibition of mammalian thioredoxin, thioredoxin reductase, and glutaredoxin by cis-diamminedichloroplatinum (II) and its major metabolite, the glutathione-platinum complex. Free Radic. Biol. Med. 2001;31:1170–1178. doi: 10.1016/S0891-5849(01)00698-0. [PubMed] [Cross Ref]
3. Mustacich D., Powis G. Thioredoxin reductase. Biochem. J. 2000;346 Pt 1:1–8. doi: 10.1042/0264-6021:3460001. [PubMed] [Cross Ref]
4. Saitoh M., Nishitoh H., Fujii M., Takeda K., Tobiume K., Sawada Y., Kawabata M., Miyazono K., Ichijo H. Mammalian thioredoxin is a direct inhibitor of apoptosis signal-regulating kinase (ASK) 1. Embo. J. 1998;17:2596–2606. doi: 10.1093/emboj/17.9.2596. [PubMed] [Cross Ref]
5. Sun Y., Rigas B. The thioredoxin system mediates redox-induced cell death in human colon cancer cells: implications for the mechanism of action of anticancer agents. Cancer Res. 2008;68:8269–8277. doi: 10.1158/0008-5472.CAN-08-2010. [PMC free article] [PubMed] [Cross Ref]
6. Kirshner J.R., He S., Balasubramanyam V., Kepros J., Yang C.Y., Zhang M., Du Z., Barsoum J., Bertin J. Elesclomol induces cancer cell apoptosis through oxidative stress. Mol. Cancer Ther. 2008;7:2319–2327. doi: 10.1158/1535-7163.MCT-08-0298. [PubMed] [Cross Ref]
7. Sasada T., Nakamura H., Ueda S., Sato N., Kitaoka Y., Gon Y., Takabayashi A., Spyrou G., Holmgren A., Yodoi J. Possible involvement of thioredoxin reductase as well as thioredoxin in cellular sensitivity to cis-diamminedichloroplatinum (II) Free Radic. Biol. Med. 1999;27:504–514. doi: 10.1016/S0891-5849(99)00101-X. [PubMed] [Cross Ref]
8. Yamada M., Tomida A., Yoshikawa H., Taketani Y., Tsuruo T. Overexpression of thioredoxin does not confer resistance to cisplatin in transfected human ovarian and colon cancer cell lines. Cancer Chemother. Pharmacol. 1997;40:31–37. doi: 10.1007/s002800050621. [PubMed] [Cross Ref]
9. Wangpaichitr M., Wu C., You M., Kuo M.T., Feun L., Lampidis T., Savaraj N. Inhibition of mTOR restores cisplatin sensitivity through down-regulation of growth and anti-apoptotic proteins. Eur. J. Pharmacol. 2008;591:124–127. doi: 10.1016/j.ejphar.2008.06.028. [PMC free article] [PubMed] [Cross Ref]
10. Wu C., Wangpaichitr M., Feun L., Kuo M.T., Robles C., Lampidis T., Savaraj N. Overcoming cisplatin resistance by mTOR inhibitor in lung cancer. Mol. Cancer. 2005;4:25. doi: 10.1186/1476-4598-4-25. [PMC free article] [PubMed] [Cross Ref]
11. Savaraj N., Wu C., Wangpaichitr M., Kuo M.T., Lampidis T., Robles C., Furst A.J., Feun L. Overexpression of mutated MRP4 in cisplatin resistant small cell lung cancer cell line: collateral sensitivity to azidothymidine. Int. J. Oncol. 2003;23:173–179. [PubMed]
12. Xu J., Tyan T., Cedrone E., Savaraj N., Wang N. Detection of 11q13 amplification as the origin of a homogeneously staining region in small cell lung cancer by chromosome microdissection. Genes Chromosomes Cancer. 1996;17:172–178. doi: 10.1002/(SICI)1098-2264(199611)17:3<172::AID-GCC5>3.0.CO;2-1. [PubMed] [Cross Ref]
13. Wangpaichitr M., Savaraj N., Maher J., Kurtoglu M., Lampidis T.J. Intrinsically lower AKT, mammalian target of rapamycin, and hypoxia-inducible factor activity correlates with increased sensitivity to 2-deoxy-d-glucose under hypoxia in lung cancer cell lines. Mol. Cancer Ther. 2008;7:1506–1513. doi: 10.1158/1535-7163.MCT-07-2334. [PMC free article] [PubMed] [Cross Ref]
14. Trachootham D., Zhou Y., Zhang H., Demizu Y., Chen Z., Pelicano H., Chiao P.J., Achanta G., Arlinghaus R.B., Liu J. Selective killing of oncogenically transformed cells through a ROS-mediated mechanism by beta-phenylethyl isothiocyanate. Cancer Cell. 2006;10:241–252. doi: 10.1016/j.ccr.2006.08.009. [PubMed] [Cross Ref]
15. Trachootham D., Zhang H., Zhang W., Feng L., Du M., Zhou Y., Chen Z., Pelicano H., Plunkett W., Wierda W.G. Effective elimination of fludarabine-resistant CLL cells by PEITC through a redox-mediated mechanism. Blood. 2008;112:1912–1922. doi: 10.1182/blood-2008-04-149815. [PubMed] [Cross Ref]

Articles from Cancers are provided here courtesy of Multidisciplinary Digital Publishing Institute (MDPI)