Journal Information
Download PDF
More article options
DOI: 10.1016/j.bjpt.2021.04.010
Full text access
Available online 10 June 2021
Dyspnea could be accurately assessed by a caregiver in hospitalized patients with respiratory diseases: Interrater reliability and agreement study
Gregory Reychlera,b,c,
Corresponding author

Corresponding author at: Service de Pneumologie, Cliniques Universitaires Saint-Luc, Avenue Hippocrate 10, Brussels 1200, Belgium.
, Marc Beaumontd,e, Anne-Claire Latiersc, Thierry Pietersa,b, Antoine Fremaultf
a Institut de Recherche Expérimentale et Clinique (IREC), Pôle de Pneumologie, ORL and Dermatologie, Université Catholique de Louvain, Brussels, Belgium
b Service de Pneumologie, Cliniques Universitaires Saint-Luc, Avenue Hippocrate 10, Brussels 1200, Belgium
c Secteur de Kinésithérapie et Ergothérapie, Cliniques Universitaires Saint-Luc, Brussels, Belgium
d Service de Réhabilitation Respiratoire, Centre Hospitalier des Pays de Morlaix, Morlaix, France
e EA3878 (GETBO), CHU Brest, Brest, France
f Service de Pneumologie, Grand Hôpital de Charleroi, Charleroi, Belgium
Received 20 March 2019. Accepted 21 April 2021
Article information
Full Text
Download PDF
Tables (3)
Table 1. Patients characteristics.
Table 2. Assessment of dyspnea using the modified Borg scale and the visual analogue scale.
Table 3. Reliability of dyspnea rating in hospitalized patients with lung disease.
Show moreShow less

The perception of dyspnea is a subjective feeling typically self-assessed by the patient. However, the assessment by a caregiver is sometimes required.


The primary aim was to compare patient self-assessment and caregiver assessment of dyspnea (interrater reliability) using the modified Borg and visual analog scale (VAS) in hospitalized patients. The secondary aim was to compare dyspnea assessment between the two scales for patients and caregiver (inter-instrument reliability).


Self-assessment of dyspnea intensity of hospitalized patients with respiratory diseases was compared with caregiver's assessment. Dyspnea intensity was measured using two scales, the modified Borg scale (0–10 scale) and the 10 cm VAS. Mean difference and 95% confidence interval (CI) between assessors (i.e. patient versus caregiver) were calculated for each scale. Inter- and intra-rater reliability was calculated using intraclass correlation coefficients (ICCs).


A total of 254 patients were recruited. The mean differences between patient and caregiver ratings were 0.31 (95% CI: 0.09, 0.53) for the modified Borg scale and 0.36 (95% CI: 0.06, 0.65) for the VAS scale. Interrater reliability was good for both scales with ICC of 0.79 (95% CI: 0.73, 0.84) for VAS and 0.82 (95% CI: 0.77, 0.86) for the modified Borg scale. The mean differences in scores between scales were 0.93 (95% CI 0.69, 1.17) for patients' ratings and 0.88 (95% CI 0.72, 1.04) for caregiver's rating. The inter-instrument reliability was moderate to good and similar for both assessors.


Dyspnea can be accurately estimated by caregivers when patients with lung diseases cannot self-report. Scores on the VAS to rate dyspnea were higher than the scores on the Borg scale.

Full Text

Dyspnea is defined as “a subjective experience of breathing discomfort that consists of qualitatively distinct sensations.”1 It is a common and complex symptom frequently assessed in clinical practice. Assessment of dyspnea is relevant to patients admitted to the emergency room or the hospital,2 community-residing individuals,3 or patients consulting their general practitioner.4 Due to the high prevalence of dyspnea in a variety of respiratory diseases, such as lung cancer, chronic obstructive pulmonary disease (COPD), and interstitial lung disease, it is considered an important patient-reported outcome in clinical and research settings.5

The mechanisms of dyspnea and contributing factors have been investigated for a long time. Nevertheless, knowledge about the neurophysiology and psychology of dyspnea increased considerably in the last decade. Psychological and cultural factors may play an important role in the perception of shortness of breath.6 Accordingly, the assessment of dyspnea has improved by notably taking into account different domains.7 But, despite new tools for assessing dyspnea, that specifically evaluate sensory and affective domains, having recently emerged,8 the visual analog scale (VAS) and modified Borg scales are still largely used.9 Both of these tools have already shown to be sensitive enough to measure these two dyspnea domains.8 However, even if they have a similar scoring scale and the tools are conceptually similar, the level of association between these two measures has not been studied.

While the perception of dyspnea is subjective and, therefore, the patient's self-assessment of dyspnea is the preferred assessment method,6 the assessment by a caregiver is sometimes required. This occurs when the patient is performing an exercise or for those who are unable to communicate their feelings. Fifty-four percent of patients referred for inpatient palliative care consultation are unable to assess their own level of dyspnea.10 To facilitate the assessment of dyspnea in these patients, the Respiratory Distress Observation Scale has been validated in terminally ill patients.11 However, there is a lack of specific tools for non-communicative patients such as those in coma or diagnosed with cerebral palsy.

Pain is a subjective symptom which may share some similarities with dyspnea. Indeed, pain and dyspnea are both associated to chronic diseases, unpleasantness, decline in quality of life, and physical sensations mediated by a common cortical network. Patients' and physicians' assessment of pain intensity has showed poor agreement.12 Similar results were found with measurements of other symptoms, especially for those considered more subjective.13,14 Nevertheless, the agreement between caregiver and patient assessment of dyspnea intensity has been poorly studied. The dyspnea duration and its variability over time could influence this agreement and these elements are partly related to the types of respiratory disease. Moreover, the feelings of emotional distress could differently influence this assessment either by the caregiver or by the patient depending on the disease and its prognosis.

The primary objective of the current study was to compare the dyspnea assessment between patients and caregivers (interrater reliability) using both the modified Borg scale and a VAS in hospitalized patients with respiratory diseases. The secondary objective was to compare dyspnea assessment scores between the two scales for the patients and caregivers (inter-instrument reliability).

MethodsStudy design

This was an observational cross-sectional study which follows the Guidelines for Reporting Reliability and Agreement Studies.15


We prospectively recruited all participants who were hospitalized between July and September 2013 in the pulmonology and oncology units of the Cliniques Universitaires Saint-Luc (Brussels, Belgium) and the Grand Hôpital de Charleroi (Charleroi, Belgium). To be included, participants had to be aged 18 years or over and diagnosed with an acute or chronic respiratory disease with or without oxygen therapy. Patients were excluded if presenting with mental or cognitive impairments or unable to understand French and if requiring non-invasive ventilation.

All included patients signed a written informed consent form according to the Declaration of Helsinki and the current guidelines for Clinical Good Practice. The study was approved by the Institutional Medical Ethics Committee (Cliniques Universitaires Saint-Luc, Brussels, Belgium - B403201317644) and was registered at (NCT03598569).


All participants were assessed during hospitalization by the same qualified caregiver. The caregiver was a physical therapist who worked in the pulmonology unit and in the rehabilitation center on a regular basis for the past 5 years and was experienced in assessing dyspnea intensity. The assessment of dyspnea by the caregiver was made through visual observation and was primarily based on breathing frequency and the recruitment of accessory muscles. This assessment was performed as a single measure just before the patient's self-assessment of dyspnea. The assessment of dyspnea intensity by the patient and the caregiver was performed while the patient was at rest. Patients were unaware of the time the caregiver's assessment took place and they remained blinded to the caregiver's rating throughout the study.


The 10 cm VAS16 and the modified (0–10) Borg scale17 were used to assess the sensory domain of dyspnea. Both scales were previously validated for this purpose.18

Visual analog scale (VAS)

The VAS is a 10 cm line anchored at the ends by statements that define dyspnea, ranging from “I am not at all short of breath” (0 cm) to “The most short of breath I have ever been” (10 cm). Participants were asked to place a vertical mark on the line to indicate how they rate their dyspnea. The distance between the start of the line and the mark placed by the participant was then measured in centimeters.

Modified Borg scale

The modified Borg scale is an 11-point numerical scale with associated verbal descriptors rating the difficulty of breathing from “Your breathing is causing you no difficulty at all” (zero point) to “Your breathing difficulty is maximal” (10 points). The patients were asked to choose the number that best describeg their feeling about breathing difficulties.

Data analysis

The SPSS 25.0 program (IBM software) was used to analyze the data. Descriptive data were expressed as mean ± SD for normally distributed variables or median (min-max) for non-normally distributed variables. Normality of the distribution was investigated by the Kolmogorov-Smirnov test. The comparison between patient's self-assessment and caregiver's assessment of dyspnea using both scales in hospitalized patients with respiratory diseases was performed by calculating the mean difference and 95% confidence interval (CI) between raters. The degree of concordance between the two assessors (interrater reliability for each scale) was calculated using intraclass correlation coefficients (ICCs) using a two-way mixed effects (single measurement) mode with consistency.19–22

For the comparison of dyspnea assessment between the two scales, for patients and caregiver, we calculated the mean difference in scores and 95% CI between the two scales. The degree of concordance between both scales (inter-instrument reliability) for both the patients and the caregiver was also calculated using a two-way mixed effects model (single measurement) with absolute agreement ICC type.22 All ICCs were expressed with their 95% CI. Values less than 0.5 indicate poor reliability, values between 0.5 and 0.75 indicate moderate reliability, values between 0.75 and 0.9 indicate good reliability, and values greater than 0.90 indicate excellent reliability.22 In a secondary analysis, the above was then conducted separately for the subgroup of patients with lung cancer and those with other lung diseases.


Two hundred and seventy one patients were considered eligible for the study. Sixteen patients were excluded because they did not understand French and one patient was excluded due to cognitive impairments. The characteristics and diagnosis of the included patients are shown in Table 1.

Table 1.

Patients characteristics.

  All  Lung cancer group  Other lung disease group 
254  121  133 
Age  66.5 ± 11.9  64.9 ± 9.9  68.0 ± 13.4 
Gender (F/M)    76/45  61 /72 
DiagnosisCOPDFibrosisAsthmaPneumoniaLung transplantBronchiectasisOther      858373324 
OxygennFlow (L/min)  740.8 ± 1.4  210.5 ± 1.4  531.0 ± 1.5 

Values are expressed as mean ± standard deviation or number of patients.

Patient versus caregiver

For the total sample, the mean differences between patient and caregiver ratings were 0.31 (95% CI: 0.09, 0.53) for the modified Borg scale and 0.36 (95% CI: 0.06, 0.65) for the VAS scale (Table 2), which indicates higher scores for patient self-assessment compared to caregiver assessment. We found good reliability for both scales with ICC of 0.79 (95% CI: 0.73, 0.84) for the VAS and 0.82 (95% CI: 0.77, 0.86) for the modified Borg scale (Table 3), which indicates good agreement between patients and caregiver for both scales. Our subgroup analyses indicated similar results for the group of patients with lung cancer. In contrast, a significant difference in rating was found between the caregiver and the patients’ self-rating for the group with other lung diseases (Table 2). Finally, there was a higher degree of concordance between patients and caregiver in the lung cancer group (ICC > 0.90) compared to those with other lung diseases (ICC between 0.50 and 0.75; Table 3).

Table 2.

Assessment of dyspnea using the modified Borg scale and the visual analogue scale.

Group/instrument  Caregiver  Patient  Mean Difference (95%CI)* 
Total sample (n = 254) 
Borg  2.4 ± 2.2  2.7 ± 2.3  0.31 (0.09, 0.53) 
VAS  3.2 ± 2.9  3.6 ± 2.8  0.36 (0.06, 0.65) 
Mean Difference (95%CI)**  0.88 (0.72, 1.04)  0.93 (0.69, 1.17)   
Lung cancer group (n = 121) 
Borg  2.0 ± 2.3  1.9 ± 2.3  −0.03 (−0.23, 0.17) 
VAS  2.2 ± 2.7  2.2 ± 2.5  −0.02 (−0.29, 0.25) 
Mean Difference (95%CI)**  0.25 (0.10, 0.40)  0.25 (0.03, 0.47)   
Other lung diseases group (n = 133) 
Borg  2.7 ± 2.0  3.3 ± 2.2  0.62 (0.24, 0.99) 
VAS  4.2 ± 2.7  4.9 ± 2.5  0.70 (0.20, 1.20) 
Mean Difference (95%CI)**  1.46 (1.22, 1.70)  1.54 (1.15, 1.93)   

VAS: visual analogue scale.

Values are expressed as mean ± standard deviation or mean difference (95% confidence interval). *Between caregiver and patient's self-rating. **Between Borg and VAS scales.

Table 3.

Reliability of dyspnea rating in hospitalized patients with lung disease.

Reliability  ICCTotal sample  ICCLung cancer group  ICCOther lung diseases group 
Caregiver vs patient self-rating       
VAS  0.79 (0.73, 0.84)  0.91 (0.87, 0.94)  0.56 (0.38, 0.69) 
Borg  0.82 (0.77, 0.86)  0.94 (0.91, 0.96)  0.64 (0.49, 0.74) 
VAS vs Borg (inter-instrument     
Caregiver  0.90 (0.74, 0.95)  0.97 (0.95, 0.98)  0.82 (0.20, 0.93) 
Patient  0.81 (0.68, 0.87)  0.93 (0.90, 0.95)  0.62 (0.26, 0.78) 

ICC: intraclass correlation coefficient, VAS: visual analogue scale.

Values are intraclass correlation coefficients (95% confidence intervals).

VAS versus modified Borg scale

For the total sample, the mean differences between the VAS and modified Borg scale were 0.93 (95% CI 0.69, 1.17) for patients' ratings and 0.88 (95% CI 0.72, 1.04) for caregiver's rating (Table 2). These results indicate higher dyspnea rating obtained with the VAS scale compared to the modified Borg scale regardless of the assessor. Our results also showed at least moderate reliability between scales (Table 3). The subgroup analyses showed similar ratings for both scales, regardless of the assessor, but the degree of concordance was higher for the lung cancer group (ICC 0.90 or greater) compared to the other lung diseases group (ICC ranging from 0.62 to 0.82).


Our results showed that the caregiver's ratings of dyspnea in hospitalized patients with respiratory diseases were similar to the patients' self-ratings. Although higher scores were obtained with patients' self-assessment compared to caregiver's assessment, the difference of less than 1 point might not be considered clinically relevant because the minimal clinically important difference (MCID) is 1-unit for the modified Borg scale and approximately 1 to 2 cm for the VAS.27 It is noteworthy, that the level of agreement between the caregiver and patients’ self-rating was better for those with lung cancer who had lower level of dyspnea compared to those with other lung conditions who had more severe dyspnea.

Although dyspnea encompasses two domains, in the present study, we focused on the sensory domain which is less subjective than the emotional domain. Secondly, the caregiver in charge of assessing dyspnea was a qualified physical therapist with 5 years of experience. Future studies are needed to investigate whether the results found in this study could be generalized to less experienced caregivers.

The majority of patients in the other lung disease group were diagnosed with COPD. Patients with COPD report more frequent and longer episodes of dyspnea than patients with lung cancer23 and can be overperceivers for this symptom.24 Differences in the medications used to treat the patients from both groups could also explain the differences in the reliability indices. Opioids have been shown to play a role in dyspnea perception25 and they are more often prescribed in patient with lung cancer. However, we did not have the rate of opioids prescription in this patient population. The patient's past history also contributes to symptom perception and may differ based on diagnosis.26

The secondary purpose of the study established that the VAS and modified Borg scales are not interchangeable to quantify dyspnea severity. If the inter-instrument reliability was good, and even excellent in the group with lung cancer as previously observed in healthy subjects and patients with COPD patients,28,29 such a difference was expected.30,31 The high level of reliability suggests a standard offset between the scales.

The strengths of this study include the large cohort of patients with dyspnea and the variety of pulmonary diseases which offered a wide range of dyspneic conditions. Our results should be interpreted with caution. An emotional influence cannot be excluded.32,33 Data were collected on hospitalized patients only, therefore making extrapolation to outpatient settings difficult in part due to potential anxiety related to hospitalization.34 No objective measurements of lung function were used to determine the validity of the data.35,36 Finally, data have limited external validity as a single examiner with five years of experience participated in the study.


We demonstrated that dyspnea can be accurately estimated by an experienced caregiver in hospitalized patients with lung diseases, using either a VAS or modified Borg scale. This is particularly the case for patients with lung cancer with lower level of dyspnea. A larger difference between raters was shown for patients with other lung diseases and more severe dyspnea. However, the VAS and Borg scales are not interchangeable, as the scores on the VAS were consistently higher than the scores on the Borg scale.


We would like to thank Sabrina Urso, Cécile De Noble, and Juliette Guislain for the support in the project. Gregory Reychler received a grant from the Institut de Recherche Expérimentale et Clinique (Université catholique de Louvain – Brussels – Belgium).

M.B. Parshall, R.M. Schwartzstein, L. Adams, R.B. Banzett, H.L. Manning, J. Bourbeau, et al.
An official American Thoracic Society statement: update on the mechanisms, assessment, and management of dyspnea.
Am J Respir Crit Care Med, 185 (2012), pp. 435-452
J.P. Stevens, T. Dechen, R. Schwartzstein, C. O'Donnell, K. Baker, M.D. Howell, et al.
Prevalence of dyspnea among hospitalized patients at the time of admission.
J Pain Symptom Manage, (2018),
A. Frostad, V. Soyseth, A. Andersen, A. Gulsvik.
Respiratory symptoms as predictors of all-cause mortality in an urban community: a 30-year follow-up.
J Intern Med, 259 (2006), pp. 520-529
T. Frese, C. Sobeck, K. Herrmann, H. Sandholzer.
Dyspnea as the reason for encounter in general practice.
J Clin Med Res, 3 (2011), pp. 239-246
M. Ekstrom, J. Sundh, K. Larsson.
Patient reported outcome measures in chronic obstructive pulmonary disease: which to use?.
Expert Rev Respir Med, 10 (2016), pp. 351-362
P.S. De, D. Van, Lemaigre V I, G. Verleden, M. Demedts.
Van den Bergh O. Dyspnea: the role of psychological processes.
Clin Psychol Rev, 24 (2004), pp. 557-581
R.B. Banzett, C.R. O'Donnell, T.E. Guilfoyle, M.B. Parshall, R.M. Schwartzstein, P.M. Meek, et al.
Multidimensional dyspnea profile: an instrument for clinical and laboratory research.
Eur Respir J, 45 (2015), pp. 1681-1691
A. von Leupoldt, B. Dahme.
Differentiation between the sensory and affective dimension of dyspnea during resistive load breathing in normal subjects.
Chest, 128 (2005), pp. 3345-3349
S. Dorman, A. Byrne, A. Edwards.
Which measurement scales should we use to measure breathlessness in palliative care? A systematic review.
Palliat Med, 21 (2007), pp. 177-191
M.L. Campbell, T. Templin, J. Walch.
Patients who are near death are frequently unable to self-report dyspnea.
J Palliat Med, 12 (2009), pp. 881-884
M.L. Campbell, T. Templin, J. Walch.
A Respiratory Distress Observation Scale for patients unable to self-report dyspnea.
J Palliat Med, 13 (2010), pp. 285-290
C. Brunelli, S. Kaasa, A.K. Knudsen, M.J. Hjermstad, A. Pigni, A. Caraceni.
Comparisons of patient and physician assessment of pain-related domains in cancer pain classification: results from a large international multicenter study.
I.J. Higginson, W. Gao.
Caregiver assessment of patients with advanced cancer: concordance with patients, effect of burden and positivity.
Health Qual Life Outcomes, 6 (2008), pp. 42
M.J. Silveira, C.W. Given, B. Given, A.M. Rosland, J.D. Piette.
Patient-caregiver concordance in symptom assessment and improvement in outcomes for patients undergoing cancer chemotherapy.
Chronic Illn, 6 (2010), pp. 46-56
J. Kottner, L. Audige, S. Brorson, A. Donner, B.J. Gajewski, A. Hrobjartsson, et al.
Guidelines for reporting reliability and agreement studies (GRRAS) were proposed.
J Clin Epidemiol, 64 (2011), pp. 96-106
J.G. Burdon, E.F. Juniper, K.J. Killian, F.E. Hargreave, E.J. Campbell.
The perception of breathlessness in asthma.
Am Rev Respir Dis, 126 (1982), pp. 825-828
G.A. Borg.
Psychophysical bases of perceived exertion.
Med Sci Sports Exerc, 14 (1982), pp. 377-381
G.A. Borg.
A category scale with ratio properties for intermodal and interindividual comparisons.
pp. 25-34
K.O. McGraw, S.P. Wong.
Forming inferences about some intraclass correlation coefficients.
Psychol Methods, 1 (1996), pp. 30-46
C.E Nickerson.
A note on “a concordance correlation coefficient to evaluate reproducibility”.
Biometrics, 53 (1997), pp. 1503-1507
P.E. Shrout, J.L. Fleiss.
Intraclass correlations: uses in assessing rater reliability.
Psychol Bull, 86 (1979), pp. 420-428
T.K. Koo, M.Y. Li.
A guideline of selecting and reporting intraclass correlation coefficients for reliability research.
J Chiropr Med, 15 (2016), pp. 155-163
V. Weingartner, C. Bausewein, I.J. Higginson, C. Scheve, F.E. Murtagh, R. Voltz, et al.
Characterizing episodic breathlessness in patients with advanced disease.
J Palliat Med, 16 (2013), pp. 1275-1279
G. Scioscia, I. Blanco, E. Arismendi, F. Burgos, C. Gistau, M.P. Foschino Barbaro, et al.
Different dyspnoea perception in COPD patients with frequent and infrequent exacerbations.
A.H. Gifford, D.A. Mahler, L.A. Waterman, J. Ward, W.J. Kraemer, B.R. Kupchak, et al.
Neuromodulatory effect of endogenous opioids on the intensity and unpleasantness of breathlessness during resistive load breathing in COPD.
J. Hoyer, A. Reusch, E. Leibing.
Accuracy of interoception and withdrawal of attention in airway diseases: bronchial asthma versus chronic obstructive bronchitis.
Verhaltenstherapie, 9 (1999), pp. 140-147
A.L. Ries.
Minimally clinically important difference for the UCSD shortness of breath questionnaire, Borg scale, and visual analog scale.
K. Bogaerts, K. Notebaert, I. Van D, S. Devriese, P.S. De, O. Van den Bergh.
Accuracy of respiratory symptom perception in different affective contexts.
J Psychosom Res, 58 (2005), pp. 537-543
A. von Leupoldt, R. Ambruzsova, S. Nordmeyer, N. Jeske, B. Dahme.
Sensory and affective aspects of dyspnea contribute differentially to the Borg scale's measurement of dyspnea.
Respiration, 73 (2006), pp. 762-768
H. Ohbayashi.
Comparison of the rapid effects of single inhalations of formoterol and tiotropium bromide on respiratory function and COPD symptoms in a randomized crossover study.
Respir Investig, 55 (2017), pp. 348-356
R.C. Boshuizen, A.D. Vincent, M.M. van den Heuvel.
Comparison of modified Borg scale and visual analog scale dyspnea scores in predicting re-intervention after drainage of malignant pleural effusion.
Support Care Cancer, 21 (2013), pp. 3109-3116
A.G. Gift, S.M. Plaut, A. Jacox.
Psychologic and physiologic factors related to dyspnea in subjects with chronic obstructive pulmonary disease.
Heart Lung, 15 (1986), pp. 595-601
S.T. Simon, C. Bausewein, E. Schildmann, I.J. Higginson, H. Magnussen, C. Scheve, et al.
Episodic breathlessness in patients with advanced disease: a systematic review.
J Pain Symptom Manage, 45 (2013), pp. 561-578
S.M.H. Al Aqqad, B. Tangiisuran, I.A. Hyder Ali, R.M.N. Md Kassim, J.L. Wong.
Tengku Saifudin TI. Hospitalisation of multiethnic older patients with AECOPD: exploration of the occurrence of anxiety, depression and factors associated with short-term hospital readmission.
Clin Respir J, 11 (2017), pp. 960-967
J.M. Marin, S.J. Carrizo, M. Gascon, A. Sanchez, B. Gallego, B.R. Celli.
Inspiratory capacity, dynamic hyperinflation, breathlessness, and exercise performance during the 6-minute-walk test in chronic obstructive pulmonary disease.
Am J Respir Crit Care Med, 163 (2001), pp. 1395-1399
S. Ozalevli, E.S. Ucan.
The comparison of different dyspnoea scales in patients with COPD.
J Eval Clin Pract, 12 (2006), pp. 532-538
Copyright © 2021. Associação Brasileira de Pesquisa e Pós-Graduação em Fisioterapia
Brazilian Journal of Physical Therapy

Subscribe to our newsletter

Article options
en pt
Cookies policy Política de cookies
To improve our services and products, we use "cookies" (own or third parties authorized) to show advertising related to client preferences through the analyses of navigation customer behavior. Continuing navigation will be considered as acceptance of this use. You can change the settings or obtain more information by clicking here. Utilizamos cookies próprios e de terceiros para melhorar nossos serviços e mostrar publicidade relacionada às suas preferências, analisando seus hábitos de navegação. Se continuar a navegar, consideramos que aceita o seu uso. Você pode alterar a configuração ou obter mais informações aqui.