The landscape of validity and reliability practices from applied performance analysts: Establishing a best practice framework

Dataset for: Whilst Theoretical Performance Analysts benefit from established validity and reliability frameworks, their practical application faces several challenges for Applied Performance Analysts (APAs). We examined the current practices and perceptions regarding validity and reliability by APAs to formulate best-practice recommendations. An open and closed-answer survey was completed by 175 APAs from a range of sports and countries, with responses analysed based on experience (earl-career APAs vs experienced APAs). Findings reveal a desire for robust validity and reliability methods, particularly among experienced APAs. Collaborative approaches amongst APAs and other stakeholders were identified when determining performance indicators, yet decisions for live data collection were primarily made by APAs. Familiarisation and reliability processes varied, specifically the frequency of checks, with experienced APAs dedicating more time to learning new systems and checking data more frequently. Despite the existing theoretical frameworks, time constraints and personnel changes pose challenges for practical implementation, with APAs calling for an applied framework. These insights inform the development of best-practice recommendations for collecting, analysing, and presenting accurate and meaningful performance analysis data in applied settings.