Why Johnny Adopts Identity-Based Software Signing: A Usability Case Study of Sigstore

Why Johnny Adopts Identity-Based Software Signing: A Usability Case Study of Sigstore Kelechi G. Kalu Purdue University [email protected] Sofia Okorafor Purdue University [email protected] Tanmay Singla Purdue University [email protected] Sophie Chen Carnegie Mellon University [email protected] Santiago Torres-Arias Purdue University [email protected] James C. Davis Purdue University [email protected] Abstract Software signing is the most robust method for ensuring the integrity and authenticity of components in a software supply chain. Legacy key-managed signing tools (e.g., OpenPGP) burdened practitioners with key management and signer identification, creating both usability challenges and security risks. A new class of identity-based signing tools have automated many of these concerns, but little is known about their usability and its effect on adoption and effectiveness in practice. A usability evaluation can clarify the extent to which identity-based designs succeed and highlight priorities for improvement. To fill this gap, we conducted a usability study of Sigstore, a pioneering and widely adopted exemplar of identity-based signing. Through interviews with 17 industry experts, we examined (1) the problems and advantages associated with practitioners’ tooling choices, (2) how and why their signingtool usage has evolved over time, and (3) the contexts that cause usability concerns. Our findings illuminate the usability factors of identity-based signing tools and yield recommendations for toolmakers, adopting organizations, and the research community. Notably, components of identity-based tooling exhibit different levels of maturity and readiness for adoption, and integration flexibility is a common pain point but potentially mitigable through plugins and APIs. Our results will help identity-based signing toolmakers further strengthen software supply chain security. 1 Introduction The reuse of software components in modern software production creates complex supply chains that are susceptible to the unauthorized introduction of code [6,32,87,100]. To mitigate this risk, engineers need provenance—evidence of actor authenticity and artifact integrity [10]. Software signing provides the strongest guarantee of provenance currently available [91] and has thus been widely advocated by academia, industry consortia, and government regulators [13,21,33,95]. Like other security practices, the effectiveness of software signing in practice depends on the availability and usability of tools that support it. Traditional signing tools such as OpenPGP are famously challenging to use [99], with several “Why Johnny Can’t Encrypt”–inspired studies consistently revealing usability challenges that limit adoption [38,75,99]. In recent years, identity-based systems (e.g., Sigstore [103]) have been introduced that automate much of the process using short-lived keys and external identity providers [15]. However, despite growing adoption driven by recent supply chain attacks and efforts to lower usability barriers [29,92,103], we lack evidence about the usability of these identity-based tools and need to understand how that affects adoption in practice [63]. The innovations in identity-based approaches to signing creates a pressing need to evaluate how usability concerns affect adoption and effectiveness. A usability evaluation can clarify the extent to which identity-based designs succeed, highlight how usability compares to legacy key-managed signing approaches, and surface priorities for improvement. To this end, we offer the first empirical study of the usability of identity-based software signing tools. Our work examines Sigstore, a pioneer identity-based signing tool that has seen rapid uptake across major open-source ecosystems [50,61,84], cloud-native projects, and corporations [59,79 – 81,98], involving a large number of engineers worldwide. We interviewed 17 experienced security practitioners to investigate the usability concerns that drive or discourage Sigstore’s adoption. We adopted an exploratory data analysis approach, because this study addresses a new class of tools that has not previously been analyzed. Our analysis centered on two aspects: practitioners’ direct experiences with Sigstore and their perceptions of its usability relative to other signing tools. Our study highlights the usability factors practitioners weigh when adopting identity-based software signing tools. We show that while identity-based signing tools ease legacy key-managed challenges, adoption is still constrained by integration hurdles, privacy concerns, and organizational factors. Second, our results provide adopting organizations with in1 sight into the readiness levels of these tools, helping them manage adoption risks. Finally, we offer toolmakers and designers concrete directions for improving identity-based signing tools. Together, these findings provide formative feedback situated in real-world usage contexts. In summary, we make the following contributions: 1. We report on the difficulties and advantages of using Sigstore, a identity-based software signing tool. 2. We discuss how practitioners’ choices of software signing tools change over time and the factors influencing those decisions. 3. We discuss how organizational deployment contexts influence the usability concerns raised by practitioners. Significance: Our work contributes to the security of software signing tools and, more broadly, to software supply chains. By unpacking the usability factors that promote and hinder Sigstore’s adoption, we inform the ongoing refinement of its workflows and provide a template for evaluating and improving similar identity-based signing solutions. Because Sigstore exemplifies other identity-based software signing tools, our findings can be generalized to guide the design and implementation of this class of software, ultimately advancing usability (and thus security) across the software supply chain. 2 Background & Related Works In this section, we review software signing technologies (§2.1) and usability assessments (§2.2). 2.1 Software Signing Software signing ensures the authorship and integrity of a software artifact by linking a maintainer’s cryptographic signature to their software artifact. From literature, the implementation of software signing is shaped by several factors: the software artifacts to be signed (e.g., driver signing [14]), the root-oftrust design (e.g., public key infrastructure/PKI [104]), and ecosystem policies (e.g., PyPi [90] and Maven [83]). 2.1.1 Software Signing Tools Software signing tools can be broadly grouped into two families [92]. Legacy key-managed tools such as GPG [1] follow a conventional public-key model, relying on long-lived asymmetric key pairs and delegating key-management responsibilities to the signer; trust is typically established through endorsements by other signers. In contrast, identity-based signing tools (described as “identity-basederation signing” by Schorlemmer et al. [92]) automate key management by issuing short-lived, ephemeral certificates tied to a signer’s identity via external OpenID Connect (OIDC) or OAuth providers. This “keyless signing” approach, exemplified by Sigstore [103], OpenPubKey [15], and SignServer [77], reduces long-term key handling and often embeds transparency and auditability through append-only logs. We discuss the architecture and workflow of identity-based signing in §4.2 using Sigstore as a case study. 2.1.2 Tooling Effects on Signing Adoption Previous studies [29,57,91] recognize that tooling influences whether and how software signing is adopted. For example, Schorlemmer et al. [91] perform a measurement study of several software package registries, identifying tooling as a key determinant of signature quality; their results indicate that “dedicated tools” lead to higher-quality software signatures. Kalu et al. [57] examine the implementation and adoption of signing in real-world organizational settings, highlighting tool usability as a central technical challenge for using software signing in practice. Their study focuses on how organizations incorporate signing into their existing software delivery processes, without centering on any particular tool. Although these works highlight tooling, and, in the case of Kalu et al., usability, as important concerns, they do so only at a high level and provide little concrete guidance on what makes a signing tool usable or unusable in practice. In this study, by contrast, we examine the specific usability of identity-based signing tooling, using Sigstore as a case study. We concretize formative usability concerns for Sigstore and extend Kalu et al.’s [57] work by specifying which features and design choices make identity-based signing tools like Sigstore usable or unusable in practice, and by translating these findings into concrete improvement guidance. 2.2 Usability Studies in Software Signing 2.2.1 Usability — Definition and Evaluation Approaches Tools enable engineers to adopt new software development practices [88]. A usable tool is one whose functionality effectively supports its intended purpose, achieving product goals [63,64]. Studies assessing implementation strategies of tools are typically framed as usability evaluations [3]. To evaluate a tool’s usability, studies typically follow a usability evaluation framework [3]. There are two kinds of approaches. Summative work measures a tool’s effectiveness [11,54], e.g., by task success rates, completion time, and user satisfaction. Formative assessments provide feedback to guide ongoing improvement [63,96]. Given our aims, we took a formative approach. We considered several formative frameworks [60,89,93,97] and found that Cresswell et al.’s four-factor framework [56] best explained our data. That framework is summarized in Figure 1. Usability depends on the context in which a tool is used. Many factors, such as organizational objectives, personnel, 2 policies, and compliance requirements, can make a tool usable in one setting and impractical in another [30,99]. This contextual aspect informs our study design. 2.2.2 Empirical Studies of Signing Usability The usability of legacy key-managed signing tools, often studied in the context of encryption, remains a significant challenge. Whitten and Tygar’s seminal study, “Why Johnny Can’t Encrypt” [99], found PGP 5.0’s interface too complex for nonexperts, a problem that persisted in PGP 9 [75], especially around key verification, transparency, and signing. Subsequent work [9,17,74] has underscored the public-key model’s inherent complexity. Several works have subsequently examined the use and automation of signing in specific domains, such as social media conversations [23,37,39]. These prior usability studies of signing tools are limited in applicability to software signing. Much of these works have focused on communication contexts such as email and message encryption, where users have less technical expertise [28,58] than software engineers securing their software supply chains. These studies also examine only legacy keymanaged signing technologies. We contribute the first usability study of identity-based software signing tools. 3 Knowledge Gaps & Research Questions Our literature review shows a gap in formative assessments of software signing tools. Moreover, no study has examined identity-based signing tools, which are structurally and functionally different from legacy key-managed signing tools. We ask: RQ1: What usability strengths and weaknesses of identitybased software signing tools shape their adoption in practice? RQ2: What factors drive practitioners to adopt, retain, or replace software signing tools over time? 4 Methodology To investigate our research questions, we adopted a case study approach. Our overall methodology is summarized in Figure 1. This section details our research design, including the study rationale (§4.1), case study context (§4.2), instrument design (§4.3), data collection (§4.4), and analysis procedures (§4.5). Our Institutional Review Board (IRB) approved this work (IRB-2023-841). 4.1 Rationale for Case Study We selected the case study method for our research because it allows for an in-depth exploratory examination of our chosen case, enabling a deeper understanding of the usability factors that influence the adoption of a software signing tool in practice [35,46,73,102], rather than a broad but superficial cross-analysis [70]. In selecting Sigstore, we adhered to the criteria outlined by Yin [102] and Crowe et al. [35]. Specifically, we considered the intrinsic value and uniqueness of the case study unit (high), access to data (adequate), and risks of participation (low). In §6we discuss generalizability. 4.2 Case Study Context – Sigstore Following guidelines outlined by Runeson & Höst [73], we describe the importance of our selected case study context and the technical details of the system. 4.2.1 Sigstore in Practice Sigstore provides keyless signing, where developers authenticate using existing OIDC identities (e.g., GitHub, Google) instead of generating and distributing long-lived keys. It issues short-lived (ephemeral) certificates that simplify key management and reduce long-term security risks compared to legacy key-managed signing tools. Within 13 months of its public release in 2021, Sigstore recorded 46 million artifact signatures [50]. It has since gained broad adoption across open-source ecosystems (e.g., PyPI [61], Maven [84], NPM [82]) and major companies (e.g., Autodesk [81], Verizon [80], Yahoo [101]). Given its rapid uptake, integration into critical ecosystems, and pioneering design features, Sigstore provides a compelling case study for examining the usability of identity-based software signing tools. 4.2.2 Sigstore Components & Workflow Following Newman et al. [103], we outline Sigstore’s architecture, components, and workflow to orient the reader. • Cosign: Command-line and library support (e.g., cosign, sigstore-java) for creating, verifying, and bundling signatures and attestations. Integrations may differ across ecosystems (e.g., GitHub Actions vs. Maven plugins). • Identity Provider: Services such as GitHub or Google that authenticate signers and issue short-lived identity tokens (e.g., via OpenID Connect). • Fulcio (Certificate Authority): Issues ephemeral signing certificates after verifying the signer’s OIDC token. • Rekor (Transparency Log): Append-only ledger that records each signing event and certificate issuance. • Monitors and Verifiers: Tools that validate the signer’s identity, certificate status, and signature integrity. 3 Figure 1: Study Methodology and Usability Framework. We developed our interview protocol by reviewing academic and grey literature on software signing, then conducted 17 practitioner interviews. Data were analyzed through thematic analysis and mapped onto Cresswell’s formative usability evaluation framework [56]. The framework organizes usability concerns into four factors: Technological (T) — technical characteristics such as performance, ease of use, and data integrity; Social/Human (P) — user experience, and correct use of features; Organizational (O) — internal factors such as training, leadership, and resource support; and Macroenvironmental (M) — external influences including regulation, economics, and ecosystem community. Figure 2: Sigstore Signing Workflow. The software author requests a certificate from a certificate authority, which confirms the signer’s identity through an identity provider. The signature and signer’s identity are recorded in a transparency log, which the verifier can monitor to confirm the validity of the signature upon downloading the signed package. Figure 2summarizes the Sigstore software signing workflow. First, the signer authenticates with an OIDC provider to obtain a short-lived token. Next, the certificate authority Fulcio verifies this token and issues a one-time signing certificate. The signer then uses Cosign to apply that certificate to the software artifact. Finally, Rekor logs the certificate and signature, and verifiers retrieve entries from both Fulcio’s log and Rekor to confirm the signer’s identity and ensure the certificate and signature remain valid. 4.2.3 Commonality with other Identity-Based Tools Sigstore, like other identity-based signing tools such as OpenPubKey [15], AWS Signer [4], and SignServer [77], shares core functionality in signer identity management and automated key handling. However, unlike Sigstore, most of these tools do not include a built-in certificate authority (e.g., OpenPubKey) or a transparency log (e.g., OpenPubKey, SignServer); instead, such services must be added via external integrations. We therefore use Sigstore as a focal case, while expecting many of our observations to transfer to similar identity-based signing tools. 4.3 Instrument Design & Development Per §2.2, our study is a formative usability study. This type of evaluation offers methodological flexibility and prioritizes input from experienced practitioners to better articulate a system’s usability concerns and produce actionable guidance [63]. To elicit long-form, context-rich opinions and detailed observations, we employed semi-structured interviews to examine various aspects of tooling usability (e.g., implementation challenges, perceived benefits, and comparisons to other signing tools), following guidelines from Saldaña [76]. This approach allowed us to pose a consistent core of questions across all participants while probing unique aspects of each participant’s circumstances. Sigstore is an emerging (though maturing) technology, and case study research recommends exploratory designs in such contexts [73,102]. To develop our interview protocol, we therefore pursued open, inductive inquiry to capture the full range of practitioner usage experiences. After this exploratory phase, we mapped our findings onto a formal usability framework, ensuring it reflected practitioners’ experiences rather than constraining data collection to preconceived categories (§4.5). Therefore, in constructing the protocol, we: 1. Reviewed grey literature, Sigstore blogs [78] and usage reports [7,51,59,79 – 81,94,98], to ground our questions in real-world implementations and updates. 2. Used a snowball review of top papers on signing primitives [29,103] and relevant empirical studies (see §2) to identify gaps and avoid redundant questions. We seeded the review with works from prominent cybersecurity and 4 Table 1: Summary of interview protocol. We present our full protocol in Appendix B. Topic Sample Questions A. Demographics What is your role in your team? B. Contextual Information (Risks & Use of Signing) Can you describe any specific software supply chain attacks (e.g., incidents with 3rd-party dependencies, code contributors, OSS) your team has encountered? How does the team use software signing to protect its source code (what parts of the process is signing required)? C. Signing Tool Usability Questions C1. What factors did the team consider before adopting this tool/method (Sigstore) over others? C2. What was the team’s previous signing practice/tool before the introduction of Sigstore? C3. How does your team implement Sigstore (which components of Sigstore does the team mostly use)? C4. Have you encountered any challenge(s) using this tool of choice? C5. Have you/your team considered switching Sigstore for another tool? software engineering venues (e.g., USENIX Security, IEEE S&P, ICSE, FSE) published in the past 10 years. Next, we refined our interview protocols through a series of initial interviews [12]. We first conducted practice interviews with the secondary authors of this work, followed by pilot interviews with the first two interview subjects. We modified our protocol after practice and pilot, affecting 8 questions1. Table 1summarizes our final interview protocol 2 . After the demographic section (Topic A), our interview instrument consisted of two main topics (B & C). Topic B comprised contextual questions about the supply chain risks that prompt signing and how signing is implemented across the software production process (e.g., where signing is required vs. optional, which artifacts are signed, and relevant workflow/policy constraints). Topic C presented questions to assess the usability of the signing tool(s) used by the participant’s organization, including perceived strengths and weaknesses, adoption rationale, implementation details, challenges, and any considerations about switching tools. 4.4 Data Collection Population: Our study investigates signing tools using a case study of the Sigstore tool ecosystem. Accordingly, our participant pool focused on current and prospective users of this tool. Furthermore, since our research questions pertain to organizational decision-making, we targeted expert practitioners who either oversee Software signing compliance or manage the infrastructure of their respective teams or organizations. 1Details are in artifact Appendix B 2 Questions in the protocol reflect our screening criteria (expert users of signing infrastructure), with later questions building on earlier answers. Table 2: Demographics. For Role, “Senior management” are senior managers/directors/executives; “Technical leader” are senior/lead/partner/principal engineers; and “Engineer” and “Manager” are junior staff. Table 10 gives organizational info. ID Role Experience Software Type P1 Research leader 5 years Internal POC software P2 Senior mgmt. 15 years SAAS security tool P3 Senior mgmt. 13 years SAAS security tool P4 Technical leader 20 years Open-source tooling P5 Engineer 2 years Internal security tooling P6 Technical leader 27 years Internal security tooling P7 Manager 6 years Security tooling P8 Technical leader 8 years Internal security tooling P9 Engineer 2.5 years SAAS security P10 Engineer 13 years SAAS security P11 Technical leader 16 years Firmware P12 Technical leader 4 years Consultancy P13 Senior mgmt. 16 years Internal security tool P14 Research leader 13 years POC security software P15 Senior mgmt. 15 years Internal security tooling P16 Senior mgmt. 15 years SAAS P17 Manager 11 years Security tooling To recruit this target population, we employed a nonprobability purposive and snowball sampling approach [5]. We leveraged the personal network of one of the authors to send an initial invitation to members of the 2023 KubeCon (hosted by the Linux Foundation) organizing committee, who were also part of the In-toto [40,53] Steering Committee (ITSC). This initial invitation yielded 6 participants. We then expanded our recruitment through snowball sampling, relying on recommendations from initial participants (4) and authors’ contacts (7) to recruit an additional 11 participants. We report on data from 17 participants.3 4 Participant Demographics: Our subjects were experienced security practitioners responsible for initiating or implementing their organization’s security controls or compliance. This gives them the relevant context to assess their organization’s strategies for adopting software signing tools. Subjects came from 13 distinct organizations, all companies. Relevant demographics are in Table 2. Thirteen participants reported their organizations use Sigstore. Two use internally developed tools and two rely on Notary v1 and PGP signing. Those four participants enrich our findings, as their experiences shed light on usability factors that hinder adoption of sigstore by non-users. Interviews: We conducted our interviews over Zoom. Each interview lasted ∼ 50 minutes. The lead author conducted these interviews. We offered each subject a $100 gift card as an incentive in recognition of their expertise. Survey Attempt: We recognize the importance of data triangulation in case studies and therefore fielded a follow-on survey to complement our interviews. We deployed a survey to validate and model the prevalence and stability of our in3 We conducted 18 interviews. Participant 18 was a senior security expert. While aware of the organization’s overall use of signing, they reported limited knowledge of the signing tool’s specific operation and deployment. 4 POC: Proof of concept software: early-stage software that may be productized 5 terview themes over time and across contexts. We tested our survey instrument in two practice runs with two members of our research team. Despite five months of repeated solicitation in the Sigstore community, we received only 13 completed responses, of which 7 were sufficiently complete for analysis (see §9). This regrettably echoes the well-documented difficulty of recruiting participants from highly specialized populations [86]. Data triangulation is encouraged but not mandatory in case study research. As Stake notes, combining multiple data sources strengthens the substantiation of constructs and hypotheses, yet practical constraints necessarily shape feasible designs [85]. We encountered such constraints. 4.5 Data Analysis We analyzed data in two stages: an interpretive–exploratory thematic analysis, and then a retrospective mapping of emergent themes onto a formative usability framework. First, we adopted an interpretive and exploratory [36] approach for initial analysis, as recommended for studies characterized by substantial uncertainty [72]. In our case, this uncertainty concerned how organizations adopt software signing as a security method, a decision that varies with organizational policies, security expectations, perceived threats, and customer requirements. This orientation motivated the use of a reflexive thematic analysis lens [8]. At the same time, we aimed to generate concrete, usable guidance for tool designers. To increase the systematicity and transparency of our process, we therefore incorporated selected coding-reliability practices, structured early coding, and agreement checks [69]. In short, reflexivity grounds our insights, and these reliability practices help make them more communicable and actionable. Second, because our work is a usability study, we situated the exploratory findings in a usability context by applying a formative usability framework retrospectively to our inductively generated results. This retrospective mapping of emergent themes onto existing constructs is an established interpretive strategy in qualitative analysis. This method uses prior constructs to structure and extend inductively derived findings while preserving empirical grounding (e.g., [19,31,43]). Memoing & Codebook Creation. Two analysts participated in this analysis stage. After transcription 5 and anonymization, we began with data familiarization [24]. During Familiarization, the main analyst 6 read all interviews, and the secondary analyst read three transcripts to calibrate. Next, the two analysts independently memoed a subset of 6 anonymized transcripts through three rounds of review (2 transcripts per round) and discussion, following the recommendations of O’Connor & Joffe [69] and Campbell et 5 Interview recordings were transcribed by www.rev.com using human transcription to address spoken accents. 6The lead author is the main analyst in this study. al. [55]. This process improved reliability of the analysis while making efficient use of resources. 7 After each round, we discussed emerging codes in meetings, which produced 506 coded memos in total. We distilled these memos into a codebook (36 categories) over several multi-hour meetings. Next, to assess the reliability of the coding process, we combined the techniques outlined by Maxam & Davis [52] and Campbell et al. [55]. Specifically, we randomly selected an unanalyzed transcript, which we coded independently. Following Feng et al.’s [42] recommendation, we used the percentage agreement measure 8 to evaluate consistency, as both analysts had contributed to developing the codebook. We obtained an 89% agreement score, suggesting a stable coding scheme [55,69]. We resolved disagreements by discussion. Codebook Revision. Following the reliability assessment of our codebook and given the substantive agreement recorded, the main analyst proceeded to code the remaining transcripts in two batches of six and five transcripts per round, respectively. The lead analysts revisited earlier coded transcripts throughout . In the first batch, six new code categories were introduced, which were subsequently reviewed by the other analyst. The same analytical process was applied to the final five transcripts, leading to the addition of four new code categories, which were subsequently reviewed by the other analyst. Notably, of the 10 codes added during this phase of codebook refinement, only three provided new insights into the data. By mutual agreement between both analysts, these were categorized as minor codes. To assess the adequacy of our sample size and analysis, we calculated code saturation for each category in our codebook. Following the recommendations of Guest et al. [45], we measured the cumulative number of new codes introduced in each interview (Figure 8). We saw saturation after interview #12. Each interview yielded a median of 15 codes. Thematic Analysis & Usability Analysis Next, following Braun & Clarke [8], we derived initial themes from our code categories (see Table 1for protocol focus). This reflexive thematic analysis provided an inductive account of participants’ experiences and practices. The lead analyst generated the themes and then discussed and refined them with feedback from the secondary analyst. From the emergent themes, we interpreted our findings using a formative usability framework. Lacking formative usability studies set in similar contexts in the Software Engineering literature, we followed advice [22,41,44] to seek theory from adjacent disciplines. Surveying candidate frameworks beyond SE, we found three groups: formative usability assessments of software systems [89]; applications of technology 7 Although there is no consensus on the ideal proportion of data to use, O’Connor and Joffe note that selecting 10–25% is typical [69]. We randomly selected six transcripts ( ∼ 35% of our dataset) to develop our initial codebook. 8(matching code assignments / total code assignments) × 100 6 Figure 3: Self-Reported Evolution of Software Signing Tools. Sankey diagram showing participants’ self-reported transitions between previously used and current signing tools in their organizations. Flows (e.g., from no signing to Sigstore) reflect participant accounts, not our observations. systems in other disciplines [56,60,97]; and general usability frameworks [93]. Among these, we selected Cresswell et al.’s four-factor TPOM framework (Technology, People, Organization, Macro-environment) [56] because it best aligned with the shape of our inductively generated themes (see Figure 1). Our intent was to synthesize those themes into higher-level conceptual groupings that correspond to established usability determinants of adoption. The TPOM categories also suggest interventions, e.g., People issues may not resolve via Technology. We then conducted a framework-informed analysis, mapping the emergent themes to the TPOM factors to refine and structure our interpretations. The lead analyst mapped our earlier generated themes and then discussed and refined the mappings with feedback from the secondary analyst. 5 Results We begin by presenting a Sankey diagram (Figure 3) that illustrates the distribution of our participants between Sigstore and non-Sigstore users, and how their teams changed signing tool usage over time. Many participants (6) transitioned from using no tools to adopting Sigstore. Others transitioned from existing tools to Sigstore: 3 from implementations that had some combinations of GPG/PGP (of which one practitioner reported combining PGP with Skopeo [71]), 2 from Notary, and 2 from proprietary or internal tools. 4 subjects did not adopt Sigstore. To understand the usability concerns that Sigstore adopters face; and why practitioners did (and did not) change tools, we structure the results by RQ using Cresswell’s framework. Table 3: Summary of Practitioner-Reported Advantages of Using Sigstore. Practitioners cited Sigstore’s technological and macroenvironmental factors as advantages. Topics & Associated Examples Subjects Technological Factors Ease of Use 8 subjects 1. Signing Workflow & Verification P1, P2, P7, P9, P14-17 2. Setting up with automated CI/CD actions P9 3. No key distribution problems P1 Use of Short-lived Keys & Certificate 3 subjects P2, P3, P15 Signer ID Management 4 subjects 1. Use of OIDC(Keyless) to authenticate signers P3, P5, P10, P12 Compatibility with Several New Technologies 4 subjects 1. Integrability with SLSA build P16 2. Integrability with several container registries/technologies P12, P14 3. Integrability with several cloud-native applications P15 Precence of a Transparency Log 3 subjects 1. Transparency logs increase security P5, P14 2. Evaluation of signing adoption using logs P9 Bundling Signatures With Provenance Attestations 2 subjects P3, P4 Reliability of Service 1 subject P7 Macroenvironmental Factors Free/Open-Source 2 subjects P7, P17 5.1 RQ1: Strengths & Weaknesses of Sigstore The first aim of our research work was to ascertain from the the experiences of our subjects the strengths and weaknesses they have experienced while using Sigstore. 5.1.1 Sigstore Strengths We extracted the benefits our participants experienced while using Sigstore from their responses. These reflections capture their firsthand experiences with Sigstore. We grouped their responses into eight categories as summarized in Table 3. When mapped to the Cresswell framework, these factors fell into two categories: Technological and Macroenvironmental. Technological Factors We identified five factors: Ease of use: This was the most frequently discussed advantage of Sigstore over other tools. Participants commonly highlighted the simplicity of the Sigstore workflow. As P7 (security tools, cloud security) put it, “I don’t really know of any other options that provide the same conveniences that Sigstore provides... It really is just one command to sign something and then one more command to verify it, and so much of the work is handled by Sigstore.” Other ease-of-use advantages mentioned were the seamless integration of the Sigstore workflow into CI/CD automation using GitHub Actions and the reduced complexity of key management. Identity management for Signers: Sigstore’s integration of OIDC identity management was a strength per our participants. In legacy key-managed software signing, signer identity is often an ad-hoc feature and may be optional for the signing authority. In Sigstore, however, this behavior is integrated with the Fulcio certificate authority as an identity management step before creating a signature. Participants 7 greatly valued the elimination of the key generation and distribution phase, as it reduced the time spent distributing keys to key servers and searching for keys, streamlining the signing process. This aligns with the intuition that identity management is easier than key management for developers [103]. P16 (SAAS, SSC security tool) highlights this point: “I can associate my identity with an OIDC identity as opposed to generat[ing] a key and keep[ing] track of that key...I could say, ‘Oh, this is signed with [Jane]’s public GitHub identity’...that’s much better.” Compatibility with Several New Technologies: A portion of our participants reported that Sigstore’s integration and compatibility with a wide range of modern technologies provided a significant advantage. The mentioned technologies included security tooling such as SLSA build attestations, as well as containerization technologies and cloud computing applications. In most cases, the resources required to set up connections to these technologies were minimal or nonexistent. P12 (consultancy, cloud OSS security) states, “a lot of clients are getting into signing their containers... Sigstore is the easy choice because it just works on any registry, and integrating it to Kubernetes is easy.” Use of Short-lived Keys/Certificates: Participants appreciated Sigstore’s enforcement of short-lived keys and certificates. Some signing tools require users to manually set expiration times for cryptographic elements such as keys and certificates at the time of generation. That customizability also introduces the risk of long-lived cryptographic materials, increasing the likelihood of compromise. Participants preferred Sigstore’s automated management of key and certificate lifetimes. As P2 (SAAS, SSC security tool) stated, “PGP, you still have to figure out the key distribution problem....Sigstore uses short-lived keys, that solves a lot of those issues.” Presence of a Transparency Log: Adopters valued the ability to audit signing actions. Sigstore’s transparency log provides a tamper-resistant record of all changes to an artifact’s metadata, a functionality not offered by other signing systems. P9 (SAAS, SSC security): “From a higher level, we’re going to use a transparency log to determine what was actually signed and what was written into that transparency log.” Macroenvironmental Factors Free/Open-Source: Sigstore’s open-source nature was also highlighted by participants as a key advantage. They noted that this reduces both setup and maintenance barriers, particularly for users leveraging Sigstore’s public deployments. P7 (security tools, cloud security): “I think it’s amazing that Sigstore is free and public.” This enthusiasm reflects both the appeal of free/open-source (FOSS) software and the practical benefits of public accessibility. However, note that open-source = free; enterprises may pay for private deployments, support, and integration. Table 4: Summary of Practitioner-Reported Difficulties Using Sigstore. Difficulties are grouped by Cresswell factors. Topics & Associated Examples Subjects Technological Factors Transparency Log Constraints 6 subjects 1. Not suitable for private setup P2, P3, P6, P14, P17 2. Use in air-gapped/offline conditions P2, P3, P9 Performance & Rate Limits 6 subjects 1. Rate limiting problems P3, P7, P14, P15 2. Latency concerns P6 Integrations & Tooling 3 subjects 1. GitLab/Jenkins P9 2. Other unsupported technologies P16 3. Attestation storage P1 Documentation (Technical Aspects) 6 subjects 1. Private instance setup documentation P9, P12, P15 2. Other documentation / usage issues P1, P14, P17 Private Instance: Infra & Cost 2 subjects 1. Infra requirements & maintenance cost P5, P6 Fulcio / Timestamping Workflow 1 subject 1. Timestamping issues P3 2. Fulcio–OIDC workflow P3 Software Libraries 1 subject 1. Unsupported software libraries P7 Social / Human Factors Log Monitoring Burden 1 subject 1. Effort to monitor logs P2 Support & Maintenance 3 subjects 1. Lack of dedicated support & maintenance P15, P2, P6 Organizational Factors Regulatory Suitability (Cross-factor) 1 subject 1. Not suited for regulated organizations — M&O P3 Macroenvironmental Factors Community Maturity/Support 1 subject 1. Limited community support P15 5.1.2 Sigstore’s Weaknessses We summarize the difficulties reported by participants in using Sigstore in Table 4. Most of these weaknesses mapped to multiple usability factors. Macroenvironmental & Organizational The primary weakness along this dimension related to Sigstore’s use in large enterprise contexts. Enterprise Adoption Limitations: Many of our participants mentioned that Sigstore is not suited for large enterprise applications. The issues reported by participants include rate limiting—where Sigstore restricts the number of signatures that can be created per unit time—along with latency concerns, where the service’s turnaround time for a large volume of signatures is problematic. Additionally, participants highlighted the lack of dedicated support and maintenance teams due to Sigstore’s open-source nature, as well as its unsuitability for organizations in regulated sectors for the same reason. P7 (security tools, cloud security): “We were relying on the public Sigstore instance, and I don’t think it could meet our signing needs in terms of just the capacity of signatures we needed as an enterprise [rate limit].” P3 (SAAS, SSC security tool): “For customers that are very large in scale, the biggest of the Fortune 100, or customers operating in very highly regulated environments, I think 8 operating your own Sigstore instance within that air gap in a private environment could be valuable. And I think it depends on the amount of software you’re producing and the frequency with which you’re producing it.” Technological We identified seven technological issues. Transparency Log Issues: While the transparency log bundled with Sigstore was highlighted as a major advantage by participants, it was also a significant concern. Their concerns stemmed from the public nature of the log, where sensitive artifact metadata from company assets could be exposed in a publicly accessible record. Another common issue raised was the log’s usability in air-gapped [67] environments. Additionally, participants noted the manual effort required to continuously monitor and manage the log. Most of these participants were in the process of experimenting with setting up the transparency log at the time of the interviews. P6 (security tooling, digital technology): “We’ve got some teams piloting Rekor, for instance, for different traceability usages...I’m all for the transparency log, but we...have to be careful who it’s transparent to at what [time].” Setting up Private Sigstore Instance: Sigstore’s customizability was a feature frequently utilized by participants. However, it was also associated with certain challenges. The primary issues identified included limited documentation on setup, scarce community usage information, and the infrastructure requirements for private deployments. Issues here were primarily related to participants’ need for support resources, which were macroenvironmental and social in nature. P15 (internal security tooling, aerospace security): “There’s not a big enough community of practitioners, or help to...deploy Fulcio, like on premises.” Other Documentation Issues: Overall, the documentation for Sigstore, along with practical usage information and support, was found to be insufficient. Participants noted that updates to the documentation lagged behind changes in the Sigstore product and that there was a lack of clarity regarding the use of different Sigstore components. P17 (security tooling, internet service) highlights this, “I would say the documentation lacks a bit. Just I think if you look at the documentation, it was updated quite a long time ago, like the ReadMe file. And so if we could have these different options saying that, okay, if you are starting over, this is how you can do rekor and Fulcio.” Integration to Other Systems: Participants also expressed a desire for a wider range of compatible technologies beyond what Sigstore currently supports. Commonly mentioned integrations included other CI/CD platforms such as Jenkins and GitLab, as well as attestation storage databases. These limitations were often attributed to Sigstore being an emerging technology, as highlighted by P16 (SAAS, SSC security tool), “The weakness is not everything supports it. There’s a lot of weaknesses around...its being a newer technology.” Fulcio Issues: Some Sigstore issues were also tied to its certificate authority’s (Fulcio) time stamping capabilities, and implementation of OIDC for keyless signing. Fulcio acts as an intermediary using OIDC identities to bind to a short-lived certificate, this was criticized by P3. P3 also commented on the lack of timestamping support in early versions of Fulcio. P3 (SAAS, SSC security tool): “We do have some concerns about the way Fulcio operates as its own certificate authority. So we’ve been looking at things like OpenPubkey as something that removes that intermediary and would allow you to do identity-based signing directly against the OIDC provider. And then I think the other big thing that we did, because Sigstore originally didn’t support it, was time-stamping. We want to ensure that the signature was created when the certificate was valid.” Offline Capabilities: Another reported limitation of Sigstore is the absence of offline verification capabilities using offline keys. This issue is related to the transparency log’s inability to function in air-gapped environments. However, in this case, participants specifically desired the ability to use Sigstore’s signing and verification features in offline scenarios. Software Libraries: The programming libraries of Sigstore were also reported as a challenge, particularly when participants sought to embed signing capabilities directly into applications. They noted the absence of well-supported, easy-touse client libraries, which forced them to rely on workarounds such as shelling out to the CLI. 5.1.3 Impact of Issues on Sigstore Component Use We assessed the effect of participants’ reported problems with Sigstore on their usage of each component. By component, nine subjects use Cosign, nine use the Fulcio certificate authority, and nine use the OIDC keyless signing and identity manager. Eight use the Rekor transparency log. Only four use Gitsign and only two make use of customized components and local Sigstore deployments. Among these, our data shed the most light on local deployments and the Rekor log. For local deployment, several participants cited difficulties with documentation and setup. Only two successfully deployed private instances, likely due to these challenges (Table 4, §5.1.2). The Rekor transparency log presents a mixed case, with both high adoption and significant reported issues. Of the eight participants who used it, three were still in the pilot phase. Although Rekor was recognized as a key factor driving adoption due to its strengths, it also faced a high number of reported issues, indicating that, despite its benefits, challenges persist for some participants. Other Components Without Mention Though the participants in the survey shared experiences with user-facing components, discussion about other elements was notably absent. In particular, there was very little mention of log wit9 Open Science We acknowledge that USENIX Security has an open science policy: that authors are expected to openly share their research artifacts by default. The research artifacts associated with this study are: • Raw transcripts of interviews • Raw transcripts of interviews • Anonymized transcripts of interviews • Interview protocol • Codebook Things we have shared: The interview protocol is a crucial part of any interview study, since it allows for the critical review of a study design as well as its replication. Our full interview protocol is included in Appendix B. Since it is semi-structured, we include all of the questions asked of all subjects, as well as examples of the follow-up questions we asked. We also share the codebook, with codes, definitions, and example quotes that we coded for each code. Things we cannot share: For subject privacy reasons, and for IRB compliance, we cannot share the raw transcripts. We are also unwilling to share the anonymized transcripts of the interviews. Given the high organizational ranks of many of our subjects, and the small size of the subject pool resulting from our recruiting strategy, we believe there is a high risk of de-anonymization even of anonymized transcripts. Therefore, we cannot share the anonymized transcripts. Artifact Repository: An anonymized artifact containing our interview protocol, codebook, etc. is available at: https://github.com/nextgenusability/identitybased-usabilty References [1] The GNU privacy guard. https://www.gnupg.org/ , December 2024. [2] A. Ferraiuolo et al. Policy transparency: Authorization logic meets general transparency to prove software supply chain integrity. In ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses, 2022. [3] A. R. Lyon et al. The cognitive walkthrough for implementation strategies (CWIS): A pragmatic method for assessing implementation strategy usability. 2, 2021. [4] Amazon Web Services. AWS signer developer guide. https://docs.aws.amazon.com/signer/latest/ developerguide/Welcome.html , 2024. Accessed: 2025-05-30. [5] S. Baltes and P. Ralph. Sampling in software engineering research: A critical review and guidelines. Empirical Software Engineering, 2022. [6] S. Benthall. Assessing software supply chain risk using public data. In IEEE Software Technology Conference (STC), 2017. [7] H. Blauzvern. How Sigstore quickly patched an upstream vulnerability. https://blog.sigstore.dev/ how-sigstore-quickly-patched-an-upstreamvulnerability-76ba84ef1122, October 2022. [8] V. Braun and V. Clarke. Using thematic analysis in psychology. Qualitative Research in Psychology, 2006. [9] C. Braz and J. Robert. Security and usability: The case of the user authentication methods. In Conference on l’Interaction Homme-Machine. ACM, April 2006. [10] C. Okafor et al. SoK: Analysis of software supply chain security by establishing secure design properties. In ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses, 2022. [11] C. Rusu et al. User experience evaluations: Challenges for newcomers. In Aaron Marcus, editor, Design, User Experience, and Usability: Design Discourse. Springer, Cham, 2015. [12] R. J. Chenail. Interviewing the investigator: Strategies for addressing instrumentation and researcher bias concerns in qualitative research. Qualitative report, 16(1):255–262, 2011. [13] Cloud Native Computing Foundation. Software supply chain best practices, May 2021. https://tinyurl. com/v1cncf. [14] D. Cooper et al. Security considerations for code signing. NIST Cybersecurity White Paper, January 2018. [15] E. Heilman et al. OpenPubkey: Augmenting OpenID connect with user held signing keys, 2023. https: //eprint.iacr.org/2023/296. [16] A. Al Hadwer et al. A systematic review of organizational factors impacting cloud-based technology adoption using technology-organization-environment framework. Internet of Things, 15:100407, 2021. [17] A. Reuter et al. Secure email - a usability study. In Financial Cryptography and Data Security, Lecture Notes in Computer Science, pages 36–46, Cham, 2020. Springer International Publishing. [18] B. Xia et al. An empirical study on software bill of materials: Where we stand and the road ahead. In IEEE/ACM International Conference on Software Engineering (ICSE), May 2023. 16 [19] C. Carroll et al. “Best fit” framework synthesis: Refining the method. BMC Medical Research Methodology, 13(1):37, 2013. [20] C. L. Okafor et al. Diverify: Diversifying identity verification in next-generation software signing. arXiv preprint arXiv:2406.15596, 2024. [21] D. Cooper et al. Protecting software integrity through code signing. Technical Report ITL Bulletin, National Institute of Standards and Technology, May 2018. [22] D. I. K. Sjoberg et al. The future of empirical methods in software engineering research. In 2007 Future of Software Engineering (FOSE ’07). IEEE, 2007. [23] E. Atwater et al. Leading Johnny to water: Designing for usability and trust. In USENIX Conference on Usable Privacy and Security, SOUPS ’15, July 2015. [24] G. Terry et al. Thematic analysis. The SAGE handbook of qualitative research in psychology, 2, 2017. [25] H. S. Bansal et al. “Migrating” to new service providers: Toward a unifying framework of consumers’ switching behaviors. Journal of the Academy of Marketing Science, 33(1), 2005. [26] J. C. Davis et al. A guide to stakeholder analysis for cybersecurity researchers. arXiv preprint arXiv:2508.14796, 2025. [27] J. Cappos et al. The update framework (TUF). https: //theupdateframework.io , 2021. Accessed: 202505-30. [28] J. Kjeldskov et al. Does time heal? A longitudinal study of usability. In Australian Computer-Human Interaction Conference 2005 (OzCHI’05), 2005. [29] K. Merrill et al. Speranza: Usable, privacy-friendly software signing. In ACM SIGSAC Conference on Computer and Communications Security, 2023. [30] M. Lennartsson et al. Exploring the meaning of usable security–a literature review. Information & Computer Security, 29(4):647–663, 2021. [31] N. K. Gale et al. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Medical Research Methodology, 13:117, 2013. [32] P. Ladisa et al. SoK: Taxonomy of attacks on opensource software supply chains. In 2023 IEEE Symposium on Security and Privacy (SP), 2023. [33] R. Chandramouli et al. Strategies for the integration of software supply chain security in DevSecOps CI/CD pipelines. US Department of Commerce, National Institute of Standards and Technology, 2024. [34] R. Verdecchia et al. Threats to validity in software engineering research: A critical reflection. Information and Software Technology, 164, 2023. [35] S. Crowe et al. The case study approach. BMC Medical Research Methodology, 11:100, June 2011. [36] S. Easterbrook et al. Selecting Empirical Methods for Software Engineering Research, page 285–311. Springer, London, 2008. [37] S. Fahl et al. Helping Johnny 2.0 to encrypt his Facebook conversations. In The Eighth Symposium on Usable Privacy and Security, SOUPS ’12, July 2012. [38] S. Ruoti et al. Why Johnny still, still can’t encrypt: Evaluating the usability of a modern PGP client. arXiv preprint arXiv:1510.08555, 2015. [39] S. Ruoti et al. Private Webmail 2.0: Simple and easyto-use secure email. In Symposium on User Interface Software and Technology, UIST ’16, 2016. [40] S. Torres-Arias et al. in-toto: Providing farm-to-table guarantees for bits and bytes. In 28th USENIX Security Symposium (USENIX Security 19), August 2019. [41] T. Lorey et al. Social science theories in software engineering research. In International Conference on Software Engineering, ICSE ’22, July 2022. [42] G. C. Feng. Intercoder reliability indices: disuse, misuse, and abuse. Quality & Quantity, 2014. [43] J. Fereday and E. Muir-Cochrane. Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International Journal of Qualitative Methods, 2006. [44] D. M. Fernández and J. Passoth. Empirical software engineering: From discipline to interdiscipline. Journal of Systems and Software, 148:170–179, 2019. [45] G. Guest et al. How many interviews are enough? An experiment with data saturation and variability. Field methods, 2006. [46] J. Gerring. What is a case study and what is it good for? American Political Science Review, 98(2):341–354, May 2004. [47] Matthew Green and Matthew Smith. Developers are not the enemy!: The need for usable security apis. IEEE Security Privacy, 14(5), 2016. 17 [48] H. Robinson H. Sharp and M. Woodman. Software engineering: community and culture. IEEE Software, 2000. [49] A. Hicks. SoK: Log based transparency enhancing technologies, 2023. https://arxiv.org/abs/2305. 01378. [50] L. Hinds and H. Blauzvern. Sigstore: Simplifying code signing for open source ecosystems. https: //openssf.org/blog/2023/11/21/sigstoresimplifying-code-signing-for-open-sourceecosystems/, November 2023. [51] J. Hutchings. Safeguard your containers with new container signing capability in github actions. https://github.blog/2021-12-06-safeguardcontainer-signing-capability-actions/ , December 2021. [52] W. P. Maxam III and J. C. Davis. An interview study on third-party cyber threat hunting processes in the US Department of Homeland Security. USENIX Security, 2024. [53] in toto and The Linux Foundation. in-toto: A framework to secure the integrity of software supply chains. https://in-toto.io/, 2023. [54] International Organization for Standardization. ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs) — part 11: Guidance on usability specification and measures, 1997. [55] J. L. Campbell et al. Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement. Sociological methods & research, 42(3):294–320, 2013. [56] K. Cresswell et al. Developing and applying a formative evaluation framework for health information technology implementations: Qualitative investigation. J Med Internet Res, 22(6), 2020. [57] K. Kalu et al. An industry interview study of software signing for supply chain security. In USENIX Security Symposium, 2025. [58] K. MacDorman et al. An improved usability measure based on novice and expert performance. International Journal of Human–Computer Interaction, 27(3), 2011. [59] F. Kammel. Signing and securing confidential Kubernetes clusters in the cloud with sigstore, August 2022. https://tinyurl.com/confidentialkubernetes. [60] L. Strifler et al. Development and usability testing of an online support tool to identify models and frameworks to inform implementation. BMC Medical Informatics and Decision Making, 24(1):182, 2024. [61] S. M. Larson. Python and Sigstore. https: //sethmlarson.dev/python-and-sigstore , 2024. Accessed: 2025-05-30. [62] Valentina Lenarduzzi et al. Open source software evaluation, selection, and adoption: a systematic literature review. In Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2020. [63] J. R. Lewis. Usability: Lessons learned...and yet to be learned. International Journal of Human-Computer Interaction, 30(9), 2014. [64] M. A. Sasse and I. Flechais. Usable security: Why do we need it? How do we get it? O’Reilly, 2005. [65] J. C. Mankins. Technology readiness assessments: A retrospective. Acta Astronautica, 65(9-10), 2009. [66] Del Nagy et al. Organizational adoption of open source software: barriers and remedies. Commun. ACM, 53(3):148–151, March 2010. [67] National Institute of Standards and Technology. Air gap. https://csrc.nist.gov/glossary/term/ air_gap, 2025. Accessed: 2025-05-30. [68] OpenPubKey. OpenPubKey: Issues. https: //github.com/openpubkey/openpubkey/issues , 2025. GitHub issues page. [69] C. O’Connor and H. Joffe. Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 2020. [70] P. Ralph. ACM SIGSOFT Empirical Standards Released. ACM SIGSOFT Software Engineering Notes, 2021. [71] Red Hat. What is Skopeo? https://www.redhat. com/en/topics/containers/what-is-skopeo , July 2022. Published July 18, 2022. [72] B. H. Rowlands. Grounded in practice: Using interpretive research to build theory. Electronic Journal of Business Research Methods, 3(1):pp81–92, 2005. [73] P. Runeson and M. Höst. Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14(2), 2009. [74] S. Ruoti et al. Confused Johnny: When automatic encryption leads to confusion and mistakes. In Ninth Symposium on Usable Privacy and Security, SOUPS ’13, pages 1–12, July 2013. 18 [75] S. Sheng et al. Why Johnny still can’t encrypt: Evaluating the usability of email encryption software. In Symposium on usable privacy and security. ACM, 2006. [76] J. Saldana. Fundamentals of qualitative research. Oxford university press, 2011. [77] SignServer Project. About SignServer: Open - Source Signing Software. https://www.signserver.org/ about/, 2025. Accessed 26 May, 2025. [78] Sigstore. Sigstore: A new standard for signing, verifying, and protecting software. https://www. sigstore.dev/. [79] Sigstore. Securing your software supply chain without changing your devops workflow. https://blog.sigstore.dev/securing-yoursoftware-supply-chain-without-changingyour-devops-workflow-e23393a5fffa , December 2022. [80] Sigstore. Security by default: How Verizon new business incubation uses Sigstore to demonstrate provenance. https://blog.sigstore.dev/securityby-default-how-verizon-new-businessincubation-uses-sigstore-to-demonstrateprovenance-7beed5714738, November 2022. [81] Sigstore. Using Sigstore to meet FedRAMP compliance at Autodesk. https://blog.sigstore. dev/using-sigstore-to-meet-fedrampcompliance-at-autodesk-6f645a920abc , November 2022. [82] Sigstore Blog. Announcing NPM support: Public beta. https://blog.sigstore.dev/npm-publicbeta/, 2024. Accessed: 2025-05-30. [83] Sonatype. Working with PGP signatures. https://central.sonatype.org/publish/ requirements/gpg/. Accessed: 2025-05-30. [84] Sonatype. Central publisher portal now validates Sigstore signatures. https://www.sonatype. com/blog/central-publisher-portal-nowvalidates-sigstore-signatures , 2024. Accessed: 2025-05-30. [85] R. E. Stake. The Art of Case Study Research. SAGE Publications, Thousand Oaks, CA, 1995. [86] C. G. Steeh. Trends in nonresponse rates, 1952–1979. Public Opinion Quarterly, 45(1), Spring 1981. [87] Synopsys. 2024 open source security and risk analysis (OSSRA) report, 2024. https: //www.synopsys.com/software-integrity/ resources/analyst-reports/open-sourcesecurity-risk-analysis.html#introMenu. [88] T. Bruckhaus et al. The impact of tools on software productivity. IEEE Software, September 1996. [89] T. C. Chan et al. A practical usability study framework using the SUS and the affinity diagram: A case study on the online roadshow website. Pertanika Journal of Science & Technology, 30(2), 2022. [90] T. Kuppusamy et al. PEP 480 – Surviving a compromise of PyPI: End-to-end signing of packages, oct 8 2014. https://peps.python.org/pep-0480/. [91] T. R. Schorlemmer et al. Signing in four public software package registries: Quantity, quality, and influencing factors. In IEEE Symposium on Security and Privacy (S&P), May 2024. [92] T. R. Schorlemmer et al. Establishing provenance before coding: Traditional and next-generation software signing. IEEE Security & Privacy, IEEE S&P Magazine ’25, (01), 2025. [93] T. S. Andre et al. Testing a framework for reliable classification of usability problems. In The Human Factors and Ergonomics Society Annual Meeting, volume 44. SAGE Publications, 2000. [94] The Sigstore Technical Steering Committee. Sigstore support in NPM launches for public beta. https: //blog.sigstore.dev/npm-public-beta/ , April 2023. [95] The White House. Executive order on improving the nation’s cybersecurity, May 2021. https://www. whitehouse.gov/briefing-room/statementsreleases/2021/05/12/executive-order-onimproving-the-nations-cybersecurity/. [96] Mary Theofanos and Whitney Quesenbery. Towards the design of effective formative test reports. J. Usability Studies, 1(1):27–45, November 2005. [97] V. Shah et al. Is my MOOC learner-centric? A framework for formative evaluation of MOOC pedagogy. International Review of Research in Open and Distributed Learning, 24(2):138–161, 2023. [98] S. J. Vaughan-Nichols. Kubernetes adopts Sigstore for supply chain security. https: //thenewstack.io/kubernetes-adoptssigstore-for-supply-chain-security/ , May 2022. [99] A. Whitten and J. D. Tygar. Why Johnny can’t encrypt: A usability evaluation of PGP 5.0. In USENIX security symposium, volume 348, 1999. 19 [100] M. Willett. Lessons of the SolarWinds hack. In Survival April–May 2021: Facing Russia, pages 7–25. Routledge, 2023. [101] Yahoo Inc. Scaling up supply chain security: Implementing Sigstore for seamless container image signing. https://www.yahooinc. com/paranoids/scaling-up-supply-chainsecurity-implementing-sigstore-forseamless-container-image-signing , 2023. Accessed: 2025-05-30. [102] R. K. Yin. Case study research and applications: Design and methods. SAGE, Los Angeles, sixth edition edition, 2018. [103] Z. Newman et al. Sigstore: Software signing for everybody. In ACM SIGSAC Conference on Computer and Communications Security, 2022. [104] V. Zimmer and M. Krau. Establishing the root of trust. https://uefi.org/sites/default/files/ resources/UEFI%20RoT%20white%20paper_ Final%208%208%2016%20(003).pdf , 2016. Accessed 26 May, 2025. Outline of Appendices The appendix contains the following material: • Appendix A: Traditional Software Signing Workflow. • Appendix B: The interview protocol. • Appendix C: The codebooks used in our analysis, with illustrative quotes mapped to each code. • Appendix D: Code Saturation Analysis. • Appendix E: Other Results Omitted from the Paper Due to Space Concerns. • Appendix E.1: Sigstore Components Mostly Used by our Sample Population. • Appendix E.2: Other Effects of Organization and Signing Tools on Identified Usability Concerns Continued from §5.2.3 • Appendix E.3: Expanded Demographic Table. • Appendix E.4: Summary of Factors Influencing Sigstore Adoption for Sigstore Users (Prior to Adoption). • Appendix E.5: Summary of Factors Influencing Consideration of Sigstore Amongst Non-Sigstore Users. • Appendix E.6: Additional Discussions & Implications of Results. • Appendix F: Survey results, omitted from the main paper due to low-quality data. Figure 6: Typical workflow for software signing and verifying signatures. The component author packages (A) and signs (B) their software. The signed package (C) and public key (D) are published. To use a package, a user downloads it (E) and its public key (F) and verifies the signature (G). A Traditional Signing Tools Workflow The signature creation process begins when the maintainer’s artifact is ready for submission. The signer generates a key pair that provides them with a private key for signing their software artifact and a public key for a verifier to validate the artifact’s signature. Once the artifact is signed, it must be submitted to a package registry (such as PyPi, npm, or Maven) or the intended party. During this phase, the signer is responsible for making their public key available and discoverable so future users can verify the signer’s identity. In the verification phase, the verifier retrieves the signer’s public key, uses it to decrypt the signature file, and checks for equivalency with the software artifact’s hashed digest. We show a typical legacy key-managed package signing workflow in Figure 6 compared to a identity-based workflow in Figure 7. B Interview Protocol Here we present the full protocol, see Table 6. We indicate the structured questions (asked of all users) and examples of follow-up questions posed in the semi-structured style. Given the nature of a semi-structured interview, the questions may not be asked exactly as written or in the same sequence, but may be adjusted depending on the flow of the conversation. Section B provides context for interpreting our results in terms of organizational cases. Section C gives our main questions for this study(and they may also include follow-up questions). 20 Table 6: Interview Protocol. Sections A–C reflect demographics, signing use/motivation context, and signing tool usability. We analyze part C in this study and use the other themes as context. Theme Description / Questions A: Demographics We ask basic demographic questions to establish participant experience and context (role, years of experience, team size, and major products). A-1: What best describes your role in your team? (Security engineer, Infrastructure, Software engineer, etc.) A-2: What is your seniority level? How many years of experience? A-3: What is the team size? A-4: What are the team’s major software products/artifacts? B: Signing Use & Context We ask about the organizational context of signing: why and how signing is used, what motivates adoption, what infrastructure supports it, and how it fits into broader supply chain security practices. Focus is on general practices, not tooling specifics. B-1: Why did the team choose to implement software signing? B-2: Is software signing the team’s major strategy to secure its supply chain? Any complementary efforts? B-3: Are these strategies influenced by regulations or standards? B-4: What challenges did your team face while integrating signing into supply chain practices? B-5: How do team members contribute to the implementation of signing? Roles/responsibilities? B-6: Do you believe signing (if implemented correctly) is sufficient to secure a supply chain? B-7: How does the presence of a signature influence dependency selection? B-8: How is authenticity of a signature verified in your process? B-9: What influences your team to discontinue a dependency? Does signature status play a role? B-10: How does your team manage third-party dependency security and vulnerabilities? B-11: Is signing a requirement for developers and contributors? If so, in which parts of the workflow? B-12: How is signing applied in your build process? B-13: How is signing applied to artifacts and deployment (binaries, repositories, pipelines)? C: Signing Tool Usability We ask about the signing tools themselves: what tools are in use, factors influencing adoption, challenges, and considerations of alternatives. Each question includes a brief explanation. C-1: What software signing tool does the team use? (Understanding current tool choices and components in practice.) C-2: What factors did the team consider before adopting this tool over others? (Identifying decision-making drivers.) C-3: What was the team’s previous signing practice before the current tool? (Capturing historical evolution of practices.) C-4: How does the team implement this tool (which components are in use)? (Understanding deployment in workflows.) C-5: What challenges or limitations has the team faced with this tool? How did you cope with these? (Surfacing usability pain points and coping strategies.) C-6: Has the team considered switching tools? If yes, which alternatives? (Exploring sustainability of current tool choices.) 21 Figure 7: Typical workflow for identity-based software signing. The component author requests a certificate from a certificate authority, which confirms the signer’s identity through an identity provider. The signature and signer’s identity are recorded in a transparency log, which the verifier can monitor to confirm the validity of the signature upon downloading the signed package. C Codebooks Table 7describes our codebook, with code definition, and example quote tagged with that code. D Code Saturation We present a saturation analysis of our studies in Figure 8. The green trendline (saturation curve) was measured by identifying the cumulative unique codes present in each interview. We observe saturation at the eleventh interview, indicating that no new unique codes were identified after participant 12. Additionally, the blue trendline (number of codes) shows that Figure 8: Saturation curve. Interviews are plotted in the order in which they were conducted. Each interview covered many topics in detail (orange line, blue line). However, as the green line shows, our results saturated (i.e., stopped observing new perspectives) around interview 11. Figure 9: Heatmap showing the organizational usability context for subjects from medium scale organizations (4 subjects). Similar to the one showed for small and large-scale organizations in Figure 4. In comparing to Figure 4, note that some columns are omitted in this figure because no subjects mentioned those aspects. the lowest number of codes was obtained from subject S4. This is also reflected in the orange trend line (unique number of codes). E Other Results In this appendix we prresent a continuation of the results presented in §5 E.1 Sigstore Components Most Commonly Used by Our Sample Population E.2 Other Effects of Organization and Signing Tools on Identified Usability Concerns In this appendix we prresent a continuation of the results presented and shown in Figure 4to show the Organizational and usage context report from participants in medium-scale organizations. E.3 Expanded Demographics Table See Table 9 E.3.1 Organizational Demographics We summarize additional organizational demographic information of our subjects in Table 10. This table provides further context for the subject’s organizational size, type of software product, and how many of our subjects belonged to each organization. 22 Table 7: Excerpts from the final codebook we used for our analysis. Code Definition Sample Quote Sigstore Criticisms Criticisms and weaknesses associated with implementing Sigstore for signing. P3: "We do have some concerns about the way Fulcio operates as its own certificate authority. So we’ve been looking at things like OpenPubkey as something that removes that intermediary and would allow you to do identity-based signing directly against the OIDC provider. " Sigstore Improvements Suggested improvements for Sigstore. P9: "I would say probably integrations within different platforms, so say you use GitLab or GitHub or Jenkins or something like that. Try to leverage some type of integration to make that, I guess, less friction for the developers to implement, something like that." Sigstore Strengths Strengths of Sigstore and reasons why subjects prefer using Sigstore. P7: "The convenience and the good user experience is probably what keeps us using Sigstore " Sigstore Alternatives Considered Alternatives of Sigstore that were or are being considered by the subject (for subjects whose primary signing implementation is Sigstore). P5: "We are also looking at other new alternatives that are coming up, like OpenPubkey. Docker just released a new standard called OpenPubKey, which is trying to solve this issue where they will not require you to have infrastructures and stuff." Sigstore Components Used Components of Sigstore (Cosign, Fulcio, etc.) being used by the subject. P2: "Mostly fulcio is what’s integrated into our product." Notary Criticism Criticisms and weaknesses Associated with Notary – promoting users to stop using them. "P5: "We were using Notary, and Notary only verifies that the image was signed using a specific key, but you cannot verify that who is the owner of the key." Table 8: Sigstore Functionalities Used by Subjects. Tool # of Subjects Cosign (Signing & Verification CLI) 9 Certificate Authority (Fulcio) 9 Keyless Signing & Identity Manager (OIDC) 9 Transparency Log (Rekor) 8 Gitsign (Commit Signing) 4 Component Customization & Local Sigstore Deployment 2 E.4 Factors Influencing Sigstore Adoption (Sigstore Users) In this appendix, we present a tabular organization of the reasons given by Sigstore users for their teams’ and organizations’ adoption of Sigstore, structured according to Cresswell et al. ’s four-factor framework. E.5 Factors Influencing Sigstore Consideration (Non-Sigstore Users). E.6 Additional Discussions & Implications of Results In addition to the implications discussed in §6, we outline further implications of our findings below. Table 9: Subject Demographics. For anonymity, we used generic job roles, not specific titles. “Senior management” refers to senior managers, directors, and executives; “Technical leader” to senior, lead, partner, and principal engineers; and “Engineer” and “Manager” to more junior staff. We highlight the type of software subject’s immediate team produces. *refers to cases where subject’s team built other types of software in addition to the main stated. Orefers to cases where subject belonged to multiple teams with other products than the one stated. ID Role Experience Software Type Duration (min) P1 Research leader 5 years Internal POC software 55 P2 Senior mgmt. 15 years SAAS security tool*O45 P3 Senior mgmt. 13 years SAAS security tool*O43 P4 Technical leader 20 years Open-source tooling 33 P5 Engineer 2 years Internal security tooling 46 P6 Technical leader 27 years Internal security tooling*O68 P7 Manager 6 years Security tooling*O48 P8 Technical leader 8 years Internal security tooling* 57 P9 Engineer 2.5 years SAAS security* 55 P10 Engineer 13 years SAAS security* 45 P11 Technical leader 16 years Firmware* 57 P12 Technical leader 4 years Consultancy 65 S13 Senior mgmt. 16 years Internal security tool 58 P14 Research leader 13 years POC security Software* 52 P15 Senior mgmt. 15 years Internal security tooling 42 P16 Senior mgmt. 15 years SAAS 60 P17 Manager 11 years Security tooling 52 23 Table 10: Organizational Demographics. To enhance anonymity, we only highlight the number of subjects in each organizational category. Letters A-L are used to depict each organization. Type Breakdown (#Organizations|#Subjects) Organizational Size (Employee Size) Small (<100) (4/6), Medium (<1500) (3/4), Large (>1500) (6/7) Product Area Digital technology (1/3), SSC Security (2/4), Social technology (1/1), Dev tools (1/1), Telecommunications (1/1), Cloud security (2/3), Aerospace Security (1/1), Internet services (2/2), Cloud + OSS Security (1/1), Cloud + Dev toools (1/1) Subject Distribution A (2), B (3), C (1), D (1), E (2), F (1), G (1), H (1), I (1), J (1), J (1), K (1), L (1), M(1) Table 11: Reasons Practitioners Choose or Switch to Sigstore Before Adoption. Practitioners’ decisions are driven primarily by human/social factors—such as community contribution, frustration with legacy tools, and usability issues—supplemented by Sigstore’s technological capabilities and reinforced by macroenvironmental pressures like regulations and trust in the CNCF ecosystem. Topics & Associated Examples Subjects Human/Social Factors Practitioners Contribute to Sigstore 6 – P2, P4, P6, P7, P9, P14 GPG Issues 4 Low adoption rates P1, P14 Steep learning curve P12, P16 Notary Issues 1 Non-demand from customers P12 Technological Factors Available Sigstore Functionalities 3 – P1, P5, P14 A transparency log, etc P5, P14 Compatibility to other Tools P1 GPG Issues 5 Key management issues & Other usability concerns P1, P6, P12, P16, P17 Compatibility with newer technology P16 Notary Issues 2 Compatibility with other tools P12 Lack of regular updates P12 Key & Identity Management P5 Proprietary Tool Issues 1 Difficult to setup P15 Macroenvironmental Factors Regulation & Standards (Cross-factor)-M&O 4 – P5, P6, P15, P17 Large Sigstore User Community 1 – P9 Inherent Trust of Creators 1 Trust of CNCF products P3 E.6.1 The Role of Community in Software Signing Tool Adoption Many participants cited their involvement in the Sigstore community as a key reason for adopting the tool (Table 11). While this may partly reflect our sampling, half of the non-Sigstore users who adopted external tools also emphasized community affiliation as a motivator. Challenges like setting up private Sigstore instances or accessing usage knowledge were frequently linked to the absence of sufficient community support (§5.1.2). Other community-related influences, such as the size of the user community and trust in the tool creators, also Table 12: Non Sigstore Users: Tooling Issues causing a consideration of Sigstore and other Next-gen tools. Tool Subjects Topic Internal tool P8, P11 Tooling Issues (1) Integration with other tools/platformsT– P8, P11 (2) Non unified tool implementation across teams O – P8, P11 (3) Signer ID Management T– P11 (4) Usage info/documentation unclear T– P11 (5) Verification not built in with tool T– P8 (6) Non transparency in signing infrastructure T– P11 (7) Security of Signing Infrastructure T– P11 (8) Engineers don’t understand the current tool P– P11 Considered (or Adopted) Fix (1) Organization building new tools with Next-gen features – P8, P11 Notary-V1 P10 Tooling Issues (1) Key ManagementT (2) Lack of an inbuilt signing policy T (3) Engineers don’t understand the current tool P– P11 (4) Integration with other tools/platformsT Considered (or Adopted) Fix (1) Organization has adopted Openpubkey for key management (1) Organization considering Sigstore, and Notary V2 PGP P13 (1) Tooling Issues (1) Key ManagementT (2) Non transparency in signing infrastructure T (3) Security of Signing Infrastructure T Considered (or Adopted) Fix (1) Use of Key discovery and delivery systems like TUF9 shaped adoption decisions. These findings echo prior work on the importance of community in developer tool adoption [48]. Toolmakers should actively foster supportive, trustworthy communities around their tools, as these networks can be pivotal to both adoption and long-term sustainability. E.6.2 Push-Pull-Barrier Definition In this appendix, we map our generated themes and subthemes to the Push, Pull, and Barrier factors from the TRL framework by Bansal et al. [25], as described in the results section. The table defines the driver type, corresponding theme, and examples. As this analysis was highly objective, a single rater performed this mapping. The table is here: Table 13. F Survey Attempt In addition to our interviews, we developed a survey distributed to Sigstore users at the 2024 SigstoreCon conference and through post-conference communications with the Sigstore user community. This aimed to gather perspectives from those who might not have had time for an interview. We exclude this data from our work due to several factors that compromise the quality of the results; these factors include respondents who completed the survey in an unrealistically 24 Table 13: Driver labels (Push/Pull/Barrier) mapped to concrete codes with factor tags (T/P/O/M) and example participant IDs. Driver Theme Code Factor Examples Barrier Policy/ownership Centralize org code signing internally O P8 Barrier Policy/ownership Organization’s open-source policy constraints O P8 Barrier Policy/ownership Organizational use cases don’t fit Sigstore O P8 Barrier Policy/ownership Tool owned by organization O P10 Barrier Risk/privacy/trust Low trust in Sigstore maintainers M P13 Barrier Risk/privacy/trust Sigstore privacy concerns T P13 Barrier Risk/privacy/trust Third-party security risk concerns T P11, P8 Pull Community/engagement Practitioners contribute to Sigstore H P2, P4, P6, P7, P9, P14 Pull Macro pressures Large Sigstore user community M/H P9 Pull Macro pressures Regulation & standards alignment M/O P5, P6, P15, P17 Pull Macro pressures Trust in CNCF/creators M P3 Pull Sigstore capabilities Compatibility with other tools/CI T P1 Pull Sigstore capabilities Transparency log & related features T P5, P14 Push General Key & identity management pain T P5 Push GPG issues (human) Low adoption rates H P1, P14 Push GPG issues (human) Steep learning curve H P12, P16 Push GPG issues (tech) Compatibility with newer technology T P16 Push GPG issues (tech) Key management & other usability pain T P1, P6, P12, P16, P17 Push Notary v1 issues Engineers don’t understand tool P P11 Push Notary v1 issues Key management T P10 Push Notary v1 issues Lack of in-built signing policy T P10 Push Notary v1 issues Lack of regular updates T P12 Push Notary v1 issues Compatibility problems T P12 Push Proprietary/Internal tool issues Difficult to set up T P15 Push Proprietary/Internal tool issues Engineers don’t understand current tool P P11 Push Proprietary/Internal tool issues Integration with other tools/platforms T P8, P11 Push Proprietary/Internal tool issues Non-transparent signing infrastructure T P11 Push Proprietary/Internal tool issues Non-unified implementations across teams O P8, P11 Push Proprietary/Internal tool issues Security concerns about signing infrastructure T P11 Push Proprietary/Internal tool issues Signer ID management problems T P11 Push Proprietary/Internal tool issues Unclear usage info/docs T P11 short amount of time and those who failed to complete the entire survey. Here, we summarize the answers to an excerpt of the questions responded to in our survey: What signing tool did your team use prior to the introduction of Sigstore? Among 7 participants who utilize Sigstore, 3/7 responded that they use PGP/GPG-based signing and 4/7 did not use any toolings prior to using Sigstore. Which of the following factors have influenced your team’s use of signing? (select all): Among 7 participants who utilize Sigstore, 4/7 selected "Organizational policies," 4/7 with "Best Practices/Other standards and recommendations" and 3/7 selected "Awareness of prominent SSC attack." Which Sigstore components does your team use? (select all): Among 7 participants who utilize Sigstore, 3 responded to this question, and all 3 participants selected Cosign, Fulcio, and Rekor. In which of the following ways have you implemented Sigstore? Among 7 participants who utilize Sigstore, 3 responded to this question, with 2 responding that they use a private deployment of Sigstore and 1 responding that they use a public instance of Sigstore. Do you use or require Sigstore (signing) at any of the following stages of your product? Among 7 participants who utilize Sigstore, 3 responded to this question. All 3 participants shared the "Build/Deployment/final product" stage as part of their answer. How do you verify Sigstore signatures? (select all): Among 7 participants who utilize Sigstore, 3 responded to this question. All 3 participants shared selecting certificate Verification as part of their answer. 2/3 selected that they check Rekor logs. 25