Azure Confidential Ledger – attestability for the masses
- Posted on July 14, 2021
- Estimated reading time 4 minutes
Azure Confidential Ledger or ‘ACL’ is a lightweight and flexible managed decentralized data platform, built on top of Azure Confidential Computing and Intel SGX. Backed by blockchain technology, multiple parties can add data entries in a secure and tamperproof way to ensure data integrity whilst allowing flexibility in the data and approach.
Azure’s confidential computing technology and hardware can secure data during processing, to ensure confidentiality of data end-to-end rather than just when at rest or in transit. This allows for processing of sensitive and regulated data in the cloud, to enable specific use cases such as cross-organizational data sharing, data combination, and processing of large datasets to train AI models without exposing the data to others.
Azure Confidential Ledger runs on similar principles to those used in Azure SQL ledger tables, which we explored previously for Microsoft Build 2021. Azure Confidential Ledger gives us the same level of trust and integrity, without the need for a managed SQL database or dedicated SQL server. Entries can be quickly added to the ledger as and when they occur, with receipts for each transaction to validate each entry. Entries can also be read from the ledger, so a full history can be established, without the ability of any party to tamper with the historical content.
Entries can vary, from the short and simple, to verbose or unstructured data formats. This provides great flexibility in what can be logged to the ledger, making Azure Confidential Ledger suitable for many use cases. Data formats can be changed or adapted over time, and entries don’t have to conform to a single standard from the point of creation, so confidential ledger can evolve with the problem space it has been implemented in, while retaining the history of data.
Azure Confidential Ledger’s benefits are its lightweight and flexible structure. It can be applied to any use case where data integrity is a core objective, where multiple parties need abilities to add and read from a shared central ledger. One use case allows us to track the data used to train a particular iteration of a machine learning model used by multiple stakeholders. This can be sent to the ledger at the point of training, to allow all parties to see when, why, and how decisions about model data sets were made. In the event of an ethical violation of the data involved, or the decisions the model makes, the source decision can be quickly found and assessed. Check out the Microsoft blog for more use cases.
Azure Confidential Ledger is extremely easy to deploy and utilise in Azure. As a fully managed Azure service, we can rely on Azure’s high standards of availability and scalability. Confidential Ledger has a python SDK that’s available to use today. There are samples available to help you get started. A .NET library is currently in the pipeline, but you can also manage your Confidential Ledger via REST calls meaning Confidential Ledger is easy to integrate into any code base or project.
Have a look at our GitHub repository to see our implementation around monitoring telemetry from vehicles of the future and look out for more posts on the underlying technical aspects and implementations as the ledger technology evolves.
Are you as enthusiastic as us? Leave your thoughts in the comments below.
Could you please elaborate more on how "processing of large datasets to train AI models without exposing the data to others" is achieved? Thanks.
Hi William, great question. With confidential compute environments you can run any processes encrypted down to the memory level of the machines. This means encrypted data can be sent to these secure compute environments and used for processing without the need for it to be exposed even while processing. Even admins with access to the compute resources will not be able to see the data, whether at rest, in transit, or undergoing processing - for example in training a model. In the case of multiple parties, these systems can be designed such that only each individual party can only see their own input data as long as they hold the key, and all organisations share the output model.