End-to-End Storage Workflow¶
This tutorial will cover the end-to-end process of creating a bucket, uploading a file, and retrieving it, in a step-by-step format.
Prerequisites¶
- Node.js v22+ installed
-
A TypeScript project
Need a starter project?
If you don't have an existing project, follow these steps to create a TypeScript project you can use to follow the guides in this section:
-
Create a new project folder by executing the following command in the terminal:
-
Initialize a
package.jsonfile using the correct command for your package manager: -
Add the TypeScript and Node type definitions to your projects using the correct command for your package manager:
-
Create a
tsconfig.jsonfile in the root of your project and paste the following configuration: -
Initialize the
srcdirectory:
-
-
The StorageHub SDK installed
Client Setup and Create a Bucket¶
Buckets group your files under a specific Main Storage Provider (MSP) and value proposition. Derive a deterministic bucket ID, fetch MSP parameters, then create the bucket. If you run the script multiple times, use a new bucketName to avoid a revert, or modify the logic to use your existing bucket in later steps.
In the following code, you will initialize the StorageHub, viem, and Polkadot.js clients on the DataHaven TestNet, sign in via SIWE (Sign in With Ethereum), and pull the MSP’s details/value proposition to prepare for bucket creation. Then you will derive the bucket ID, confirm it doesn’t exist, submit a createBucket transaction, wait for confirmation, and finally query the chain to verify that the new bucket’s MSP and owner match our account.
The following sections will build on this snippet, so it's important to start here to properly configure the client and ensure the bucket is created correctly. If you'd prefer to step through the steps to create a bucket individually, please see the Create a Bucket guide.
import '@storagehub/api-augment';
import { ApiPromise, WsProvider } from '@polkadot/api';
import { types } from '@storagehub/types-bundle';
import {
StorageHubClient,
initWasm,
FileManager,
ReplicationLevel,
} from '@storagehub-sdk/core';
import { createReadStream, createWriteStream, statSync, existsSync, mkdirSync } from 'node:fs';
import path from 'node:path';
import { Readable } from 'node:stream';
import { TypeRegistry } from '@polkadot/types';
import type { AccountId20, H256 } from '@polkadot/types/interfaces';
import {
MspClient,
} from '@storagehub-sdk/msp-client';
import type { DownloadResult } from '@storagehub-sdk/msp-client';
import {
createPublicClient,
createWalletClient,
defineChain,
http,
} from 'viem';
import { privateKeyToAccount } from 'viem/accounts';
async function run() {
// For anything from @storagehub-sdk/core to work, initWasm() is required
// on top of the file
await initWasm();
// --- viem setup ---
// Define DataHaven chain, as expected by viem
const chain = defineChain({
id: 55931,
name: 'DataHaven Testnet',
nativeCurrency: { name: 'Mock', symbol: 'MOCK', decimals: 18 },
rpcUrls: {
default: { http: ['https://services.datahaven-testnet.network/testnet'] },
},
});
// Define account from a private key
const account = privateKeyToAccount('INSERT_PRIVATE_KEY_HERE' as `0x${string}`);
// Create a wallet client using the defined chain, account, and RPC URL
const walletClient = createWalletClient({
chain,
account,
transport: http('https://services.datahaven-testnet.network/testnet'),
});
// Create a public client using the defined chain and RPC URL
const publicClient = createPublicClient({
chain,
transport: http('https://services.datahaven-testnet.network/testnet'),
});
// --- Polkadot.js API setup ---
const provider = new WsProvider('wss://services.datahaven-testnet.network/testnet');
const polkadotApi: ApiPromise = await ApiPromise.create({
provider,
typesBundle: types,
noInitWarn: true,
});
// --- Bucket creating logic ---
const baseUrl = 'https://deo-dh-backend.testnet.datahaven-infra.network/';
const httpConfig = { baseUrl: baseUrl };
// Setup session provider for MSP authentication
let sessionToken: string | undefined;
const sessionProvider = async () =>
sessionToken ? ({ token: sessionToken, user: { address: account.address } } as const) : undefined;
// Connect to MSP Client
const mspClient = await MspClient.connect(httpConfig, sessionProvider);
// Check MSP Health Status
const mspHealth = await mspClient.info.getHealth();
console.log('MSP service health:', mspHealth);
// Initialize StorageHub Client
const storageHubClient = new StorageHubClient({
rpcUrl: 'https://services.datahaven-testnet.network/testnet',
chain: chain,
walletClient: walletClient,
filesystemContractAddress:
'0x0000000000000000000000000000000000000404' as `0x${string}`,
});
// Derive bucket ID
const bucketName = 'INSERT_BUCKET_NAME_HERE';
const address = account.address;
const bucketId = (await storageHubClient.deriveBucketId(
address,
bucketName
)) as string;
console.log('Derived bucket ID: ', bucketId);
// Check that the bucket doesn't exist yet
const bucketBeforeCreation = await polkadotApi.query.providers.buckets(
bucketId
);
console.log(
'Bucket before creation is empty: ',
bucketBeforeCreation.isEmpty
);
// Get basic MSP information from the MSP including its ID
const mspInfo = await mspClient.info.getInfo();
const mspId = mspInfo.mspId as `0x${string}`;
console.log('MSP ID:', mspId);
// Choose one of the value props retrieved from the MSP
const valueProps = await mspClient.info.getValuePropositions();
if (!Array.isArray(valueProps) || valueProps.length === 0) {
throw new Error('No value props available from this MSP.');
}
// For simplicity, this selects the first valueProp in the list
const valueProp = valueProps[0];
console.log('Chosen value prop: ', valueProp);
// Get the ID of the chosen value prop
const valuePropId = valueProp.id as `0x${string}`;
console.log('Chosen value prop id: ', valuePropId);
// Define if bucket should be private or public
const isPrivate = false;
// Create bucket on chain
const txHash: `0x${string}` | undefined = await storageHubClient.createBucket(
mspId as `0x${string}`,
bucketName,
isPrivate,
valuePropId
);
console.log('createBucket() txHash:', txHash);
if (!txHash) {
throw new Error('createBucket() did not return a transaction hash');
}
// Wait for transaction receipt
const receipt = await publicClient.waitForTransactionReceipt({
hash: txHash,
});
console.log('Bucket creation receipt:', receipt);
if (receipt.status !== 'success') {
throw new Error(`Bucket creation failed: ${txHash}`);
}
const bucketAfterCreation = await polkadotApi.query.providers.buckets(
bucketId
);
console.log('Bucket after creation exists', !bucketAfterCreation.isEmpty);
// Unwrap bucket in order to read its data
const bucketData = bucketAfterCreation.unwrap();
console.log('Bucket data:', bucketData);
// Check if the retrieved bucket's MSP ID matches the initial MSP ID you retrieved
console.log(
'Bucket mspId matches initial mspId:',
bucketData.mspId.toString() === mspId
);
// Check if the retrieved bucket's userId (owner address) matches the initial address you used to create the bucket
console.log(
'Bucket userId matches initial bucket owner address:',
bucketData.userId.toString() === address
);
// Next section code here
// Disconnect the Polkadot API at the very end
await polkadotApi.disconnect();
}
await run();
Issue a Storage Request¶
Ensure your file is ready to upload. In this demonstration, we're using a .jpg file named hello.jpg stored in the current working directory, i.e., the same as the typescript project files, /src/.
Register your intent to store a file in your bucket and set its replication policy. Initialize FileManager, compute the file’s fingerprint, fetch MSP info (and extract peer IDs), choose a replication level and replica count, then call issueStorageRequest.
// Prior section code here
// --- File fingerprinting (hello.jpg) ---
const fileName = 'hello.jpg';
const filePath = path.join(process.cwd(), fileName);
if (!existsSync(filePath)) {
throw new Error(`Could not find ${fileName} at ${filePath}`);
}
const fileSize = statSync(filePath).size;
console.log(`Using file path: ${filePath}`);
const fileManager = new FileManager({
size: fileSize,
stream: () => Readable.toWeb(createReadStream(filePath)) as ReadableStream<Uint8Array>,
});
const fingerprint = await fileManager.getFingerprint();
console.log(`File size: ${fileSize} bytes`);
console.log(`Fingerprint: ${fingerprint.toHex()}`);
// --- Issue storage request for the file ---
const { mspId: infoMspId, multiaddresses } = await mspClient.info.getInfo();
if (!multiaddresses?.length) {
throw new Error('MSP multiaddresses are missing');
}
function extractPeerIDs(multiaddresses: string[]): string[] {
return (multiaddresses ?? [])
.map((addr) => addr.split('/p2p/').pop())
.filter((id): id is string => !!id);
}
const peerIds: string[] = extractPeerIDs(multiaddresses);
if (peerIds.length === 0) {
throw new Error('MSP multiaddresses had no /p2p/<peerId> segment');
}
const replicationLevel = ReplicationLevel.Custom;
const replicas = 1;
const fileSizeBigInt = BigInt(fileManager.getFileSize());
const storageTxHash: `0x${string}` | undefined = await storageHubClient.issueStorageRequest(
bucketId as `0x${string}`,
fileName,
fingerprint.toHex() as `0x${string}`,
fileSizeBigInt,
(infoMspId ?? mspId) as `0x${string}`,
peerIds,
replicationLevel,
replicas
);
console.log('issueStorageRequest() txHash:', storageTxHash);
if (!storageTxHash) {
throw new Error('issueStorageRequest() did not return a transaction hash');
}
const storageReceipt = await publicClient.waitForTransactionReceipt({ hash: storageTxHash });
if (storageReceipt.status !== 'success') {
throw new Error(`Storage request failed: ${storageTxHash}`);
}
// Next section code here
Verify If Storage Request Is On-Chain¶
Derive the deterministic file key, query on-chain state, and confirm the request exists and matches your local fingerprint and bucket.
// Prior section code here
// --- Compute file key (owner, bucketId, fileName) ---
const registry = new TypeRegistry();
const owner = registry.createType('AccountId20', account.address) as AccountId20;
const bucketIdH256 = registry.createType('H256', bucketId) as H256;
const fileKey = await fileManager.computeFileKey(owner, bucketIdH256, fileName);
console.log('Computed file key:', fileKey.toHex());
// --- Verify storage request on chain ---
const storageRequest = await polkadotApi.query.fileSystem.storageRequests(fileKey);
if (!storageRequest.isSome) {
throw new Error('Storage request not found on chain');
}
// Read storage request data
const storageRequestData = storageRequest.unwrap();
console.log('Storage request data:', storageRequestData);
console.log('Storage request bucketId:', storageRequestData.bucketId.toString());
console.log(
'Storage request fingerprint should be the same as initial fingerprint',
storageRequestData.fingerprint.toString() === fingerprint.toString()
);
// Next section code here
Authenticate with SIWE and JWT¶
In this section, you'll trigger the SIWE flow: the connected wallet signs an EIP-4361 message, the MSP verifies it, and returns a JWT session token. Save that token as sessionToken and reuse it for subsequent authenticated requests.
// Prior section code here
// Trigger the SIWE (Sign-In with Ethereum) flow.
// This prompts the connected wallet to sign an EIP-4361 message,
// which the MSP backend verifies to issue a JWT session token
const siweSession = await mspClient.auth.SIWE(walletClient);
console.log('SIWE Session:', siweSession);
// Store the obtained session token for future authenticated requests
sessionToken = (siweSession as { token: string }).token;
// Next section code here
Upload a File¶
Send the file bytes to the MSP, linked to your storage request. Confirm that the upload receipt indicates a successful upload.
// Prior section code
// --- Upload file to MSP ---
const fileBlob = await fileManager.getFileBlob();
const uploadReceipt = await mspClient.files.uploadFile(
bucketId,
fileKey.toHex(),
fileBlob,
address,
fileName
);
console.log('File upload receipt:', uploadReceipt);
if (uploadReceipt.status !== 'upload_successful') {
throw new Error('File upload to MSP failed');
}
// Next section code here
Retrieve Your Data¶
Download the file by its deterministic key from the MSP and save it locally.
// Prior section code here
// Must be authenticated to download files
const downloadResponse: DownloadResult = await mspClient.files.downloadFile(
fileKey.toHex()
);
// Check if the download response was successful
if (downloadResponse.status !== 200) {
throw new Error(`Download failed with status: ${downloadResponse.status}`);
}
// Sanitize the filename to prevent path traversal attacks
// Extract only the base filename without any directory separators
const sanitizedFileName = path.basename(fileName);
// Define the local path where the downloaded file will be saved
// Here it is resolved relative to the current module's URL.
const downloadPath = new URL(
`../../files/${sanitizedFileName}_downloaded.jpg`, // make sure the file extension matches the original file
import.meta.url
).pathname;
// Ensure the directory exists before writing the file
const downloadDir = path.dirname(downloadPath);
mkdirSync(downloadDir, { recursive: true });
// Create a writable stream to the target file path
// This stream will receive binary data chunks and write them to disk.
const writeStream = createWriteStream(downloadPath);
// Convert the Web ReadableStream into a Node.js-readable stream
const readableStream = Readable.fromWeb(downloadResponse.stream as any);
// Pipe the readable (input) stream into the writable (output) stream
// This transfers the file data chunk by chunk and closes the write stream automatically
// when finished.
readableStream.pipe(writeStream);
// Wait for the write stream to finish before proceeding
await new Promise<void>((resolve, reject) => {
writeStream.on('finish', resolve);
writeStream.on('error', reject);
});
console.log('Downloaded file saved to:', downloadPath);
// Final code here to Disconnect the Polkadot API and run the function
// You already configured this in the first section code.
Putting It All Together¶
The code containing the complete series of steps from issuing a storage request to retrieving the data is available below. As a reminder, before running the full script, ensure you have the following:
- Tokens to pay for the storage request on your account
- A file to upload, such as
hello.jpg
View complete script
import '@storagehub/api-augment';
import { ApiPromise, WsProvider } from '@polkadot/api';
import { types } from '@storagehub/types-bundle';
import {
StorageHubClient,
initWasm,
FileManager,
ReplicationLevel,
} from '@storagehub-sdk/core';
import { createReadStream, createWriteStream, statSync, existsSync, mkdirSync } from 'node:fs';
import path from 'node:path';
import { Readable } from 'node:stream';
import { TypeRegistry } from '@polkadot/types';
import type { AccountId20, H256 } from '@polkadot/types/interfaces';
import {
MspClient,
} from '@storagehub-sdk/msp-client';
import type { DownloadResult } from '@storagehub-sdk/msp-client';
import {
createPublicClient,
createWalletClient,
defineChain,
http,
} from 'viem';
import { privateKeyToAccount } from 'viem/accounts';
async function run() {
// For anything from @storagehub-sdk/core to work, initWasm() is required
// on top of the file
await initWasm();
// --- viem setup ---
// Define DataHaven chain, as expected by viem
const chain = defineChain({
id: 55931,
name: 'DataHaven Testnet',
nativeCurrency: { name: 'Mock', symbol: 'MOCK', decimals: 18 },
rpcUrls: {
default: { http: ['https://services.datahaven-testnet.network/testnet'] },
},
});
// Define account from a private key
const account = privateKeyToAccount('INSERT_PRIVATE_KEY' as `0x${string}`);
// Create a wallet client using the defined chain, account, and RPC URL
const walletClient = createWalletClient({
chain,
account,
transport: http('https://services.datahaven-testnet.network/testnet'),
});
// Create a public client using the defined chain and RPC URL
const publicClient = createPublicClient({
chain,
transport: http('https://services.datahaven-testnet.network/testnet'),
});
// --- Polkadot.js API setup ---
const provider = new WsProvider('wss://services.datahaven-testnet.network/testnet');
const polkadotApi: ApiPromise = await ApiPromise.create({
provider,
typesBundle: types,
noInitWarn: true,
});
// --- Bucket creating logic ---
const baseUrl = 'https://deo-dh-backend.testnet.datahaven-infra.network/';
const httpConfig = { baseUrl: baseUrl };
// Setup session provider for MSP authentication
let sessionToken: string | undefined;
const sessionProvider = async () =>
sessionToken ? ({ token: sessionToken, user: { address: account.address } } as const) : undefined;
// Connect to MSP Client
const mspClient = await MspClient.connect(httpConfig, sessionProvider);
// Check MSP Health Status
const mspHealth = await mspClient.info.getHealth();
console.log('MSP service health:', mspHealth);
// Initialize StorageHub Client
const storageHubClient = new StorageHubClient({
rpcUrl: 'https://services.datahaven-testnet.network/testnet',
chain: chain,
walletClient: walletClient,
filesystemContractAddress:
'0x0000000000000000000000000000000000000404' as `0x${string}`,
});
// Derive bucket ID
const bucketName = 'INSERT_BUCKET_NAME_HERE';
const address = account.address;
const bucketId = (await storageHubClient.deriveBucketId(
address,
bucketName
)) as string;
console.log('Derived bucket ID: ', bucketId);
// Check that the bucket doesn't exist yet
const bucketBeforeCreation = await polkadotApi.query.providers.buckets(
bucketId
);
console.log(
'Bucket before creation is empty: ',
bucketBeforeCreation.isEmpty
);
// Get basic MSP information from the MSP including its ID
const mspInfo = await mspClient.info.getInfo();
const mspId = mspInfo.mspId as `0x${string}`;
console.log('MSP ID:', mspId);
// Choose one of the value props retrieved from the MSP
const valueProps = await mspClient.info.getValuePropositions();
if (!Array.isArray(valueProps) || valueProps.length === 0) {
throw new Error('No value props available from this MSP.');
}
// For simplicity, this selects the first valueProp in the list
const valueProp = valueProps[0];
console.log('Chosen value prop: ', valueProp);
// Get the ID of the chosen value prop
const valuePropId = valueProp.id as `0x${string}`;
console.log('Chosen value prop id: ', valuePropId);
// Define if bucket should be private or public
const isPrivate = false;
// Create bucket on chain
const txHash: `0x${string}` | undefined = await storageHubClient.createBucket(
mspId as `0x${string}`,
bucketName,
isPrivate,
valuePropId
);
console.log('createBucket() txHash:', txHash);
if (!txHash) {
throw new Error('createBucket() did not return a transaction hash');
}
// Wait for transaction receipt
const receipt = await publicClient.waitForTransactionReceipt({
hash: txHash,
});
console.log('Bucket creation receipt:', receipt);
if (receipt.status !== 'success') {
throw new Error(`Bucket creation failed: ${txHash}`);
}
const bucketAfterCreation = await polkadotApi.query.providers.buckets(
bucketId
);
console.log('Bucket after creation exists', !bucketAfterCreation.isEmpty);
// Unwrap bucket in order to read its data
const bucketData = bucketAfterCreation.unwrap();
console.log('Bucket data:', bucketData);
// Check if the retrieved bucket's MSP ID matches the initial MSP ID you retrieved
console.log(
'Bucket mspId matches initial mspId:',
bucketData.mspId.toString() === mspId
);
// Check if the retrieved bucket's userId (owner address) matches the initial address you used to create the bucket
console.log(
'Bucket userId matches initial bucket owner address:',
bucketData.userId.toString() === address
);
// --- File fingerprinting (hello.jpg) ---
const fileName = 'hello.jpg';
const filePath = path.join(process.cwd(), fileName);
if (!existsSync(filePath)) {
throw new Error(`Could not find ${fileName} at ${filePath}`);
}
const fileSize = statSync(filePath).size;
console.log(`Using file path: ${filePath}`);
const fileManager = new FileManager({
size: fileSize,
stream: () => Readable.toWeb(createReadStream(filePath)) as ReadableStream<Uint8Array>,
});
const fingerprint = await fileManager.getFingerprint();
console.log(`File size: ${fileSize} bytes`);
console.log(`Fingerprint: ${fingerprint.toHex()}`);
// --- Issue storage request for the file ---
const { mspId: infoMspId, multiaddresses } = await mspClient.info.getInfo();
if (!multiaddresses?.length) {
throw new Error('MSP multiaddresses are missing');
}
function extractPeerIDs(multiaddresses: string[]): string[] {
return (multiaddresses ?? [])
.map((addr) => addr.split('/p2p/').pop())
.filter((id): id is string => !!id);
}
const peerIds: string[] = extractPeerIDs(multiaddresses);
if (peerIds.length === 0) {
throw new Error('MSP multiaddresses had no /p2p/<peerId> segment');
}
const replicationLevel = ReplicationLevel.Custom;
const replicas = 1;
const fileSizeBigInt = BigInt(fileManager.getFileSize());
const storageTxHash: `0x${string}` | undefined = await storageHubClient.issueStorageRequest(
bucketId as `0x${string}`,
fileName,
fingerprint.toHex() as `0x${string}`,
fileSizeBigInt,
(infoMspId ?? mspId) as `0x${string}`,
peerIds,
replicationLevel,
replicas
);
console.log('issueStorageRequest() txHash:', storageTxHash);
if (!storageTxHash) {
throw new Error('issueStorageRequest() did not return a transaction hash');
}
const storageReceipt = await publicClient.waitForTransactionReceipt({ hash: storageTxHash });
if (storageReceipt.status !== 'success') {
throw new Error(`Storage request failed: ${storageTxHash}`);
}
// --- Compute file key (owner, bucketId, fileName) ---
const registry = new TypeRegistry();
const owner = registry.createType('AccountId20', account.address) as AccountId20;
const bucketIdH256 = registry.createType('H256', bucketId) as H256;
const fileKey = await fileManager.computeFileKey(owner, bucketIdH256, fileName);
console.log('Computed file key:', fileKey.toHex());
// --- Verify storage request on chain ---
const storageRequest = await polkadotApi.query.fileSystem.storageRequests(fileKey);
if (!storageRequest.isSome) {
throw new Error('Storage request not found on chain');
}
// Read storage request data
const storageRequestData = storageRequest.unwrap();
console.log('Storage request data:', storageRequestData);
console.log('Storage request bucketId:', storageRequestData.bucketId.toString());
console.log(
'Storage request fingerprint should be the same as initial fingerprint',
storageRequestData.fingerprint.toString() === fingerprint.toString()
);
// Trigger the SIWE (Sign-In with Ethereum) flow.
// This prompts the connected wallet to sign an EIP-4361 message,
// which the MSP backend verifies to issue a JWT session token
const siweSession = await mspClient.auth.SIWE(walletClient);
console.log('SIWE Session:', siweSession);
// Store the obtained session token for future authenticated requests
sessionToken = (siweSession as { token: string }).token;
// --- Upload file to MSP ---
const fileBlob = await fileManager.getFileBlob();
const uploadReceipt = await mspClient.files.uploadFile(
bucketId,
fileKey.toHex(),
fileBlob,
address,
fileName
);
console.log('File upload receipt:', uploadReceipt);
if (uploadReceipt.status !== 'upload_successful') {
throw new Error('File upload to MSP failed');
}
// Must be authenticated to download files
const downloadResponse: DownloadResult = await mspClient.files.downloadFile(
fileKey.toHex()
);
// Check if the download response was successful
if (downloadResponse.status !== 200) {
throw new Error(`Download failed with status: ${downloadResponse.status}`);
}
// Sanitize the filename to prevent path traversal attacks
// Extract only the base filename without any directory separators
const sanitizedFileName = path.basename(fileName);
// Define the local path where the downloaded file will be saved
// Here it is resolved relative to the current module's URL.
const downloadPath = new URL(
`../../files/${sanitizedFileName}_downloaded.jpg`, // make sure the file extension matches the original file
import.meta.url
).pathname;
// Ensure the directory exists before writing the file
const downloadDir = path.dirname(downloadPath);
mkdirSync(downloadDir, { recursive: true });
// Create a writable stream to the target file path
// This stream will receive binary data chunks and write them to disk.
const writeStream = createWriteStream(downloadPath);
// Convert the Web ReadableStream into a Node.js-readable stream
const readableStream = Readable.fromWeb(downloadResponse.stream as any);
// Pipe the readable (input) stream into the writable (output) stream
// This transfers the file data chunk by chunk and closes the write stream automatically
// when finished.
readableStream.pipe(writeStream);
// Wait for the write stream to finish before proceeding
await new Promise<void>((resolve, reject) => {
writeStream.on('finish', resolve);
writeStream.on('error', reject);
});
console.log('Downloaded file saved to:', downloadPath);
// Disconnect the Polkadot API at the very end
await polkadotApi.disconnect();
}
await run();
Notes on Data Safety¶
Uploading a file does not guarantee network-wide replication. Files are considered secured by DataHaven only after replication to a Backup Storage Provider (BSP) is complete. Tooling to surface replication status is in active development.
Next Steps¶
| Created: October 17, 2025