Streamni benchmark qilamiz (nodejs).

Habibov Ulug'bek
5 min readJul 28, 2024

--

Assalamu Alaykum o`tgan maqolada streamlar haqida to`liq gaplashdik va ularni tezlik va xotirada foydasi tegishini bildirib o`tdik bugun bir muammo olib stream haqiqatdan foydalimi yoqmi buni shu muammo orqali tekshirib ko`ramiz(benchmarking orqali ).

Bu maqolani boshlashdan oldin oldingi maqolani streamlarni yaxshiroq tushunish uchun tavsiya qilaman. link

Muammo uchun o`sha mashhur hamda nodejs dokumentatsiyasida berilgan 1 dan milliongacha bo`lgan sonlarni faylga yozish va o`qishni tanlab oldim .

Unda qani boshladik ;)

Benchmarking Natijalari komputeringizda boshqacha chiqishi mumkin (cpu va RAMga bog`liq).

Birinchi bo`lib biz o`zimiz Nodejsda streamlarni bilmagan holda bu misolga qanday yondashamiz ? Miyamga kelgan birinchi fikr bu fs modul orqali faylga yozish .Birinchi bo`lib shundan boshlaymiz. Bu holda bizda 3 xil usul bor callback funksiyalardan foydalanish ,sinxron funksiyalardan foydalanish yoki promise funksiyalaridan foydalanish .

Promise funksiyalari orqali tekshirib ko`ramiz

unda birinchi bo`lib kodni yozib olamiz .

const fs = require("node:fs/promises");

// Execution Time: 9.3s
// Memory Usage: 50-60MB
(async () => {
console.time("fs promise solution");
try {
const fileHandler = await fs.open("test.txt", "w");

for (let i = 0; i < 1000000; i++) {
await fileHandler.write(` ${i} `);
}
} catch (error) {
console.error("An error occurred:", error);
} finally {
console.timeEnd("fs promise solution");
}
})();

(Agarda IIFE haqida kop`roq bilmoqchi bo`lsangiz link ).

memory usage

ko`rib turganingizdek yuqoridagi ma’lumotni faylga yozishda promise solution uchun.

Execution Time: 9.3s.

Memory Usage: 50–60MB sarflandi

Callback funksiyalari orqali tekshirib ko`ramiz

const fs = require("node:fs");

// Execution Time: 1.4s
// Memory Usage: 1GB
(async () => {
console.time("fs callback solution");
fs.open("test.txt", "w", (err, fd) => {
if (err) {
console.error("Error opening file:", err);
return;
}
for (let i = 0; i < 1000000; i++) {
fs.write(fd, ` ${i} `, (err) => {
if (err) {
console.error("Error writing to file:", err);
}
});
}
console.timeEnd("fs callback solution");
});
})();

callback versiyadagi yechimda 1.4 soniya va 1GB atrofida xotira sarflandi .

Execution Time: 1.4s.

Memory Usage: 1GB sarflandi

Agarda biz 1 million emas 10 million raqamni yozsak u holat xotira yetishmasligi tufayli pastdagi error kelib chiqadi.

Ko`rib turganingizdek callback holatida xotirada joy yetishmasligi tufayli process o`chirib yuborildi.

Sinxron funksiyalari orqali tekshirib ko`ramiz

const fs = require("node:fs");

// Execution Time: 1.8s
// Memory Usage: 40MB
(async () => {
console.time("fs sync solution");
fs.open("test.txt", "w", (err, fd) => {
if (err) {
console.error("Error opening file:", err);
return;
}

for (let i = 0; i < 1000000; i++) {
const buff = Buffer.from(` ${i} `, "utf-8");
try {
fs.writeSync(fd, buff);
} catch (writeErr) {
console.error("Error writing to file:", writeErr);
return;
}
}

console.timeEnd("fs sync solution");
});
})();

Sinxron yechimda esa :

Execution Time: 1.8s.

Memory Usage: 40MB sarflandi.

Tepada 3 ta yechim ko`rdik bularning hammasini har xil yaxshi va yomon tomonlari bor .Agarda yoziladigan sonlarni oshirsak nima bo`ladi ?

  1. Promise solution xotira deyarli o`zgarmay turadi ishlaydigan vaqt ko`payadi .
  2. Callback solution xotira ko`payib xotira yetmasligi natijasida process to`xtatiladi.
  3. Sinxron xotira o`zgarmasdan qoladi ammo vaqt ko`payadi (bu yerda faqat raqam yozilmoqda agar ko`proq vaqt talab qilinadigan task orqali sinxron yozilsa promise solutiondan ko`p vaqt ketadi).

Nodejs dokumentatsiya bo`yicha callback eng tez ishlaydigan usul va bu benchmark orqali ko`rindi ammo bir necha callbacklar xotiraga yozilishi tufayli xotiradan ancha joy olinadi .Ammo tepadagi benchmarkingdan nega sinxron usul promise yechimdan tezroq ishlayapti degan savol keladi chunki raqam yozish ko`p vaqtni olmaydigan bo`lganligi uchun sinxron promise ochib resolve qilish vaqti ketmaganligi evaziga vaqtdan yutmoqda.

Stream solutions

Endi oldingi maqolada aytganimizdek bu turdagi muammolar uchun eng to`g`ri yechim bu streamlar orqali .Biz solishtirish uchun raqamlarni bilib oldik endi stream orqali yechim yozishga o`tamiz.

const fs = require("node:fs/promises");

(async () => {
try {
console.time("naive stream solution");
const fileHandle = await fs.open("test.txt", "w");

const stream = fileHandle.createWriteStream();

for (let i = 0; i < 1000000; i++) {
const buff = Buffer.from(` ${i} `, "utf-8");
stream.write(buff);
}

stream.end(); // Close the stream

await new Promise((resolve, reject) => {
stream.on("finish", resolve);
stream.on("error", reject);
});

console.timeEnd("naive stream solution");
} catch (error) {
console.error("An error occurred:", error);
}
})();

Execution Time: 220ms
Memory Usage: 240MB

Bu yechimda ko`rib turganingizdek vaqt ancha tezlashdi 6 marta atrofida ammo biroz ko`proq xotira evaziga .Bu oxirgi yechimmi ?

Yo’q bu oxirgi yechim emas bu yerda biz streamlarda bir narsaga amal qilmayapmiz yani buffer faqat to`lganidan keyingina uni faylga yozish .Bu yechimda har bitta raqam uchun har doim yozilmoqda va bu xotiradan joy olmoqda.

Note: Production da tepadagi yechim tavsiya etilmaydi

Yaxshiroq stream yechimi.

const fs = require("node:fs/promises");

// Execution Time: 245ms
// Memory Usage: 30MB
(async () => {
console.time("better stream solution");
try {
const fileHandle = await fs.open("test.txt", "w");

const stream = fileHandle.createWriteStream();

let i = 0;

const numberOfWrites = 1000000;

const writeMany = () => {
while (i < numberOfWrites) {
const buff = Buffer.from(` ${i} `, "utf-8");

if (i === numberOfWrites - 1) {
return stream.end(buff);
}

i++;

if (!stream.write(buff)) break;
}
};

writeMany();

stream.on("drain", () => {
writeMany();
});

stream.on("finish", () => {
fileHandle.close();
console.timeEnd("better stream solution");
});
} catch (error) {
console.error("An error occurred:", error);
}
})();

Execution Time: 245ms
Memory Usage: 30MB

Kurib turganingizdek shunchki streamdan foydalanish emas streamlardan to`gri foydalanish bizga performance yaxshigina ko`tarishga yordam beradi.

Agarda maqola yoqqan bo`lsa chapak chaling (ko`p chalsayam bo`ladi 50 tagacha).

Yana shunday maqolalar o`qishni xohlasangiz medium da follow tugmasini bosib qo`ying.

Xato va kamchiliklar uchun uzr !!!

Kodlarni ushbu githab repoda topishingiz mumkin =>Repo

linkedin.com => Ulug’bek Habibov | LinkedIn

telegram channel => @habibov_ulugbek

--

--

Habibov Ulug'bek
Habibov Ulug'bek

Written by Habibov Ulug'bek

Software Engineer | Backend Nodejs Developer

No responses yet