Streamni benchmark qilamiz (nodejs).

5 min readJul 28, 2024

Assalamu Alaykum o`tgan maqolada streamlar haqida to`liq gaplashdik va ularni tezlik va xotirada foydasi tegishini bildirib o`tdik bugun bir muammo olib stream haqiqatdan foydalimi yoqmi buni shu muammo orqali tekshirib ko`ramiz(benchmarking orqali ).

Bu maqolani boshlashdan oldin oldingi maqolani streamlarni yaxshiroq tushunish uchun tavsiya qilaman. link

Muammo uchun o`sha mashhur hamda nodejs dokumentatsiyasida berilgan 1 dan milliongacha bo`lgan sonlarni faylga yozish va o`qishni tanlab oldim .

Unda qani boshladik ;)

Benchmarking Natijalari komputeringizda boshqacha chiqishi mumkin (cpu va RAMga bog`liq).

Birinchi bo`lib biz o`zimiz Nodejsda streamlarni bilmagan holda bu misolga qanday yondashamiz ? Miyamga kelgan birinchi fikr bu fs modul orqali faylga yozish .Birinchi bo`lib shundan boshlaymiz. Bu holda bizda 3 xil usul bor callback funksiyalardan foydalanish ,sinxron funksiyalardan foydalanish yoki promise funksiyalaridan foydalanish .

Promise funksiyalari orqali tekshirib ko`ramiz

unda birinchi bo`lib kodni yozib olamiz .

const fs = require("node:fs/promises");

// Execution Time: 9.3s
// Memory Usage: 50-60MB
(async () => {
  console.time("fs promise solution");
  try {
    const fileHandler = await fs.open("test.txt", "w");

    for (let i = 0; i < 1000000; i++) {
      await fileHandler.write(` ${i} `);
    }
  } catch (error) {
    console.error("An error occurred:", error);
  } finally {
    console.timeEnd("fs promise solution");
  }
})();

(Agarda IIFE haqida kop`roq bilmoqchi bo`lsangiz link ).

memory usage

ko`rib turganingizdek yuqoridagi ma’lumotni faylga yozishda promise solution uchun.

Execution Time: 9.3s.

Memory Usage: 50–60MB sarflandi

Callback funksiyalari orqali tekshirib ko`ramiz

const fs = require("node:fs");

// Execution Time: 1.4s
// Memory Usage: 1GB
(async () => {
  console.time("fs callback solution");
  fs.open("test.txt", "w", (err, fd) => {
    if (err) {
      console.error("Error opening file:", err);
      return;
    }
    for (let i = 0; i < 1000000; i++) {
      fs.write(fd, ` ${i} `, (err) => {
        if (err) {
          console.error("Error writing to file:", err);
        }
      });
    }
    console.timeEnd("fs callback solution");
  });
})();

callback versiyadagi yechimda 1.4 soniya va 1GB atrofida xotira sarflandi .

Execution Time: 1.4s.

Memory Usage: 1GB sarflandi

Agarda biz 1 million emas 10 million raqamni yozsak u holat xotira yetishmasligi tufayli pastdagi error kelib chiqadi.

Ko`rib turganingizdek callback holatida xotirada joy yetishmasligi tufayli process o`chirib yuborildi.

Sinxron funksiyalari orqali tekshirib ko`ramiz

const fs = require("node:fs");

// Execution Time: 1.8s
// Memory Usage: 40MB
(async () => {
  console.time("fs sync solution");
  fs.open("test.txt", "w", (err, fd) => {
    if (err) {
      console.error("Error opening file:", err);
      return;
    }

    for (let i = 0; i < 1000000; i++) {
      const buff = Buffer.from(` ${i} `, "utf-8");
      try {
        fs.writeSync(fd, buff);
      } catch (writeErr) {
        console.error("Error writing to file:", writeErr);
        return;
      }
    }

    console.timeEnd("fs sync solution");
  });
})();

Sinxron yechimda esa :

Execution Time: 1.8s.

Memory Usage: 40MB sarflandi.

Tepada 3 ta yechim ko`rdik bularning hammasini har xil yaxshi va yomon tomonlari bor .Agarda yoziladigan sonlarni oshirsak nima bo`ladi ?

Promise solution xotira deyarli o`zgarmay turadi ishlaydigan vaqt ko`payadi .
Callback solution xotira ko`payib xotira yetmasligi natijasida process to`xtatiladi.
Sinxron xotira o`zgarmasdan qoladi ammo vaqt ko`payadi (bu yerda faqat raqam yozilmoqda agar ko`proq vaqt talab qilinadigan task orqali sinxron yozilsa promise solutiondan ko`p vaqt ketadi).

Nodejs dokumentatsiya bo`yicha callback eng tez ishlaydigan usul va bu benchmark orqali ko`rindi ammo bir necha callbacklar xotiraga yozilishi tufayli xotiradan ancha joy olinadi .Ammo tepadagi benchmarkingdan nega sinxron usul promise yechimdan tezroq ishlayapti degan savol keladi chunki raqam yozish ko`p vaqtni olmaydigan bo`lganligi uchun sinxron promise ochib resolve qilish vaqti ketmaganligi evaziga vaqtdan yutmoqda.

Stream solutions

Endi oldingi maqolada aytganimizdek bu turdagi muammolar uchun eng to`g`ri yechim bu streamlar orqali .Biz solishtirish uchun raqamlarni bilib oldik endi stream orqali yechim yozishga o`tamiz.

const fs = require("node:fs/promises");

(async () => {
  try {
    console.time("naive stream solution");
    const fileHandle = await fs.open("test.txt", "w");

    const stream = fileHandle.createWriteStream();

    for (let i = 0; i < 1000000; i++) {
      const buff = Buffer.from(` ${i} `, "utf-8");
      stream.write(buff);
    }

    stream.end(); // Close the stream

    await new Promise((resolve, reject) => {
      stream.on("finish", resolve);
      stream.on("error", reject);
    });

    console.timeEnd("naive stream solution");
  } catch (error) {
    console.error("An error occurred:", error);
  }
})();

Execution Time: 220ms
Memory Usage: 240MB

Bu yechimda ko`rib turganingizdek vaqt ancha tezlashdi 6 marta atrofida ammo biroz ko`proq xotira evaziga .Bu oxirgi yechimmi ?

Yo’q bu oxirgi yechim emas bu yerda biz streamlarda bir narsaga amal qilmayapmiz yani buffer faqat to`lganidan keyingina uni faylga yozish .Bu yechimda har bitta raqam uchun har doim yozilmoqda va bu xotiradan joy olmoqda.

Note: Production da tepadagi yechim tavsiya etilmaydi

Yaxshiroq stream yechimi.

const fs = require("node:fs/promises");

// Execution Time: 245ms
// Memory Usage: 30MB
(async () => {
  console.time("better stream solution");
  try {
    const fileHandle = await fs.open("test.txt", "w");

    const stream = fileHandle.createWriteStream();

    let i = 0;

    const numberOfWrites = 1000000;

    const writeMany = () => {
      while (i < numberOfWrites) {
        const buff = Buffer.from(` ${i} `, "utf-8");

        if (i === numberOfWrites - 1) {
          return stream.end(buff);
        }

        i++;

        if (!stream.write(buff)) break;
      }
    };

    writeMany();

    stream.on("drain", () => {
      writeMany();
    });

    stream.on("finish", () => {
      fileHandle.close();
      console.timeEnd("better stream solution");
    });
  } catch (error) {
    console.error("An error occurred:", error);
  }
})();