Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Performance issue installing packages with large number of Folders #7080

Closed
2 tasks done
bigtimebuddy opened this issue Dec 13, 2023 · 5 comments
Closed
2 tasks done
Labels
Bug thing that needs fixing Needs Triage needs review for next steps Release 9.x work is associated with a specific npm 9 release

Comments

@bigtimebuddy
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

This issue exists in the latest npm version

  • I am using the latest npm

Current Behavior

We have a large package (~75K files) that is taking a long time to install (1-2 minutes). I was able to debug this issue more thoroughly and have discovered an interesting performance nuance for packages with a large number of folders. It seems that folders create ~10x slow-down installing than a comparative package of the same size but with a flat structure.

Expected Behavior

I would expect that a flat or nested file structure to have roughly the same performance profile when unpacking and installing.

Steps To Reproduce

I created two versions of a dummy package (@bigtimebuddy-dev/install-stress-test):

  • 1.2.0 - Contains about 60K junk files in a flat structure (i.e., ./dist/*.txt)
  • 1.3.0 - Contains about 60K junk files in nested folders (i.e., ./dist/*/*/*/*.txt)

Time results with v1.2.0

> time npm install @bigtimebuddy-dev/install-stress-test@1.2.0 
4.69s user 11.34s system 150% cpu 10.643 total

Time results with v1.3.0

> time npm install @bigtimebuddy-dev/install-stress-test@1.3.0
85.58s user 14.90s system 113% cpu 1:28.24 total

Full Reproduction Steps

mkdir consume-stress-test
cd consume-stress-test
npm init --yes
time npm install @bigtimebuddy-dev/install-stress-test@1.3.0

Environment

  • npm: 9.9.2
  • Node.js: 18.17.1
  • OS Name: MacOS 12.6
  • System Model Name: MacBook Pro (M1)
  • npm config:
; node bin location = /Users/********/.nvm/versions/node/v18.17.1/bin/node
; node version = v18.17.1
; npm local prefix = /Users/********/Desktop/consume-stress-test
; npm version = 9.9.2
; cwd = /Users/********/Desktop/consume-stress-test
; HOME = /Users/********
@bigtimebuddy bigtimebuddy added Bug thing that needs fixing Needs Triage needs review for next steps Release 9.x work is associated with a specific npm 9 release labels Dec 13, 2023
@kchindam-infy
Copy link

@bigtimebuddy U mentioned the issue on performance of installing packages with large number of folders. checked the issue what u mentioned with two different time result packages. installing package 1.2.0 it takes ~19s and for the 1.3.0 it takes ~57s both are less than 1mintute.

@kchindam-infy
Copy link

Closing this issue due to inactivity, feel free to open the issue if you need more info.

@bigtimebuddy
Copy link
Author

I feel this issue should remain open as it has not been addressed in anyway.

Also, this is not an issue that is seen with other package managers.

@kchindam-infy kchindam-infy reopened this Dec 30, 2024
@kchindam-infy
Copy link

@bigtimebuddy Hi installing the different packages with large no of folders takes the time less than a minute based on the specified environment. can u provide the detailed information to reproduce the issue

@kchindam-infy
Copy link

Closing the issue due to inactivity. Feel free to open the isssue if you need more information

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug thing that needs fixing Needs Triage needs review for next steps Release 9.x work is associated with a specific npm 9 release
Projects
None yet
Development

No branches or pull requests

2 participants
  NODES
COMMUNITY 2
Project 3
USERS 5